JP2012128578A - Portable terminal and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To synthesize a virtual object with an appropriate inclination relation even when the size and distortion of a marker cannot be correctly recognized due to disturbance or the like.
An image analysis unit 103 analyzes a marker in shooting data and recognizes a shooting direction with respect to the marker, and a sensor unit 104 detects a sensor value indicating a posture of the terminal. Then, the smoothing processing unit 107 performs a smoothing process on the shooting direction based on the sensor value based on the shooting direction of the marker and the shooting direction based on the sensor value. The virtual object position determination unit 109 corrects the inclination of the virtual object using the shooting direction based on the sensor value smoothed by the smoothing processing unit 107, and matches the virtual object whose inclination is corrected to the position of the marker. Composite to shooting data. The display unit 102 displays shooting data in which virtual objects are combined.
[Selection] Figure 1

Description

  The present invention relates to a portable terminal and an image processing method for combining and displaying image data such as a virtual object on captured image data.

  In recent years, services using AR (Augmented Reality) technology have been developed and provided. For example, a technique is known in which additional information associated with an object such as a marker in an image acquired by a camera is acquired, and the acquired additional information is combined with the object and displayed. In this technique, additional information called a so-called virtual object is stored in association with an object such as a marker in each user's mobile terminal. As a technique common to such a technique, a technique of detecting a palm (skin color part) and synthesizing image data such as a character in the part is known (see, for example, Patent Document 1).

JP 2006-262980 A

  However, when the marker is configured in a plane, it may appear distorted or tilted depending on the shooting direction. In such a case, the mobile terminal can determine that the marker was taken with a certain tilt method, and the virtual object is processed so as to have a tilt according to the tilt, and the processed virtual object is aligned with the marker position. To synthesize.

  For example, an example of a marker is shown in FIG. FIG. 13A shows an example in which the marker receives external light, and shows a state in which a part of the marker marked D in the black frame is missing due to external light. In this case, it may be determined that the marker is taken from an oblique direction.

  FIG. 13B shows an example in which a part of the marker is turned up by wind or the like when the marker is formed of a soft member such as paper. Even in this example, a part of the marker is missing. In this case as well, it may be determined that the marker was taken from an oblique direction.

  FIG. 13C shows an example in which the marker is shifted left and right due to camera shake. In this case as well, the marker may be distorted or its size may be mistaken. Depending on the degree of distortion and size misrecognition, it may be determined that the marker was taken from an oblique direction or taken from near (or far away). May have been.

  When it is determined that the marker is photographed from an oblique direction as described above, the virtual object is processed according to the inclination of the marker and synthesized as shown in FIG. FIG. 14A shows an example in which a marker and a virtual object are synthesized with a correct inclination relationship. FIG. 14B shows an example in which, when a part of the marker is missing due to, for example, external light, the composition processing is performed with the inclination of the virtual object shifted.

  In this way, there is a problem that the size and distortion of the marker cannot be recognized correctly as a result of the influence of camera shake and disturbance (external light, vibration of the image analysis object). As a result, there is a problem that a virtual object cannot be correctly combined with a marker.

  Therefore, in the present invention, in order to solve the above-described problem, even if the size and distortion of the marker cannot be correctly recognized due to disturbance or the like, the portable object can synthesize a virtual object with an appropriate inclination relationship. It is an object to provide a terminal and an image processing method.

  In order to solve the above-described problem, a portable terminal of the present invention includes a photographing unit that captures photographing data, and a recognition unit that analyzes a marker in the photographing data captured by the photographing unit and recognizes a photographing direction with respect to the marker. And a sensor value detecting means for detecting a sensor value indicating the attitude of the terminal, a photographing direction detected by the recognizing means, and a photographing direction based on the sensor value of the sensor value detecting means. A smoothing unit that performs a smoothing process on the shooting direction, a correction unit that corrects the tilt of the object using the shooting direction based on the sensor value smoothed by the smoothing unit, and a tilt that is corrected by the correction unit. Combining means for combining the corrected object with the shooting data according to the position of the marker, and shooting data in which the object is combined by the combining means And a display means for displaying.

  Further, the image processing method of the present invention includes a photographing step for capturing photographing data, a recognition step for analyzing a marker in the photographing data captured in the photographing step, recognizing a photographing direction with respect to the marker, and a posture of the terminal. In the imaging direction of the marker, based on the sensor value detection step for detecting the sensor value indicated, the imaging direction detected in the recognition step, and the imaging direction based on the sensor value detected in the sensor value detection step A smoothing step for performing a smoothing process, a correction step for correcting the tilt of the object using a shooting direction based on the sensor value smoothed by the smoothing step, and the tilt is corrected in the correction step. A synthesis step of synthesizing the object with the shooting data according to the position of the marker; and Object comprises a display step of displaying the captured data synthesized, the in formation step.

  According to the present invention, the marker in the shooting data is analyzed, the shooting direction with respect to the marker is recognized, and the sensor value indicating the attitude of the terminal is detected, and the recognized shooting direction of the marker and the sensor value are detected. The smoothing process is performed on the shooting direction based on the sensor value based on the shooting direction. Then, the inclination of the object is corrected using the shooting direction based on the smoothed sensor value, the object whose inclination is corrected is combined with the shooting data in accordance with the position of the marker, and the shooting data in which the object is combined is obtained. indicate. Accordingly, even when the inclination of the marker cannot be correctly recognized due to external light or the like, the inclination of the object to be combined can be determined according to the inclination of the marker and can be combined.

  Further, in the portable terminal of the present invention, the smoothing means changes to a transition state in each detection history of the shooting direction of the marker recognized by the recognition means and the shooting direction based on the sensor value of the sensor value detection means. Based on this, it is preferable to perform a smoothing process.

  According to the present invention, the smoothing process can be performed with high accuracy by performing the smoothing process based on the transition state in each detection history of the shooting direction of the marker and the shooting direction based on the sensor value.

  In the portable terminal of the present invention, the smoothing means may change the transition state in the detection history of the sensor value detected by the sensor value detection means in the detection history of the shooting direction of the marker recognized by the recognition means. If they do not match the state, it is preferable to perform a smoothing process.

  According to the present invention, when the transition state in the sensor value detection history does not match the transition state in the shooting direction detection history based on marker recognition, smoothing processing is performed. As a result, it can be determined that there is an erroneous recognition in the marker recognition, and the error can be reduced by the smoothing process and reflected in the inclination correction of the object.

  Further, in the mobile terminal of the present invention, the smoothing parameter that increases as the difference between the shooting direction based on the sensor value detected by the sensor value detection unit and the shooting direction based on the analysis result of the recognition unit increases. It is preferable to perform a smoothing process using.

  According to the present invention, the smoothing process is performed using the smoothing parameter that increases as the difference between the shooting direction based on the detected sensor value and the shooting direction based on the analysis result increases. Smoothing processing can be performed, and the tilt error due to the marker recognition error can be corrected more appropriately.

  In addition, the mobile terminal of the present invention analyzes an imaging unit that captures imaging data, a marker in the imaging data captured by the imaging unit, and an analysis object larger than the marker, and the marker and the analysis object Recognizing means for recognizing the photographing direction with respect to each, smoothing means for performing a smoothing process on the photographing direction of the marker based on each of the marker recognized by the recognizing means and the photographing direction of the analysis object; A correction unit that corrects the inclination of an object using a shooting direction based on the marker smoothed by the smoothing unit, and an object whose inclination is corrected by the correction unit is combined with shooting data according to the position of the marker. Synthesizing means, and display means for displaying photographing data in which the object is synthesized by the synthesizing means. .

  Further, the image processing method of the present invention includes a photographing step for capturing photographing data, a marker in the photographing data captured in the photographing step and an analysis object larger than the marker, and the marker and the analysis object A recognition step for recognizing the shooting direction with respect to each of the above, a smoothing step for performing a smoothing process on the shooting direction of the marker based on each of the marker recognized in the recognition step and the shooting direction of the analysis object, and A correction step of correcting the tilt of the object using the shooting direction based on the marker smoothed in the smoothing step, and the object whose tilt is corrected in the correction step according to the position of the marker in the shooting data A synthesis step to synthesize, and an object in the synthesis step It comprises a display step of displaying the combined shot data.

  According to the present invention, the marker in the imaging data and the analysis object larger than the marker are analyzed, the imaging direction with respect to each of the marker and the analysis object is recognized, and the recognized marker and the imaging direction of the analysis object are recognized. Based on each of the above, smoothing processing is performed on the shooting direction of the marker, and the tilt of the object is corrected using the shooting direction based on the smoothed marker. Then, the object whose inclination is corrected can be combined with the shooting data in accordance with the position of the marker, and the shooting data with the object combined can be displayed.

  Accordingly, even when the inclination of the marker cannot be correctly recognized due to external light or the like, the inclination of the object to be combined can be determined according to the inclination of the marker and can be combined. Furthermore, it is not necessary to provide a device such as a sensor, and the configuration can be simplified.

  According to the present invention, even when the inclination of the marker cannot be correctly recognized due to external light or the like, the inclination of the object to be combined can be determined according to the inclination of the marker and can be combined.

It is a block diagram which shows the function of the portable terminal 100 of this embodiment. 2 is a hardware configuration diagram of the mobile terminal 100. FIG. It is explanatory drawing which shows the pattern in the case of misrecognizing. 10 is a graph showing the analysis result of the marker analyzed by the image analysis unit 103. 6 is a graph showing a transition in a predetermined time unit of an imaging direction detected by a sensor unit 104. 6 is an explanatory diagram illustrating a specific example of comparison processing in a comparison unit 105. FIG. 5 is a flowchart showing processing of a comparison unit 105. It is a figure which shows the database of the smoothing parameter DB part. It is a figure which shows the management table which memorize | stores a virtual object. It is explanatory drawing which showed typically a mode that it combined with the inclination of the virtual object according to the recognition result of a marker. 5 is a flowchart showing processing of the mobile terminal 100. It is explanatory drawing which showed the imaging | photography data in the modification. It is explanatory drawing which shows the recognition example of a marker. It is the schematic diagram which represented typically the synthetic | combination process of a marker and a virtual object.

  Embodiments of the present invention will be described with reference to the accompanying drawings. Where possible, the same parts are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a block diagram illustrating functions of the mobile terminal 100 according to the present embodiment. As shown in FIG. 1, the mobile terminal 100 includes an imaging unit 101 (imaging unit), a display unit 102 (display unit), an image analysis unit 103 (recognition unit), a sensor unit 104 (sensor value detection unit), a comparison unit 105, A smoothing parameter DB unit 106, a smoothing processing unit 107 (smoothing unit), a virtual object storage unit 108, and a virtual object position determination unit 109 (correction unit, synthesis unit) are included. The portable terminal 100 is configured by hardware shown in FIG.

  FIG. 2 is a hardware configuration diagram of the mobile terminal 100. As shown in FIG. 2, the portable terminal 100 shown in FIG. 1 physically includes a CPU 11, a RAM 12 and a ROM 13 that are main storage devices, an input device 14 such as a keyboard and a mouse that are input devices, and an output such as a display. The computer 15 includes a device 15, a communication module 16 that is a data transmission / reception device such as a network card, an auxiliary storage device 17 such as a hard disk, and the like. Each function described in FIG. 1 has an input device 14, an output device 15, and a communication module 16 under the control of the CPU 11 by reading predetermined computer software on hardware such as the CPU 11 and the RAM 12 shown in FIG. 2. This is realized by reading and writing data in the RAM 12 and the auxiliary storage device 17. Hereinafter, each functional block will be described based on the functional blocks shown in FIG.

  The photographing unit 101 is a part that captures photographing data photographed by a camera or the like.

  The display unit 102 is a part that displays the shooting data captured by the shooting unit 101.

  The image analysis unit 103 is a part that performs image analysis on the shooting data captured by the shooting unit 101. Specifically, the image analysis unit 103 recognizes the presence or absence of a marker in the captured data, and analyzes the recognized marker size and distortion amount. Information indicating the size of the marker to be analyzed, the appearance of the marker (an overview when viewed from the front), and the like are stored in advance in a storage unit (not shown). By using these pieces of information, the image analysis unit 103 can grasp the distance to the marker, the degree of distortion of the marker, and the like, and can grasp from which direction the marker was photographed. This image analysis method uses a general technical method. A virtual object position determination unit 109, which will be described later, can determine the size / tilt (posture) of the virtual object based on the shooting direction and shooting distance of the marker.

  Further, the image analysis unit 103 may not be able to correctly recognize the marker. For example, marker distortion and size may be erroneously recognized due to external light or camera shake. FIG. 3 shows a pattern for erroneous recognition. As shown in FIG. 3, there is a state in which a part of the marker becomes white (so-called white blur) due to external light or the like. For example, as shown in FIG. 3A, when a part of the upper left corner of the marker is white, the image analysis unit 103 erroneously recognizes that the marker is viewed obliquely from about 15 degrees to the right. May end up. That is, the marker is erroneously recognized as shown in FIG.

  In addition, as shown in FIG. 3B, when most of the left side of the marker is white, the image analysis unit 103 mistakenly assumes that the marker is viewed obliquely from about 20 degrees to the left. It may be recognized. That is, the marker is erroneously recognized as shown in FIG.

  In order to prevent this, the image analysis unit 103 stores analysis result information in units of a predetermined time as information for smoothing processing as shown in FIG. FIG. 4 is a graph showing the analysis result (imaging direction) of the marker analyzed by the image analysis unit 103. In the example illustrated in FIG. 4, the analysis result is obtained when the mobile terminal 100 images the marker at an angle of 45 degrees above the front. As shown in FIG. 4, by analyzing the image of the marker in the shooting data, it can be recognized that the shooting was performed from an angle of approximately 45 degrees, and thus the shooting direction can be recognized as 45 degrees. Then, this shooting direction is stored in predetermined time units. FIG. 4 shows a state of shaking up and down around 45 degrees due to camera shake and the like. The photographing direction stored here is used as information for determining whether or not smoothing processing is necessary in the comparison unit 105 described later. In FIG. 4, the shooting angle is used as the shooting direction as the analysis result. However, the present invention is not limited to this, and the shooting direction or the distance to the marker may be used, or a combination of these may be used. Good. Thereby, it can be set as a highly accurate analysis result.

  The sensor unit 104 includes an angle sensor, a gyro sensor, an acceleration sensor, a geomagnetic sensor, a position sensor (GPS module), and the like. As information for grasping the shooting direction with respect to the marker of the shooting unit 101, a sensor is used. A value is acquired every predetermined time, a shooting direction is calculated based on the sensor value, and this shooting direction is stored in time series. FIG. 5 is a graph showing the transition of the shooting direction detected by the sensor unit 104 in a predetermined time unit. In this manner, sensor values are sequentially acquired and stored every predetermined time. The photographing direction stored here is used as information for determining whether or not smoothing processing is necessary in the comparison unit 105 described later. The sensor unit 104 may detect the shooting direction as the shooting direction in addition to the shooting angle, and store it in time series.

  The comparison unit 105 is a part that compares the analysis result (shooting direction) acquired and stored in the image analysis unit 103 with the shooting direction calculated based on the sensor value acquired and stored in the sensor unit 104. . In this comparison unit 105, the analysis result of the image analysis unit 103 and the sensor value of the sensor unit 104 are compared in the same time zone, and if both match, it is determined that the image analysis result is correct, If they do not match, it is determined that the image analysis result is incorrect.

  FIG. 6 is an explanatory diagram illustrating a specific example of comparison processing in the comparison unit 105. 6A shows a graph of the analysis result of the image analysis unit 103, and FIG. 6B shows a graph of the sensor value of the sensor unit 104. The comparison unit 105 can determine that the portion indicated by the reference symbol X1 in FIG. 6A and the portion indicated by the reference symbol X2 in FIG. Therefore, it can be determined that the analysis result is correct for this portion (time zone). Moreover, the part shown with the code | symbol Y1 in Fig.6 (a) and the part shown with the code | symbol Y2 in FIG.6 (b) do not correspond. Therefore, it can be determined that the analysis result is not correct for this portion (time zone). In this embodiment, a plurality of shooting directions in a time zone having a certain width are used, but one shooting direction in a certain time may be used.

  Details of the processing of the comparison unit 105 will be further described. FIG. 7 is a flowchart showing the processing of the comparison unit 105. First, the analysis result of the image analysis unit 103 is acquired by the comparison unit 105 (S101). Then, in the analysis result of the image analysis unit 103, it is determined whether or not there is a change from the previous result (S102). That is, it is determined whether or not there has been a change in the shooting direction based on the recently acquired analysis result and the shooting direction based on the currently acquired analysis result. Note that one shooting direction in a certain time may be compared, or a plurality of shooting directions in a certain time zone may be compared.

  Then, when it is determined that there is a change, the shooting direction based on the sensor value of the sensor unit 104 and the shooting direction based on the analysis result of the image analysis unit 103 in the same time zone (or the same time) are obtained. They are compared (S103). Here, if it is determined that the shooting directions match, it is determined as a comparison result pattern (1) (S105). If it is determined that the shooting directions do not match, it is determined as a comparison result pattern (2) (S106).

  If it is determined in S102 that there is no change in the shooting direction, the shooting direction based on the sensor value of the sensor unit 104 and the shooting direction based on the analysis result of the image analysis unit 103 at the same time are compared. (S104). If it is determined that the shooting directions coincide with each other, the comparison result pattern (3) is determined (S107). If it is determined that the shooting directions do not match, it is determined as a comparison result pattern (4) (S108).

  In this way, the comparison unit 105 can obtain a comparison result pattern, and the smoothing processing unit 107 performs a smoothing process using a smoothing parameter determined in advance based on the comparison result pattern.

  The smoothing parameter DB unit 106 is a part that stores the smoothing parameter associated with each comparison result pattern, and stores, for example, the database shown in FIG. FIG. 8 is a diagram illustrating a database of the smoothing parameter DB unit 106.

  As shown in FIG. 8, the weighting coefficient A (t) and the retroactive value N are stored for each comparison result pattern. The weighting coefficient A (t) and the retroactive place N will be described later. In the pattern (4), it is preferable not to perform the smoothing process for the following reason. That is, there are the following cases as pattern (4). First, in the first case, the portable terminal 100 is fixed and held with respect to the marker. As a result, there is no change in the value (imaging direction) calculated by image analysis, but an error may occur in the sensor unit due to the influence of disturbance. Accordingly, there is no change in the calculation result by the image analysis, but the result obtained by the sensor unit 104 is inconsistent.

  As a second case, the mobile terminal 100 is moved with respect to the marker. Originally, the value obtained by the image analysis result changes, but it may be affected by illumination or the like simultaneously with the movement of the marker. In that case, as a result, there may be a case where there is no change in the value obtained by the image analysis. On the other hand, since the mobile terminal 100 is moving, the value of the sensor changes. Accordingly, there is no change in the calculation result by the image analysis, but the result obtained by the sensor unit 104 is inconsistent.

  Considering these cases, it can be determined that there is no significant change in the image analysis result due to disturbance such as illumination. Therefore, only the case where the marker is fixed to the mobile terminal 100 is practically possible, and it can be determined that the sensor unit 104 has an error. Therefore, it is concluded that smoothing processing is unnecessary.

  Further, the magnitude of the smoothing parameter may be changed according to the difference between the shooting direction based on the sensor value of the sensor unit 104 and the shooting direction based on the analysis result of the image analysis unit 103. That is, as the difference is larger, it can be determined that the reliability of the shooting direction calculated from the image analysis result is lower (the error is larger). Therefore, in order to reduce the error, the influence of the shooting direction obtained in the past is increased by setting the weighting coefficient A (t) and the retroactive value N large.

ここでは、ある時間tの撮影方向をf(t)とし、ある時間tの平滑化した撮影方向をF(t)とする。また、どこまで遡ってセンサ値を利用するかを示す値をnとする。例えば、ある時間tに、前回の撮影方向を利用する場合はn=1となる。遡った撮影方向に対する重み付け係数をa(n)とする。ｉは、ｔを基準にして、遡った過去の撮影方向を示す。そして、F(t)の値から仮想オブジェクトをディスプレイ部１０２に表示するための傾き（姿勢）をX(t)とする。 The smoothing processing unit 107 uses the smoothing parameters (the weighting coefficient A (t) and the retroactive value N) determined based on the comparison result in the comparison unit 105, and based on the analysis result of the image analysis unit 103. This is the part that performs the smoothing process in the shooting direction. Specifically, the following arithmetic processing (1) is performed.

Here, the shooting direction at a certain time t is f (t), and the smoothed shooting direction at a certain time t is F (t). A value indicating how far the sensor value is used is assumed to be n. For example, when using the previous shooting direction at a certain time t, n = 1. The weighting coefficient for the retrospective shooting direction is a (n). i indicates a past shooting direction retroactive with reference to t. Then, the inclination (posture) for displaying the virtual object on the display unit 102 from the value of F (t) is assumed to be X (t).

In the smoothing parameter DB unit 106 in FIG. 8, 0.2 is set as the weighting coefficient A (t) and 1 is set as the retroactive value N in the pattern (2). And if this is applied to the above formula (1),
F (t) = f (t) + a (t-1) * f (t-1) = f (t) + 0.2 * f (t-1)
It becomes.

  The virtual object storage unit 108 is a part that stores virtual objects. FIG. 9 shows a management table for storing virtual objects. In this management table, the file name of the image data of the virtual object is stored in association with the marker information as a mark. Based on this management table, the virtual object position determination unit 109 can determine which marker the virtual object is to be synthesized in.

  The virtual object position determination unit 109 reads a virtual object corresponding to the marker included in the shooting data from the virtual object storage unit 108 and determines a position near the marker or a predetermined position based on the marker. Thus, the virtual object and the shooting data are combined so that the virtual object is arranged at the position. In this combining process, the inclination (posture) of the virtual object is corrected according to the shooting direction of the marker. A specific example will be described with reference to FIG. 10 schematically showing a state in which the object is synthesized with the inclination of the virtual object according to the marker recognition result. FIG. 10A shows the inclination of the virtual object when the image is taken from 45 degrees above the front of the marker. In this manner, the virtual object can be represented as viewed from above the front at an angle of 45 degrees in front of the virtual object according to the analysis result (angle information) analyzed by the image analysis unit 103.

  Similarly, FIG. 10B shows the inclination of the virtual object when the image is taken from an angle of 20 degrees above the front of the marker. In this case as well, the inclination of the virtual object can be corrected in accordance with the inclination of the marker.

  FIG. 10C shows the inclination of the virtual object when the image is taken from the diagonal left direction of the marker and obliquely above 45 degrees. In this case as well, the inclination of the virtual object can be corrected in accordance with the inclination of the marker.

  Furthermore, the virtual object position determination unit 109 corrects the inclination (posture) of the virtual object based on the recognition result smoothed by the smoothing processing unit 107, that is, according to the degree of marker distortion, and corrects the correction. The virtual object in the state is combined with the shooting data.

  The virtual object corrected in this way is combined with the shooting data and displayed on the display unit 102.

  The mobile terminal 100 configured as described above can correct the inclination of the virtual object based on the marker included in the shooting data, and can display the combined image with the shooting data.

  Next, processing of the mobile terminal 100 configured as described above will be described. FIG. 11 is a flowchart showing processing of the mobile terminal 100.

  First, the photographing unit 101 is activated and photographing data is captured (S201). Then, the marker included in the shooting data is analyzed by the image analysis unit 103, and the shooting direction is calculated based on the analysis result such as the size / distortion of the marker (S202). Further, a sensor value is acquired by the sensor unit 104 (S203).

  Then, the analysis result of the image analysis unit 103 and the sensor value of the sensor unit 104 are compared by the comparison unit 105 (S205). As a result of this comparison, when it is determined that the pattern is (2) (S205: Pattern (2)), the smoothing parameter DB unit 106 reads the weighting coefficient A (t) and the retroactive value N that are smoothing parameters. (S206).

  Then, the smoothing processing unit 107 performs a smoothing process on the analysis result obtained from the image analysis unit 103 using the weighting coefficient A (t) and the retroactive value N (S207). Based on the smoothed analysis result, the virtual object position determination unit 109 corrects the inclination (posture) of the virtual object and synthesizes the captured data so that the virtual object is arranged in the vicinity of the corresponding marker. Is done. Then, shooting data in which a virtual object having an appropriate posture is combined is displayed on the display unit 102 (S208).

  Note that it is preferable to reduce the power consumption if the timing of composing the virtual object composition position is changed depending on the pattern.

  For example, when it is determined that the comparison result pattern is the pattern (3) and the pattern (4), the analysis result of the image analysis unit 103 is not changed from the previous result, and is thus combined with the display unit 102. The change in the synthesized position of the virtual object to be performed is small. For this reason, when it is determined as the pattern (3) and the pattern (4), the time interval for updating the composite position of the virtual object on the display unit 102 may be increased.

  That is, a predetermined reference value T1 is set, and if it is determined in S205 that the pattern of the comparison result is the pattern (3) or (4), the comparison unit 105 sets the time updated last time before moving to S208. It is determined whether the time determined by the predetermined reference value T1 has elapsed, and if it has elapsed, the composition processing in S208 is performed. If it has not elapsed, the composition process is not performed.

  On the other hand, when the comparison result pattern is determined to be the pattern (1) and the pattern (2), the analysis result of the image analysis unit 103 is changed from the previous result, so that the virtual combined with the display unit 102 The change in the composition position of the object is large. For this reason, when it is determined as the pattern (1) or the pattern (2), the time interval for updating the composite position of the virtual object on the display unit 102 may be shortened. For example, if a predetermined reference value T2 (<T1) is set and it is determined in S205 that the pattern of the comparison result is pattern (1), the predetermined reference value is calculated from the time last updated before proceeding to S208. It is determined whether the time set in T2 has elapsed. If the time has elapsed, the composition processing in S208 is performed. If it has not elapsed, the composition process is not performed.

  Similarly, if it is determined in S205 that the pattern of the comparison result is the pattern (2), before the process of S206, the comparison unit 105 elapses the time determined by the predetermined reference value T2 from the previously updated time. If it has elapsed, the extraction process in S206 and the subsequent smoothing process in S207 are performed. If it has not elapsed, the extraction process and the smoothing process are not performed.

  In this way, it is possible to reduce power consumption by changing the time interval for updating the composite position of the virtual object. Note that the time for updating the composite position may be stored in the virtual object position determination unit 109 or may be stored in the comparison unit 105 as the time for comparison processing.

  By performing the smoothing process in this way, a virtual object having an appropriate inclination can be combined with the shooting data even if there is a recognition error for the marker.

  Next, a modified example of the mobile terminal 100 of the present embodiment will be described with reference to FIG. In the portable terminal 100 of the above-described embodiment, the sensor value of the sensor unit 104 is used when correcting the inclination of the virtual object according to the inclination of the marker. However, it is not limited to using the sensor value of the sensor unit 104. FIG. 12 is an explanatory diagram showing the state of the shooting data when an object such as a desk much larger than the marker is recognized. As shown in FIG. 12, an object to be analyzed such as a desk is recognized and its inclination is calculated, and the inclination of the virtual object is corrected using the calculated inclination of the object to be analyzed. The inclination with respect to the desk which is the object may be registered in advance, or the imaging direction with respect to the desk can be recognized by recognizing the entire desk or the edge of the desk. For this, a general method of image processing for recognizing an object may be used.

  Then, by comparing the shooting direction with respect to the object and the shooting direction with respect to the marker, it is possible to determine whether the analysis result with respect to the marker is correct or not, and based on the determination, it is possible to determine whether smoothing processing is necessary. As a result, the sensor unit 104 is not required and the configuration is simple.

  Next, functions and effects of the mobile terminal 100 of the present embodiment will be described. In the portable terminal 100 of the present embodiment, the image analysis unit 103 analyzes a marker in the shooting data and recognizes the shooting direction with respect to the marker, while the sensor unit 104 detects a sensor value indicating the attitude of the terminal. Then, the smoothing processing unit 107 sets the imaging direction based on the sensor value based on the imaging direction of the marker recognized by the image analysis unit 103 and the imaging direction based on the sensor value detected by the sensor unit 104. A smoothing process is performed on the image. The virtual object position determination unit 109 reads the virtual object from the virtual object storage unit 108, corrects the inclination of the virtual object using the shooting direction based on the sensor value smoothed by the smoothing processing unit 107, and corrects the inclination. The obtained virtual object is combined with the shooting data according to the position of the marker. The display unit 102 displays shooting data in which virtual objects are combined. Thereby, even when the inclination of the marker cannot be correctly recognized due to external light or the like, the inclination of the virtual object to be combined can be determined according to the inclination of the marker and can be combined. Note that although the shooting direction has been described here as an example, the present invention is not limited to this, and the shooting direction and the shooting distance to the marker may be combined.

  In the mobile terminal 100 of the present embodiment, the image analysis unit 103 stores the shooting direction of the recognized marker in time series, and the sensor unit 104 displays the shooting direction calculated based on the detected sensor value. Remember in time series. And the smoothing process part 107 can perform a smoothing process accurately by performing a smoothing process based on the transition state in each detection history of the imaging | photography direction of a marker and the imaging | photography direction based on a sensor value. it can.

  In this portable terminal 100, the larger the difference between the imaging direction of the marker recognized by the image analysis unit 103 and the imaging direction based on the sensor value, the larger the smoothing parameter (for example, the weighting factor A (t) and Using the retroactive value N), the smoothing processor 107 can perform a smoothing process. Therefore, the inclination of the virtual object can be set to the actual inclination of the marker.

  Further, in the mobile terminal 100, the smoothing processing unit 107 performs a smoothing process using a smoothing parameter that increases in accordance with the amount of change in the transition state in the detection history of the sensor value detected by the sensor unit 104. Thus, an appropriate smoothing process can be performed, and a tilt error due to a marker recognition error can be corrected more appropriately.

  Further, in the mobile terminal 100 according to the modification, the image analysis unit 103 analyzes a marker and an analysis target (for example, a desk) larger than the marker in the shooting data, and shoots each of the marker and the analysis target. Recognize direction. Then, the smoothing processing unit 107 performs a smoothing process on the shooting direction of the marker based on each of the marker recognized by the image analysis unit 103 and the shooting direction of the analysis target. The virtual object position determination unit 109 reads the virtual object from the virtual object storage unit 108 and corrects the inclination of the virtual object using the smoothed shooting direction of the marker. Then, the virtual object whose inclination is corrected can be combined with the shooting data in accordance with the position of the marker, and the shooting data with the combined virtual object can be displayed.

  Thereby, even when the inclination of the marker cannot be correctly recognized due to external light or the like, the inclination of the virtual object to be combined can be determined according to the inclination of the marker and can be combined. Furthermore, it is not necessary to provide a device such as a sensor, and the configuration can be simplified.

DESCRIPTION OF SYMBOLS 100 ... Portable terminal, 101 ... Imaging | photography part, 102 ... Display part, 103 ... Image analysis part, 104 ... Sensor part, 105 ... Comparison part, 106 ... Smoothing parameter DB part, 107 ... Smoothing process part, 108 ... Virtual object Storage unit 109... Virtual object position determination unit.

Claims (7)

Photographing means for capturing photographing data;
Recognizing means for analyzing a marker in photographing data captured by the photographing means and recognizing a photographing direction with respect to the marker;
Sensor value detection means for detecting a sensor value indicating the attitude of the terminal;
Smoothing means for performing a smoothing process on the shooting direction of the marker based on the shooting direction detected by the recognition means and the shooting direction based on the sensor value of the sensor value detection means;
Correction means for correcting the inclination of the object using the shooting direction based on the sensor value smoothed by the smoothing means;
A synthesizing unit that synthesizes the object whose inclination is corrected by the correcting unit with the shooting data according to the position of the marker;
Display means for displaying shooting data in which the object is synthesized by the synthesis means;
A mobile terminal comprising: The smoothing means includes
The smoothing process is performed based on a transition state in each detection history of the imaging direction of the marker recognized by the recognition unit and the imaging direction based on the sensor value of the sensor value detection unit. The mobile terminal according to 1. The smoothing means includes
When the transition state in the detection history of the sensor value detected by the sensor value detection means does not match the transition state in the detection history of the shooting direction of the marker recognized by the recognition means, smoothing processing is performed. The portable terminal according to any one of claims 1 to 3, wherein   Smoothing processing is performed using a smoothing parameter that increases as the difference between the shooting direction based on the sensor value detected by the sensor value detection unit and the shooting direction based on the analysis result of the recognition unit increases. The portable terminal according to any one of claims 1 to 3, wherein Photographing means for capturing photographing data;
Recognizing means for analyzing a marker in the photographing data captured by the photographing means and an analysis object larger than the marker, and recognizing a photographing direction for each of the marker and the analysis object;
Smoothing means for performing a smoothing process on the imaging direction of the marker based on each of the imaging direction of the marker and the analysis object recognized by the recognition means;
Correction means for correcting the tilt of the object using the shooting direction based on the marker smoothed by the smoothing means;
A synthesizing unit that synthesizes the object whose inclination is corrected by the correcting unit with the shooting data according to the position of the marker;
Display means for displaying shooting data in which the object is synthesized by the synthesis means;
A mobile terminal comprising: Shooting steps to capture shooting data,
Recognizing a marker in the shooting data captured in the shooting step and recognizing a shooting direction with respect to the marker;
A sensor value detection step for detecting a sensor value indicating the attitude of the terminal;
A smoothing step of performing a smoothing process on the shooting direction of the marker based on the shooting direction detected in the recognition step and the shooting direction based on the sensor value detected in the sensor value detection step;
A correction step of correcting the inclination of the object using a shooting direction based on the sensor value smoothed by the smoothing step;
A synthesis step of synthesizing the object whose inclination is corrected in the correction step with the shooting data according to the position of the marker;
A display step for displaying shooting data in which the object is combined in the combining step;
An image processing method comprising: Shooting steps to capture shooting data,
A recognition step of analyzing a marker in the shooting data captured in the shooting step and an analysis object larger than the marker, and recognizing a shooting direction with respect to each of the marker and the analysis target;
A smoothing step of performing a smoothing process on the imaging direction of the marker based on each of the marker recognized in the recognition step and the imaging direction of the analysis object;
A correction step of correcting the inclination of the object using a shooting direction based on the marker smoothed in the smoothing step;
A synthesis step of synthesizing the object whose inclination is corrected in the correction step with the shooting data according to the position of the marker;
A display step for displaying shooting data in which the object is combined in the combining step;
An image processing method comprising:

Images