JP2006271658A - Program, information storage medium, and image pickup display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a new interest in imaging using image recognition processing.
In a portable game apparatus 1000, when a camera unit 7 is activated, a finder image is displayed on an image area AR1 and a detection result (ghost reaction) for a ghost by a ghost sensor is displayed on a display 1. . That is, the strength of the ghost reaction is determined based on the similarity between the feature data of the finder image and the feature data previously associated with each ghost, and the determined ghost response strength is displayed on the gauge MT as a meter. At the same time, a message corresponding to the gauge value MN is displayed in the message area AR2. When the shutter is turned on in a state where the ghost reaction is strong (for example, a state where the gauge value is “75” or more), the ghost GS1 corresponding to the subject OB1 is synthesized and displayed on the captured image.
[Selection] Figure 2

Description

  The present invention allows a computer to control the display of a finder image captured by an imaging means on a display device, and to control the display by combining a captured image with a captured image captured by a shutter operation. The present invention relates to a program for performing display control as if a character exists in space.

An image recognition technique for recognizing the content of a captured image by performing image recognition on the captured image is used in various fields. For example, as an application example of this image recognition technology, Patent Document 1 discloses a game device that performs a card game and a fortune-telling device that performs tarot fortune-telling. In such an apparatus, various kinds of cards such as playing cards and tarot cards and recognition objects such as dice are imaged using a camera, and the arrangement and the like are recognized by pattern recognition processing for performing image recognition on the captured image.
JP 2002-49909 A

  The technology of Patent Document 1 described above assumes that the shape and content of the recognition target are predetermined, but the recognition target is not particularly limited, and a new interest in imaging is realized by using image recognition processing. Isn't it possible? The present invention has been made in view of the above circumstances.

The first invention for solving the above-described problems is
A finder image taken by an imaging means (for example, the camera unit 7 of FIG. 1 and the imaging unit 200 of FIG. 5) is displayed on a display device (for example, the display 1 of FIG. 1, the image display unit 430 of FIG. 5). A program (e.g., a program for performing display control as if a character exists in real space by performing display control by combining the image of the character with the captured image captured by the shutter operation and performing display control) , A game program 510) of FIG.
Of the composite image conditions for each character (for example, feature data 524b of the ghost information 524 in FIG. 8) that are image conditions for determining whether or not to synthesize a character image with a captured image, Condition determining means for determining whether or not the viewfinder image satisfies any composite image condition (for example, the image recognition unit 312, the similarity calculation unit 314, and the ghost reaction determination unit 316 in FIG. 5; step S 1 in FIG. 14) Steps ST1 to T15 in FIG.
Notification control means (for example, display control unit 322 in FIG. 5; step T17 in FIG. 15) that performs a predetermined notification process as to whether or not a character exists based on the determination result of the condition determination means,
When a shutter operation is performed in a state in which a notification process that the character is present is performed by the notification control unit, a composite display control unit that combines and controls display of the character image with the captured image captured by the shutter operation (For example, display control unit 322 in FIG. 5; steps S7 in FIG. 14: YES to S13),
As a program for causing the computer to function.

The eleventh invention
Imaging means (for example, the camera unit 7 in FIG. 1 and the imaging unit 200 in FIG. 5) and display means for displaying the finder image taken by the imaging means (for example, the display 1 in FIG. 1 and the image display in FIG. 5). Unit 430), and by combining the captured image captured by the shutter operation and capturing the image of the character and causing the display unit to perform display control, display control as if the character exists in real space is performed. An image pickup display device to be performed (for example, the portable game device 1000 in FIGS. 1 and 5),
Whether the finder image satisfies any of the composite image conditions among the composite image conditions for each character, which is an image condition for determining whether to synthesize the character image with the captured image. Condition determination means for determining whether or not
Based on the determination result of the condition determination means, notification control means for performing a predetermined notification processing of whether or not a character exists;
When a shutter operation is performed in a state in which a notification process that the character is present is performed by the notification control unit, a composite display control unit that combines and controls display of the character image with the captured image captured by the shutter operation When,
It is an image pick-up display device provided with.

  According to the first or eleventh invention, it is determined whether or not the finder image satisfies any one of the composite image conditions for each character, and the character exists based on the determination result. A predetermined notification process for determining whether or not to perform is performed. Then, when a shutter operation is performed in a state where a notification process indicating that a character exists is performed, the character image is combined with the captured image captured by the shutter operation, and display control is performed.

  That is, when a finder image is displayed, whether or not a character exists is informed based on the finder image, and if a shutter operation is performed in a state in which the character is informed, the character is displayed in the captured image. It is possible to realize a new interest in imaging as if a character that exists in real space is captured, such as being displayed in a composite manner. At this time, if the finder images are substantially the same, the determination results as to whether or not the finder images satisfy any of the composite image conditions are the same, so the same notification is given. For this reason, an effect as if the character exists at a certain fixed place (subject position) is realized. In addition, the effect is as if the character present in the surroundings is detected by the notification performed based on the determination result of whether or not the finder image satisfies each of the composite image conditions, and the detection reaction is notified. Is realized.

The second invention is the program of the first invention, wherein
The composite display control means corresponds to a composite image condition that is determined to be satisfied by the condition determination means when a shutter operation is performed in a state in which the notification processing that the character is present is performed by the notification control means. This is a program for causing the computer to function so as to display and control an image of a character to be synthesized with a captured image captured by the shutter operation.

  According to the second aspect of the present invention, when the shutter operation is performed in a state where the notification process that the character exists is performed, the character image corresponding to the composite image condition that has been determined to satisfy the finder image is obtained. Then, display control is performed by synthesizing the captured image captured by the shutter operation.

  That is, if the finder images are substantially the same, the composite image conditions for determining that the finder image is satisfied are the same. Therefore, when the shutter operation is performed in this state, the same character image is always combined and displayed on the captured image. Is done. Therefore, when the same subject is imaged, an effect as if the character existing for each subject is determined, such as always displaying the same character, is realized.

The third invention is the program of the first or second invention,
Each of the composite image conditions includes a plurality of image conditions,
Condition satisfaction determination that the condition determination means determines a degree of satisfaction that the finder image satisfies each of the combined image conditions based on whether or not each image condition included in the combined image condition is satisfied. Means (for example, similarity calculation unit 314, ghost reaction determination unit 316 in FIG. 5; steps T1 to T3 in FIG. 15),
The notification control means notifies the notification content according to the satisfaction degree determined by the condition satisfaction degree determination means,
Is a program for causing the computer to function.

  According to the third aspect of the invention, the degree of satisfaction that the finder image satisfies each composite image condition is determined based on whether or not each image condition included in the composite image condition is satisfied. The notification content corresponding to the degree of satisfaction is notified. Therefore, the satisfaction level is expressed in stages, and notification according to the degree of the sensory response to the character is realized. In this case, for example, the more the image conditions that the finder image satisfies, the higher the degree of satisfaction, and thus a sensing reaction that is easy to understand intuitively.

A fourth invention is the program of the third invention, wherein
The notification control means is a program for causing the computer to function so as to perform notification processing by changing notification contents over time when the viewfinder image does not change.

  According to the fourth aspect, when the finder image does not change, the notification content is changed with the passage of time, and the notification process is performed. When the viewfinder image does not change, the determination result as to whether or not the viewfinder image satisfies any composite image condition does not change, and thus the notification content does not change. Therefore, in such a case, it is possible to realize an effect as if the moving character is sensed by changing the notification contents with the passage of time.

A fifth invention is a program according to any one of the first to fourth inventions,
The condition determining means is
A feature extraction unit (for example, the image recognition unit 312 in FIG. 5; step S1 in FIG. 14) that performs a feature extraction process for extracting a feature portion of the finder image;
Feature part determination means for determining whether or not the image of the feature part extracted by the feature extraction means satisfies any composite image condition (for example, the similarity calculation unit 314 and the ghost reaction determination unit 316 in FIG. 5). Steps T1 to T15 in FIG.
A program for causing the computer to function so as to have

  According to the fifth aspect, the feature portion of the finder image is extracted, and it is determined whether or not the extracted feature portion image satisfies any composite image condition. That is, the character to be displayed in combination is determined based on the characteristic part of the finder image.

A sixth invention is the program of the fifth invention, wherein
Each composite image condition includes at least a shape condition of the object as an image condition,
The feature extraction means extracts the shape of an object shown in the viewfinder image as a feature part,
The feature portion determination means determines whether or not the object shape extracted by the feature extraction means satisfies an object shape condition included in a composite image condition;
Is a program for causing the computer to function.

  According to the sixth aspect, the shape of the object shown in the finder image is extracted as the feature portion, and it is determined whether or not the extracted object shape satisfies the object shape condition included in the composite image condition Is done. That is, at least the character to be displayed in combination is determined based on the shape of the imaged object.

The seventh invention is the program of the fifth or sixth invention,
The notification control unit is a program for causing the computer to function so as to perform different notification processing depending on the size and / or position of the feature portion extracted by the feature extraction unit in the viewfinder image.

  According to the seventh aspect, different notification processes are performed depending on the size and / or position of the extracted feature portion in the finder image. Therefore, for example, a notification indicating that the degree of the sensing reaction to the character is stronger as the size of the extracted feature portion in the finder image is larger or the position in the finder image is closer to the center of the image. Intuitively understandable notifications such that the position of the subject, which is the characteristic part, coincides with the imaging direction, that is, the position of the subject is in front, or the closer the subject is, the stronger the sensing response to the character. it can.

An eighth invention is the program of the first to seventh inventions,
The notification control means is a program for causing the computer to function so as to have gauge display control means for performing predetermined gauge display as the notification processing.

  According to the eighth aspect of the invention, a predetermined gauge display is performed as a notification process that a character exists.

A ninth invention is a program according to any one of the first to eighth inventions,
A plurality of virtually wearable character sensors that detect whether or not a character exists in the imaging direction of the imaging means are set in advance according to the detectable character,
A mounting sensor selection means for selecting a character sensor to be virtually mounted among the plurality of character sensors according to a user's selection operation;
Function the computer as
The condition determination means sets a composite image condition corresponding to a character that can be detected by the character sensor selected by the wearing sensor selection means from the plurality of composite image conditions as a determination target condition.
Is a program for causing the computer to function.

  According to the ninth aspect of the present invention, it is assumed that, among a plurality of character sensors that can be virtually mounted and that detect whether or not a character is present in the imaging direction of the imaging means, which is set in advance, it is virtually mounted. A character sensor is selected according to a user's selection operation, and a composite image condition corresponding to a character that can be sensed by the selected character sensor is set as a condition for determining whether or not the finder image is satisfied.

  That is, if the character sensor to be virtually mounted is different, the condition for determining whether or not the finder image is satisfied is different. For this reason, even with the same finder image, the contents of the notification as to whether or not a character exists are different, and an interesting effect is realized in which the displayed character is different even when the same subject is imaged. The

  A tenth aspect of the invention is a computer-readable information storage medium (for example, the storage unit 500 in FIG. 5) storing the program of any one of the first to ninth aspects.

  Here, the “information storage medium” is a storage medium such as a hard disk, MO, CD-ROM, DVD, memory card, or IC memory that can be read by a computer. Therefore, according to the tenth aspect, by causing the computer to read the information stored in the information storage medium and executing the arithmetic processing, the same effect as any of the first to ninth aspects can be obtained. Can play.

  According to the present invention, when a finder image is displayed, whether or not a character exists is informed based on the finder image, and imaging is performed when a shutter operation is performed in a state in which the presence of the character is informed. It is possible to realize a new interest in imaging as if an image of a character existing in real space is captured, such as a character being synthesized and displayed on an image. In addition, at this time, an effect as if the surrounding character is sensed and this sensing reaction is notified is realized, and an effect as if the character exists in a certain place (subject position) Is realized.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. In the following, a case where the present invention is applied to a portable game device will be described. However, embodiments to which the present invention can be applied are not limited thereto.

[Appearance of game device]
FIG. 1 is an external view showing an example of a portable game apparatus 1000 in the present embodiment. FIG. 4A shows a front view, and FIG. 4B shows a rear view. As shown in the figure, the portable game apparatus 1000 includes a display 1, a speaker 3, and operation keys 5 on the front surface of a substantially rectangular casing. In addition, the camera unit 7 is provided on the upper side of the housing, and an insertion port for inserting the game cartridge 10 is provided on the lower side.

  The display 1 is a color liquid crystal display with a backlight such as a TFT display, and displays various information, images, and the like. The display 1 functions as an electronic finder when the camera unit 7 is activated, and displays a finder image by the camera unit 7 as a moving image.

  The operation key 5 is used for the player to input a game operation, and one of the operation keys 5 functions as a shutter key for inputting an imaging instruction when the camera unit 7 is activated.

  The camera unit 7 has a photographic lens 8 formed of glass or the like provided on the back side thereof, and an imaging element such as a CMOS or a CCD, and digitally converts image data imaged by the imaging element through the photographic lens 8. Convert to data and output. In addition, although this camera unit 7 is comprised integrally with the housing | casing, it is good also as being comprised so that attachment or detachment is possible.

  In addition, the portable game apparatus 1000 includes a reading device that reads game information (game program and game data) stored in the game cartridge 10 inserted into the insertion slot, and other external devices such as other portable game apparatuses 1000. It has a communication device for connecting and performing wireless / wired communication, and a control device equipped with a CPU, IC memory, and the like. The CPU mounted on the control device executes various processes based on programs and data read from the IC memory and the game cartridge 10, operation signals from the operation keys 5, image data input from the camera unit 7, and the like. Then, an image signal such as a game screen and a sound signal of a game sound are generated. Then, the generated image signal is output to the display 1 to display the game screen, and the sound signal is output to the speaker 3 to output the game sound.

  Information (system program, game program, game data, etc.) necessary for the portable game apparatus 1000 to execute a game is stored in an IC memory or a game cartridge 10 mounted on the control apparatus. More specifically, the system program is stored in the IC memory of the control device, and the game program and game data are stored in the game cartridge 10. Therefore, the player can cause the portable game apparatus 1000 to execute a different game by exchanging the game cartridge 10.

[Overview]
In the game of this embodiment, a game effect is performed as if a ghost (character) exists around the player in the real space.

2-4 is a figure for demonstrating the game outline | summary of this embodiment.
As shown in FIG. 2A, when the camera unit 7 of the portable game apparatus 1000 is activated, an image (finder image) captured by the photographing lens 8 is displayed on the display 1 in the image area AR1 at the center thereof. Is displayed.

  Further, in the game of the present embodiment, as one of the game effects, an effect as if the portable game device 1000 includes a ghost sensor (character sensor) that detects the presence of a ghost around the device is performed. Is called. This ghost sensor has a function of sensing a character that exists within a predetermined range centered on the shooting direction of the camera unit 7. Then, the detection result by the ghost sensor (hereinafter referred to as “ghost reaction”) is displayed (notified) on the display 1 together with the finder image. This finder image is a moving image.

  Specifically, the strength (degree) of the ghost reaction is digitized into an integer in the range of “0” to “100” and displayed on the gauge MT (gauge display). ) MN is displayed as a percentage. The larger the gauge value MN, the stronger the ghost reaction. That is, the minimum value “0” of the gauge value indicates that there is no ghost response, and the maximum value “100” indicates that the ghost response is strongest. The gauge value MN represents whether or not the finder image satisfies a composite image condition that is a determination condition as to whether or not to display (appear) a ghost, and also represents the degree of satisfaction. That is, if the gauge value MN is equal to or greater than a predetermined value (for example, “75”), it indicates that it is satisfied, and if not, it indicates that it is not satisfied. For example, FIG. 5A shows that the gauge value MN is “85”, the ghost reaction is relatively strong, and the composite image condition is satisfied.

  On the display 1, a message corresponding to the strength of the ghost reaction is displayed in the message area AR2. Similar to the gauge value MN, this message indicates whether or not the finder image satisfies the composite image condition and indicates the general satisfaction level. For example, in the figure, a message “Two ghost hannowes” indicating that the ghost reaction is relatively strong and the composite image condition is satisfied is displayed.

  When the operation key 5 is operated and the imaging instruction is input (shutter ON) in such a state where the ghost reaction is relatively strong as described above, the ghost is displayed on the image (captured image) captured by the shutter ON. The image of That is, a ghost “appears”. In FIG. 9A, a ghost GS1 is displayed. As a standard of the shutter ON timing at which a ghost can be displayed, the gauge value is somewhat large (for example, “75” or more), or the message is “Toy Ghost Hannowed”. When the ghost reaction is weak (for example, less than “75”), the ghost is not displayed.

  At this time, the portable game device 1000 performs image recognition processing on the finder image using a predetermined image recognition method to extract feature data, and the extracted feature data and each of a plurality of ghosts prepared in advance are extracted. The degree of similarity with the feature data is calculated, and the degree of ghost reaction strength is determined based on the calculated degree of similarity. When an imaging instruction is input (shutter ON), one of the plurality of ghosts is selected based on the determined degree of ghost reaction intensity, and an image of this ghost is captured by the shutter ON. It is combined with the captured image and displayed on the display 1.

  The strength of the ghost reaction varies depending on the position of the subject in the finder image. Specifically, since the ghost sensor senses a character that exists in the imaging direction of the camera unit 7, the ghost response shows a stronger ghost response as the subject position is closer to the center of the image, and the ghost response becomes weaker as the distance from the center of the image increases. It will become.

  For example, as shown in FIG. 2 (a), when the subject OB1 is positioned approximately in the center of the finder image, the gauge value is “85”, indicating a relatively strong ghost reaction, but as shown in FIG. 2 (b). Thus, when the position of the subject OB1 is located in the left part of the finder image, the gauge value is “70”, indicating a weak ghost reaction compared to the case of FIG.

  Further, when the subject is different, the ghost reaction and the ghost displayed when the shutter is turned on are different. For example, as shown in FIG. 3A, when the imaging direction of the camera unit 7 is pointed in the finder image toward the subject OB2, the gauge value is “10” and the subject OB1 shown in FIG. Shows a weak ghost reaction. In the case shown in FIG. 5B, since the ghost reaction is weak, no ghost is displayed even when the shutter is turned on.

  As shown in FIG. 4, when the subject OB3 is captured in the finder image, the gauge value is “90”, which indicates a stronger ghost reaction than the case of the subject OB1 shown in FIG. When the shutter is turned on in this state, the display 1 displays the ghost GS2 combined with the captured image captured by the shutter being turned on.

  In both cases shown in FIGS. 3 and 4, when the position of the subject in the finder image changes from the center of the image to the peripheral side, the ghost reaction becomes weak.

[Function configuration]
FIG. 5 is a block diagram illustrating an example of a functional configuration of the portable game apparatus 1000 according to the present embodiment. According to the figure, the portable game apparatus 1000 functionally includes an operation input unit 100, an imaging unit 200, a processing unit 300, an image display unit 430, a sound output unit 440, and a storage unit 500. The communication unit 600 is configured.

  The operation input unit 100 receives an operation instruction from the player, and outputs an operation signal corresponding to the operation to the processing unit 300. In particular, in the present embodiment, an imaging instruction is detected and output to the processing unit 300. In FIG. 1, the operation key 5 corresponds to this.

  The imaging unit 200 includes an imaging element such as a CMOS or CCD, a photographing lens that forms an image of a light beam of a subject on the imaging element, and an A / D conversion that converts an electrical signal (analog image signal) output from the imaging element into a digital signal. A circuit is included, and image data of the subject (finder image data) is output to the processing unit 300. In FIG. 1, the camera unit 7 corresponds to this.

  The processing unit 300 performs various arithmetic processes such as overall control of the portable game apparatus 1000, game progress control, and image generation. This function is realized by, for example, an arithmetic device such as a CPU (CISC type, RISC type), ASIC (gate array, etc.) and its control program. In FIG. 1, a CPU or the like mounted on a built-in control device corresponds to this.

  The processing unit 300 mainly includes a game calculation unit 310 that performs calculation processing related to game execution, an image generation unit 330 that generates a game image based on various data obtained by the processing of the game calculation unit 310, And a sound generator 340 for generating game sounds such as sound effects and BGM.

  The game calculation unit 310 performs various game processes based on game information (programs and data) read from the storage unit 500, operation signals input from the operation input unit 100, image data input from the imaging unit 200, and the like. Execute. In the present embodiment, the game calculation unit 310 includes an image recognition unit 312, a similarity calculation unit 314, a ghost reaction determination unit 316, an appearance determination unit 318, and a display control unit 322.

  The image recognition unit 312 performs image recognition processing on the image data input from the imaging unit 200 at predetermined time intervals, and extracts feature data of the finder image. In this case, the image recognition unit 312 performs image recognition processing on the finder image data using the image recognition method specified by the image recognition method information 522.

  Here, image recognition methods performed by the image recognition unit 312 include shape recognition, color recognition, and pattern recognition. In shape recognition, for example, a Hough transform is performed on captured image data to detect a straight line component and its strength, a shape of a subject is recognized by a combination of the detected straight lines, and a feature portion based on the recognized shape is extracted. In the color recognition, the color of the entire captured image or the feature portion recognized by the shape recognition is recognized. In the pattern recognition, for example, a Fourier transform is performed on the captured image data to extract a frequency component, and the entire captured image or the pattern of the feature portion recognized by the shape recognition is recognized. In the present embodiment, image recognition is performed on captured image data by using one or a plurality of image recognition methods, and feature data of the captured image is extracted for each image recognition method.

  The image recognition method information 522 is information that designates an image recognition method for a captured image. FIG. 6 shows an example of the data configuration of the image recognition method information 522. According to the figure, the image recognition method information 522 stores an image recognition method 522a and a weight 522b in association with each other. The weight 522b is a value indicating the importance of the corresponding image recognition method, and based on this value, the total value of the similarity between the feature data of the captured image and the feature data of each ghost is calculated.

  In the image recognition method information 522 shown in the figure, “shape recognition” and “color recognition” are stored as image recognition methods. Therefore, according to the image recognition method information 522 shown in the figure, the image recognition unit 312 performs shape recognition processing on the captured image data to extract the feature data of the “shape” of the subject, and the color for the captured image data. Recognition processing is performed to extract feature data of “color” of the recognized shape portion.

  The image recognition result by the image recognition unit 312 is stored in the image recognition result information 538. FIG. 7 shows an example of the data configuration of the image recognition result information 538. According to the figure, the image recognition result information 538 stores an image recognition method 538a and feature data 538b extracted by the image recognition method in association with each other. The image recognition method 538b is an image recognition method designated by the image recognition method information 522. For example, in the same figure, feature data on “shape” and “color” corresponding to the image recognition method specified by the image recognition method information 522 shown in FIG. 6 is stored.

  The similarity calculation unit 314 calculates the similarity between the feature data of the finder image data and the feature data of each ghost specified by the ghost information 524 based on the image recognition result by the image recognition unit 312. Specifically, for each ghost, for each image recognition method, the similarity between the feature data and the feature data of the finder image is calculated according to a predetermined calculation formula. Since the similarity calculation method is a known method using pattern matching or the like, for example, detailed description is omitted. Then, a total value of the similarities is calculated by multiplying the calculated similarity for each image recognition method by the weight specified by the image recognition method information 522.

  The ghost information 524 is information regarding the characteristics of each ghost that can appear. FIG. 8 shows an example of the data structure of the ghost information 524. According to the figure, the ghost information 524 stores a name (ghost name) 524a and feature data 524b in association with each ghost. The feature data 524b stores a value corresponding to each image recognition method specified by the image recognition method information 522. For example, in the same figure, feature data about “shape” and “color” which are image recognition methods specified by the image recognition method information 522 shown in FIG. 6 is stored.

  The similarity calculated by the similarity calculation unit 314 is stored in the calculated similarity information 542. FIG. 9 shows an example of the data structure of the calculated similarity information 542. According to the figure, the calculated similarity information 542 stores the name (ghost name) 542a and the calculated similarity 542b in association with each ghost specified by the ghost information 524. The similarity 542b includes the similarity for each image recognition method specified by the image recognition method information 522 and the total value of the similarities. For example, the figure includes similarities for “shape” and “color”, which are image recognition methods specified by the image recognition method information 522 shown in FIG. 6, and total values of these similarities. The similarity 542b is expressed by a decimal value in the range of “0.0” to “1.0”, where the minimum value “0.0” represents a complete mismatch and the maximum value “1.0” is It represents an exact match.

  The ghost reaction determination unit 316 determines a ghost reaction based on the image recognition result by the image recognition unit 312. Specifically, referring to the calculated similarity information 542, the largest value is selected from the total values of the similarities for each ghost. Then, based on the selected value (the maximum value of the total value of similarities, hereinafter referred to as “maximum similarity”), the gauge value is determined according to the relationship shown in FIG.

FIG. 10A is a graph showing the relationship between the maximum similarity and the gauge value. The horizontal axis is the maximum similarity A, and the vertical axis is the gauge value G. According in FIG. 6 (a), the ghost reaction determination unit 316, the maximum similarity A is if the first less than the threshold value T ₁ of the predetermined minimum gauge value G value to "0". The maximum similarity A is the first threshold value above T _1, and if the ₂ less than a predetermined second threshold value T, and calculates the gauge value G according to the following equation (1). Further, if the maximum similarity A second threshold value T ₂ or more, the gauge value G between the maximum value "100". However, the first threshold value T ₁ <the second threshold value T ₂ .
G = {(A−T ₁ ) / (T ₂ −T ₁ )} × 100 (1)

The values of the first threshold T ₁ and the second threshold T ₂ are stored in the threshold information 526. FIG. 10B shows an example of the data configuration of the threshold information 526. According to FIG. 5B, the threshold information 526 stores values of a first threshold 526a and a second threshold 536b.

Further, the ghost reaction determination unit 316 refers to the similarity calculation coefficient information 528, and calculates a coefficient f corresponding to the position in the finder image of the subject that is the target of feature data extraction by the image recognition unit 312. Multiply by G to calculate the final gauge value G _f (= G × f).

  The similarity calculation coefficient information 528 is information on the coefficient f by which the gauge value G is multiplied. FIG. 11A shows an example of the data configuration of the similarity calculation coefficient information 528. According to the figure, the similarity calculation coefficient information 528 stores the subject position 528a in the viewfinder image and the coefficient 528b in association with each other.

  The subject position 528a in the finder image is the position of the shape portion in the finder image recognized by the image recognition unit 312 immediately before the shutter is turned on, and a plurality of positions set in the finder image as shown in FIG. It is expressed as an area. That is, according to FIG. 4B, the viewfinder image includes an area A that is the central portion of the image, an area B that is the peripheral area of the area A, and a peripheral area of the area B that is the periphery of the image. It is divided into area C, which is an area. In each area, a coefficient f in the range of “0.0” to “1.0” is set. Specifically, in the order of area A, area B, and area C close to the center of the image, A large value is set.

  In addition, when the position of the subject in the finder image, that is, the shape recognition portion extends over a plurality of areas, the determination is made in the priority order of area A, area B, and area C.

When an imaging instruction is input from the operation input unit 100 (shutter ON), the appearance determination unit 318 determines a ghost to appear based on the ghost reaction determined by the ghost reaction determination unit 316. Specifically, it is determined gauge value G _f calculated by the ghost reaction determination unit 316 depending on whether it is above a predetermined occurrence threshold, whether the viewfinder image satisfies a composite image condition. If it is equal to or greater than the appearance threshold, the composite image condition is satisfied, so that the ghost having the maximum total value of similarity is selected with reference to the calculated similarity information 542, and it is determined that the selected ghost appears. On the other hand, if the gauge value is less than the appearance threshold, the composite image condition is not satisfied, and it is determined that no ghost appears.

  This appearance threshold value is stored in the appearance determination condition information 532. FIG. 12 shows an example of the data configuration of the appearance determination condition information 532. As shown in the figure, the appearance determination condition information 532 stores an appearance threshold value. The appearance threshold value is expressed by an integer value in the range of “0” to “100”, and the larger the appearance threshold value, the smaller (lower) the appearance probability of the ghost.

The display control unit 322 controls the image generation unit 330 to control image display on the image display unit 430. Specifically, a finder image based on the image data input from the imaging unit 200 is displayed in the image area AR1, and a display based on the ghost reaction determined by the ghost reaction determination unit 316 is performed. That is, updates the display of the gauge MT to represent the gauge value G _f calculated by the ghost reaction determining unit 316 updates the display of the gauge value MN. Further, by referring to the message information 534 to display a message corresponding to the gauge value G _f in the message area AR2.

The message information 534 is message information to be displayed in the message area AR2. FIG. 13 shows an example of the data structure of message information 534. According to the figure, the message information 534 stores the gauge value (534a) and the message (534b) in association with each other. The display control unit 322 displays the message area AR2 by selecting a message corresponding to the gauge value G _f.

  Further, when an imaging instruction is input from the operation input unit 100, the display control unit 322 synthesizes a captured image with a ghost image determined to appear by the appearance determination unit 318 and displays the synthesized image in the image area AR1. When the appearance determination unit 318 determines that no ghost appears, only the captured image is displayed in the image area AR1. The captured image is a finder image captured when an imaging instruction is input from the operation input unit 100, and is stored in the captured image information 534.

  The image display unit 430 displays a game image based on the image signal from the processing unit 300. This function is realized by hardware such as CRT, LCD, ELD, for example. In FIG. 1, the display 1 corresponds to this.

The sound generation unit 340 generates game sounds such as sound effects and BGM used during the game, and outputs the generated game sound signal to the sound output unit 440.
The sound output unit 440 outputs game sounds such as BGM and sound effects based on the sound signal from the sound generation unit 340. This function is realized by, for example, a speaker. In FIG. 1, the speaker 3 corresponds to this.

  The storage unit 500 stores a system program for realizing various functions for causing the processing unit 300 to control the portable game apparatus 1000 in an integrated manner, a program and data necessary for executing the game, and the like. Used as a work area of the processing unit 300, temporarily stores calculation results executed by the processing unit 300 according to various programs, input data input from the operation input unit 100, and the like. This function is realized by various IC memories, hard disks, CD-ROMs, DVDs, MOs, RAMs, VRAMs, and the like. In FIG. 1, ROM, RAM, and game cartridge 10 mounted on the control device correspond to this.

  Further, the storage unit 500 stores a game program 510 and game data for causing the processing unit 300 to function as the game calculation unit 310. The game data includes image recognition method information 522, ghost information 524, threshold information 526, similarity calculation coefficient information 528, appearance determination condition information 532, message information 534, captured image information 536, and image recognition. Result information 538, calculated similarity information 542, and ghost capture information 544 are included.

  The ghost capture information 544 is information such as the type and number of ghosts that have been “captured” so far. The capture of the ghost is realized by executing a predetermined ghost capture process for the appearing ghost.

  The communication unit 600 connects to a predetermined communication line and performs data communication with an external device such as another portable game device 1000. This function is realized by a wireless communication module such as BlueTooth (registered trademark) or IrDA, a modem, TA, a cable communication cable jack, a control circuit, and the like. In FIG. 1, the built-in communication device corresponds to this.

[Process flow]
FIG. 14 is a flowchart for explaining the flow of the game process in the present embodiment. This process is realized by the game calculation unit 310 executing a process according to the game program 510.

  According to the figure, the image recognition unit 312 performs image recognition processing on the finder image data by the image recognition method specified by the image recognition method information 522 at every predetermined time, and extracts feature data of this finder image ( Step S1). Next, a ghost reaction determination process is executed (step S3).

FIG. 15 is a flowchart for explaining the flow of the ghost reaction determination process.
According to the figure, in the ghost reaction determination process, the similarity calculation unit 314 refers to the image recognition result information 538 and performs the process of loop A for each ghost of the ghost information 524. That is, in the loop A, for each designated image recognition method, the similarity between the two data is calculated according to a predetermined calculation formula based on the ghost feature data of interest and the extracted finder image feature data. (Step T1). Then, the similarity for each image recognition method calculated is multiplied by the weight set for each image recognition method and added to calculate a total value of similarity (step T3).

  When the processing of loop A for all ghosts is completed, the ghost reaction determination unit 316 then selects the largest value from the calculated total values of similarities (step T5). Based on the selected value (maximum similarity), the gauge value is determined with reference to the threshold value information 526.

That is, it is determined whether or not the maximum similarity A is first thresholds T ₁ or more, if not the first threshold value above T ₁ (Step T7: NO), the minimum gauge value G to "0" ( Step T9). If the maximum similarity A is the first threshold T ₁ or more (step T7: YES), subsequently, it is determined whether the maximum similarity A is the second threshold value T ₂ or more, the second threshold value T If it is ₂ or more (step T11: YES), the gauge value G is set to the maximum value “100” (step T13). Also, if the maximum similarity A is not the second threshold value _{T 2} or more (Step T13: NO), calculates the gauge value G according to the equation (1) (step T15).

Subsequently, the ghost reaction determination unit 316 refers to the similarity calculation coefficient information 528 and multiplies the calculated gauge value G by the coefficient f corresponding to the position of the subject in the finder image to obtain the final gauge value G _f. Is calculated (step T17).
When the above processing is performed, the ghost reaction determination processing ends.

  When the ghost reaction determination process ends, the display control unit 322 performs a ghost reaction display process. In other words, the display of the gauge MT is updated based on the calculated gauge value, the display of the gauge value MN is updated, and the message corresponding to the gauge value is displayed in the message corresponding to the gauge value by referring to the message information 534. The display is updated (step S5).

  Subsequently, the game calculation unit 310 determines whether or not an imaging instruction (shutter ON) is input from the operation input unit 100, and if it is input (step S7: YES), the appearance determination unit 318 generates a ghost. Judge whether to make it appear. That is, with reference to the appearance determination condition information 532, it is determined whether or not the gauge value is greater than or equal to the appearance threshold. If the gauge value is equal to or greater than the appearance threshold, the calculated similarity information 542 is referred to and it is determined that a ghost having the highest total similarity value appears (step S9).

  If it is determined that a ghost appears (step S11: YES), then the display control unit 322 combines the ghost image determined to appear with the captured image and causes the image display unit 430 to display the image (see FIG. Step S13). Next, the game calculation unit 310 performs a predetermined ghost capturing process for the ghost that has appeared based on an operation signal from the operation input unit 100 (step S15). Thereafter, it is determined whether or not to end the game. If the game is not ended (step S17: NO), the process returns to step S1, and if the game is ended (step S17: YES), this process ends.

[Action / Effect]
As described above, according to the present embodiment, on the display 1 of the portable game apparatus 1000, the image (finder image) captured by the photographing lens 8 is displayed in the image area AR1 and exists in the real space. A detection result (ghost reaction) by the ghost sensor for detecting the character is displayed. That is, the degree of the strength of the ghost reaction is digitized and displayed on the gauge MT (gauge display). This value (gauge value) MN is displayed, and a message corresponding to the gauge value MN is displayed in the message area AR2. Is displayed. Then, when the shutter is turned on by operating the operation key 5 in a state where the ghost reaction is strong (for example, the state where the gauge value MN is “75” or more), an image (captured image) captured by the shutter ON is displayed in the image area AR1. ), A ghost corresponding to the subject is synthesized and displayed.

At this time, the portable game apparatus 1000 performs image recognition processing on the finder image to extract feature data of the finder image, and for each ghost specified by the ghost information 524, the ghost feature for each image recognition method. The similarity between the data and the extracted feature data of the finder image is calculated. Next, the gauge value G is calculated with reference to the threshold value information 526 based on the largest value (maximum similarity) among the total similarity values for the ghosts, and further, the similarity calculation coefficient information 528 is referred to. Then, the final gauge value _Gf is calculated by multiplying the calculated gauge value G by the coefficient f corresponding to the position of the subject in the finder image. Then, the meter displays the gauge value G _f to gauge MT, and displays as a gauge MT, with reference to the message information 534 to display a message corresponding to the gauge value G _f in the message area AR2.

  Therefore, a finder image is displayed on the display 1, and a display corresponding to a ghost reaction that is a sensing reaction to a ghost existing in the vicinity is made based on the finder image. Then, when the shutter is turned on in a state where the ghost reaction is strong, it is possible to realize a new interest as if a ghost that exists in real space is imaged, such that the ghost is synthesized and displayed on the captured image. Further, at this time, an effect as if the ghost exists at a certain fixed place (subject position) such that the same ghost is displayed when the same subject is imaged is realized.

[Modification]
The application of the present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention.

(A) Displaying the position of a subject For example, an image recognition portion by image recognition processing may be shown in a finder image. Specifically, for example, when a predetermined operation is performed, such as pressing the shutter key halfway, by changing the display form of the shape part recognized by shape recognition and the part other than this shape part, The recognition part is clarified visually. For example, the brightness, hue, and resolution of the recognized shape portion are changed. In this way, the player grasps the image part that is the object of image recognition, and, for example, captures the image part so that it appears in the center of the image, so that a ghost appears more reliably. It becomes possible.

(B) Calculation Method of Total Value of Similarity In the embodiment described above, the total value of the similarity for each ghost is set to the similarity for each image recognition method, and the weight set for the image recognition method is set. Although the calculation is performed by multiplication and addition, the calculation may be performed as follows. That is, (1) an average value of similarities for each image recognition method may be set as a total value, and (2) a maximum value among similarities for each image recognition method may be set as a total value.

(C) Changing the Gauge Value If the finder image does not change as the same subject is continuously photographed, the gauge value may be changed as time passes. When the finder image does not change, the characteristic data of the finder image does not change, and the calculated gauge value does not change. Therefore, in such a case, the calculated gauge value is changed so as to be increased or decreased periodically or randomly according to a sine function, for example, within a predetermined fluctuation range.

  By doing so, an effect is realized as if the ghost is moving and sensing it. Note that the determination that the viewfinder image has not changed is made by, for example, no change in the feature data. Also, at this time, by changing the period to be changed and the amount of increase / decrease (amplitude) according to the magnitude of the gauge value, for example, the greater the gauge value, the shorter the period or the greater the amount of increase / decrease, An effect that senses a ghost with a sensitivity corresponding to the strength of the ghost reaction is realized.

(D) Size of the subject in the finder image In the above-described embodiment, the coefficient f corresponding to the position of the subject in the finder image (the position of the shape recognition portion) is added to the gauge value G corresponding to the maximum similarity. Although the final gauge value _Gf is calculated by multiplication, the coefficient e may be multiplied by a coefficient e corresponding to the size of the subject in the finder image instead of the coefficient f. In this case, for example, by increasing the value of the coefficient e as the subject size in the finder image increases, the ghost sensor produces an effect as if the ghost sensor senses a ghost located closer to it with higher sensitivity. Realized.

(E) A plurality of ghost sensors It is good also as sensing a ghost by preparing a plurality of ghost sensors and using one ghost sensor selected from these. In this case, each ghost sensor is associated with one or more detectable ghosts in advance. Then, for each ghost corresponding to the ghost sensor being used, the similarity between the feature data and the feature data of the finder image is calculated, and the gauge value is calculated based on the calculated similarity for each ghost. In this way, when the ghost sensor to be used is changed, an effect that different ghosts can be sensed is realized.

  At this time, the ghost subject to appearance determination may be (1) a ghost corresponding to the ghost sensor being used. In this case, among the ghosts corresponding to the ghost sensor being used, it is determined that the ghost having the largest total similarity value calculated and having a predetermined appearance threshold value or more appears. Therefore, a ghost appears only when the ghost response by the ghost sensor being used is strong.

  Further, (2) not only the ghost corresponding to the ghost sensor being used, but also all ghosts may be subject to appearance determination. In this case, the degree of similarity with the finder image is calculated for all ghosts, and it is determined that a ghost having the largest total value of the calculated degrees of similarity and having a predetermined appearance threshold value or more will appear. Therefore, even when the shutter is turned on when the ghost response by the ghost sensor being used is weak, a ghost may appear, which means that the ghost sensor being used cannot sense the ghost. . That is, it is possible to make the user feel more realistic about the relationship between the ghost and the ghost sensor.

(F) Notification of presence / absence of character In the above-described embodiment, the strength of the ghost reaction is meter-displayed (gauge display) with the gauge MN as the notification of whether the character exists. It is good also as alerting | reporting by the method of. For example, when the finder image satisfies the composite image condition, the hue (RGB value) and brightness of the entire finder image may be changed. In this case, the degree of satisfaction of the composite image condition may be expressed by the amount of change in hue or luminance.

(G) Character In the above-described embodiment, a ghost has been described as an example of a character. However, other characters such as an imaginary creature such as a human, an animal, a robot, or a monster, or an anime character may be used. Of course.

An example of the appearance of a portable game device of an embodiment. 1 is a schematic diagram of an embodiment. 1 is a schematic diagram of an embodiment. 1 is a schematic diagram of an embodiment. 2 is a functional configuration example of a portable game device. The data structural example of image recognition method information. The data structural example of image recognition result information. The data structural example of ghost information. The data structural example of calculation similarity information. (A) Gauge value calculation method, (b) Data configuration example of threshold information. (A) Data configuration example of similarity calculation coefficient information, (b) Area setting example in finder image. The data structural example of appearance determination condition information. Data configuration example of message information. The flowchart of the game processing of embodiment. The flowchart of the ghost reaction determination process performed during a game process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1000 Portable game device 100 Operation input part 200 Imaging part 300 Processing part 310 Game operation part 312 Image recognition part 314 Similarity calculation part 316 Ghost reaction determination part 318 Appearance determination part 322 Display control part 330 Image generation part 340 Sound generation part 430 Image display unit 440 Sound output unit 500 Storage unit 510 Game program 522 Image recognition method information 524 Ghost information 526 Threshold information 528 Similarity calculation coefficient information 532 Appearance determination condition information 534 Message information 536 Captured image information 538 Image recognition result information 542 Calculation similarity Degree Information 544 Ghost Capture Information 600 Communication Department

Claims (11)

By causing the computer to control the display of the finder image captured by the imaging means on the display device, and to control the display by combining the image of the character with the captured image captured by the shutter operation, the character can be displayed in real space. A program for controlling the display as if it exists,
Whether the finder image satisfies any of the composite image conditions among the composite image conditions for each character, which is an image condition for determining whether to synthesize the character image with the captured image. Condition determination means for determining whether
A notification control means for performing a predetermined notification process as to whether or not a character exists based on the determination result of the condition determination means;
When a shutter operation is performed in a state in which a notification process that the character is present is performed by the notification control unit, a composite display control unit that combines and controls display of the character image with the captured image captured by the shutter operation ,
A program for causing the computer to function as   The composite display control means corresponds to a composite image condition that is determined to be satisfied by the condition determination means when a shutter operation is performed in a state in which the notification processing that the character is present is performed by the notification control means. The program according to claim 1, wherein the computer is caused to function so as to display and control an image of a character to be synthesized with a captured image captured by the shutter operation. Each of the composite image conditions includes a plurality of image conditions,
Condition satisfaction determination that the condition determination means determines a degree of satisfaction that the finder image satisfies each of the combined image conditions based on whether or not each image condition included in the combined image condition is satisfied. Having means,
The notification control means notifies the notification content according to the satisfaction degree determined by the condition satisfaction degree determination means,
The program according to claim 1 or 2 for causing the computer to function as described above.   The program according to claim 3, wherein the notification control unit causes the computer to function to perform notification processing by changing notification contents over time when the viewfinder image does not change. The condition determining means is
Feature extraction means for performing feature extraction processing to extract a feature portion of the finder image;
A feature portion determination means for determining whether or not the image of the feature portion extracted by the feature extraction means satisfies any composite image condition;
The program as described in any one of Claims 1-4 for functioning the said computer so that it may have. Each composite image condition includes at least a shape condition of the object as an image condition,
The feature extraction means extracts the shape of an object shown in the viewfinder image as a feature part,
The feature portion determination means determines whether or not the object shape extracted by the feature extraction means satisfies an object shape condition included in a composite image condition;
The program according to claim 5 for causing the computer to function as described above.   7 or 6 for causing the computer to function so that the notification control unit performs different notification processing depending on the size and / or position of the feature portion extracted by the feature extraction unit in the viewfinder image. The program described in.   The program according to any one of claims 1 to 7 for causing the computer to function so that the notification control unit includes a gauge display control unit that performs predetermined gauge display as the notification process. A plurality of virtually wearable character sensors that detect whether or not a character exists in the imaging direction of the imaging means are set in advance according to the detectable character,
A mounting sensor selection means for selecting a character sensor to be virtually mounted among the plurality of character sensors according to a user's selection operation;
Function the computer as
The condition determination means sets a composite image condition corresponding to a character that can be detected by the character sensor selected by the wearing sensor selection means from the plurality of composite image conditions as a determination target condition.
The program as described in any one of Claims 1-8 for functioning the said computer like this.   The computer-readable information storage medium which memorize | stored the program as described in any one of Claims 1-9. An image pickup means and a display means for displaying a finder image photographed by the image pickup means are provided, and a character image is combined with a picked-up image captured by a shutter operation, and the display means controls display. An image pickup and display device that performs display control as if a character exists in real space,
Whether the finder image satisfies any of the composite image conditions among the composite image conditions for each character, which is an image condition for determining whether to synthesize the character image with the captured image. Condition determination means for determining whether or not
Based on the determination result of the condition determination means, notification control means for performing a predetermined notification processing of whether or not a character exists;
When a shutter operation is performed in a state in which a notification process that the character is present is performed by the notification control unit, a composite display control unit that combines and controls display of the character image with the captured image captured by the shutter operation When,
An image pick-up display device.

Images