JP2010067002A - Information processing apparatus, method, and program - Google Patents

Abstract

Provided is an image in which a user enters the world of a virtual space and works with a character in the virtual space.
An output image creation unit 31 estimates a position of a user subject according to an operation determination result of an operation determination unit 16 (S3), and includes all or a part of the user subject from an image received from a camera 200. A partial image is extracted (S4). Further, the output image creation means 20 determines the positions of the ball object and the bat object when the swinging motion of the operation unit 5 is detected from the operation determination of the operation determination means 16 (S5). Further, the output image creation means 20 creates an output image by synthesizing the ball object and the bat object with the basic background image for output in accordance with the determined positions of the character and the bat object (S7).
[Selection] Figure 8

Description

  The present invention relates to an apparatus, a method, and a program for performing various types of information processing according to user operations.

  In Patent Literature 1 and Patent Literature 2, game results and the like are printed with a user's face photo. Patent Document 3 relates to a trajectory image synthesizing apparatus for video objects, and synthesizes a trajectory of a ball with video in a golf relay or the like.

In addition, patent documents 4-7 are mentioned as prior art relevant to this invention. Patent Document 4 discloses a system for measuring a swing. Patent Document 5 discloses a technique for realistically displaying a batting action of a baseball game character. Patent Document 6 discloses a technique for extracting a human subject from an image. Patent Document 7 discloses a technique for extracting a subject's face.
JP 2006-86823 A JP 2000-138858 A JP 2006-287911 A Japanese Patent Laid-Open No. 2002-210055 JP 2001-204963 A JP 2006-311505 A JP 2008-42801 A

  In Patent Documents 1 and 2, there is a description that a scene in a game is stored and printed in combination with a user's face photograph, but it is not always possible to capture and print a user who performs a dynamic operation at an accurate timing. . Also, simply printing the user's face side by side on the game screen cannot give a sense of realism that the user was actually playing the game. In addition to the user's face, it is also impossible to take out and print a subject partial image including an operating hand or foot. In Patent Document 3, a motion is estimated from a video and an image is synthesized. However, there is a restriction that a motion of a photographing camera must be used.

  The present invention makes it possible to provide an image as if a user has entered the world of a virtual space and is working with a character in the virtual space. Further preferably, the present invention enables the photographer's intention to be reflected in the image processing with higher accuracy based on an action taken by the photographer when photographing an image.

  An information processing apparatus according to the present invention includes a motion detection unit that detects a user's motion, and a progress management unit that controls the progress of a scene according to a scenario in a virtual space based on the user's motion detected by the motion detection unit A storage unit that stores a basic background image for output according to a scene in which the progress management unit can control the progress, an imaging instruction unit that instructs a predetermined imaging device to start or end imaging, and an imaging instruction An image extraction unit that extracts a basic background image for output corresponding to a scene controlled by the progress management unit at the time when the unit instructs the start of imaging is acquired by the imaging device in response to an instruction from the storage unit and the imaging instruction unit An image for output is created by combining an image input unit that inputs an image from the imaging device, a part or all of the image input by the image input unit, and the basic background image for output extracted by the image extraction unit. Create output image And, equipped with a.

  According to the present invention, when a shooting is instructed at a desired timing while a scenario is in progress according to the user's operation, the user is instructed to take an image at that time, and the obtained image is instructed to take an image. Create an output image by combining with the scene at the time. As a result, an image is created as if the user has entered the virtual space world and is moving with the character in the virtual space.

  A position detection unit that detects a user's operation position, and the output image creation unit outputs the image input by the image input unit at a position on the basic output image for output corresponding to the user's operation position detected by the position detection unit. Synthesize some or all.

  In this way, the user's subject image can be accurately synthesized at the same position as the character in the virtual space.

  The output basic background image includes an image in which the user character is captured from a desired viewpoint in the virtual space.

  The basic output background image includes an image that captures the user from a viewpoint opposite to the display video in the virtual space.

  An operation determining unit that determines whether or not a predetermined operation related to the progress of the scene has started or ended based on the user's operation detected by the operation detecting unit. In response to determining that a predetermined operation related to the progress of the scene has started or ended, the imaging apparatus is instructed to start or end imaging.

  In this way, it is possible to start or end shooting in accordance with the timing of the specific operation of the user.

  A user extraction unit that extracts a subject image of a user from an image input by an image input unit based on a user's operation detected by the motion detection unit and / or a user operation position detected by the position detection unit. The image creating unit synthesizes the user's subject image extracted by the user extracting unit and the output basic background image extracted by the image extracting unit to create an output image.

  In this way, the user's subject image can be accurately extracted.

  The imaging instruction unit instructs the imaging apparatus to set a shooting interval and / or a shutter speed according to the user action detected by the action detection unit.

  In this way, it is possible to set a shooting interval and / or a shutter speed suitable for the user's operation.

  The imaging instruction unit instructs the imaging device to start or end imaging in response to an input operation to the operating device corresponding to a user other than the user.

  In this way, another user can capture an image of the user at an arbitrary timing while watching the user's facial expressions and actions.

  An output unit is provided that outputs the output image created by the output image creation unit to a desired electronic device.

  The information processing method according to the present invention includes a step in which a computer detects a user's action, a step of controlling the progress of a scene according to a scenario in a virtual space based on the detected user's action, and a control of the progress. A step of storing a basic background image for output corresponding to a scene that can be performed, a step of instructing a predetermined imaging device to start or end imaging, and a scene controlled at the time of instructing the start of imaging A step of extracting a corresponding output basic background image, a step of inputting an image acquired by the imaging device in response to an instruction to start imaging, a part or all of the input image, and an output for extraction Generating an output image by combining the basic background image.

  The information processing program according to the present invention is capable of detecting a user's action, controlling a progress of a scene according to a scenario in a virtual space, and controlling the progress based on the detected user's action. A step of storing a basic background image for output according to a possible scene, a step of instructing a predetermined imaging device to start or end imaging, and an output corresponding to a scene controlled at the time of instructing the start of imaging A step of extracting a basic background image for use, a step of inputting an image acquired by the imaging device in response to an instruction to start imaging, a part or all of the input image, and an extracted basic background image for output Are made to be executed by a computer and a step of creating an output image.

  According to the present invention, when a shooting is instructed at a desired timing while a scenario is in progress according to the user's operation, the user is instructed to take an image at that time, and the obtained image is instructed to take an image. Create an output image by combining with the scene at the time. As a result, an image is created as if the user has entered the virtual space world and is moving with the character in the virtual space.

<First Embodiment>
FIG. 1 shows the appearance of a video processing system 1000 according to this embodiment according to a preferred embodiment of the present invention. The video processing system 1000 includes a video processing device 100, an operation unit 5, and a camera 200. The operation unit 5 is connected to the video processing apparatus 100 via Bluetooth, infrared communication, or the like. The operation unit 5 is a device independent of the video processing device 100 from the viewpoint of improving operability, and is preferably wirelessly connected to the video processing device 100, but there is no problem even if it is connected by wire. .

  A camera 200 is installed in the vicinity of the user P. The camera 200 may be installed anywhere as long as the user P can be captured as a subject. For example, the user may be placed on the monitor 23 case to catch the user from the front, or the user may be seen from the side or the back on the left, right, or back of the user. The camera 200 and the video processing device 100 are connected by various communication means such as USB and wireless LAN, and the CPU 10 of the video processing device 100 is instructed by the CPU 10 of the video processing device 100 for the shooting timing of the camera 200. An image captured by the camera 200 is transmitted to the communication unit 4 of the video processing apparatus 100 and stored in the RAM 42.

  For simplification of explanation, the installation position of the camera 200 captures the subject from the front. However, if the user wants to capture the subject from other positions, the position is designated from the operation unit 5 or the non-user operation unit 6. You may be able to do it.

  Speakers for outputting BGM and sound signals of sound effects transmitted from the video processing apparatus 100 are attached to the left and right sides of the monitor 23.

  FIG. 2 is a block diagram of the video processing apparatus 100. The video processing apparatus 100 includes a CPU 10, a ROM 41, and a RAM 42. The CPU 10 implements a plurality of functions to be described later based on the OS and video processing program recorded in the ROM 41 and video processing data stored in the RAM 42. The ROM 41 stores an OS for causing each unit of the video processing system 1000 to perform basic operations. The RAM 42 is used as a work area for temporarily storing various video processing data read from the semiconductor memory 4 as necessary. The semiconductor memory 4 may be detachable, and different types of video processing programs may be exchanged so that various videos, for example, a video game or a virtual world experience can be enjoyed. Alternatively, various media reading devices such as a CDROM drive may be provided in the video processing device 100, and the video processing program may be loaded from the CD ROM installed in the reading device to the RAM 42 and executed. Alternatively, the video processing apparatus 100 may download the video processing program from a predetermined server connected via various networks such as the Internet to the RAM 42 and execute it. In this embodiment, the type of video is a baseball game. However, it can also be applied to a game of hitting a ball with something such as tennis or golf.

  The video processing apparatus 100 includes a GPU (Graphics Processing Unit) 21 and a frame buffer 22. The GPU 21 writes a CG image composed of a combination of polygons to the frame buffer 22 based on the calculation result of the CPU 10. The CG image drawn by the GPU 21 is temporarily recorded in the frame buffer 22. The CG image recorded in the frame buffer 22 is output to the monitor 23 and displayed. By continuously performing drawing processing by the GPU 21 and recording in the frame buffer 22, a CG image including a moving image element, for example, a motion of a character is displayed on the monitor 23.

  The video processing apparatus 100 also includes an SPU (Sound Processing Unit) 31. The SPU 31 outputs analog signals of music and sound effects to the display 23 based on sound data such as music data and various sound effect data recorded in the semiconductor memory 4 (or an alternative such as a CDROM or a network download server). Then, this is reproduced by the speaker of the display 23.

  The communication unit 4 receives operation information from the operation unit 5. The operation unit 5 is an instruction input unit for operating an object such as a bat, a golf club, or a racket, which is a virtual handling item in a character or a character game space, a virtual world, or other virtual space, in the virtual space. However, the operation unit 5 does not have to be a virtual operation object, and can function as a device that simply detects a user's action (stepping or the like).

  The operation unit 5 includes an operation detection unit 51, a position detection unit 52, and a controller 53. The operation detection unit 51 detects various operations stored in the operation unit 5, such as operations such as lifting, lowering, shaking, turning, tilting, and twisting in real time, and outputs them to the CPU 10. Specifically, the motion detection unit includes a triaxial (X, Y, Z axis) acceleration sensor 701 and a rotation sensor that detects angular velocity and angular acceleration of elevation angle and azimuth angle. The detection period of the user movement is arbitrary, but if it is short, it is ideal if the instantaneous movement can be continuously observed and a sampling rate equivalent to that of a known swing measurement system (for example, Japanese Patent Laid-Open No. 2002-210055) can be secured. It is.

  The position detection unit 52 is a means for the user to detect a designated position on the screen displayed on the monitor 23. The position detection unit 52 includes a CMOS sensor or the like. The position information sent from the position detection unit 52 to the CPU 10 is expressed by, for example, two-dimensional or more coordinates in the real space set on the screen of the monitor 23. If the position is continuously specified, the trajectory of the operation can be specified. Thereby, the swing position of the bat corresponding to the ball character displayed on the screen can be designated.

  FIG. 2 shows a processing unit performed by the CPU 10 according to the program in the ROM 41. The processing units of the CPU 10 include imaging control means 11, throwing control means 13, hit determination means 14, sound output control means 15, operation determination means 16, situation determination means 17, object display control means 18, and output image creation means 31. There is.

  The imaging control means 11 instructs the camera 200 connected via the communication unit 4 to start or end the imaging, and the camera 200 starts or ends imaging in response to this instruction.

  The pitch control means 13 controls the pitch from the pitcher character in the game space to the batter character. The pitcher character is a user who is not operated by a human being called an NPC or a CPU, and the pitching control means 13 controls the action. The contents of the control include the pitch start timing, ball type, speed, control, and the like.

  When pitching is started by the pitching control means 13, the object display control means 18 instructs the GPU 21 to generate a video showing the ball object going from the pitcher character to the batter box toward the batter character. This is the movement trajectory and movement speed of the ball character from the current position (mound) of the pitcher character toward the current position of the batter character (home base or a batter box in the vicinity thereof) in the coordinate system introduced into the game space. And instructing the generation of an image in which the position coordinates of the ball character sequentially move along with it. It is not necessary to display the pitcher character and the batter character on one screen at the same time. As is well known, the pitcher character is displayed with the background of the mound before and after the pitcher character is thrown toward the batter character (see FIG. 1), and when the ball object approaches the batter character, The GPU 21 may generate a video that displays the ball object together with the background of the batter box and output it to the monitor 23.

  The batter character has a bat object, and the user can control the timing, speed, and trajectory of swinging the bat object by the operation unit 5. The operation determining means 16 determines the trajectory and speed of the bat object in the game space in response to the operation unit 5 detecting a motion swung by the user. However, if the detection results of the position detection unit 52 and the position detection unit 52 are combined, it is possible to detect an operation in which the operation unit 5 is not shaken largely and moved to a certain position like a bunt operation and then the operation unit 5 is stopped. it can.

  For example, the operation determination unit 16 determines the head speed of the bat object from the detected angular velocity of the azimuth angle. Further, the position detection unit 52 detects the height position of the operation unit 5 when the operation unit 5 is swung. For example, when the monitor 23 is 20 inches (about 50 cm) and the height of the operation unit 5 is measured from the bottom of the screen of the monitor 23 and R1 = 0 to 20 cm, the position of the bat object in the game space is “ “Low”. Alternatively, if the height of the operation unit 5 is measured from the ground and R2 = 20 to 35 cm, it is determined as “medium (strike zone)”. Or if the height of the operation part 5 is measured from the ground and R3 = 35 cm or more, it is determined as “high”. R1, R2, and R3 can be adjusted according to the height of the user. This is because the user's height is registered in advance from the operation unit 5, and the CPU 10 shifts the position of R2 upward or downward according to the ratio of the average height and the registered height in the above R1 to R3, and strikes. You can make the zone higher or lower to suit your height.

  The hit determination means 14 determines whether or not the ball object released from the pitcher character hits the bat object operated by the user. For example, the hit determination means 14 indicates that the ball object trajectory at the time of detecting the swinging motion of the operation unit 5 is in the vicinity of the home base to which the coordinates in the virtual game space are assigned, and in the game space at that time After determining the height of the ball object from among “low”, “medium”, and “high”, the height of the ball object and the height of the bat object are compared. It is determined as “missing”.

  When the hit determination means 14 determines “winning”, the situation determination means 17 determines the virtual head speed v of the bat object when hit according to the angular velocity Θ detected by the operation determination means 16. The equation v = rΘ is used, but r is actually substituted with a value (for example, 85 cm) corresponding to the length of the bat. r may be specified from the operation unit 5. The situation determination unit 17 further determines a batting result, that is, a home run, a hit, and a file. This detects a motion vector in the direction of the Z-axis (axis perpendicular to the ground surface) in addition to the rotation operation, and determines the swing type: upper swing, down swing, level swing. If the head speed is equal to or lower than a predetermined threshold, the foul is determined, and if the head speed is equal to or higher than the predetermined threshold (for example, 120 km / h) and the level swing, the home run is determined.

  The sound output control means 15 instructs the SPU 31 to output a sound effect corresponding to the hit determination result of the hit determination means 14 or the batting result of the situation determination means 17. For example, a very loud cheer for a home run, an ordinary cheer for a hit, an ordinary noise for a foul, and an output of disappointing noise for an empty swing are instructed.

  The object display control unit 18 instructs the GPU 21 to output a video corresponding to the hit determination result of the hit determination unit 14 or the batting result of the situation determination unit 17. If it is a home run, the batter character will go around the base. If it is a hit, the batter character will come out. If it is a foul, the ball object will fly off the field. The GPU 21 is instructed to output a video showing the appearance.

  The progress control means 19 replaces the defensive and attacking teams, starts pitching the pitcher character, catches and sends the fielder's batting ball, counts the batter's score, runs the runner, the number of teams out, Depending on the batting result of the batter, the team score on the user side or CPU side is counted for each inning, and this is repeated up to a predetermined number of times (9 times). When the inning reaches the predetermined number of times, the game ends. Declare.

  The background DB 43 is composed of a hard disk unit or semiconductor memory 4 and the like, and the current game progress status managed by the progress control means 19, the hit judgment result of the hit judgment means 14 or the batting result of the situation judgment means 17, or the operation unit 5. The basic background image for output corresponding to all the patterns that can be taken by the designated installation position of the camera 200 is stored in advance. The output background image may be the same as or different from the game video image used by the GPU 21. Usually, an image obtained by capturing a virtual game space from a viewpoint opposite to the image for the game video is set as the basic background image for output. For example, even if the game image is an image of the pitcher viewed from the batter, the basic background image for output is an image as if the vicinity of the batter box where the batter (user) is present is viewed from the pitcher. The distance from the user to the viewpoint may be selected from the operation unit 5, and the output image creation unit 31 zooms in or out the output basic background image according to this distance, and the batter box ( Processing may be performed so that the vicinity of the user present position) is large and the vicinity of the batter box (user present position) is small from a far viewpoint.

  Further, the basic background image for output corresponds to the installation position of the camera 200. For example, the basic background image for output is four shootings for front imaging, left side imaging, right side imaging, and rear imaging. Further, it is assumed that the image is classified into positions, and further classified according to four batting results for fouling, home run, hit, and idling for each classification of each photographing position. Further, the output basic background image is associated with reference position information for designating a position on the output basic background image corresponding to the position of the character in the game space. In the baseball game, the reference position information is the position coordinates of the batter box for the right batter or left batter selected by the user. The output basic background image for front-side imaging is an image as seen from the pitcher near the batter box where the batter (user) exists, as described above. The output basic background image for right side imaging, left side imaging, or rear side imaging is located near the batter box where the batter (user) is located from the 1st dugout side, 3rd dugout side, or back side. It is an image as seen.

  For example, in FIG. 3, batter box Br (rectangular area), which is the standing position of the right batter character, is the reference position information. The reference position information changes according to the designation of right batter play or left batter play from the operation unit 5. If right batter play is designated, Br is used as reference position information, and if left batter is designated, Bl is used as reference position information. Since the user in FIG. 4A is a right batter, Br may be selected from the operation unit 5 as reference position information.

  Even if the position of the user character in the game space changes every moment as in a tennis game, the reference position information reflects the position of the user in the game space. This is obtained, for example, in a virtual three-dimensional game space by capturing the user character from a viewpoint opposite to the image for the game video and projecting the coordinates of the user character onto the basic background image for output on the two-dimensional plane. The coordinates of the given user character may be used as the reference position information.

  The output image creation means 31 outputs the basic background image for output corresponding to the hit determination result of the hit determination means 14 or the batting result output from the situation determination means 17 or the installation position of the camera 200 designated from the operation unit 5. Extracted from the background DB 43 and stored in the RAM 42.

  Further, the output image creation unit 31 determines the positions of the ball object and the bat object when the swinging motion of the operation unit 5 is detected from the operation determination of the operation determination unit 16. As described above, the position of the ball object is determined from “low”, “medium”, and “high” in the game space from the ball object trajectory when the swing operation is detected. For the bat object, the position of the bat object is determined from “low”, “medium”, and “high” from the above-described swing type.

  The output image creation means 31 creates an output image by synthesizing the ball object and the bat object with the basic background image for output according to the determined positions of the character and the bat object. If it is a level swing, it is almost parallel to the ground. If it is a downswing, the head part is tilted to the ground. If it is an upper swing, the head part is tilted to the top. The image of the bat object is combined with the basic background image for output so as to match the position. FIGS. 5A and 5B each show an example of a bat object and an output basic background image in which the bat object is synthesized. In addition to the bat object, an accessory image such as a hat or a uniform may be synthesized on the user's body. In addition, a batter box not selected by the user at the time of composition may be cut out from the basic background image for output and omitted. In FIG. 5B, the left batter batter box is omitted.

  If the operation unit 5 or the user's arm is hidden behind the object and is not shown in the photographed image, the arm position may not be accurately grasped from the photographed image, and the position of the bat object may not be known. Then, the position is determined based on the operation determination result, and the bat object can be synthesized at a more appropriate position.

  The imaging control unit 11 instructs the camera 200 to determine the imaging timing in response to the operation determination unit 16 detecting the start (takeback) of the swinging operation of the operation unit 5. The camera 200 starts shooting in response to an instruction from the imaging control unit 11 and transmits the obtained image data to the game apparatus 100.

  An imaging timing instruction by the imaging control means 11 may be one time or plural, and continuous shooting is repeated at a predetermined continuous shooting interval / predetermined shutter speed in accordance with an instruction of one imaging timing. May be performed.

  The shooting may be started when the swing motion of the operation unit 5 is detected, but may be too late to capture the instantaneous motion. Therefore, the imaging control means 11 is performed when the game has progressed to a predetermined scene. In the case of a baseball game, the imaging control unit 11 starts imaging in response to the ball character released by the pitcher character approaching the vicinity of the batter character in the game space in accordance with the control of the pitching control unit 13. Instruct. Alternatively, the imaging control means 11 simply instructs the start of shooting when a predetermined time (for example, 2 seconds) has elapsed since the pitcher character released the ball character, and during that period, the frame image is displayed at a predetermined frame rate. The photographing may be continued, and the end of photographing may be instructed at a predetermined time (for example, after one second) after the swing operation of the operation unit 5 is detected.

  Alternatively, when the operation detected by the operation unit 5 is a hitting preparation operation (takeback), the imaging control unit 11 instructs the start of imaging. The takeback is an operation for creating a “finger” in the horizontal direction, and a series of swing operation models (for example, as shown in FIG. 9) are registered in advance in the ROM 41 by the user via the operation unit 5 and image pickup control means. 11 compares the motion model and the feature quantity of the motion detected by the operation unit 5. If both match with a certain probability, it is determined that there is a takeback, and the start of imaging is instructed.

  Alternatively, the imaging control unit 11 starts / stops imaging in response to another user inputting a shooting start / stop instruction from a non-user operation unit 6 different from the operation unit 5 operated by the user by pressing a button or the like. May be indicated. This is effective when, for example, it is desired to take a picture of a batter waiting for a pitch. However, an output basic background image corresponding to an arbitrary input timing is prepared in the background DB 43 in advance, and the output image creation means 31 needs to extract this when creating the output image.

  The shooting interval for continuous shooting is set as follows. First, prior to the start of the game, the progress control means 19 displays a message or the like on the monitor 23 to prompt the user to perform an empty swing motion. In response to this, the user performs a swinging swing motion. The operation determination unit 16 measures the total swing operation time in response to the operation unit 5 detecting that the acceleration of the rotational operation exceeds zero (in response to the angular acceleration becoming positive). Is started, and the timing of the total swing operation time is ended in response to the fact that the operation unit 5 detects that the acceleration of the rotational operation has become zero. Then, the total swing operation time is measured and stored in the RAM 42. If it also serves to register the swing motion model, the detected motion trajectory and speed of the operation unit 5 may be stored together in the RAM 42. If the number of shots is specified by the setting value recorded in advance in the ROM 41 or the setting value input from the operation unit 5, the imaging control unit 11 shoots the number of image data during the total swing operation time. The shooting interval is determined as follows. If the number of shots is 6, and the total swing operation time is 1.5 seconds, 1.5 / 6 = 0.25 seconds is determined as the shooting interval, and this is instructed to the camera 200.

  Further, the imaging control unit 11 determines an optimum shutter speed according to the rotation speed detected by the operation unit 5. For example, if the detected rotation speed is about 120 km / h, the shutter speed is determined to be 1/100 s. If there is a restriction on the performance of the camera 200, the setting of the fastest shutter speed closest to 1 / 100s is instructed.

  4A and 4B show an example of a real space user and a captured image E that are subjects.

  The output image creation means 31 analyzes the image of the RAM 42 received from the camera 200, and the entire body of the user (the whole body; see, for example, Japanese Patent Laid-Open No. 2006-311505) or a part (face, for example, Japanese Patent Laid-Open No. 2008). -42801).

  In order to accurately extract the user's body in motion from the image data, the user's posture may be estimated based on the determination result by the operation determination means 16, and the user with the estimated posture may be extracted from the image data. For example, this is as follows. In the bat swing motion, the arm moves greatly because it rotates around the body axis, but the head does not move greatly. Therefore, as illustrated in FIG. 6, first, a known face detection detects a rectangular face area F having a width w × height h, and this face area F is detected by a predetermined length in a predetermined direction (for example, in the vertical direction). 4 times) to obtain an extension region PL. This may be determined based on the ratio of standard face length to overall body size. Then, a region including the closed region and the extended region PL formed by connecting the upper and lower ends E1 and E2 of the extended region PL and the detection position X of the operation unit 5 with line segments is defined as a user detection region. To do. When it is desired to extract only the upper body of the user, the extension area may be doubled from the face area F in the vertical direction. From this user detection area, a human body part is further extracted by edge detection or the like, and user image data is extracted. Since the user is indoors, if there are various interiors around the object, the edge component of the object and the person mixes to reduce the accuracy of the person extraction, but the person extraction area is limited by using the position of the operation unit 5. As a result, it is expected that the extraction accuracy of the image data in the shape of only the moving user is improved.

  In order to further improve the extraction accuracy, a skeleton model that represents a series of actions according to the type of game (if the game is baseball, for example, a bat swing as shown in FIG. 10 of JP-A-2001-204963) is prepared. Then, this is compared with the image data from the camera 200 to obtain a partial image (user extracted image) corresponding to the user. Specifically, for example, the position of the face of the skeleton model is aligned with the face area F, the position of the skeleton model hand during the bat swing operation is aligned with the position X of the controller, and the user body model area set based on this skeleton model Is extracted to obtain a user-extracted image. The type of game can be other than baseball, and in addition to sports with swing movements such as tennis and golf, sports that repeat similar movements such as marathons can be extracted by extracting them as a skeleton model. Is possible.

  Next, the output image creation means 31 combines a partial image corresponding to the user (user extracted image) with the output background image. At that time, the composite position of the partial image follows the reference position information. For example, in FIG. 7, the bottom of the user-extracted image is positioned and synthesized in the batter box Br designated for the right batter play.

  FIG. 8 is a flowchart showing a series of image processing executed by the CPU 10. Upon receiving a game start instruction from the controller 53 of the operation unit 5, the progress control means 19 starts game progress control (S1). At this time, the registration of the model swing action and the preceding / lasting of the user side team are determined. The progress control means 19 instructs the pitch control means 13 to start pitching by the CPU user when the attack inning of the user side team is reached. In response to this, the pitch control means 13 controls the pitcher character of the CPU user to perform a pitching action. The imaging control means 11 controls the imaging timing as described above (S2).

  The output image creation unit 31 estimates the position of the user subject according to the operation determination result of the operation determination unit 16 (S3), and extracts a partial image including all or part of the user subject from the image received from the camera 200. (S4).

  Further, the output image creation unit 31 determines the positions of the ball object and the bat object when the swinging motion of the operation unit 5 is detected from the operation determination of the operation determination unit 16. If it is a game of a type in which the coordinates of the user character in the game space are changed, the coordinates are converted into coordinates on the basic image for output (S5). The output image creation means 31 extracts a basic background image for output corresponding to the game scene in progress at the start of shooting from the background DB 43 (S6). However, this extraction may be in progress of the game or after the game is over. In order to extract after the game is over, the output image creation means 31 stores a flag indicating a scene at the time when it is determined that an instruction to start shooting is input in the RAM 42, and this flag is used after the game is over. An output basic background image corresponding to the displayed scene is extracted from the background DB 43. When a plurality of imaging instructions are input, a flag corresponding to each input may be stored, and an output basic background image for only the scene indicated by the flag selected from the operation unit 5 may be extracted. In this way, only necessary basic background images for output can be extracted, which is efficient. In order to extract in real time, the output image creating means 31 may immediately extract an output basic background image corresponding to a scene at the time when it is determined that an instruction to start photographing has been input from the background DB 43.

  Further, the output image creation means 31 composes the ball object and the bat object with the basic background image for output according to the determined positions of the character and the bat object, and creates an output image (S7).

  This output image is stored in the RAM 42. Thereafter, the data can be output to a printer 300 connected via the communication unit 4, a print order receiving server (not shown), a personal computer, a PDA, or other electronic device, and printed out or stored (S8).

  In this way, the user who is performing the game operation is captured instantaneously, and the user-extracted image is synthesized with the background corresponding to the game space in which the user exists at that moment, so that the user is in the game space. Can be created and output.

<Second Embodiment>
In the first embodiment, a type of game mainly involving a hitting operation has been described. However, even in a game involving an action other than a hit like a marathon, a user subject is accurately extracted from an image taken at a suitable timing, Can be combined with the game background.

  As illustrated in FIG. 10, the operation unit 5 is attached to the user's belt so as not to interfere with the user's stepping action.

  The progress control means 19 determines whether or not the operation detected from the operation unit 5 is a running operation (periodic vertical movement), and on a predetermined running route according to the pitch speed of the running operation. Controls the current position of the user runner character. That is, the speed of the vertical movement pitch is proportional to the running speed. Further, the object display control means 18 instructs the GPU 21 to display an image around the runner's current running point on a predetermined running route in accordance with the current position of the user runner character controlled by the progress control means 19. Also, it instructs the SPU 31 to perform appropriate BGM playback.

  The progress control means 19 determines the running status of the running route of the CPU character indicating a general runner other than the user runner character, and the object display control means 18 instructs the GPU 21 to display the general runner character accordingly. For example, the progress control means 19 assigns a running speed to each of one or more general runners (CPU characters) at random, and virtually maps the coordinates of the general runners along the running route from the start to the goal while maintaining the running speed. The object display control means 18 instructs the GPU 21 to display the character image of the general runner only when the current position of the general runner approaches the current position of the user runner character. When the general runner and the user runner are close to each other, the progress control means 19 may adjust the traveling speed of the general runner so that the general runner is pulled out of the user runner and makes a close contact.

  The imaging control means 11 inquires of the progress control means 19 periodically (for example, every second) whether or not the user runner character is trying to overtake or overtake the general runner. In response to this inquiry, the progress control means 19 determines whether the current position of the user runner character is close to a current position of a general runner within a predetermined range (behind the general runner, running in parallel, or And the result of the determination is returned to the imaging control means 11.

  The imaging control unit 11 instructs the camera 200 to perform imaging when the progress control unit 19 returns Yes (approaching). At that time, the user is probably desperately trying to pass the general runner, so it is suitable for creating an image that captures the state of the user. Alternatively, the imaging control means 11 inquires of the progress control means 19 whether or not the user runner starts from the starting point of the running route, before or after the user runner leaves the goal point of the running route, and so on. Depending on the response, the camera 200 may be instructed to shoot.

  The output image creating means 31 extracts from the background DB 43 an output basic background image having a landscape corresponding to the current run point of the runner on the predetermined running route managed by the progress control means 19 (see FIG. 11). .

  Further, the output image creation unit 31 extracts a user extracted image from the image data received from the camera 200. As in the first embodiment, a skeleton model representing a user's stepping action is prepared, and the position of the face of the skeleton model is aligned with the face region F, and the position of the hand of the skeleton model is aligned with the position X of the controller. An image data portion corresponding to the user body model region set based on the skeleton model is extracted to obtain a user extracted image (see FIG. 12).

  Then, the output image creation means 31 combines the user extracted image and the background image to obtain an output image (see FIG. 13). An accessory image such as a bib may be synthesized on the user's body. This composite image is stored in the RAM 42. Thereafter, the output image is stored in the RAM 42. Thereafter, the data can be output to a printer 300 connected via the communication unit 4, a print order receiving server (not shown), a personal computer, a PDA, or other electronic devices, and printed out or stored.

  In addition, the present invention can be applied to any game that can detect the user's action, such as golf, fishing, and cooking.

  In addition, the present invention is not limited to the concept of a “game” that competes for points, but may be any device that outputs a video related to a virtual space according to a scenario of video output of a character and a virtual world according to a user's operation. Anything can be applied. That is, the progress control means 19 does not handle only “games” that compete for scores.

Diagram showing the appearance of the video processing system Block diagram of video processing device The figure which shows an example of the background basic image for an output of 1st Embodiment. The figure which shows an example of the user picked-up image of 1st Embodiment, and a user extraction image The figure which shows an example of the background basic image for output which the object and the object were synthesize | combined The figure which shows an example of a user detection area | region The figure which shows an example of the image for an output of 1st Embodiment. Image processing flowchart Diagram showing an example of swing motion model The figure which shows an example of the user picked-up image of 2nd Embodiment. The figure which shows an example of the background basic image for output of 2nd Embodiment. The figure which shows an example of the user extraction image of 2nd Embodiment. The figure which shows an example of the image for an output of 2nd Embodiment.

Explanation of symbols

5: operation unit, 100: video processing device, 200: camera

Claims (11)

An operation detection unit for detecting a user's operation;
Based on the user action detected by the action detection unit, a progress management unit that controls the progress of the scene according to the scenario in the virtual space;
A storage unit for storing a basic background image for output corresponding to a scene in which the progress management unit can control the progress;
An imaging instruction unit that instructs a predetermined imaging device to start or end imaging;
An image extraction unit that extracts a basic background image for output corresponding to a scene controlled by the progress management unit when the imaging instruction unit instructs to start imaging, from the storage unit;
An image input unit that inputs an image acquired by the imaging device in response to an instruction from the imaging instruction unit from the imaging device;
An output image creation unit that creates an output image by synthesizing a part or all of the image input by the image input unit and the output basic background image extracted by the image extraction unit;
An information processing apparatus comprising: A position detection unit for detecting a user's operation position;
The output image creating unit synthesizes a part or all of the image input by the image input unit at a position on the output basic background image corresponding to the operation position of the user detected by the position detection unit. The information processing apparatus according to 1.   The information processing apparatus according to claim 1, wherein the basic background image for output includes an image capturing a user character from a desired viewpoint in a virtual space.   The information processing apparatus according to claim 3, wherein the basic background image for output includes an image that captures a user from a viewpoint opposite to a display video in a virtual space. An operation determination unit that determines whether or not a predetermined operation related to the progress of the scene has started or ended based on the user's operation detected by the operation detection unit;
With
The imaging instruction unit instructs the imaging device to start or end imaging in response to the operation determining unit determining that a predetermined operation related to the progress of the scenario has started or ended. 5. The information processing apparatus according to any one of 4. A user extraction unit that extracts a user's subject image from an image input by the image input unit based on a user operation detected by the operation detection unit and / or a user operation position detected by the position detection unit; ,
The output image creation unit synthesizes the user's subject image extracted by the user extraction unit and the output basic background image extracted by the image extraction unit, and creates an output image. An information processing apparatus according to claim 1.   The information processing apparatus according to claim 1, wherein the imaging instruction unit instructs the imaging apparatus to set a shooting interval and / or a shutter speed according to a user operation detected by the operation detection unit.   The information processing apparatus according to claim 1, wherein the imaging instruction unit instructs the imaging apparatus to start or end imaging in response to an input operation to an operation apparatus corresponding to a user other than the user.   The information processing apparatus according to claim 1, further comprising: an output unit that outputs the output image created by the output image creation unit to a desired electronic device. Computer
Detecting user actions; and
Controlling the progress of a scene along a scenario in a virtual space based on the detected user action;
Storing a basic background image for output according to a scene capable of controlling the progress;
Instructing a predetermined imaging device to start or end imaging;
Extracting a basic background image for output corresponding to a scene controlled at the time of instructing the start of imaging;
Inputting an image acquired by the imaging device in response to an instruction to start imaging from the imaging device;
Creating an output image by synthesizing a part or all of the input image and the extracted basic background image for output; and
Information processing method to execute. Detecting user actions; and
Controlling the progress of a scene along a scenario in a virtual space based on the detected user action;
Storing a basic background image for output according to a scene capable of controlling the progress;
Instructing a predetermined imaging device to start or end imaging;
Extracting a basic background image for output corresponding to a scene controlled at the time when the start of imaging is instructed;
Inputting an image acquired by the imaging device in response to the imaging start instruction from the imaging device;
Creating an output image by synthesizing a part or all of the input image and the extracted basic background image for output; and
An information processing program that causes a computer to execute.

Images