JP2019134204A - Imaging apparatus - Google Patents

Abstract

【Task】
The photographer automatically determines the composition suitable for the shooting location and does the shooting without focusing on the shooting.
[Solution]
Imaging means, storage means for storing a plurality of compositions, spatial information acquisition means for acquiring three-dimensional spatial information of a shooting location in advance before shooting, and position and orientation information acquisition means for sequentially acquiring position and orientation information of the imaging device And a selection means for selecting a composition suitable for shooting from the plurality of compositions, and an imaging control means for controlling the imaging means, wherein the imaging control means has a composition of an object scene captured by the imaging means, When it is determined that the composition selected by the selection means substantially matches, photographing is performed.
[Selection] Figure 1

Description

  The present invention relates to a method for detecting a decisive moment using three-dimensional information of a field.

  Conventionally, there is a need for a photographer to perform automatic shooting at a decisive moment without being conscious of camera shooting. For example, when taking a picture of the critical moment of your child in soccer, it has been difficult to watch it with the naked eye because it has been necessary to follow the child through the viewfinder. Therefore, there is a demand for a technique in which a camera automatically predicts a decisive moment and performs automatic shooting while watching with the naked eye.

  In automatic shooting, a composition determined by the positional relationship between the camera, the subject, and the background is important. For example, Patent Document 1 proposes a technique in which a desired composition is stored in advance in a moving robot camera so that the desired composition can be obtained even when the camera moves.

  On the other hand, in recent years, a technique that can acquire accurate three-dimensional information of a structure using a 3D laser scanner has been proposed. The three-dimensional information is point group data or 3D model data of color information such as three-dimensional position information and RGB data when a certain point in space is the origin. Hereinafter, in the present specification, the three-dimensional information is referred to as field information.

Japanese Patent No. 4833445

  The automatic photographing technique proposed in Patent Document 1 stores three-dimensional position information of a subject and calculates camera parameters that give a desired composition from the relative positional relationship with a moving camera. is there. In this method, when the position of the subject is fixed, shooting is performed with a desired composition, but when the subject moves violently in sports such as soccer, the subject moves off the screen and cannot be shot.

  In view of the above background and problems, an object of the present invention is to propose a technique for detecting a decisive moment using field information and performing automatic photographing.

  In order to solve the above problems, the present invention provides an imaging unit, a storage unit that stores a plurality of compositions, a spatial information acquisition unit that acquires in advance three-dimensional spatial information of a shooting location before shooting, and the imaging device. Position and orientation information acquisition means for sequentially acquiring position and orientation information, selection means for selecting a composition suitable for shooting from the plurality of compositions, and imaging control means for controlling the imaging means, the imaging control means, When it is determined that the composition of the object scene captured by the imaging unit substantially matches the composition selected by the selection unit, shooting is performed.

  According to the present invention, by detecting the decisive moment by applying the field information, it is possible to perform automatic shooting by determining a desired composition without the photographer focusing on shooting.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. The figure explaining the processing flow of Example 1. The figure explaining an example of the calculation method of a camera position and orientation The figure explaining an example of the image showing a decisive moment The figure explaining the subject extraction method using field information Example of composition corresponding to Tables 1-3

[Example 1]
In the first embodiment of the present invention, a use case is assumed in which a decisive moment is automatically photographed with a soccer ball (hereinafter referred to as a ball camera) loaded with a camera. The application of the present embodiment is not limited to the ball camera, but can be applied to all cameras whose composition can be changed, such as a normal hand-held camera, wearable camera, and robot camera.

  Hereinafter, an imaging apparatus suitable for the present embodiment will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus suitable for the first embodiment. The imaging unit 101 includes a lens and an imaging element, and outputs a subject image as an electrical signal. The camera signal processing unit 102 forms a video signal from the electrical signal output from the imaging unit 101. The camera signal processing circuit 102 includes an A / D conversion circuit, an auto gain control circuit (AGC), and an auto white balance circuit (not shown), and forms a digital signal. The three-dimensional spatial information holding unit 103 holds field information.

  The position / orientation calculation unit 104 calculates the position / orientation of the camera based on the output of the camera signal processing unit 102 and the three-dimensional space information obtained from the three-dimensional space information unit 103. The composition information holding unit 105 holds composition information representing a decisive moment. This composition information is held in a format associated with the output of the position / orientation calculation unit 104. The composition information selection unit 106 selects composition information from the composition information holding unit 105 based on the output of the position / orientation calculation unit 104. The composition information detection unit 107 detects composition information from the image data output from the camera signal processing unit 102.

  The deterministic moment determination unit 108 determines whether or not it is a definitive moment based on the composition information selected by the composition information selection unit 106 and the composition information obtained from the composition information detection unit 107. The image recording unit 109 records the image data output from the camera signal processing unit 102 when the deterministic moment determination unit 108 determines that it is a decisive moment.

  FIG. 2 shows an operation flow of the image processing apparatus according to the first embodiment of the present invention. First, in step S <b> 201, a captured image is acquired from the camera signal processing unit 102.

  Next, in step S <b> 202, field information is acquired from the three-dimensional spatial information holding unit 103. As described above, field information is a collection of point data in which position information (x, y, z) and color information (R, G, B) are uniquely determined when an origin in space is determined. (Hereinafter referred to as a point cloud).

  In the present embodiment, it is a point cloud representing surrounding structures such as a ground for playing soccer or a spectator seat. The field information can be acquired in advance using, for example, a laser scanner.

  In step S203, the position / orientation calculation unit 104 calculates the position / orientation of the camera using the captured image acquired in step S201 and the field information acquired in step S202. An example of a method for calculating the position and orientation of the camera will be described with reference to FIG. First, feature points are extracted from the captured image and feature quantities are described. As a method for extracting feature points from the captured image, a known corner detection technique can be used.

  In FIG. 3A, 301 indicates a captured image, 302 indicates a feature point extracted from the captured image, and 303 indicates a camera coordinate system. The coordinates of the extracted feature points are expressed in the camera coordinate system. Here, 302 is expressed as p = (u, v).

  On the other hand, in FIG. 3B, 304 indicates a feature point included in the field information, 305 indicates a world coordinate system, and 306 indicates a point obtained by projecting 304 on the image. 304 is expressed in the world coordinate system, and 306 is expressed in camera coordinates. Here, 304 is represented as S = (x, y, z), and 306 is represented as p ′ = (u ′, v ′).

  Calculating the position and orientation of the camera is nothing but finding the correspondence between the camera coordinate system and the world coordinate system. This correspondence is expressed using a camera external parameter matrix M. Hereinafter, this matrix M is referred to as camera position and orientation change. The camera position / orientation change M is expressed by Equation 1 using the rotation matrix R and the translation vector t.

  Here, the feature quantities 302 and 306 may be associated by nearest neighbor search, and M may be calculated such that the error between the coordinates p 302 and the coordinates p ′ of 306 is minimized. As an index of error, for example, the sum of squares of distances can be used. p ′ is obtained by the following equations 2 and 3 using S.

  Here, (x ′, y ′, z ′) is a coordinate representing S in the camera coordinate system, and f is a focal length of the camera.

As described above, the camera position / orientation can be calculated as a change in camera position / orientation from the origin of the field based on the field information and the captured image. If the initial position / orientation M0 of the camera can be calculated, the current position / orientation _Mn of the camera can be efficiently calculated by using the camera position / orientation _Mn-1 (initial value is M0) one frame before. Is possible. For example, assuming that the amount of change between M _n−1 and M _n is small, the correspondence range of the feature points 302 and 306 can be narrowed down.

  Further, when the camera position and orientation are calculated at higher speed and higher accuracy, information on GPS and gyro sensors can be used as an initial value for calculating M. Further, there may be a case where no feature point is included in the captured image. For example, when only the ground is shown. In such a case, a method of arranging an artificial marker in the field in advance can be considered. If an invisible marker is used, it is possible to arrange it without damaging the landscape.

  Next, in step S204, it is determined whether there is a difference from the calculation result of the previous frame for the camera position and orientation calculated in step S203. If the camera is stationary, no change in position and orientation occurs, so there is no difference. When the position and orientation of the camera has changed, the process proceeds to step S205. When the camera position / posture does not change, the process proceeds to step S206.

  In step S205, the composition information selection unit 106 selects composition information from the composition information holding unit 105 based on the position and orientation of the camera calculated in step S203. The composition information is held in association with the position and orientation of the camera, and instructs what kind of image is determined as an image representing a decisive moment for each camera position and orientation.

  An example of an image representing a decisive moment will be described with reference to FIG. In FIG. 4A, 401 indicates the soccer ground, and 402, 403, and 404 indicate the position and orientation of the ball camera at different times. In 402, since a scene is assumed in which the player kicks the ball toward the goal, for example, as shown in FIG. 4B, the moment when the player kicks the ball toward the ball can be considered as a decisive moment.

  On the other hand, in 403, a scene is assumed in which the goalkeeper (hereinafter referred to as GK) stops the shot. For example, as shown in FIG. It is possible.

  The features of the ball camera as in this embodiment are that the distance between the camera and the player is close, the player's eyes are always directed at the camera, and shooting at various angles is born by the movement of the ball. It is done. These features make it possible to obtain a powerful video as compared to when shooting from outside the soccer ground, as well as to hide the decisive moment behind the player.

  Next, an example of composition information for designating an image representing a decisive moment will be described using Tables 1 to 3. In Table 1, it is assumed that the camera position and orientation 402 and the focal length in FIG. Further, the type of subject, the location of the subject, and the size of the subject are used as composition information. These pieces of information are information obtained by the composition information detection unit 107, and an acquisition method will be described later. The camera position / posture 402 assumes a scene in which the player kicks the ball as one decisive moment. Therefore, for example, “person” is selected as the type of subject. By doing so, in a scene where a person is not shown as a subject, it is not determined as a decisive moment.

  Next, the subject position means an area in the image where the subject is to be photographed. For example, the center of the screen is selected. By doing so, in a scene where the subject is reflected at the edge of the screen, it is not determined as a decisive moment. Furthermore, the size of the subject means the ratio of the subject to the screen. For example, 70% or more of the screen is selected. By doing so, in a scene where the size of the subject is small and less than 70% of the screen, it is not determined as a decisive moment.

  As described above, in Table 1, an image representing a decisive moment is designated by the type of subject, the location of the subject, and the size of the subject.

  An example of the composition corresponding to Table 1 is shown in FIG. In Table 2, in order to specify the decisive moment more accurately or to reflect the user's intention more accurately, in Table 2, subject motion and subject posture are added as composition information. These pieces of information are information obtained by the composition information detection unit 107, and an acquisition method will be described later.

  The movement of the subject means movement of the subject between a plurality of consecutive frames. For example, when shooting a scene in which the player kicks the ball, the movement “kick” is designated. By doing so, in a scene where the action of “kick” does not exist, it is not determined as a decisive moment.

  The posture of the subject means the posture of the subject in a certain frame. For example, at the moment of kicking the ball, the posture is to stand on the ground with one foot. Therefore, the posture “standing on one foot” is designated. In this way, when the player's feet are on the ground, it is not determined as a decisive moment.

  As described above, in Table 2, the image representing the decisive moment is designated by the motion of the subject and the posture of the subject in addition to the type of subject, the position of the subject, and the size of the subject.

  An example of a composition corresponding to Table 2 is shown in FIG. 6A is characterized in that the motion and posture of the subject are specified. In Table 3, in order to specify the decisive moment more accurately or to reflect the user's intention more accurately, in Table 3, the expression of the subject, subject distance, and pressure information are added as composition information. These pieces of information are information obtained by the composition information detection unit 107, and an acquisition method will be described later.

  Regarding the facial expression of the subject, for example, when shooting a scene where the player kicks the ball with a smile, the facial expression “smile” is designated. In this way, in a scene where there is no player's smile, it is not determined as a decisive moment. In addition, with respect to the subject distance, the ball camera has a feature that the distance from the player always approaches or moves away. In the scene where the player kicks the ball, the distance becomes zero at the moment of kicking, and then the distance from the player increases.

  In order to shoot a player's kicking moment from the ball camera, there is an appropriate distance range corresponding to the focal length, and it is only necessary to specify the distance range. By doing so, in a scene where the distance to the player is too close or too far away, it is not determined as a decisive moment. Furthermore, with respect to pressure information, a large amount of pressure is applied to the ball camera at the moment the player kicks the ball. In a scene where a player kicks a ball, the pressure becomes maximum at the moment of kicking, and then the pressure decreases.

  As in the case of the subject distance, in order to capture the moment when the player kicks from the ball camera, there is an appropriate pressure range corresponding to the focal length. By doing so, in a scene where the pressure applied to the ball camera is too large or too small, it is not determined as a decisive moment.

  As described above, in Table 3, an image representing a decisive moment is represented by subject expression, subject distance, pressure information in addition to subject type, subject position, subject size, subject action, subject posture. Specified.

  An example of the composition corresponding to Table 3 is shown in FIG. A feature of FIG. 6B is that the facial expression of the subject is designated. Although not shown in FIG. 6C, subject distance and pressure information are also taken into account.

  Here, there may be a plurality of composition information associated with one camera position and orientation. For example, in the camera position / posture 402, in addition to kicking the ball with a foot as one decisive moment, a scene where the ball is headed can be considered. In that case, separate motion and orientation candidates for the subject are prepared. For example, as the action, the ball is hit with the head, so “head butt” is selected. As the posture, it is conceivable to head while flying in the air, so “floating both feet” is selected. Further, since it is considered that the pressure on the ball camera is smaller than when the ball is kicked with the foot, a lower pressure is selected.

  Furthermore, in the camera position / posture 402, if the focal length is tele, the state of GK may be captured. For example, a scene in which GK skips an instruction toward a player is assumed as one decisive moment. Therefore, “GK” is selected as the subject type and “pointing” is selected as the posture. In addition, in a soccer ground, countless composition information representing a decisive moment is scattered. For example, in the camera position and orientation 403 in FIG. 4A, a scene where GK jumps onto a ball that is likely to enter the goal and saves is assumed as a decisive moment.

  Therefore, as the composition information for the input camera position / posture 403, as shown in Table 5, the subject type is selected as “GK”, the motion is “jump”, and the posture is “arm extension”, which is decisive. It will be judged as a moment. In addition, in the camera position / posture 404 of FIG. 4A, a scene where a player throws in is assumed as a decisive moment. Therefore, the composition information for the input camera position / posture 404 is selected such that the type of subject is “player” and the action is “throw”, and this is determined to be a decisive moment.

  In this way, it is possible to instantly determine the decisive moment by narrowing down the composition information scattered innumerably in the field based on the field information and the camera position and orientation. The composition information may be held in association with the camera position and orientation by machine learning. If there is no composition information associated with the camera position and orientation calculated in step S203, the process proceeds to S203 without selecting composition information, or is associated with the closest camera position and orientation. What is necessary is just to transfer to S206 so that composition information may be referred.

  In step S206, the composition information detection unit 107 detects composition information from the captured image acquired in step S201. As described above, the composition information includes subject expression, subject distance, and pressure information in addition to the type of subject, the position of the subject, the size of the subject, the motion of the subject, and the posture of the subject. For example, if the subject is a person, a known person detection technique can be used. Also, personal authentication such as detecting GK is possible by registering the faces of participating players in advance. The position of the subject is calculated as, for example, the barycentric coordinates of the circumscribed rectangle of the detected subject. The size of the subject is calculated by, for example, a ratio between the area of the circumscribed rectangle of the subject and the area of the entire screen.

  The motion and posture of the subject can be detected by using a known gesture authentication technique. An example of the gesture authentication technique will be described. First, a distance image is acquired for each frame. The distance image can be detected by using a distance measuring sensor or from a captured image by a known stereo matching method.

  Next, based on the acquired distance image, a humanoid region is extracted and classified into body parts in units of pixels. For the classification, a boosting learning algorithm such as AdaBoost can be used. Then, based on the body part, the arrangement (joint position) of the part in the three-dimensional space is estimated so that the kinematic constraint and the temporal consistency are maintained. Gesture authentication can be performed by comparing the displacement of the three-dimensional position of the joint position with a desired motion pattern to be authenticated.

  The expression of the subject can be detected by a known expression recognition technique. For example, in smile recognition, a face area is first detected by face detection. Next, feature points are detected in the face region, and the coordinates of the end points of both eyes and mouth are specified. The characteristics of smiles include narrowing eyes, lowering the corners of the eyes, wrinkles around the mouth, and raising the corners of the mouth. In order to capture these features, feature quantities are calculated in the vicinity of the feature point coordinates. Then, the facial expression is identified from the feature amount. For identification, a boosting learning algorithm such as AdaBoost can be used.

  The subject distance can be detected by using a distance measuring sensor. Or it can detect with a well-known stereo matching method from a captured image. The pressure on the ball camera can be detected by using a pressure sensor. Here, if the field information is used, subject extraction can be performed with high accuracy and at high speed. For example, using the field information obtained in step S202 and the camera position and orientation obtained in step S203, a background image can be generated by a known method.

  FIG. 5 is a schematic diagram for extracting the main subject from the background image and the captured image. In FIG. 5, a subject that does not exist in the field information is extracted from the difference image (c) obtained by taking the difference between the background image (a) and the captured image (b).

  In step S207, the composition information narrowed down in step S205 is compared with the composition information detected in step S206. If the composition information matches (substantially matches), it is determined that the captured image captures a decisive moment, and the process proceeds to step S208. If the composition information does not match, the process proceeds to step S203. Finally, in step S208, the image recording unit 109 records the captured image acquired in step S201.

  As described above, a decisive moment can be determined and an image can be recorded. In this embodiment, only when the composition information matches in S207, the image is recorded in S208. However, the image is always recorded, and whether or not the composition information matches is tagged in S207. The image may be edited based on the tag.

  As described in the present embodiment, there are an infinite number of decisive moments in a soccer ground, but when the position and orientation of the camera are determined, the decisive moments can be appropriately narrowed down.

  As a result, the definitive moment can be captured by waiting for the narrowed decisive moment and recording the image when the captured image matches the condition.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  In the above, an example of a ball camera and a soccer ground has been described as a combination of a camera and a shooting location, but the present invention is not limited to this. The ball camera can be replaced with a wearable camera worn by players or a camera held by the audience. In the case of a ball camera, the position and orientation of the camera change in conjunction with the movement of the ball.

  On the other hand, in the case of a wearable camera, the position and orientation of the camera change in conjunction with the movement of the player. The camera held by the spectator changes its position and orientation in conjunction with the shooting position and direction of the spectator.

  In either case, composition information representing a decisive moment can be narrowed down from field information and camera position and orientation changes. For example, in a wearable camera, when the player is near the goal, a scene in which the defender comes to take the ball is assumed as the decisive moment. In the case of a camera held by a spectator, when shooting with the camera facing the goal direction, a scene in which the GK stops the ball that is likely to enter the goal is assumed as a decisive moment.

  By defining these critical moments as composition information as described above, the effects of the present invention can be obtained. In addition, the shooting place called a soccer ground can be replaced with another place such as a home or a park.

  For example, at home or in a park, the present invention can be used for the purpose of capturing a decisive moment of a subject such as a child or a pet that is difficult to predict behavior.

  As an example, a scene where a cat yawns on a sofa at home is assumed as a decisive moment, and this is defined as composition information as described above.

  Similarly, a moment when a child slides down from a slide in a park is assumed as a critical moment, and this is defined as composition information as described above. If the above-described ball camera is used as a camera for photographing, the effects of the present invention can be obtained. As described above, the ball camera may be replaced with a camera held by a person or a mobile robot camera.

  As described above, in the present invention, the combination of the camera and the shooting location is not limited, and the decisive moment is automatically detected by using the field information and the camera position and orientation in all the cameras whose composition can change. Can be understood.

DESCRIPTION OF SYMBOLS 101 Image pick-up part 102 Camera signal processing part 103 Three-dimensional spatial information holding part 104 Position and orientation calculation part 105 Composition information holding part 106 Composition information selection part 107 Composition information detection part 108 Deterministic instantaneous determination part 109 Image recording part

Claims (5)

An imaging device that captures an image when the composition of the object scene captured by the imaging means matches a predetermined composition,
Storage means for storing a plurality of compositions;
Spatial information acquisition means for acquiring three-dimensional spatial information of the shooting location in advance before shooting;
Position and orientation information acquisition means for sequentially acquiring the position and orientation information of the imaging device;
Selection means for selecting a composition suitable for photographing by the imaging device from a plurality of compositions stored in the storage means based on the spatial information and the position and orientation information;
An imaging control unit that controls the imaging unit to perform imaging when it is determined that the composition of the object scene captured by the imaging unit matches the composition selected by the selection unit;
An imaging apparatus comprising: The imaging apparatus according to claim 1,
The position and orientation information includes the position, orientation, and focal length of the image pickup apparatus at the shooting location. The imaging apparatus according to claim 1,
The imaging apparatus according to claim 1, wherein the selection unit selects a composition based on the three-dimensional space information of the shooting location and the subject information. The imaging apparatus according to claim 1,
It also has a pressure sensor,
The image pickup apparatus, wherein the selection unit selects a composition based on a pressure applied to the image pickup apparatus. The imaging apparatus according to claim 1,
When the imaging control means determines that the composition of the object scene captured by the imaging means matches the composition selected by the selection means, the imaging control means controls the imaging means to tag the captured image. An imaging device.