JP2019061502A - Information processing device, information processing method, and program - Google Patents

Abstract

To restrain cost relating to the calibration of a camera.SOLUTION: An information processing device 1 comprises: a calibration section 13 for, on the basis of the correspondence between a photographed image coordinate representing a positional coordinate in a photographed image and a looking-down map coordinate representing a positional coordinate in a looking-down map corresponding to a real space, identifying a conversion parameter for converting the photographed image coordinate to the looking-down map coordinate; and a coordinate conversion section 15 for converting the photographed image coordinate to the looking-down map coordinate. The coordinate conversion section converts the photographed image coordinate of a target object in a first photographed image captured at a first camera positional angle, to the looking-down map coordinate of the target object in the looking-down map. The calibration section identifies, on the basis of the correspondence between the photographed image coordinate of the target object in a second photographed image captured at a second camera positional angle and the looking-down map coordinate of the target object in the looking-down map, a second conversion parameter corresponding to the second camera positional angle.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  In order to map the position of an object included in a captured image captured by a camera arranged so as to look down obliquely from above onto an image viewed from directly above (herein referred to as an overhead map), projective transformation Transformation parameters such as a matrix are used. For example, when a plurality of objects are installed in the real space corresponding to the eyelid map, the conversion parameter is specified from the correspondence between the position coordinates of the plurality of objects in the captured image and the position coordinates of the plurality of objects in the eyelid map It is possible. Determining conversion parameters in this manner is called calibration.

  The calibration as described above may be performed only once, provided that the arrangement of the cameras is not changed. Further, Patent Document 1 below discloses a technique for calibrating camera parameters using images captured before and after the camera arrangement is changed.

JP, 2017-78923, A

  However, the technique described in the above-mentioned Patent Document 1 targets the change in the posture (angle) of the camera as the calibration target, and can not cope with the change in the camera position. Therefore, in the case of changing the arrangement of the camera or adding the camera for the purpose of changing or enlarging the range to be map-ized, as described above, the position coordinates of the plurality of objects in the captured image and the map of the plurality of objects It is necessary to recalibrate from the correspondence with the position coordinates in. Under the present circumstances, in order to obtain the position coordinate in the eyelid map of the said several object, the measurement in real space or installation of the object in the predetermined position of real space etc. were needed, and the cost was large.

  Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to provide a new and improved information processing apparatus capable of suppressing the cost involved in calibration, and information processing A method and a program are provided.

  In order to solve the above problems, according to one aspect of the present invention, captured image coordinates that are position coordinates in a captured image captured by a camera, and eyelid map coordinates that are position coordinates in an eyelid map corresponding to real space And a calibration unit that specifies a conversion parameter for converting the captured image coordinates into the overhead map coordinates based on the correspondence between the cameras, and the converted image coordinates using the conversion parameter. A coordinate conversion unit configured to convert the eyelid map coordinates, wherein the coordinate conversion unit is a first image captured by the first camera arrangement using a first conversion parameter corresponding to a first camera arrangement; Converting the captured image coordinates of the target object in the captured image into the overhead map coordinates of the target object in the overhead map; The unit further comprises: the captured image coordinates of the target object in a second captured image captured in a second camera configuration different from the first camera configuration; and the overhead map coordinates of the target object in the overhead map An information processing apparatus is provided which specifies a second conversion parameter corresponding to the second camera arrangement based on the correspondence.

  The information processing apparatus may further include a target object specifying unit that specifies the captured image coordinates of the target object.

  The target object identification unit detects the captured image coordinates of a target object candidate from the captured image, and captures the captured image coordinates of the detected target object candidate based on the determination of the operator. It may be specified as image coordinates.

  The information processing apparatus may further include a correction unit that corrects the conversion parameter specified by the calibration unit using a reference object with the eyelid map coordinates known.

  The correction unit is a known reference coordinate that is the known eyelid map coordinates of the reference object, and the eyelid map coordinates of the reference object obtained by the coordinate conversion unit converting the captured image coordinates of the reference object The conversion parameter may be corrected based on the distance.

  The correction unit may further include a determination unit that determines whether the distance is within a predetermined range, and may correct the conversion parameter according to a result of the determination by the determination unit.

  The correction unit determines the eyelid map coordinates of the target object based on a difference between the known reference coordinates and the conversion reference coordinates when the determination unit determines that the distance is not within the predetermined range. The image processing apparatus may further include a coordinate correction unit that corrects the conversion parameter based on the eyelid map coordinates of the target object corrected by the coordinate correction unit.

  The number of reference objects required for the correction unit to correct the conversion parameter may be smaller than the number of target objects required for the calibration unit to specify the second conversion parameter.

  The information processing apparatus may be configured to determine whether the number of target objects included in both the first captured image and the second captured image is smaller than a predetermined threshold value. In the sensor coordinate system corresponding to the position sensor, the eyelid map coordinates of the first target object in the eyelid map obtained by converting the captured image coordinates of the target object, the sensing of the position sensor The second target object in the eyelid map based on the position coordinates of the first target object and the position coordinates of a second target object in the sensor coordinate system obtained based on the sensing of the position sensor The information processing apparatus further includes a sensor information management unit that acquires the eyelid map coordinates, and the calibration unit is configured to receive the second target object in the second captured image. Based on the correspondence between the bird's-eye map coordinates of the second target object in serial captured image coordinates and the bird's eye map may identify a second conversion parameters.

  The transformation parameters may include a projection transformation matrix or a perspective projection transformation matrix.

  In order to solve the above problems, according to another aspect of the present invention, the correspondence between the captured image coordinates that are position coordinates in a captured image captured by a camera and the eyelid map coordinates that is position coordinates in an eyelid map And a calibration unit that specifies a conversion parameter for converting the captured image coordinates to the overhead map coordinates based on the arrangement of the cameras, and using the conversion parameter to convert the captured image coordinates to the overhead map An information processing method executed by an information processing apparatus including a coordinate conversion unit configured to convert coordinates into coordinates, which is imaged by the first camera arrangement using a first conversion parameter corresponding to a first camera arrangement. Converting the captured image coordinates of the target object in the first captured image into the overhead map coordinates of the target object in the overhead map And the captured image coordinates of the target object in a second captured image captured in a second camera configuration different from the first camera configuration and the overhead map coordinates of the target object in the overhead map Providing a second conversion parameter corresponding to the second camera arrangement based on the correspondence.

  In order to solve the above problems, according to another aspect of the present invention, the correspondence between the captured image coordinates that are position coordinates in a captured image captured by a camera and the eyelid map coordinates that is position coordinates in an eyelid map And a calibration function for specifying a conversion parameter for converting the captured image coordinates to the overhead map coordinates based on the arrangement of the cameras, and using the conversion parameter to convert the captured image coordinates to the overhead map A program for causing a computer to realize a coordinate conversion function of converting into coordinates, wherein the coordinate conversion function uses the first conversion parameter corresponding to a first camera arrangement to execute the first camera arrangement. And the captured image coordinates of the target object in the first captured image captured by The calibration function converts the captured image coordinates of the target object in a second captured image captured in a second camera arrangement different from the first camera arrangement and the eyelid map A program is provided for identifying a second transformation parameter corresponding to the second camera arrangement based on the correspondence with the eyelid map coordinates of the target object.

  As described above, according to the present invention, it is possible to suppress the cost involved in calibration.

It is a block diagram showing an example of composition of information processor 8 concerning a comparative example of an embodiment of the present invention. It is an explanatory view for explaining a method for acquiring correspondence with an image pick-up picture coordinate and a map map coordinate. It is a block diagram showing an example of composition of information processor 1 concerning a 1st embodiment of the present invention. It is a flowchart figure which shows the operation | movement outline which concerns on the embodiment. It is a flowchart figure which shows the detailed operation | movement of initial stage calibration. It is an explanatory view for explaining initial calibration. It is an explanatory view for explaining initial calibration. It is a flowchart figure which shows the detailed operation | movement of recalibration which concerns on the embodiment. It is an explanatory view for explaining re-calibration concerning the embodiment. It is an explanatory view for explaining re-calibration concerning the embodiment. It is a block diagram showing an example of composition of information processor 2 concerning a 2nd embodiment of the present invention. It is a flowchart figure which shows the operation | movement outline which concerns on the embodiment. It is a flowchart figure which shows the detailed operation | movement of correction | amendment of conversion parameter. It is an explanatory view for explaining correction of a conversion parameter. It is an explanatory view for explaining correction of a conversion parameter. It is a block diagram which shows the structural example of the information processing apparatus 3 which concerns on the 3rd Embodiment of this invention. It is a flowchart which shows an example of the operation | movement which concerns on the recalibration which concerns on the embodiment. It is an explanatory view for explaining re-calibration concerning the embodiment. It is an explanatory view for explaining re-calibration concerning the embodiment. It is an explanatory view showing hardware constitutions of an information processor 1000 concerning an embodiment of the present invention.

  The present invention will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown. In the present specification and the drawings, components having substantially the same functional configuration will be assigned the same reference numerals and redundant description will be omitted.

  Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by attaching different alphabets to the same reference numerals. However, when it is not necessary to distinguish each of a plurality of components having substantially the same functional configuration, only the same reference numerals will be given.

<< 1. Background >>
In describing the embodiments of the present invention, first, the background that led to the creation of the embodiments of the present invention will be described.

  2. Description of the Related Art In construction sites, commercial facilities, etc., the position of an object such as a person is detected or the position is tracked from a captured image captured by a camera. In order to be viewed and grasped by humans and used for various analyzes, etc., it is desirable to obtain position coordinates in an eyelid map corresponding to real space and viewed from directly above. However, since it is difficult to arrange the camera so as to look down from directly above, a camera arranged so as to look down obliquely from the upper side is often used.

  Therefore, mapping of the position coordinates of an object obtained from a captured image taken by a camera arranged to look down from such a diagonal upper side onto a map of the eyelid is performed (transformed to position coordinates in the map of eyelid) ing. For such mapping, transformation parameters such as a projection transformation matrix or a perspective projection transformation matrix are used.

  For example, assuming that a projection transformation matrix from a captured image to an eyelid map is H and position coordinates in the captured image are G (x, y), position coordinates F (x, y) in the eyelid map are obtained by the following equation (1) It is possible.

  The conversion parameter (for example, the projective transformation matrix) from the captured image to the overhead map is a position coordinate in the captured image (hereinafter sometimes referred to as the captured image coordinate) and a position coordinate in the overhead map (hereinafter referred to as the overhead map coordinate) It is possible to obtain | require by using two or more correspondence with (it may). The determination of the conversion parameter from the captured image to the overhead map in this way is hereinafter referred to as overhead map calibration, or simply calibration.

  As described above, in order to perform the calibration, it is necessary to prepare a plurality of correspondences between the captured image coordinates and the eyelid map coordinates. Therefore, for example, calibration is performed by preparing a plurality of correspondences between captured image coordinates and eyelid map coordinates using a plurality of objects installed in a real space corresponding to the eyelid map.

  Hereinafter, an object used for calibration in this manner is referred to as a target object. The target object is not limited to an object newly installed for calibration, and, for example, an object already existing in real space may be used as the target object.

  Here, with reference to FIG. 1 and FIG. 2, an example of the information processing apparatus that performs calibration using a plurality of target objects in this way will be described as a comparative example of the embodiment of the present invention.

  FIG. 1 is a block diagram showing an example of the configuration of an information processing apparatus 8 according to the present comparative example. As shown in FIG. 1, the information processing apparatus 8 according to this comparative example includes a control unit 810, an image acquisition unit 860, an operation unit 870, an output unit 880, and a storage unit 890. The information processing apparatus 8 according to the present comparative example is a information processing apparatus that performs calibration for obtaining a conversion parameter from an imaged image to an eyelid map, and obtains an eyelid map coordinate of an object shown in the imaged image using the projective transformation matrix. is there. Hereinafter, a schematic configuration of the information processing device 8 will be described.

  The control unit 810 controls each component of the information processing device 8. Further, as illustrated in FIG. 1, the control unit 810 according to the present comparative example also functions as an object position specifying unit 811, a calibration unit 813, and a coordinate conversion unit 815.

  The object position specifying unit 811 specifies position coordinates (captured image coordinates) in the captured image of an object shown in the captured image provided from the image acquisition unit 860 that acquires the captured image. Although a method of specifying captured image coordinates by the object position specifying unit 811 is not particularly limited, for example, captured image coordinates may be specified based on a user operation using the operation unit 870, or captured image coordinates may be specified using image processing technology. It may be done.

  The object position specifying unit 811 provides the captured image coordinates to the calibration unit 813 or the coordinate conversion unit 815. For example, when performing an operation relating to calibration, the object position specifying unit 811 provides the captured image coordinates of the target object to the calibration unit 813. In addition, after calibration is completed, when it is desired to obtain an eyelid map position of an object shown in a captured image, the object position specifying unit 811 provides the captured image coordinates of the object to the coordinate conversion unit 815.

  The calibration unit 813 specifies a conversion parameter for converting the captured image coordinates to the map coordinates based on the correspondence between the captured image coordinates and the map coordinates of the map. For example, if the transformation parameter is a projective transformation matrix, the projective transformation matrix can be specified using the above equation (1) if the correspondence between four or more sets of captured image coordinates and eyelid map coordinates is obtained. It is. For example, the calibration unit 813 specifies (calculates a 3 × 3 projective transformation matrix (an example of transformation parameters) by solving simultaneous equations obtained from the correspondences between four or more sets of captured image coordinates and map coordinates of an eyelid. May).

  Here, a method for obtaining the correspondence between the captured image coordinates and the eyelid map coordinates will be described with reference to FIG. FIG. 2 is an explanatory diagram for explaining a method for obtaining correspondence between captured image coordinates and eyelid map coordinates.

FIG. 2 shows a real space R100 viewed from directly above, in which the camera 9 and target objects T101 to T104 are installed. Further, as shown in FIG. 2, the real space coordinate system with the origin point O _R in the real space R100 is set.

  An image captured by the camera 9 is, for example, like a captured image G100 shown in FIG. Position coordinates (captured image coordinates) P101 to P104 of target objects T101 to T104 in the captured image G100 are specified by the object position specifying unit 811 as described above.

Moreover, the overhead map corresponding to real space R100 becomes like the overhead map M100 shown in FIG. Bird's-eye map M100 has an origin _{O M} corresponding origin _{O R} in the real space R100, each position in the overhead map M100 with each location in the real space R100 is compatible with a predetermined scale. Therefore, in order to obtain the position coordinates (eyelid map coordinates) in the eyelid map M100 of the target objects T101 to T104, for example, it is sufficient to know the real space positions of the target objects T101 to T104.

  The target objects T101 to T104 may be placed, for example, at predetermined positions where the real space position is known. Alternatively, when the target objects T101 to T104 are installed at positions where the real space position is unknown, the real space positions of the target objects T101 to T104 may be measured by a known positioning method. The positioning method may be, for example, a method performed manually or an automatic method using a positioning device such as a GPS receiver. Information on the real space position of the target object may be input through a communication unit (not shown) or may be input by the operation of the operation unit 870 by the user (operator).

  Information on the real space position of the target object obtained in this manner is provided to the calibration unit 813, and the calibration unit 813 sets the eye map coordinates F101 to F104 of the target object T101 to T104 based on the real space position. It is possible to get Note that the user may specify overhead map coordinates from the real space position, and in such a case, the user may operate the operation unit 870 to directly input overhead map coordinates.

  As described above, the calibration unit 813 can use the correspondence between the captured image coordinates P101 to P104 and the eyelid map coordinates F101 to F104 for calibration. The conversion parameter specified by the calibration unit 813 is stored in the conversion parameter DB 891 stored in the storage unit 890.

  Returning to FIG. 1, the description will be continued. The coordinate conversion unit 815 performs coordinate conversion to convert captured image coordinates provided from the object position specifying unit 811 into eyelid map coordinates using the conversion parameter. The coordinate conversion unit 815 performs such coordinate conversion using the conversion parameter stored in the conversion parameter DB 891 stored in the storage unit 890, that is, the conversion parameter obtained by the calibration of the calibration unit 813 described above. It is also good. The eyelid map coordinates of the object obtained by the coordinate conversion of the coordinate conversion unit 815 are output from, for example, the output unit 880.

  The schematic configuration of the information processing apparatus 8 according to the present comparative example has been described above. As described above, in order to perform calibration by the information processing device 8, it is necessary to obtain correspondence between captured image coordinates and eyelid map coordinates, and to obtain eyelid map coordinates, the real space position of the target object is obtained. You need to get it.

  In order to obtain the real space position of the target object, it is necessary to set the position at the predetermined position as described above, to perform positioning manually, or to use a positioning device such as a GPS receiver. When a positioning device is used, the manufacturing cost of the target object is increased. In some cases, it is impossible to use a positioning device indoors or underground.

  On the other hand, placing the target object at a predetermined position or manually positioning leads to an increase in the working cost for calibration. If it is premised that the arrangement of the camera is not changed, one calibration is sufficient, but the arrangement of the camera may be changed or a new camera may be added. For example, at the construction site, it is conceivable to change the arrangement of the camera when the construction range moves, or to add the camera when the construction range becomes wide. In addition, also in a commercial facility, it is conceivable to change the arrangement of cameras or add cameras according to the condition of the floor and the like.

  In the case where the camera arrangement is changed or the camera is added in this way, it is necessary to perform calibration each time the camera arrangement is changed or the camera is added. Therefore, placing the target object at a predetermined position or manually positioning it every time the camera is changed or added to the camera greatly increases the operation cost.

  Therefore, the inventor of the present invention has made the embodiment of the present invention, focusing on the above circumstances. The information processing apparatus according to each embodiment of the present invention described below stores the eyelid map coordinates of the target object obtained in the calibrated camera arrangement, and uses it for calibration in the new camera arrangement. It is possible to suppress Below, each embodiment of the present invention which realizes the effect concerned is described in detail one by one.

<< 2. Detailed Description of Each Embodiment >>
<2-1. First embodiment>
(Overview)
First, an outline of the first embodiment of the present invention will be described. The information processing apparatus according to the present embodiment is a target object in a captured image captured by a camera arrangement in which a conversion parameter has been specified, that is, a camera arrangement in which calibration has already been performed (hereinafter may be referred to as a first camera arrangement). The captured image coordinates of are converted into overhead map coordinates. The information processing apparatus according to the present embodiment temporarily stores the eyelid map coordinates obtained in this manner, and performs, for example, a change in camera arrangement or an addition of a camera. Then, in the information processing apparatus according to the present embodiment, the captured image of the target object in the captured image captured by the camera layout after changing the camera layout or after adding the camera (hereinafter may be referred to as a second camera layout) Calibration is performed based on the correspondence between the coordinates and the stored map coordinates of the eye.

  By performing the calibration in this manner, it is possible to suppress the cost involved in the calibration in the second camera arrangement. Hereinafter, a configuration example of the information processing apparatus according to the present embodiment will be described with reference to FIG.

(Example of configuration)
FIG. 3 is a block diagram showing an exemplary configuration of the information processing apparatus 1 according to the first embodiment of the present invention. As shown in FIG. 3, the information processing apparatus 1 according to the present embodiment includes a control unit 10, an image acquisition unit 60, an operation unit 70, an output unit 80, and a storage unit 90.

  The control unit 10 controls each component of the information processing apparatus 1. Further, as shown in FIG. 3, the control unit 10 according to the present embodiment also functions as an object position specifying unit 11, a calibration unit 13, and a coordinate conversion unit 15. The functions of the object position specifying unit 11, the calibration unit 13, and the coordinate conversion unit 15 that the control unit 10 has will be described later.

  The image acquisition unit 60 is an interface that is connected to a camera (not shown) and acquires a captured image captured by the camera. The image acquisition unit 60 may be connected to a plurality of cameras to acquire captured images from the plurality of cameras. The image acquisition unit 60 provides the acquired captured image to the control unit 10.

  Note that the information processing apparatus 1 itself may include a camera. In such a case, the information processing apparatus 1 may include an imaging unit including a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), instead of or in addition to the image acquisition unit 60.

  The operation unit 70 receives an input by a user (operator) of the information processing apparatus 1. For example, the operation unit 70 may be realized by a mouse, a keyboard, a touch panel, a button, a switch, a lever, a dial, or the like.

  The output unit 80 is, for example, a display, and displays and outputs the position and the like of the object in the eyelid map according to the purpose. For example, if the object is stationary, the position of the object may be displayed and output on the map, and if the object is moving, the locus may be displayed and output based on the map coordinates of each frame. Good.

  The storage unit 90 stores programs and data used for the operation of the information processing apparatus 1. In addition, the storage unit 90 stores a conversion parameter DB 91 and a position coordinate DB 93 as shown in FIG.

  The conversion parameter DB 91 stores conversion parameters for converting captured image coordinates into overhead map coordinates, which are provided from a calibration unit 13 described later. The position coordinate DB 93 stores the eyelid map coordinates of the target object provided by the coordinate conversion unit 15 described later. The storage unit 90 according to the present embodiment is different from the storage unit 890 shown in FIG. 1 in that the position coordinate DB 93 is stored.

  The overall configuration of the information processing apparatus 1 according to the present embodiment has been described above. Subsequently, functions of the object position specifying unit 11, the calibration unit 13, and the coordinate conversion unit 15 included in the control unit 10 included in the information processing device 1 will be described.

  The object position specifying unit 11 specifies captured image coordinates of an object (including a target object) shown in the captured image provided from the image acquisition unit 60. The object position specifying unit 11 may specify the captured image coordinates of the object based on the user operation using the operation unit 870, or may specify the captured image coordinates by image processing technology. For example, when the captured image coordinates are specified by image processing technology, the object position specifying unit 11 detects an object from the captured image using a method such as template matching, and the lowermost position in the region of the object in the captured image. The coordinates may be specified as captured image coordinates of the object.

  Further, the object position specifying unit 11 may detect the captured image coordinates of the target object candidate from the captured image, and output (for example, display) the captured image coordinates of the detected target object candidate to the output unit 80. The user (operator) who has confirmed the output of the output unit 80 may determine the target object from among the target object candidates via the operation unit 70. Then, based on the determination of the operator via the operation unit 70, the object position specifying unit 11 may specify the determined captured image coordinates of the target object candidate as the captured image coordinates of the target object.

  With such a configuration, captured image coordinates of the target object can be identified with higher accuracy. If there is an error in specifying the captured image coordinates of the target object, there is a possibility that the calibration accuracy may be greatly affected. Therefore, the calibration accuracy can be improved by specifying the captured image coordinates of the target object with high accuracy. .

  The calibration unit 13 specifies a conversion parameter for converting the captured image coordinates to the map coordinates based on the correspondence between the captured image coordinates and the map coordinates of the map. The transformation parameters specified by the calibration unit 13 may be, for example, a projection transformation matrix. The calibration unit 13 can specify, for example, a projection transformation matrix in the same manner as the calibration unit 813 according to the comparative example illustrated in FIG.

  When the calibration unit 13 according to the present embodiment performs calibration in the camera arrangement when the camera is first installed, for example, the real space position of the target object or the eyelid map coordinates by the user operation via the operation unit 70 It may be input.

  However, when the eyelid map coordinates of the target object are stored in the position coordinate DB 93 of the storage unit 90, the calibration unit 13 according to the present embodiment is the eyelid of the target object stored (stored) in the position coordinate DB 93. Perform calibration using map coordinates. The eyelid map coordinates of the target object stored in the position coordinate DB 93 are eyelid map coordinates obtained by coordinate conversion performed by the coordinate conversion unit 15 described later.

  The coordinate conversion unit 15 performs coordinate conversion to convert captured image coordinates provided from the object position specifying unit 11 into eyelid map coordinates using conversion parameters. The coordinate conversion unit 15 performs such coordinate conversion using the conversion parameter stored in the conversion parameter DB 91 stored in the storage unit 90, that is, the conversion parameter obtained by the calibration of the calibration unit 13 described above. It is also good. The eyelid map coordinates of the object obtained by the coordinate conversion of the coordinate conversion unit 15 are output from the output unit 80, for example.

  In addition, the coordinate conversion unit 15 according to the present embodiment uses the first conversion parameter corresponding to the first camera arrangement to pick up an image of the target object in the first pickup image picked up by the first camera arrangement. The image coordinates are transformed into the eye map coordinates of the target object in the eye map. Further, the coordinate conversion unit 15 provides the eyelid map coordinates obtained in this manner to the storage unit 90, and stores the same in the position coordinate DB 93.

  With such a configuration, the calibration unit 13 uses the eyelid map coordinates of the target object stored in the position coordinate DB 93 as described above when performing calibration in the second camera arrangement different from the first camera arrangement. It becomes possible. That is, the calibration unit 13 determines between the captured image coordinates of the target object in the second captured image captured by the second camera layout different from the first camera layout and the overhead map coordinates of the target object in the overhead map. Based on the correspondence, it is possible to specify a second conversion parameter corresponding to the second camera arrangement.

  According to the configuration, when performing calibration in the second camera arrangement, it is not necessary to perform positioning of the target object in the real space, so the operation cost is suppressed.

(Operation example)
The configuration example of the information processing apparatus 1 according to the present embodiment has been described above. Subsequently, an operation example of the present embodiment will be described with reference to FIGS.

  FIG. 4 is a flowchart showing an outline of operation according to the present embodiment. As shown in FIG. 4, first, calibration (hereinafter referred to as initial calibration) for determining conversion parameters in the camera arrangement when the camera is first installed is performed (S10). Details of the initial calibration in step S10 will be described later with reference to FIGS.

Subsequently, when the camera layout is changed or added (YES in S12), calibration for obtaining conversion parameters in the camera layout after the camera layout change or after the camera addition (hereinafter referred to as recalibration) is It is performed (S14). The detailed operation of the recalibration in step S14 will be described later with reference to FIGS.

  On the other hand, when neither the arrangement change of the camera nor the addition is performed (NO in S12), the operation ends. Although not shown, as described above, the process of displaying and displaying the position of an object such as a person appearing in the captured image may be included between the step S10 and the step S12. .

  The outline of the operation according to the present embodiment has been described above. Subsequently, the initial calibration of step S10 shown in FIG. 4 will be described in more detail with reference to FIGS. FIG. 5 is a flowchart showing the detailed operation of the initial calibration. 6 and 7 are explanatory diagrams for explaining the initial calibration.

  FIG. 6 shows a real space R1 and a captured image G1 captured by a camera 5A installed in the real space R1. Further, FIG. 7 shows a real space R1 and an overhead map M1 corresponding to the real space R1. In addition, the area | region MA has shown the area | region corresponding to the imaging range of the camera 5A among the eyelid map M1. Further, the real space R1 shown in FIG. 6 is a plan view seen from directly above, and the real space R1 shown in FIG. 7 is a side view seen from the side.

  As shown in FIG. 5, first, target objects T1 to T4 are placed in the real space R1 (S100). Subsequently, the real space positions of the target objects T1 to T4 are measured, and the eyelid map coordinates F1 to F4 respectively corresponding to the real space positions of the target objects T1 to T4 are input by a user operation via the operation unit 70, for example. (S102).

  Subsequently, the image acquisition unit 60 acquires a captured image G1 captured by the camera 5A (S104). In addition, after the captured image G1 is acquired, the target objects T1 to T4 may be removed. Subsequently, the object position specifying unit 11 specifies captured image coordinates P1 to P4 of the target objects T1 to T4 (S106).

  Subsequently, the calibration unit 13 specifies conversion parameters based on the correspondence between the captured image coordinates P1 to P4 and the eyelid map coordinates F1 to F4 (S108). The conversion parameter identified in step S108 is stored (stored) in the conversion parameter DB 91 of the storage unit 90 (S110).

  The initial calibration of step S10 has been described above. Subsequently, the recalibration in step S14 shown in FIG. 4 will be described in more detail with reference to FIGS. FIG. 8 is a flowchart showing the detailed operation of the recalibration according to the present embodiment. 9 and 10 are explanatory diagrams for explaining the recalibration according to the present embodiment.

  9 and 10 show an example in which recalibration is performed first after the initial calibration is performed, that is, an example in which the camera layout change or addition is performed first. ing. When the arrangement change of the camera is performed, the camera 5A and the camera 5B shown in FIGS. 9 and 10 may be the same camera 5, and the camera 5A indicates the camera 5 before the arrangement change, and the camera 5B It may be understood that the camera 5 is shown after the arrangement change. Further, when the camera is added, the camera 5A and the camera 5B are different cameras, and the camera 5B is added.

  Since calibration in the camera arrangement of the camera 5A has already been completed in step S10, the camera arrangement of the camera 5A will be referred to as the first camera arrangement described above, and the camera arrangement of the camera 5B as the second camera arrangement described above. .

  In FIG. 9, a first captured image G11 captured in the first camera arrangement by the real space R10 and the camera 5A installed in the real space R10, and a second captured image by the camera 5B installed in the real space R10. The second captured image G12 captured in the camera arrangement is shown. Further, FIG. 10 shows a real space R10, and a weir map M11 and a weir map M12 corresponding to the real space R10. In the overhead map M11, the area MA indicates the area corresponding to the imaging range of the camera 5A, and the area MB on the overhead map M12 indicates the area corresponding to the imaging range of the camera 5B. Further, the real space R10 shown in FIG. 9 is a plan view seen from directly above, and the real space R10 shown in FIG. 10 is a side view seen from the side. The eyelid map M11 and the eyelid map M12 shown in FIG. 10 have the same coordinate system as the eyelid map M1 shown in FIG.

  As shown in FIG. 8, first, target objects T11 to T14 are placed in the real space R10 (S140). Subsequently, the image acquisition unit 60 acquires a first captured image G11 captured by the camera 5A in the first camera arrangement (S142). Subsequently, the object position specifying unit 11 specifies captured image coordinates P111 to P114 of the target objects T11 to T14 in the first captured image G11, respectively (S144).

  Subsequently, using the first conversion parameter stored (stored) in the conversion parameter DB 91 of the storage unit 90, the coordinate conversion unit 15 converts the captured image coordinates P111 to P114 into the eyelid map coordinates F11 to F14, respectively. (S146). The first conversion parameter used in step S146 is a conversion parameter corresponding to the camera arrangement (first camera arrangement) of the camera 5A, and is, for example, the conversion parameter stored in step S110 shown in FIG. May be

  The eyelid map coordinates F11 to F14 obtained in step S146 are stored (stored) in the position coordinate DB 93 of the storage unit 90 (S147).

  Subsequently, the arrangement of the camera or the addition of a camera is performed (S148). Subsequently, the second captured image G12 captured by the camera 5B in the second camera arrangement is acquired by the image acquisition unit 60 (S150). Subsequently, the object position specifying unit 11 specifies captured image coordinates P121 to P124 of the target objects T11 to T14 in the second captured image G12, respectively (S152).

  Subsequently, the calibration unit 13 performs a second conversion based on the correspondence between the captured image coordinates P121 to P124 specified in step S152 and the eyelid map coordinates F11 to F14 stored in the position coordinate DB 93 in step S147. The parameters are identified (S154). The second conversion parameter specified in step S154 is stored (stored) in the conversion parameter DB 91 of the storage unit 90 (S156).

  The recalibration in step S14 has been described above. Note that, as described with reference to FIG. 4, the recalibration in step S14 is repeated until camera repositioning or camera addition is not performed. Therefore, the camera arrangement (second camera arrangement) of the camera 5B described with reference to FIGS. 8 to 10 and the specified second conversion parameter are the first when the recalibration is performed next time. And the first conversion parameter.

  Further, as shown in FIG. 10, the target objects T11 to T14 described above are arranged so as to be included in the imaging ranges of both the camera 5A and the camera 5B. As shown in FIG. 8, when the arrangement of the cameras is changed or the camera is added after the target objects T11 to T14 are arranged, the camera 5B is set so that the target objects T11 to T14 are included in the imaging range of the camera 5B. The installation of should be done.

(Supplement)
The first embodiment of the present invention has been described above. According to the present embodiment, by using the eyelid map coordinates of the target object obtained in the first calibrated camera arrangement for calibration in the second camera arrangement, calibration for the second camera arrangement is performed. It is possible to suppress such costs. Although the example in which the number of target objects is four has been described above, the number of target objects may be different depending on the conversion parameter and the number necessary for the method of coordinate conversion. In addition, the number of target objects may be larger than the number required for conversion parameters and coordinate conversion methods, and such a configuration may enable more accurate calibration.

2-2. Second embodiment>
(Overview)
In the first embodiment, an example has been described in which recalibration is repeatedly performed each time camera relocation or camera addition is performed. However, when the number of repetitions of recalibration increases, an error may be accumulated between the eyelid map coordinates of the target object obtained by coordinate conversion and the eyelid map coordinates corresponding to the real space position of the target object, which may increase. is there.

  Thus, an example in which correction is performed using an object whose position in the real space is known, that is, the eyelid map coordinates are known (hereinafter, referred to as a reference object) will be described as a second embodiment.

(Example of configuration)
FIG. 11 is a block diagram showing a configuration example of the information processing device 2 according to the second embodiment of the present invention. The information processing apparatus 2 according to the present embodiment includes a control unit 20, an image acquisition unit 60, an operation unit 70, an output unit 80, and a storage unit 90, as shown in FIG. Note that among the components shown in FIG. 11, components having substantially the same functional configuration as the components shown in FIG. 3 are given the same reference numerals, and thus the description thereof is omitted.

  The control unit 20 controls each component of the information processing device 2. Further, as shown in FIG. 11, the control unit 20 according to the present embodiment also functions as an object position specifying unit 21, a calibration unit 23, a coordinate conversion unit 25, and a correction unit 27.

  The object position specifying unit 21 specifies, as in the case of the object position specifying unit 11 described with reference to FIG. 3, captured image coordinates of an object shown in a captured image provided from the image acquisition unit 60. However, the captured image coordinates identified by the object position identifying unit 21 differ from the object position identifying unit 11 in that the captured image coordinates identified by the object position identifying unit 21 include captured image coordinates of the reference object. The object position specifying unit 21 may specify the captured image coordinates of the reference object by the same method as the method of identifying the captured image coordinates of the target object.

  Similar to the calibration unit 13 described with reference to FIG. 3, the calibration unit 23 converts the captured image coordinates into map coordinates based on the correspondence between the captured image coordinates and map map coordinates. Identify the parameters However, in addition to the function of the calibration unit 13, the calibration unit 23 is a function of specifying conversion parameters as correction candidates using the corrected eyelid map coordinates of the target object provided by the correction unit 27 described later. Have.

  Similar to the coordinate conversion unit 15 described with reference to FIG. 3, the coordinate conversion unit 25 uses coordinate conversion to convert the captured image coordinates provided from the object position specifying unit 21 into eyelid map coordinates. Have a function to However, in addition to the function of the coordinate conversion unit 15, the coordinate conversion unit 25 has a function of converting captured image coordinates of an object into eyelid map coordinates under the control of the correction unit 27 described later. The coordinate conversion unit 25 according to the present embodiment converts the captured image coordinates of the reference object into overhead map coordinates using the conversion parameter stored in the conversion parameter DB 91 under the control of the correction unit 27 described later. Further, the coordinate conversion unit 25 according to the present embodiment converts captured image coordinates of the reference object into eyelid map coordinates using conversion parameters as correction candidates under the control of the correction unit 27 described later.

  The correction unit 27 corrects the conversion parameter specified by the calibration unit 23 using a reference object whose eyelid map coordinates are known. As shown in FIG. 11, the correction unit 27 includes a distance specifying unit 271, a determination unit 273, a coordinate correction unit 275, and a correction control unit 277.

  The distance specifying unit 271 obtains the known eyelid map coordinates of the reference object (hereinafter referred to as known reference coordinates) and the eyelid map coordinates of the reference object obtained by converting the captured image coordinates of the reference object by the coordinate conversion unit 25. The distance to (hereinafter referred to as conversion reference coordinates) is specified. The distance specified by the distance specifying unit 271 may be, for example, a Euclidean distance between the known reference coordinates and the conversion reference coordinates. However, the distance specified by the distance specifying unit 271 is not limited to the Euclidean distance.

  The determining unit 273 determines whether the distance specified by the distance specifying unit 271 is within a predetermined range.

  When the determination unit 273 determines that the distance is not within the predetermined range, the coordinate correction unit 275 determines the difference between the known reference coordinates and the conversion reference coordinates, that is, based on the vector from the conversion reference coordinates to the known reference coordinates. And correct the eyelid map coordinates of the target object. For example, the coordinate correction unit 275 sets the target so that the vector from the eyelid map coordinates before correction of the target object to the eyelid map coordinates after correction is the same as or substantially the same as the vector from the conversion reference coordinates to the known reference coordinates. Correction of the eyelid map coordinates of the object may be performed. That is, the coordinate correction unit 275 may obtain the after-correction map coordinates of the target object by adding the difference between the known reference coordinates and the conversion reference coordinates to the before-correction overhead map coordinates of the target object.

  The correction control unit 277 controls the operations of the distance specifying unit 271, the determination unit 273, and the coordinate correction unit 275 described above. Further, the correction control unit 277 controls part of the operations of the calibration unit 23 and the coordinate conversion unit 25 in order to correct the conversion parameter.

  For example, the correction control unit 277 controls the coordinate conversion unit 25 to convert the captured image coordinates of the reference object into conversion reference coordinates using the conversion parameter stored in the conversion parameter DB 91. Further, the correction control unit 277 controls the distance specifying unit 271 to specify the distance using the conversion reference coordinates obtained by the coordinate conversion unit 25.

  Furthermore, the correction control unit 277 controls the determination unit 273 to determine whether the distance specified by the distance specification unit 271 is within a predetermined range. Then, the correction control unit 277 corrects the conversion parameter in accordance with the determination result of the determination unit 273.

  For example, when the determination unit 273 determines that the distance is not within the predetermined range, the correction control unit 277 controls the coordinate correction unit 275 to correct the eyelid map coordinates of the target object. Further, the eyelid map coordinates of the target object corrected by the coordinate correction unit 275 are provided to the calibration unit 23, and conversion parameters as correction candidates are specified.

  Note that the correction control unit 277 repeats the series of processes described above until the determination unit 273 determines that the distance is within the predetermined range, and a correction candidate such that the distance finally falls within the predetermined range The conversion parameter which becomes the above may be stored in the position coordinate DB 93 of the storage unit 90 as a corrected conversion parameter.

  In addition, in the case where the coordinate conversion unit 25 converts the captured image coordinates of the reference object into conversion reference coordinates in the second and subsequent processing in the repetition, the conversion unit is specified by the calibration unit 23 as a conversion parameter. Conversion parameters as correction candidates are used.

  Also, there may be a plurality of reference objects. As the number of reference objects is larger, it may be possible to perform more accurate correction. However, if there is at least one or more reference objects, the correction unit 27 can perform the correction of the conversion parameters described above. That is, the number of reference objects required for the correction unit 27 to correct the conversion parameters is smaller than the number (at least 4 or more) of target objects required for the calibration unit 23 to specify the second conversion parameters. Good. Therefore, it is possible to perform the above-described correction with less operation cost than positioning the eyelid map coordinates of all target objects.

(Operation example)
The configuration example of the information processing apparatus 2 according to the present embodiment has been described above. Subsequently, an operation example of the present embodiment will be described with reference to FIGS. 12 to 15.

  FIG. 12 is a flowchart showing the outline of the operation according to this embodiment. As shown in FIG. 12, first, initial calibration is performed to obtain conversion parameters in the camera arrangement when the camera is first installed (S20). The initial calibration in step S20 is the same as the initial calibration in the first embodiment described with reference to FIGS. 5 to 7, and thus detailed description will be omitted.

Subsequently, when the camera layout change or addition is performed (YES in S22), recalibration is performed to obtain conversion parameters in the camera layout after the camera layout change or after the camera addition (S24). The recalibration in step S24 is the same as the recalibration in the first embodiment described with reference to FIGS. 8 to 10, and thus detailed description will be omitted.

  Subsequently, in step S26, it is determined whether or not correction is to be performed. The determination as to whether to perform the correction may be performed based on, for example, a user operation via the operation unit 70, or may be automatically performed based on a predetermined condition. For example, correction may be performed when the reference object appears in the captured image when recalibration in step S24 is performed, or when the number of repetitions of recalibration in step S24 reaches a predetermined number. A correction may be made.

  If it is determined that the correction is not to be performed (NO in step S26), the process returns to step S22. On the other hand, when it is determined that the correction is to be performed (YES in step S26), the correction unit 27 corrects the second conversion parameter obtained in the recalibration in step S24 (S28). The detailed operation of step S28 will be described later with reference to FIGS. 13 to 15.

  On the other hand, when neither arrangement change of the camera nor addition is performed (NO in S22), the operation ends. Although not shown, a process of displaying and outputting the position of an object such as a person appearing in a captured image may be included between the step S20 and the step S22 on a map.

  The outline of the operation according to the present embodiment has been described above. Subsequently, the correction of step S28 shown in FIG. 12 will be described in more detail with reference to FIGS. FIG. 13 is a flowchart showing a detailed operation of correction of conversion parameters. 14 and 15 are explanatory diagrams for explaining the correction of the conversion parameter.

  In FIGS. 14 and 15, the camera arrangement of the camera 5A is the first camera arrangement described above, and the camera arrangement of the camera 5B is the second camera arrangement described above. Further, when the arrangement of the cameras is changed in the recalibration in step S20, the cameras 5A and 5B shown in FIGS. 14 and 15 may be the same camera 5, and the cameras 5A are arranged It may be understood that the camera 5 B before the change is shown and the camera 5 B shows the camera 5 after the change of arrangement. In addition, in the case where a camera is added in the recalibration in step S20, the cameras 5A and 5B are different cameras, and the camera 5B is a camera to which the camera 5B is added.

  In FIG. 14, the real space R20, the captured image G21 captured in the first camera arrangement by the camera 5A installed in the real space R20, and the second camera arrangement by the camera 5B installed in the real space R20 The captured image G22 captured is shown. Further, FIG. 15 shows a real space R20, and a weir map M21 and a weir map M22 corresponding to the real space R20. In the overhead map M21, the area MA indicates an area corresponding to the imaging range of the camera 5A, and in the overhead map M22, the area MB indicates an area corresponding to the imaging range of the camera 5B. The real space R20 shown in FIG. 14 is a plan view seen from directly above, and the real space R20 shown in FIG. 15 is a side view seen from the side.

  In addition, at the time before the operation of step S28 described below is performed, the target objects T21 to T24 have already been arranged in the recalibration at step S24. Further, in the recalibration in step S24, captured image coordinates P211 to P214 of the target objects T21 to T24 in the captured image G21 and captured image coordinates P221 to P224 of the target objects T21 to T24 in the captured image G22 have already been specified. . Further, in the recalibration in step S24, the eyelid map coordinates F21 to F24 of the target objects T21 to T24 are already obtained by the conversion using the first conversion parameter.

  As shown in FIG. 13, first, a known eyelid map coordinate C25 (known reference coordinate) corresponding to the real space position of the reference object B20 is input by a user operation via the operation unit 70, for example (S282). The known reference coordinates may be stored in advance in the storage unit 90.

  Subsequently, the object position specifying unit 21 specifies a captured image coordinate P220 of the reference object B20 in the captured image G22 (S283).

  Subsequently, the coordinate conversion unit 25 converts the captured image coordinates P220 of the reference object B20 into overhead map coordinates F25 (conversion reference coordinates) using the conversion parameter (S284). When step S284 is performed first, the coordinate conversion unit 25 may use the second conversion parameter stored in the conversion parameter DB 91 of the storage unit 90 as the conversion parameter. When step S284 is performed for the second time or later, the coordinate conversion unit 25 may use, as a conversion parameter, a conversion parameter serving as a correction candidate identified in step S292 described later.

  Subsequently, the distance specifying unit 271 of the correction unit 27 specifies the distance between the input eyelid map coordinates C25 (known reference coordinates) and the eyelid map coordinates F25 (conversion reference coordinates) obtained by the conversion (S286) .

  Subsequently, the determination unit 273 of the correction unit 27 determines whether the distance specified in step S286 is within a predetermined range (S288).

  If it is determined that the distance is not within the predetermined range (NO in S288), the coordinate correction unit 275 of the correction unit 27 corrects the eyelid map coordinates F21 to F24 of the target object T21 to T24 (S290). For example, in step S290, the coordinate correction unit 275 causes the vector from the eyelid map coordinates F21 to F24 before correction to the eyelid map coordinates C21 to C25 after correction to be eyelid map coordinates C25 from the eyelid map coordinates F25 (conversion reference coordinates). Correction of the eyelid map coordinates of the target object may be performed so as to be the same as or substantially the same as the vector to the known reference coordinates.

  Subsequently, based on the correspondences between the captured image coordinates P221 to P224 already obtained in step S24 and the post-correction map coordinates C21 to C25 obtained in step S290, the calibration unit 23 sets correction candidates as correction candidates. Conversion parameters are identified (S292).

  Under the control of the correction control unit 277 of the correction unit 27, the processes of steps S284 to S292 described above are repeated until it is determined in step S288 that the distance is within the predetermined range. Then, when it is determined that the distance is within the predetermined range (YES in S288), the correction control unit 277 corrects the conversion parameter as the correction candidate specified in step S292 last performed at that time. It is stored (stored) in the conversion parameter DB 91 of the storage unit 90 as the acquired second conversion parameter (S294).

  The correction of the conversion parameter in step S28 has been described above. In the example shown in FIGS. 14 and 15, the reference object B20 is included in the imaging range of the camera 5A, but the reference object B20 only needs to be included in the imaging range of the camera 5B. It does not have to be included in the imaging range of.

(Supplement)
The second embodiment of the present invention has been described above. According to the present embodiment, even when an error occurs due to repeated recalibration, the operation cost is suppressed by correcting the conversion parameters using a smaller number of reference objects than the number of target objects. While, it becomes possible to specify conversion parameters with high accuracy.

  In the second embodiment described above, when it is determined that the correction can not be performed, the correction control unit 277 controls the output unit 80 to notify the user that the correction is impossible. You may Then, when notified that the correction is impossible, the calibration may be performed by the same method as the initial calibration described with reference to FIGS. 5 to 7.

  For example, the correction control unit 277 can not perform correction when it is not determined that the distance is within the predetermined range in step S288 even if the processes of steps S284 to S292 illustrated in FIG. 13 are repeated a predetermined number of times or more. It may be determined that it is possible. Alternatively, the correction control unit 277 performs correction when the distance specified by the distance specifying unit 271 is larger than a predetermined range (a range larger than the predetermined range used in the determination of step S 288 shown in FIG. 13). It may be determined that is impossible.

  With such a configuration, even if an error that can not be corrected occurs due to repeated recalibration, the user is notified as such, and calibration is performed in the same manner as in the initial calibration, and high accuracy is achieved. It is possible to specify various conversion parameters.

<2-3. Third embodiment>
(Overview)
In the first and second embodiments described above, a common target object is used in the first camera arrangement and the second camera arrangement. That is, the number of target objects included in both the first captured image captured by the first camera arrangement and the second captured image captured by the second camera arrangement is required to specify the conversion parameter. If the number is less than 10%, recalibration can not be performed. Therefore, for example, in the case where there is no overlap in the imaging range in the first camera arrangement and the imaging range in the second camera arrangement, or when the overlap is very small, the first embodiment and the second embodiment will be described. It was not possible to do such a recalibration.

  Therefore, in the following, as a third embodiment, by using a position sensor such as light detection and ranging (LiDAR), a common target object is not used for the first camera arrangement and the second camera arrangement. An example of recalibration will be described.

(Example of configuration)
FIG. 16 is a block diagram showing a configuration example of the information processing device 3 according to the third embodiment of the present invention. The information processing device 3 according to the present embodiment includes a control unit 30, a sensor information acquisition unit 50, an image acquisition unit 60, an operation unit 70, an output unit 80, and a storage unit 90, as shown in FIG. Note that among the components shown in FIG. 16, components having substantially the same functional configuration as the components shown in FIG. 3 are given the same reference numerals, and thus the description thereof is omitted.

  The control unit 20 controls each component of the information processing device 3. The control unit 30 according to the present embodiment also functions as an object position specifying unit 31, a calibration unit 33, a coordinate conversion unit 35, and a sensor information management unit 39, as shown in FIG.

  The object position specifying unit 31 specifies, as in the case of the object position specifying unit 11 described with reference to FIG. However, in addition to the function of the object position specifying unit 11, the object position specifying unit 31 includes a first pickup image obtained by the first camera arrangement and a second pickup image obtained by the second camera arrangement. And the function of specifying the number of target objects (hereinafter referred to as a common target object) included in both. For example, the number of common target objects may be identified by an image recognition method, or may be identified by a user operation via the operation unit 70. The object position specifying unit 31 notifies the sensor information management unit 39 of the number of common target objects specified.

  Similar to the calibration unit 13 described with reference to FIG. 3, the calibration unit 33 performs conversion for converting the captured image coordinates into the map coordinates based on the correspondence between the captured image coordinates and the map coordinates with the map Identify the parameters

  However, in addition to the function of the calibration unit 13, the calibration unit 33 has a function of specifying conversion parameters according to the control of a sensor information management unit 39 described later.

  For example, the calibration unit 33 has a function of specifying a conversion parameter for converting position coordinates (hereinafter referred to as sensor coordinates) in the sensor coordinate system corresponding to the position sensor into the map coordinate in accordance with the control of the sensor information management unit 39 Have. The sensor coordinate system corresponding to the position sensor is, for example, a coordinate system which differs from the coordinate system of the eyelid map in terms of scale and angle in the horizontal direction (Z-axis direction). Therefore, it is possible to specify a conversion parameter for converting sensor coordinates into map coordinates by using the correspondence between two or more sets of sensor coordinates and map coordinates. The calibration unit 33 may use, for example, an eyelid map coordinate in an eyelid map obtained by converting an imaged image coordinate of a target object (hereinafter, referred to as a first target object) included in a first imaged image. Such conversion parameters may be specified based on the correspondence with the sensor coordinates of the target object of Note that the number of first target objects is, for example, 2 or more from the above-described condition. Further, the calibration unit 33 may store (store) conversion parameters for converting sensor coordinates into overhead map coordinates in the conversion parameter DB 91 of the storage unit 90.

  In addition, under the control of the sensor information management unit 39, the calibration unit 33 controls the captured image coordinates of the target object (hereinafter referred to as a second target object) included in the second captured image, and the second in the eyelid map. A second transformation parameter is identified based on the correspondence with the eyelid map coordinates of the target object. When the transformation parameter is a projective transformation matrix, the number of second target objects is four or more.

  Similar to the coordinate conversion unit 15 described with reference to FIG. 3, the coordinate conversion unit 35 uses coordinate conversion to convert the captured image coordinates provided from the object position specifying unit 21 into eyelid map coordinates. Have a function to However, in addition to the function of the coordinate conversion unit 15, the coordinate conversion unit 35 converts the sensor coordinates of the second target object included in the second captured image into bar map coordinates under the control of the sensor information management unit 39 described later. , And has a function to be provided to the sensor information management unit 39. When the coordinate conversion unit 35 converts the sensor coordinates of the second target object into eyelid map coordinates, the calibration unit 33 specifies and converts the sensor coordinates stored in the conversion parameter DB 91 into eyelid map coordinates. Conversion parameters may be used.

  The sensor information management unit 39 manages sensor information acquired by the sensor information acquisition unit 50 described later from the position sensor, and controls execution of recalibration based on the sensor information. The sensor information includes, for example, sensor coordinates of the target object. The sensor coordinates of the target object may be obtained directly from the position sensor by the sensor information obtaining unit 50, or may be specified based on the information obtained from the position sensor. For example, when only the information on the position and the shape sensed by the sensor information acquired by the sensor information acquisition unit 50 from the position sensor is included, the sensor information management unit 39 determines the sensor coordinates of the target object from the information on the sensing position and the shape. May be identified. The sensor information management unit 39 may cause the output unit 80 to output sensor information, and specify sensor coordinates of a target object based on a user operation by the user who has confirmed the sensor information via the operation unit 70.

  The sensor information management unit 39 may execute recalibration based on the sensor information when the number of common target objects provided from the object position specifying unit 31 is smaller than a threshold. The predetermined threshold may be, for example, the number of sets of correspondence between the captured image coordinates required for the calibration unit 33 to specify the conversion parameter and the map coordinates of the ridge. For example, if the transformation parameter specified by the calibration unit 33 is a projection transformation matrix, the predetermined threshold may be 4.

  When the number of common target objects is smaller than the threshold value, the sensor information management unit 39 may control the calibration unit 33 to specify a conversion parameter for converting sensor coordinates into overhead map coordinates. Furthermore, the sensor information management unit 39 controls the coordinate conversion unit 35 to convert the sensor coordinates of the second target object included in the second captured image into the eye map coordinates using the conversion parameter, and the eye map. The coordinates may be acquired from the coordinate conversion unit 35. Then, the sensor information management unit 39 controls the calibration unit 33, and based on the correspondence between the captured image coordinates of the second target object and the eyelid map coordinates of the second target object, the second conversion parameter is calculated. It may be specified.

  With such a configuration, it is possible to suppress the operation cost for recalibration even when there are not a sufficient number of common target objects.

  The sensor information acquisition unit 50 is an interface that is connected to a position sensor (not shown) and acquires sensor information obtained by sensing of the position sensor. The sensor information acquisition unit 50 may be connected to a plurality of position sensors to acquire sensor information from the plurality of position sensors. The sensor information acquisition unit 50 provides the acquired sensor information to the control unit 30.

  Note that the information processing device 3 itself may include a position sensor. In such a case, the information processing apparatus 3 may include a position sensor such as LiDAR instead of or in addition to the sensor information acquisition unit 50.

(Operation example)
The configuration example of the information processing device 3 according to the present embodiment has been described above. Subsequently, an operation example of the present embodiment will be described. The operation outline of the present embodiment is substantially the same as the operation outline of the first embodiment described with reference to FIG. 4, and thus the description of the operation outline of the present embodiment will be omitted. The difference between the operation of the present embodiment and the operation of the first embodiment is the difference in the operation related to recalibration corresponding to step S14 shown in FIG. Therefore, in the following, the operation related to the recalibration according to the present embodiment will be described with reference to FIGS. 17 to 19.

  FIG. 17 is a flowchart showing an example of the operation related to the recalibration according to the present embodiment. FIGS. 18 and 19 are explanatory diagrams for describing recalibration according to the present embodiment.

  When the arrangement of the camera is changed, the camera 5A and the camera 5B shown in FIGS. 18 and 19 may be the same camera 5, and the camera 5A indicates the camera 5 before the arrangement change, and the camera 5B May be understood to indicate the camera 5 after the change of arrangement. Further, when the camera is added, the camera 5A and the camera 5B are different cameras, and the camera 5B is added. That is, the camera arrangement of the camera 5A is a first camera arrangement, and the camera arrangement of the camera 5B is a second camera arrangement.

  In FIG. 18, the real space R30, a first captured image G31 captured in a first camera arrangement by the camera 5A installed in the real space R30, and a second captured image by the camera 5B installed in the real space R30. The second captured image G32 captured in the camera arrangement is shown. Further, FIG. 19 shows a real space R30, and a weir map M31 and a weir map M32 corresponding to the real space R30. In the overhead map M31, the area MA indicates the area corresponding to the imaging range of the camera 5A, and in the overhead map M32, the area MB indicates the area corresponding to the imaging range of the camera 5B. The real space R30 shown in FIG. 18 is a plan view seen from directly above, and the real space R30 shown in FIG. 19 is a side view seen from the side.

  Moreover, as shown in FIG. 18, the position sensor 6 is installed in the real space R30. The position sensor 6 senses information of an object present in the range D30 in the real space R30 using LiDAR or the like, and provides the information to the information processing device 3.

  As shown in FIG. 17, first, target objects T31 to T36 are placed in the real space R30 (S302). Subsequently, the image acquisition unit 60 acquires a first captured image G31 captured by the camera 5A in the first camera arrangement (S304). Subsequently, the object position specifying unit 31 specifies the captured image coordinates P311 and P312 of the first target objects T31 and T32 in the first captured image G31 (S308).

  Subsequently, using the first conversion parameter stored (stored) in the conversion parameter DB 91 of the storage unit 90, the coordinate conversion unit 35 converts the captured image coordinates P311 and P312 into eyelid map coordinates F31 and F32, respectively. (S312). The eyelid map coordinates F31 and F32 obtained in step S312 are stored (stored) in the position coordinate DB 93 of the storage unit 90 (S313).

  Subsequently, the arrangement of the camera or the addition of the camera is performed (S314). Subsequently, the image acquisition unit 60 acquires a second captured image G32 captured by the camera 5B in the second camera arrangement (S316). Subsequently, the object position specifying unit 31 specifies captured image coordinates P323 to P326 of the second target objects T33 to T36 in the second captured image G32, respectively (S318).

  Subsequently, it is determined whether the number of common target objects included in both the first captured image G31 and the second captured image G32 is equal to or greater than a predetermined threshold (S320). In the example shown in FIG. 18 and FIG. 19, the number of common target objects is 0, which is smaller than the predetermined threshold (4 in the above example) (NO in S320), so the process proceeds to step S322.

  In step S322, the sensor information acquisition unit 50 acquires sensor information from the position sensor 6 (S322). Subsequently, under the control of the sensor information management unit 39, the calibration unit 33 determines, based on the correspondence between the sensor coordinates of the first target objects T31 and T32 included in the first captured image and the eye map coordinates F31 and F32. A conversion parameter for converting sensor coordinates into overhead map coordinates is specified (S324). Furthermore, the coordinate conversion unit 35 converts the sensor coordinates of the second target object T33 to T36 included in the second captured image G32 into eyelid map coordinates F33 to F36, respectively, according to the control of the sensor information management unit 39.

  Subsequently, based on the correspondence between the captured image coordinates P323 to P326 specified in step S318 and the eyelid map coordinates F33 to F36 obtained by the coordinate conversion in step S326, the calibration unit 33 performs a second conversion parameter. Are identified (S332). The second conversion parameter specified in step S332 is stored (stored) in the conversion parameter DB 91 of the storage unit 90 (S334).

  Although the above description is given along the example shown in FIG. 18 and FIG. 19, when it is determined in step S320 that the number of common target objects is larger than the threshold, the process proceeds to step S332. If it is determined in step S320 that the number of common target objects is greater than the threshold, the series of processes shown in FIG. 17 are the first embodiment described with reference to FIG. 8 except for the determination process of step S320. It is similar to the process of recalibration according to the embodiment.

(Supplement)
The third embodiment of the present invention has been described above. According to the present embodiment, by using the position sensor, a sufficient number of common target objects included in the captured image do not exist before and after the change in the camera arrangement or between the existing camera and the added camera. Even if it exists, it becomes possible to suppress the operation cost concerning recalibration.

  Note that although an example in which recalibration based on sensor information is performed when the number of common target objects is smaller than the threshold has been described above, the present embodiment is not limited to such an example. For example, re-calibration may always be performed based on the above-described sensor information, or whether or not re-calibration based on the sensor information may be switched by user operation via the operation unit 70.

  Moreover, it is also possible to combine the second embodiment described above with the third embodiment.

<< 3. Hardware configuration >>
The embodiments of the present invention have been described above. The information processing such as the object position specifying processing, the calibration processing, and the coordinate conversion processing described above is realized by cooperation of software and hardware of the information processing device 1, the information processing device 2, and the information processing device 3. The hardware configuration of the information processing apparatus 1000 will be described below as an example of the hardware configuration of the information processing apparatus 1, the information processing apparatus 2, and the information processing apparatus 3 that are information processing apparatuses according to the embodiment of the present invention.

  FIG. 20 is an explanatory view showing the hardware configuration of the information processing apparatus 1000 according to the embodiment of the present invention. As shown in FIG. 20, the information processing apparatus 1000 includes a central processing unit (CPU) 1001, a read only memory (ROM) 1002, a random access memory (RAM) 1003, an input device 1004, and an output device 1005. , Storage apparatus 1006, and communication apparatus 1007.

  The CPU 1001 functions as an arithmetic processing unit and a control unit, and controls the overall operation in the information processing apparatus 1000 according to various programs. The CPU 1001 may be a microprocessor. The ROM 1002 stores programs used by the CPU 1001, calculation parameters, and the like. The RAM 1003 temporarily stores programs used in the execution of the CPU 1001, parameters that appropriately change in the execution, and the like. These are mutually connected by a host bus configured of a CPU bus and the like. Mainly, the functions of the control unit 10, the control unit 20, the control unit 30, etc. are realized by the cooperation of the CPU 1001, the ROM 1002 and the RAM 1003 and the software.

  The input device 1004 generates an input signal based on an input by a user such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever, and an input control circuit that generates an input signal based on the input by the user. And so on. The user of the information processing apparatus 1000 can input various types of data to the information processing apparatus 1000 and instruct processing operations by operating the input device 1004. The input device 1004 corresponds to the operation unit 70.

  The output device 1005 includes, for example, a display device such as a liquid crystal display (LCD) device, an OLED device and a lamp. Furthermore, the output device 1005 includes an audio output device such as a speaker and headphones. For example, the display device displays a captured image, a generated image, and the like. On the other hand, the audio output device converts audio data and the like into audio and outputs the audio. The output device 1005 corresponds to the output unit 80.

  The storage device 1006 is a device for storing data. The storage device 1006 may include a storage medium, a recording device that records data in the storage medium, a reading device that reads data from the storage medium, and a deletion device that deletes data recorded in the storage medium. The storage device 1006 stores programs executed by the CPU 1001 and various data. The storage device 1006 corresponds to the storage unit 90.

  The communication device 1007 is, for example, a communication interface configured of a communication device or the like for connecting to a communication network. Also, the communication device 1007 may include a wireless local area network (LAN) compatible communication device, a long term evolution (LTE) compatible communication device, a wire communication device performing wired communication, or a Bluetooth (registered trademark) communication device.

<< 4. End >>
As described above, according to the embodiment of the present invention, it is possible to suppress the cost involved in the calibration accompanying the change in the arrangement of the cameras or the addition of the cameras.

  Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that those skilled in the art to which the present invention belongs can conceive of various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also fall within the technical scope of the present invention.

  For example, although the above embodiment mainly describes an example in which the transformation parameter for transforming captured image coordinates into overhead map coordinates is a projective transformation matrix, the present invention is not limited to this example. For example, conversion parameters for converting captured image coordinates to overhead map coordinates may include a perspective projection conversion matrix.

  In addition, the steps in the above embodiment do not necessarily have to be processed chronologically in the order described as the flowchart diagram. For example, each step in the process of the above embodiment may be processed in an order different from the order described as the flowchart diagram, or may be processed in parallel.

  Further, according to the above-described embodiment, a computer for causing the hardware such as the CPU 1001, the ROM 1002, and the RAM 1003 to exhibit the same function as each configuration of the information processing device 1, the information processing device 2, and the information processing device 3 described above. A program can also be provided. Also provided is a recording medium having the computer program recorded thereon.

1, 2, 3 information processing apparatus 5 camera 6 position sensor 10, 20, 30 control unit 11, 21, 31 object position specifying unit 13, 23, 33 calibration unit 15, 25, 35 coordinate conversion unit 27 correction unit 39 sensor Information management unit 50 Sensor information acquisition unit 60 Image acquisition unit 70 Operation unit 80 Output unit 90 Storage unit 271 Distance identification unit 273 Determination unit 275 Coordinate correction unit 277 Correction control unit

Claims (12)

Corresponding to the arrangement of the camera, based on correspondence between captured image coordinates that are position coordinates in a captured image captured by a camera and eyelid map coordinates that are position coordinates in an eyelid map corresponding to real space A calibration unit that specifies a conversion parameter for converting coordinates into the map coordinates, and a coordinate conversion unit that converts the captured image coordinates into the map coordinates using the conversion parameters.
The coordinate conversion unit uses the first conversion parameter corresponding to a first camera arrangement to select the image coordinates of the target object in the first image captured by the first camera arrangement. Convert to the eyelid map coordinates of the target object in the map;
The calibration unit is configured to adjust the captured image coordinates of the target object in a second captured image captured by a second camera arrangement different from the first camera arrangement and the eyelid map of the target object in the eyelid map An information processing apparatus that specifies a second conversion parameter corresponding to the second camera arrangement based on correspondence with coordinates.   The information processing apparatus according to claim 1, wherein the information processing apparatus further includes a target object specifying unit that specifies the captured image coordinates of the target object.   The target object identification unit detects the captured image coordinates of a target object candidate from the captured image, and captures the captured image coordinates of the detected target object candidate based on the determination of the operator. The information processing apparatus according to claim 2, wherein the information processing apparatus is specified as image coordinates.   The information processing apparatus according to any one of claims 1 to 3, further comprising: a correction unit configured to correct the conversion parameter specified by the calibration unit using a reference object whose eyelid map coordinates are known. The information processing apparatus according to any one of the above.   The correction unit is a known reference coordinate that is the known eyelid map coordinates of the reference object, and the eyelid map coordinates of the reference object obtained by the coordinate conversion unit converting the captured image coordinates of the reference object The information processing apparatus according to claim 4, further comprising: a distance specifying unit that specifies a distance between the conversion reference coordinate and the correction coordinate, and correcting the conversion parameter based on the distance.   The correction unit further includes a determination unit that determines whether the distance is within a predetermined range, and the conversion parameter is corrected according to the result of the determination by the determination unit. The information processing apparatus according to claim 1.   The correction unit determines the eyelid map coordinates of the target object based on a difference between the known reference coordinates and the conversion reference coordinates when the determination unit determines that the distance is not within the predetermined range. The information processing apparatus according to claim 6, further comprising a coordinate correction unit that corrects the correction parameter, based on the eyelid map coordinates of the target object corrected by the coordinate correction unit.   The number of reference objects required for the correction unit to correct the conversion parameter is smaller than the number of target objects required for the calibration unit to specify the second conversion parameter. The information processing apparatus as described in any one of -7. The information processing apparatus may be configured to determine whether the number of target objects included in both the first captured image and the second captured image is smaller than a predetermined threshold value. In the sensor coordinate system corresponding to the position sensor, the eyelid map coordinates of the first target object in the eyelid map obtained by converting the captured image coordinates of the target object, the sensing of the position sensor The second target object in the eyelid map based on the position coordinates of the first target object and the position coordinates of a second target object in the sensor coordinate system obtained based on the sensing of the position sensor It further comprises a sensor information management unit for acquiring the eyelid map coordinates,
The calibration unit is configured to perform a second operation based on the correspondence between the captured image coordinates of the second target object in the second captured image and the overhead map coordinates of the second target object in the overhead map. The information processing apparatus according to any one of claims 1 to 8, wherein a conversion parameter is specified.   The information processing apparatus according to any one of claims 1 to 9, wherein the conversion parameter includes a projection conversion matrix or a perspective projection conversion matrix. The captured image coordinates correspond to the location of the camera based on the correspondence between the captured image coordinates that are position coordinates in the captured image captured by the camera and the overhead map coordinates that are position coordinates in the overhead map Information processing method executed by an information processing apparatus, comprising: a calibration unit that specifies a conversion parameter for converting into the image; and a coordinate conversion unit that converts the captured image coordinates into the eyelid map coordinates using the conversion parameter And
The captured image coordinates of a target object in a first captured image captured by the first camera layout using a first conversion parameter corresponding to a first camera layout are compared with the target image in the eyelid map. Converting to the eyelid map coordinates;
Based on the correspondence between the captured image coordinates of the target object in the second captured image captured in a second camera configuration different from the first camera configuration and the overhead map coordinates of the target object in the overhead map Specifying a second conversion parameter corresponding to the second camera arrangement;
Information processing methods, including: The captured image coordinates correspond to the location of the camera based on the correspondence between the captured image coordinates that are position coordinates in the captured image captured by the camera and the overhead map coordinates that are position coordinates in the overhead map A program for causing a computer to realize a calibration function of specifying a conversion parameter for converting into a coordinate, and a coordinate conversion function of converting the captured image coordinates into the eyelid map coordinates using the conversion parameters. ,
The coordinate conversion function allows the captured image coordinates of a target object in a first captured image captured by the first camera layout to be captured using the first conversion parameter corresponding to the first camera layout. Convert to the eyelid map coordinates of the target object in the map;
The calibration function is configured to include the captured image coordinates of the target object in a second captured image captured by a second camera arrangement different from the first camera arrangement and the eyelid map of the target object in the eyelid map A program for specifying a second conversion parameter corresponding to the second camera arrangement based on correspondence with coordinates.

Images