JP2014039201A - Method of remote control by using roi during use of a plurality of cameras - Google Patents

Abstract

A remote control method for controlling and operating an actuator by performing visual feedback while using a plurality of cameras is provided.
A management apparatus transmits an entire area of an image captured by a plurality of cameras to a control apparatus via a non-guaranteed network, and the control apparatus includes the received images captured by the plurality of cameras. The management device 12 obtains and sets a cut-out area in the image in which the control target 13 is shown, and the management apparatus 12 cuts out the image captured by each camera 11 based on the set cut-out area. And the integrated image is transmitted as ROI to the control device 14 via the guaranteed network, and the control device 14 generates a control signal for controlling the control target 13 based on the ROI, and passes the guaranteed network via the guaranteed network. It transmits to the management apparatus 12.
[Selection] Figure 1

Description

  The present invention relates to an invention related to a method for remotely controlling and operating a controlled object such as a robot, an actuator, or a machine. Specifically, the situation of the controlled object is photographed by a plurality of cameras, and the image is transmitted to a control device on a computer at a remote place via a network. Then, the state of the controlled object is estimated by processing and analyzing the image data, and the estimated state is used for control / operation.

  As a method of controlling actuators such as robots and machinery, a control method called visual feedback is known (Non-Patent Documents 1 and 2). This method estimates the state of the controlled object by image processing and image analysis, and controls the controlled object based on that information. By transferring the image used for this estimation to a remote high-performance computer, It is also possible to apply visual feedback to remote control.

Yoshihiro Nakabo, Idaku Ishii, and Masatoshi Ishikawa: High Speed Target Tracking Using 1ms visual Feedback System, Video E Conf. Robotics and Automation (Minneapolis, 1996. 24-24-26) Yoshihiro Nakabo, Akira Ishii, Masatoshi Ishikawa: 1ms target tracking system using massively parallel and ultra-high-speed vision, Journal of the Robotics Society of Japan, Vol. 15, no. 3, pp. 417-421 (1997)

  By the way, when only one camera can be used, the effective scenes in the real environment are limited because of the limitation of the physical area that can be captured by the camera and the possibility that there are obstacles between the camera and the controlled object such as the actuator. It is thought that it is done. Therefore, if visual feedback can be used while using multiple cameras, it is possible to control and operate the actuator even in a more physically wide area or in an environment where there are obstacles in shooting the actuator. It is thought that it becomes.

  Therefore, an object of the present invention is to use visual feedback while using a plurality of cameras.

  Of the images captured by the camera, only the image area (ROI: Region Of Interest) of particular interest for processing and analysis required for control is provided via a guaranteed network with a high frame rate, low delay, and low information. Consider transferring data with loss, and stably controlling and operating robots and machinery. Here, when images obtained from a plurality of cameras are used, the ROI is appropriately handed over between the shooting areas of the respective cameras as the controlled object moves between the shooting areas of the respective cameras. At this time, the calculation related to the takeover is performed by a remote computer so that a load is not applied to the computer and the management device on the actuator side as much as possible.

  The control system according to the present invention transmits captured images captured by a plurality of cameras to a control device via a non-guaranteed network, and cuts the captured images from the captured image based on a clipping region set for each camera from the control device. A management device that integrates the cut-out image into an ROI image showing a control object, transmits the ROI image to the control device via a guaranteed network, and controls the control object according to a control signal from the control device; The cut-out area included in the ROI image showing the control object is obtained from the captured image received from the management apparatus for each camera, and the ROI and the cut-out area are transmitted to the management apparatus to obtain the ROI image. The control target is controlled based on the received ROI image, which is transmitted to the management apparatus via a guaranteed network. It generates the control signal, and a control device for controlling the controlled object by sending the control signal to the management device via a guaranteed network.

  The control method according to the present invention includes a captured image acquisition procedure for causing a control device to transmit captured images captured by a plurality of cameras to a management device via a non-guaranteed network, and a control device to control from the received captured images. A cut-out area transmission procedure for obtaining a cut-out area included in the ROI in which the target is shown for each camera, and transmitting the ROI and the cut-out area to the management apparatus, and the control device based on the cut-out area received by the captured image ROI image acquisition procedure for integrating the cut image cut out from the image into the ROI image and transmitting it to the management device via a guaranteed network, and a control signal for the control device to control the control object based on the received ROI image And transmitting the control signal to the management device via the guaranteed network A control signal transmission procedure of controlling, in this order.

  The control apparatus according to the present invention includes a captured image acquisition unit that acquires captured images captured by a plurality of cameras from a management apparatus via a non-guaranteed network, and an ROI in which a control target is reflected from the captured images. A ROI detection unit that obtains a cut region for each camera, and transmits the ROI and the cut region to the management device; and a cut image cut from the captured image based on the cut region is integrated into the ROI image; An ROI image acquisition unit for acquiring an image from the management device via a guaranteed network, a control signal for controlling the control target based on the ROI image, and the control signal being transmitted to the management device via the guaranteed network A control signal generation unit for transmitting to

  The control program according to the present invention includes a captured image acquisition step in which a captured image acquisition unit acquires a captured image captured by a camera from a management device via a non-guaranteed network, and an ROI detection unit controls from the captured image. A cut-out area detecting step for obtaining a cut-out area including the ROI in which the object is reflected for each camera, and transmitting the ROI and the cut-out area to the management device; and a ROI detection unit based on the cut-out area A ROI image acquisition step of acquiring a ROI image obtained by integrating the cut image cut out from the ROI into the ROI from the management device via a guaranteed network, and a control signal generation unit controls the control object based on the ROI image Control that generates a control signal and transmits the control signal to the management device via a guaranteed network Control program for No. executing a generating step, an order to the computer.

  According to the present invention, visual feedback can be used while using a plurality of cameras. By allowing the ROI to be inherited between a plurality of cameras, the actuator can be stably moved in a physical area that is as large as the number of cameras. In particular, since the present invention handles all calculations related to takeover by a remote computer, the calculation load on the actuator side is small (low cost and energy saving of the management device), and the transferred ROI image itself is the same as in the case of a single camera. Therefore, it is possible to mount without losing the advantages of a single device, such as an increase in the communication capacity for ROI images.

It is a schematic diagram and a sequence diagram of a control system concerning this embodiment. It is a block block diagram of the control system which concerns on this embodiment. It is explanatory drawing of each block of the control system which concerns on this embodiment. An example of ROI and subROI setting processing when a control target is included in a captured image is shown. An example of ROI and subROI setting processing when the control target extends over a plurality of captured images is shown. It is explanatory drawing which shows an example of the calculation method of subROI when a control object straddles a some picked-up image.

  Embodiments of the present invention will be described with reference to the accompanying drawings. The embodiments described below are examples of the present invention, and the present invention is not limited to the following embodiments. In the present specification and drawings, the same reference numerals denote the same components.

  The outline of the system of the present invention and the overall processing flow are shown in FIGS. 1 and 2 (system outline diagram and sequence diagram). Details of processing based on the functional block, ROI, and subROI are shown in FIG. 3 and FIG. 4 (functional block diagram and processing of ROI and subROI). Further, FIG. 5 (ROI, subROI detection and setting flowchart example) shows a ROI detection flowchart, and FIG. 6 (subROI setting flowchart implementation example (two camera system)) shows a subROI setting flowchart in the two-camera system.

  The control system according to the present embodiment observes the control target 13 and its environment with a plurality of cameras 11, transmits the image data to the control device 14 via the network, and processes and analyzes the image in the control device 14. By performing, the state of the controlled object 13 is estimated in real time. At this time, when the control target 13 exists across the imaging regions of the plurality of cameras 11, an image region ROI necessary for control is constructed using information on the control target 13 obtained from the estimation. Therefore, setting information for setting a subROI smaller than the ROI in each camera 11 is generated by the remote computer 15 and the setting information is transferred to the management apparatus 12 on the side where the control target 13 exists. The management device 12 on the control target 13 side sets subROIs in the respective cameras 11, integrates these subROIs as one ROI, and transfers them to the remote control device 14.

  Note that the processing of image analysis and recognition (extraction of the control target 13) may be performed on an image obtained by combining the images of the plurality of cameras 11 into one image. In this case, since each camera 11 is fixed, the image synthesizing process can be easily executed by a fixed calculation. Further, the image composition process may be executed by either the management device 12 or the control device 14.

  The control device 14 includes an inverse encoding unit 41 of the non-guaranteed network 91, an inverse encoding unit 42 of the guaranteed network 92, an image analysis unit 43 of a captured image that functions as a captured image acquisition unit, and an ROI detection unit 44. And an ROI image analysis unit 45, a state estimation unit 46, and a control signal generation unit 47. The management device 12 includes an ROI image generation unit 21, a compression encoding unit 22 of the non-guaranteed network 91, a compression encoding unit 23 of the guarantee type network 92, and a control signal I / F 24.

The management device 12 transmits the captured images captured by the plurality of cameras 11 to the control device 14 via the non-guaranteed network 91, and cuts the captured images from the captured images based on the cropping area set for each camera 11 from the control device 14. The cut image is integrated into the ROI image in which the control target 13 is reflected, the ROI image is transmitted to the control device 14 via the guaranteed network 92, and the control target is controlled according to the control signal from the control device 14.
The control device 14 obtains, for each camera 11, a clipping region (subROI) included in the ROI image in which the control target 13 is reflected from the captured image received from the management device 12, and obtains ROI setting information and subROI setting information in the management device 12. Transmitting and transmitting the ROI image to the management apparatus 12 via the guaranteed network 92, generating a control signal for controlling the control target 13 based on the received ROI image, and managing the control signal via the guaranteed network 92 The control object 13 is controlled by transmitting to the apparatus.

  The control method according to the present embodiment includes a captured image acquisition procedure (S101), a cutout region transmission procedure (S102, S103, S104), an ROI image acquisition procedure (S105, S106, S107), and a control signal transmission procedure (S108). , S109).

  The management apparatus 12 transmits the captured image captured by the camera 11 to the control apparatus 14 via the non-guaranteed network 91 (S101). The ROI detection unit 44 obtains an area where the control target 13 is shown from the received captured image (S102). The ROI detection unit 44 performs the image processing, the image analysis, etc. on the image obtained from the camera 11 transferred via the non-guaranteed network 91, and the screen of the detected control target 13 The ROI setting information and the subROI setting information are generated based on the above-described position information, size information, or physical state such as the speed of the control target 13. The ROI setting information and subROI setting information are transmitted to the management apparatus 12 (S103). The ROI setting information and the subROI setting information are transferred to the management apparatus 12 via the guaranteed network 92 and used by the ROI image generation unit 21.

  The ROI image generation unit 21 sets the ROI and the subROI according to the ROI setting information and the subROI setting information (S104). The ROI image generation unit 21 cuts out a part of each image obtained from the plurality of cameras 11 based on the received ROI setting information and subROI setting information, and integrates the cut images based on the ROI setting information. At this time, the ROI image generation unit 21 may copy the image. After step S104, the management apparatus 12 continues to transmit the captured image captured by the camera 11 as a stream to the control apparatus 14 via the non-guaranteed network 91 (S105). At the same time, the management apparatus 12 transfers the cut and integrated image as a stream to the control apparatus 14 via the guaranteed network 92 through processes such as compression and encoding (S106).

  The non-guaranteed network 91 transfers the image obtained from the camera 11 to the control device 14 (S301). This image is presented to the user on a monitor or the like (S302). The image analysis unit 43 performs image processing and analysis, and the ROI detection unit 44 detects the control target 13 and sets the surrounding area (S306). The management apparatus 12 actually sets the ROI and subROI (S308).

  The guaranteed network 92 transfers the ROI image to the control device 14, transfers the control signal generated in the control device 14 to the management device 12, and receives the ROI setting information and the subROI setting information generated in the control device 14. The data is transferred to the management device 12 (S303). The image analysis unit 45 performs image processing and image analysis of the ROI image, and the state estimation unit 46 estimates the state of the control target 13 (S107, S304). The control signal generation unit 47 generates a control signal for the control target 13 based on the state estimated by the state estimation unit 46 (S108, S305). The control signal generation unit 47 transmits a control signal to the management device 12. And the management apparatus 12 transmits a control signal to the control object 13 (S109, S307).

  Thereafter, S102, S103, and S104 are repeated for the image obtained from the camera 11. S107, S108, and S109 are repeated for the ROI image. These two iterations are performed independently.

  In order to control and operate the control target 13 itself, it is sufficient to know the control target 13 and its peripheral area in the image data obtained from the camera 11. Therefore, in the present invention, only the ROI image focusing only on the control target 13 and its periphery is transmitted to the control device 14 individually from the image obtained from the camera 11. As a result, image data smaller than the captured image of the camera 11 itself passes through the network. In addition, if the image is small, the calculation cost for encoding and compression is small, so that an encoding method and compression method with a higher compression rate can be used. Accordingly, the amount of image data transmitted to the control device 14 can be considerably reduced, and the required transmission capacity is reduced.

  This ROI image data needs to be transmitted to the control device 14 quickly and with little information loss in order to know the state of the control target 13 quickly and accurately. Therefore, a guarantee type network 92 is used for the transmission path of the ROI image data. Since the amount of data actually transmitted is reduced by the above processing, the required bandwidth of the guaranteed network 92 can be reduced.

  Further, the captured image itself obtained from the camera 11 needs to be transferred to the control device 14 side in order for the user to determine an operation instruction and to set an ROI at the time of system activation. However, even if the image is somewhat distorted, it can be corrected by the user himself and need not be transmitted to the control device 14 promptly. Therefore, the image is transmitted via a non-guaranteed network 91 such as the Internet. Is done.

FIG. 4 shows a flow of image area setting processing in the control device.
It is determined whether or not data of captured images from a plurality of cameras 11 has been received (S401). If it has been received, the reverse encoding unit 41 reversely encodes the data and decodes the captured image (S402).
Next, the image analysis unit 43 acquires captured images captured by the plurality of cameras 11 from the management apparatus 12 via the non-guaranteed network 91, and performs image processing such as color space adjustment and filter processing (S403). .
Next, the ROI detection unit 44 performs image processing and image analysis in order to detect the control target 13 (S404).

When the control target 13 is recognized in step S405, the ROI detection unit 44 detects the coordinates (x, y) of the center of the control target 13 on the image and the sizes (x ₁ , y ₁ ) in the respective axial directions ( S406).
Next, the ROI detection unit 44 calculates an appropriate peripheral region ε according to the characteristics of the control target 13 (S407). Here, the peripheral region ε is calculated by, for example, calculating the maximum moving speed of the control target 13 × the cut image frame period × the safety factor.
Next, the ROI detection unit 44 sets an ROI having a size (x _{1 + ε} , y _{1 + ε} ) centered on the coordinates (x, y) (S408). Then, a subROI corresponding to the set ROI is calculated for each camera 11. The subROI calculation method will be described later.
Next, the ROI detection unit 44 transfers the ROI setting information and the corresponding subROI setting information for each camera 11 to the management apparatus 12 (S409). Then, the process proceeds to step S401.

When the captured image data is not received in step S401, the ROI detection unit 44 acquires the current state of the control target 13 from the state estimation unit 46, and detects the speed and the position in the ROI image ( S410).
Next, the ROI detection unit 44 determines whether or not the ROI needs to be updated (S411). For example, when the coordinate value of the control target 13 and the distance between the edges of the closest ROI image are less than a certain threshold value, it is determined that the ROI needs to be updated. Further, when (current speed of control target 13 × cut image frame) + (coordinates of current control target 13) protrudes from the ROI in the x direction or the y direction, it is determined that the ROI needs to be updated.

If it is determined in step S411 that the ROI needs to be updated, the ROI is shifted so that the center of the control target 13 comes to the center in the ROI (S412). Then, a subROI corresponding to the set ROI is calculated for each camera 11. The subROI calculation method will be described later.
Next, the set ROI and the corresponding subROI are transferred to the management apparatus 12 (S413). Then, the process proceeds to step S401.

  If it is determined in step S411 that ROI update is not necessary, the process proceeds to step S401.

A specific example of the ROI setting method in the ROI detection unit 44 will be described with reference to FIG.
The state information of the control target 13 is acquired from the state estimation unit 46 (S501), and ROI setting information is acquired (S502). Then, it is determined whether or not the ROI extends over a plurality of images (S503).
When the ROI extends over a plurality of images (Yes in S503), the moving direction of the control target 13 is calculated (S504). The subROI of each camera 11 is calculated according to the moving direction (S505).
When the ROI does not straddle a plurality of images (No in S503), it is detected which camera 11 the ROI completely fits in (S510). The subROI and ROI of the camera 11 to be accommodated are set as the same area (S511).
After step S505 or S511, the ROI setting information and the subROI setting information are transmitted to the management apparatus 12 (S506), and the ROI image generation unit 21 sets the subROI for each camera 11 (S507), and acquires the subROI image from the camera 11. (S508). The ROI image generation unit 21 generates a ROI image by integrating a plurality of subROI images, and transmits the ROI image to the control device 14. As a result, the control device 14 receives the ROI image from the management device 12.

Here, the subROI of each camera 11 can be calculated as follows. FIG. 6 shows an example when the control target 13 moves from the camera 1 to the camera 2. Assume that the imaging size of each camera is X × Y. The size of the ROI and _{X r} × _{Y r.} The coordinates of the control target 13 are (x, y). The origin of the camera 1 is (0, 0), and the origin of the camera 2 is (x ₂ , y ₂ ).

The origin of the subROI of the camera 1 is (x−X _r / 2, y−Y _r / 2).
The size of the subROI of the camera 1 is X _r × Y _r when the ROI does not straddle two images, and (X _r / 2 + (X−x)) when the ROI straddles two images. × is a _{Y r.}

The origin of the subROI of the camera 2 is (X, y−Y _r / 2).
The size of the subROI of the camera 2 is 0 × 0 when the ROI does not straddle two images, and (X _r / 2− (X−x)) × when the ROI straddles two images. it is a Y _r.

Further, whether or not to straddle can be determined as straddling when X _r / 2> (X−x) is satisfied and not straddling when not satisfying. When not straddling, the subROI and ROI of the camera 1 match.

  As described above, in the present invention, the management device 12 transmits an image obtained by capturing the control target 13 with the plurality of cameras 11 to the control device 14 via the non-guaranteed network 91, and the control device 14 receives the image. In the remote control system that generates a control signal for controlling the control target 13 based on the image and transmits the control signal to the control target 13 via the guaranteed network 92 and the management device 12, the management device 12 captures images taken by the plurality of cameras 11. Are transmitted to the control device 14 via the non-guaranteed network 91, and the control device 14 displays an image in which the control target 13 is reflected in the regions corresponding to the received images taken by the plurality of cameras 11. The management device 12 obtains and sets the cut area in the image, and the management device 12 cuts out the image captured by each camera 11 based on the set cut area. The regions are integrated, and the integrated image is transmitted as an ROI to the control device 14 via the guaranteed network 92. The control device 14 generates a control signal for controlling the control target 13 based on the ROI, and the guaranteed network 92 To the management device.

  The camera 11 is a general and inexpensive sensor that knows the state of robots and machines in remote places and their environment, and is very useful for the future. It is thought that applications that take advantage of these features will be born in the future. . However, since there are a lot of unnecessary information and areas in the image information obtained from the camera 11, the burden on the communication device, the network, and the like is unnecessarily increased. The present invention enables state estimation and control with high real-time performance and high accuracy using an inexpensive guaranteed network 92, and can be expected to contribute to the realization of new applications and services. Here, by using the present invention, it becomes possible to introduce a plurality of cameras 11 into a remote control system that uses the ROI to estimate the state of the actuator. As a result, the system can be adapted to the case where the actuator is operated in a wide physical region where it is difficult to photograph with one camera 11, or to an environment where there is an obstacle in photographing the actuator.

  Furthermore, since the present invention performs all calculations related to the setting of the subROI with a remote computer, the load on the actuator side management device is small. Therefore, it can be expected to use an expensive device and avoid an increase in energy consumption. In the present invention, since the transferred ROI image (subROI integrated) itself is transferred as one image, the communication capacity and the number of communication sessions are not increased.

  The control device 14 of the present invention can be realized by a computer and a program, and can be recorded on a recording medium or provided through a network.

  The present invention can be applied to the information communication industry.

11: Camera 12: Management device 13: Control target 14: Control device 15: Computer 21: ROI image generation unit 22, 23: Compression encoding unit 24: Control signal I / F
41, 42: Decoding unit 43: Image analysis unit 44: ROI detection unit 45: Image analysis unit 46: State estimation unit 47: Control signal generation unit 91: Non-guaranteed network 92: Guaranteed network

Claims (4)

Captured images taken by multiple cameras are sent to the control device via a non-guaranteed network, and the control object shows a cut image cut from the captured image based on a cut area set for each camera from the control device. An ROI (Region Of Interest) image, a management device that transmits the ROI image to the control device via a guaranteed network, and controls the control target according to a control signal from the control device;
For each camera, the cut-out area included in the ROI image showing the control target is obtained from the captured image received from the management apparatus, and the ROI and the cut-out area are transmitted to the management apparatus to guarantee the ROI image. A control signal for controlling the control object based on the received ROI image, and transmitting the control signal to the management device via the guaranteed network. A control device for controlling the controlled object;
A control system comprising: A captured image acquisition procedure for causing the control device to transmit captured images captured by a plurality of cameras to the management device via a non-guaranteed network;
A control device, for each camera, obtains a cutout area included in the ROI in which the control target is shown from the received captured image, and sends the ROI and the cutout area to the management device.
ROI image acquisition procedure in which the control device integrates the cut image cut from the captured image based on the received cut region into the ROI image and transmits the ROI image to the management device via the guaranteed network;
A control device generates a control signal for controlling the control target based on the received ROI image, and transmits the control signal to the management device via a guaranteed network, thereby transmitting the control signal for controlling the control target Procedure and
The control method which has these in order. A captured image acquisition unit that acquires captured images captured by a plurality of cameras from a management device via a non-guaranteed network;
A ROI detection unit that obtains, for each camera, a cutout area including a ROI in which a control target is reflected from the captured image, and transmits the ROI and the cutout area to the management device;
A ROI image acquisition unit that integrates a cut image cut from the captured image based on the cut region into the ROI image, and acquires the ROI image from the management device via a guaranteed network;
A control signal generation unit that generates a control signal for controlling the control target based on the ROI image, and transmits the control signal to the management device via a guaranteed network;
A control device comprising: The captured image acquisition unit acquires a captured image captured by the camera from the management device via a non-guaranteed network, and
A ROI detection unit that obtains, for each camera, a cutout region including a ROI in which a control target is reflected from the captured image, and sends the ROI and the cutout region to the management device;
A ROI image acquisition step in which the ROI detection unit acquires an ROI image obtained by integrating the ROI image cut out from the captured image based on the cut-out region into the ROI from the management device via a guaranteed network;
A control signal generating step for generating a control signal for controlling the control target based on the ROI image, and transmitting the control signal to the management device via a guaranteed network;
Is a control program that causes a computer to execute in order.

Images