JP2003067048A - Information processing equipment - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To enable an operator to intuitively grasp an operation target and an operation result in a remote driving monitoring system or the like. When a display position of a button is moved sideways while strongly pressing a finger on an image display area in which an image from a camera is displayed, a man-machine server is displayed.
Indicates that each time the finger is moved, the knob 4 is controlled via the control computer 50.
A command is issued to an actuator that controls the knob 24, and the knob 424 is actually moved in accordance with the movement of the finger. Knob 424
As a result, the needle of the meter 422 also touches. The movements of the knob 424 and the needle of the meter 422 are photographed by the camera 60 and are displayed on an image on the image display area 200. This allows the operator to use the knob 424 with his / her finger.
Get the feeling of moving.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a man-machine interface (hereinafter, simply referred to as a man-machine) using a video image (hereinafter, simply referred to as a video), and more particularly to a video input from a video camera (hereinafter simply referred to as a man-machine interface) A video operation method and device, a video search method and device, a video processing definition method and device, and a power plant, which are suitable for monitoring and operating a remote object by operating a camera image). Chemical plant, steel rolling mill, building,
The present invention relates to a remote driving monitoring method and system for roads, water and sewer systems, etc.

[0002]

2. Description of the Related Art In order to safely operate a large-scale plant system represented by a nuclear power plant, an operation monitoring system having an appropriate man-machine is indispensable. The plant is operated and maintained by the operator's three tasks of “monitoring”, “decision” and “operation”. The operation monitoring system needs to be equipped with a man-machine to facilitate these three tasks of the operator. In the "monitoring" work, it is necessary to be able to grasp the state of the plant immediately and accurately.
At the time of "judgment", the operator must be able to quickly refer to the information and the information that is the basis for the judgment. At the time of "operation",
It is essential to have an operating environment in which the operation target and the result of the operation can be intuitively understood, and the operation intended by the operator can be executed accurately and quickly.

The man-machine of the conventional operation monitoring system is
Let's take a look at each work of “monitoring”, “judgment” and “operation”.

(1) Monitoring The state inside the plant can be grasped by monitoring data from various sensors for detecting pressure and temperature and images from video cameras arranged at various places in the plant. Values from various sensors are displayed in various forms on a graphic display or the like. Trend graphs and pergraphs are also widely used. On the other hand, the image from the video camera is often displayed on a dedicated monitor different from the graphic display. 4 in the plant
It is not uncommon for zero or more cameras to be installed. The operator monitors the various parts of the plant while switching cameras and controlling the camera direction and lens. In normal surveillance work, the operator rarely sees the image from the camera, and at present the utilization rate of the camera image is low.

(2) Judgment When an abnormality occurs in the plant, the operator comprehensively examines a large amount of information obtained from the sensors and cameras,
You have to quickly and accurately determine what is happening in your plant. In the current operation monitoring system, since data from various sensors and images from cameras are managed independently, it is difficult to refer to them while associating them with each other, which puts a heavy burden on the operator.

(3) Operation The operation is performed using the buttons and levers on the operation panel. Recently, there are many systems that can be operated by combining a graphic display and a touch panel and selecting menus and figures displayed on the screen. However, buttons and levers on the operation panel, menus and figures on the display are abstract forms that are separated from things on site, and the operator recalls their functions and the results of the operations. Can be difficult. That is, there is a problem that it is not immediately known which lever can be operated to perform a desired operation, and it is difficult to intuitively grasp what operation command is sent to which device in the plant when a certain button is pressed. Also,
Since the operation panel is configured independently of the monitor such as the camera, there is a problem that the device becomes large.

[0007]

As described above, the prior art has the following problems.

(1) Remote operation using keys, buttons, levers on the operation panel, or remote operation using menus or icons on the screen makes it difficult to convey the sense of presence in the field. It is difficult to grasp intuitively and there is a high possibility that it will be operated incorrectly.

(2) The operator has to directly switch the camera or perform remote control, and when monitoring is performed using a large number of cameras, it is not possible to easily select the camera that shows the desired point. Also, by operating a remote camera,
It takes a lot of work to make the part you want to see well visible.

(3) A screen for displaying an image from a video camera, a screen for referring to other data, and a screen or a device for operating are different from each other, or the entire device becomes large. There is a problem that it is difficult to cross-reference with.

(4) Although the camera image has a high effect of transmitting a sense of realism, it has a drawback that the operator cannot intuitively grasp the structure in the camera image because it has a large amount of information and is not abstracted. On the other hand, the graphic display makes it possible to emphasize important parts, simplify unnecessary parts, and abstract only essential parts to display, but it is an expression separated from actual things and events. However, the operator may not be able to easily recall the relationship between the graphic representation and the actual thing or event.

(5) Video information from the camera and other information
(For example, data such as pressure and temperature) are managed independently of each other, and cross-reference cannot be easily performed. Therefore, it is difficult to judge the situation comprehensively.

An object of the present invention is to solve the above problems of the prior art and achieve at least one of the following (1) to (6).

(1) In a remote operation monitoring system, etc.,
To allow the operator to intuitively understand the operation target and operation result.

(2) To be able to easily see the image of the place to be monitored without being bothered by the selection of the camera and the remote operation of the camera.

(3) To reduce the size of the remote operation monitoring system and save space.

(4) To make use of the advantages of each of the camera image and graphics and to make up for the disadvantages.

(5) To be able to quickly refer to different types of information. For example, making it possible to immediately refer to the temperature of the part currently monitored by the camera image.

(6) To easily design and develop a man-machine that achieves the above object.

[0020]

According to the present invention, the above objects (1) to (5) are solved by a method having the following steps.

(1) Object designation step An object in the video image displayed on the screen (hereinafter, referred to as an object) is a pointing device (hereinafter, referred to as an object).
It is specified using an input means such as (PD). Video images are input from a remotely placed video camera or played from a storage medium (optical video disc, video tape recorder, computer disc, etc.). As the pointing device, for example, a touch panel, tablet, mouse, eye tracker, gesture input device, or the like is used. Before designating the object, the specifiable object in the video may be clearly displayed by the composite display of graphics.

(2) Processing Execution Step Processing is executed based on the object designated by the object designation step. The contents of the processing are as follows, for example.

Sending an operation command that causes a specified object to move or results similar to when the specified object is operated. For example, when the designated object is a button, the button is actually pushed down, or an operation command that gives the same result as when the button is pushed down is sent.

Switching the video based on the specified object. For example, by operating a remote camera,
Makes the specified object look better. Move the camera so that the specified object appears in the center of the image, or control the lens so that the specified object appears larger. In another example, the image of the camera that captures the specified object from another angle is switched to, or the image of the camera that shows the object related to the specified object is switched to.

The graphics are composite-displayed on the video so that the designated object is clearly displayed.

Display information related to the specified object. For example, the manual, maintenance information, structure diagram, etc. of the specified object are displayed.

Display a list of processes that can be executed on the specified object by using a menu. The menu can also be expressed graphically. That is, some figures are composite-displayed on the image, the composite-displayed figures are selected by the PD, and the next process is executed based on the selected figures.

Further, according to the present invention, the object (2) is to provide a search key designating step for designating a search key by inputting text or graphics, and a thing that matches the search key designated by the search key designating step. It is also solved by a method having a video search step of displaying the video video being displayed.

According to the present invention, the object (6) is a video display step of displaying a video image input from a video camera, an area designating step of designating an area on the video displayed by the video display step, A process defining step of defining a process in the area specified by the area specifying step.

An object in the video image on the screen is directly designated and an operation command is sent to the designated object.
The operator gives an operation instruction while looking at a video image of a real image of the object. If the operation instruction causes a visible movement of the object, the movement is directly reflected in the camera image. In this way, by directly operating the live-action image, the operator can perform remote operation as if he / she was actually working on site. As a result, the operator can intuitively understand the operation target and the operation result, and can reduce erroneous operations.

A camera is selected or an operation command is sent to the camera based on the object in the image specified on the screen. This makes it possible to obtain the optimum image for monitoring the object by simply specifying the object in the image. That is, the operator only needs to specify what he / she wants to see, and does not need to select a camera or operate the camera remotely.

When a direct operation is performed on an object in a video, graphics are appropriately displayed and synthesized on the video. For example, when the user specifies an object, a graphic display is performed so as to clearly indicate which object has been specified. As a result, the operator can confirm that the operation intended by the operator is being performed reliably. If a plurality of processes can be executed on the specified object, a menu for selecting a desired process is displayed.
This menu may be composed of figures.
By selecting the figure displayed as a menu,
The operator can have a stronger sense of actually operating the object.

Information is displayed based on the object in the video image designated on the screen. With this, the information related to the object in the video can be referred to only by designating the object. The situation can be easily determined while referring to the video and other information at the same time.

Text or a figure is input as a search key, and an image showing an object matching the input search key is displayed. Text is input from a character input device such as a keyboard, a voice recognition device, a handwritten character recognition device, or the like. The figure is input using the PD, or the figure data already created by some method is input. Also,
A text or graphic in the video may be designated as the search key. When the image to be searched is a camera image, the camera is selected based on the search key, the direction of the camera and the lens are controlled, and the search key is displayed.
It is also possible to clearly show where in the video the portion matching the search key is, by appropriately synthesizing the graphics with the video showing the object matching the search key. In this way, by displaying the image based on the search key,
The operator can obtain the desired image by simply showing what he wants to see with words or figures.

By displaying an image, designating a region on the image, and defining a process for the designated region, the contents of the process to be executed when an object in the image is designated are defined. . This makes it possible to create a man-machine that directly manipulates things in the video.

[0036]

BEST MODE FOR CARRYING OUT THE INVENTION A plant operation monitoring system according to an embodiment of the present invention will be described with reference to FIGS. 1 to 24.

The overall structure of this embodiment will be described with reference to FIG. In FIG. 1, 10 is a display serving as display means for displaying graphics and images, 12 is a pressure-sensitive touch panel serving as input means mounted on the entire surface of the display 10, 14 is a speaker for outputting voice, and 20 is an operation. A man-machine server that provides a man-machine for a person to monitor and operate the plant, a switcher 30 for selecting one video input and a voice input from a plurality of video inputs and a plurality of voice inputs, and 50 A control computer that controls equipment in the plant and collects data from sensors, 52 is an information system local area network that connects the control computer 50 to the man-machine server 20, other terminals, and computers (hereinafter Referred to as LAN)
(Eg LAN as defined by IEEE802.3), 54
Is a control system LAN (for example, IEEE802.0) that connects the control computer 50 with various devices to be controlled and various sensors.
LAN (as specified in 2.4), 60, 70, 80 are placed in various places in the plant, and an industrial video camera (hereinafter simply referred to as camera) for capturing and inputting an object to be controlled, 62, 72, and 82 are cameras 60 and 7 in accordance with commands from the control computer 50, respectively.
Controllers for controlling the directions of 0 and 80 and lenses, 64, 74 and 84 are cameras 60, 70 and
Microphones attached to 80, 90 and 92 are various sensors for knowing the state of each part of the plant, and 94 and 96 are actuators that control various devices in the plant in accordance with commands from the control computer 50.

The pressure-sensitive touch panel 12 is a type of PD. When an arbitrary position on the touch panel 12 is pressed with a finger, the coordinates of the pressed position and the pressed pressure are reported to the man-machine server. The touch panel 12 is attached to the entire surface of the display 10. The touch panel 12 is transparent, and the display content of the display 10 behind the touch panel 12 can be seen. This allows the operator to display 1
It is possible to specify by touching the object displayed on 0 with a finger. In this embodiment, as the operation of the touch panel 12, three types of (1) lightly pressing, (2) strongly pressing, and (3) dragging are used. To drag means to move the finger while pressing the touch panel 12 with the finger.
Although a pressure sensitive touch panel is used as the PD in this embodiment,
Other devices may be used. For example, a non-pressure-sensitive touch panel, tablet, mouse, light pen, eye tracker, gesture input device, keyboard, etc. may be used.

A plurality of images photographed by the cameras 60, 70, 80 are selected by the switcher 30 and displayed on the display 10 via the man-machine server 20. The man-machine server 20 controls the switcher 30 via a communication port such as RS232C to select an image from a desired camera. In this embodiment, when video is selected, input audio from the microphones 64, 74, 84 and the like are also selected at the same time. That is, when the camera is selected, the sound is also switched to the input from the microphone attached to the selected camera. The voice input from the microphone is output from the speaker 14. Of course, you can also switch the input from the microphone and camera independently. The man-machine server 20 can combine graphics with the image from the camera. The man-machine server 20 also sends an operation command to the control computer via the information system LAN 52 to specify the orientation of the camera and the control of the lens. This camera orientation and lens control will be referred to as camera work.

Further, the man-machine server, in accordance with the instruction of the operator, through the control computer 50, the sensor 90,
Input data from 92, etc.
Remotely operate 4, 96, etc. Man-machine server 20
Controls the movements and functions of various equipment in the plant by sending operation commands to the actuator.

The configuration of the man-machine server will be described with reference to FIG. In FIG. 2, 300 is a CPU, 310 is a main memory, 320 is a disk, 330 is a PD, an I / I for connecting the touch panel 12, the switcher 30, and the like.
O, 340 is a graphics frame buffer that stores the display data generated by the CPU 300, 360 is a digitizer that digitizes the input analog video information, and 370 is the digitized video information that is the output of the digitizer 360. Frame buffer for video to store,
380 is a display 1 that combines the contents of the graphic frame buffer 340 and the video frame buffer.
It is a blend circuit displayed at 0.

The video information input from the camera is displayed on the display 10 after being combined with the graphics generated by the man-machine server 20. The graphic frame buffer 340 has data of R, G, and B colors and data called α value corresponding to each pixel on the display 10. The α value specifies, for each pixel of the display 10, how to combine the image information in the video frame buffer 370 and the graphic display data in the graphic frame buffer 340. The function of the Brent circuit 380 can be expressed by d = f (g, v, α). Here, g and α are color information and an α value of one pixel in the graphic frame buffer 340, v is color information of a pixel corresponding to g in the video frame buffer 370, and d is g and v. Is the color information of the pixel as a result of combining. In this system, the following formula is used as the function f.

[0043] f (g, v, α) = 255 {αg + (1-α) v} Of course, another function may be used as the function f.

The graphic frame buffer 340 has a so-called double buffer structure.
The double buffer has a buffer for two screens, and the buffer to be displayed on the display 10 can be selected at any time. The buffer displayed on the display 10 is called a front buffer, and the buffer not displayed is called a back buffer. Switching between the front and back buffers can be done instantly. Flickering at the time of drawing can be eliminated by drawing graphics in the back buffer and switching the back buffer to the front buffer when the drawing is completed. The contents of either buffer can be read or written from the CPU at any time.

FIG. 3 shows an example of the display screen structure of the display 10. In FIG. 3, 100 is a display 10.
Display screen, 110 is a menu area for designating commands relating to the entire system, 150 is a data display area for displaying data from sensors and various materials and data relating to the plant, 130 is the entire plant and each part. Is a screen display area for displaying a configuration diagram, a structural drawing, a design drawing, etc., and 200 is a video display area for displaying a video image input from the camera.

FIG. 4 shows an example of the display form of the drawing display area 130. In FIG. 4, 132 is a menu for issuing a command that clearly indicates the position of the sensor, and 134 is one object on the drawing designated by the operator.
When the operator selects an object in the drawing displayed on the screen display area 130, information from the sensor related to the object is displayed on the data display area 150 or the video display area 200. For example, if a camera is defined as a sensor associated with the specified object, the image input from the camera is displayed in the image display area 20.
Displayed at 0. Also, for example, when a hydraulic pressure sensor is defined as a sensor related to the specified object, graphics that clearly indicate the current hydraulic pressure value,
A trend graph showing the change of the hydraulic pressure value so far is displayed in the data display area 150. When the finger is strongly pressed on the touch panel 12, the object on the drawing displayed at the pressed position is designated. Nothing happens if the sensor associated with the specified object is not defined.
In FIG. 4, the display position of the object 134 is strongly pressed with a finger. When the object is pressed with a finger, it is highlighted so that the operator can confirm that the object has been designated. In the example shown in FIG. 4, a camera 60 that reflects the object 134 is displayed on the object 134.
And a microphone 64 for inputting a sound around the object 134 is defined as an associated sensor, and when the object 134 is designated, the image display area 200 is displayed.
An image showing the object 134 is displayed on the screen, and the sound around the object 134 is output from the speaker 14.

FIG. 5 shows a display form of the image display area 200 when the object 134 is designated on the drawing display area 130 and the objects 13 arranged in the plant.
4 shows the correspondence relationship with 4. In FIG. 5, 202 to 210
Is a menu for setting the camerawork of the camera that is currently shooting the video, and 220 is a menu for clearly designating the object that can be designated in the video. 202 is a menu for setting the orientation of the camera. By selecting the menu 202, the camera can be panned left and right or up and down. Two
Reference numeral 04 denotes a menu for controlling the lens of the camera to zoom in the image, 206 denotes a menu for controlling the lens of the camera to zoom out the image, and 208 denotes resetting the camera work to the previous camera work. , 210 is a menu for resetting to the first camera work.

400 to 424 are attached to the object 134. Or various objects in the vicinity. Four
00 is a valve, and 410 and 420 are objects 134.
The above-mentioned character display, 412 is a voltage meter, 4
Reference numeral 14 is a button for turning on the power, 416 is a button for turning off the power, 422 is a meter showing the hydraulic pressure, and 424 is a slider knob for adjusting the hydraulic pressure. Valve 400,
The buttons 414, 416 and the knob 424 are not only an operating device that can be actually operated by hand, but also an operating device that can be remotely operated by issuing an operation command from the man-machine server 20.

When the operator lightly presses the inside of the image display area 200 with the finger, the camera work is set so that the object displayed at the position of the finger can be easily seen. FIG. 6 shows how the camera work is set so that the meter 412 is in the middle of the image when the meter 412 is lightly touched with the finger in the image display area 200. When the meter 412 is designated by the operator as shown in the left diagram of FIG. 6, the orientation of the camera 60 is set so that the meter 412 appears in the center of the image, and the lens of the camera 60 is moved so that the meter 412 zooms in. It is controlled and the image changes as shown in the right figure of FIG. By simply touching the object on the screen, the operator can set the camera work so that the object can be seen more clearly, and the remote operation of the camera is not bothered. In FIG. 6, reference numeral 502 is a graphic echo for clearly indicating that the meter 412 has been designated. The graphic echo 502 is erased when the operator removes the finger from the touch panel 12. In this way, the man-machine can be improved by synthesizing the graphic display on the camera image.

In FIG. 7, the image display area 200 includes a valve 4
When 00 is lightly touched with a finger, the camera work is set so that the valve 400 is located at the center of the image.
When the valve 400 is designated by the operator as shown in the left diagram of FIG. 7, the image changes to a large image in the center of the valve 400 as shown in the right diagram of FIG. In FIG. 7, reference numeral 504 is a graphic echo for clearly indicating that the valve 400 has been designated. The graphic echo 504 is erased when the operator releases the finger from the touch panel 12. Other objects 410, 414, 416, 420, 42
The same operation can be performed for 2,424.

When the operator strongly presses the inside of the image display area 200 with the finger, the object displayed at the position of the finger can be operated. 8 to 10 show examples of operating an object. FIG. 8 shows an example of operating the button 414. When the position where the button 414 is displayed on the image display area 200 is strongly pressed with a finger, the button 414 is pushed down from the man-machine server 20 to the actuator for operating the button 414 remote via the control computer 50. Is sent, and the button 414 at the remote site is actually pushed down. Button 414 is depressed,
As a result, how the needle of the meter 412 touches is displayed on the image display area 200 by the camera 60. As a result, the operator feels as if he or she actually pressed the button on the screen.

FIG. 9 shows an example of operating the slider knob 424 by dragging a finger on the touch panel 12. On the image display area 200, when the position where the button 414 is displayed is strongly pressed with the finger and the finger is moved sideways, the knob 424 displayed in the image also moves in accordance with the movement of the finger. As a result of the movement of the knob 424, the meter 422
The needle also moves. At this time, the man-machine server 20 issues a command to the actuator for controlling the knob 424 via the control computer 50 each time the finger moves, and actually moves the knob 424 in accordance with the movement of the finger. As a result, the operator feels as if he / she is actually moving the knob 424 with his / her finger.

FIG. 10 shows an example in which, when a plurality of operations can be performed on an object, a desired operation is selected by selecting a graphic that is compositely displayed on the image. Figure 1
At 0, 510 and 520 are valves 400 and
When the display position of is strongly pressed with a finger, it is a figure composited and displayed on the image. When the operator strongly presses the graphic 510, the man-machine server 20 issues an operation command to the actuator via the control computer 50 to rotate the valve 400 to the left. On the contrary, when the figure 512 is strongly pressed with a finger, the man-machine server sends an operation command for rotating the valve 400 to the right to the actuator. The rotation of the valve 400 is displayed in the image display area 200 captured by the camera 60. Along with the rotation of the valve 400, the graphic 51
The display of 0, 512 may be rotated.

FIG. 11 shows a method of specifying an operable object. Since not all objects shown in the image can be operated, a means for clearly indicating operable objects is required. In FIG. 11, 514 to 524 are
Objects 400, 412, 414, 41 respectively
6, 422 and 424 are graphics displayed to clearly show that they are operable. In the case of this embodiment,
The extrapolation rectangle of the object is displayed. Of course, various other display methods such as displaying more realistic graphics can be considered for the explicit display of the object.

FIG. 12 shows an example of inputting text and searching for a video in which the text is displayed. 12
, 530 is a graphic displayed on the image, 600 is a search sheet for executing a text search, 610 is a next menu for searching for another image matched by the search key, and 620 is the end of the search. An end menu for designating ".", And 630 is a text input area for inputting to the search menu. When a menu designating a search is selected in the menu area 110, the search sheet 600 is displayed on the display screen 100. When a text serving as a search key is input to the text input area 630 from the keyboard and the return key is pressed, the search is started. The man-machine server searches for a camera that can project an object including the search key, sets the searched camera in camera work so that the search key is clearly visible, and displays the image from the searched camera in the video display area 200. Graphics 530 are composite-displayed on the part that matches the search key in the video to clearly indicate the part that matches the search key. The video search using the text as the search key allows the operator to reflect the desired object by words.
With this method, it is possible to quickly find the monitoring target without switching the cameras or operating the cameras remotely. Although the keyboard is used for inputting text in this embodiment, other input means such as a voice recognition device or a handwritten character recognition device may be used. Further, in the present embodiment, the text is used as the search key, but a graphic may be used as the search key to search for an image in which a graphic that matches the graphic of the search key is displayed.

A method for realizing this embodiment will be described with reference to FIGS. 13 to 24. The main function of this embodiment is to specify an object in a video and execute an operation based on the object. FIG. 17 shows a flow chart of a program that realizes this function. When the touch panel 12 is pressed on the video display area 200, the object displayed at the pressed position (the position on the screen designated by the operator using the PD such as the touch panel is called an event position) is displayed. Identify (step 1000). When the object can be identified (when the object exists at the event position) (step 1010), the operation defined corresponding to the object is executed (step 1020).

The object shown at the event position is identified by referring to the model of the object to be photographed and the camera information. The model to be photographed is data on the shape and position of the object to be photographed. The camera information is data on how the camera is shooting the object to be shot, that is, the position, orientation, angle of view, orientation, etc. of the camera.

Various methods are conceivable for modeling the object to be photographed. In this embodiment, (1) three-dimensional model,
(2) Two two-dimensional models are used together. 2 above
Below is a summary of the two models and their advantages and disadvantages.

(1) Three-dimensional model A model in which the shape and position of the object to be photographed are defined in a three-dimensional coordinate system, and the advantage is that the object can be identified in correspondence with any camera work. That is, the object can be operated while freely operating the camera. The disadvantage is that since the model has to be defined in a three-dimensional space, the process of creating the model and identifying the object is more complicated than the 2D model. However, recently, CAD is often used for designing plants and designing and arranging devices in the plant.
Utilizing these data makes it easy to create a three-dimensional model.

(2) Two-dimensional model A model that defines the shape and position of an object in a two-dimensional coordinate system (display plane) for each specific camera work. The advantage is that it is easy to create a model. You can define a model by drawing a figure on the screen. The disadvantage is that you can only operate on camerawork images for which a model has been defined in advance. In order to increase the degree of freedom in camera operation, it is necessary to define the shape and position of an object on the corresponding plane for each more camera work. In many operation monitoring systems, there are many cases where the places to be monitored are decided beforehand. In that case, the number of camera works is determined, so that the disadvantage of the two-dimensional model does not matter.

A method of identifying an object based on a three-dimensional model will be described with reference to FIGS. 13 to 16. Figure 5
The object to be photographed by the camera 60 shown in FIG.
FIG. 13 shows an example of modeling with yz (called the world coordinate system). Here, the shape of each object is modeled by a plane, a rectangular parallelepiped, a cylinder, or the like. Of course, more kinds of three-dimensional basic shapes such as a sphere and a tetrahedron may be used. Further, not only a combination of basic shapes but also a more precise shape model may be used. Objects to be operated 400, 410, 412, 414, 416, 420, 4
22 and 424 are planes 800 and 8 on the model, respectively.
10,812,814,816,820,822,82
It is modeled as 4.

Correspondence between the image captured by the camera and the three-dimensional model will be described with reference to FIG. Imaging by a camera is an operation of projecting an object arranged in a three-dimensional space onto a two-dimensional plane (image display area 200). That is, the image displayed in the image display area 200 is a perspective projection of an object arranged in a three-dimensional space onto a two-dimensional plane. When the two-dimensional orthogonal coordinate system XsYs taken on the screen is called the screen coordinate system, the image taken by the camera is one point (x,
It is possible to formulate y, z) as a number (1) that maps one point (Xs, Ys) on the screen coordinate system.

[0063]

[Equation 1]

The matrix T in equation (1) is called a view transformation matrix. Each element of the view conversion matrix T includes camera information (camera position, posture, orientation, angle of view) and video display area 2
Given a size of 00, it is uniquely determined. Camera information is given in the world coordinate system. In FIG. 14, the position of the camera corresponds to the coordinates of the center Oe of the lens, the orientation of the camera corresponds to the vector OeYe, and the orientation of the camera corresponds to the vector OeZe.

The object identifying process is a process for determining which point on the world coordinate system is projected onto the point P on the screen coordinate system when one point P on the screen coordinate system is designated. As shown in FIG. 15, all points on the extended line of the straight line connecting the center Oc of the camera lens and the point P on the screen coordinate system are projected onto the point P. Among the points on the straight line, the point actually projected on the image display area 200 by the camera is the intersection of the straight line with the object 1 closest to the center Oe of the lens. In FIG. 15, an intersection point P1 between the object 1 and the straight line 840 is projected by the camera onto a point P on the image display area 200. That is, assuming that the event position is P, the object 1 will be identified.

Techniques for obtaining a view transformation matrix T from camera information and techniques for perspectively displaying a model defined in the world coordinate system on the screen coordinate system based on the view transformation matrix T and displaying it are well known in the field of graphics. Technology. In perspective projection, the process of projecting the surface of the object close to the camera onto the screen and not projecting the surface hidden from the camera by other objects onto the screen is called hidden surface removal (hi
ddensurfaceelimination) or visible surface determination (Visible
-Surface Determination), many algorithms have already been developed. These techniques are described, for example, in Foley.vanDam.Feiner.Hughes, "Computer Graphi.
Published by cs Principles and Practice ”Addison Wesley (19
90), and Newman.Sproull, “Principles of Interact.
ive Computer Graphics ”, published by McGraw-Hill (197
It is explained in detail in 3). Also, in many so-called graphic workstations, graphic functions such as view conversion matrix setting from camera information, perspective projection, and hidden surface removal are pre-installed by hardware and software, and these can be processed at high speed. .

In the present embodiment, these graphic functions are used to perform object identification processing. In the three-dimensional model, the surface of the object to be operated is color-coded in advance, and it is possible to identify which object the surface belongs to by the color. For example, in FIG.
3, planes 800, 810, 812, 814, 8
Different colors are set to 16, 820, 822 and 824 respectively. The color set for each object will be called an ID color. FIG. 16 shows the procedure of the identification processing using this ID-colored three-dimensional model. First, the current camera information is inquired (step 1300), and the view conversion matrix is set based on the inquired camera information (step 1310). The man-machine server 20 always manages the current camera state, and when a camera information inquiry is made, camera information is returned according to the current camera state. Of course, the current camera state may be managed by the camera control controller. In step 1320,
Based on the view conversion matrix set in step 1310, the color-coded model is drawn in the back buffer of the graphic frame buffer 340. In this drawing, perspective projection and hidden surface removal processing are performed. Since the drawing is performed in the back buffer, the drawn result is not displayed on the display 10. When the drawing is completed, the pixel value of the back buffer corresponding to the event position is read (step 1330). The pixel value is the ID color of the object projected at the event position. The ID color has a one-to-one correspondence with the object, and the object can be identified.

A method of identifying an object based on a two-dimensional model will be described with reference to FIGS. 18 to 24. Two
The dimensional model defines the position and shape of the object after being projected from the world coordinate system to the screen coordinate system. When the orientation and the angle of view of the camera change, the position and shape of the object projected on the screen coordinate system change. Therefore, in the two-dimensional model, it is necessary to have data on the shape and position of the object for each camera work. In this embodiment, the object is modeled by a rectangular area. That is, an object in a camerawork is represented by the position and size of the rectangular area in the screen coordinate system. Of course, you may model by using other figures (for example, a polygon or a free curve).

FIG. 18, FIG. 19 and FIG. 20 show the correspondence between the camerawork and the two-dimensional model. The left view of each drawing shows the display form of the image display area 200 for each camera work. In addition, the right diagram of each figure shows a two-dimensional model of an object corresponding to each camera work. In the left diagram of FIG. 18, objects 410, 412, 414 and 414 on the image
Numerals 416, 420, 422, and 424 are rectangular areas 710, 712, 714, and 71 in the two-dimensional model shown on the right.
6,720,722,724. Corresponding to one camera work, a set of rectangles modeling an object is called an area frame. The area frame 1 corresponding to the camerawork 1 is rectangular areas 710, 712, 71.
4, 716, 720, 722, 724.
19 and 20 show examples of area frames for different camera works. In FIG. 19, the area frame 2 corresponding to the camera work 2 is a rectangular area 740, 742, 74.
It is composed of 6,748. Rectangular areas 740, 742
746 and 748 are objects 412 and 41, respectively.
6, 424, 422. Similarly, in FIG. 20, the area frame 3 corresponding to the camera work 3 is composed of a rectangular area 730. The rectangular area 730 corresponds to the object 400. Even the same object corresponds to another rectangular area if the camera work is different. For example,
The object 416 corresponds to the rectangular area 716 in the case of camerawork 1 and corresponds to 742 in the case of camerawork 2.

The data structure of the two-dimensional model is shown in FIGS. 22, 23 and 24. In FIG. 22, reference numeral 1300 is a camera data table that stores data corresponding to each camera. The camera data table 1300 stores data of camera work that can be operated on an object in a video and data of a region frame corresponding to each camera work.

In FIG. 23, 1320 is camerawork data. The camerawork data includes a horizontal angle that is the horizontal direction of the camera, a vertical angle that is the vertical direction of the camera, and an angle of view that indicates the degree of zooming. Here, it is assumed that the posture of the camera and the position of the camera are fixed. When the posture of the camera and the position of the camera can be remotely controlled, data for controlling them can be added to the camera work data 1320. The camerawork data 1320 is used to set the camera to a predefined camerawork. That is, the man-machine server 20 sends the camera work data to the camera control controller to remotely operate the camera. The camerawork data 1320 is not directly necessary for the object identification process.

FIG. 24 shows the data structure of the area frame. The area frame data is composed of the number of areas forming the area frame and data on each rectangular area. The area data includes the position of the rectangular area in the screen coordinate system (x,
y), the size of the rectangular area (w, h), the active state of the object, the action, and additional information. The active state of an object is data indicating whether the object is active or inactive. When an object is inactive, it is not identified. Only active objects are identified. A pointer to the event / motion correspondence table 1340 is stored in the motion. In the event / action correspondence table 1340, the object is PD
The action to be performed when specified by is stored in pairs with the event. Here, the event is PD
The operation type is designated. For example, the event is different when the pressure-sensitive touch panel 12 is strongly pressed and when it is lightly pressed. When an event occurs, the object at the event location is identified and the action paired with the event / action pair defined in the object that matches the event that occurred occurs.
The additional information of the area frame stores a pointer to additional information 1350 of the object that cannot be represented only by the rectangular area. There are various types of additional information. For example, there is a text drawn on an object, a color, an object name, and related information (for example, a device manual, maintenance information, design data, etc.). This allows the object to be searched for based on the text drawn on the object and the related information of the specified object to be displayed.

FIG. 21 shows a procedure for identifying an object using a two-dimensional model. First, a plurality of frames corresponding to the current camera work are stored in the camera data table 130.
Search from 0 (step 1200). Next, the area including the event position is searched from the areas forming the area frame. That is, the position and size data of each area stored in the plurality of frame data is compared with the event position (step 1220), and when the area at the event position is found, the number is returned to the upper processing system. The upper processing system checks whether or not the found plurality is in the active state, and if it is in the active state, executes the operation defined corresponding to the event. Step 1220 is repeated until either a region containing the event location is found or all regions in the region frame have been examined (step 1210).

The two-dimensional model is defined using a two-dimensional model definition tool. The two-dimensional model definition tool has the following functions.

(1) Camera selection function A function of selecting an arbitrary camera arranged in the plant and displaying an image from the camera on the screen. There are the following methods for selecting the camera.

By designating an object on the layout plan of the plant displayed on the screen, the camera that projects the object is designated.

Designate the location of the camera on the plant layout displayed on the screen.

Specify an identifier such as a camera number or name.

(2) Camera work setting function A function of remotely controlling the camera selected by the camera selection function to set the orientation and angle of view of the camera.

(3) Graphic drawing function A function of drawing a graphic on the image displayed on the screen. Drawing is done by combining basic graphic elements such as rectangles, circles, polygonal lines, and free-form curves. With this function, the image of the object is used as the underlay and the outline of the object is drawn.

(4) Event / action pair definition function A function of designating at least one figure drawn by the figure drawing function and defining an event / action pair for it.
The event is defined by selecting a menu or entering the event name as text. The operation is described by selecting a predefined operation from a menu or using a description language. As such a description language, for example, the description language UIDL described in “User Interface Construction Support System with Meta User Interface” described in IPSJ Journal, Volume 30, No. 9, pages 1200 to 1210. To use.

The two-dimensional model definition tool creates a two-dimensional model by the following procedure.

Step 1: Designation of camera and camera work (1) Select a camera using the camera selection function,
Display the image on the screen. Next, using the camera work setting function (2), camera work is set and an image of a desired place is obtained.

Procedure 2: Defining the outline of the object The outline of the object to be defined as an object in the object in the image displayed in the procedure 1 is drawn by using the (2) graphic drawing function.

Procedure 3: Define event / action pair Use procedure (4) Event / action pair definition function to perform procedure 2
Select at least one figure drawn in, and define a pair of event and action.

Step 4: Saving Definition Contents Save definition contents as required. Definition contents are shown in Figure 22,
The data structure shown in FIGS. 23 and 24 is saved. When it is desired to create a two-dimensional model for another camera or another camera work, steps 1 to 4 are repeated.

By having a model of an object, it is possible to know where and how an object is displayed in the image. As a result, it is possible to display information related to the object on the basis of the position and shape of the object in the image, and to search the image of the object. An example is given below.

The object name, function, operation manual, maintenance method, etc. are compositely displayed on or near the object. For example, issuing a command will display the name of each function shown in the video.

An object created by graphics is synthesized and displayed on a live-action image as if it were actually viewed by a camera.

Searching the additional information of the object based on the input keyword, and setting the camera and camera work so as to reflect the corresponding object.

The internal structure of the object that cannot be displayed by the camera is displayed in a composite manner on the object in the video. For example, the state of the water flow in the pipe is simulated based on the data obtained from other sensors, and the result is synthetically displayed on the pipe shown in the live-action image. Similarly, a Grafax representing the state of the flame in the boiler (for example, a temperature distribution map created based on the information obtained from the sensor) is compositely displayed on the boiler in the image.

The object of interest at present is specified by graphics. For example, when an anomaly is detected by the sensor, graphics are composited and displayed on the object in the image. Also, graphics are combined with the object in the image related to the data displayed in the trend graph so that the relationship between the data and the object in the image can be immediately understood.

[0093]

The effects of the present invention are as follows (1) to (6)
To achieve at least one of.

(1) In a remote operation monitoring system or the like, the operator can intuitively understand the operation target and the operation result, and the erroneous operation is reduced.

(2) It is possible to easily see the image to be monitored without being bothered by camera selection and remote operation of the camera.

(3) Operations can be performed on the surveillance image. Therefore, it is not necessary to separate the monitoring monitor and the operation panel, and the remote operation monitoring system can be downsized and the space can be saved.

(4) By synthesizing and displaying graphics on the camera image, it is possible to make the best use of the respective advantages and to supplement the disadvantages. In other words, it is possible to emphasize important parts while conveying the realism of the scene.

(5) A display for quickly referencing different kinds of information. For example, it is possible to display a trend graph showing the values of the sensors related to the portion by simply specifying the portion currently monitored by the camera image. This allows
You will be able to comprehensively judge the situation at the site.

(6) It becomes possible to easily design and develop a man-machine so as to directly operate an image.

[Brief description of drawings]

FIG. 1 is an overall configuration of a plant operation monitoring system according to the present invention.

FIG. 2 is a hardware configuration of a man-machine server.

FIG. 3 is a screen configuration example of a plant operation monitoring system according to the present invention.

FIG. 4 is a screen display form of a drawing display area.

FIG. 5 is a diagram showing a correspondence between a screen display form of a video display area and a site.

FIG. 6 is a diagram showing an example of camerawork setting by object designation.

FIG. 7 is a diagram showing an example of camera work setting by object designation.

FIG. 8 is a diagram showing an example of a button operation by specifying an object.

FIG. 9 is a diagram showing an example of a slider operation by designating an object.

FIG. 10 is a diagram showing an example of an operation by graphic selection.

FIG. 11 is a diagram showing an explicit example of operable objects.

FIG. 12 is a diagram showing an example of video search using a search key.

FIG. 13 is a diagram showing an example of a three-dimensional model.

FIG. 14 is a diagram showing a relationship between a three-dimensional model and an image on a screen.

FIG. 15 is a diagram showing a relationship between points on the screen and objects.

FIG. 16 is a diagram showing a procedure of object identification processing using a three-dimensional model.

FIG. 17 is a diagram showing a procedure of an implementation method of the embodiment.

FIG. 18 is a diagram showing a relationship between a two-dimensional model and camera work.

FIG. 19 is another diagram showing a relationship between a two-dimensional model and camera work.

FIG. 20 is another diagram showing the relationship between a two-dimensional model and camera work.

FIG. 21 is a diagram showing a procedure of object identification processing using a two-dimensional model.

FIG. 22 is a diagram showing the structure of a camera data table.

FIG. 23 is a diagram showing a structure of camera work data.

FIG. 24 is a diagram showing a data structure of a region frame.

[Explanation of symbols]

10 ... Display, 12 ... Pressure sensitive touch panel, 14 ...
Speakers, 20 ... Man-machine server, 30 ... Switcher, 40 ... Keyboard, 50 ... Control computer, 60 ... Video camera, 62 ... Camera control controller, 64 ...
Microphone, 90, 92 ... Sensor, 94, 96 ... Actuator, 100 ... Display screen, 130 ... Drawing display area, 15
0 ... data display area, 200 ... video display area, 600 ...
Text search sheet.

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Koichiro Tanikoshi             4026 Kujimachi, Hitachi City, Ibaraki Japan             Tachi Works Hitachi Research Laboratory (72) Inventor Masayasu Futagawa             4026 Kujimachi, Hitachi City, Ibaraki Japan             Tachi Works Hitachi Research Laboratory (72) Inventor Shinya Tanifuji             4026 Kujimachi, Hitachi City, Ibaraki Japan             Tachi Works Hitachi Research Laboratory (72) Inventor Atsuhiko Nishikawa             5-2-1 Omika-cho, Hitachi City, Ibaraki Prefecture             Ceremony company Hitachi Ltd. Omika factory F term (reference) 5E501 AC02 BA03 CA03 CB05 EA13                       FA14 FA37 FA42 FA46 FB25                       FB28 FB44                 5H223 AA01 AA03 CC03 DD09 EE08                       FF03                 5K048 AA04 BA22 BA23 EB02 EB15                       FB05 HA01 HA21

Claims (4)

[Claims] 1. An information processing apparatus having a function of displaying, on a display, a live-action image being captured by a camera, wherein shape information of an object shown in the real-image is displayed on a display position of the object. Shape information storage means for correspondingly storing, position designating means for designating a position in the photographed image displayed on the display, and when the position is designated by the position designating means, stored in the shape information storage means An information processing apparatus, comprising: an object identifying means for determining which object in a live-action image is designated based on the obtained information. 2. The information processing apparatus according to claim 1, further comprising processing execution means for executing processing corresponding to the identified thing when the thing is identified by the thing identifying means. 3. The method according to claim 2, wherein the processing execution means has a graphics display means for superimposing and displaying graphics on a real image of the object based on shape information corresponding to the identified object. Information processing device. 4. The shape information storage means according to any one of claims 1, 2 and 3, wherein the shape information is rectangular, circular,
An information processing device characterized in that it is expressed and stored as one of a polygon, a free curve, a polygonal line, or a combination thereof.