JP3238205B2 - Screen tagging method and device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[Industrial applications] Video images (hereinafter simply referred to as "images")
The present invention relates to a man-machine interface (hereinafter, also simply referred to as “man-machine”) system using a video camera (hereinafter, simply referred to as “camera image”). The present invention relates to a method and an apparatus for providing a means for associating various kinds of preset information with a subject constituting the subject.

[0002]

2. Description of the Related Art As a conventional plant monitoring method, for example, a memo is put on a device, an operation panel, a display, a system monitoring panel, or the like existing at a site to call a plant user's attention. At this time, various contents are described on a paper with glue adhered on the back side, and the paper is attached to the target device. For example, the failure state of the equipment, precautions for use, indication that the test is being performed, various changes, and the like are described on paper.
These papers (memos) are used for contacting other workers, memorandums of oneself, and the like. Affixing a memo to a device in this way is referred to as "tag hanging", and a memo is referred to as a "tag". On the other hand, in a plant monitoring and control system, remote control and centralization of monitoring and control are progressing. Various devices in the plant are centrally monitored and controlled from an operation room located at a remote point, and the operator performs most of the operation and monitoring work via a display in the operation room and input devices such as various instructions. Do. In such a work mode, it is very inconvenient to use the conventional bill hanging. This is because it is necessary to actually go to the site in order to attach a tag to the device at the site or to refer to a tag attached to the device at the site. Further, there may be a case where an object to be hung is present only on the display screen. For example, the conventional system monitoring panel is being replaced with a system diagram display on a display screen, and in that case, it is necessary to perform billing on the system diagram displayed on the screen.
However, even if a memo is pasted on the display screen in order to change the display content of the display as needed, it does not mean that the memo is pasted on a display object on the screen. Hypert
ext '89 Proceedings (Pittsburgh, Pennsylvania, No
v. 5-7, 1989) ACM, New York, 1989, `` InterNote: Extend
ing a Hypermedia framework to support annotative c
ollaboration '' (Catlin, T., Bush, PE, Yankelovic
h, N.) discloses a technique for annotating a document displayed on a screen. This technique can be interpreted as a type of technique for electronically tagging a selected portion of a document displayed on a screen. However, the above-mentioned document only discloses a technique of tagging document data, and screen components other than editing targets such as document data, for example, components of a man-machine interface (menu,
There is no disclosure of a technique for tagging a subject or the like shown in a video image, such as an icon, a graph, or a meter.

[0003]

As described above, the prior art has the following problems. First, it is necessary to actually go to the site when tagging the equipment at the site, and similarly, it is necessary to actually go to the site when referring to the tag attached to the device at the site. Extremely bad. Second, annotations, memos, and the like cannot be added to screen components other than the data to be edited (for example, document data) (that is, “tagging” cannot be performed). Therefore,
SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problems of the prior art and achieve the following matters. That is, first, it is necessary to attach a tag to a device at a site while being in an operation room and to refer to the attached tag. Second, it is an object to provide a device that can attach a tag to a screen component other than the data to be edited. In particular, it is possible to attach a memo, an annotation, and the like to a subject shown on a video image.

[0004]

SUMMARY OF THE INVENTION In order to achieve one of the above objects, the present invention provides a method for transmitting information including sound or image.
Remember that it is associated with the subject, and
When the image of the subject is included in the video being played
In the video image, the information is related to the subject.
Display graphics to indicate that
A screen tag hanging method is provided .

[0005]

According to the screen tag hanging method of the present invention, the object
Is included in the video being played,
A graphic that indicates that information is associated with the
Is displayed in the video image.

Embodiments of the present invention will be described below. First, information (hereinafter referred to as "tag" or "memo") using various media such as text, graphics, audio, and video.
Create Next, on the video image displayed on the screen, a subject or an area to be associated with the information is specified. The means for performing the association includes the designated subject,
Alternatively, information corresponding to the position on the video is stored. Next, if the currently displayed image has a tag-applied subject or an area, the fact that the tag is applied is clearly indicated by a method such as displaying graphics in a video image. . In addition, when a designated place is designated by means for clearly indicating a bill hanging state, bill information is output. If the information is text, drawing, video image or the like, it is displayed on the screen, and if it is audio, necessary information is output from a speaker or the like. Of course, even if the tagging state is not specified, if the tagged subject or area is specified, the information (items described on the tag) is output as long as there is information to be referred to. As described above, it is possible to provide a device and a method for attaching a tag to a device at a site and referring to the attached tag while being in an operation room, and for attaching a tag to a screen component other than data to be edited. In particular, a memo, an annotation, and the like can be added to a subject displayed on a video image.

A plant operation monitoring system according to one embodiment of the present invention will be described with reference to FIGS. Needless to say, the present invention can be applied to various remote operation monitoring systems such as a power plant, a chemical plant, a steel rolling mill, a building, a road, and a water and sewage system. The tag hanging function in the present embodiment is a function of pasting a memo (hereinafter, also referred to as a “tag”) on a display object on the screen.
Tags can be described using various media such as audio, video, text, graphics, and the like. Wall hangings have various uses, such as messages to other operators and memorandums for themselves. It is possible to attach a tag to not only a graphic display on the screen but also a video image.
If the surveillance video at the site is tagged, the actual tag at the site can be simulated and replaced. FIG.
FIG. 1 shows an example of an overall configuration diagram of a plant monitoring control system. Each operator has a workstation (hereinafter referred to as
Operation may be performed as necessary while monitoring the state of the plant using the “WS”. One workstation 100 used by an operator for operation monitoring is shown in the figure.

The workstation 100 includes an audio processing hardware 120, speakers 121 and 122 with built- in amplifiers, and a microphone 123 for inputting, storing, and reproducing audio. The audio processing hardware 120 has an A / D conversion function for converting an analog audio signal into a digital signal, and a D / D converter for converting a digital audio signal into an analog signal.
It has an A conversion function and a signal processing function for compressing / expanding a digital audio signal. The audio signal from the microphone 123 or the microphone 211 at the site where the plant is located is converted into a digital signal by the audio processing hardware 120, further compressed, and processed by the workstation 10.
0 is recorded on a memory, a disk or the like. Conversely, the recorded digital audio is expanded and A / D converted by the audio processing hardware 120, and then output from the speakers 121 and 122. The microphone 123 is mainly used by an operator for recording his own voice, inputting operation commands, and the like. The microphone 123 is arranged at a position where the operator can easily input voice. When it is desired to perform voice input from a place away from the workstation 100, for example, a wireless microphone may be used. 1 video synthesis Hard 110 in the video signal of the workstation 1 0 0, a means for synthesizing displaying the television signal, the camera 111 is for operating personnel mainly record their video is there. As a device for inputting a television signal to the video synthesizing hardware 110, a camera 111, a group of cameras (for example, a camera 210) for monitoring a site where a plant is present, a laser disk player (LD) 220, a video tape recorder (VTR) 22.
1 etc. are conceivable. Compared to these television signals, video signals of the workstation 1 0 0 it has a high resolution. Therefore, the two cannot be synthesized as they are. Therefore, the video synthesizing hardware 110 converts the low-resolution television signal to match the resolution of the workstation 100, and then synthesizes both. The video compositing hardware 110 has a so-called chroma key function. The chroma key function of the work station 1 0 0 video signals, only the portion of the color you there specified, a function for imposing a video signal. For example, when "black" is designated as the chroma key, the video signal is displayed only in the black portion on the screen. No video signal is displayed on the parts other than black,
As a result, the image of the site being shot by the camera 210 is displayed on the display screen of the workstation 100, and the workstation 1 is displayed on the image.
00 will be able to synthesize and display the drawn graphics. In this system, it is the same as the workstation 100 so that a plurality of operators can operate in cooperation with each other.
Alternatively, workstations having equivalent functions are mutually connected via the network 10. The state of various devices in the plant (including, of course, the information on the tag) is transmitted to each workstation via the network 10. Conversely, an operation command from each workstation to each device is also transmitted via the network 10.
Such a network can be easily configured with an information transmission medium such as an optical fiber. The workstation 100 has a local driver 130 and a shared media server 1400 as software for accessing various media devices. Here, as the media device, general devices (camera, LD, VTR, CD-ROM driver (CD), etc.) for inputting, recording, processing, and outputting video, audio, and the like can be considered. Local driver 1
30, the shared media server 140 is a communication port (RS232C, SCSI
) To control each media device. The local driver 130 is a media device connected to each workstation (a media device that is not shared with other workstations, such as the video synthesis hardware 110, the audio processing hardware 120, and the CD-ROM driver 131).
Is installed on each workstation. Shared media server 1
Reference numeral 40 denotes software for controlling a media device shared by a plurality of workstations. For example, the shared media server 140 is installed only on the workstation 100. Other workstations are connected via the network 10 to the shared media server 1
40, and access the shared media device.
The shared media devices include cameras (for example, camera 210, etc.), microphones (for example, microphone 211, etc.), LD220, VTR22
1. Matrix switcher 20 for data switching
0, control switcher 250, camera controller 25
1, 252, and the like. The camera 210, the microphone 211, and the like are provided for visually and audibly confirming a state in the plant from the operation room. Data matrix switcher 200
Is a switching means that can be connected in an arbitrary matrix state in a port configuration of M inputs to N inputs using audio signals and video signals as input signals. Therefore, the data matrix switcher 200 can distribute a signal from one input terminal to a plurality of output terminals. Thereby, for example, the image of the camera 210 can be distributed to a plurality of displays and can be viewed from each display.

[0008] The LD 220 stores manual video information. The VTR 221 is connected to both the input side and the output side of the switcher 200. As a result, the video of the camera monitoring the site and the video of the camera 111 can be recorded, and the recorded video can be output to the display screen of the workstation 200. The operator can use the camera 111 to capture, for example, a video being spoken and record the captured video on the VTR 221, so that a video message can be transmitted to other operators. Of course, instead of recording the video on the VTR 221, the video may be digitized, compressed, and stored in the memory of the workstation, a disk, or the like, similarly to the audio. In FIG. 1, camera controllers 251 and 252 are means for controlling cameras (such as the camera 210) installed on the site, and a control switcher 250 is installed on the site together with the camera controllers 251 and 252. This is a means for switching the connection with the camera. Camera controller 25
1, 252 and the control switcher 250 are connected to the communication port of the workstation 100, respectively.
The camera controllers 251 and 252 are means for freely setting the direction and the angle of view of the camera connected via the control switcher 250. Shared media server 140
Is a client process (C
P1), the camera at the site (for example, camera 2)
10), the control right of the camera 210 is assigned to the CP1 in the following procedure. First, the first procedure will be described. It is checked whether or not the control right of the camera 210 that has requested to control is assigned to another client process. If the control right has already been assigned, the camera 210 notifies the fact that the camera is in use to CP1, and ends the control right assignment process. If the control right has not been assigned, the process proceeds to the second procedure. The second procedure searches for a camera controller that is not used by another client process. When all the camera controllers are in use, the fact is notified to CP1, and the control right assignment process is terminated. If there is a camera controller that is not in use, it is assigned to CP1. Finally, in step 3, a command is output to the control switcher 250, and a controller (hereinafter, referred to as a controller 251) assigned to CP1 is connected to a control line of the camera 210. After the control right is obtained by the above procedure, for example, C
When there is a request from P1 to pan the camera 210 to the left (refer to rotating the camera in the horizontal direction), the shared media server 140 sends an operation command to the camera controller 251 to pan to the left. Upon receiving the operation command, the camera controller 251 drives and controls the motor of the camera platform (referred to as a camera mount having a drive unit) of the camera 210 to pan the camera 210 to the left. C
When the use of the camera 210 ends, P1 informs the shared media server 140 of that fact and releases the control right of the camera 210. By controlling the cameras in the above-described procedure, an arbitrary camera can be controlled from a plurality of client processes with a smaller number of camera controllers than the number of cameras installed at the site. That is, it is only necessary to prepare camera controllers as many as the number of cameras to be operated at the same time. FIG. 2 shows an example of the monitoring screen displayed on the display screen of the workstation 100. 2, a display screen 500 of the workstation 100 includes a plant configuration diagram 510, a graphic display of control data, a camera, and an LD 22.
0, a video image 520 displaying a video signal from the VTR 221 or the like, a billing icon 530, a billing menu 5
31, a pointer 501 displayed in conjunction with a pointing device such as a mouse. In this embodiment, a mouse is used as an example of the pointing device. The mouse is provided with two buttons, which are called a left button and a right button, respectively. In particular, when it is not necessary to distinguish between the left and right buttons, they are simply described as buttons. Of course, it goes without saying that another pointing device such as a tablet, a touch panel, and a trackball may be used instead of the mouse.

In this embodiment, the following operation is performed using a mouse. Operations that "click" is, pushes down the mouse button, an operation to release the button that has been as it is pushed down without moving the position of the mouse. This operation is mainly used to specify a position in the screen 500, a display object, and the like. The operation of “drag” is an operation of depressing a mouse button and moving the mouse as it is. The drag operation ends by releasing the pressed button. By dragging, 2 in the screen 500
Points can be specified. In other words, there are two points: the position of the pointer 501 when dragging is started (when the button is pressed down) and the position of the pointer 501 when dragging is completed (when the button is released). Operate the mouse and hover over the display on the screen 500,
Placing the pointer 501 and clicking (or dragging) the mouse is referred to as clicking (or dragging) a display object. The tag hanging menu 531 includes, for example, “tag hanging sheet”, “link”, “icon display”, “search”, “cut”, “paste”, and the like as selection items.

The function of each item is as follows. The item "tag sheet" is a process for opening a tag sheet which is a tool for creating and editing tags. Item "Link"
Is a process of associating a bill with an object (referred to as a link). The item “icon display” is a process of displaying an icon indicating that a card is hung (referred to as a bill icon). When this item is selected, the item name changes to "Icon hidden". If you select "Hide Icon",
The display of the bill icon disappears, and the item name changes to “icon display”. The item “search” is a process of performing a keyword search based on the contents of a bill. When a keyword is input, a tag including the keyword and an object to which the tag is linked are displayed. The item “cut” is a process of deleting a tag corresponding to a currently selected tag icon and storing data of the tag in a kind of buffer called a paste buffer. The item “paste” is a process for linking the tag stored in the paste buffer to the currently selected object. The following is an outline of a standard procedure for tagging an object on the screen. First, the item "tag sheet" in the tag menu 531 is selected to open the tag sheet 600 shown in FIG. The billing menu 531 is called by clicking the billing icon 530.

In order to select an item of the tag hanging menu 531, a desired item may be clicked with a mouse. (Procedure 1) Next, a tag is created and edited using the tag hanging sheet 600. The bill includes graphic (including text) data, audio data, video data, and the like. The tag sheet 600 has a figure editing function, a voice recording function, a video recording function, and the like in order to edit the data. (Step 2 above) Next, the created tag is saved, and the tag hanging sheet 600 ends. When the note sheet 600 is closed, a note icon is displayed at a default position on the screen 500.

Next, the created bill is linked to the object. Drag the displayed tag icon and move it over the object you want to tag,
When the item 1 “Link” is selected, the bill is linked to the object. (Step 3 above) In the above procedure, the objects include the following. First, the components of the graphics display 703 will be described. For example, user interface tools such as figures, menus, and icons, graphs, images, and the like. Further, a group of a plurality of components can also be an object. Next, the subject of the video image 520 and the devices and objects shown in the video image are also included. Furthermore, the video image itself is included. Each scene of a video image can also be an object. If the camera parameters, that is, the shooting direction of the camera (changes by a pan operation or the like), the angle of view (changes by a zoom operation), and the like change, the target scene also changes. A scene corresponding to a certain camera parameter also becomes one object (referred to as a scene object). Further, a video at a certain time of the video video stored in the LD 220, the VTR 221 or the like is also a scene object. A scene object shot by a camera is called a live video object, and a scene object of a stored video image is called a stored video object. Therefore, an area on a video image (scene object) and an area (including a point) in the scene object can also be an object. The area can be defined, for example, by the shape of a figure. Next, with reference to FIG. 3, a description will be given of a method of creating and editing a tag using a tag hanging sheet.
FIG. 3 shows an area 610 for inputting and displaying the title of the tag, an area 611 for inputting and displaying the name of the creator of the tag, an area 612 for inputting and displaying the address of the tag, and displaying the date and time when the tag is being generated. 613, figure editing area 620, figure editing area 62
Scroll bars 621 and 622 for scrolling the drawing drawn in 0, left and right, up and down, a drawing menu 625 for selecting a drawing object drawn in the drawing editing area 620, and enlargement or reduction of the drawing drawn in the drawing editing area 620 A menu 626 for displaying, a menu 630 for specifying a recording / reproduction target (audio or video), and an icon 631 for controlling recording or reproduction of audio or video.
To 635, management menu 64 for managing bill data
0, a list bar 650 for scrolling the list display 650 and the list display 650 of the object names linked to the tag currently being created. First, the items of the graphic menu 625 and their functions are described below. Item "character"
Is a function for drawing characters. After selecting this item,
When one point in the editing area 620 is clicked with the mouse and characters are input from the keyboard, the characters input from the clicked position are drawn. When the pointer 501 is moved to a position other than the character string currently being input and the mouse is clicked, the character string input ends.

The item "straight line" is a function for drawing a straight line. When the mouse is dragged in the editing area 620 after selecting this item, a straight line is drawn connecting the position of the pointer 501 at the start of dragging and the position of the pointer 501 at the end of dragging. The item “rectangle” is a function for drawing a rectangle. After selecting this item, if the mouse is dragged in the editing area 620, the pointer 50 at the start of dragging
A rectangle is drawn with the position of No. 1 and the position of the pointer 501 at the end of dragging as diagonal vertices. Item `` ellipse ''
Is a function for drawing an ellipse. After selecting this item,
When the mouse is dragged in the editing area 620, an ellipse is drawn which is inscribed in a rectangle whose diagonal is the position of the pointer 501 at the start of dragging and the position of the pointer 501 at the end of dragging. The item “polyline” is a function for drawing a polygonal line. After selecting this item, the mouse is repeatedly moved and clicked in the editing area 620, and when the mouse is finally clicked twice at the same position, the pointer 50 when clicked is clicked.
A polygonal line formed by sequentially connecting the positions 1 with straight lines is drawn. The item “polygon” is a function for drawing a polygon.
After selecting this item, the mouse is repeatedly moved and clicked in the editing area 620. Finally, when the mouse is clicked twice at the same position, the position of the pointer 501 at the time of the click is sequentially connected with a straight line, and the last point and the first point are connected. A polygon is created by connecting the points with straight lines. The item “handwriting” is a function for drawing a freehand line. When the mouse is dragged, a line connecting the passing points of the pointer 501 is drawn. Editing area 62
The figure drawn on the top 0 can be selected and moved using the mouse. By moving the mouse to move the pointer 501 on a desired figure and clicking the left button, the figure is selected. When the mouse is dragged on the editing area 620, all the figures included in the rectangular area having the diagonal vertices between the position of the pointer 501 at the start of dragging and the position of the pointer 501 at the end of dragging are selected. become.
On the other hand, if the mouse is moved to move the pointer 501 on a desired figure, the left button is pressed down, and the mouse is dragged, the figure moves together with the pointer 501. When the right button of the mouse is pressed down while the pointer 501 is on the editing area 620, a graphic object editing menu (not shown) is displayed. When the mouse is moved while keeping the right button pressed, the pointer 501 is moved to a desired item in the graphic object editing menu, and when the right button is released, the desired item is selected. When the right button is released, the display of the graphic object edit menu is deleted.
Hereinafter, the items of the graphic object editing menu and their functions are shown. The item “delete” is an instruction to delete the selected graphic and store the graphic in the paste buffer at the same time. The item “copy” is an instruction to copy the selected figure into the paste buffer. The item “Paste”
This is an instruction to draw the contents of the paste buffer at the position of the pointer 501 when the mouse is clicked most recently.
The item “rotation” is an instruction to rotate a selected figure by a desired angle around a desired position. The item “enlargement / reduction” is an instruction for enlarging or reducing a figure selected to a desired size. The item “synthesis” is an instruction for grouping a plurality of selected graphics. The grouped figures can be edited and operated as one figure. The item “decomposition” is an instruction to decompose a grouped figure into individual figures. Next, a method of leaving a memo by voice using a bill will be described with reference to FIG. First, the item “voice” in the menu 630 is selected. Next, when the icon 635 is clicked, the display of the icon 635 is highlighted. The highlighting of the icon 635 indicates that it is in the voice recording or video recording mode. When entering the voice recording mode, the tag hanging sheet 600 requests the shared media server 140 to connect the output port of the microphone 123 and the input port of the voice processing hardware 120. Upon receiving the request, the shared media server 140 performs a switching process of the data switching matrix switcher 200 via a communication port provided in the workstation 100. Next, when the icon 633 is clicked, recording starts. Speaking into the microphone 123 is recorded as voice data of the bill. The sound input from the microphone 123 is subjected to so-called A / D conversion by the sound processing hardware 120, further compressed, and recorded in a memory (not shown), a disk (not shown) or the like in the workstation 100. . To end the recording of the sound, the user clicks the icon 632. When the recording is completed, the highlighting of the icon 635 is released, and the mode may be shifted to the reproduction mode. When confirming the recorded contents, the user first clicks the icon 631 and moves the pointer to the head of the recorded data. Here, the “pointer” means a position on data at which recording or reproduction is started. When the icon 633 is clicked, the recorded audio data is reproduced from the position of the pointer. That is, audio data stored in a memory, a disk, or the like is read from the position of the pointer, decompression processing and D / A conversion processing are performed by the audio processing hardware 120, and audio is output from the speakers 121 and 122. You. Further, in the case of additionally recording voice, first, the icon 634 may be clicked to move the pointer to the end of the recording data. Next, if the icons 635 and 633 are sequentially clicked, additional recording is started. Next, a method of leaving a memo using a video on a bill will be described. First, the item “video” in the menu 630 is selected. Next, when the icon 635 is clicked, the display of the icon 635 is emphasized. At this time, the tag sheet 600 connects the shared media server 140 to (1) the output port of the microphone 123 and the input port of the audio processing hardware 120, (2) the output port of the camera 121, and the image synthesis. Input port of hardware 110 and V
Request connection to the input port of TR 221.
Upon receiving the request, the shared media server 140 communicates with the data switching matrix switcher 200 via a communication port provided in the workstation 100.
Is performed. Next, when the icon 633 is clicked, recording starts. At this time, the tag sheet 600 is transmitted to the shared media server 140 by the VTR 22.
Request the start of recording 1. The shared media server 140 that has received the request instructs the VTR 221 to start recording via a communication port provided in the workstation 100. At the same time, a tag object (described later) is generated on the memory in the workstation 100, and the recording start address of the VTR 221 is stored in the tag information storage field of the tag object. The address is specified by, for example, a frame number. Of course, designation by a time code or the like is also possible. Camera 111 and microphone 123
, The video and audio of the speaker will be recorded on the VTR 221. To end the video recording, click the icon 632. When the recording is completed, the highlighted display of the icon 635 is released, and the mode shifts to the reproduction mode.

At the end of recording, a recording end address is newly stored in the tag information storage field of the tag object generated at the start of recording. To check the recorded content, the user first clicks the icon 631 and moves the pointer to the beginning of the recorded data. That is, a request is made to the shared media server 140 to rewind the VTR 221 to the recording start address stored in the tag object. When the icon 633 is clicked, the reproduction of the VTR 221 is started from the rewind position. Further, when additionally recording video information, first, the icon 63
By clicking 4, the VTR 221 may be "fast-forwarded" to the beginning of the unrecorded portion of the video tape. Next, if the icons 635 and 633 are sequentially clicked, additional recording is started. Now, the bill created through the above-described processing is managed using the menu 640. The main functions of the menu 640 are described below. “New” is a function for creating a new bill. “Open” is a function for opening an existing tag. “Save” is a function of saving the tag currently being edited. “End” is a function of ending the sheet hung sheet 600. With the list display 650, not only the object on which the tag currently being edited is hung can be displayed, but also the object to be hung can be specified or changed. The link display 650 has an ordinary text editing function, and can input an object name, change or modify an already displayed object name, using a mouse, a keyboard, or the like. Next, a procedure for referring to a bill is described below. First, the item "icon display" of the tag hanging menu 531 is selected to display a tag icon (procedure 1).
Of course, if the bill icon is already displayed, this procedure is unnecessary. Next, the tag icon of the tag to be referred to is clicked twice consecutively (this is called "double-click") (procedure 2). When double-clicked, the tag hanging sheet 600 is opened, and data of the double-clicked tag is displayed. That is, the title, creator, and address of the bill are displayed in areas 610, 611, and 612, respectively. In addition, the names of all objects on which bills are hung are displayed in the list 650. If the tag has graphic data, the graphic data is displayed in the graphic editing area 620. If the bill has audio data, the item “voice” of the menu 630 is in a selected state, and if the bill has video data, the item “video” is in a selected state.

If both data are present, the item "video" is selected by default (of course,
The item “voice” may be selected in a default state.) When the reproduction icon 633 is clicked, the data selected in the menu 630 is reproduced. In the above procedure 2, the tag icon is double-clicked to open the tag hanging sheet 600. However, after clicking the tag icon once and selecting the item "tag hanging sheet" of the tag menu 531, the tag hanging sheet is also displayed. The sheet 600 can be opened. The above procedure 1 is not always necessary.
In other words, even if the icon is not displayed, even if the object on which a tag is hung is directly clicked with the mouse and then the item “slatch sheet” of the tag hanger menu 531 is selected, the tag hanger sheet 600 is opened. be able to. FIG. 4 shows an example of a screen at the time of displaying the bill icon in the above procedure 1. In the figure, bill icons 700 and 70
1, 702 and 703 are exemplified. The bill icon is composed of a combination of three symbols, that is, a symbol representing paper, a symbol representing a speaker, and a symbol representing a person (circle and triangle). Each symbol represents the type of data included in the bill. A paper symbol represents graphic data, a speaker symbol represents audio data, and a human symbol represents video data. That is, in FIG.
Reference numerals 0, 701, 702, and 703 indicate that the corresponding tags have only audio data, have only graphic data, have graphic data and video data, and have graphic data and audio data. The tag can be searched based on the title, creator's name, destination, date, name of the object on which the tag is placed, text entered in the graphic editing area 620, and the like. Tags that match the search conditions are displayed along with the objects on which the tags are placed. For example, when a tag is hung on the subject of the video image 520, the video of the subject is also displayed on the video image 520 at the same time that the tag is searched. FIG. 5 is a conceptual diagram of a realizing method in a case where a subject is attached to a video image. In FIG.
Video image 750, video image 7 taken at 51
50, a modeling object 761 having information on what is being projected, a graphic object 760 defining which region of the video image 750 corresponds to the modeling object, and a modeling object 761
Shows the tag object 762 linked to. The modeling object 761 defines what is shown in the video image. The graphic object 760 defines a position on the video image 750 where the modeling object 761 is shown. The operator has a video image 7
When a position on the position 50 is designated, the graphic object at that position is identified (this refers to checking the indicated target object), and the modeling object to which the graphic object belongs, that is, the subject is identified. Is done. If camera parameters such as the shooting direction, posture, and angle of view of the camera are changed, the shape, position, and the like of the subject and the subject on the video change. Of course, if a plurality of cameras are installed, the subject differs for each camera. Therefore,
Corresponding to the camera and its camera parameters,
A graphic object 760, a modeling object, and the like are defined. In the case of stored video images,
Since the scene changes for each playback section, a graphic object 760, a modeling object, and the like are defined corresponding to the playback section. The tag is an area on the video image 750,
Graphic object 760, modeling object 76
1 can be linked to In the figure, the tag object 762 is linked to the modeling object 761. FIG. 6 shows a detailed data structure example of this embodiment. First, live video objects 800 and 801 defined for each scene will be described. One live video object corresponds to one scene. Further, the live video object has the following attributes. The “active state” indicates whether or not an interactive operation (operation between the operator and the workstation) is possible by a binary flag. When “false” (for example, “0”), no interactive operation is accepted. When “true” (for example, “1”), the interactive operation is accepted. Here, enabling the interactive operation is referred to as “activation”, and disabling the interactive operation is referred to as “inactivation”.
This information is possessed by all objects. “Area information” is an attribute that specifies an area on the screen of the workstation 100. The image of the camera is displayed in the area on the screen specified by the area information. “Camera information” is an attribute configured to include a camera identifier to be photographed and camera parameters. The camera identifier specifies one of a plurality of cameras, and the camera parameters include information including a camera position, an angle of view, a shooting direction (horizontal angle and vertical angle), a posture, and the like when shooting a certain scene. Consists of That is, a scene is determined by the camera information. The “object list” stores a pointer to a graphic object defined for a certain scene or a pointer to a tag object hanging on a certain scene. “Additional information” is information other than the above related to the scene, for example, the type of camera,
Information such as a maintenance history is stored. The stored video image scene 840 shown in FIG. 7 corresponds to the stored video image scene instead of the live video object. The stored video object 840 has a video information attribute instead of camera information. The video information attribute includes an identifier of a video device used for reproduction and information on a reproduction section of the scene. In the playback section, a playback start address and an end address are specified by a frame number or a time code. Other attributes are the same as the live video object. Here, the live video object and the stored video object are collectively referred to as a video object. FIG.
Includes a graphic object 810 defined for each scene.
-813 are also shown. The graphic object is data corresponding to various graphic elements such as a circle, a rectangle, a polygon, and a line. The outline and position of the subject are defined by one or a plurality of graphic objects. Such a graphic object has the following attributes. The “active state” indicates whether or not an interactive operation (operation between the operator and the workstation) is possible by a binary flag. When “false” (for example, “0”), no interactive operation is accepted. "true"
In the case of (for example, “1”), the interactive operation is accepted.

When a live video object is activated, all graphic objects registered in the object list are also activated. “Area information” defines an area occupied by the graphic object on the screen of the workstation 100. When the area defined by the area information is specified, the graphic object is selected. “Part-of” is a pointer to a modeling object whose graphic object represents a region. When a graphic object is selected on the screen, par
This means that the modeling object whose pointer is stored in the t-of information is also selected. Generally part-
“of” stores a pointer to a parent object.
Here, the parent object indicates an object composed of a plurality of other objects (called child objects). The “additional information” refers to information related to graphics other than those described above, for example, information such as the type, color, and presence / absence of painting. Modeling object 820
6 to 823 are shown in FIG. The modeling object is a conceptual object corresponding to the subject, and has information on the subject. The modeling object is composed of at least one graphic object or another modeling object. For example, the burner object 820 is composed of a graphic object 810, the fuel pipe object 821 is composed of graphic objects 811 and 821, and the boiler object 823 is composed of a burner object 820 and a fuel pipe object 821. When the driver points to a point on the video, a graphic object including the point in the area is identified, and a modeling object to which the graphic object is linked is identified. As described above, the link between the graphic object and the modeling object is a part-of-part of the graphic object.
This can be realized by storing a pointer to the modeling object in the attribute.

The modeling object has the following attributes. The “active state” indicates whether or not an interactive operation (operation between the operator and the workstation) is possible by a binary flag. When “false” (for example, “0”), no interactive operation is accepted. When “true” (for example, “1”), the interactive operation is accepted.

When a graphic object as a child object is activated, a modeling object as a parent object is also activated. "Tag object"
Is a pointer to a tag object. When a plurality of tags are hung on one modeling object, the tags are connected in a list. “Part-of” is a pointer to a parent modeling object. In the example shown in FIG. 6, the boiler object is a parent object of the burner object 820 and the fuel pipe object 821, and a pointer to the boiler object 823 is stored in each of the part-of attributes. In the “additional information”, information on the subject other than the above, for example, the burner object 820 has design information, manual information, maintenance information, and the like on the burner as additional information. Further, the tag objects 830, 831
Is shown in FIG. The bill object is the billing sheet 600
It has the following attributes. The “active state” indicates whether or not an interactive operation (operation between the operator and the workstation) is possible by a binary flag. When “false” (for example, “0”), no interactive operation is accepted. When “true” (for example, “1”), the interactive operation is accepted.

When the tag object is activated, a tag icon is displayed on the screen. “Area information” is information for specifying a display area of a bill icon corresponding to a bill object on the screen of the workstation 100. The “tag object” is a pointer to a subsequent tag object. When a plurality of tags are placed on one modeling object, the tag objects are connected in a list using this attribute. When there is no subsequent tag, for example, “0” is set. The “tag information” is information relating to the content of the tag created by the tag hanging sheet 600, and includes a pointer to graphic data, a start address and an end address of audio data, a start address and an end data of recorded data, and the like. It is composed. An “invalid address” is stored in the data that has not been created. Here, the “invalid address” is an address that is not used as an address to actual data. "Additional information"
It has information other than the above, for example, information on the creator of the bill, the creation date and time, the number of references, the referrer, the reference date and time, and the like. FIG. 8 shows an example of a processing procedure for displaying a scene. Less than,
Each processing procedure will be described in detail. First, a video object corresponding to a designated scene is activated (step 1100).

Next, an instruction to activate is sent to all the objects registered in the object list of the activated video object. If the object registered in the object list is a graphic object, the activation state is set to “true” in response to the activation command. The activated object is part-o
An instruction to activate is sent to the parent object specified by the f attribute, and the parent object is also activated. If the pointer to the tag object is registered in the tag object attribute, the activated modeling object sends a command to activate the tag object to activate the tag object. The activated tag object activates the subsequent tag object if a subsequent tag object exists (that is, if a pointer is registered in the tag object attribute). Upon receiving the activation instruction, the tag object not only sets the active state to “true” but also displays a tag icon on the screen (step 11).
10).

Next, scene display processing is performed separately depending on whether the designated scene is a live image from a camera or an image stored by the VTR 221 or the LD 220 or the like.
In the case of live video, the following steps 1130 to 1
The processing of 140 is performed, and in the case of a stored image, the processing of the following steps 1150 to 1160 is performed (step 1120). In step 1130, the shared media server 140 is requested to connect the camera specified by the camera information attribute of the live video object to the video compositing hardware 110. In step 1140, a request is made to the shared media server 140 to set the camera parameters specified by the camera information attribute of the live video object in the camera. In step 1150, if the specified scene is a stored video, the video device (VTR 22) specified by the video information attribute of the stored video object
1 or the LD 220) and the video compositing hardware 110 by requesting the shared media server 140 to connect. In step 1160, the shared media server 140 is requested,
It instructs the video device to repeat playback between the start address and the end address specified by the video information attribute of the stored video object. Finally, the area information of the video object is called, the display position of the video is instructed to the video composition hardware 110 via the local media driver 130, and the composite display of the video on the screen of the workstation 100 is started. The video is displayed in a color different from the display color of the bill icon using the chroma key function (step 1170). Thereby, the tag icon displayed in step 1110 is displayed on the video. In the processing procedure shown in FIG. 8, when displaying an image, a bill icon is also displayed (see step 1110). Of course, the tag icon is not displayed at the time of displaying the image, and at the time of the request from the operator (for example, when the item “icon display” of the tag hanging menu 531 is selected),
A bill icon may be displayed. In the data structure shown in FIG. 6, the video object and the modeling object can be linked to the tag object. That is, the video object is linked to the tag object by setting a pointer to the tag object in the object list attribute. The modeling object links with the tag object by setting a pointer to the tag object in the tag object attribute. When a tag object is linked to a video object, the tag icon is displayed only in the scene specified by the video object. When a tag object is linked to a modeling object, a tag icon is displayed in all scenes in which a subject corresponding to the modeling object is displayed. Of course, objects other than the video object and the modeling object (graphic objects, other objects not shown in FIG. 6, for example, user interface tools such as menus and icons, graphs, images, etc.) are provided with tag object attributes, It is good also as composition which can be linked with a bill object. As described above, according to the present embodiment , the following effects can be obtained. First, the operator, while in the operation room, makes notes on the equipment at the site,
You will be able to refer to the memo. By associating and adding memos to the devices shown in the video of the surveillance camera, the work can be performed as if the user were actually placing a bill at the site. Second, notes can be managed by a computer such as a workstation or personal computer. As a result, there are advantages in that the reliability of storage and the like can be improved and that the search can be speeded up, as compared with the paper tagging. Third,
It is possible to attach cards using media such as audio and video as well as text. Thereby, a memo can be easily created by recording, recording, or the like. Fourth, notes can be added not only to on-site equipment but also to man-machine components on the screen. As a result, it is possible to transmit notes on screen operation. Fifth, communication between operators can be promoted. That is, when the operator is changed,
By creating the items to be sent to the next operator in correspondence with the displayed items on the screen, it is possible to send the items easily and reliably.

[0022]

Tags hanging on, according to the present invention is in the far-septum device can be simulated on the screen, the driver while are in the control room, memo dated to the site of the equipment, can see the like of the memo.

[Brief description of the drawings]

FIG. 1 is an example of an overall configuration diagram of a plant monitoring and control system according to the present invention.

FIG. 2 is an explanatory diagram of a display example on a display of the plant monitoring and control system according to the present invention.

FIG. 3 is an explanatory diagram of a display example of a bill hanging sheet.

FIG. 4 is an explanatory diagram of a display example of a bill icon.

FIG. 5 is an explanatory diagram showing a concept of realizing a tag attached to a video image.

FIG. 6 is an explanatory diagram of an example of a data structure for tagging a video image.

FIG. 7 is an explanatory diagram of an example of a data structure of a stored video object.

FIG. 8 is an explanatory diagram of a processing example when displaying a scene.

[Explanation of symbols]

10 ... network, 100 ... workstation, 1
10: video synthesis hardware, 111: camera, 120: audio processing hardware, 121: speaker, 122: speaker, 1
23: microphone, 130: local media driver, 1
40: Shared media driver, 200: Matrix switcher for data switching, 210: Camera, 211
... microphone, 220 ... laser disk (LD), 221 ...
Video tape recorder (VTR), 250 control switcher, 251 camera controller, 252 camera controller, 500 display screen, 501 pointer, 510 plant configuration diagram, 520 screen 500
Video image displayed on the screen, 530 ... billing icon, 5
31: bill hanging menu, 600: bill hanging sheet, 610
... Area for inputting and displaying the title of the tag, 611 ... Area for inputting and displaying the name of the creator of the tag, 612 ... Area for inputting and displaying the address of the tag, 613 ... Date and time display when the tag is created , 620 ... (graphic) editing area, 621 ... scroll bar, 622 ... scroll bar, 625 ... graphic menu, 626 ... menu for enlargement or reduction, 6
30, a menu for designating a recording / reproducing object, 631 ...
Icon, 632 ... icon, 633 ... icon, 63
4, icon, 635, icon, 640, management menu, 650, object name list display, 651, scroll bar, 700, bill icon, 701, bill icon, 702, bill icon, 703, bill icon, 750
… Video image, 751 camera, 760… graphic object, 761… modeling object (boiler object), 762… bill object, 800… live video object, 801… live video object, 810… graphic object, 811… graphic object, 812 ... graphic object, 813 ... graphic object, 820 ... modeling object, 821 ... modeling object, 822 ... modeling object, 823 ... modeling object, 830 ... bill object, 831 ... bill object, 840 ... accumulated video object

──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Masayasu Futagawa 4026 Kuji-cho, Hitachi City, Ibaraki Prefecture Within Hitachi Research Laboratory, Hitachi, Ltd. (72) Inventor Masato Hotta 4026 Kuji-machi, Hitachi City, Ibaraki Prefecture Hitachi, Ltd. (72) Inventor Shinya Yato 4026 Kuji-cho, Hitachi City, Ibaraki Prefecture Within Hitachi Research Laboratory, Hitachi, Ltd. (72) Inventor Atsuhiko Nishikawa 5-2-1 Omikamachi, Hitachi City, Ibaraki Prefecture, Hitachi, Ltd.Omika Plant ( 72) Inventor Atsuhiko Hirota 5-2-1, Omika-cho, Hitachi City, Ibaraki Prefecture Inside Hitachi, Ltd. Omika Plant (72) Inventor Harumi Uchigasaki 4026 Kuji-cho, Hitachi City, Ibaraki Prefecture Hitachi, Ltd. References JP-A-3-242686 (JP, A) JP-A-5-324233 (JP, A) JP-A-5-197355 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G09G 5/00 G05B 23/02 G06F 3/14

Claims (10)

(57) [Claims] 1. Information including voice or image is related to a subject.
Stores that are attached, an image of the object is included in the video image being displayed
In such a case, the information is provided to the subject in the video image.
Show associated graphics
A method of hanging a screen tag. 2. The screen tag hanging method according to claim 1, wherein when the graphics is instructed, the graphics are displayed.
Outputting information associated with the subject corresponding to the screen. Wherein when displaying the video image of the object photographed by the imaging means, create more pre Me information <br/> report on medium containing voice or video, the information is, the plurality of image pickup means The imaging means number indicating which one is used, and the fact that there is a relation with the imaging means setting parameter including the setting position of the imaging means are stored, and based on the stored information, the information created beforehand is stored in the subject. create graphics <br/> box indicating that the associated screen tags hanging wherein the storing. 4. A screen tag hanging method according to claim 3, wherein, if an imaging unit setting parameter is selected, displays the graphics associated with the imaging means setting parameters, the instruction means the graphics displayed when instructed, the screen label threading method and outputting the information of the previously created. When 5. A display video images of the object photographed by the imaging means, create more pre Me information <br/> report on medium containing voice or video, the information is specified on the display image It is stored that there is a relationship with the area, and based on the stored information,
A screen tagging method, wherein graphics indicating that the previously created information is associated with the subject are created and stored. 6. A screen tag hanging method according to claim 5, wherein, if the specified area on the Viewing image is selected, displays the graphics associated with the specified region on the display image, the display It has been when instructed by the instruction means graphics screen label threading method and outputting the information of the previously created. 7. The method according to claim 1, wherein a plurality of scenes of the subject are photographed and stored as video images, and the graphics are displayed in association with each scene. A screen tagging method characterized in that: Photographing an object at 8. The imaging unit, when displaying a video image of the subject, to create a by <br/> Ri予Me information on medium containing voice or video, the object of the information A screen tag that displays a graphic indicating that the pre-created information is associated with the subject based on the stored information. a multiplying method, to display only the video images of the Utsushitai, then only when there is some indication of the subject graphic to said instruction unit
It is determined whether the scan is over, if the graphics is trained, the screen label threading method and outputting the information associated with the graphics. 9. A screen label hooking device for chromatic display means of the captured image, comprising means for storing information associated with the object, the display means, the image of the subject in the captured image being displayed Where is included
In case, the pre-Symbol Information to the subject is associated
Is displayed in the displayed captured image.
A screen tag hanging device. Te 10. The method of claim 9 screen label hanging device smell according <br/>, comprising means for instructing a part of the captured image, before Symbol Display means, means for indicating a portion of the captured image With the graphic
If is instructed, it outputs information that has been stored in association with the graphic screen label hanging device, characterized in that.