JP2013150171A - Recording device, recording system and recording method - Google Patents

Abstract

An object of the present invention is to record the details of human behavior with high accuracy.
An image captured by an imaging device is input, a specific object is detected from the image, the captured image is associated with information on the detected specific object, and the detected information on the specific object is used. A recognition rule is set, and according to the set recognition rule, a video recognition process is performed on the video associated with the information on the specific object used when setting the recognition rule, and the video recognition result is recorded.
[Selection] Figure 5

Description

  The present invention relates to a recording apparatus, a recording system, and a recording method.

  A household account book, diary, etc. are used to record the behavior of people during their lives. Even when these records are taken on a computer, they are usually made manually according to the stored and issued receipts, which is an obstacle to many people trying to record their actions. ing.

  In order to remove such obstacles, a system described in Non-Patent Document 1 has been studied. In this system, a camera is attached to the head, the behavior of the wearer is photographed, the photographed image is analyzed, and the behavior of the wearer is recorded.

EYEWATCHME-3D Hand and object tracking for insight out activity analysis, IEEE Computer Society Conferencing on Computer Vision and Pattern Recognition 9 Rapid Object Detection Using a Boosted Cascade of Simple Features, Proceedings of the 2001, IEEE Computer Society Conference on Computer Vision and Cognition.

  In the system shown in Non-Patent Document 1, it is necessary to always perform shooting by operating a camera in order to record the behavior of the wearer. In addition, it is necessary to analyze all the captured images and recognize what actions have been taken.

  However, if photographing is always performed, the battery of the camera that performs photographing becomes extremely exhausted. In addition, since the system does not know what context the wearer is in, the action of the photographed wearer must be classified as one of all possible actions, and where the break of action is You must also recognize it. In addition, it is necessary to classify an object to be photographed as one of all kinds of objects in the world. This increases the difficulty of classifying the wearer's behavior and lowers the accuracy of the classification.

  The present invention has been made in view of such problems, and an object of the present invention is to record human action content with high accuracy.

  The above object is achieved by the following apparatus or method.

That is,
An input means for inputting video shot by the photographing device;
Detecting means for detecting a specific object from the video;
Associating means for associating the input video with the information of the detected specific object;
Setting means for setting a recognition rule using information of the detected specific object;
In accordance with the set recognition rule, a video recognition result recording unit that performs video recognition processing on a video associated with information on a specific object used when setting the recognition rule, and records a video recognition result;
A recording apparatus comprising:

Or an input step in which the input means inputs the video imaged by the imaging device;
A detecting step for detecting a specific object from the video;
An associating unit associating the input video with the information on the detected specific object;
A setting step in which a setting means sets a recognition rule using information on the detected specific object;
A video recognition result in which video recognition result recording means performs video recognition processing on the video associated with the information of the specific object used when setting the recognition rule according to the set recognition rule, and records the video recognition result Recording step;
A recording method comprising:

  Moreover, the said objective is achieved by the following systems or methods.

That is, a recording system including a photographing device and a server device,
The imaging device
Photographing means for photographing images;
Detecting means for detecting a specific object from the video;
The server device includes transmission means for transmitting the captured video and the information of the detected specific object,
The server device
Receiving means for receiving the transmitted video and the transmitted information on the specific object from the imaging device;
Association means for associating the received video with the received information on the specific object;
Setting means for setting a recognition rule using the received information on the specific object;
In accordance with the set recognition rule, a video recognition result recording unit that performs video recognition processing on a video associated with information on a specific object used when setting the recognition rule, and records a video recognition result;
A recording system comprising:

Alternatively, a photographing step in which the photographing means of the photographing device photographs a video,
A detecting step of detecting a specific object from the video by the detecting means of the photographing apparatus;
A transmission step in which the transmission unit of the imaging device transmits the captured video and the information of the detected specific object to a server device;
A receiving step in which the receiving unit of the server device receives the transmitted video and the transmitted information on the specific object from the imaging device;
An association step in which the recording means of the server device associates the received video with the received information on the specific object;
A setting step in which the setting unit of the server device sets a recognition rule using the received information on the specific object;
The video recognition result recording means of the server device performs video recognition processing on the video associated with the information on the specific object used when setting the recognition rule according to the set recognition rule, and records the video recognition result And a video recognition result recording step.

  According to the present invention, it is possible to record a person's action content with high accuracy.

It is the figure which showed roughly the flow which records an image | video in the action recording system. It is the figure which showed roughly the flow which performs video recognition and records an action in the action recording system. It is the figure which showed the specific example of the form of a mounting | wearing type device. It is the figure which showed the object relevant to the action which a user wants to record in life. FIG. 3 is a diagram showing a configuration of a wearable device 701. 10 is a flowchart describing a procedure of how the wearable device 701 performs shooting. 10 is a flowchart illustrating a procedure of how the wearable device 701 recognizes a captured image. It is the flowchart which showed the flow of the process of recording of a mirror image. It is the flowchart which showed the flow of the recording process of body temperature. It is the flowchart which showed the flow of the recording process of taking. It is the figure which showed an example of the system configuration etc. of a recording system.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Embodiment 1>
The action recording system according to the present embodiment has an imaging unit and records the action of the person to be recorded as a video. In addition, a specific object used by the person to be recorded in life is detected by some means, and recorded in association with the captured video. Furthermore, when creating a behavior record by analyzing the captured video, video recognition is performed using different recognition rules depending on the detected object, and the video recognition result is recorded as a database in a memory or the like as a behavior record. .

  FIG. 1 is a diagram schematically showing a flow of recording video in an action recording system.

  In S901, when action recording is started, video is taken and recorded (input).

  In S902, a specific object is detected by some means.

  In S903, the captured image and the information of the detected specific object are associated and recorded in a memory or the like.

  FIG. 2 is a diagram schematically showing a flow of performing action recognition and recording actions in the action recording system.

  In S101, when video recognition is started, a video recognition rule is set according to the information on the detected specific object.

  In S102, a video recognition process is performed on the captured video associated with the information on the specific object according to the set video recognition rule.

  In S103, the video recognition result is recorded in a memory or the like as a database (video recognition result recording).

  Hereinafter, the processing and the like will be described more specifically. In the present embodiment, a body-mounted device with a camera used for health care is described. By using this wearable device, the user can automatically record behaviors related to health management, such as medication, in daily life without taking notes.

  FIG. 3 shows the appearance of the wearable device in the present embodiment. In FIG. 3, the pendant device 701 and the user wear an ear-mounted device 702 and the like. Although the pendant device 701 and the ear-mounted device 702 are different in form, both are devices having the same function and perform the same operation. Note that the form of the device is not limited to this, and any form may be used as long as it has an imaging unit, and the space in front of the user always enters the field of view of the imaging unit and moves with the user. Good. In the following description, it is assumed that the user wears the pendant wearable device 701.

  FIG. 4 is a diagram showing objects related to actions that the user wants to record in his / her life in the present embodiment. The mirror 801 in FIG. 4 is used daily by the user. An RFID tag 811 having a unique ID is attached to the mirror 801 in advance. A thermometer 802 is used when a user wants to measure body temperature. An RFID tag 812 having a unique ID is attached to the thermometer 802 in advance. Reference numeral 803 denotes a medicine bottle, which is a translucent container in which medicines commonly used by users are placed. An RFID tag 813 having a unique ID is attached to the medicine bottle 803 in advance.

  The wearable device 701 shown in this embodiment records actions related to the three objects shown in FIG. That is, when the user of the wearable device 701 uses the mirror 801, the user's face image reflected in the mirror and the date and time are recorded. By taking this record, the daily facial image of the user can be continuously recorded, and the image can be used in the health management system outside the wearable device 701. Similarly, when a user measures body temperature, the body temperature and date / time can be recorded, and when a user takes medicine, the date / time can be recorded and used by an external health management system.

  In the present embodiment, the specific object detection unit 101 includes an RFID tag attached to the specific object and an RFID tag reader built in the wearable device 701. However, this may be configured as long as it can detect a specific object without contact. For example, a camera built in the wearable device 701 captures a two-dimensional code attached to the specific object. The ID may be acquired.

  FIG. 5 is a diagram showing the configuration of the wearable device 701 in the present embodiment. Hereinafter, the configuration of the wearable device 701 in the present embodiment will be described with reference to the drawings.

  The specific object detection unit 101 detects one or more specific objects used by the user. In the present embodiment, the specific objects are the mirror 801, the thermometer 802, and the medicine bottle 803 shown in FIG. In this embodiment, the specific object detection unit 101 includes an RFID tag that is attached to each of a mirror, a thermometer, and a medicine bottle, and an RFID tag reader that is built in the wearable device 701. The RFID tag operates with no power supply, and unique ID information is read from the RFID tag reader. The reading range of the electronic tag reader is set in the front direction of the wearable device 701, and the reading distance is set to about 60 cm. With this setting, when the user wearing the wearable device 701 holds a specific object, the ID information of the electronic tag attached to the specific object is read. Therefore, the wearable device 701 can specify the type of the specific object in the field of view of the built-in imaging unit.

  In the present embodiment, the specific object detection unit 101 includes an RFID tag attached to the specific object and an RFID tag reader built in the wearable device 701. However, this may be configured as long as it can detect a specific object without contact. For example, a camera built in the wearable device 701 captures a two-dimensional code attached to the specific object. The ID may be acquired.

  The imaging unit 102 includes a camera module built in the wearable device 701. The imaging unit 102 can capture a moving image. The imaging range of the imaging unit 102 is adjusted to an angle approximately equal to the reading range of the RFID tag reader built in the wearable device 701.

  The recording unit 103 records the specific object ID information detected by the specific object detection unit 101 and the moving image captured by the imaging unit 102 in association with each other. The recording unit 103 is configured by a flash memory or the like.

  The video recognition unit 104 recognizes the content of the moving image stored in the recording unit 103 and stores the recognition result in the database 106 stored in a memory or the like. The video recognition unit 104 is realized by a microprocessor built in the wearable device 701 executing a program.

  The recognition rule setting unit 105 sets a recognition rule when the video recognition unit 104 recognizes a moving image and stores the recognition result in the database 106 stored in a memory or the like. The recognition rule setting unit 105 is realized when a microprocessor built in the wearable device 701 executes a program.

  The database 106 is a result recognized by the video recognition unit 104. The database 106 is stored in a memory such as a flash memory. However, the database 106 may be any medium as long as it can record digital information. For example, the database 106 may be stored in a memory such as a hard disk.

  FIG. 6 is a flowchart showing a procedure of how the wearable device 701 performs shooting in the present embodiment. The procedure described in FIG. 6 is realized by a control board built in the wearable device 701 and a microprocessor on the board executing a program. Hereinafter, the procedure for the wearable device 701 to perform shooting will be described with reference to FIG.

  In S201, when the wearable device 701 is turned on, it is determined whether the specific object detection unit 101 recognizes the tag attached to the specific object. If the tag is recognized, the process proceeds to S202, and if not, the process proceeds to S204.

  In S202, it is determined whether the wearable device 701 is shooting. If it is already shooting, the process proceeds to S201, and if not, the process proceeds to S203.

  In S203, moving image shooting by the imaging unit 102 is started, and the moving image shot in the recording unit 103 is recorded. At this time, the detected tag information and the date and time information at that time are also recorded in the recording unit 103 in a pair with the recorded moving image. When shooting is started, the process proceeds to S201.

  In S204, it is determined whether the wearable device 701 is shooting. If shooting is in progress, the process proceeds to S205, and if not, the process proceeds to S201.

  In S205, it is determined whether or not 30 seconds have elapsed since the photographing start time. This determination is performed so that a large number of sliced images are not captured when the tag is at the limit of the reading range. If 30 seconds have elapsed, the process proceeds to S206, and if not, the process proceeds to S201.

  In S206, shooting is stopped.

  As described above, by executing the procedure shown in the flowchart of FIG. 6, it is possible to realize the wearable device 701 that performs imaging only when there is a specific object in the vicinity of the user. It is possible to take a picture while using it in the hand.

  FIG. 7 is a flowchart showing a procedure of how the wearable device 701 recognizes a captured image in this embodiment. The procedure shown in FIG. 7 is realized by the microprocessor built in the wearable device 701 executing a program. Hereinafter, a procedure in which the wearable device 701 recognizes an image will be described with reference to FIG.

  In S <b> 301, when the wearable device 701 is turned on, the video recognition unit 104 first captures a moving image stored in the recording unit 103 and has not yet been recognized. Take it out. In addition, the recognition rule setting unit 105 extracts tag information recorded as a pair from the recording unit 103 with the moving image. If there is no moving image that has not been recognized in S301, the process waits in S301 until a new moving image is recorded.

  In S302, the recognition rule setting unit 105 uses the tag information to determine which specific object has been detected when the moving image is captured. If the tag information matches the tag information attached to the mirror 801, the tag information attached to the medicine bottle 803 is matched to S303 if it matches the tag information attached to the thermometer 802. If so, the process proceeds to S305.

  In S303, a mirror image is recorded.

  FIG. 8 is a flowchart showing a flow of mirror image recording processing. The flow of the mirror image recording process will be described with reference to FIG. In recording a mirror image, a process of extracting a face image of the user reflected in the mirror when the user of the wearable device 701 uses the mirror is performed.

  In S401, face recognition processing is performed for each frame of the recorded moving image. In addition, for a frame in which a face is detected, a higher score is assigned as the face is larger. This face recognition process may be performed by any method that can detect a face from an image. For example, the face recognition process is realized by using a Viola-Jones face detector shown in Non-Patent Document 2. Also good.

  In S402, a higher score is assigned to the frame in which the face is detected as the face faces the front. This process may be performed by any method as long as it can detect the orientation of the face. For example, a plurality of face detectors corresponding to the orientation of the face are prepared and applied in advance.

  In S403, the sum of the scores calculated in S401 and S402 is compared, and the frame having the maximum score is selected. From the selected frame, an image is cut out with a rectangle including a face. The cut face image and the time when the face image was taken are recorded in the database 106 as a pair.

  Through the above procedure, the image recognition unit 104 cuts out the face image of the user reflected in the mirror and records it in the database 106 together with the date and time when the image was taken. In the present embodiment, the face image of the user is cut out by the face detector, but the face image may be cut out by other processing. For example, the contour region of the mirror may be detected by means such as edge detection, and the face detector may be used only within the mirror region. Further, an individual identification procedure by face authentication may be added after face detection, and the face image may be cut out only when it is confirmed that the face of the user of the wearable device 701 is reflected.

  In S304, the body temperature is recorded.

  FIG. 9 is a flowchart showing the flow of the body temperature recording process. The flow of the body temperature recording process will be described with reference to FIG. In the recording of the body temperature, the user of the wearable device 701 performs a process of recording the measured body temperature information after using the thermometer.

  In S501, the thermometer detection process is performed from the last frame of the recorded sequence of the recorded video. This process may be performed by any method that can detect a thermometer. For example, as shown in Non-Patent Document 1, it is realized by an object detection and tracking method using a 3D model of a thermometer provided in advance. can do. In the present embodiment, it is known in advance that there is a high possibility that the thermometer is reflected in the screen based on the tag information. Therefore, the object detection and tracking may be performed using only the 3D model of the thermometer. When the frame in which the thermometer is detected is found, the process proceeds to S502.

  In S502, it is determined whether or not the region of the temperature display unit is detected within the region of the detected thermometer in the image. This process may be performed by any method as long as the temperature display unit can be detected. For example, the process can be realized by the same method as that used in S501. When the temperature display unit is detected, the process proceeds to S503. If the temperature display unit is not detected because the temperature display unit is on the opposite side as viewed from the user, the process proceeds to S501.

  In step S503, the body temperature display in the detected temperature display area in the image is read. This reading is performed by an OCR (Optical Character Recognition) technique. For example, when performing object detection and tracking using a 3D model in S501, the relative three-dimensional position / posture of the thermometer with respect to the camera has been clarified. Therefore, the accuracy of OCR can be further improved by applying the conversion process to the image of the body temperature display region using the information. When the body temperature display is read as a result of the OCR, the body temperature information thus read and the time at that time are paired and recorded in the database 106, and the body temperature recording process ends. If the body temperature display cannot be read, the process proceeds to S501.

  Through the above procedure, the image recognition unit 104 reads the temperature display of the thermometer and records it in the database 106 together with the date and time when the image was taken.

  In S305, the dose is recorded.

  FIG. 10 is a flowchart showing the flow of processing for taking a dose. In the following, the flow of processing for taking a dose will be described with reference to FIG. In the medication recording, a process of recording the time when the user of the wearable device 701 took the medicine in the medicine bottle and the amount of medicine remaining in the medicine bottle is performed.

  In S601, a detection process of a medicine bottle standing upright with respect to the screen is performed from the last frame of the recording sequence of the recorded video. The reason why the detection process is limited to the upright medicine bottle is to make it easier to grasp the remaining amount of medicine in the process of S602 described later. This process may be performed by any method that can detect a medicine bottle. For example, as shown in Non-Patent Document 1, an object detection and tracking method using a 3D model of a medicine bottle given in advance. Can be realized. In this embodiment, since it is known in advance that there is a high possibility that a medicine bottle appears in the screen based on tag information, object detection and tracking may be performed using only the 3D model of the medicine bottle. If a frame in which an upright medicine bottle is detected is found, the process proceeds to S602.

  In S602, a process for recognizing the remaining amount in the medicine bottle is performed. This process may be performed by any method that can detect the remaining amount of the medicine in the medicine bottle. For example, the color detection process is performed in the area of the medicine bottle in the image using the medicine color. The remaining amount can be grasped by a method of obtaining the ratio of the drug color area in the medicine bottle. At this time, since the medicine bottle stands upright, the region connected to the bottom of the medicine bottle can be determined as the medicine portion in consideration that the lower direction of the image is the direction of gravity. When the remaining amount of medicine is detected, the read remaining amount of medicine and the time at that time are paired and recorded in the database 106, and the process of recording medication is completed.

  Through the above procedure, the image recognition unit 104 grasps the remaining amount in the medicine bottle and records it in the database 106 together with the date and time when the image was taken.

  The first embodiment has been described above. By wearing the body-mounted device described in the present embodiment, it becomes possible to automatically take a record for health management in daily life.

  In the present embodiment, the wearable device is used for health management. However, the wearable device is not limited to this application, and can be used for recording various actions using objects. For example, when an RFID tag is attached to a wallet and the RFID tag is detected, the usage history of the wallet can be automatically recorded by applying a recognition rule for recognizing an action related to the wallet. . In addition, record the usage history of your mobile phone, record the contents of the refrigerator, record the contents of the bag, record breastfeeding, and record the seal with a seal. Or record the usage history of the medical device.

  In this embodiment, the time and the result of video recognition are recorded in the database. However, the wearable device may be provided with a position detection device such as GPS, and the position may be recorded together.

  In this embodiment, all processing is performed in the wearable device. However, a part of the processing such as recognition processing is placed on the server side via the wireless network, and information is transmitted from the wearable device. It is good also as what performs a process. The wearable device and the server device may perform transmission / reception via wireless communication. FIG. 11 is a block diagram showing such a configuration. In this configuration, the wireless transmission unit 201 is added to the wearable device 701, and the video captured by the imaging unit 102 and the specific object ID information detected by the specific object detection unit 101 are transmitted via wireless communication. The wirelessly transmitted information is received by the server device including the wireless reception unit 202. The server device includes a recording unit 103, a video recognition unit 104, a recognition rule setting unit 105, and a database 106. In this configuration, the processing performed by each block and the flow of processing are the same as those already described.

  In the present embodiment, the video recognition rule corresponding to the specific object is determined in advance, and the combination of the specific object and the video recognition rule when the specific object is detected is also determined. . However, this may be configured to be variable, for example, video recognition rules may be created using an editor on a computer. A combination of a specific object and a video recognition rule may be created in advance using an editor on a computer.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

  As mentioned above, according to each embodiment mentioned above, a person's action content can be recorded with high accuracy.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

101 specific object detection unit, 102 imaging unit, 103 recording unit, 104 video recognition unit, 105 recognition rule setting unit, 106 database

Claims (12)

An input means for inputting video shot by the photographing device;
Detecting means for detecting a specific object from the video;
Associating means for associating the input video with the information of the detected specific object;
Setting means for setting a recognition rule using information of the detected specific object;
In accordance with the set recognition rule, a video recognition result recording unit that performs video recognition processing on a video associated with information on a specific object used when setting the recognition rule, and records a video recognition result;
A recording apparatus comprising:   The recording apparatus according to claim 1, wherein the recording apparatus is a body-mounted device.   The recording apparatus according to claim 1, wherein the detection unit detects the specific object by reading an electronic tag attached to the specific object with an electronic tag reader.   The setting means sets any one of a recognition rule for recognizing a mirror image, a recognition rule for recognizing body temperature, and a recognition rule for recognizing medication taking according to information on a specific object detected by the detection means. The recording apparatus according to claim 1, wherein the recording apparatus is a recording apparatus. A recording system including an imaging device and a server device,
The imaging device
Photographing means for photographing images;
Detecting means for detecting a specific object from the video;
The server device includes transmission means for transmitting the captured video and the information of the detected specific object,
The server device
Receiving means for receiving the transmitted video and the transmitted information on the specific object from the imaging device;
Association means for associating the received video with the received information on the specific object;
Setting means for setting a recognition rule using the received information on the specific object;
In accordance with the set recognition rule, a video recognition result recording unit that performs video recognition processing on a video associated with information on a specific object used when setting the recognition rule, and records a video recognition result;
A recording system comprising:   The recording system according to claim 5, wherein the photographing apparatus is a body-mounted device.   The recording system according to claim 5 or 6, wherein the detection unit detects the specific object by reading an electronic tag attached to the specific object with an electronic tag reader.   The setting means sets any one of a recognition rule for recognizing a mirror image, a recognition rule for recognizing body temperature, and a recognition rule for recognizing medication taking according to information on a specific object detected by the detection means. The recording system according to claim 5, wherein the recording system is a recording system.   The recording system according to claim 5, wherein the photographing apparatus and the server apparatus perform transmission / reception via wireless communication. An input step in which the input means inputs the video imaged by the imaging device;
A detecting step for detecting a specific object from the video;
An associating unit associating the input video with the information on the detected specific object;
A setting step in which a setting means sets a recognition rule using information on the detected specific object;
A video recognition result in which video recognition result recording means performs video recognition processing on the video associated with the information of the specific object used when setting the recognition rule according to the set recognition rule, and records the video recognition result Recording step;
A recording method comprising: A photographing step in which the photographing means of the photographing device photographs a video;
A detecting step of detecting a specific object from the video by the detecting means of the photographing apparatus;
A transmission step in which the transmission unit of the imaging device transmits the captured video and the information of the detected specific object to a server device;
A receiving step in which the receiving unit of the server device receives the transmitted video and the transmitted information on the specific object from the imaging device;
An association step in which the recording means of the server device associates the received video with the received information on the specific object;
A setting step in which the setting unit of the server device sets a recognition rule using the received information on the specific object;
The video recognition result recording means of the server device performs video recognition processing on the video associated with the information on the specific object used when setting the recognition rule according to the set recognition rule, and records the video recognition result And a video recognition result recording step.   The program for making a computer perform each step of the recording method of Claim 10 or 11.

Images