JP2017016347A - Information processing device, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly display information necessary to a user, such as a doctor.SOLUTION: An information processing device includes portion specification means for specifying a portion of a patient on the basis of a photographed image of the patient, photographed by photographing means, position specification means for specifying a portion position in the real space of the portion, and display processing means for displaying patient information on the portion specified by the portion specification means in display means in association with the position of the display means corresponding to the portion position of the real space on the basis of a positional relation between the display means and the photographing means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  Conventionally, with the spread of computerization in the medical field, it has become possible to manage a large amount of information such as images and documents related to patients using a device such as a PC. For example, it is possible to use patient information such as displaying patient information on a display installed in an examination room, and a doctor confirming the display and returning the line of sight to the patient again for treatment. At this time, there is a problem that a motion of the doctor moving his / her line of sight occurs and an operation of associating the displayed information with the patient occurs. On the other hand, Patent Document 1 discloses a technique in which a doctor wears a wearable display and displays information on devices that are not included in the view of the doctor on the wearable display. This allows the doctor to see information about the device that is not visible on the spot without diverting the field of view from the patient.

JP 2015-19678 A

  However, in the technique of Patent Document 1, information on the device can be viewed simultaneously with the patient, but there is a problem that information other than the information displayed on the device cannot be displayed. In addition, for example, when there are a plurality of patients and measuring devices in the same space, there is a problem that information measured for patients other than the patient in charge of the doctor can be displayed.

  The present invention has been made in view of such problems, and an object thereof is to appropriately display information necessary for a user such as a doctor.

  Therefore, the present invention provides a part specifying means for specifying a part of the patient based on a photographed image of the patient photographed by the photographing means, a position specifying means for specifying a part position of the part in the real space, Patient information related to the part specified by the part specifying means is displayed on the display means in association with the position of the display means corresponding to the part position in the real space based on the positional relationship between the display means and the imaging means. Display processing means.

  According to the present invention, it is possible to appropriately display information necessary for a user such as a doctor.

It is a figure which shows a medical system. It is a flowchart which shows a display process. 2 is an explanatory diagram of a usage pattern of an information processing apparatus 100. FIG. It is a figure which shows the example of a display of augmented reality. It is a figure which shows the example of a display of the display apparatus which concerns on another example. It is a flowchart which shows the display process which concerns on 2nd Embodiment. It is a figure which shows the relationship between an apparatus and a patient. It is a flowchart which shows the display process which concerns on 3rd Embodiment.

(First embodiment)
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a medical system as an information processing system according to the first embodiment. The medical system includes an information processing device 100 and a server device 110. The information processing apparatus 100 and the server apparatus 110 are connected via a network 120. The information processing apparatus 100 is a wearable apparatus having a transmissive display unit, and displays augmented reality in which information is superimposed on a real space to a user such as a doctor who is a wearer (hereinafter simply referred to as a user). It is a device to provide. In the following embodiment, a case where the user is a doctor will be described as an example. However, the user is not limited to a doctor and may be a medical worker. As the information processing apparatus 100, for example, a glasses-type apparatus or a head-mounted apparatus such as a head-mounted display can be used. The user can visually recognize the augmented reality in which the information displayed on the display unit is superimposed on the real space through the display unit. In the present embodiment, the information processing apparatus 100 displays patient information such as medical treatment contents and examination results regarding the patient under examination on the display unit. The server device 110 stores and manages patient information.

  In the present embodiment, patient information is stored in the server device 110, and the information processing device 100 receives patient information from the server device 110 as appropriate and displays it. However, the patient information may be stored in any device of the medical system, and the device that stores the patient information is not limited to the server device 110. As another example, the information processing apparatus 100 may store and manage patient information. As another example, the medical system may distribute storage and management of patient information to a plurality of devices.

  The information processing apparatus 100 includes a CPU 101, a memory 102, a storage device 103, a display device 105, a photographing unit 106, a microphone 107, and a communication I / F unit 108. These units are connected to each other by a bus 109. The CPU 101 executes various functions to be described later by executing a program 104 stored in the storage device 103. The memory 102 temporarily stores programs and data read by the CPU 101 from the storage device 103. The memory 102 is also used as an area for the CPU 101 to execute various programs.

  The storage device 103 stores an operating system (OS), various programs, control programs, data, and the like. Note that the control program 104 is stored as part of the data in the storage device 103. Examples of the program include programs related to various recognition processes and display of patient information. Examples of data include images and documents related to patients. Functions and processes of the information processing apparatus 100 described later are realized by the CPU 101 reading out a program stored in the storage device 103 and executing the program. The display device 105 displays various information. The display device 105 is a transmissive display device that does not interfere with the see-through view. The imaging unit 106 captures an image. The microphone 107 collects sound.

  The information processing apparatus 100 receives an instruction from the user via the photographing unit 106 and the microphone 107. That is, the photographing unit 106 and the microphone 107 function as an input unit. Specifically, the user inputs an instruction to the information processing apparatus 100 by a predetermined gesture or utterance. On the other hand, the information processing apparatus 100 accepts an instruction by gesture recognition based on an image photographed by the photographing unit 106 and accepts an instruction by voice recognition for the sound collected by the microphone 107. Note that the information processing apparatus 100 may include an input device such as a physical button in addition to the photographing unit 106 and the microphone 107.

  Next, a configuration and processing for the information processing apparatus 100 to display information superimposed on an object existing in the real space will be described. The imaging unit 106 is provided in a position and orientation such that the direction of the line of sight when the user who wears the information processing apparatus 100 sees the real space through the display device 105 substantially matches the imaging direction. It shall be. Thereby, the imaging | photography part 106 can image | photograph the image of the real space which the user is seeing. As another example, the photographing unit 106 may be installed such that the photographing direction and the wearer's line-of-sight direction have a certain relationship. The photographing unit 106 captures an image including the object, and the CPU 101 detects the reality of the object based on the position of the object in the captured image, the position of the information processing apparatus 100, and the orientation of the information processing apparatus 100 (the line of sight of the user). Specify a position in space. Then, the CPU 101 specifies a position on the display device 105 corresponding to the position of the object in the real space based on the position of the object in the real space, and the display information in which information is arranged at the specified position on the display device 105. Is generated and displayed on the display device 105.

  FIG. 2 is a flowchart showing display processing by the information processing apparatus 100. FIG. 3 is an explanatory diagram of a usage pattern of the information processing apparatus 100. As shown in FIG. 3, the doctor A wears the information processing apparatus 100 when the patient B is examined. Then, the patient B is seen through the display device 105 of the information processing apparatus 100. At this time, the patient information of the patient B is displayed on the display device 105. Therefore, the doctor A can check the patient information of the patient B at the same time as viewing the patient B through the display device 105. Hereinafter, processing of the information processing apparatus 100 at this time will be described.

  In S <b> 200, the CPU 101 of the information processing apparatus 100 specifies a medical treatment area handled by a doctor who is a user (medical treatment area specifying process). Here, the medical treatment area is information indicating medical departments that can be examined by a doctor. In addition, the medical treatment area | region should just be information which specifies the medical treatment content, and the classification method is not limited to embodiment. For example, the medical treatment area may be further subdivided such as a specific part of the internal medicine or a specific disease. Specifically, the CPU 101 receives an instruction to input a medical area from the user's utterance, and specifies the medical area according to the input instruction. As another example, the storage device 103 stores a medical treatment area in advance in association with user identification information, and the CPU 101 identifies a user based on biometric authentication or the like, and associates the identified user with the identified user. A medical treatment area may be specified.

  Next, in S201, the CPU 101 waits until the presence of the patient is confirmed and the patient is identified. When the CPU 101 confirms the presence of the patient and identifies the patient (YES in S201), the process proceeds to S202. Specifically, when the CPU 101 receives patient identification information (patient ID) via the input unit (acceptance process), the CPU 101 confirms the presence of the patient and determines that the patient has been identified. As another example, the CPU 101 may confirm the presence of the patient and identify the patient from the image feature amount or the like by the image recognition process on the captured image captured by the capturing unit 106. Here, the process of S201 is an example of a patient confirmation process.

  In S <b> 202, the CPU 101 preferentially extracts (extracts) patient information corresponding to the patient identified in S <b> 201 and patient information related to the medical treatment area specified in S <b> 200 from the patient information stored in the server device 110. Processing), receive. Next, in S203, the CPU 101 specifies a patient's part by image recognition processing on the captured image (part specifying process). In the present embodiment, the CPU 101 specifies three parts of the head, upper body, and lower body, but the part to be specified is not limited to this. As another example, the CPU 101 may specify a part in a finer unit such as an arm or a neck.

  Next, in S204, the CPU 101 specifies a part position in the real space based on the part position where the part is specified in the captured image (position specifying process). Note that the CPU 101 determines the position of the part on the photographed image based on the correspondence between the position on the photographed image and the position on the real space from information such as the photographing direction, the line-of-sight direction, and the position of the information processing apparatus 100. It is assumed that the position is converted into a part position in the real space. For example, the CPU 101 identifies the position of the part in the real space based on the relationship between the position of the photographing unit 106 that is an example of the photographing direction and the position of the display device 105 that is an example of the position of the information processing apparatus 100. More specifically, the CPU 101 specifies a part position on the display device 105 in the real space based on the relationship between the position of the imaging unit 106 and the position of the display device 105.

  Next, in S205, the CPU 101 specifies to which part each patient information extracted in S202 corresponds. The patient information includes information indicating a part, and the CPU 101 specifies which part the patient information is related to from the information indicating the part. The unit of the part included in the patient information may not match the unit of the part specified in S203. Also in this case, the CPU 101 specifies the upper body as a part according to a predetermined correspondence, for example, if it is patient information related to the lung.

  Then, the CPU 101 displays each patient information in association with the position of the display device 105 corresponding to the position of the part in the real space (display process). Note that the CPU 101 can use the positional relationship between the display device 105 and the imaging unit 106 to align patient information displayed on the display device 105 with the patient's part existing in the real space. Further, the CPU 101 displays patient information that is not associated with any part in a form that is not associated with any part. As described above, the CPU 101 confirms the presence of the patient and identifies the patient, and displays the patient information superimposed on the real space at a timing that matches the condition. When a moving image is shot by the shooting unit 106, steps S203 to 205 may be repeated.

  FIG. 4 is a diagram illustrating a display example of augmented reality visually recognized by the user via the display device 105. Balloons 400 a and 400 b are displayed on the display device 105 with reference to positions on the display device 105 corresponding to the position of each part of the patient B visually recognized by the user through the display device 105. Further, in each of the balloons 400a and 400b, patient information 401a and 401b corresponding to the part is shown. For example, the balloon 400a is displayed with the position on the display device 105 corresponding to the position of the head of the patient B as a base point, and the patient information 401a shown in the balloon 400a is patient information regarding the head. The balloon 400b is displayed with the position on the display device 105 corresponding to the position of the upper body part of the patient B as a base point. The patient information 401b shown in the balloon 400b is related to the upper body such as a lung radiograph. Patient information. The balloons 400a and 400b are an example of related information that associates patient information with a part position.

  In the frame 402, patient information 403 that does not correspond to any part is displayed. As described above, the CPU 101 displays patient information that does not correspond to any part in the frame 402 having a different display mode from the balloons 400a and 400b. That is, the CPU 101 displays the patient information other than the patient information related to the part specified in the part specifying process (S203) in a format not associated with the part position. Thereby, the user can grasp that the content displayed in the frame 402 is information not related to the part.

  The CPU 101 arranges and displays any patient information at a position that does not overlap the patient B existing in the real space. That is, the CPU 101 displays patient information so as not to overlap the position of the display device 105 corresponding to the patient B in the real space. Further, the CPU 101 arranges patient information related to a part at a position as close as possible to the position of the display device 105 corresponding to the corresponding part position. That is, the CPU 101 displays the corresponding patient information at a position determined based on the part position.

  As described above, the information processing apparatus 100 according to the present embodiment displays the patient information related to the part in association with the part position of the patient. Therefore, the user can easily grasp the relationship between the patient information and the part of the patient to be treated that exists in the real space. Further, the information processing apparatus 100 can appropriately display information necessary for the user by narrowing down the patient information to be displayed based on the medical treatment area of the doctor as the user. For example, patient information regarding dentistry is unnecessary for a doctor who is in charge of orthopedic surgery. On the other hand, the information processing apparatus 100 can preferentially display orthopedic information except for dental information by narrowing down as described above.

  As a first modification of the first embodiment, the information processing apparatus 100 is an apparatus that provides augmented reality, and may include a non-transmissive display unit that covers the entire user's field of view. Good. In this case, the information processing apparatus 100 displays the captured image captured by the imaging unit 106 on the display device 105. Then, the information processing apparatus 100 displays corresponding patient information in association with the patient's part on the captured image. In this case, the information processing apparatus 100 may identify a part position in the photographed image and display the patient information superimposed on the photographed image in association with the part position.

  As a second modification, the information processing apparatus 100 is not limited to an apparatus that provides augmented reality. For example, as illustrated in FIG. 5, the information processing apparatus 100 may include a display device 105 having a size similar to that of a patient. In this case, it is assumed that the display device 105 is installed at a predetermined position such as a position adjacent to a position where the patient sits at the time of medical examination, a position behind the sitting position, or the like. In addition, the imaging unit 106 is installed at a position where an area including the position where the patient sits can be imaged. Furthermore, it is assumed that the positional relationship between the imaging unit 106 and the display device 105 is known. And according to this positional relationship, CPU101 displays patient information on the display apparatus 105 so that it may relate with the site | part position in the real space specified from the patient's picked-up image image | photographed by the imaging | photography part 106. FIG.

  As a third modified example, the information processing apparatus 100 may further include a detection device that detects a movement at the time when the doctor moves his / her face when he / she examines the patient and gazes at the affected part. In this case, when detecting an upward movement, the CPU 101 may narrow down the display target part to the head, and if the downward movement is detected, the display target part may be narrowed down to the lower body. And CPU101 is good also as displaying the patient information about the site | part narrowed down.

(Second Embodiment)
In the medical system according to the second embodiment, the information processing apparatus 100 not only confirms the presence of a patient but also displays patient information when confirming the presence of a specific device with the patient. Hereinafter, differences of the medical system according to the second embodiment from the medical system according to the first embodiment will be described.

  FIG. 6 is a flowchart showing display processing by the information processing apparatus 100 according to the second embodiment. Of the processes shown in FIG. 6, the same processes as the processes of the display process according to the first embodiment described with reference to FIG. In the second embodiment, the CPU 101 advances the process to S600 after the process of S200. In S <b> 600, the CPU 101 confirms the presence of the patient based on the captured image, identifies the patient, and waits until a predetermined specific device is recognized together with the patient (device confirmation process). Here, as the specific equipment, an examination table such as a chair or a bed used for the examination can be cited.

  If the CPU 101 recognizes the presence of the equipment together with the patient (YES in S600), the process proceeds to S202. As described above, the information processing apparatus 100 according to this embodiment displays the patient information at a timing that matches the condition, with the patient being confirmed and the equipment being confirmed as a condition for displaying the patient information. Can do. That is, the information processing apparatus 100 can display patient information at a more appropriate timing for the user.

  FIG. 7 is a diagram illustrating the relationship between the device and the patient. At the time of medical examination, the patient often sits on a chair 700 as shown in FIG. 7A or lies on a bed 701 as shown in FIG. 7B. Therefore, by limiting the display timing of the patient information of the information processing apparatus 100 to a case where the patient is confirmed at the same time as such a device, it is possible to prevent the patient information from being displayed at an unnecessary timing, and at the time of the examination Information can be displayed. The remaining configuration and processing of the medical system according to the second embodiment are the same as the configuration and processing of the medical system according to the first embodiment.

  As a first modification of the second embodiment, the server device 110 stores patient information and related device information in association with each other, for example, associating a device “bed” with patient information related to an electrocardiogram. May be. In this case, the information processing apparatus 100 may narrow down the patient information extracted in S202 based on the equipment confirmed in S600. Thereby, patient information corresponding to the equipment can be preferentially displayed.

  As a second modification, the information processing apparatus 100 determines the positional relationship between the part and the equipment such that the head is recognized above the position of the chair, for example, when recognizing the patient's part. May be used. Thereby, a part can be specified with higher accuracy.

(Third embodiment)
In the medical system according to the third embodiment, the information processing apparatus 100 displays patient information when confirming a patient and recognizing a predetermined keyword. Hereinafter, differences of the medical system according to the third embodiment from the medical system according to the first embodiment will be described. FIG. 8 is a flowchart showing display processing by the information processing apparatus 100 according to the third embodiment. Of the processes shown in FIG. 8, the same processes as the processes of the display process according to the first embodiment described with reference to FIG. In the third embodiment, when the CPU 101 confirms the presence of the patient and identifies the patient in S201 (YES in S201), the process proceeds to S800.

  In S800, the CPU 101 stands by until a keyword related to the medical treatment area is recognized from the patient's voice (keyword recognition process). Note that the keyword is set in advance in the storage device 103 for each medical treatment area. For example, “lung”, “stomach”, and the like are set as keywords for the medical treatment area “internal medicine”. As another example, a word included in patient information may be used as a keyword. If the CPU 101 recognizes the keyword (YES in S800), the process proceeds to S202. If the keyword has not been recognized (NO in S800), the CPU 101 advances the process to S201. As described above, the information processing apparatus 100 according to the present embodiment can display the patient information at a timing that matches the condition, with the condition that the patient is confirmed and the keyword is recognized as a condition for displaying the patient information. it can. That is, the information processing apparatus 100 can display patient information at a more appropriate timing for the user. Note that other configurations and processes of the medical system according to the third embodiment are the same as the configurations and processes of the medical system according to the other embodiments.

  As a modification of the third embodiment, the CPU 101 may narrow down patient information to be extracted based on the keyword recognized in S700 in S202. For example, it is assumed that the CPU 101 recognizes the keyword “lung” related to “internal medicine” from the patient's utterance “lung is... In this case, patient information related to the lung is extracted from the patient information and displayed preferentially. Thereby, unnecessary information can be further excluded.

  As described above, according to each embodiment described above, information necessary for a user such as a doctor can be appropriately displayed.

<Other embodiments>
Although the embodiment has been described in detail above, the present invention can take an embodiment as a system, apparatus, method, program, recording medium (storage medium), or the like. Specifically, the present invention may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a Web application, etc.), or may be applied to a device composed of a single device. good.

  Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) that records a program code (computer program) of software that implements the functions of the above-described embodiments is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

100 Information processing apparatus 101 CPU
105 display device 110 server device

Claims (19)

A part specifying means for specifying the part of the patient based on a photographed image of the patient taken by the photographing means;
Position specifying means for specifying a part position in the real space of the part;
Patient information related to the part specified by the part specifying means is associated with the position of the display means corresponding to the part position in the real space based on the positional relationship between the display means and the imaging means, and is displayed on the display means. An information processing apparatus comprising display processing means for displaying. The display means is a transmissive type,
2. The display processing unit according to claim 1, wherein the display processing unit displays the patient information on the display unit in association with the position of the part of the real space that is visible by the user through the display unit. Information processing device.   The display processing means displays the patient information in association with the part position by displaying the patient information and related information for associating the patient information with the part position. The information processing apparatus according to 1 or 2.   The information according to claim 1, wherein the display processing unit displays the patient information in association with the part position by displaying the patient information at a position determined with the part position as a reference. Processing equipment.   The information processing apparatus according to claim 1, wherein the display processing unit displays the patient information at a position that does not overlap with the position of the part.   The said display process means displays patient information other than the patient information regarding the site | part specified by the said site | part specification means in the format which is not linked | related with the said site | part position. Information processing device. A medical area specifying means for specifying a medical area of a doctor;
Extracting means for extracting patient information related to the medical treatment area from patient information of the patient;
The information processing apparatus according to claim 1, wherein the display processing unit displays the patient information extracted by the extraction unit. Patient confirmation means for confirming the presence of the patient and identifying the patient;
The information processing apparatus according to claim 1, wherein the display processing unit displays the patient information on condition that the patient is confirmed.   The information processing apparatus according to claim 8, wherein the patient confirmation unit confirms the presence of the patient by image recognition in the captured image and identifies the patient. Further comprising a receiving means for receiving patient identification information from the user;
The information processing apparatus according to claim 8, wherein the patient confirmation unit confirms the presence of the patient and identifies the patient based on the identification information. It further has equipment confirmation means for confirming the presence of predetermined equipment together with the patient,
The information processing apparatus according to any one of claims 8 to 10, wherein the display processing unit displays the patient information on the condition that the patient is confirmed and the equipment is confirmed. . It further has a keyword recognition means for recognizing a predetermined keyword,
The information processing apparatus according to claim 1, wherein the display processing unit displays the patient information on condition that the keyword is recognized. Based on the patient's image captured by the imaging means, a part specifying means for specifying the patient's part;
Position specifying means for specifying a part position in a captured image of the part;
And display processing means for displaying on the display means the patient information related to the part specified by the part specifying means in association with the position of the part on the photographed image and superimposed on the photographed image. Information processing device. A part specifying means for specifying the part of the patient based on a photographed image of the patient taken by the photographing means;
Position specifying means for specifying a part position in the real space of the part;
Patient information related to the part specified by the part specifying means is associated with the position of the display means corresponding to the part position in the real space based on the positional relationship between the display means and the imaging means, and is displayed on the display means. An information processing system comprising display processing means for displaying. Based on the patient's image captured by the imaging means, a part specifying means for specifying the patient's part;
Position specifying means for specifying a part position in a captured image of the part;
And display processing means for displaying on the display means the patient information related to the part specified by the part specifying means in association with the position of the part on the photographed image and superimposed on the photographed image. Information processing system. An information processing method executed by an information processing apparatus,
A part specifying step for specifying a part of the patient based on a photographed image of the patient photographed by the photographing means;
A position specifying step of specifying a position of the part in the real space;
Patient information related to the part specified in the part specifying step is associated with the position of the display unit corresponding to the part position in the real space based on the positional relationship between the display unit and the imaging unit, and is displayed on the display unit. A display processing step of displaying the information processing method. An information processing method executed by an information processing apparatus,
A part specifying step for specifying a part of the patient based on a photographed image of the patient photographed by the photographing means;
A position specifying step for specifying the position of the part in the captured image of the part;
A display processing step of displaying on the display means the patient information related to the part specified in the part specifying step in association with the part position on the captured image and superimposed on the captured image. Information processing method. Computer
A part specifying means for specifying the part of the patient based on a photographed image of the patient taken by the photographing means;
Position specifying means for specifying a part position in the real space of the part;
Patient information related to the part specified by the part specifying means is associated with the position of the display means corresponding to the part position in the real space based on the positional relationship between the display means and the imaging means, and is displayed on the display means. A program for functioning as display processing means for displaying. Computer
Based on the patient's image captured by the imaging means, a part specifying means for specifying the patient's part;
Position specifying means for specifying a part position in a captured image of the part;
A program for functioning as display processing means for displaying patient information on a part specified by the part specifying means in association with the position of the part on the photographed image and superimposed on the photographed image.