WO2025141689A1 - Image classification device for endoscope, image classification method, image classification program, and search method - Google Patents

Abstract

This image classification device for an endoscope comprises: a site determination unit that determines anatomical sites in a human body in accordance with captured images sequentially obtained in a process of inserting or extracting the endoscope into or from the body; a feature determination unit that determines, regarding at least one determination item, site distinctive features which are features of the determined sites, and obtains site distinctive feature information; and an association unit that records, in a manner associated with the captured images, the site distinctive feature information for the respective sites.

Description

Endoscopic image classification device, image classification method, image classification program, and search method

The present invention relates to an endoscopic image classification device, an image classification method, an image classification program, and a search method that classify and organize images obtained during the process from the insertion to the removal of an endoscope for use in examinations, diagnoses, etc.

An endoscope is a device that is inserted inside the body and allows observation of diseased areas that cannot be seen from the outside. An endoscope has an insertion part that is inserted into a body cavity, and an imaging device is provided, for example, at the tip of the insertion part. In an examination using an endoscope, a doctor sequentially displays images acquired by imaging the tip of the insertion part, and adjusts the position of the tip of the insertion part while checking the displayed images, thereby diagnosing the state of health or disease.

The imaging device continues to capture images throughout the entire process from insertion to removal of the insertion part, and in endoscopic examinations, in addition to images used for diagnosis, a huge amount of information is also obtained, including images of each part (organ) of the human body that passes through.

A technology for managing acquired image information is proposed in Japanese Patent Application Laid-Open Publication No. 2002-253539 (hereinafter referred to as Patent Document 1). This proposal discloses a technology for identifying medical images and adding the imaging attribute information of the most matching category as management information.

JP 2002-253539 A

The proposal of Patent Document 1 is a technology that automatically adds imaging attribute information, such as imaging device, imaging part, and imaging direction, to a medical image as management information. The device of Patent Document 1 makes it easy to distinguish which part was imaged from which direction by which device, such as an X-ray device or an MRI device, for example, whether the head or chest was imaged. The technology of Patent Document 1 makes it possible to determine which part an image is from, making it easy to search for past cases of a specific subject.
However, in the proposal of Patent Document 1, although the acquired management information is effective for diagnosing the subject himself, a complicated search operation is required to use the information as reference information for diagnosing others.
The present invention aims to provide an endoscopic image classification device, image classification method, image classification program, and search method that can accumulate medical information obtained during the process from insertion to removal of an endoscope and classify each part based on images of the parts according to their characteristics, thereby obtaining information that is extremely useful for diagnosis and treatment.

An endoscopic image classification device according to one aspect of the present invention includes a part determination unit that determines anatomical parts of the human body according to captured images obtained sequentially during the process of inserting an endoscope into or removing it from the body, a feature determination unit that determines part-specific features that are characteristics of the determined part for one or more determination items to obtain part-specific feature information, and an association unit that records the part-specific feature information for each part in association with the captured image.

An image classification method according to one aspect of the present invention determines anatomical parts of the human body according to captured images etc. obtained sequentially during the process of inserting an endoscope into or removing it from the body, determines part-specific features that are characteristics of the determined parts for one or more determination items to obtain part-specific feature information, and records the part-specific feature information for each part in association with the captured image.

An image classification program according to one aspect of the present invention causes a computer to execute the following steps: determine anatomical parts of the human body according to captured images, etc., obtained sequentially during the process of inserting an endoscope into or removing it from the body; determine part-specific features, which are characteristics of the determined parts, for one or more determination items to obtain part-specific feature information; and record the part-specific feature information for each part in association with the captured image.

An image classification method according to another aspect of the present invention determines a number of anatomical parts of the human body according to captured images obtained sequentially during the process of inserting an endoscope into or removing it from the body, and in order to obtain regional features that are characteristic of each of the multiple parts, a database is referenced that organizes determination items that indicate the features to be obtained for each of the multiple parts, and the regional features are determined for each of the determined parts, and regional feature information is obtained based on the regional features so that the determined regional features can be recorded in association with the captured images.

The search method according to one aspect of the present invention searches for an endoscopic image acquired in a second case using part-specific characteristic information that indicates the characteristics of each anatomical part through which an endoscope passes during insertion into or removal from the body in a first case.

The present invention has the effect of accumulating medical information acquired during the process from the insertion to the removal of the endoscope, and classifying the information according to the characteristics of each part based on the images of each part, thereby obtaining information that is extremely useful for diagnosis and treatment.

1 is a block diagram showing an endoscopic image classification device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing an example of an input image. FIG. 11 is an explanatory diagram for explaining an inference model that enables part determination. FIG. 13 is an explanatory diagram showing an example in which the correspondence between each part and each determination item is summarized in a table format. FIG. 5 is an explanatory diagram showing an example of the shape of the large intestine. 4 is a flowchart for explaining the operation of the embodiment. 4 is a flowchart for explaining the operation of the embodiment. FIG. 1 is an explanatory diagram for explaining an example of an endoscopic examination. FIG. 9 is an explanatory diagram for explaining an image acquired in the example of FIG. 8 . 4 is a flowchart for explaining the operation of the embodiment.

Below, an embodiment of the present invention will be described in detail with reference to the drawings.

(Embodiment)
FIG. 1 is a block diagram showing an endoscopic image classification device according to an embodiment of the present invention. Video obtained by endoscopic examination includes images of various parts from outside the body to various organs such as the throat, esophagus, stomach, and duodenum, and is extremely rich information, but is not fully utilized. In this embodiment, each part is classified by a plurality of features based on the image of each part, the classification results are accumulated, and the accumulated classification results are made available as information useful for diagnosis. For example, by classifying the digestive system by features and accumulating the classification results, it is possible to determine the characteristics of the digestive system of a person who is prone to digestive diseases, which can be useful for diagnosis and prevention.

A part is an anatomical part that can identify a specific anatomical feature. The unit of a part may be an organ with a specific function. Large organs such as the large intestine and stomach have further medically subdivided names (for the stomach, cardia, fundus, body, antrum, pylorus, etc.; for the large intestine, ascending colon, transverse colon, descending colon, sigmoid colon, rectum, etc.). Therefore, a part of the subdivided parts may be used as a part, or an organ may be subdivided in a way that is different from the medical method of subdivision.

In FIG. 1, medical images such as endoscopic images acquired by an endoscope (not shown) are input to the endoscopic image classification device 10. The endoscope has an insertion section that is inserted into a body cavity, and an imaging device is provided at the tip of the insertion section. The imaging device includes an imaging element such as a CCD or CMOS sensor, and photoelectrically converts an optical image from a subject to obtain an imaging signal. This imaging signal may be supplied to a video processor (not shown) for signal processing, and then supplied to the endoscopic image classification device 10. In addition, a channel (a connection path connecting the inside of the body cavity and the outside of the body) for passing instruments such as treatment tools is provided inside the tube of the endoscope, and ultrasonic sensors can be used through it, and various sensors and transmitters can also be provided at the tip of the endoscope, and it is possible to obtain information other than images as the endoscope is inserted into the body in cooperation with such sensors or other devices. The linked device may be an in-hospital system, and in some cases, the doctor's voice, comments, and terminal operations during the endoscopic examination may be available. Since the video information of the endoscope is time-series information, it is sufficient to make it available in accordance with the timing of the video. These may be recorded in association with the image as "accompanying information" in FIG.
The endoscopic classification device 10 is described here as a special device that classifies images obtained from many endoscopes, but it may also be directly connected to an endoscope to classify the results of an endoscopic examination in real time, or it may work in conjunction with a display control unit 16 to provide a guide that applies its functions during an endoscopic examination.

The medical images input to the endoscopic image classification device 10 are images of the inside of the body acquired by endoscopic examination or the like. These input images, for example endoscopic images, may be supplied directly from a video processor, or may be images that have been recorded in a recording device and then read out and supplied. In FIG. 1, an example of a video is shown as the image (input image) input to the endoscopic image classification device 10, but in addition to a series of continuously acquired images, still images may also be acquired and input as necessary in response to the operation of a medical professional or under specific conditions.

Figure 2 is an explanatory diagram showing an example of an input image. In Figure 2, frames f1 to f6 of a series of images Pe that are acquired successively are shown in square frames. Frame f4 contains an image portion of the diseased area, shown by an oval. In normal endoscopic examinations, the image of frame f4 is used for diagnosis, etc., and the other frames f1 to f3, f5, and f6 are generally not used for purposes other than inserting the endoscope.

Figure 1 shows an example in which many images including video images of three persons A to C are input. For persons A to C, the images acquired in 2020 are input images P2020A to P2020C, respectively, and the images acquired in 2021 are input images P2021A to P2021C, respectively.

In addition to video, accompanying information may be added to the input image. The accompanying information includes operation information, sensor information, endoscopic device setting information, information on the behavior of the doctor performing the endoscopic examination and the caregiver during the endoscopic examination, information on the treatment tools used, and other diagnostic results, pathology results, interview results, etc.

It may also include information about the hospital and surgery room, patient information such as the date and time, age, race, and gender, information about the doctor in charge and medical staff who participated in the examination, etc. It may also include information about other equipment used during the examination. For example, recently it has become possible to record conversations between doctors and medical staff using microphones, etc., so it may be possible to record such information so that it can be associated with the image frames.

Similar to this, it may also include the various sensor information mentioned above. For example, various switches are provided on the operation section of the endoscope (not shown), and operation information such as the operation of these switches may be added to the image data of the endoscopic image as accompanying information and input. Also, operation information based on the operation of the air and water supply device may be added as accompanying information. If information indicating which timing of the video frame these are from is included, this can also be used for the endoscopic image classification of the present application.

In addition, since special light observation has different characteristics from normal images, it may be better to treat special light observation images and normal light observation images separately for various judgments. Therefore, information for distinguishing whether each frame of the captured image is a special light observation image or a normal light observation image may be added to the image information as accompanying information. Also, information acquired by the endoscope insertion shape observation device may be added to the image information as accompanying information.

The endoscopic image classification device 10 includes a body part determination unit 12, a body part feature determination unit 13, an active operation determination unit 14, an association unit 15, a display control unit 16, a search unit 17, and a recording control unit 18. Each component of the endoscopic image classification device 10 may be configured by a processor using a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit), etc., and may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of the functions by a hardware electronic circuit.

In this embodiment, the part determination unit 12 of the endoscopic image classification device 10 sequentially determines which part of the body is imaged by the input image. The part determination unit 12 determines each part of the human body from endoscopic images sequentially acquired by the endoscope during the endoscope insertion and removal process, i.e., from the start of the endoscope's insertion to the completion of its removal. The part determination unit 12 can not only directly identify the anatomical features of each part from the image of each part, but can also determine the part when the anatomical features of each part can be inferred from information on other parts whose anatomical features have been identified.

For example, the part determination unit 12 determines the part by image analysis and inference processing of the endoscopic image. For example, the part determination unit 12 can identify specific parts such as the nasal cavity or stomach by determining the shape of the lumen from image features. In addition, for example, the part determination unit 12 can determine specific parts such as the large intestine by determining changes in image features (image history) that show that the insertion part progresses while bending, and can also determine whether it is the descending colon or the transverse colon.

The part determination unit 12 may also determine the characteristic color or shape of the part, or the pattern of blood vessels or the like visible on the surface by referring to a database (not shown). Another method is to use an external receiver to receive the transmission results of a magnetic or other transmitter attached to the tip or insertion part of the endoscope, and determine where on the body the tip of the endoscope is located, thereby determining which part the image was taken of, or to determine the part from the insertion depth and twist of the endoscope. Representative images of each part (images with adjusted image quality, such as those found in papers) and data showing the image characteristics of each part may be recorded on a recording medium (not shown), and the part determination unit 12 may determine each part by referring to the information recorded on the recording medium.

The part determination unit 12 may also determine the part by AI (artificial intelligence) processing.

FIG. 3 is an explanatory diagram for explaining an inference model that enables part determination.
When determining the part by AI processing, the part determination unit 12 employs a network corresponding to each part. In the example of FIG. 3, two networks N1 and N2 for inferring the first and second parts are shown. A large amount of images corresponding to the input and output are provided as teacher data to the networks N1 and N2, respectively. In the example of FIG. 3, a plurality of images (rectangular frames) in the first part are input as teacher data to the network N1, and a plurality of images (rectangular frames) in the second part are input as teacher data to the network N2. For example, the input images are annotated with the part names corresponding to each image.

By learning from large amounts of training data, the network design of networks N1 and N2 is determined so that an output corresponding to each input is obtained. That is, in the example of Figure 3, when an image of a first part is input, network N1 outputs information on the part name of the first part, and when an image of a second part is input, network N2 outputs information on the part name of the second part. That is, network N1 constructs an inference model that detects the part name of the first part from the input image, and network N2 constructs an inference model that detects the part name of the second part from the input image.

The correct answer determination unit 12a determines whether the inference results from networks N1 and N2 are correct, and if they are incorrect, performs learning control to reconstruct the network design of networks N1 and N2.

"Deep learning" is a multi-layered structure of the "machine learning" process using neural networks. A typical example is the "forward propagation neural network" which sends information from front to back and makes a judgment. In its simplest form, it has three layers: an input layer consisting of m1 neurons, a middle layer consisting of m2 neurons given by parameters, and an output layer consisting of m3 neurons corresponding to the number of classes to be discriminated. The neurons in the input layer and middle layer, and the middle layer and output layer are each connected by a connection weight, and a bias value is added to the middle layer and output layer, making it easy to form logic gates. Three layers are sufficient for simple discrimination, but if there are many middle layers, it is also possible to learn how to combine multiple features in the machine learning process. In recent years, 9 to 152 layers have become practical in terms of the time it takes to learn, the accuracy of judgment, and energy consumption.

A variety of well-known networks may be used as the network N1 used for machine learning. For example, R-CNN (Regions with CNN features) or FCN (Fully Convolutional Networks) using CNN (Convolution Neural Network) may be used. This involves a process called "convolution" that compresses the features of an image, operates with minimal processing, and is strong in pattern recognition. In addition, a "recurrent neural network" (fully connected recurrent neural network) that can handle more complex information and allows information analysis whose meaning changes depending on the order or sequence, and in which information flows in both directions, may be used.

To realize these technologies, conventional general-purpose calculation processing circuits such as CPUs and FPGAs can be used, but because much of the processing in neural networks involves matrix multiplication, GPUs and Tensor Processing Units (TPUs), which are specialized for matrix calculations, may also be used. In recent years, such artificial intelligence (AI) dedicated hardware "neural network processing units (NPUs)" have been designed to be integrated and embedded with CPUs and other circuits, and may form part of the processing circuit.

Furthermore, in addition to deep learning, various well-known machine learning methods may be employed to obtain an inference model. For example, there are methods such as support vector machines and support vector regression. The learning here involves calculating the weights, filter coefficients, and offsets of the classifier, and other methods use logistic regression processing. When making a machine make a judgment, a human must teach the machine how to make the judgment. In this embodiment, a method is used to derive a judgment on an image using machine learning, but it is also possible to apply a rule-based method that applies rules acquired by humans through experience and heuristics to make a specific judgment.

In this embodiment, the characteristics of each part determined by the part determination unit 12 (hereinafter referred to as part-specific characteristics) are determined, and classification is performed according to the determination results. The items used to determine the part-specific characteristics include the presence or absence of an affected area, passing time, number of frames acquired, number of times the lumen direction was lost, number of times the insertion and removal direction was redone, color characteristics, deformation during air supply, degree of residue, length, thickness, color, wrinkles, movement, elasticity, twisting, bending, etc.

Figure 4 is an explanatory diagram showing examples of the correspondence between each body part and each assessment item, classified in a table format. Examples of body parts shown are the anus, rectum, sigmoid colon, and descending colon.

The "presence or absence of an affected area" judgment item is based on size, location, number, color, etc. "Passing time," "number of frames acquired," "number of times the lumen was lost," and "number of times the insertion/removal direction was redone" indicate the state of the endoscope insertion part when it was inserted or removed, and respectively indicate the time required to pass the area, the number of frames acquired at the area, the number of times the lumen (deep area) was lost when imaging the area, and the number of times the insertion part was inserted and removed when imaging the area. "Color characteristics" indicate the characteristics of the color and color change of the area, and "deformation during air supply" indicates the characteristics of the state in which the shape of the area changes due to air supply to the area. From the state of deformation due to swelling of the lumen tissue when air is supplied, characteristics such as whether the tissue is elastic, hard, or soft can also be determined. The same can be said for water supply, and characteristics such as the effect of water supply can be recorded. In other words, the endoscopic image classification device of the present application is characterized in that the judgment item database (DB) 40 for referring to judgment items indicating the characteristics to be judged for the above-mentioned parts includes features (judgment items) for judging the part-specific characteristics of the image changes according to active treatment. Active treatment refers to active action on the object that differs from normal observation (water supply, air supply, irradiation with special light (and image judgment at that time), dyeing (and image judgment at that time), action by treatment tool, etc.) in addition to normal screening and diagnostic observation.

"Level of residue" indicates the amount of fecal matter remaining. Of course, if the color, size, location, and other characteristics of the residue are specifically noted and recorded, it may be possible to use this information to identify the characteristics of the internal structure of the digestive tract, digestive ability, and identify illnesses.

In addition to air supply, it is also possible to determine whether cleaning with water supply is effective. In other words, these can be said to classify the reactions of measures that actively act on the lumen (active manipulation) as characteristics. Since such active manipulation may or may not be performed for each case, and the areas where it is applicable vary, it may not be possible to compare the sensations of cases. Therefore, in the judgment item DB40, test items such as supplying air to this area to check the softness of the lumen can be entered, and a guide can be displayed during the examination to encourage the examiner to perform the operation for each case, making it a standardized test item.

When an endoscopic examination is performed by connecting an endoscope directly to the endoscopic image classification device 10, the judgment item DB 40 may record information on the active operation recommended for each of the above anatomical parts, and display a guide to the endoscope operator according to that information in accordance with the display device that displays the endoscopic image. The display control unit 16 is provided with such a composite display function.

The judgment items "length," "thickness," "color," "wrinkles," "movement," "elasticity," "twisting," and "flexibility" indicate the characteristics of the part's length, thickness, color, wrinkles, movement, elasticity, twisting, and flexing. One method for judging length, etc. is to judge the insertion and removal time or fine structures such as blood vessels as patterns, and determine how long this continues from the images. It is also possible to use a method in which the lumen length is inferred from endoscopic images using an inference model that has been trained using images of the endoscope moving inside the lumen and the lumen length obtained from X-rays, etc., as training data. Other characteristics such as the number, size, and spacing of the folds in the intestine may also be recorded.

The part feature judgment unit 13 judges the part feature, which is a feature of the part judged by the part judgment unit 12, for one or more judgment items. That is, the part feature judgment unit 13 as a feature judgment unit judges the part feature for various judgment items for each part to obtain part feature information. The part feature judgment unit 13 may obtain part feature information that classifies and shows the judgment results of the part feature in text format or table format. For example, the part feature judgment unit 13 may judge each judgment item by image analysis or AI processing of the input image. The part feature judgment unit 13 may judge the part feature based on the image change of each frame of the image obtained sequentially. As shown in FIG. 4, there are cases where judgment items common to all parts are set, but the judgment items may be different for each part. In addition, table-format information may be held for each part with common judgment items. The judgment item DB 40 is a database that stores information on judgment items for each part. The part-specific characteristic determination unit 13 may read out the determination items stored in the determination item DB 40 for each part, and make a determination for the read-out determination items. In other words, the endoscopic image classification device of the present application has a database for referencing features (determination items) for determining part-specific characteristics for each of multiple parts through which the endoscope passes.

The active operation determination unit 14 obtains active operation information by determining signals based on active operation by the surgeon when inserting the endoscope into the body. Possible active operations include, for example, taking still images by operating the operation unit of the endoscope, and operating devices for supplying air or water into the body. The active operation determination unit 14 obtains active operation information as a determination result of active operation by receiving operation signals based on operations on these devices. Note that the active operation determination unit 14 may obtain a determination result of active operation by image analysis of the input image. It may also be possible to determine from the change in the image over time that water has been supplied (e.g., an image of water splashing is obtained and then subsides) or air has been supplied (e.g., tissue swells and returns to normal).

The association unit 15 records the part-specific features for each part in association with the input image. That is, the association unit 15 associates each of the judgment results of the part determination unit 12, the part-specific feature determination unit 13, and the active operation determination unit 14 with the input image. The association unit 15 controls the recording control unit 18 to record the associated results in the recording device 30. For example, the association unit 15 may record each of these judgment results in association as metadata of a still image or video file. As a result, part information indicating which part was captured, part-specific feature information for that part in, for example, a table format or text format, and active operation information performed at the timing of shooting that frame are associated and recorded in a still image file or in image information of each frame of a video file. Note that the associated information associated with the input image is also associated and recorded as metadata in the image file as it is. In addition, when the part determination unit 12 determines the part by AI processing, information on the inference model used for the judgment may also be associated and recorded in association with the image.

The recording control unit 18 controls the recording of data in accordance with the association unit 15. The recording device 30 has a recording medium (not shown), and records the associated image files under the control of the recording control unit 18. In this way, images of many people who have undergone endoscopic examinations are recorded with associated body part information, body part characteristic information, and active operation information for each frame, in addition to the accompanying information added at the imaging stage.

The display control unit 16 controls the display of the display device 20. The display device 20 is a display device such as an LCD (liquid crystal display), and performs various displays under the control of the display control unit 16. The display control unit 16 can display the search results of the search unit 17, which will be described later, on the display screen.

The search unit 17 searches for images based on user operations on an input device (not shown). For example, the search unit 17 may search for images that have common body part information and common body part characteristic information. That is, the search unit 17 can perform a search process based on the body part characteristic information for captured images of the body part determined by the body part determination unit 12 to extract captured images of similar cases. The search unit 17 may also search for images of the same body part of the same person captured at different times. The search unit 17 provides the search results to the display control unit 16, which causes the search display to be displayed on the display screen of the display device 20.

FIG. 1 shows an example of a display on the display screen of the display device 20. In the example of FIG. 1, an image Pe1 of a certain part currently (2023) acquired for subject α is displayed. Within image Pe1, an image Pe1a of a lesion such as a polyp is displayed. The search unit 17 searches for an image having part information and part characteristic information that match the part information and part characteristic information associated with image Pe1. For example, it is assumed that the part of image Pe1 is determined to be the large intestine by the part determination unit 12.

FIG. 5 is an explanatory diagram showing examples of the shape of the large intestine. Example E1 shows an example of a large intestine with little twisting or bending, Example 2 shows an example of a large intestine with a lot of twisting, and Example 3 shows an example of a large intestine with a lot of bending. The part-specific characteristic information also includes information on the shape characteristics of such parts. Even if the same part is present, symptoms may differ between organs with different sizes and shapes. Therefore, in this embodiment, images with the same part-specific characteristics are searched for, even if they are the same part.

The search unit 17 searches for images having the part information and part-specific characteristic information corresponding to the image Pe1. That is, images of the same part, the large intestine, that have similar shapes, sizes, elasticity, etc., and similar affected areas, i.e., images with the same part-specific characteristics, are obtained as search results. Note that, depending on how the part-specific characteristics are classified, the search unit 17 may be configured to obtain, as search results, not only images with the same part-specific characteristics, but also images with similar part-specific characteristics.

In the example of FIG. 1, image Pe2 is searched for and displayed as an image having part information and part-specific characteristic information corresponding to image Pe1. Image Pe2 includes image Pe2a of a lesion similar to image Pe1a of image Pe1. Note that image Pe2 was captured in 2020. Furthermore, search unit 17 searches for progress information on the treatment content and condition of the affected area for the person of whom image Pe2 was captured, for example, whether there is an image obtained by capturing the same area in 2021 after image Pe2 was captured, and displays the search results. Image Pe3 shows an image from this search, and image Pe3 includes image Pe3a showing that the lesion shown in image Pe2a has grown in size.

In other words, the search unit 17 can search for endoscopic images acquired in the second case using the part-specific characteristic information obtained for each anatomical part through which the endoscope passes during insertion into or removal from the body in the first case.

According to the display in Figure 1, for example, a doctor can determine that there is a possibility that the lesion in subject α will also expand, since the lesion has subsequently expanded in a person with a lesion similar to that in the large intestine of subject α and the same site-specific characteristics. In addition, it is also possible to identify the patient using the site-specific characteristic information. Therefore, for example, even if patient information such as the patient's name is not available, the use of the site-specific characteristic information makes it possible to identify the patient and record the case, etc.

(operation)
Next, the operation of the embodiment configured as above will be described with reference to Fig. 6 to Fig. 10. Fig. 6, Fig. 7 and Fig. 10 are flow charts for explaining the operation of the embodiment. Fig. 8 is an explanatory diagram for explaining an example of an endoscopic examination, and Fig. 9 is an explanatory diagram for explaining an image acquired in the example of Fig. 8.

Figure 8 shows the oral cavity, larynx, and pharyngeal parts of human body B. Figure 8 shows an upper gastrointestinal endoscopy or bronchial examination using oral endoscopy in which an insertion portion 51 is inserted through the mouth. As shown in Figure 8, in an examination or treatment using an endoscope 50, a doctor operates the endoscope 50 to insert the insertion portion 51 of the endoscope 50 into human body B. A bending portion (not shown) is provided at the tip of the insertion portion 51, and the doctor operates a bending knob 50a or the like provided on the endoscope 50 to bend the bending portion or to move the insertion portion 51 forward or backward, thereby inserting the tip of the insertion portion 51 to the area to be observed.

FIG. 9 shows an example of an image acquired during the endoscopic examination of FIG. 8. The square frames in FIG. 8 indicate the frames that are acquired, and when the endoscope is inserted, many images of the tongue, trachea, esophagus, etc. are acquired in addition to the area to be observed. In this way, the insertion portion 51 passes through various areas before reaching the esophagus, and images of the various areas are acquired.

When the endoscope is inserted, images taken up to the specific area to be observed may not be used for diagnosis, etc. However, when the endoscope passes through various areas on the way to the area to be observed, images of areas other than the area to be observed are also acquired when the endoscope is inserted or removed. That is, in addition to the endoscopic images used for diagnosis, many endoscopic images are acquired during the process of inserting or removing the endoscope. These endoscopic images may contain information that is extremely useful for the next examination or for treating others with similar symptoms.

However, simply recording these endoscopic images makes it extremely difficult to extract useful information from the huge number of images. Therefore, in this embodiment, the characteristics of each part are determined and the determination results are recorded in association with the images as metadata, making it easier to extract useful information.

FIG. 6 shows an example in which such determination of region-specific characteristics is performed when the endoscope is inserted, but it may also be performed during screening or observation. In addition, after the endoscope is removed, the endoscopic images recorded in a recording device (not shown) may be provided to the endoscopic image classification device 10 to determine the region-specific characteristics.

In the example of FIG. 6, the endoscopic image classification device 10 determines in S1 whether or not the endoscope is being inserted. When the endoscope is inserted (YES in S1), the part determination unit 12 performs part determination and determines whether or not part determination has been performed (S2). When part determination has been performed, the special light observation image is treated separately in the next S3, and then the part feature determination unit 13 detects part features. In special light observation, an organ illuminated with special light such as infrared light is imaged. When determining part features, it is not possible to simply compare an image obtained by imaging an organ illuminated with normal light (white light) with an image obtained by imaging with special light, so the special light observation image is treated separately. In other words, for the special light observation image, part features are determined within a group of special light observation images, and the image is used for searching within the group. The endoscopic image classification device 10 may determine whether or not the image is a special light observation image by image analysis, or, if accompanying information indicating whether or not the image is a special light observation image is added to the image, this accompanying information may be used to determine whether or not the image is a special light observation image.

Note that the part determination unit 12 and part feature determination unit 13 may not only perform part determination and part feature determination by image analysis of the input image or AI processing, but may also perform these determinations using associated information and active operation information. For example, for deformation during air supply, which is one of the determination items for part feature, information on the air supply operation is obtained from active operation information, and part feature determination is performed by image analysis of the deformation of the organ during the air supply operation period. That is, the active operation information in this case is related to the timing of the captured image, and valid determination results can be obtained by determining part features according to changes in the image corresponding to the active operation. Part determination may also be performed using associated information based on the output of an insertion shape observation device.

When the part-specific features are detected by the part-specific feature determination unit 13 (YES in S4), the association unit 15 organizes the part-specific features by part and records them in association with the endoscopic image (S5). For example, the association unit 15 may record the part information and part-specific feature information as text, associated with each frame as image metadata. When recording is completed, processing returns to S1. Processing also returns to S1 if NO is determined in S2 and S4. When recording in S5, not only the part information and part-specific feature information are recorded, but also various information such as accompanying information and active operation information contained in the input image, and the type of logical model used to determine the part, etc.

In this way, the vast amount of image information obtained from endoscopic examinations is organized by features specific to each part, so that in a specific case, it is possible to search for endoscopic images from other cases using the feature information specific to each anatomical part that the endoscope passes through during insertion into or removal from the body.

If the endoscopic image classification device 10 determines in S1 that it is not the time for insertion (NO in S1), it determines in S11 whether it is the time for observation. If the endoscopic image classification device 10 determines in S11 that it is not the time for observation (NO in S11), it performs screening in S12. Note that screening refers to the act of imaging a relatively wide area to search for a lesion, etc. Also, the act of imaging a specific area in detail and checking it, for example by imaging close up or zooming in on the specific area, is called "observation".

The endoscopic image classification device 10 can determine the time of insertion, screening, and observation by image analysis. For example, assume that the insertion section 51 advances inside the lumen. In this case, the image part of the back of the lumen (back in the direction of the lumen length) where the illumination light from the endoscope tip does not reach has a low-luminance lumen cross-sectional shape (often approximately circular), and when the insertion section 51 advances inside the lumen, this image part is located approximately in the center of the endoscopic image, and continuous images are obtained in which the pattern of the lumen wall moves toward the periphery of the image. In addition, when the endoscope tip is bent toward the target wall surface from a state in which the low-luminance lumen cross-sectional shape part indicating the back of the lumen is located in the center of the image, the low-luminance part that was in the center moves to the periphery of the screen, and this state can be determined by image analysis. In addition, when a specific patterned area is observed for a long time by moving closer or further away, or by changing the viewing direction, similar patterns such as blood vessels and irregularities of lesions are captured in the continuous images, and the observation state can be determined by the change in these patterns.

In S13, the results of the screening are displayed on the display device 20 and recorded in the recording device 30. Furthermore, during observation (YES in S11), the endoscopic image classification device 10 performs lesion determination and differentiation (S14) using known image processing, AI (artificial intelligence) processing, etc., and displays the results on the display device 20 and records them in the recording device 30 (S15). After the processing of S13 and S15, the process returns to S1.

In this way, the recording device 30 stores image files that are endoscopic images of many people who have undergone endoscopic examinations, with information such as body part information, body part characteristic information, associated information, and active operation information associated with the images. The image information recorded in the recording device 30 can be classified by the information associated with the images, and image searches are possible based on the information associated with the images.

(movie)
The flow of Fig. 7 corresponds to the flow of Fig. 6, and is an example of detecting site-specific features from a video. In Fig. 7, the same steps as those in the flow of Fig. 6 are given the same reference numerals, and the description thereof will be omitted. Note that, although the flow of Fig. 7 also shows an example in which site determination and site-specific feature determination are performed when an endoscope is inserted, site determination and site-specific feature determination may be performed when the endoscope is removed, during endoscopic observation, during screening, or after the endoscopy is completed.

In S21, imaging by the endoscope is started, determination of the part and features of each part (feature determination) is started, display of the captured image is started, and recording of the captured image is started. In S22, the endoscopic image classification device 10 determines whether or not it is time to insert the endoscope. When it is inserted (YES in S22), the variable N indicating the part is initialized to 1, and it is determined to start determination of the Nth part (S24). When determination of the Nth part is started (YES in S24), in S25, the endoscopic image classification device 10 matches the conditions at the time of imaging, such as whether special light observation or normal observation, and acquires an image of the Nth part. Incidental information and active operation information are also acquired at the same time as the image.

The part determination unit 12 determines each part using the acquired image, associated information, and active operation information. In S26, the part feature determination unit 13 determines part features for each part using the acquired image, associated information, and active operation information. In S26, the endoscopic image classification device 10 determines whether the determination process for the Nth part has been completed. If the determination of the Nth part has not been completed (NO in S26), the process returns to S25 to continue determining the parts and part features.

When the determination of the Nth part is completed (NO in S26), in S27, the endoscopic image classification device 10 acquires timing information and track information. The timing information is information on the imaging time for the Nth part, and the track information is information that indicates the position of the recording unit (track) in the image information of the Nth part.

The association unit 15 organizes the part-specific characteristic information for the Nth part, and provisionally records it in a table format or text format in the image information as metadata (S28). This part-specific characteristic information is recorded in association with the endoscopic image in correspondence with timing information and track information. By using the timing information and track information, the part-specific characteristic information for the Nth part can be easily extracted from the recorded image information. Here, when recording metadata in text format, since the patterns of the recorded text can be counted in advance, a code corresponding to each text pattern can be assigned and the code can be recorded as metadata. In this case, the correspondence information between the text pattern and the code is appropriately shared with the system that references the code.

In S29, the endoscopic image classification device 10 increments the variable N indicating the part, and in S30, determines whether the examination has ended. If the examination has not ended (NO in S30), the process returns to S22 and is repeated. When images of all anticipated parts have been taken and the examination has ended (YES in S30), the association unit 15 controls the recording control unit 18 in S31 to have the recording device 30 record part-specific characteristic information for all parts in table format or text format for each part in the image file (image file acquired during examination) as metadata, and then ends the examination. As a result, the recording control unit 18 records tabular information as part-specific characteristic information describing the characteristics of each part for each of the multiple parts passed through during the insertion or removal process.

If the result of S22 is NO, the operation is the same as in FIG. 6.

In this way, the recording device 30 stores image files that are endoscopic images of many people who have undergone endoscopic examinations, with information such as body part information, body part characteristic information, associated information, and active operation information associated with the images. The image information recorded in the recording device 30 can be classified by the information associated with the images, and image searches are possible based on the information associated with the images.

(Use of classified and accumulated information)
FIG. 10 shows a method for using image information recorded in the recording device 30 (data confirmation flow).

The endoscopic image classification device 10 receives an examination specification for specifying the image information to be used by an input device (not shown). In S41, the endoscopic image classification device 10 is in a standby state for the specification of an examination for a certain subject. For example, the user can specify the target image information by specifying the person's name, the examination date and time, the type of examination such as an upper endoscopy examination, etc. When an examination is specified (YES in S41), the search unit 17 searches for an image file corresponding to the specified examination (S42). The search unit 17 provides the image file of the search result to the display control unit 16, which displays the image on the display device 20.

The search unit 17 determines whether or not the user has specified a part of the image that he or she wishes to check (part specification) (S43). If no part has been specified, the search unit 17 controls the display control unit 16 to start video playback of the displayed image (S46), and then determines whether or not a playback stop operation has been performed (S47). In S47, the search unit 17 continues video playback until the user performs a stop operation, and if a stop operation has been performed, the process proceeds to S44, and if no stop operation has been performed, the process proceeds to S51. In S51, the search unit 17 determines whether the process has ended, and if not, returns to S43.

If a stop operation is performed during video playback (YES in S47), the search unit 17 displays a still image of the part displayed at the time of the stop operation (S44). Also, if in S43 the user inputs a part such as the stomach or large intestine using an input device (not shown) (YES in S43), a still image of the part specified by this input is displayed (S44).

In this way, an image of the specified subject, of the part of the body that the doctor or other person wishes to check, is displayed. In S45, the search unit 17 determines whether or not an operation has been performed to check similar cases. If no such checking operation has been performed (NO in S45), the process proceeds to S51, and if a checking operation has been performed, the process proceeds to S48.

In S48, the search unit 17 searches for videos with similar characteristics. That is, the search unit 17 searches the vast amount of image information stored in the recording device 30 for an image portion that is based on the body part information and has a body part characteristic based on the body part characteristic information (S48), reads out the image portion based on timing information and track information that specify the position of the image portion, and displays it as a still image on the display device 20 (S49) (see image Pe2 in Figure 1). The search unit 17 may also use accompanying information and active operation information in this search process.

Furthermore, if there is progress information for the subject whose similar image was captured in S48, the search unit 17 searches for the progress information and displays the searched information (see image Pe3 in Figure 1).

In this manner, in this embodiment, each body part is classified according to its body part features based on the image of each body part, and the classification results are accumulated. By searching for images based on the body part features, it becomes easy to search for images with similar body part features. This makes it easier to extract not only one's own examination images with similar body part features, but also other people's examination images similar to one's own examination images, which can be useful for diagnosis and prevention.

In addition, information on the movements of the surgeon (doctor performing the examination) or the movements of the assistant during an endoscopic examination or treatment or surgery using an endoscope may be used to search for similar images, including the procedure. Similarly, images may be searched for using setting information on the treatment tools and equipment used in the examination, treatment, or surgery.

In this embodiment, gastrointestinal endoscopic examination has been used as an example, but the present invention can also be applied to endoscopes other than those inserted into the gastrointestinal tract (in fields such as surgery, otolaryngology, gynecology, and urology), such as those used in laparoscopic surgery and thoracoscopic surgery.
Not only in the case of a single connected tube such as the intestine, but also in the case of a region such as the bronchi, the tubes are branched. In this case, it is preferable to add information on which tube has been selected.

Also, due to its nature, an endoscope travels back and forth within the body when inserted and removed, and the characteristics can be classified separately for when it is inserted and when it is removed. In that case, two tables like the one in Figure 4 will be created for one case. The characteristics can change when the lumen is pushed or pulled by friction, and so these can be organized in a table as average values.

The present invention is not limited to the above-described embodiments, and can be embodied by modifying the components in the implementation stage without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining the multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be deleted. Furthermore, components from different embodiments may be appropriately combined.

Furthermore, among the technologies described herein, many of the controls and functions mainly described in the flowcharts can be set by a program, and the above-mentioned controls and functions can be realized by having a computer read and execute the program. The program can be recorded or stored in whole or in part as a computer program product on portable media such as flexible disks, CD-ROMs, nonvolatile memories, or storage media such as hard disks and volatile memories, and can be distributed or provided at the time of product shipment or via portable media or communication lines. A user can easily realize the endoscopic image classification device of this embodiment by downloading the program via a communication network and installing it on a computer, or by installing it from a recording medium on a computer.

Claims (20)

a part determination unit that determines an anatomical part of a human body according to images sequentially obtained during the process of inserting the endoscope into the body or removing the endoscope from the body;
a feature determination unit that determines a feature by region, which is a feature of the determined region, for one or more determination items to obtain feature information by region;
an association unit that associates the part-specific characteristic information for each of the parts with the captured image and records the information;
An endoscopic image classification device comprising: a recording control unit that records information in a table format as the site-specific characteristic information describing the characteristics of each of a plurality of sites through which the device passes during the insertion or removal process;
The endoscopic image classification device according to claim 1 , further comprising: a database for referring to the judgment items for judging the site-specific characteristics for each of a plurality of sites passed through during the insertion or removal process;
The endoscopic image classification device according to claim 1 , further comprising: The database includes the judgment items for judging the site-specific characteristics of the image change following an active treatment.
The endoscopic image classification device according to claim 1 . the database includes information of recommended active manipulations at the anatomical site;
a display control unit that displays a guide to an endoscope operator according to the information on the active operation;
The endoscopic image classification device according to claim 3, further comprising: a search unit that searches for an endoscopic image acquired in a second case by using the site-specific characteristic information obtained for each anatomical site through which the endoscope passes during the insertion or removal of the endoscope into or from the body in a first case;
The endoscopic image classification device according to claim 1 , further comprising: an active operation determination unit that determines a signal based on an operation by an operator while the endoscope is inserted into a body and obtains active operation information;
The associating unit associates part information indicating the part, the part feature information, and the active operation information with the captured image and records them.
The endoscopic image classification device according to claim 1 . The captured image has accompanying information added thereto,
the associating unit associates the body part information, the body part feature information, the active operation information, and the accompanying information with the captured image and records them;
The endoscopic image classification device according to claim 7 . The feature determination unit obtains part-specific feature information indicating the result of the determination of the part-specific features in a text format, a code format, or a table format.
The endoscopic image classification device according to claim 1 . the associating unit adds the part-by-part characteristic information in the table format to the captured image as metadata when recording a file of the captured image, and records the information.
The endoscopic image classification device according to claim 9 . The associating unit associates the part-specific characteristic information with timing information of the captured image and records the information in association with the captured image.
The endoscopic image classification device according to claim 9 . the feature determination unit determines the site-specific features based on image changes in each frame of the sequentially acquired captured images;
The endoscopic image classification device according to claim 1 . acquiring the active operation information related to a timing of the captured image, and determining the body part feature according to a change in the image corresponding to the active operation;
The endoscopic image classification device according to claim 1 . a search unit that performs a search process based on the part-by-part characteristic information on the captured images of the part determined by the part determination unit to extract captured images of similar cases;
The endoscopic image classification device according to claim 1 , further comprising: a display control unit for displaying a search result of the search unit on a display device;
The endoscopic image classification device according to claim 9, further comprising: The search unit searches for progress information of a subject from whom an image of the similar case has been captured, if such progress information exists;
The display control unit displays the progress information.
The endoscopic image classification device according to claim 15 . determining an anatomical location in the human body according to images sequentially obtained during the process of inserting the endoscope into the body or removing it from the body;
determining a region-specific characteristic, which is a characteristic of the determined region, for one or more determination items to obtain region-specific characteristic information;
The part-specific characteristic information for each of the parts is recorded in association with the captured image.
13. An image classification method comprising: On the computer,
determining an anatomical location in the human body according to images sequentially obtained during the process of inserting the endoscope into the body or removing it from the body;
determining a region-specific characteristic, which is a characteristic of the determined region, for one or more determination items to obtain region-specific characteristic information;
The part-specific characteristic information for each of the parts is recorded in association with the captured image.
An image classification program to carry out the procedure. determining a plurality of anatomical parts of the human body according to images sequentially obtained during the process of inserting the endoscope into the body or removing the endoscope from the body;
In order to obtain site-specific characteristics for each of the plurality of sites, a database is referred to that organizes judgment items indicating characteristics to be obtained for each of the sites;
determining the region-specific characteristics for each of the determined regions, and obtaining region-specific characteristic information based on the region-specific characteristics so that the determined region-specific characteristics can be recorded in association with the captured image;
13. An image classification method comprising: A search method characterized by searching for endoscopic images acquired in a second case using part-specific characteristic information indicating the characteristics of each anatomical part through which an endoscope passes during insertion or removal of the endoscope into or from the body in a first case.

Images