JP7619893B2 - ENDOSCOPIC APPARATUS, ENDOSCOPIC SYSTEM, OPERATION METHOD OF ENDOSCOPIC APPARATUS, AND PROGRAM - Google Patents

Description

The present invention relates to an endoscope device, an endoscope system, an operation method for an endoscope device, and a program.

Industrial endoscope devices are used to inspect the inside of a specimen for damage, corrosion, and the like. The specimens include boilers, pipes, and aircraft engines. The endoscope device has a function for generating videos of the specimen. The following technology is known for automatically editing videos. This technology uses Visual Simultaneous Localization and Mapping (Visual SLAM) or the like to estimate the position and orientation of the tip of the endoscope when each frame of the video is generated. This technology also associates information indicating the position and orientation with each frame and manages this information.

The technology disclosed in Patent Document 1 provides a method for editing video generated when shooting is performed with a camera moving at a constant speed. The technology thins out frames based on shooting position information associated with each frame in the video. As shown in Figures 4 and 17 of Patent Document 1, the technology extracts multiple frames generated at regular distances and stitches the frames together. In this way, the technology generates a video similar to the video generated when shooting is performed with a camera moving at a constant speed.

The technology disclosed in Patent Document 2 provides a method for changing the playback speed of a video. The technology performs image processing to extract a period during which a characteristic portion appears in the frame. The technology slows down the playback speed of the video during that period. This causes the characteristic portion of the video to be played in slow motion.

JP 2001-290820 A Patent No. 6663491

In conventional technology, the burden on the user is great when viewing the examination video generated during the examination. When the user views an examination video that takes a long time to shoot, the viewing takes a long time, which places a great burden on the user. In particular, when the subject is complex and large, the shooting time of the examination video is very long. In addition, the speed at which the endoscope is inserted into the subject (insertion speed) is not constant. The user may view the video by fast-forwarding. In general fast-forwarding playback, the playback speed of the video is constant, but the endoscope is inserted into the subject at an apparent speed that corresponds to the actual insertion speed. As a result, when the examination video is played back, there are scenes in which the subject in the image moves too fast or too slow, making it difficult for the user to view the examination video.

The technology disclosed in Patent Document 1 thins out video frames so that the camera speed remains constant. However, this technology may cause the user to miss important parts.

The technology disclosed in Patent Document 2 extracts feature points by analyzing an image, and emphasizes the period during which the feature points appear in the frame. For example, the technology slows down the playback speed of the video so that the video is played in slow motion during that period. The location that the inspector focuses on during the inspection is not necessarily a location that has such feature points. Therefore, when the inspection video is played back, slow motion playback is not necessarily performed at the location that the inspector focuses on. As a result, the viewer of the inspection video may focus on a location different from the location that the inspector focuses on. When the viewer approves the inspection results, they are unable to correctly evaluate the locations that are important to the inspector. This reduces the efficiency of the entire inspection.

The present invention aims to provide an endoscope device, an endoscope system, an operation method for an endoscope device, and a program that can reduce the burden on the user when playing back a video and allow the user to easily check important points when playing back a video.

The device has an insertion section that can be inserted into a subject, the insertion section having an image sensor that generates a video including two or more frames, and a control section, the control section having an information receiving section that receives operation information indicating the content of an operation performed by a user, an information generating section that generates length information indicating the length of a portion of the insertion section that is inserted into the subject when each of the two or more frames is generated, and a video editing section that edits the video, and the video editing section divides the two or more frames into one or more frames in a first section and one or more frames in a second section that are different from the first section based on the operation information. and one or more frames in a second section, the first section including a timing when a user performs a predetermined operation, the video editing unit reduces the number of frames per unit length in the second section based on the length information, the video editing unit adjusts the number of frames in the first section so that the first number of frames is greater than the second number of frames, the first number of frames indicates the number of frames per unit length in the first section, and the second number of frames indicates the number of frames per unit length in the second section.

In the endoscope device of the present invention, the video editing unit adjusts the number of frames in the second section so that the apparent speed at which the insertion section is inserted into the subject during playback of the video in the second section is constant.

In the endoscope device of the present invention, the information generating unit uses the two or more frames to estimate the position of the image sensor at the time when each of the two or more frames was generated, and estimates the length based on the estimated position.

In the endoscope device of the present invention, the information generating unit estimates the position by using sensor information in addition to the two or more frames, and the sensor information includes at least one of first sensor information generated by a first sensor, second sensor information generated by a second sensor, and third sensor information generated by a third sensor, the first sensor information indicating the length, the second sensor information indicating the acceleration of the tip of the insertion section, and the third sensor information indicating the angular velocity of the tip.

In the endoscope device of the present invention, the video editing unit deletes a portion of the two or more frames so that the trajectory of the position change is the shortest.

In the endoscope device of the present invention, the information generating unit generates the length information based on sensor information generated by a sensor that detects the length.

In the endoscope device of the present invention, the operation information includes at least one of an instruction to acquire a still image, an instruction to save a still image, an instruction to perform a measurement, an instruction to bend the insertion section, an instruction to change the image processing settings, an instruction to attach a label to the video, and an instruction to attach information to a still image.

In the endoscopic device of the present invention, the information includes notes and wound evaluation results.

In the endoscope device of the present invention, the second section includes one or more subsections, and if the number of frames per unit length in at least one of the one or more subsections is greater than a predetermined value, the video editing unit thins out at least one frame in at least one of the one or more subsections.

In the endoscope device of the present invention, the second section includes two or more subsections, and if the number of frames per unit length in at least one of the two or more subsections is equal to or less than a predetermined value, the video editing unit adds at least one frame to at least one of the two or more subsections.

In the endoscope device of the present invention, the one or more frames in the first section include one frame generated at the timing when the user performs the specified operation, one or more frames generated before the timing, and one or more frames generated after the timing.

The present invention includes an endoscope device and a control device, the endoscope device having an insertion section that can be inserted into a subject, the insertion section having an image sensor that generates a video including two or more frames, an information receiving section that receives operation information indicating the content of an operation performed by a user, and a transmission section that transmits the video to the control device, the control device having a receiving section that receives the video from the endoscope device, an information generating section that generates length information indicating the length of the portion of the insertion section that is inserted into the subject when each of the two or more frames is generated, and a video editing section that edits the video, the video editing section editing the two or more frames based on the operation information. An endoscope system that classifies frames into one or more frames in a first section and one or more frames in a second section different from the first section, the first section includes a timing when a user performs a predetermined operation, the video editing unit reduces the number of frames per unit length in the second section based on the length information, the video editing unit adjusts the number of frames in the first section so that the first number of frames is greater than the second number of frames, the first number of frames indicates the number of frames per unit length in the first section, and the second number of frames indicates the number of frames per unit length in the second section.

The present invention relates to an operation method for an endoscope device having an insertion section that can be inserted into a subject, the insertion section having an image sensor that generates a video including two or more frames, and a control section, the operation method including an information receiving step in which an information receiving section of the control section receives operation information indicating the content of an operation performed by a user, an information generating step in which an information generating section of the control section generates length information indicating the length of a portion of the insertion section that is inserted into the subject when each of the two or more frames is generated, and a video editing step in which a video editing section of the control section edits the video, the video editing step including editing the video by dividing the two or more frames into one or more frames in a first section and one or more frames in a second section based on the operation information. and one or more frames in a second section different from the first section, the first section including the timing when the user performed a predetermined operation; a second section editing step in which the video editing unit reduces the number of frames per unit length in the second section based on the length information; and a first section editing step in which the video editing unit adjusts the number of frames in the first section so that the first number of frames is greater than the second number of frames, the first number of frames indicating the number of frames per unit length in the first section, and the second number of frames indicating the number of frames per unit length in the second section.

The present invention is a program for causing a computer of an endoscope device having an insertion section that can be inserted into a subject, the insertion section having an image sensor that generates a video including two or more frames, and a control section, to execute an information receiving step of receiving operation information indicating the content of an operation performed by a user, an information generating step of generating length information indicating the length of a portion of the insertion section that is inserted into the subject when each of the two or more frames is generated, and a video editing step of editing the video, in which the video editing step divides the two or more frames into one or more frames in a first section and one or more frames in a second section different from the first section based on the operation information. a classification step of classifying the first section into one or more frames between the first section and the second section, the first section including the timing at which the user performed a predetermined operation; a second section editing step of reducing the number of frames per unit length in the second section based on the length information; and a first section editing step of adjusting the number of frames in the first section so that the first number of frames is greater than the second number of frames, the first number of frames indicating the number of frames per unit length in the first section, and the second number of frames indicating the number of frames per unit length in the second section.

According to the present invention, an endoscope device, an endoscope system, an operation method for an endoscope device, and a program can reduce the burden on a user when playing back a video, and can allow a user to easily check important points when playing back a video.

1 is a perspective view showing an overall configuration of an endoscope apparatus according to a first embodiment of the present invention; 1 is a block diagram showing an internal configuration of an endoscope apparatus according to a first embodiment of the present invention. 5 is a flowchart showing a procedure of a moving image recording process according to the first embodiment of the present invention. 5 is a flowchart showing a procedure of a video editing process according to the first embodiment of the present invention. 1 is a diagram illustrating a moving image before a moving image editing process is executed according to a first embodiment of the present invention; 4A and 4B are diagrams illustrating a moving image after a moving image editing process has been executed according to the first embodiment of the present invention. FIG. 2 is a block diagram showing an internal configuration of an endoscope apparatus according to a first modified example of the first embodiment of the present invention. FIG. 13 is a diagram showing importance levels given to operation information in the first modified example of the first embodiment of the present invention. 13 is a flowchart showing a procedure of a video editing process according to a second modified example of the first embodiment of the present invention. 13 is a flowchart showing a procedure of a video editing process according to a third modified example of the first embodiment of the present invention. FIG. 13 is a diagram illustrating a moving image before a process according to a third modified example of the first embodiment of the present invention is executed. FIG. 13 is a diagram illustrating a moving image after a process according to a third modified example of the first embodiment of the present invention is executed. FIG. 11 is a block diagram showing an internal configuration of an endoscope apparatus according to a second embodiment of the present invention. 13A and 13B are diagrams illustrating a state of an insertion section in a second embodiment of the present invention. FIG. 4 shows frames acquired in a second embodiment of the present invention. FIG. 4 shows frames acquired in a second embodiment of the present invention. FIG. 4 shows frames acquired in a second embodiment of the present invention. 10 is a flowchart showing a procedure of a moving image recording process according to a second embodiment of the present invention. FIG. 13 is a block diagram showing an internal configuration of an endoscope apparatus according to a first modified example of the second embodiment of the present invention. 13 is a flowchart showing a procedure of a moving image recording process in a second modified example of the second embodiment of the present invention. FIG. 11 is a block diagram showing a configuration of an endoscope system according to a third embodiment of the present invention. 13 is a flowchart showing a procedure of a moving image recording process according to a third embodiment of the present invention. 10 is a flowchart showing a procedure of a process executed by a control device according to a third embodiment of the present invention. 13 is a flowchart showing a procedure of a moving image recording process in a modified example of the third embodiment of the present invention. 13 is a flowchart showing a procedure of a moving image recording process in a modified example of the third embodiment of the present invention. FIG. 13 is a block diagram showing a configuration of an endoscope system according to a fourth embodiment of the present invention.

The following describes an embodiment of the present invention with reference to the drawings.

(First embodiment)
A first embodiment of the present invention will be described below. Fig. 1 shows the appearance of an endoscope device 1 according to the first embodiment. The endoscope device 1 shown in Fig. 1 has an insertion section 2, a control section 3, an operation section 4, and a display section 5.

The endoscope device 1 captures an image of a subject (specimen) and generates an image. In order to observe various specimens, the examiner can replace the optical adapter attached to the tip of the insertion section 2, select a built-in image processing program, and add an image processing program. The specimen is an industrial product.

The insertion section 2 is inserted into the subject. The insertion section 2 is a long, thin tube that can be bent from the tip 20 to the base end. The insertion section 2 captures an image of the subject and outputs an image signal to the control section 3. An optical adapter is attached to the tip 20 of the insertion section 2. The control section 3 is a control device that includes a storage section that stores the insertion section 2. The operation section 4 accepts user operations on the endoscope device 1. The display section 5 has a display screen, and displays images of the subject acquired by the insertion section 2, operation menus, etc. on the display screen.

The operation unit 4 is a user interface (input device). For example, the operation unit 4 is at least one of a button, a switch, a key, a mouse, a joystick, a touchpad, a trackball, and a touch panel. The display unit 5 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 5 may be a touch panel. In that case, the operation unit 4 and the display unit 5 are integrated. The user touches the screen of the display unit 5 by using a part of the body (e.g., a finger) or a tool. Below, an example in which the operation unit 4 and the display unit 5 of the endoscope device 1 are used will be described. An external terminal different from the endoscope device 1 may have an operation unit or a display unit, and the operation unit or the display unit may be used.

Figure 2 shows the internal configuration of the endoscope device 1. The configurations of the insertion section 2 and the control section 3 are shown in Figure 2. The insertion section 2 shown in Figure 2 has a bending section 21 and an image sensor 22. The control section 3 has an information receiving section 30, a memory section 31, an information generating section 32, and a video editing section 33.

The schematic configuration of the endoscope device 1 will be described. The insertion section 2 can be inserted into the subject. The image sensor 22 generates a video including two or more frames. The information receiving section 30 receives operation information indicating the content of the operation performed by the user. The information generating section 32 generates length information indicating the length of the portion of the insertion section 2 inserted into the subject when each of the two or more frames is generated. The video editing section 33 edits the video. The video editing section 33 classifies the two or more frames into one or more frames in a first section and one or more frames in a second section different from the first section based on the operation information. The first section includes the timing when the user performed a predetermined operation. The video editing section 33 reduces the number of frames per unit length in the second section based on the length information. The video editing section 33 adjusts the number of frames in the first section so that the first number of frames is greater than the second number of frames. The first number of frames indicates the number of frames per unit length in the first section. The second number of frames indicates the number of frames per unit length in the second section.

The detailed configuration of the endoscope device 1 will be described. The bending section 21 has wires and a drive device for bending the insertion section 2. The bending section 21 bends the insertion section 2 based on operation information output from the information receiving section 30. The operation information indicates an instruction to bend the insertion section 2 (bending instruction).

The imaging element 22 is disposed at the tip 20 of the insertion section 2. The imaging element 22 is an image sensor such as a CCD sensor or a CMOS sensor. The imaging element 22 performs imaging and continuously generates two or more frames of the subject. Each of the two or more frames is an image. The two or more frames constitute a video. The imaging element 22 outputs each generated frame to the control unit 3. The imaging element 22 also generates one or more still images based on operation information output from the information receiving unit 30, and outputs the generated still images to the control unit 3. The operation information indicates an instruction to acquire a still image (still image acquisition instruction).

The user inputs operation information to the endoscope device 1 by operating the operation unit 4. The operation unit 4 outputs the information input by the user. The information receiving unit 30 receives the operation information output from the operation unit 4. The operation information is an instruction related to video recording, an instruction to acquire a still image, or an instruction to bend. The instruction related to video recording is a recording start instruction to start video recording or a recording end instruction to end video recording. Other examples of operation information will be described in the first modified example of the first embodiment.

When the information receiving unit 30 receives operation information other than instructions related to video recording, the information receiving unit 30 adds the operation information to a frame of the video. When the information receiving unit 30 receives a still image acquisition instruction, the information receiving unit 30 outputs the still image acquisition instruction to the imaging element 22. When the still image acquisition instruction is output from the information receiving unit 30, the imaging element 22 generates one or more still images. When the information receiving unit 30 receives a bending instruction, the information receiving unit 30 outputs the bending instruction to the bending section 21. When the bending instruction is output from the information receiving unit 30, the bending section 21 bends the insertion section 2.

The storage unit 31 is a volatile or non-volatile recording medium. For example, the storage unit 31 is at least one of a RAM (Random Access Memory), a DRAM (Dynamic Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), and a flash memory.

The storage unit 31 stores a video file including two or more frames generated by the imaging element 22. Operation information is added to one or more of the two or more frames. Furthermore, length information is added to each of the two or more frames. The storage unit 31 also stores one or more still images generated by the imaging element 22.

The memory unit 31 may be a recording medium that can be attached to and detached from the endoscope device 1. Therefore, the endoscope device 1 does not need to have the memory unit 31.

The information generating unit 32 performs image processing using two or more frames generated by the imaging element 22. The image processing is Visual SLAM or structure from motion (SfM), etc. The information generating unit 32 estimates the position of the tip 20 of the insertion unit 2 by performing self-position estimation in the image processing. The information generating unit 32 also generates position information. The position information includes coordinate information and length information. The information generating unit 32 adds the position information to each frame.

The coordinate information indicates the coordinates (position) of the tip 20 of the insertion section 2 when each of the two or more frames is generated. The image sensor 22 is disposed at the tip 20. Therefore, the coordinate information indicates the coordinates (position) of the image sensor 22 when each of the two or more frames is generated.

The length information indicates the length (insertion length) of the portion of the insertion section 2 that is inserted into the subject when each of the two or more frames is generated. The information generating unit 32 calculates the insertion length based on the coordinate information.

In the first embodiment, the insertion length indicates the distance between a first position of the tip 20 and a second position of the tip 20. The first position indicates the position of the tip 20 when the reference frame is generated. The second position indicates the position of the tip 20 when a frame that is one or more frames later than the reference frame is generated.

The video editing unit 33 processes the video. When the information receiving unit 30 receives a recording start instruction, the video editing unit 33 starts recording the video. When the information receiving unit 30 receives a recording end instruction, the video editing unit 33 ends recording the video. After the video recording ends, the video editing unit 33 executes a video editing process. The video editing unit 33 edits the video file in the video editing process.

The information receiving unit 30, the information generating unit 32, and the video editing unit 33 may be composed of at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). The information receiving unit 30, the information generating unit 32, and the video editing unit 33 may include one or more processors. The information receiving unit 30, the information generating unit 32, and the video editing unit 33 may include one or more logic circuits.

The computer of the endoscope device 1 may load a program and execute the loaded program. The program includes instructions that define the operation of the information receiving unit 30, the information generating unit 32, and the video editing unit 33. In other words, the functions of the information receiving unit 30, the information generating unit 32, and the video editing unit 33 may be realized by software.

The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer that holds the program to the endoscope device 1 via a transmission medium or by a transmission wave in the transmission medium. The "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The above function may be realized by a combination of a program already recorded in the computer and a difference program.

The video recording process in the first embodiment will be explained using Figure 3. Figure 3 shows the steps of the video recording process.

The image sensor 22 continuously generates frames. The display unit 5 continuously displays the frames to display a real-time image of the subject.

Video recording does not start until the information receiving unit 30 receives a recording start instruction. When the user inputs a recording start instruction to the endoscope device 1 by operating the operation unit 4, the information receiving unit 30 receives the recording start instruction (step S100). At this time, video recording starts.

After video recording has started, the frames output from the image sensor 22 are stored in the memory unit 31 (step S101).

The user inputs operation information to the endoscope device 1 by operating the operation unit 4. The operation information is a still image acquisition instruction or a bending instruction. After step S101, the information receiving unit 30 receives the operation information (step S102). Step S102 may be executed before step S101 is executed.

After step S102, the information receiving unit 30 adds the operation information to the frame stored in the storage unit 31 in step S101. For example, the information receiving unit 30 embeds the operation information in the frame (step S103).

The operation information may be separated from the frame stored in the storage unit 31. Information that associates the operation information with the frame may be added to the frame.

If the user does not input operation information to the endoscope device 1, steps S102 and S103 are not executed. Therefore, operation information may not necessarily be added to all of two or more frames.

When the user inputs a recording end instruction to the endoscope device 1 by operating the operation unit 4 at a desired timing, the information receiving unit 30 receives the recording end instruction (step S104). Steps S101 to S103 are repeated until the information receiving unit 30 receives the recording end instruction.

After the information receiving unit 30 receives the instruction to end recording, the video editing unit 33 generates a video file that includes the frames stored in the storage unit 31 (step S105). The video file is stored in the storage unit 31.

After step S105, the information generating unit 32 performs self-location estimation by using two or more frames included in the video file. As a result, the information generating unit 32 generates location information for each of the two or more frames. The information generating unit 32 adds the generated location information to each frame. For example, the information generating unit 32 embeds the location information in the frame (step S106).

The location information may be separate from the frame stored in the memory unit 31. Information that associates the location information with the frame may be added to the frame.

After step S106, the video editing unit 33 executes the video editing process (step S107). When the video editing process ends, the video recording process ends.

The video editing process in step S107 will be explained using Figure 4. Figure 4 shows the steps of the video editing process.

The video editing unit 33 classifies two or more frames of the video into one or more frames in a first section and one or more frames in a second section different from the first section, based on the operation information added to the frames of the video (step S107a).

The predetermined operation in the first embodiment is an operation of acquiring a still image or an operation of bending the insertion portion 2. In other words, the predetermined operation is an operation of inputting a still image acquisition instruction to the endoscope device 1 or an operation of inputting a bending instruction to the endoscope device 1.

When an area that the user is interested in is shown in the image, the user inputs a still image acquisition instruction to the endoscope device 1 to acquire a still image of that area. Alternatively, when that area is shown in the image, the user inputs a bending instruction to the endoscope device 1 to observe that area in detail. When a still image acquisition instruction or a bending instruction is input to the endoscope device 1, there is a high possibility that the area that the user is interested in is shown in the image.

The first interval and the second interval are each a time interval (period). The first interval includes the timing when the user performs a specified operation. The first interval corresponds to the period during which the user performs the specified operation. One or more frames in the first interval include frames generated at a timing corresponding to the timing (operation timing) when the user performs the specified operation. The one or more frames in the first interval correspond to an image showing an area that the user focuses on. The image sensor 22 generates frames in each frame period. For example, the one or more frames in the first interval include frames generated during the frame period during which the user performs the specified operation.

The one or more frames in the first section may include the frame generated at the operation timing as well as frames before and after the frame. For example, the one or more frames in the first section may include one or more frames before the frame generated at the operation timing. In other words, the one or more frames in the first section may include one or more frames at an insertion length between a first insertion length and a second insertion length. The first insertion length indicates the insertion length in the frame generated at the operation timing. The second insertion length is shorter than the first insertion length by a predetermined length.

The one or more frames in the first section may include one or more frames following the frame generated at the operation timing. In other words, the one or more frames in the first section may include one or more frames at an insertion length between the first insertion length and the third insertion length. The first insertion length indicates the insertion length in the frame generated at the operation timing. The third insertion length is a predetermined length longer than the first insertion length.

The user may input information indicating the size of the subject to the endoscope device 1 by operating the operation unit 4. Alternatively, the user may input information indicating the length of the first section to the endoscope device 1 by operating the operation unit 4. The video editing unit 33 may set the length of the first section based on the information input to the endoscope device 1.

The second section corresponds to a period during which the user does not perform a specified operation. The second section does not include a time when the user performs a specified operation. One or more frames in the second section correspond to an image that does not include an area that the user focuses on.

For example, the second interval is longer in time than the first interval. The number of frames in the second interval is greater than the number of frames in the first interval.

The video editing unit 33 adjusts the number of frames per unit length of the insertion length in steps S107b and S107c described below. The unit length indicates a predetermined length in the unit system (coordinate system) of the insertion length. Since length information is added to each frame, the insertion length is associated with each frame. The value of the insertion length indicates the position of each frame in the unit system of the insertion length. For example, the insertion length L1 is associated with frame F1, and the insertion length L2 is associated with frame F2. If the difference between the insertion length L1 and the insertion length L2 is the same as the unit length, the number of frames per unit length is the number of frames from frame F1 to frame F2.

After step S107a, the video editing unit 33 adjusts the number of frames in the second section based on the length information added to each frame of the video. Specifically, the video editing unit 33 reduces the number of frames per unit length of the insertion length (step S107b).

Because the number of frames in the second section is reduced, the playback time of the video in the second section is shortened. This reduces the burden on the user viewing the video.

The video editing unit 33 may adjust the number of frames in the second section so that the speed at which the insertion unit 2 is inserted into the subject (insertion speed) is constant when the video is played back. In this case, the video editing unit 33 may thin out frames or add frames. The video editing unit 33 may adjust the number of frames in the second section so that the insertion speed when the video is played back in the second section is faster than the insertion speed when the video is played back in the first section and varies over time.

If the number of frames per unit length of the insertion length is greater than a predetermined number in at least a portion of the second section, the video editing unit 33 thins out frames. In other words, the video editing unit 33 deletes some of the frames in the second section. For example, if the amount of movement of the subject between two or more consecutive frames in the second section is small, the video editing unit 33 deletes some of those two or more frames.

If the number of frames per unit length of the insertion length in a portion of the second section is smaller than a predetermined number, the video editing unit 33 adds frames. The video editing unit 33 may duplicate one or more frames in the second section and add the one or more frames to the second section. The video editing unit 33 may interpolate frames in the second section based on two or more frames in the second section. Even when frames are added to a portion of the second section, the number of frames in the entire second section is reduced.

When the video in the second section is played, the subject in the image moves at a constant speed. This reduces the burden on the user viewing the video.

After step S107b, the video editing unit 33 adjusts the number of frames in the first section so that the first number of frames is greater than the second number of frames adjusted in step S107b (step S107c). The first number of frames indicates the number of frames per unit length of the insertion length in the first section. The second number of frames indicates the number of frames per unit length of the insertion length in the second section. When step S107c is executed, the video editing process ends.

For example, the video editing unit 33 reduces the number of frames in the first section by deleting some of the frames in the first section. Alternatively, the video editing unit 33 increases the number of frames in the first section by adding frames to the first section.

The video editing unit 33 may adjust the number of frames in the first section so that the insertion unit 2 remains still for at least a part of the first section. The video editing unit 33 may adjust the number of frames in the first section so that, in the video in the first section, the insertion unit 2 gradually slows down and the insertion unit 2 remains still. Furthermore, the video editing unit 33 may adjust the number of frames in the first section so that, in the video in the first section, the insertion unit 2 gradually speeds up after the insertion unit 2 remains still.

Because the first number of frames is greater than the second number of frames, the first subject is slower than the second subject. The first subject is captured in frames in the first section. The second subject is captured in frames in the second section. Therefore, the video in the first section is played in slow motion. A viewer of the video can easily check the important points that the inspector focused on.

The order of steps S107b and S107c is not limited to the order shown in FIG. 4. Step S107c may be executed before step S107b is executed. In that case, the final second number of frames in step S107b is calculated in advance, and the second number of frames is used in step S107c.

Figure 5 shows a schematic diagram of a video before the video editing process is performed. Figure 6 shows a schematic diagram of a video after the video editing process is performed.

The insertion portion 2 is inserted in the right direction in Figs. 5 and 6. Line L11 shown in Figs. 5 and 6 indicates the trajectory of movement of the tip 20 inserted into the subject IT11. Multiple marks on line L11 indicate the position of the tip 20 when the image sensor 22 generates a frame.

Position P11 in Figures 5 and 6 indicates the position of tip 20 when the user performs a specified operation. At this time, image sensor 22 generates frames. A first interval SE11 and a second interval SE12 are shown in Figures 5 and 6. The video in the first interval SE11 includes frames acquired when tip 20 is at position P11. The video in the first interval SE11 also includes two or more frames acquired when tip 20 is near position P11. The video in the second interval SE12 includes two or more frames acquired when tip 20 is at a position some distance away from position P11.

In the examples shown in Figures 5 and 6, the video editing unit 33 reduces the number of frames in each of the first section SE11 and the second section SE12. The video editing unit 33 adjusts the number of frames in the second section SE12 so that the speed at which the insertion section 2 is inserted into the subject is constant. The video editing unit 33 adjusts the number of frames in the first section SE11 so that the first number of frames in the first section SE11 is greater than the second number of frames in the second section SE12. The first number of frames and the second number of frames indicate the number of frames per unit length of the insertion length.

The operation method of the endoscope device of each aspect of the present invention has an information receiving step, an information generating step, and a video editing step. In the information receiving step (step S102), the information receiving unit 30 receives operation information indicating the content of the operation performed by the user. In the information generating step (step S106), the information generating unit 32 generates length information indicating the length of the portion of the insertion unit 2 inserted into the subject when each of the two or more frames is generated. In the video editing step (step S107), the video editing unit 33 edits the video. The video editing step has a classification step, a second section editing step, and a first section editing step. In the classification step (step S107a), the video editing unit 33 classifies the two or more frames into one or more frames in a first section and one or more frames in a second section different from the first section based on the operation information. The first section includes the timing when the user performs a predetermined operation. In a second section editing step (step S107b), the video editing unit 33 reduces the number of frames per unit length in the second section based on the length information. In a first section editing step (step S107c), the video editing unit 33 adjusts the number of frames in the first section so that the first number of frames is greater than the second number of frames. The first number of frames indicates the number of frames per unit length in the first section. The second number of frames indicates the number of frames per unit length in the second section.

Each aspect of the present invention may include the following modified example: The video editing unit 33 adjusts the number of frames in the second section so that the apparent speed at which the insertion unit 2 is inserted into the subject while the video in the second section is being played back is constant.

Each aspect of the present invention may include the following modified examples: The information generating unit 32 uses two or more frames to estimate the position of the image sensor 22 at the timing when each of the two or more frames is generated, and estimates the insertion length based on the estimated position.

Each aspect of the present invention may include the following modified examples: The one or more frames in the first section include one frame generated at the timing when the user performs a predetermined operation, one or more frames generated before that timing, and one or more frames generated after that timing.

In the first embodiment, the video editing unit 33 reduces the number of frames per unit length of the insertion length in the second section. In addition, the video editing unit 33 adjusts the number of frames in the second section so that the speed at which the insertion portion 2 is inserted into the subject is constant when the video is played back. Therefore, the endoscope device 1 can reduce the burden on the user when playing back the video. In the first embodiment, the video editing unit 33 adjusts the number of frames in the first section so that the first number of frames is greater than the second number of frames. Therefore, the endoscope device 1 can allow the user to easily check important points when playing back the video.

The viewer of the video checks whether the examination is being performed according to a predetermined procedure. The viewer checks whether predetermined locations on a subject having a complex structure have been examined. In the video that has not been edited in the first embodiment, the speed at which the insertion part 2 is inserted into the subject varies. For example, there are cases in which the work is not performed because the examiner is checking a manual or a drawing. In such cases, the insertion part 2 is stationary.

When fast-forward playback is performed at a constant speed without editing in the first embodiment, the video is easy to view during periods when insertion is not being performed. On the other hand, when a video is played during a period when smooth insertion is being performed, the insertion portion 2 is inserted quickly. Therefore, it is difficult for the viewer to confirm whether or not a predetermined location has been examined. The endoscope device 1 keeps the speed at which the insertion portion 2 is inserted into the subject constant during playback of the video in the second section, regardless of the actual speed at which the insertion portion 2 is inserted into the subject. Therefore, the viewer of the video can easily determine whether or not the examination is being performed according to a predetermined procedure.

The frames in the first section are likely to show the area that the user focuses on. The video in the first section is played in slow motion. This allows the viewer of the video to easily check the important points that the examiner focused on. On the other hand, the frames in the second section are unlikely to show the area that the user focuses on. The video in the second section is played so that the speed of the insertion part 2 is faster than the actual speed. This allows the viewer of the video to view the video in a short time.

There are cases where a large amount of video is stored for a long period of time as evidence of an examination. In such cases, it is desirable to reduce the amount of data in the video file. For example, if a user temporarily pauses an examination while recording a video, there will be a period during which there is little change in the position of the insertion portion 2. Because the number of frames during that period is reduced, the amount of unnecessary data is reduced. The endoscope device 1 can reduce the amount of data in the video file while retaining important data.

(First Modification of the First Embodiment)
A first modified example of the first embodiment of the present invention will be described. The endoscope device 1 shown in Fig. 2 is changed to an endoscope device 1a shown in Fig. 7. Fig. 7 shows the configuration of the endoscope device 1a. The control unit 3 shown in Fig. 2 is changed to a control unit 3a shown in Fig. 7. A description of the same configuration as that shown in Fig. 2 will be omitted.

The control unit 3a has an information receiving unit 30, a memory unit 31, an information generating unit 32, a video editing unit 33, and a processing unit 34. The information receiving unit 30, the information generating unit 32, the video editing unit 33, and the processing unit 34 may be composed of at least one of a processor and a logic circuit. The information receiving unit 30, the information generating unit 32, the video editing unit 33, and the processing unit 34 may include one or more processors. The information receiving unit 30, the information generating unit 32, the video editing unit 33, and the processing unit 34 may include one or more logic circuits.

The operation information in the first modified example of the first embodiment includes the following instructions in addition to the still image acquisition instruction and curvature instruction in the first embodiment. The operation information includes an instruction to save a still image (still image save instruction), an instruction to perform measurement (measurement execution instruction), an instruction to change the settings of image processing (setting change instruction), an instruction to attach a label to a moving image (label attachment instruction), and an instruction to attach information to a still image (information attachment instruction). The processing unit 34 executes processing according to the operation information.

The user inputs a still image acquisition instruction to the endoscope device 1a, and then inputs a still image storage instruction to the endoscope device 1a. When the information receiving unit 30 receives the still image storage instruction, the information receiving unit 30 outputs the still image storage instruction to the processing unit 34. When the still image storage instruction is output from the information receiving unit 30, the processing unit 34 stores one or more still images generated by the imaging element 22 in the memory unit 31. One instruction may indicate both a still image acquisition instruction and a still image storage instruction.

After the still image is stored in the memory unit 31, the user inputs an instruction to execute a measurement to the endoscope device 1a. When the information receiving unit 30 receives the instruction to execute a measurement, the information receiving unit 30 outputs the instruction to execute a measurement to the processing unit 34. When the instruction to execute a measurement is output from the information receiving unit 30, the processing unit 34 executes a measurement by using the still image stored in the memory unit 31. The processing unit 34 measures the size and depth of the subject shown in the still image.

The user inputs a label attachment instruction to the endoscope device 1a at any timing during video recording. For example, the label indicates the type of subject shown in the image. The types are "surface scratch," "chip," and "hole." When the information receiving unit 30 receives a label attachment instruction, the information receiving unit 30 outputs the label attachment instruction to the processing unit 34. When the label attachment instruction is output from the information receiving unit 30, the processing unit 34 adds the label to a frame of the video. When the user specifies a label, playback of the video starts from the frame to which the label is attached.

When the insertion portion 2 approaches the examination location, the user bends the insertion portion 2 to the bending portion 21 by inputting a bending instruction to the endoscope device 1a. This allows the user to adjust the composition of the image to be captured. The user then inputs a setting change instruction to the endoscope device 1a in order to obtain an image that is easy to see. The setting change instruction indicates an instruction to change the zoom (angle of view) setting or the brightness setting. When the information receiving unit 30 receives the setting change instruction, the information receiving unit 30 outputs the setting change instruction to the image sensor 22. The image sensor 22 changes the zoom or brightness setting.

When a user inputs an instruction to attach information into the endoscope device 1a, any information is attached to the still image. For example, the information indicates content such as "Scar 1 at inspection location 1." A label registered in advance in the endoscope device 1a may be attached to the still image as a memo. The label may be "surface scratch," "chip," or "hole." When the information receiving unit 30 receives an instruction to attach information, the information receiving unit 30 outputs the instruction to attach information to the processing unit 34. When the instruction to attach information is output from the information receiving unit 30, the processing unit 34 attaches the information to the still image stored in the memory unit 31.

The information attached to the still image may be an evaluation result of the area shown on the subject. The evaluation result corresponds to the size and depth of the wound shown in the still image. For example, the evaluation result indicates wound rank 1 or wound rank 2. The size and depth of the wound are measured by using the measurement function. The depth of the wound indicates the distance from the tip 20 of the insertion portion 2 to the wound. The processing unit 34 may automatically generate the evaluation result. For example, if the size of the wound is 5 mm or more, the processing unit 34 may generate evaluation information corresponding to wound rank 1. The user may specify a wound rank based on the automatically measured size or depth of the wound. The processing unit 34 may generate evaluation information corresponding to the wound rank specified by the user.

Importance is assigned to the operation information. Figure 8 shows the importance assigned to the operation information. The importance is indicated by a number. The higher the number indicating the importance, the higher the importance.

The user performs the examination in a predetermined order. Typically, the user inputs the following instructions to the endoscope device 1a in this order: a label attachment instruction, a bending instruction, a setting change instruction, a still image acquisition instruction, a still image storage instruction, an information attachment instruction for attaching a note to the still image, a measurement instruction, and an information attachment instruction for attaching an evaluation result to the still image. The examination procedure includes operations related to these instructions. When a later operation in the procedure is performed, the user is more likely to focus on an area in the image.

The video editing unit 33 sets a first section for each piece of operation information. Since two or more pieces of operation information are used, two or more first sections are set. The video editing unit 33 executes step S107c shown in FIG. 4 for each first section. At this time, the video editing unit 33 adjusts the number of frames in each first section so that the number of frames corresponds to the importance of the operation information.

The higher the importance, the greater the number of frames after adjustment. The playback speed of the video in the first section including the first timing is slower than the playback speed of the video in the first section including the second timing. The first timing indicates the timing when a first operation is performed. The second timing indicates the timing when a second operation, which has a lower importance than the first operation, is performed.

The video editing unit 33 does not need to use all of the above operation information. The video editing unit 33 may use only one of the two or more types of operation information described above. The video editing unit 33 may use two or more of the two or more types of operation information described above.

Each aspect of the present invention may include the following modified examples. The operation information includes at least one of an instruction to obtain a still image, an instruction to save the still image, an instruction to perform a measurement, an instruction to bend the insertion portion 2, an instruction to change the image processing settings, an instruction to attach a label to a moving image, and an instruction to attach information to a still image. The information attached to the still image includes a note and a scratch evaluation result.

In the first modification of the first embodiment, the video editing unit 33 adjusts the number of frames in the first section depending on the type of operation information. Therefore, the endoscope device 1a can allow the user to easily check important parts during video playback.

(Second Modification of the First Embodiment)
A second modification of the first embodiment of the present invention will be described by using the endoscope device 1 shown in FIG.

The video editing unit 33 divides the second section into two or more subsections. Alternatively, the video editing unit 33 sets one subsection in the entire second section. The video editing unit 33 adjusts the number of frames in each subsection based on the number of frames per unit length of the insertion length. If the number of frames per unit length of the insertion length in a subsection is greater than a predetermined value, the video editing unit 33 thins out at least one frame in that subsection. If the number of frames per unit length of the insertion length in a subsection is equal to or less than a predetermined value, the video editing unit 33 adds at least one frame to that subsection.

For example, the predetermined value is stored in advance in the storage unit 31. When the video editing process is executed, the user may input the predetermined value to the endoscope device 1 by operating the operation unit 4.

The video editing process in the second modified example of the first embodiment will be described using FIG. 9. FIG. 9 shows the procedure of the video editing process. The video editing process shown in FIG. 9 is executed in step S107 shown in FIG. 3. The description of the same process as that shown in FIG. 4 will be omitted.

After step S107a, the video editing unit 33 selects one subsection in the second section (step S107d).

After step S107d, the video editing unit 33 determines whether the number of frames per unit length of the insertion length in the partial section selected in step S107d is greater than a predetermined value (step S107e).

If the video editing unit 33 determines in step S107e that the number of frames per unit length of the insertion length is greater than a predetermined value, the video editing unit 33 thins out frames in the partial section selected in step S107d (step S107f).

If the video editing unit 33 determines in step S107e that the number of frames per unit length of the insertion length is equal to or less than a predetermined value, the video editing unit 33 adds frames to the partial section selected in step S107d (step S107g). At this time, the video editing unit 33 may duplicate one or more frames in the partial section and add the one or more frames to the partial section. The video editing unit 33 may also interpolate frames in the partial section based on two or more frames in the partial section. Even when frames are added to the partial section, the number of frames in the entire second section is reduced.

After step S107f or step S107g, the video editing unit 33 determines whether processing has been completed for all partial sections (step S107h). If neither step S107f nor step S107g has been performed for one or more partial sections, step S107d is performed. If step S107f or step S107g has been performed for all partial sections, step S107c is performed.

When the video editing unit 33 adds frames to one or more subsections of the second section, the video editing unit 33 thins out frames in one or more other subsections of the second section. To reduce the number of frames in the entire second section through adjusting the number of frames, the number of frames added is smaller than the number of frames thinned out.

The video editing unit 33 may determine whether the number of frames in the entire second section has decreased through the adjustment of the number of frames in the second section. If the number of frames in the entire second section has increased, the video editing unit 33 may decrease the predetermined value used in step S107e and repeat the adjustment of the number of frames in the second section.

In the second modified example of the first embodiment, the endoscope device 1a shown in FIG. 7 may be used. The video editing unit 33 may adjust the number of frames in the first section by using the method in the first modified example of the first embodiment.

Each aspect of the present invention may include the following variations. The second section includes one or more subsections. If the number of frames per unit length of the insertion length in at least one of the one or more subsections is greater than a predetermined value, the video editing unit 33 thins out at least one frame in at least one of the one or more subsections.

Each aspect of the present invention may include the following variations. The second section includes two or more subsections. If the number of frames per unit length of the insertion length in at least one of the two or more subsections is equal to or less than a predetermined value, the video editing unit 33 adds at least one frame to at least one of the two or more subsections.

In the second modified example of the first embodiment, the video editing unit 33 adjusts the number of frames in a partial section of the second section. This allows the video editing unit 33 to keep the speed at which the insertion portion 2 is inserted into the subject constant during video playback.

(Third Modification of the First Embodiment)
A third modification of the first embodiment of the present invention will be described by using the endoscope device 1 shown in FIG.

During an examination, the user may fail to insert the insertion portion 2 into the subject. In that case, the user pulls back the insertion portion 2 slightly and advances it again. Alternatively, the user pulls out the insertion portion 2 from the subject and reinserts it into the subject. Therefore, the insertion portion 2 may pass through the same location in the subject more than once. Viewers of the video do not need to view the same location multiple times. In a third modified example of the first embodiment, the endoscope device 1 reduces the frames generated through multiple captures.

The subject may have a branched structure. Therefore, the insertion length when the tip 20 is at a first location in the subject may be the same as the insertion length when the tip 20 is at a second location in the subject. The first location is captured in the first frame, and the second location is captured in the second frame. The insertion length calculated from the first frame is the same as the insertion length calculated from the second frame. It is difficult for the video editing unit 33 to distinguish between the first frame and the second frame based on the insertion length.

The video editing unit 33 determines the position of the tip 20 based on the coordinate information added to each frame of the video. The coordinate information of the first location is different from the coordinate information of the second location. Therefore, the video editing unit 33 can distinguish between the first frame and the second frame. The video editing unit 33 deletes a portion of two or more frames included in the video so that the trajectory of the change in the position of the tip 20 is the shortest.

The video editing process in the third modified example of the first embodiment will be described using FIG. 10. FIG. 10 shows the procedure of the video editing process. The video editing process shown in FIG. 10 is executed in step S107 shown in FIG. 3. The description of the same process as that shown in FIG. 4 will be omitted.

The video editing unit 33 deletes frames so that the trajectory of the change in the position of the tip 20 is the shortest (step S107i). After step S107i, step S107a is executed.

The order of steps S107i and S107a is not limited to the order shown in FIG. 10. Step S107a may be executed before step S107i is executed.

Figures 11 and 12 show the processing in step S107i. Insertion unit 2 is inserted to the right in Figures 11 and 12. Line L21 in Figure 11 shows the trajectory of movement of tip 20 in the video before step S107i is executed. Line L22 in Figure 12 shows the trajectory of movement of tip 20 in the video after step S107i is executed. The trajectory shown by line L21 or line L22 is equivalent to the trajectory of the change in position of image sensor 22. Multiple marks on line L21 and line L22 show the position of tip 20 when image sensor 22 generated the frame.

In the region indicated by line L23 in FIG. 11, the insertion portion 2 passes through the same location in the subject twice. The video editing unit 33 deletes frames so that the trajectory of the change in position of the tip 20 from position P21 to position P22 is the shortest. At this time, the video editing unit 33 deletes frames generated in the region indicated by line L23. After step S107i is executed, the video does not include frames generated in the region indicated by line L23.

Two or more frames in a video are connected to each other in the order in which they were generated. Before frames are deleted, the frame generated at position P23 and the frame generated at position P24 are connected to each other. In addition, the frame generated at position P25 and the frame generated at position P26 are connected to each other.

The video editing unit 33 changes the connections between frames. After the frames are deleted, the video editing unit 33 connects the frame generated at position P23 and the frame generated at position P26 to each other.

In step S107i, the video editing unit 33 does not need to consider the time at which each frame was generated. The video editing unit 33 determines the position of the tip 20 based only on the coordinate information added to each frame. The video editing unit 33 selects frames to be deleted based on the position of the tip 20. The video editing unit 33 deletes the selected frames.

The video editing unit 33 may delete one or more of the three or more frames included in the video. The trajectory of the change in the position of the image sensor 22 for two or more frames excluding the one or more deleted frames will be the shortest. The video editing unit 33 may change the connection between two or more frames excluding the one or more deleted frames.

By changing the connections between frames, the order in which frames are played may change. For example, a third frame is generated, then a fourth frame is generated. If the connections between frames are not changed, the third frame is played, then the fourth frame is played. If the connections between frames are changed, the fourth frame may be played, then the third frame.

The user may increase the amount of light or change the image processing settings while recording a video. The image processing settings are zoom (angle of view) settings or brightness settings. When the video editing unit 33 changes the connections between frames, the brightness of the images may differ significantly between the two connected frames. Alternatively, the angle of view of the images may differ significantly between the two connected frames. This may make the video difficult to view.

The video editing unit 33 may execute the following processes to suppress changes in image brightness or angle of view. For example, the video editing unit 33 selects two frames to be connected to each other based on the luminance values of the frames. Alternatively, the video editing unit 33 selects two frames to be connected to each other based on the setting values of image processing.

In the third modified example of the first embodiment, the endoscope device 1a shown in FIG. 7 may be used. The video editing unit 33 may adjust the number of frames in the first section by using the method in the first modified example of the first embodiment.

The video editing unit 33 may execute steps S107d to S107h shown in FIG. 9 instead of step S107b.

Each aspect of the present invention may include the following modified examples. The information generating unit 32 uses two or more frames to estimate the position of the image sensor 22 at the timing when each of the two or more frames was generated. The video editing unit 33 deletes a portion of the two or more frames so as to minimize the trajectory of the change in the position of the image sensor 22.

The video in the third modified example of the first embodiment may include three or more frames. The three or more frames may include one or more frames in the first section and two or more frames in the second section. The video editing unit 33 may delete some of the two or more frames in the second section so that the trajectory of the change in the position of the image sensor 22 in the second section is the shortest.

In the third modified example of the first embodiment, the video editing unit 33 deletes a portion of two or more frames that show the same location. Therefore, the endoscope device 1 can reduce the burden on the user when playing back a video.

Second Embodiment
A second embodiment of the present invention will be described. The endoscope device 1 shown in Fig. 2 is changed to an endoscope device 1b shown in Fig. 13. Fig. 13 shows the configuration of the endoscope device 1b. The insertion section 2 shown in Fig. 2 is changed to an insertion section 2b shown in Fig. 13. A description of the same configuration as that shown in Fig. 2 will be omitted.

The insertion section 2 has a bending section 21, an imaging element 22, and a sensor 23. The sensor 23 is disposed at the tip 20 of the insertion section 2. For example, the sensor 23 is an acceleration sensor or a gyro sensor (angular velocity sensor). The acceleration sensor detects the acceleration of the tip 20. The sensor 23 may be a one-axis acceleration sensor. The one-axis acceleration sensor detects the acceleration in the longitudinal direction of the insertion section 2. The gyro sensor detects the angular velocity of the tip 20. The sensor 23 may be a combination of an acceleration sensor and a gyro sensor. The sensor 23 may be a six-axis inertial sensor that detects three-axis acceleration and three-axis angular velocity.

The sensor 23 measures a physical quantity and outputs sensor information indicating that physical quantity. The physical quantity is acceleration or angular velocity. The sensor information indicates the acceleration of the tip 20 or the angular velocity of the tip 20.

When the information receiving unit 30 receives operation information other than instructions related to video recording, the information receiving unit 30 adds the operation information to a frame of the video. The information receiving unit 30 also adds sensor information output from the sensor 23 to the frame.

An example will be described in which the information generating unit 32 uses sensor information indicating acceleration. The information generating unit 32 calculates the insertion length (first tentative insertion length) by using the method in the first embodiment. At this time, the information generating unit 32 detects feature points from the image and uses information on the feature points. The information generating unit 32 also calculates the insertion length (second tentative insertion length) by using the sensor information. Specifically, the information generating unit 32 calculates the amount of change in the position of the tip 20 based on the acceleration. The information generating unit 32 calculates the second tentative insertion length based on the amount of change.

The information generating unit 32 calculates the final insertion length based on the first tentative insertion length and the second tentative insertion length. For example, the information generating unit 32 calculates a weighted average of the first tentative insertion length and the second tentative insertion length, and uses the calculated weighted average as the final insertion length. The information generating unit 32 sets the weight of the weighted average according to the number of feature points. When the number of feature points is sufficiently large, the information generating unit 32 sets the weight of the first tentative insertion length to be greater than the weight of the second tentative insertion length. When the number of feature points is small, the information generating unit 32 sets the weight of the second tentative insertion length to be greater than the weight of the first tentative insertion length.

By performing the above process, the information generating unit 32 estimates the insertion length at the timing when each of the two or more frames is generated. The information generating unit 32 adds position information including the calculated insertion length to each frame.

Using Figures 14 to 17, an example in which the information generating unit 32 uses sensor information indicating angular velocity will be described. Figure 14 shows states ST11, ST12, and ST13 when the insertion portion 2 is inserted into the subject IT12. The insertion portion 2 is inserted in a direction DR11 shown in Figure 14, and is withdrawn in a direction DR12 shown in Figure 14.

Figure 15 shows frames F11, F12, and F13 generated by the imaging element 22 in state ST11. Figure 16 shows frames F14, F15, and F16 generated by the imaging element 22 in state ST12. Figure 17 shows frames F17, F18, and F19 generated by the imaging element 22 in state ST13. Regions RG11 and RG12 in Figures 14 to 17 typically show the regions of the subject IT12.

In state ST11, the insertion portion 2 is not bent. Regions RG11 and RG12 at the rear of the subject IT12 are captured in frames F11, F12, and F13. When the tip 20 of the insertion portion 2 is in the position shown in FIG. 14, the image sensor 22 generates frame F11. After the tip 20 moves in direction DR11 from the position shown in FIG. 14, the image sensor 22 generates frame F12. After the tip 20 moves in direction DR12 from the position shown in FIG. 14, the image sensor 22 generates frame F13.

In state ST12, the insertion portion 2 is curved. Regions RG11 and RG12 on the inner surface of the subject IT12 are captured in frames F14, F15, and F16. When the tip 20 of the insertion portion 2 is in the position shown in FIG. 14, the image sensor 22 generates frame F14. After the tip 20 moves in direction DR11 from the position shown in FIG. 14, the image sensor 22 generates frame F15. After the tip 20 moves in direction DR12 from the position shown in FIG. 14, the image sensor 22 generates frame F16.

In state ST13, the insertion portion 2 is curved. Regions RG11 and RG12 on the inner surface of the subject IT12 are captured in frames F17, F18, and F19. When the tip 20 of the insertion portion 2 is in the position shown in FIG. 14, the image sensor 22 generates frame F17. After the tip 20 moves in direction DR11 from the position shown in FIG. 14, the image sensor 22 generates frame F18. After the tip 20 moves in direction DR12 from the position shown in FIG. 14, the image sensor 22 generates frame F19.

When the insertion length changes, the frame generated by the image sensor 22 changes according to the direction of the tip 20. For example, when the tip 20 moves in the direction DR11 in state ST11, region RG11 moves leftward and region RG12 moves rightward, as shown in frames F11 and F12. When the tip 20 moves in the direction DR11 in state ST12, region RG11 and region RG12 move rightward, as shown in frames F14 and F15. On the other hand, when the tip 20 moves in the direction DR11 in state ST13, region RG11 and region RG12 move leftward, as shown in frames F17 and F18. As described above, the relationship between the movement direction of the tip 20 and the movement direction of the region in the frame changes according to the direction of the tip 20.

The information generating unit 32 calculates the insertion length (provisional insertion length) by using the method in the first embodiment. The information generating unit 32 corrects the provisional insertion length by using the angular acceleration indicated by the sensor information. As a result, the information generating unit 32 calculates the final insertion length.

The video recording process in the second embodiment will be described using FIG. 18. FIG. 18 shows the procedure for the video recording process. The description of the same process as that shown in FIG. 3 will be omitted.

After step S102, the information receiving unit 30 adds the operation information and sensor information to the frame stored in the storage unit 31 in step S101. For example, the information receiving unit 30 embeds the operation information and sensor information in the frame (step S110). After step S110, step S104 is executed.

After step S105, the information generating unit 32 calculates the insertion length based on the video file and the sensor information. The information generating unit 32 generates position information including coordinate information and length information (insertion length), and adds the generated position information to each frame. For example, the information generating unit 32 embeds the position information in the frame (step S111). After step S111, step S107 is executed.

Instead of the control unit 3, a control unit 3a shown in FIG. 7 may be used. The video editing unit 33 may adjust the number of frames in the first section by using the method in the first modified example of the first embodiment.

The video editing unit 33 may execute the video editing process shown in FIG. 9 or FIG. 10 instead of the video editing process shown in FIG. 4.

If halation or the like occurs in the image, it is difficult for the information generating unit 32 to detect feature points from the image. In that case, it is difficult for the information generating unit 32 to estimate the position of the tip 20 of the insertion unit 2 by using the image.

In the second embodiment, the information generating unit 32 calculates the insertion length by using sensor information in addition to the video file. This improves the accuracy of estimating the insertion length.

(First Modification of the Second Embodiment)
A first modified example of the second embodiment of the present invention will be described. The endoscope device 1 shown in Fig. 2 is changed to an endoscope device 1c shown in Fig. 19. Fig. 19 shows the configuration of the endoscope device 1c. The control unit 3 shown in Fig. 2 is changed to a control unit 3c shown in Fig. 19. A description of the same configuration as that shown in Fig. 2 will be omitted.

The control unit 3c has an information receiving unit 30, a memory unit 31, an information generating unit 32, a video editing unit 33, and a sensor 35. The sensor 35 is an imaging element (image sensor) or a rotary encoder.

An example will be described in which the sensor 35 is an imaging element. Predetermined information indicating the insertion length is displayed on the surface of the insertion portion 2. The predetermined information is a scale, a number, a mark, or the like. The imaging element generates an image in which the predetermined information is displayed. The imaging element outputs the generated image as sensor information. The information generation unit 32 detects the insertion length by analyzing the image.

An example in which the sensor 35 is a rotary encoder will be described. Two rollers are arranged to sandwich the insertion portion 2. The two rollers are in contact with the insertion portion 2. When the insertion portion 2 moves, the two rollers rotate. The rotary encoder detects the amount of rotation of at least one of the two rollers to detect the insertion length. The rotary encoder outputs sensor information indicating the detected insertion length. The information generating unit 32 detects the insertion length (second tentative insertion length) based on the sensor information.

The procedure for the video recording process in the first variant of the second embodiment is the same as that shown in FIG. 18.

The insertion section 2 may be changed to the insertion section 2b shown in FIG. 13. That is, the endoscope device 1c may have a sensor 23 and a sensor 35. The sensor arranged in the endoscope device 1c may be any one of an acceleration sensor, a gyro sensor, a six-axis inertial sensor, an imaging element, and a rotary encoder. Alternatively, the sensor arranged in the endoscope device 1c may be a combination of two or more sensors.

Each aspect of the present invention may include the following modifications. The information generating unit 32 estimates the position of the imaging element 22 by using sensor information in addition to two or more frames. The sensor information includes at least one of first sensor information generated by a first sensor, second sensor information generated by a second sensor, and third sensor information generated by a third sensor. The first sensor information indicates an insertion length. The second sensor information indicates an acceleration of the tip 20 of the insertion portion 2. The third sensor information indicates an angular velocity of the tip 20.

There are cases where the bending section 21 bends the insertion section 2 without changing the insertion length. As the insertion section 2 bends, the frames generated by the image sensor 22 change. In this case, the insertion length calculated by the information generating section 32 may change.

In the first modification of the second embodiment, the information generating unit 32 calculates the insertion length by using sensor information indicating the insertion length in addition to the video file. This improves the accuracy of estimating the insertion length.

(Second Modification of the Second Embodiment)
A second modification of the second embodiment of the present invention will now be described. In the second modification of the second embodiment, an endoscope apparatus having at least one of the sensor 23 shown in Fig. 13 and the sensor 35 shown in Fig. 19 is used.

The information generating unit 32 calculates or detects the insertion length by using sensor information generated by at least one of the sensors 23 and 35, without using video frames.

The video recording process in the second modified example of the second embodiment will be described using FIG. 20. FIG. 20 shows the procedure for the video recording process. The description of the same process as that shown in FIG. 3 will be omitted.

After step S102, the information generating unit 32 calculates or detects the insertion length by using the sensor information. The information generating unit 32 generates position information including length information (insertion length) and adds the generated position information together with operation information to each frame. For example, the information generating unit 32 embeds the position information and operation information in the frame (step S120). After step S120, step S104 is executed.

The information generating unit 32 may use the sensor information output from the sensor 23 to calculate the position of the tip 20 of the insertion unit 2 when each of the two or more frames is generated. The information generating unit 32 may generate position information including coordinate information indicating the calculated position and length information.

After step S105, step S106 shown in FIG. 3 is not executed, and step S107 is executed.

Instead of the control unit 3 shown in FIG. 13, the control unit 3a shown in FIG. 7 may be used. Alternatively, the control unit 3c shown in FIG. 19 may have the processing unit 34 shown in FIG. 7. The video editing unit 33 may adjust the number of frames in the first section by using the method in the first modified example of the first embodiment.

The video editing unit 33 may execute the video editing process shown in FIG. 9 or FIG. 10 instead of the video editing process shown in FIG. 4.

Each aspect of the present invention may include the following modifications: The information generating unit 32 generates length information based on sensor information generated by a sensor. The sensor detects the length of the insertion portion 2 inserted into the subject when each of the two or more frames is generated.

The state of the subject in the frame affects the accuracy of self-location estimation using two or more frames of a video. In a second variant of the second embodiment, the information generator 32 can calculate or detect the insertion length regardless of the state of the video frames.

Third Embodiment
A third embodiment of the present invention will be described. Fig. 21 shows the configuration of an endoscope system 10 according to the third embodiment. The endoscope system 10 shown in Fig. 21 has an endoscope device 1d and a control device 6. Description of the same configuration as that shown in Fig. 2 will be omitted.

The endoscope device 1d has an insertion section 2 and a control section 3d. The insertion section 2 shown in FIG. 21 is the same as the insertion section 2 shown in FIG. 2. The control section 3d has an information receiving section 30, a memory section 31, and a transmission section 36.

For example, the control device 6 is a cloud server. The control device 6 has a receiving unit 60, a memory unit 61, an information generating unit 62, and a video editing unit 63.

The transmitting unit 36 and the receiving unit 60 are each a communication device having a communication circuit. The transmitting unit 36 and the receiving unit 60 perform communication. The transmitting unit 36 and the receiving unit 60 may be connected to each other by a cable. The transmitting unit 36 and the receiving unit 60 may perform wireless communication. The transmitting unit 36 transmits a video file stored in the memory unit 31 to the receiving unit 60. The receiving unit 60 receives the video file transmitted by the transmitting unit 36.

The storage unit 61 is a volatile or non-volatile recording medium. For example, the storage unit 61 is at least one of a RAM, a DRAM, an SRAM, a ROM, an EPROM, an EEPROM, and a flash memory. The storage unit 61 stores video files received by the receiving unit 60. The information generating unit 62 has a function similar to that of the information generating unit 32 shown in FIG. 2. The video editing unit 63 has a function similar to that of the video editing unit 33 shown in FIG. 2.

The information generating unit 62 and the video editing unit 63 may be configured with at least one of a processor and a logic circuit. The information generating unit 62 and the video editing unit 63 may include one or more processors. The information generating unit 62 and the video editing unit 63 may include one or more logic circuits.

The video recording process in the third embodiment will be described using FIG. 22. FIG. 22 shows the procedure for the video recording process executed by the endoscope device 1d. The description of the same process as that shown in FIG. 3 will be omitted.

After step S105, the transmission unit 36 transmits the video file stored in the storage unit 31 to the reception unit 60 (step S130). When step S130 is executed, the video recording process ends.

The process executed by the control device 6 will be explained using FIG. 23. FIG. 23 shows the procedure of the process executed by the control device 6.

The receiving unit 60 receives the video file transmitted by the transmitting unit 36 (step S200). The video file received by the receiving unit 60 is stored in the memory unit 61.

After step S200, the information generating unit 62 performs self-location estimation by using two or more frames included in the video file. As a result, the information generating unit 62 generates location information for each of the two or more frames. The information generating unit 62 adds the generated location information to each frame. For example, the information generating unit 62 embeds the location information in the frame (step S201).

After step S201, the video editing unit 63 executes a video editing process (step S202). The video editing process in step S202 is the same as the video editing process shown in FIG. 4. When step S202 ends, the process shown in FIG. 23 ends.

The control unit 3d shown in FIG. 21 may have the processing unit 34 shown in FIG. 7. The video editing unit 63 may adjust the number of frames in the first section by using the method in the first modified example of the first embodiment.

The video editing unit 63 may execute the video editing process shown in FIG. 9 or FIG. 10 instead of the video editing process shown in FIG. 4.

The endoscope device 1d may have at least one of the sensor 23 shown in FIG. 13 and the sensor 35 shown in FIG. 19. In the video recording process shown in FIG. 22, steps S110 and S111 shown in FIG. 18 may be executed. In the video recording process shown in FIG. 22, step S120 shown in FIG. 20 may be executed.

In the third embodiment, the endoscope system 10 can reduce the burden on the user when playing back a video. Also, in the third embodiment, the endoscope system 10 can allow the user to easily check important points when playing back a video.

Generally, image processing for self-position estimation requires a large amount of computational resources. If that image processing is performed in an endoscope device, the current consumption of the endoscope device increases, or the endoscope device becomes larger. Because the control device 6 performs that image processing, the occurrence of such problems is suppressed.

(Modification of the third embodiment)
A modified example of the third embodiment of the present invention will be described. In the modified example of the third embodiment, the endoscope device 1d transmits frames generated by the image sensor 22 to the control device 6 in real time. In the modified example of the third embodiment, the endoscope device 1d does not need to have the storage unit 31.

The video recording process in the modified example of the third embodiment will be described using Figures 24 and 25. Figure 24 shows the procedure for the video recording process executed by the endoscope device 1d. A description of the same process as that shown in Figure 3 will be omitted.

After step S103, the transmitting unit 36 transmits the frame to the receiving unit 60 (step S140). After step S140, step S104 is executed. If the information receiving unit 30 receives an instruction to end recording, the video recording process ends.

Figure 25 shows the procedure for video recording processing executed by the control device 6. Explanation of the same processing as that shown in Figure 23 will be omitted.

The receiving unit 60 receives the frame transmitted by the transmitting unit 36 (step S210). The frame received by the receiving unit 60 is stored in the memory unit 61.

Step S210 is repeated until the receiving unit 60 receives all the frames transmitted from the transmitting unit 36. When the receiving unit 60 receives all the frames transmitted from the transmitting unit 36 (step S211), the video editing unit 63 generates a video file including the frames stored in the storage unit 61 (step S212). The video file is stored in the storage unit 61. After step S212, step S202 is executed.

In a modification of the third embodiment, the endoscope device 1d does not need to store video files. This saves storage capacity in the endoscope device 1d.

(Fourth embodiment)
A fourth embodiment of the present invention will be described. The endoscope system 10 shown in Fig. 21 is changed to an endoscope system 10e shown in Fig. 26. Fig. 26 shows the configuration of the endoscope system 10e. The description of the same configuration as that shown in Fig. 21 will be omitted.

The endoscope device 1d shown in FIG. 21 is changed to an endoscope device 1e shown in FIG. 26. The control unit 3d shown in FIG. 21 is changed to a control unit 3e shown in FIG. 26. The control unit 3e has an information receiving unit 30, a memory unit 31, a transmitting unit 36, and a receiving unit 37.

The control device 6 shown in FIG. 21 is changed to a control device 6e shown in FIG. 26. The control device 6e has a receiving unit 60, a storage unit 61, an information generating unit 62, a video editing unit 63, an information accepting unit 64, and a transmitting unit 65.

The information receiving unit 64 has the same function as the information receiving unit 30 shown in FIG. 1. The control device 6e has an operation unit not shown in FIG. 26. A user inputs operation information to the control device 6e by operating the operation unit of the control device 6. The operation unit of the control device 6 outputs the information input by the user. The information receiving unit 64 receives the operation information. The information receiving unit 64 outputs the operation information to the transmission unit 65.

The transmitting unit 65 and the receiving unit 37 are each a communication device having a communication circuit. The transmitting unit 65 and the receiving unit 37 perform communication. The transmitting unit 65 and the receiving unit 37 may be connected to each other by a cable. The transmitting unit 65 and the receiving unit 37 may perform wireless communication. The transmitting unit 65 transmits operation information output from the information receiving unit 64 to the receiving unit 37. The receiving unit 37 receives the operation information transmitted by the transmitting unit 65. The receiving unit 37 outputs the received operation information to the information receiving unit 30.

The transmitter 36 and the receiver 37 may be integrated into a single communication device. The transmitter 65 and the receiver 60 may be integrated into a single communication device.

The information receiving unit 30 receives the operation information output from the receiving unit 37. After receiving the operation information, the information receiving unit 30 performs the same operation as in the first embodiment.

The endoscope device 1e executes the process shown in FIG. 22, and the control device 6e executes the process shown in FIG. 23. Alternatively, the endoscope device 1e executes the process shown in FIG. 24, and the control device 6e executes the process shown in FIG. 25.

The information receiving unit 30 does not need to add the operation information to the frame. The information receiving unit 64 may add the operation information to the frame stored in the storage unit 61.

The fourth embodiment provides the same effect as the third embodiment.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their variations. Addition, omission, substitution, and other modifications of the configuration are possible without departing from the spirit of the present invention. Furthermore, the present invention is not limited to the above description, but is limited only by the scope of the attached claims.

REFERENCE SIGNS LIST 1, 1a, 1b, 1c, 1d, 1e Endoscope device 2, 2b Insertion section 3, 3a, 3c, 3d, 3e Control section 4 Operation section 5 Display section 6, 6e Control device 10, 10e Endoscope system 20 Tip 21 Bending section 22 Imaging element 23, 35 Sensor 30, 64 Information receiving section 31, 61 Storage section 32, 62 Information generating section 33, 63 Video editing section 34 Processing section 36, 65 Transmission section 37, 60 Reception section

Claims (14)

an insertion section that can be inserted into a subject, the insertion section having an image sensor that generates a moving image including two or more frames;
A control unit;
having
The control unit is
an information receiving unit that receives operation information indicating the content of an operation performed by a user;
an information generating unit that generates length information indicating a length of a portion of the insertion portion that is inserted into the subject when each of the two or more frames is generated;
A video editing unit that edits the video;
having
the video editing unit classifies the two or more frames into one or more frames in a first section and one or more frames in a second section different from the first section based on the operation information, the first section including a timing at which a user performed a predetermined operation;
the video editing unit reduces a number of frames per unit length in the second section based on the length information;
an endoscope apparatus, wherein the video editing unit adjusts the number of frames in the first section so that a first number of frames is greater than a second number of frames, the first number of frames indicating the number of frames per unit length in the first section, and the second number of frames indicating the number of frames per unit length in the second section. The endoscope device according to claim 1 , wherein the video editing unit adjusts the number of frames in the second section so that an apparent speed at which the insertion section is inserted into the subject during playback of the video in the second section is constant. The endoscope device according to claim 1 , wherein the information generation unit estimates a position of the imaging element at a timing when each of the two or more frames is generated by using the two or more frames, and estimates the length based on the estimated position. 4. The endoscope device according to claim 3, wherein the information generating unit estimates the position by using sensor information in addition to the two or more frames, the sensor information including at least one of first sensor information generated by a first sensor, second sensor information generated by a second sensor, and third sensor information generated by a third sensor, the first sensor information indicating the length, the second sensor information indicating an acceleration of a tip of the insertion portion, and the third sensor information indicating an angular velocity of the tip. The endoscope apparatus according to claim 3 , wherein the video editing unit deletes a part of the two or more frames so as to minimize a trajectory of the change in position. The endoscope apparatus according to claim 1 , wherein the information generating unit generates the length information based on sensor information generated by a sensor that detects the length. The endoscopic device according to claim 1, wherein the operation information includes at least one of an instruction to acquire a still image, an instruction to save a still image, an instruction to perform a measurement, an instruction to bend the insertion portion, an instruction to change image processing settings, an instruction to attach a label to the video, and an instruction to attach information to a still image. The endoscopic device of claim 7 , wherein the information includes notes and wound assessment results. the second interval includes one or more subintervals;
The endoscope device according to claim 1 , wherein when the number of frames per unit length in at least one of the one or more partial sections is greater than a predetermined value, the video editing unit thins out at least one frame in at least one of the one or more partial sections. the second interval includes two or more subintervals;
The endoscope device according to claim 9 , wherein when the number of frames per unit length in at least one of the two or more partial sections is equal to or less than a predetermined value, the video editing unit adds at least one frame to at least one of the two or more partial sections. The endoscope device according to claim 1, wherein the one or more frames in the first section include one frame generated at the timing when the user performs the specified operation, one or more frames generated before the timing, and one or more frames generated after the timing. An endoscope device and a control device are provided.
The endoscope device includes:
an insertion section that can be inserted into a subject, the insertion section having an image sensor that generates a moving image including two or more frames;
an information receiving unit that receives operation information indicating the content of an operation performed by a user;
A transmission unit that transmits the video to the control device;
having
The control device includes:
a receiving unit that receives the video from the endoscope device;
an information generating unit that generates length information indicating a length of a portion of the insertion portion that is inserted into the subject when each of the two or more frames is generated;
A video editing unit that edits the video;
having
the video editing unit classifies the two or more frames into one or more frames in a first section and one or more frames in a second section different from the first section based on the operation information, the first section including a timing at which a user performed a predetermined operation;
the video editing unit reduces a number of frames per unit length in the second section based on the length information;
an endoscope system, wherein the video editing unit adjusts the number of frames in the first section so that a first number of frames is greater than a second number of frames, the first number of frames indicating the number of frames per unit length in the first section, and the second number of frames indicating the number of frames per unit length in the second section. 1. A method for operating an endoscope apparatus including an insertion section that can be inserted into a subject, the insertion section having an image sensor that generates a moving image including two or more frames, and a control section, the method comprising the steps of:
The method of operation includes:
an information receiving step in which an information receiving unit of the control unit receives operation information indicating the content of an operation performed by a user;
an information generating step in which an information generating unit of the control unit generates length information indicating a length of a portion of the insertion portion inserted into the subject when each of the two or more frames is generated;
a video editing step in which a video editing unit of the control unit edits the video;
having
The video editing step includes a classification step in which the video editing unit classifies the two or more frames into one or more frames in a first section and one or more frames in a second section different from the first section based on the operation information, the first section including a timing at which a user performed a predetermined operation;
a second section editing step of reducing a number of frames per unit length in the second section based on the length information by the video editing unit;
a first section editing step of adjusting a number of frames in the first section so that a first number of frames is greater than a second number of frames, the first number of frames indicating a number of frames per unit length in the first section, and the second number of frames indicating a number of frames per unit length in the second section;
A method for operating an endoscope apparatus comprising the steps of: A computer of an endoscope apparatus including an insertion section that can be inserted into a subject and has an image sensor that generates a moving image including two or more frames, and a control section,
an information receiving step of receiving operation information indicating the content of an operation performed by a user;
an information generating step of generating length information indicating a length of a portion of the insertion section inserted into the subject when each of the two or more frames is generated;
a video editing step of editing the video;
A program for executing
the video editing step includes a classification step of classifying the two or more frames into one or more frames in a first section and one or more frames in a second section different from the first section based on the operation information, the first section including a timing at which a user performed a predetermined operation;
a second section editing step of reducing the number of frames per unit length in the second section based on the length information;
a first section editing step of adjusting a number of frames in the first section so that a first number of frames is greater than a second number of frames, the first number of frames indicating a number of frames per unit length in the first section, and the second number of frames indicating a number of frames per unit length in the second section;
A program having:

Images