JP7556715B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

In the medical field, patients are photographed using medical imaging devices such as X-ray CT (Computed Tomography) devices, MRI (Magnetic Resonance Imaging) devices, and PET (Positron Emission Tomography) devices. By observing the medical images in detail, the anatomical structure and functional information of various types of organs in the patient's body can be obtained, and this information can be used for diagnosis and treatment.

Among the various types of organs that make up the human body, there are some that move relative to the surrounding tissues. For example, the lungs move due to respiratory movement, and the heart moves to circulate blood throughout the body. It is known that even the same organ moves relative to the surrounding tissues (direction and amount of movement) differently depending on the position inside or on the surface of the organ (hereinafter referred to as intra-organ position) due to the structure, presence or absence of lesions, etc. Here, there is a demand from users (doctors, etc.) to visualize the difference in the direction and amount of movement (hereinafter referred to as movement information) of the target organ depending on the intra-organ position (i.e., visualize the distribution of the direction and amount of movement) from medical images. This is to recognize intra-organ positions that have abnormal movement, and to help discover diseases and lesions. For example, there is a demand to identify the adhesion position between the pleura on the surface of the lung from medical images by visualizing the movement information due to the respiratory movement of the lung at each position on the surface of the lung.

Patent Document 1 discloses a technique for performing image registration between time-series three-dimensional images (volume data) of the lungs, and calculating the amount of surface position slippage caused by respiratory movement, which is closely related to pleural adhesions on the lung surface.

JP 2016-67832 A

In medical images of moving organs taken at different times, the density values (pixel values) of the organs can vary from image to image. This is because the amount of air and blood contained in the tissue changes depending on the state of the organ, and contrast conditions (for example, the presence or absence of contrast agent and differences in the blood concentration of contrast agent) change over time. Such fluctuations (differences) in density values are problematic because they can cause failure or reduced accuracy in registering images (i.e. matching pixels together). This problem is particularly pronounced in the case of algorithms that match pixels between images based on the similarity of pixel density values.

The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a technology for performing pixel matching with high accuracy between images of a moving object.

The present disclosure includes an information processing device having an image acquisition unit that acquires a first image and a second image captured at different times of a moving object, and an image processing unit that performs an alignment process between the first image and the second image, wherein the image processing unit performs a conversion process to convert pixel values of at least one of the first image and the second image based on an estimated air content ratio of pixels so that a difference between statistics of pixel values of the object region in the first image and statistics of pixel values of the object region in the second image is reduced, and performs a first alignment process to acquire a correspondence relationship of pixels within the object region between the first image and the second image based on the pixel values converted by the conversion process.

The present disclosure includes an information processing method comprising the steps of acquiring a first image and a second image captured at different times of a moving object, and performing a registration process between the first image and the second image, wherein in the registration process, a conversion process is performed to convert pixel values of at least one of the first image and the second image based on an estimated pixel air content ratio so that a difference between statistics of pixel values of the object region in the first image and statistics of pixel values of the object region in the second image is reduced, and a first registration process is performed to acquire a correspondence relationship of pixels within the object region between the first image and the second image based on the pixel values converted by the conversion process.

The present disclosure includes a program for causing a computer to execute each step of the above information processing method.

The present invention makes it possible to perform highly accurate pixel matching between images of moving objects.

1 is a diagram showing a device configuration of an information processing system according to a first embodiment; FIG. 2 is a flowchart showing an overall processing procedure in the first embodiment. FIG. 2 is a diagram for explaining 3D CT data of a subject; FIG. 11 is a flowchart showing an overall processing procedure in a second embodiment. FIG. 5 is a flowchart showing the processing procedure of step S2040 in FIG. 4. FIG. 13 is a flowchart showing an overall processing procedure in a third embodiment.

Below, a preferred embodiment of the information processing device disclosed in this specification will be described in detail with reference to the attached drawings. However, the components described in this embodiment are merely examples, and the technical scope of the information processing device disclosed in this specification is determined by the scope of the claims, and is not limited to the individual embodiments described below. Furthermore, the disclosure of this specification is not limited to the embodiments described below, and various modifications are possible based on the spirit of the disclosure of this specification, and these are not excluded from the scope of the disclosure of this specification. In other words, configurations that combine the examples and their modifications described below are all included in the embodiments disclosed in this specification.

In this specification, the term "image" is used as a concept that encompasses both "two-dimensional image" and "three-dimensional image (also called volume data)." Time-series data consisting of multiple images captured at different times (a set of multiple two-dimensional images or a set of multiple three-dimensional images) may also be called an "image." Additionally, the term "pixel" is used as a concept that encompasses both the smallest element that constitutes a two-dimensional image and the smallest element that constitutes a three-dimensional image (also called a voxel).

[First embodiment]
The information processing system according to the first embodiment of the present invention provides a function to assist users such as doctors and engineers in medical institutions in grasping and diagnosing the adhesion state of the pleura of a subject to be examined. Specifically, the system provides a function to generate an observation image that can easily visually recognize the sliding state of the pleura, which is a type of feature value related to the movement of the subject's lungs, as lung movement information. More specifically, the system measures the movement of the pleural part (lung contour part) of the subject's lungs from time-series 3DCT data (4DCT data, 4-dimensional CT images) of the subject's lungs moving due to breathing, and visualizes the sliding state of the part as a slippage map. Note that "slip" refers to the relative movement between a moving object (such as a lung) and its surrounding area, and the "slippage map" is information that represents the distribution of the amount of slippage between the object area and the surrounding area (boundary).

In time series data obtained by photographing a moving object such as the lungs at different times, the density value (brightness value, pixel value) of the object may vary between images. For example, in the case of the lungs, the density value in the lung field differs depending on the breathing state (mainly the ventilation volume) of the subject. Since the difference in density value may lead to failure or reduced accuracy of the registration process between images, in this embodiment, the difference in the density value of the object between images is corrected before the registration process is performed. Specifically, the pixel values of one or both images are converted (corrected) so that the statistical difference in the pixel values of the object area between the two images is reduced, and the correspondence of pixels in the object area between the two images is obtained based on the converted pixel values. This allows for highly accurate pixel correspondence between images of moving objects.

Figure 1 is a diagram showing the overall configuration of an information processing system according to one embodiment of the present invention. The information processing system includes an information processing device 10, an examination image database 30, and an examination image capturing device 40, which are communicatively connected to each other via communication means. In this embodiment, the communication means is configured as a LAN (Local Area Network) 50, but may also be a WAN (Wide Area Network). In addition, the communication means may be connected via a wired connection or a wireless connection.

The examination image database 30 stores a plurality of examination images and their associated information for a plurality of patients. The examination images are medical images captured by imaging diagnostic devices such as CT and MRI, and can be two-dimensional images, three-dimensional images, or four-dimensional images, which are moving images of three-dimensional images. Each image can be in various formats, such as monochrome or color. In this embodiment, the examination image database 30 stores 4DCT data of the lungs of the subject. The examination image database 30 stores, as the associated information of the examination image, the patient name (patient ID), examination date information (the date the examination image was captured), the name of the imaging modality of the examination image, and the like. Each examination image and its associated information is assigned a unique number (examination image ID) to enable identification from others, and the information processing device 10 can read out the information based on the number.

The information processing device 10 acquires information held by the examination image database 30 via the LAN 50. The examination image acquisition unit 100 acquires the examination image of the subject captured by the examination image capture device 40 and held by the examination image database 30. The region extraction unit 110 analyzes the examination image acquired by the examination image acquisition unit 100 and extracts the lung region of the subject. The average density value calculation unit 120 calculates an average value as a statistic of density values (pixel values) of the examination image in the lung region of the subject based on the examination image acquired by the examination image acquisition unit 100 and the lung region of the subject extracted by the region extraction unit 110. The density value correction unit 130 calculates a corrected image in which the density value of the examination image acquired by the examination image acquisition unit 100 is corrected based on the average density value of the lung region of the subject calculated by the average density value calculation unit 120. The displacement field calculation unit 140 calculates a displacement field caused by the respiration of the subject based on the examination image acquired by the examination image acquisition unit 100 and the corrected image calculated by the density value correction unit 130. The map generation unit 150 generates a map of the slippage of the lung contour caused by the subject's respiration based on the lung region of the subject calculated by the region extraction unit 110 and the displacement field of the subject calculated by the displacement field calculation unit 140. The display control unit 160 controls the display of the slippage map calculated by the map generation unit 150 on the display device 60. In this embodiment, the examination image acquisition unit 100 constitutes an image acquisition unit that acquires an image of the object. In addition, the region extraction unit 110, the average density value calculation unit 120, the density value correction unit 130, the displacement field calculation unit 140, and the map generation unit 150 constitute an image processing unit that performs image registration processing and the like.

The information processing device 10 can be configured by a computer including a processor (CPU, GPU, etc.), memory (RAM, ROM, etc.), storage (HDD, SSD, etc.), communication IF, input/output IF, etc. In this case, the functions of each part shown in FIG. 1 are realized by the processor reading and executing a program stored in the storage. Note that the configuration of the information processing system shown in FIG. 1 is merely an example. For example, the information processing device 10 may have a storage unit (not shown) and may have the function of the inspection image database 30. In addition, the information processing device 10 may be configured by a general-purpose computer (personal computer, server computer, etc.) or a dedicated computer. In addition, the information processing system of FIG. 1 may be realized by implementing the necessary functions in a computer (also called a console) included in an imaging system such as a CT device or an MRI device.

Next, the overall processing procedure by the information processing device 10 in this embodiment will be described in detail with reference to FIG. 2. In addition, the following will be described as an example in which time-series 3DCT data (4DCT data, 4-dimensional CT images) are used as the examination images, but the implementation of the present invention is not limited to this. It is sufficient that there are multiple images obtained by photographing a moving object at different times. For example, as long as there are multiple 3D images (volume data) of the subject's lungs photographed at different respiratory phases, they do not have to be time-series images (moving images) photographed in a single examination. In addition, instead of CT images, MRI images or ultrasound images may be used.

(Step S1000): Acquisition of 4DCT Data In step S1000, the examination image acquisition unit 100 acquires 4DCT data of the lungs of the subject from the examination image database 30. The 4DCT data in this embodiment is time-series three-dimensional volume data, and is data of the dynamics of the subject's breathing. More specifically, 4DCT data consisting of a plurality of 3DCT data of two or more time phases including the subject's inhalation position (e.g., maximum inhalation position) and exhalation position (e.g., maximum exhalation position) is acquired. In this embodiment, a specific example will be described in which 3DCT data of three time phases is acquired. Here, the 3DCT data of each time phase is represented as I_t (1≦t≦3). In addition, in the description of this embodiment, each 3DCT data is also represented as a function that takes I_t (x, y, z) and a position in the image as arguments and returns the pixel value of the position. In addition, in this embodiment, a case will be described as an example in which 3DCT data of the inspiration phase is I_1, 3DCT data of the expiration phase is I_3, and 3DCT data of an intermediate phase is I_2.

Figures 3A to 3C are diagrams conceptually showing three-phase 3DCT data acquired by the examination image acquisition unit 100 in this embodiment. Image 200 in Figure 3A is an image of a coronal section of 3DCT data I_1, depicting the contour 204 of the lung field 202, which is the target area. Image 210 in Figure 3B is an image of a coronal section of 3DCT data I_2, and image 220 in Figure 3C is an image of a coronal section of 3DCT data I_3. The lung fields 202, 212, and 222 in each image have different image density values due to differences in the breathing state of the subject at the time the respective images were captured. Note that in Figures 3A to 3C, the closer to white the image, the higher the density value. In other words, the lung field 222 in the 3DCT data I_3 in the expiratory position has a higher density value than the lung field 202 in the 3DCT data I_1 in the inhalation position. Furthermore, the lung field 212 in the 3DCT data I_2, which shows the respiratory state during that time, has a value between the density values of the lung field 202 and the lung field 222.

In this embodiment, the case where three-phase 3DCT data including two phases of inspiration and expiration is used as described above will be described as an example, but the present invention is not limited to this. As long as the movement of the lung field due to the subject's breathing can be captured, 3DCT data of other respiratory phases may be used. Here, imaging may be performed at a cycle sufficiently faster than the respiratory cycle, and an image of a desired respiratory phase may be extracted as a processing target image from among the multiple images obtained. In addition, an operation by a user may be accepted, and an image to be processed may be selected accordingly. Alternatively, imaging may be performed at the timing of a desired respiratory phase in synchronization with the patient's breathing. In addition, a case where time-series images captured by other modalities such as MRI or ultrasound images are obtained may also be an embodiment of the present invention.

(Step S1010): Extraction of Lung Field Area In step S1010, the area extraction unit 110 executes a process of extracting a lung field area for each of the multiple 3DCT data acquired in step S1000. The process of extracting a lung field area from 3DCT data can be realized using a known image processing method. For example, a method of applying an arbitrary threshold process to pixel values may be used, or a known segmentation method such as a graph cut method may be used. A segmentation (semantic segmentation) method using a neural network such as Deep Learning may be used. In addition, a user may manually extract a lung area using a graphic drawing software (not shown), or a lung area extracted by a known image processing method may be manually corrected by the user. In this embodiment, a method of applying an arbitrary threshold process to pixel values is used to extract a lung field area for all time phases of 3DCT data acquired in step S1000. Specifically, all pixels in the 3DCT data are binarized based on whether the pixel value is within a predetermined threshold range, and then post-processing such as isolated point removal and smoothing is performed to extract the lung field area. Then, the extraction result of the lung field area for each time phase is calculated as a mask image M_t (1≦t≦3). The mask image M_t is information representing the lung field area related to each of the 3DCT data I_t (1≦t≦3), and is data in which the pixel value is 1 when the corresponding position of the 3DCT data is a lung field, and the pixel value is 0 otherwise. In the description of this embodiment, the mask image M_t is also expressed as a function that takes M_t (x, y, z) and the pixel position in the image as arguments and returns the pixel value of the position, that is, the value of whether the position is a lung field area or not. In this embodiment, M_t is held as volume data discretized to the same extent as the 3DCT data I_t.

In the above explanation, a mask image is calculated as an extraction result of the lung field area from the 3DCT data, but the present invention is not limited to this. For example, a set of pixels in the lung field may be calculated, or the coordinate values of the contour position surrounding them may be calculated.

(Step S1020): Calculation of average density value In step S1020, the average density value calculation unit 120 executes a process of calculating an average density value in the region of the lung field extracted in step S1010 for each of the multiple 3DCT data acquired in step S1000. Specifically, the calculation shown in formula (1) is executed for each of the multiple 3DCT data. In formula (1), Ω represents the entire image (all pixels) of the 3DCT data. C_ave_t is the average density value of the 3DCT data I_t.

In the above process, the average density value is calculated as the statistical quantity of the pixels in the lung field of the 3DCT data of each time phase, but the statistical quantity is not limited to the average density value, and any representative value obtained or calculated from the pixel values of the pixel group in the lung field may be used. For example, the mode, median, maximum value, minimum value, etc. of the pixels in the lung field of the 3DCT data of each time phase may be calculated. Moreover, these values are not limited to being calculated only from the area in the lung field, and the average density value of the entire 3DCT data of each time phase may be calculated. Conversely, the statistical quantity of some pixels may be calculated instead of the statistical quantity of all pixels in the lung field.

(Step S1030): Density Value Correction In step S1030, the density value correction unit 130 executes a process of correcting (converting) density values based on the average density value in the lung field in each 3DCT data acquired in step S1020 for each of the multiple 3DCT data acquired in step S1000. More specifically, a correction process shown in formula (2) is performed on the density values of the entire 3DCT data of each time phase so that the average density value in the lung field of all the 3DCT data becomes the same as the average density value in the lung field of the 3DCT data of a reference time phase (e.g., inspiration position). In formula (2), C_ave_1 is the average density value of the reference time phase (e.g., inspiration position), C_ave_t is the average density value of the 3DCT data I_t before correction, and I'_t is the 3DCT data after correction.

By processing equation (2), the pixel values of I_t are transformed so that the difference in the statistics of pixel values between the lung field region (first object region) in I_t (first image) and the lung field region (second object region) in I_1 (second image) is reduced. This transformed image I'_t is used to align the pixels of the lung field regions between the images (obtain the pixel correspondence).

The 3DCT data of the reference time phase (inspiration position) can be set to I'_1 = I_1 without performing the processing of equation (2). In addition, the above describes an example in which the density value of the entire 3DCT data of each time phase is corrected, but the present invention is not limited to this. For example, only the pixels within the lung field of the 3DCT data of each time phase may be corrected using a method similar to that described above. In addition, the reference time phase does not have to be the inspiration position.

In the above explanation, a reference time phase is set and correction is performed so that the average density value in the lung field of the 3DCT data of each time phase is the same as the average density value in the lung field of the 3DCT data of the reference time phase, but the implementation of the present invention is not limited to this. For example, without setting a reference time phase, the 3DCT data of all time phases may be corrected so that the average density value in the lung field of the 3DCT data of each time phase is aligned with a predetermined reference density value Cref that has been set in advance. In that case, the reference density value Cref can be substituted for C_ave_1 in formula (2).

(Step S1040): Image Registration in the Lung Field In step S1040, the displacement field calculation unit 140 performs registration processing in the lung field between the 3DCT data I_1 at the inspiration position and the 3DCT data I_3 at the expiration position based on the multiple 3DCT data I'_t (1≦t≦3) after density value correction. In this embodiment, the registration between I'_1 and I'_2 and the registration between I'_2 and I'_3 are performed, and the registration results between I'_1 and I'_3 are obtained by integrating these results. In this way, instead of directly calculating the registration between the image at the inspiration position and the image at the expiration position, indirectly calculating it using an image in an intermediate state (time phase) is advantageous for improving the accuracy of the registration. That is, since the amount of movement (position difference) of the lungs is large between the inspiration position and the expiration position, it is difficult to grasp the correspondence between pixels, and the possibility of failing to search for corresponding pixels increases. In this regard, by sandwiching an intermediate image between images, the amount of movement (difference in position) between the images is relatively small, making it possible to facilitate alignment and improve accuracy. Note that, as the number of intermediate images increases, alignment becomes easier, but this leads to an increase in processing time. Therefore, it is advisable to determine the number of intermediate images by taking into consideration the balance between accuracy and processing time.

As a result of the alignment process, the displacement field calculation unit 140 obtains a displacement field D_inner(x,y,z). Any known method may be used to integrate multiple displacement fields. D_inner(x,y,z) is a function that calculates the amount of displacement for converting an arbitrary three-dimensional position (x,y,z) in the lung field in the 3DCT data I_1 at the inspiration position into a corresponding three-dimensional position (x',y',z') in the 3DCT data I_3 at the expiration position. In other words, the displacement field D_inner(x,y,z) represents the correspondence between pixels in the object region (lung field region) between the two images.

The above-mentioned alignment between I'_1 and I'_2 and alignment between I'_2 and I'_3 may be performed using any known inter-image alignment method. There are various methods for aligning two images, such as a method of matching pixels based on the similarity of pixel values (density values) and a method of matching areas based on edges and textures. In this embodiment, a method of matching pixels based on the similarity of pixel values (density values) is adopted. This is because high-speed processing is possible. For example, when aligning the first image and the second image, a process of adding deformation to the lung field area of the first image and evaluating the similarity of the pixel values (density values) between the second image and the first image after deformation is repeated to find the optimal solution with the highest similarity. In this method, the deformation of the three-dimensional region can be expressed, for example, using FFD (Free Form Deformation), and the similarity of pixel values (density values) can be evaluated using SSD (Sum of Square Difference) or the like.

When performing this alignment, it is desirable to perform image processing on each image to emphasize the image features within the depicted lung field, and then use the results to perform alignment. For example, window conversion may be performed to emphasize the differences in density values of the air and parenchymal tissue contained in the lung field, i.e., the image features within the lung field. Window conversion is a process in which only a portion of the range of the original image is extracted and converted to a specified density resolution. By changing the center (window level) and size (window width) of the extracted range (called the window), it is possible to emphasize the desired image features in the original image. It is used in situations such as generating a display image (e.g., an 8-bit image) from a wide-range image (e.g., a 10- to 16-bit image) such as a medical image.

For example, in the case of general CT data where air is -1000 and water is 0, the image features within the lung field can be emphasized by setting the density value at the center of the window to -700 and the window width to 1000. In this embodiment, by performing such window conversion, an image in which the image features within the lung field are emphasized is generated, and alignment processing is performed on this image. This makes it possible to improve the accuracy of alignment within the lung field.

The window conversion settings may be stored in advance in the information processing device 10, or may be set by a user operation. Also, the settings may be based on the 3DCT data acquired in step S1000 or the associated information stored in the inspection image database 30.

In the above description, the density value correction process for each 3DCT data is performed in step S1030, and the window conversion is performed in step S1040. However, these processes may be combined and executed in step S1040. That is, the process in step S1030 may be omitted, and the window center of the window conversion executed in step S1040 may be changed and set for each 3DCT data. Specifically, the window center may be set based on C_ave_t (1≦t≦3) so that the density values in the lung field of each 3DCT data are the same.

(Step S1050): Inter-image registration of lung field In step S1050, the displacement field calculation unit 140 performs registration processing of the lung field between the 3DCT data I_1 at the inhalation position and the 3DCT data I_3 at the exhalation position based on the multiple 3DCT data I_t (1≦t≦3) acquired in step S1000. The lung field is the area around the lung field (object area). In this embodiment, the registration between I_1 and I_2 and the registration between I_2 and I_3 are performed, respectively, and the results are integrated to obtain the registration result between I_1 and I_3. Since the specific processing method is the same as the method described in step S1040, a detailed description is omitted here. Note that the change in density value of pixels outside the lung field due to breathing is small, so it is preferable to use the 3DCT data before converting the pixel value.

Through the above processing, the displacement field calculation unit 140 acquires the displacement field D_outer(x, y, z). D_outer(x, y, z) is a function that calculates the amount of displacement that converts any three-dimensional position (x, y, z) outside the subject's lung field in the 3DCT data I_1 at the inspiration position into a corresponding three-dimensional position (x', y', z') in the 3DCT data I_3 at the expiration position. In other words, the displacement field D_outer(x, y, z) represents the correspondence between pixels in the surrounding area between the two images.

The above-mentioned alignment between I_1 and I_2 and alignment between I_2 and I_3 may be performed using any known inter-image alignment method. The same method as exemplified in the lung field alignment process may be used. When performing this alignment, it is desirable to perform image processing to emphasize the image features outside the lung field depicted in each image, and use the results to perform alignment. For example, window conversion may be performed so that the difference in density value of the tissue around the lung field, that is, the image features outside the lung field, are emphasized. By setting the density value of the window center to 0 and the window width to 400, etc., the image features of the soft tissue outside the lung field can be emphasized. In addition, by setting the density value of the window center to 200 and the window width to 2000, etc., the image features of the bones outside the lung field can be emphasized. In this embodiment, an image in which the image features outside the lung field are emphasized is generated by performing the window conversion exemplified above, and the alignment process is performed on the image.

The window conversion settings may be stored in advance in the information processing device 10, or may be set by a user operation. Also, the settings may be based on the 3DCT data acquired in step S1000 or the associated information stored in the inspection image database 30.

In the above description, an example has been described in which an image in which image features outside the lung field are emphasized is generated based on the 3DCT data I_t (1≦t≦3) and the displacement field D_outer(x,y,z) is calculated, but the present invention is not limited to this. For example, an image in which image features outside the lung field are emphasized may be generated based on the 3DCT data I'_t (1≦t≦3) after density value conversion, as in step S1040, and the displacement field D_outer(x,y,z) may be calculated.

(Step S1060): Calculation of Slippage Map In step S1060, the map generator 150 executes a process of generating a map of the amount of slippage caused by breathing on the lung surface (lung contour) of the subject. This process is executed based on the lung mask image M_1 at the inspiration position calculated in step S1010, the displacement field D_inner(x, y, z) calculated in step S1040, and the displacement field D_outer(x, y, z) calculated in step S1050. More specifically, this can be implemented by the method described in Patent Document 1. That is, the displacement field D_out at the contour portion of the lung mask image M_1 is calculated based on the displacement field D_inner(x, y, z) calculated in step S1050.
The difference between the two displacement fields, er(x, y, z) and D_inner(x, y, z), is calculated, and the difference is generated as a three-dimensional map, a slippage map S(x, y, z). In this embodiment, the slippage map S(x, y, z) is a function that returns the amount of slippage at a position in the image coordinate system of the 3DCT data at the inspiration position as an argument. More specifically, it is held as volume data discretized to the same extent as the 3DCT data. The map generating unit 150 performs a process of storing the generated slippage map S(x, y, z) in a storage unit (not shown) or the examination image database 30 in association with the examination image of the subject, as necessary.

(Step S1070): Display of slippage map In step S1070, the display control unit 160 performs control to display the slippage map S(x, y, z) calculated in step S1060 on the display device 60. Specifically, an image (observation image) for observing the slippage map is generated, and the image is controlled to be displayed on the display device 60. The observation image can be generated, for example, as a surface rendering image in which the slippage map is converted into a grayscale or color map on the three-dimensional lung field contour shape of the subject. Note that the above-mentioned method is merely one example of the present invention, and any method for displaying the slippage map, or even if the display itself is not performed, can be one embodiment of the present invention.

The information processing device 10 in this embodiment performs processing using the method described above. This allows for highly accurate alignment of 3DCT data representing different respiratory states, making it possible to provide the user with a slippage map that is effective in understanding the adhesion state of the subject's pleura and assisting in diagnosis.

(Modification 1-1): Modification further adding density value correction due to volume change In the explanation of step S1040 of the first embodiment, a specific example of performing alignment in the lung field using the image I'_t after pixel value conversion was explained, but the implementation of the present invention is not limited to this. For example, after executing the process of S1040 described in the first embodiment, a process of further correcting the displacement field D_inner (x, y, z) which is the processing result can be executed. For example, based on the displacement field D_inner (x, y, z) calculated by the above process, a local volume change in the lung field is calculated, and based on the calculated local volume change, the pixel value of each position in the lung field is further corrected. Specifically, the density value of the part where the volume expands locally can be lowered, and conversely, the density value of the part where the volume contracts locally can be increased. Then, based on the corrected image, a further alignment process may be executed to correct the displacement field D_inner (x, y, z). Also, based on the displacement field corrected by this process, the displacement field may be corrected by repeating the above-mentioned process. According to this, in addition to the alignment within the lung field described in the first embodiment, it is possible to estimate local volume changes in the lung field region of the subject and perform concentration value correction based on that, thereby achieving higher accuracy of alignment.

(Modification 1-2): Variation of pixel value correction method In the first embodiment of the present invention, the average density value of each lung field of a plurality of 3DCT data is calculated in step S1020, and in step S1030, correction is performed to shift the density value of each image so as to absorb the difference. However, the present invention is not limited to this. For example, in step S1020, in addition to the average density value C_ave_t of the lung field, the variance C_div_t of the density value can also be calculated for each 3DCT data. In this case, in step S1030, a process of correcting the density value by linear transformation (shifting and scaling) may be performed so that the average density value and the variance of the density value of the lung field of all the 3DCT data become the same as those at the inspiration position. The linear transformation for matching the average value and the variance value to the desired value can be performed by a known method. According to the above method, not only the average density value but also the variance of density values of the 3DCT data at each time phase can be made to match with the 3DCT data at the inspiration position, so that the image can be corrected to be more similar to the 3DCT data at the inspiration position, which has the effect of enabling more accurate alignment within the lung field.

The method of correcting the density values is not limited to the above method. For example, for each 3DCT data, the density values may be linearly converted so that the average density value in the lung field matches the average density value in the lung field of the 3DCT data at the inspiration position, while keeping the density value representing the air region (-1000 for general CT data) unchanged.

The implementation of the present invention is not limited to the above-described case where density values are corrected by a uniform linear transformation of the region to be corrected. For example, the mixture ratio of air and tissue parenchyma of each pixel in the lung field may be estimated based on the density value of that pixel, and the magnitude of the correction of the density value may be changed according to the air content ratio. This has the effect of making a correction that reflects the local change in ventilation volume associated with breathing in the lungs of an actual subject.

In addition to the above method, the density value may be corrected based on the difference in the volume of the lung field area of each 3DCT data calculated in step S1010. For example, the larger the difference between the volume of the lung field in the 3DCT data in the inhalation position and the volume of the lung field in the 3DCT data to be corrected, the larger the correction amount of the density value. According to this method, it is possible to perform a more accurate correction of the density value, which reflects the phenomenon that the content ratio of parenchymal tissue in the lung field increases as the volume decreases due to the discharge of air from the lung field of the subject by breathing. The method of correcting the density value is not limited to the above method. For example, the lung field may be further divided into multiple partial regions, and different corrections may be performed for each divided partial region. For example, each of the multiple lung lobes constituting the lung may be divided as a partial region, the average of the density value for each lobe may be calculated, and the density value of the integrated lung field may be corrected based on the average value. Alternatively, the left and right lungs may be divided as partial regions and the same correction as above may be performed. This has the effect of allowing for highly accurate correction that takes into account differences in ventilation volume between lung lobes and between the left and right lungs.

(Modification 1-3): In the case of two time phases or in the case of four or more time phases In the first embodiment of the present invention, the case where 3DCT data of three time phases is acquired in step S1000 has been described as an example, but the present invention is not limited to this. For example, in step S1000, only 3DCT data of two time phases, that is, the inspiration position and the expiration position, may be acquired. In this case, in the process of step S1030, the density value of the 3DCT data of the inspiration position and the expiration position is corrected. Then, in step S1040, the displacement field in the lung field between the inspiration position and the expiration position is calculated by aligning the images. In addition, in step S1050, the displacement field outside the lung field is calculated by aligning the 3DCT data of the two time phases acquired in step 1000. The process when the 3DCT data of two time phases is input is performed by the method described above.

The implementation of the present invention is not limited to the above method, and 3DCT data of four or more time phases may be acquired in step S1000. Even in this case, the slippage map of the subject can be calculated using the same process as in the first embodiment.

[Second embodiment]
A second embodiment of the present invention will be described. In the first embodiment, an image with corrected density values is generated, and the image is used to perform intrapulmonary alignment processing. However, the present invention is not limited to this. In the second embodiment, an image with corrected density values is not generated, but rather, pixel-by-pixel density value correction is performed during intrapulmonary alignment processing.

The overall configuration of the information processing system according to the second embodiment of the present invention is the same as that shown in FIG. 1, which illustrates the overall configuration of the information processing system according to the first embodiment. A detailed description will be omitted here.

The overall processing procedure of the information processing device 10 in this embodiment will be described in detail using Figure 4.

(Step S2000) to (Step S2020)
Steps S2000 to S2020 are similar to steps S1000 to S1020 in the first embodiment, and detailed description thereof will be omitted here.

(Step S2040)
In step S2040, the displacement field calculation unit 140 performs a registration process in the lung field between the 3DCT data I_1 at the inspiration position and the 3DCT data I_3 at the expiration position based on the multiple 3DCT data I_t (1≦t≦3) acquired in step S2000. In this embodiment, the registration between I_1 and I_2 and the registration between I_2 and I_3 are performed, and the registration results between I_1 and I_3 are obtained by integrating these results. As a result of the above process, the displacement field calculation unit 140 acquires a displacement field D_inner (x, y, z). Any known method may be used to integrate multiple displacement fields. Here, D_inner (x, y, z) is a coordinate transformation function that transforms an arbitrary three-dimensional position (x, y, z) in the lung field of the subject in the 3DCT data I_1 at the inspiration position into a corresponding three-dimensional position (x', y', z') in the 3DCT data I_3 at the expiration position.

In this embodiment, the above-mentioned alignment between I_1 and I_2 and the alignment between I_2 and I_3 are performed using the FFD (Free Form Deformation) method. However, the present invention is not limited to this, and may be performed using, for example, the LDDMM (large deformation diffeomorphic metric) method.
Alternatively, other deformation registration methods such as a 3D image mapping method may be used.

This process will be described in detail with reference to the process flow in FIG. 5. In the following description, an example will be described in which the displacement field D_inner12(x, y, z) is calculated as the alignment result between I_1 and I_2. The alignment process between I_2 and I_3 performed in step S2040 can also be performed in the same manner as the process described in detail below.

(Step S20400): Initialization In step S20400, the displacement field calculation unit 140 executes a process of initializing the displacement field D_inner12 (x, y, z). Specifically, the control amounts of all control points of the FFD are initialized to a zero vector. In this embodiment, the number of control points of the FFD is set to N, and a three-dimensional control amount parameter is set to each control point. That is, the displacement field is expressed by a total of 3N control amount parameters. In this embodiment, the control amount parameter is represented as V. Here, V is 3N-dimensional vector format data that stores the control amount of the FFD. In addition, in this step, the displacement field calculation unit 140 executes a process of calculating the image similarity between I_1 and I_2. Any known method may be used to calculate the image similarity, but in this embodiment, the image similarity E_org is calculated by SSD (Sum of Square Difference) shown in formula (3) as an example.

In formula (3), the "+C_ave_1-C_ave_2" portion is a density value correction term for aligning density values between the 3DCT data I_1 and I_2. Thus, in the registration process of this embodiment, density value correction for each pixel is performed simultaneously in the calculation for calculating the similarity between images. Note that W is a function of the window transformation of the image. In this embodiment, in order to calculate the image similarity by emphasizing the image features within the lung field, a window transformation is performed that emphasizes the difference in density values of the air and parenchymal tissue contained in the lung field, that is, the image features within the lung field. As an example, a transformation is performed with the density value at the window center set to -700 and the window width set to 1000.

The process from step S20401 to step S20405 is then repeated.

(Step S20401): Initialization of Control Point Index i In step S20401, the displacement field calculation unit 140 initializes to 1 the index value i of the control point that is to be the target of the processes from step S20403 to step S20404.

(Step S20403): Calculation of Similarity After Control Amount Change In step S20403, the displacement field calculation unit 140 calculates a displacement field when the control amount of the control point of index i is changed slightly, and calculates the image similarity between I_1 and I_2 based on the calculated displacement field.

Specifically, first, a controlled variable parameter V'_i is generated after minute fluctuations in the element value of the i-th dimension of the controlled variable parameter V. Here, a fluctuation is applied that increases the parameter value by δ.

Next, the displacement field calculation unit 140 calculates the displacement field D_inner12_i(x, y, z) based on the control amount parameter V'_i to which the fluctuation has been applied. This process is similar to a known method using the FFD method, so a detailed description will be omitted.

Furthermore, the displacement field calculation unit 140 calculates the similarity between the 3DCT data I_2 and I_1 when the 3DCT data I_2 is transformed based on the displacement field D_inner12_i(x, y, z) calculated by the above-mentioned method. Specifically, the image similarity E_i is calculated by processing of equation (4).

Note that dx_i represents the amount of displacement of D_inner12_i(x, y, z) in the x-axis direction. Similarly, dy_i represents the amount of displacement in the y-axis direction, and dz_i represents the amount of displacement in the z-axis direction. The effects of the density value correction term and window conversion are the same as those described in equation (3).

By the above process, the image similarity E_i is calculated when the i-th control parameter is varied.

(Step S20404): Incrementing Index In step S20404, the displacement field calculation unit 140 executes a process of incrementing the index value i of the control amount parameter.

(Step S20405): Judgment (i>3N)
In step S20405, the displacement field calculation unit 140 determines whether or not the index value i of the control amount parameters exceeds the total number of control amount parameters, 3N. If it does, the process proceeds to step S20406, and if not, the process returns to step S20403.

By performing the processes from step S20403 to step S20405 described above, the image similarity E_i (1≦i≦3N) is calculated when the element values of all dimensions of the control variable parameter V are varied.

(Step S20406): Updating the Displacement Field In step S20406, the displacement field calculation unit 140 executes a process of updating the displacement field D_inner12(x, y, z) based on the image similarities E_org and E_i (1≦i≦3N). Specifically, the control amount parameter V is updated based on the image similarities E_org and E_i (1≦i≦3N), and the displacement field D_inner12(x, y, z) is updated based on the updated control amount parameter V.

The control amount parameter V can be updated based on the image similarities E_org and E_i (1≦i≦3N) using, for example, the steepest descent method. That is, the amount of change E_i-E_org in image similarity relative to the change in the value of each of the 3N elements that make up the control amount parameter V is calculated, and the value of each element of the control amount parameter V is corrected in proportion to the amount of change. Then, D_inner12(x, y, z) is updated based on the corrected control amount parameter V.

(Step S20407): Calculation of Similarity After Displacement Field Update In step S20407, the displacement field calculation unit 140 executes a process of calculating the image similarity between I_1 and I_2 based on the displacement field D_inner12(x, y, z) corrected in step S20406, and updating E_org. The method of calculating the image similarity is the same as the calculation method shown in formula (3), and detailed description will be omitted.

(Step S20408): Convergence determination In step S20408, the displacement field calculation unit 140 determines whether or not to end the process of step S2040 based on the similarity calculated in step S20407. Specifically, if the image similarity E_org is greater than a predetermined threshold, the process of step S2040 ends, and if not, the process returns to step S20401.

The processing of step S2040 is executed through the processing of steps S20400 to S20408 described above.

Next, the information processing device 10 executes steps S2050 to S2070, which are the same as steps S1050 to S1070 in the first embodiment. A detailed description will be omitted.

The above process executes the process in the second embodiment of the present invention. This method, unlike the first embodiment, has the advantage that it is possible to perform the alignment process more efficiently because it is not necessary to generate the image I'_t after density value correction and store it in a memory (not shown).

(Variation 2-1)
In the description of this embodiment, as an example of a method for calculating the image similarity E_org in step S20400, the pixel value of I_1 and the pixel value of I_2 after density value correction are converted by the same window conversion function W as shown in formula (3). However, the implementation of the present invention is not limited to this. For example, as shown in formula (5), the pixel value of I_1 and the pixel value of I_2 may be converted by using different window conversion functions. For example, the window center of the window conversion function W_1 for I_1 may be set to C_ave_1, and the window center of the window conversion function W_2 for I_2 may be set to C_ave_2. In addition, the window width of W_1 may be set based on the variance of the density value inside the lung field of I_1, and the window width of W_2 may be set based on the variance of the density value inside the lung field of I_2.

In addition, in step S20403, the calculation shown in equation (4) has been described as an example of a method for calculating image similarity E_i. However, E_i may also be calculated by calculating equation (6) using window transformation functions W_1 and W_2, similar to equation (5).

According to the method described above, window conversion is performed on I_1 and I_2 according to the difference in the distribution of density values inside the lung field, so that the processing in step S2040 has the effect of enabling more accurate alignment.

[Third embodiment]
A third embodiment of the present invention will be described. Unlike the first embodiment in which a sliding amount map of the subject's lungs is generated as lung movement information, this embodiment generates a motion amount map of the subject's lungs as lung movement information.

The overall configuration of the information processing system according to this embodiment is the same as that shown in FIG. 1, which explains the overall configuration of the information processing system according to the first embodiment. A detailed explanation will be omitted here.

The overall processing procedure by the information processing device 10 in this embodiment will be described in detail using Figure 6.

(Step S3000) to (Step S3040)
In steps S3000 to S3040, the information processing device 10 executes the same processes as those in steps S1000 to S1040 executed by the information processing device of the first embodiment, and detailed description thereof will be omitted here.

(Step S3060)
In step S3060, the map generator 150 executes a process of generating a map of the amount of movement due to breathing on the lung surface (lung contour) of the subject. This process is executed based on the lung mask image M_1 at the inhalation position calculated in step S3010 and the displacement field D_inner (x, y, z) calculated in step S3040. More specifically, the displacement amount of the displacement field D_inner (x, y, z) at the contour portion of the lung mask image M_1 is calculated, and it is generated as a three-dimensional map, a motion amount map S (x, y, z). In this embodiment, the motion amount map S (x, y, z) is a function that returns the amount of movement at a position in the image coordinate system of the 3DCT data at the inhalation position as an argument. More specifically, it is held as volume data discretized to the same extent as the 3DCT data. This motion amount map S (x, y, z) represents the distribution of the amount of movement of the lung between two time phases (states). If necessary, the map generating unit 150 performs a process of storing the generated motion amount map S(x, y, z) in a storage unit (not shown) or in the examination image database 30 in association with the examination image of the subject.

(Step S3070)
In step S3070, the display control unit 160 performs control for displaying the motion amount map S(x, y, z) calculated in step S3060 on the display device 60. A specific control method can be performed in the same manner as the display control of the slippage amount map described in step S1070 of the first embodiment. A detailed description will be omitted.

The information processing device 10 of this embodiment performs processing using the method described above. This allows for highly accurate alignment of 3DCT data representing different respiratory states, making it possible to provide the user with a motion map that is effective in understanding the state of pleural adhesions in the subject and in assisting in diagnosis.

(Variation 3-1)
In the description of this embodiment, the case where a motion amount map is generated in step S3060 has been described as an example, but the implementation of the present invention is not limited to this. For example, a ventilation amount map of the lungs may be calculated as the process of step S3060. In this case, the magnitude of the volume change amount at each position in the lung field may be visualized based on the displacement field D_inner (x, y, z) in the lung field calculated in step S3040. For example, in this embodiment in which the displacement field D_inner (x, y, z) representing the displacement from the inspiration position to the expiration position is calculated, if the subject has normal lung function, the displacement amount in the lung field is a displacement in which the volume contracts greatly. On the other hand, if the contraction of the local volume is small, etc., a disturbance in the ventilation function of the lung at that location is suspected. As an example of the implementation of the present invention, an image that visualizes the state of the local volume change from the displacement field in the lung field may be generated so that the user can visually recognize the presence of such a location.

In addition, the processes from step S3000 to step S3040 can be performed for any purpose of aligning three-dimensional images of the lungs captured at different respiratory phases. This allows accurate alignment to be performed with reduced alignment errors caused by changes in density values in the lung field due to differences in respiratory phases.

[Other embodiments]
The above-described embodiment is merely an example of the present invention, and the configuration and scope of the present invention are not limited to the above-described embodiment. The processes of multiple embodiments may be combined, or the processes of multiple modified examples may be combined.

In the above embodiment, a processing example in which the present invention is applied to the alignment of lung field images has been described, but the application of the present invention is not limited to lung field images, and the alignment process of the present invention can be preferably applied to any object that moves and whose image density value can change. For example, the present invention can be applied to the alignment of heart images with different contrast conditions. Specifically, the present invention may be applied to the alignment between an image of the heart photographed in a contrast-enhanced state (a state in which a contrast agent is introduced) and an image of the same part photographed without contrast (a state in which a contrast agent is not introduced). The present invention may also be applied to the alignment between an image of the heart photographed in a state in which the blood concentration of the contrast agent is higher than a predetermined value and an image of the heart photographed in a state in which the blood concentration of the contrast agent is lower than a predetermined value. The present invention may also be applied to images of organs other than the lungs and the heart.

The disclosed technology can be embodied, for example, as a system, device, method, program, or recording medium (storage medium). Specifically, it may be applied to a system consisting of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or it may be applied to an apparatus consisting of a single device.

Needless to say, the object of the present invention can be achieved by the following: A recording medium (or storage medium) on which is recorded software program code (computer program) that realizes the functions of the above-mentioned embodiments is supplied to a system or device. The storage medium is, of course, a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-mentioned embodiments, and the recording medium on which the program code is recorded constitutes the present invention.

10: Information processing device 100: Inspection image acquisition unit 110: Region extraction unit 120: Average density value calculation unit 130: Density value correction unit 140: Displacement field calculation unit 150: Map generation unit

Claims (17)

an image acquisition unit that acquires a first image and a second image captured at different times of a moving object;
an image processing unit that performs a registration process between the first image and the second image,
The image processing unit includes:
performing a conversion process for converting pixel values of at least one of the first image and the second image based on the estimated air content ratio of the pixel so that a difference between statistics of pixel values of the region of the object in the first image and statistics of pixel values of the region of the object in the second image becomes small;
An information processing device characterized by performing a first alignment process to obtain a correspondence between pixels within the object area between the first image and the second image based on pixel values converted by the conversion process. The image processing unit further includes:
performing a second registration process for acquiring a correspondence relationship between pixels in a surrounding region between the first image and the second image, the surrounding region being a region around the object;
The information processing device according to claim 1 , further comprising: acquiring a distribution of the amount of slippage between the object region and the surrounding region based on a corresponding relationship between pixels in the object region and a corresponding relationship between pixels in the surrounding region. The information processing device according to claim 2, characterized in that the image processing unit performs the second alignment process to obtain the correspondence of pixels in the surrounding area between the first image and the second image using the first image and the second image before pixel values are converted. The image processing unit further includes:
An information processing device as described in any one of claims 1 to 3, characterized in that a distribution of the amount of movement of the object between the state of the first image and the state of the second image is obtained based on the correspondence of pixels within the area of the object between the first image and the second image. The information processing device according to any one of claims 1 to 4, characterized in that the image processing unit performs the conversion process to convert pixel values of at least one of the first image and the second image so that the difference between the statistics of pixel values of the object region in the first image and the statistics of pixel values of the object region in the second image is reduced and image features within the object region are emphasized. The image processing unit includes:
converting pixel values of at least one of the first image and the second image so that a difference between a statistical amount of pixel values of the region of the object in the first image and a statistical amount of pixel values of the region of the object in the second image becomes small; and
6. The information processing apparatus according to claim 5, further comprising: a window conversion step for converting the first image and/or the second image into a window image using a predetermined window. The image processing unit includes:
performing a window transformation on the first image using a first window set based on statistics of pixel values of the region of the object in the first image;
6. The information processing apparatus according to claim 5, further comprising: a window conversion unit that converts the second image into a second window that is set based on statistics of pixel values of the region of the object in the second image. the image acquisition unit acquires an intermediate image captured in a state between the first image and the second image,
The image processing unit includes:
converting pixel values of at least two images among the intermediate state image, the first image, and the second image so that a difference in statistics of pixel values of the region of the object among the intermediate state image, the first image, and the second image is reduced;
obtaining a first correspondence relationship, which is a correspondence relationship of pixels in the region of the object between the first image and the intermediate state image, and a second correspondence relationship, which is a correspondence relationship of pixels in the region of the object between the intermediate state image and the second image, based on the converted pixel values;
The information processing device according to any one of claims 1 to 7, characterized in that the result of the first alignment process indicating the correspondence of pixels within the area of the object between the first image and the second image is generated by integrating the first correspondence and the second correspondence. The image processing unit includes:
After obtaining a result of the first registration process indicating a correspondence relationship of pixels within a region of the object between the first image and the second image,
calculating a volume change of the object between the first image and the second image based on the correspondence relationship;
correcting transformed pixel values of the first image and/or the second image based on a volume change of the object;
The information processing device according to any one of claims 1 to 8, characterized in that the correspondence between pixels within the area of the object between the first image and the second image is corrected based on the corrected pixel values. The information processing device according to any one of claims 1 to 9, characterized in that the image processing unit performs pixel correspondence between the first image and the second image based on the similarity of pixel values. The information processing device according to any one of claims 1 to 10, characterized in that the object is a lung, and the first image and the second image are images taken at different respiratory phases. The information processing device according to any one of claims 1 to 10, characterized in that the first image and the second image are images taken under different contrast conditions. The information processing device according to any one of claims 1 to 12, characterized in that the first image and the second image are CT data. The information processing device according to any one of claims 1 to 13, characterized in that the first image and the second image are images captured at different times. The information processing device according to any one of claims 1 to 14, characterized in that the image processing unit does not convert pixel values of areas in the first image and the second image that correspond to air regions. acquiring a first image and a second image of a moving object captured at different times;
performing a registration process between the first image and the second image;
In the alignment process,
performing a conversion process for converting pixel values of at least one of the first image and the second image based on the estimated air content ratio of the pixel so that a difference between statistics of pixel values of the region of the object in the first image and statistics of pixel values of the region of the object in the second image becomes small;
An information processing method, comprising: executing a first alignment process to obtain a correspondence between pixels within the object area between the first image and the second image based on pixel values converted by the conversion process. A program for causing a computer to execute each step of the information processing method according to claim 16 .

Images