JP2014238344A - Image processing system, image processor and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a user to determine whether an object within a cell candidate area is a target cell only by referring to an image of the cell candidate area in a picked-up image.SOLUTION: An imaging part (12) images a sample including a target cell in each of a plurality of focal distances, a candidate area image acquisition part (22) acquires an image of a cell candidate area being a candidate for an area with the target cell photographed in each of picked-up images that are imaged, and an information for visual confirmation display part (28) displays a plurality of the images acquired by the candidate area image acquisition part (22). In addition, the information for visual confirmation display part (28) receives an input of a user's determination result about whether the target cell is photographed in the cell candidate area.

Description

  The present invention relates to an image processing system, an image processing apparatus, and a program.

  When diagnosing a prenatal fetus, fetal nucleated red blood cells (NRBCs, hereinafter referred to as target cells) contained in a very small amount in the maternal blood are detected and used. Since the number of NRBCs present in maternal blood is extremely small, it is a heavy burden to detect NRBCs by visual inspection. Therefore, as described in Patent Document 1 below, a technique for mechanically detecting NRBCs by searching a target image for cells that match conditions such as color, shape, positional relationship, and area ratio of NRBCs. Has been proposed.

Japanese Patent No. 4346923

  An object of the present invention is to enable a user to determine whether or not an object in a cell candidate region is a target cell by only referring to an image of the cell candidate region in a captured image.

  The invention according to claim 1 for solving the above-described problem is that the target cell in each of the imaging unit that images the sample including the target cell at a plurality of focal lengths, and the captured image captured by the imaging unit. An acquisition unit that acquires an image of a cell candidate region that is a candidate for the reflected region, a display unit that displays a plurality of the images acquired by the acquisition unit, and whether or not the target cell is captured in the cell candidate region An image processing system including a means for receiving an input of a user's determination result relating to the above.

  The image processing system according to claim 2 is the image processing system according to claim 1, wherein the display unit aligns the plurality of images according to a focal distance when each captured image is captured. It is characterized by displaying a list in order.

  The image processing system according to claim 3 is the image processing system according to claim 1, wherein the display unit focuses each of the plurality of images when each captured image is captured. It displays in order in the display order according to distance.

  The image processing system according to claim 4 is the image processing system according to any one of claims 1 to 3, wherein the display unit displays the plurality of images as the cell candidates in any captured image. It is characterized by being displayed together with an image of a peripheral area including the area.

  The image processing system according to claim 5 is the image processing system according to any one of claims 1 to 4, wherein the display unit focuses the plurality of images when each captured image is captured. It is characterized by being displayed together with information indicating the distance.

  Moreover, the invention according to claim 6 for solving the above-described problem is that the imaging unit captures a region in which the target cell is captured in each of the captured images obtained by imaging the sample including the target cell at each of a plurality of focal lengths. Acquisition means for acquiring images of candidate cell candidate areas, display means for displaying the plurality of images acquired by the acquisition means, and user judgment regarding whether or not the target cells are shown in the cell candidate areas And an image processing apparatus including means for receiving an input of a result.

  Moreover, the invention according to claim 6 for solving the above-described problem is that the imaging unit captures a region in which the target cell is captured in each of the captured images obtained by imaging the sample including the target cell at each of a plurality of focal lengths. Acquisition means for acquiring an image of a candidate cell candidate area, display means for displaying a plurality of the images acquired by the acquisition means, and determination result of a user regarding whether or not the target cell is shown in the cell candidate area It is a program for causing a computer to function as means for receiving input.

  According to the first, sixth, and seventh aspects of the invention, the user can determine whether or not the object in the cell candidate region is a target cell only by referring to the image of the cell candidate region in the captured image. it can.

  According to the second to fifth aspects of the present invention, it is easier to determine whether or not the object in the cell candidate region is a target cell, as compared with the case where this configuration is not provided.

It is a figure showing the system configuration figure of the image processing system concerning this embodiment. It is a figure which shows the screen displayed on a display apparatus. It is a figure which shows the screen displayed on a display apparatus. It is a figure which shows the example of the image in each focal distance of various cells. It is a figure for demonstrating a display mode (1). It is a figure for demonstrating a display mode (2). It is a figure for demonstrating a display mode (3). It is a figure which shows a mode that a display mode switches. It is a functional block diagram which shows the function group implement | achieved by an image processing system. It is a figure which shows an example of a pixel block, a nucleus candidate area | region, and a cell candidate area | region. It is a figure which shows an example of the memory content of an area | region candidate database. It is a figure which shows a mode that each candidate area | region image is displayed in order. It is a flowchart which shows an example of the process performed with an image processing apparatus. It is a figure for demonstrating the display mode (4) of a detailed display.

  Hereinafter, examples of embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram of an image processing system 1 according to the present embodiment. As shown in FIG. 1, the image processing system 1 includes an optical microscope 2, an image processing device 4, and a display device 6. The image processing device 4 is connected to each of the optical microscope 2 and the display device 6 so that data communication is possible.

  The optical microscope 2 images a sample on a slide glass placed on a sample stage with a CCD camera via an optical system such as an objective lens. In the present embodiment, a sample obtained by applying maternal blood on a slide glass and performing May-Giemsa staining is used as the sample. As a result, fetal nucleated red blood cells in maternal blood are stained blue-violet. Hereinafter, nucleated red blood cells are referred to as target cells.

  The image processing device 4 is, for example, a personal computer, and acquires a captured image captured by the optical microscope 2. The captured image includes images of various cells contained in the maternal blood. The nuclei of target cells (nucleated erythrocytes) are stained slightly darker than the nuclei of other cells by Mei Giemsa staining.

  Further, the image processing device 4 identifies a cell candidate region that is a candidate for a region in which the target cell is shown, and displays a list of images of each identified cell candidate region on the display device 6.

  The display device 6 displays a list of images of each cell candidate region specified by the image processing device 4. FIG. 2A is a diagram illustrating an example of a screen displayed on the display device 6. As shown in the figure, a list of images of the specified cell candidate regions is displayed on the screen.

  The user refers to each image displayed on the screen in the prenatal diagnosis. A visual result input button 11 is provided at the upper right of the image of each cell candidate region, and the user clicks the visual result input button 11 related to the cell candidate region for which it is determined whether or not the target cell is shown, and the determination is made. Enter the result.

  By the way, it may not be possible to determine the presence or absence of a target cell simply by referring to an image. Therefore, when it is not possible to make a judgment only by referring to the image, it is conceivable to perform an operation of observing the change in the size of the nucleus of the cell in the image with the optical microscope 2 while changing the focal length. Since the nuclei of cells other than the target cells (such as lymphocytes, large white blood cells, and erythrocytes with dust attached) are flat, the size of the image of the nucleus is almost constant regardless of the focal length. This is because the nucleus of the nucleated erythrocyte is not flat, and the size of the image of the nucleus portion changes depending on the focal length. In FIG. 3, the example of the image in each focal distance of various cells was shown. The numerical value described below each image represents the focal length. Here, the focal distance is expressed as a distance in the downward direction from the autofocus position of the focal position. The symbol “+” indicates that the focal position is below the autofocus position, and the symbol “−” indicates that the focal position is above the autofocus position. As shown in FIG. 3, the size of the nucleus image of the nucleated red blood cell that is the target cell changes according to the focal length.

  However, it is troublesome to perform the operation of observing the change in the size of the image of the nucleus portion with the optical microscope 2.

  In this respect, in this image processing system 1, this labor is omitted. That is, in this image processing system 1, the optical microscope 2 is provided with a focusing mechanism. Moreover, the optical microscope 2 images a sample at each of a plurality of focal lengths.

  In addition, buttons 7a and 7b for selecting an image display form are provided on the screen shown in FIG. 2A. When the button 7b is selected, as shown in FIG. 2B, a list of images at each focal length is displayed for each cell candidate region. The numerical value below the image indicates the focal length. Therefore, the user can determine whether or not the object in the cell candidate region is a target cell by referring to the image of the cell candidate region in each captured image without performing the above operation. Yes.

  Further, on the screen shown in FIG. 2A, detailed display is performed regarding the cell candidate region selected by the user by clicking. Here, buttons 10a, 10b, and 10c for selecting a display mode of detailed display are provided. When the button 10a is selected, detailed display is performed in the display mode (1). FIG. 4A is a diagram for explaining the display mode (1). In the display mode (1), as shown in FIG. 4A, a list of images at each focal length is displayed for the cell candidate region selected by the user. Again, the numerical value below the image indicates the focal length. Therefore, by selecting the button 10a, the user can determine whether or not the object in the cell candidate region is a target cell by referring to the image of the cell candidate region in each captured image. .

  When the button 10b is selected, detailed display is performed in the display mode (2). FIG. 4B is a diagram for explaining the display mode (2). In the display mode (2), as shown in FIG. 4B, an image of a peripheral region including the cell candidate region selected by the user is displayed. When the button 10c is selected, detailed display is performed in the display mode (3). FIG. 4C is a diagram for explaining the display mode (3). In the display mode (3), as shown in FIG. 4C, an image at each focal length and an image of the peripheral region are displayed for the cell candidate region selected by the user. The user can switch the display mode of the detailed display among the display modes (1) to (3) by switching the button to be selected from the buttons 10a, 10b, and 10c. FIG. 5 shows how the display mode is switched.

  Hereinafter, a technique for allowing a user to determine whether an object in a cell candidate region is a target cell by referring to an image of the cell candidate region in a captured image will be described.

  FIG. 6 is a functional block diagram showing a functional group realized by the image processing system 1. In the image processing system 1, an imaging unit 12, a sample image acquisition unit 14, a sample image storage unit 16, a cell candidate region determination unit 18, a region candidate database 20 (hereinafter referred to as a region candidate DB 20), a candidate region image acquisition unit 22, A visual confirmation information generating unit 24, a visual confirmation information storage unit 26, and a visual confirmation information display unit 28 are realized. The imaging unit 12 is realized by the optical microscope 2.

  Functions other than the imaging unit 12 are realized by the image processing device 4. Functions other than the imaging unit 12 are image processing that is a computer including a control unit such as a microprocessor, a storage unit such as a memory, and an input / output unit that transmits and receives data to and from an external device such as an operation reception unit that receives a user operation. The apparatus 4 is realized by reading and executing a program stored in a computer-readable information storage medium (for example, an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, a flash memory, etc.). The program may be supplied to the image processing apparatus 4 as a computer via a data communication network such as the Internet.

  Each function will be described below. The imaging unit 12 images the sample on the slide glass surface at each of a plurality of predetermined focal lengths, and outputs a plurality of captured images (hereinafter referred to as sample images) to the sample image acquisition unit 14. . The specimen image is image data, and the focal distance when it is imaged is included as header information.

  The sample image acquisition unit 14 acquires data of each of the plurality of sample images from the imaging unit 12 and stores each acquired sample image in the sample image storage unit 16. The focal length is expressed as a downward distance from the autofocus position of the focal position.

  Moreover, the cell candidate area | region determination part 18 determines the cell candidate area | region mentioned above. Specifically, the cell candidate region determination unit 18 reads one of the plurality of sample images as a reference sample image from the sample image storage unit 16, and among the reference sample images, the nucleus corresponding to the nucleus of the target cell. Extract candidate areas. More specifically, the cell candidate region determination unit 18 extracts a pixel block of significant pixels from the reference specimen image, and extracts a circumscribed rectangular region of each pixel block as a nucleus candidate region. Here, a significant pixel is a pixel whose pixel value (RGB value) is within a predetermined range. And the cell candidate area | region determination part 18 determines a cell candidate area | region based on the nucleus candidate area | region for every nucleus candidate area | region. For example, the cell candidate region determination unit 18 determines, for each nucleus candidate region, a rectangular region having a given size including the nucleus candidate region as the cell candidate region. FIG. 7 shows an example of the pixel block 30, the nucleus candidate region 32, and the cell candidate region 34. In FIG. 7, a cell candidate region 34 including the upper left nucleus candidate region 32 is shown.

  Further, the cell candidate region determination unit 18 registers the determined cell candidate region 34 in the region candidate DB 20. That is, the cell candidate region determination unit 18 stores the determined record of the cell candidate region 34 in the region candidate DB 20. The record includes the area ID of the cell candidate area 34. The record includes coordinate data indicating the cell candidate region 34. The record also includes a numerical value (hereinafter referred to as a score) indicating the probability that the target cell is shown in the cell candidate area 34. Further, the record includes a value of a visual flag (here, “0”) indicating a user's determination result regarding whether or not the target cell is shown in the cell candidate area 34. Here, the cell candidate region determination unit 18 cuts out an image in the cell candidate region 34 from the reference sample image, and outputs an output value when the image feature amount of the cut out image is input to a discriminator learned in advance. Obtained as the score of region 34. As an example of the image feature amount, for example, a HOG (Histograms of Oriented Gradients) feature amount is raised. The greater the score value, the higher the probability that the target cell is shown in the cell candidate region 34.

  FIG. 8 is a diagram showing the stored contents of the area candidate DB 20. As shown in the figure, the region candidate DB 20 includes a region ID field, a coordinate data field, a score field, and a visual result field. The region ID field stores the region ID of the cell candidate region 34. In the coordinate data field, coordinate data indicating the cell candidate region 34 is stored in association with the region ID of the cell candidate region 34. The coordinate data includes the position coordinates of the representative point (for example, the center or the upper left vertex) of the cell candidate region 34 and the size (width W and height H) of the cell candidate region 34. The coordinate data may be data expressed in pixels or data expressed in millimeters.

  In the score field, the score of the cell candidate region 34 is stored in association with the region ID of the cell candidate region 34. In the visual result field, the value of the visual flag is stored in association with the area ID of the cell candidate area 34. The value “2” indicates that the user has determined that the target cell is shown in the cell candidate region 34, and the value “1” indicates that it is unclear whether the target cell is shown in the cell candidate region 34. The value “0” indicates that the user has determined that the user has not yet viewed the cell candidate area 34 or that the target cell is not shown in the cell candidate area 34. In addition to the visual flag, a user comment regarding the cell candidate region 34 may be stored in the visual result field.

  Note that the cell candidate region determination unit 18 may register all the determined cell candidate regions 34 in the region candidate DB 20, or if the target cell is reflected in the determined cell candidate region 34, the above-described classifier identifies the target cell. Only the cell candidate region 34 may be registered in the region candidate DB 20. Here, the cell candidate region 34 identified as having the target cell reflected is, for example, a cell candidate region 34 having a score equal to or higher than a threshold, or a score “1” to “N” (where “N” is “ 200 ") is the second largest cell candidate region 34.

  Hereinafter, each cell candidate region 34 registered in the region candidate DB 20 will be referred to as a cell candidate region X, and the description will be continued.

  The candidate area image acquisition unit 22 acquires an image of the cell candidate area X in each of the plurality of sample images captured by the imaging unit 12 as a candidate area image. Specifically, the candidate area image acquisition unit 22 cuts out an image of the cell candidate area X as a candidate area image from each sample image stored in the sample image storage unit 16. The candidate area image is image data, and the focal distance when the specimen image including the candidate area image is captured is included as header information.

  The visual confirmation information generation unit 24 generates visual confirmation information based on each of the plurality of candidate area images acquired by the candidate area image acquisition unit 22. Specifically, the visual confirmation information generation unit 24 generates information including a plurality of candidate region images acquired by the candidate region image acquisition unit 22 as visual confirmation information, and uses the generated visual confirmation information as a cell candidate region. The information is stored in the visual confirmation information storage unit 26 in association with the region ID X.

  The visual confirmation information display unit 28 displays an image of the cell candidate region X in the reference sample image on the display device 6. Specifically, the visual confirmation information display unit 28 displays an image of the cell candidate region X in the reference sample image on the display device 6 when the button 7a is selected (see FIG. 2A).

  The visual confirmation information display unit 28 displays a plurality of candidate region images acquired by the candidate region image acquisition unit 22 on the display device 6 based on the visual confirmation information. For example, when the button 7b is selected, the visual confirmation information display unit 28 displays a plurality of candidate region images included in the visual confirmation information on the display device 6 (see FIG. 2B). Further, for example, the visual confirmation information display unit 28 displays a plurality of candidate region images included in the visual confirmation information on the display device 6 when the button 10a is selected and the cell candidate region X is selected by the user. (See FIG. 4A). Further, for example, when the button 10c is selected and the cell candidate region X is selected by the user, the visual confirmation information display unit 28 displays a plurality of candidate region images included in the visual confirmation information in the reference specimen image. The image is displayed on the display device 6 together with the peripheral region image including the cell candidate region X (see FIG. 4C).

  Here, the information display part 28 for visual confirmation displays each candidate area | region image with the information which shows a focal distance when each sample image is imaged. Specifically, the visual confirmation information display unit 28 displays each candidate area image in association with the focal length included in the candidate area image (see FIGS. 2B, 4A, and 4C). Further, the visual confirmation information display unit 28 displays each candidate region image in a list in an order of alignment (for example, ascending or descending order of focal length) according to the focal length when each specimen image is captured (FIG. 2B). FIG. 4A and FIG. 4C). The visual confirmation information display unit 28 displays each candidate area image in order in a display order (for example, ascending or descending order of the focal distance) according to the focal length included in each candidate area image. May be. The state where each candidate area image is displayed one by one is shown. Arrows indicate transitions of displayed candidate area images.

  Further, the visual confirmation information display unit 28 receives an input of a user's judgment result regarding whether or not the target cell is shown in the cell candidate region X. Here, the visual confirmation information display unit 28 determines whether or not the target cell is shown in the cell candidate region X when the visual result input button 11 (see FIGS. 2A and 2B) related to the cell candidate region X is clicked. The input of the user's judgment result about the is accepted.

  Further, the visual confirmation information display unit 28 stores information indicating the received determination result of the user in the visual result field in association with the region ID of the cell candidate region X. Here, when receiving the input of the determination result that the target cell is reflected in the cell candidate region X, the visual confirmation information display unit 28 is stored in the visual result field in association with the region ID of the cell candidate region X. The value of the visual flag is updated to “2”, and when the input of the determination result that it is unclear whether the target cell is shown in the cell candidate region X is received, the value of the visual flag is updated to “1”. .

  And extraction of a target cell will be performed based on the memory content of area | region candidate DB20.

  As described above, in the image processing device 4, the visual confirmation information display unit 28 displays, on the display device 6, an image of the cell candidate region 34 when a sample is imaged at each focal length. The Therefore, the user can determine whether or not the object in the cell candidate region 34 is a target cell by referring to each image. In addition, since the images are displayed in the order according to the focal length, it becomes easy to grasp the change in the size of the core portion at a glance, and the above-described determination is facilitated.

  FIG. 10 is a flowchart illustrating an example of processing executed by the candidate area image acquisition unit 22, the visual confirmation information generation unit 24, and the visual confirmation information display unit 28 in the image processing device 4. In S101, the image processing apparatus 4 reads a plurality of sample images stored in the sample image storage unit 16. In S101, the image processing apparatus 4 reads the coordinate data of each of the plurality of cell candidate regions registered in the region candidate DB 20.

  Then, the image processing device 4 initializes the visually confirmed region list stored in the storage unit (S102). Then, the image processing apparatus 4 executes steps S103 to S106 with each cell candidate region registered in the region candidate DB 20 as a cell candidate region (i).

  That is, in S103, the image processing apparatus 4 acquires the image of the cell candidate region (i) in each specimen image as the candidate region image by the candidate region image acquisition unit 22. In step S <b> 103, the image processing apparatus 4 generates visual confirmation information including a plurality of acquired candidate area images from the visual confirmation information generation unit 24 and stores the information in the visual confirmation information storage unit 26.

  For example, when the cell candidate region (i) is selected by the user, the image processing device 4 reads the visual confirmation information stored in the visual confirmation information storage unit 26 by the visual confirmation information display unit 28. Each of the plurality of candidate area images included in the visual confirmation information is displayed on the display device 6 (S104). For example, in S104, the image processing apparatus 4 displays a list of each candidate area image in an arrangement order corresponding to the focal length included in each candidate area image (see FIG. 4A). Further, for example, in S104, the image processing apparatus 4 displays each candidate area image in order in a display order corresponding to the focal length included in each candidate area image (see FIG. 9). Further, for example, in S104, the image processing apparatus 4 displays each candidate area image together with an image of a peripheral area including the cell candidate area (i) in the reference specimen image (see FIG. 4C).

  Then, the image processing apparatus 4 receives an input of a determination result of the user regarding whether or not the target cell is shown in the cell candidate region (i) by the visual confirmation information display unit 28. Further, based on the received input, it is determined whether or not the user has determined that the target cell is shown in the cell candidate region (i) (S105).

  When the user determines that the target cell is shown in the cell candidate area (i) (Y in S105), the value of the visual flag stored in the visual result field in association with the area ID of the cell candidate area (i) is set to “ Update to 2 ”. Further, the cell candidate area (i) is registered in the visually confirmed area list (S106).

  On the other hand, when the user determines that the target cell is not shown in the cell candidate region (i) (N in S105), the image processing apparatus 4 sets another cell candidate region as the cell candidate region (i) and the steps after S103. Execute.

  Finally, in S107, the image processing apparatus 4 outputs a visually confirmed region list.

  In addition, embodiment of this invention is not restricted only to the said embodiment.

  For example, the case where nucleated red blood cells are target cells has been described above, but cells other than nucleated red blood cells may be target cells. That is, the present invention can also be applied when cells other than nucleated red blood cells are target cells.

  In addition, for example, the image processing apparatus 4 does not perform detailed display in the display mode (2) when the button 10b is selected (see FIG. 4B), but performs detailed display in the display mode (4). Also good. FIG. 11 is a diagram for explaining the display mode (4). As shown in the figure, in the display mode (4), each image of the peripheral area at each focal length is displayed in order.

  DESCRIPTION OF SYMBOLS 1 Image processing system, 2 Optical microscope, 4 Image processing apparatus, 6 Display apparatus, 7a, 7b, 10a, 10b, 10c Button, 11 Visual result input button, 12 Imaging part, 14 Specimen image acquisition part, 16 Specimen image storage part , 18 Cell candidate region determination unit, 20 region candidate DB, 22 candidate region image acquisition unit, 24 visual confirmation information generation unit, 26 visual confirmation information storage unit, 28 visual confirmation information display unit, 30 pixel block, 32 nuclei Candidate region, 34 cell candidate region.

Claims (7)

Imaging means for imaging a sample containing target cells at each of a plurality of focal lengths;
An acquisition unit that acquires an image of a cell candidate region that is a candidate for a region in which the target cell is captured in each of the captured images captured by the imaging unit;
Display means for displaying the plurality of images acquired by the acquisition means;
Means for receiving an input of a user's judgment result regarding whether or not the target cell is shown in the cell candidate region;
Including an image processing system. The display means includes
Displaying a list of the plurality of images in an order of alignment according to the focal length when each captured image is captured;
The image processing system according to claim 1. The display means includes
Displaying each of the plurality of images sequentially in a display order corresponding to the focal length when each captured image is captured;
The image processing system according to claim 1. The display means includes
Displaying the plurality of images together with an image of a peripheral region including the cell candidate region in any captured image;
The image processing system according to any one of claims 1 to 3. The display means includes
Displaying the plurality of images together with information indicating a focal length when each captured image is captured;
The image processing system according to claim 1, wherein: An acquisition unit that acquires an image of a cell candidate region that is a candidate for a region in which the target cell is captured in each of the captured images obtained by imaging the sample including the target cell at each of a plurality of focal lengths;
Display means for displaying the plurality of images acquired by the acquisition means;
Means for receiving an input of a user's judgment result regarding whether or not the target cell is shown in the cell candidate region;
An image processing apparatus. An acquisition unit that acquires an image of a cell candidate region that is a candidate for a region in which the target cell is captured in each of the captured images obtained by imaging the sample including the target cell at each of a plurality of focal lengths,
Display means for displaying the plurality of images acquired by the acquisition means;
Means for receiving an input of a determination result of a user regarding whether or not the target cell is shown in the cell candidate region;
As a program to make the computer function as.

Images