JP2011010621A - Image processing method in observation of cultured product, image processing program and image processor - Google Patents

Abstract

Provided is an image processing means capable of reliably detecting a region of a culture medium from a culture container even when the culture medium changes color with changes in time and environment.
An observation image obtained by capturing an image of a culture medium in a culture container located in an observation field with an imaging device is obtained for a culture container having a culture medium used for culturing the culture inside, and is copied to the observation image. Color space information in the sampling area is set in advance by setting the sampling area specified by the observer as one area of the culture medium and the background part from the area of the culture medium in the culture vessel and the background part excluding the culture medium. Based on the above, by estimating the color distribution of the medium region in the observation image by statistical processing, the region of the medium and the background portion are discriminated to identify the medium region.
[Selection] Figure 1

Description

  The present invention relates to an image processing means for detecting a region of a medium used for culturing a culture from a culture container.

  In recent years, with the development of assisted reproduction technology (ART), it has been practiced to observe the growth state of cultured fertilized eggs by in vitro fertilization. A culture microscope is mentioned as an example of the apparatus which observes the conditions of cultures, such as a fertilized egg (for example, refer patent document 1). The culture microscope includes a culture device (invecutor) that forms a suitable environment for culturing fertilized eggs, and a microscopic observation system that microscopically observes the state of the fertilized eggs in a culture container housed in the culture device. The observation image of the fertilized egg is acquired at regular intervals, and the user can recognize the fertilized egg by visual observation, and then automatically observe, record, and manage the fertilized egg growth state. The

  In such an apparatus, when observing the growth state of a fertilized egg in a culture container, it is necessary to first detect the fertilized egg to be observed. Then, after detecting the medium immersed in mineral oil (medium drop) in the culture container from the whole observation image, a plurality of microscopic observation images of the medium are obtained while changing the observation visual field position in the culture medium. The flow is such that fertilized eggs, which are observation objects injected into the culture medium one by one based on images, are detected as being distinguished from other foreign substances. In order to recognize a fertilized egg in this way, microscopic observation in a high-magnification visual field is generally required. However, as the observation magnification increases, the observation visual field (observation range) becomes narrower. In contrast, a large number of microscopic observation images need to be taken. For this reason, if the area of the medium can be detected in advance by macro observation prior to detection of the fertilized egg by micro observation, the area for acquiring the micro observation image (high-magnification image) in the micro observation is also limited. It becomes possible to shorten.

JP 2004-229619 A

  The area of the medium contained in the culture container is visually determined by the observer himself from the entire observation image of the culture container. When observing in time series, it is necessary to specify the medium every time. there were. However, the culture medium itself is very light in color and has the characteristics that the color changes (pH changes) with the passage of time and the environment. Therefore, the culture is based on simple colors that are determined uniformly. It was difficult to accurately detect the culture medium by accurately discriminating between the plurality of culture media contained in the container and the background portion (such as transparent mineral oil having almost no color).

  The present invention has been made in view of such problems, and it is possible to reliably detect a region of a culture medium from a culture container even when the culture medium changes color with changes in time or environment. It is an object of the present invention to provide an image processing means that is made possible.

  According to the first aspect exemplifying the present invention, with respect to a culture container having a culture medium used for culturing a culture inside, an observation of the culture medium in the culture container located in the observation visual field taken by an imaging device Sampling areas designated by the observer as one area of the culture medium and the background part are preset from the area of the culture medium in the culture vessel and the background part excluding the culture medium, which are captured in the observation image. Then, based on the color space information in the sampling region, by estimating the color distribution of the medium region in the observation image by statistical processing, the region of the medium and the background portion is discriminated to identify the medium region A culture observation image processing method is provided.

  According to a second aspect exemplifying the present invention, there is provided an image processing program for causing a computer to function as an image processing device that can be read by a computer and that is captured by an imaging device and acquires an image and performs image processing. A step of acquiring an observation image obtained by photographing the medium in the culture container located in the observation visual field with an imaging device for the culture container having a medium used for culturing the culture inside, and imprinting in the observation image A step of recording a sampling area designated by the observer as one area of the medium and the background part from the area of the culture medium in the culture vessel and the background part excluding the medium, and color space information in the sampling area Based on the above, the color distribution of the area of the culture medium in the observation image is estimated by statistical processing, thereby distinguishing the area of the culture medium from the background part. And identifying a region of the medium, the culture observation image processing program, characterized in that to achieve and outputting the identification result about the medium to the computer is provided.

  According to the third embodiment illustrating the present invention, an imaging device for photographing a culture container having a medium used for culturing a culture inside, a medium and a medium in the culture container imaged in the observation image, Based on the color space information in the sampling area and the storage unit that records the sampling area specified by the observer as one area of the culture medium and the background part from the area with the removed background part, observation by statistical processing By estimating the color distribution of the area of the medium in the image, the area between the medium and the background part is discriminated, and the image analysis unit for identifying the area of the medium and the identification result regarding the medium determined by the image analysis unit are externally displayed. An image processing apparatus for observing a culture is provided, characterized in that the image processing apparatus is configured to include an output unit that outputs to the medium.

  According to such an image processing method, an image processing program, and an image processing apparatus for observing a culture, even if the culture medium in the culture container changes color with the passage of time or environment, By accurately extracting the color distribution, it is possible to detect the medium region in the culture vessel with high accuracy.

It is a flowchart which shows the outline | summary of an image processing program. It is a general | schematic block diagram of the culture observation system shown as an example of application of this invention. It is a block diagram of the said culture observation system. (A) is a top view of a culture container, (B) is a perspective view which shows a dish. It is a structural example of the display image of a user interface. It is a figure which shows HSV color space. These are an H image, an S image, and a V image when the observation image is HSV separated. It is a graph which shows the distribution state of hue H and saturation S of a sampling area. 1 is a block diagram illustrating a schematic configuration of an image processing apparatus. It is a figure which illustrates the output result displayed on a display panel.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. As an example of a system to which the image processing apparatus according to the present embodiment is applied, a schematic configuration diagram and a block diagram of a culture observation system are shown in FIGS. 2 and 3, respectively.

  The culture observation system BS roughly observes the culture chamber 2 provided in the upper part of the housing 1, the shelf-like stocker 3 that accommodates and holds a plurality of culture containers 10, and the sample in the culture container 10. An observation unit 5, a transfer unit 4 for transferring the culture vessel 10 between the stocker 3 and the observation unit 5, a control unit 6 for comprehensively controlling the operation of the system, and an operation panel 7 provided with an image display device. Etc.

The culture chamber 2 is a chamber for forming a culture environment, and is kept sealed after the sample is charged in order to prevent environmental changes and contamination. Along with the culture chamber 2, a temperature adjustment device 21 that raises and lowers the temperature in the culture chamber 2, a humidifier 22 that adjusts humidity, and a gas supply device 23 that supplies a gas such as CO ₂ gas or N ₂ gas. A circulation fan 24 for making the entire environment of the culture chamber 2 uniform, an environmental sensor 25 for detecting the temperature, humidity, carbon dioxide concentration, etc. of the culture chamber 2 are provided. The operation of each device is controlled by the control unit 6, and the culture environment defined by the temperature, humidity, carbon dioxide concentration, etc. of the culture chamber 2 is maintained in a state that matches the culture conditions set on the operation panel 7.

  The stocker 3 is formed in a shelf shape that is partitioned into a plurality of parts in the front-rear direction and the vertical direction in FIG. A unique address is set for each shelf. For example, when the front-rear direction is A to C rows and the vertical direction is 1 to 7 rows, the A-row 5 rows shelf is set as A-5. The

  As the culture vessel 10, an appropriate one such as a flask, a dish, or a well plate is selected according to the type and purpose of the culture. In this embodiment, as shown in FIG. The structure provided with the dish 10a and the holder 10b holding the dish 10a is illustrated, and as shown in FIG. 4 (B), the fertilized egg J as a culture is a medium containing a pH indicator such as phenol red. It is injected into each dish 10a together with the drop A. On the bottom of the dish 10a, one or more medium drops A of about 20 μl dropped by a pipette or the like are formed (three are shown in FIG. 4B), and the medium drop A is colorless in the dish 10a. It is in a state of being soaked with transparent mineral oil O. In each medium drop A, for example, one fertilized egg J collected at the same time from the same mother for external fertilization is inserted. The culture vessel 10 is assigned a code number and is stored in association with the designated address of the stocker 3.

  The transfer unit 4 is provided inside the culture chamber 2 so as to be movable in the vertical direction and is moved up and down by the Z-axis drive mechanism. The transfer unit 4 is attached to the Z stage 41 so as to be movable in the front-rear direction and is moved by the Y-axis drive mechanism. The Y stage 42 that is moved back and forth, the X stage 43 that is attached to the Y stage 42 so as to be movable in the left-right direction and is moved left and right by the X-axis drive mechanism, and the like, is supported by lifting the culture vessel 10 to the tip side of the X stage 43 A support arm 45 is provided. The transport unit 4 has a moving range in which the support arm 45 can move between the entire shelf of the stocker 3 and the observation unit 5. The X-axis drive mechanism, the Y-axis drive mechanism, and the Z-axis drive mechanism are configured by, for example, a servo motor with a ball screw and an encoder, and the operation thereof is controlled by the control unit 6.

  The observation unit 5 includes a first illumination unit 51 that illuminates the sample from the lower side of the sample stage 15, a second illumination unit 52 that illuminates the sample from above the sample stage 15 along the optical axis of the microscopic observation system, and a sample from below. 3, a macro observation system 54 that performs macro observation of the sample, a micro observation system 55 that performs micro observation of the sample, an image processing apparatus 100 (see FIG. 9), and the like. The sample stage 15 is made of a light-transmitting material and has a transparent window 16 in the observation area. The sample stage 15 is composed of a fine drive stage that can be moved in the XY direction (horizontal plane direction) and the Z direction (vertical direction) by operation control from the control unit 6, and the culture vessel 10 placed on the upper surface thereof. Is moved in the XY directions, so that the culture vessel 10 can be inserted on the optical axis of the macro observation system 54 or on the optical axis of the microscopic observation system 55.

  The first illumination unit 51 is a surface-emitting light source provided on the lower frame 1 b side, and backlight-illuminates the entire culture vessel 10 from the lower side of the sample stage 15. The second illumination unit 52 includes a light source such as an LED and an illumination optical system including a phase ring, a condenser lens, and the like. The second illumination unit 52 is provided in the culture chamber 2 and receives light from the microscopic observation system 55 from above the sample stage 15. The sample in the culture vessel 10 is illuminated along the axis. The third illumination unit 53 superimposes light emitted from each of the light sources, such as a plurality of LEDs and mercury that emit light having a wavelength suitable for epi-illumination observation and fluorescence observation, on the optical axis of the microscopic observation system 55. An illumination optical system composed of a beam splitter, a fluorescent filter, and the like to be disposed in the lower frame 1b located below the culture chamber 2, and the light of the microscopic observation system 55 from below the sample stage 15 The sample in the culture vessel 10 is illuminated along the axis.

  The macro observation system 54 includes an observation optical system 54 a and an imaging device 54 c such as a CCD camera that captures an image of the sample imaged by the observation optical system 54 a and is positioned above the first illumination unit 51. Provided in the culture chamber 2. The macro observation system 54 captures an entire observation image (macro image) from above the culture vessel 10 that is backlit by the first illumination unit 51.

  The microscopic observation system 55 includes an observation optical system 55a composed of an objective lens, an intermediate zoom lens, a fluorescent filter, and the like, and an imaging device 55c such as a cooled CCD camera that takes an image of a sample imaged by the observation optical system 55a. And disposed inside the lower frame 1b. The second illumination unit 52 and the microscopic observation system 55 constitute a phase difference observation microscope. A plurality of objective lenses and intermediate zoom lenses are provided, and are configured to be set to a plurality of magnifications using a displacement mechanism such as a revolver or a slider (not shown in detail). In this embodiment, at least two magnifications for low-magnification observation and high-magnification observation are switched so that magnification can be changed. The microscopic observation system 55 displays the sample in the culture vessel 10 such as a phase difference image by the transmitted light of the sample illuminated by the second illumination unit 52 and a fluorescence image by the fluorescence emitted from the sample illuminated by the third illumination unit 53. A microscopic observation image (micro image) is taken.

  The image processing apparatus 100 performs A / D conversion on signals input from the imaging device 54c of the macro observation system 54 and the imaging device 55c of the microscopic observation system 55, and performs various image processes to perform an entire observation image or a microscopic observation image. Image data is generated. Further, the image processing apparatus 100 performs color space information conversion and image analysis on the image data of these observation images (entire observation image and microscopic observation image), and identifies the medium drop A and the fertilized egg J. . Specifically, the image processing apparatus 100 is constructed by executing an image processing program stored in the ROM of the control unit 6 described below. The image processing apparatus 100 will be described in detail later.

  The control unit 6 includes a CPU 61 that executes processing, a ROM 62 that stores and stores control programs and control data of the culture observation system BS, a RAM 63 that temporarily stores observation conditions and image data, and the like. Control the operation. Therefore, as shown in FIG. 3, the constituent devices of the culture chamber 2, the transport device 4, the observation unit 5, and the operation panel 7 are connected to the control unit 6. In the RAM 63, the environmental conditions of the culture chamber 2 according to the observation program, the observation schedule, the observation type and observation position in the observation unit 5, the observation magnification, and the like are set and stored. Further, the RAM 63 is provided with an image data storage area for recording image data photographed by the observation unit 5, and index data including the code number of the culture vessel 10 and photographing date / time are associated with the image data and recorded. Is done.

  The operation panel 7 is provided with an operation panel 71 provided with input / output devices such as a keyboard and a switch, and a display panel 72 for displaying an operation screen, image data, and the like. An operation command or the like is input. The communication unit 65 is configured in accordance with a wired or wireless communication standard, and data can be transmitted to and received from a computer or the like externally connected to the communication unit 65.

  In the culture observation system BS schematically configured as described above, the CPU 61 controls the operation of each part based on the control program stored in the ROM 62 according to the setting conditions of the observation program set on the operation panel 7, and the culture vessel 10 The sample inside is automatically captured. That is, when the observation program is started by a panel operation on the operation panel 71 (or a remote operation via the communication unit 65), the CPU 61 reads each condition value of the environmental conditions stored in the RAM 63, and from the environment sensor 25. The environmental state of the culture chamber 2 to be input is detected, and the temperature adjustment device 21, the humidifier 22, the gas supply device 23, the circulation fan 24, etc. are operated according to the difference between the condition value and the actual measurement value. Feedback control is performed on the culture environment such as temperature, humidity, and carbon dioxide concentration.

  Further, the CPU 61 reads the observation conditions stored in the RAM 63 and operates the X, Y, Z stages 41, 42, 43 of the transport unit 4 based on the observation schedule to observe the culture vessel 10 to be observed from the stocker 3. The sample is transported to the sample stage 15 of the unit 5 and observation by the observation unit 5 is started. For example, when the observation set in the observation program is macro observation, the culture vessel 10 transported from the stocker 3 by the transport unit 4 is positioned on the optical axis of the macro observation system 54 and placed on the sample stage 15. Then, the light source of the first illumination unit 51 is turned on, and the entire observation image is taken by the imaging device 54c from above the culture vessel 10 that is backlit. The signal input from the imaging device 54c to the control unit 6 is processed by the image processing device 100 to generate a whole observation image, and the image data is stored in the image data storage area of the RAM 63 together with index data such as the shooting date and time. Is done.

  In addition, when the observation set in the observation program is micro observation of the sample at a specific position in the culture vessel 10, the specific position in the culture vessel 10 that has been transported by the transport unit 4 is indicated by the light of the microscopic observation system 55. Position on the axis and place on the sample stage 15, turn on the light source of the second illumination unit 52 or the third illumination unit 53, and cause the imaging device 55c to take a microscopic observation image by transmitted illumination, epi-illumination, and fluorescence. . A signal photographed by the imaging device 55c and inputted to the control unit 6 is processed by the image processing device 100 to generate a microscopic observation image (phase difference image, fluorescent image, etc.), and the image data is an index such as a photographing date / time. Stored in the image data storage area of the RAM 63 together with the data.

  The CPU 61 sequentially executes the above-described whole observation image photographing and microscopic observation image photographing based on the observation schedule set in the observation program. The image data stored in the RAM 63 is read from the RAM 63 in response to an image display command input from the operation panel 71. For example, an entire observation image, a microscopic observation image at a specified time, an analysis result of image analysis, and the like are displayed on the display panel 72. Is displayed.

  In the culture observation system BS configured as described above, when observing the growth state of the fertilized egg J in the culture vessel 10 (dish 10a), the fertilized egg to be observed must be detected. In general, the culture vessel 10 is photographed by the macro observation system 54 to obtain a whole observation image (macro image), and after the medium drop A is detected from the macro image, the inside of the medium drop A is microscopically observed. The fertilized egg J, which is an observation object that has been photographed in accordance with 55 and injected into the medium drop A one by one, is discriminated as another foreign substance and the fertilized egg is identified. In order to recognize the fertilized egg J, it is generally necessary to perform microscopic observation in a high-magnification visual field. However, as the observation magnification increases, the observation visual field (observation range) becomes narrower. Need to shoot. For this reason, prior to detection of a fertilized egg by microscopic observation, if the region of medium drop A containing the fertilized egg can be detected in advance by macro observation, the region for acquiring a high-magnification image in microscopic observation is also limited, and the observation time It becomes possible to shorten the whole.

  Conventionally, the designation of the medium drop A contained in the culture vessel 10 has been visually determined by the observer himself / herself from a macro image by the macro observation system 54. It was necessary to specify it every time. However, since the medium drop A itself is very light in color and has a characteristic that the color changes (pH changes) with the passage of time, the culture container 10 is based on a simple color determined uniformly. It was difficult to accurately detect the medium drop A by accurately discriminating between the plurality of medium drops A contained therein and the background portion thereof (such as mineral oil O which is transparent and has almost no color) B.

  In order to correct such a problem, the image processing method for observing the culture by the image processing apparatus 100 samples the color space information of the medium drop A and the background portion B from the macro image obtained by photographing the entire culture container 10, By estimating the color distribution of the medium drop A by statistical processing, all the areas of the plurality of medium drops A are automatically detected. In the following, this image processing method will be described from the basic concept.

(Specify sampling area for medium and background)
First, sampling is performed to create a color frequency distribution of the medium drop A and the background portion B based on the color space information of the observation image. FIG. 5 illustrates a display image of the user interface displayed on the display panel 72, which is displayed on the display panel 72 based on a bird view image (entire observation image of the culture vessel 10) taken by the macro observation system 54. In the “observation image” frame 720, the observer uses at least one region of the medium drop A and the background portion B in the “observation image” frame 720 using a mouse or the like provided on the operation panel 71 or the like. Specify one by one. FIG. 5 shows a state in which partial areas (rectangular areas) of the medium drop A and the background part B in the lower right dish 10a are designated one by one. In order to increase the accuracy of determination described later, it is desirable to make the sampling area as wide as possible.

(HSV separation)
For the captured entire observation image (color image), the color space information of the image is converted from the normal RGB color space to the HSV color space by, for example, arithmetic processing using a known conversion formula or a conversion table. Separate each component. The HSV color space is a color space represented by three components of hue (Hue), saturation (Saturation), and lightness (Value), and can be represented by a conical coordinate system as shown in FIG. The medium drop A and the background part B are considered to have almost the same lightness V in the HSV color space, so that they are discriminated from each other by the remaining two hues H and saturation S without being affected by the change in brightness. can do. Therefore, by limiting the difference in color distribution between the medium drop A and the background portion B to the representation of the two-dimensional color space (HS plane), the processing load of the medium drop A extraction can be reduced. In the following description, the values of hue H, saturation S, and lightness V may be referred to as H value, S value, and V value.

  FIG. 7 shows an image obtained by HSV separation of the entire observation image. The hue H generally returns -180 to 180 values, and has a periodic boundary condition (after 180, returns to -180). Therefore, the average value of the sampled hue H is converted into a distribution having a central value. There is a need to. In the present embodiment, considering that the image gradation is 256 gradations from 0 to 255, the hue H value returns 0 to 180, so the average value of the sampled hue H is 90 (0 to 0). Offset processing is performed so that the center value becomes 180).

Further, as a random noise reduction, a smoothing Gaussian filter is applied to both the H image and the S image. The Gaussian filter is an averaging filter that calculates a rate using a Gaussian distribution function (Gaussian function) so that the weight closer to the target pixel is increased and the weight is decreased as the distance is increased. Here, the weight of the Gaussian filter is determined by a Gaussian function G (x, y) defined by the following equation (1), where (x, y) is the center coordinate of the pixel of interest.
Here, the parameter σ is a value indicating the spatial spread width of the Gaussian filter, the filter size is a radius 3σ (σ = 1), and in this embodiment, a 7 × 7 matrix including the central pixel is exemplified.

(Estimated color distribution of medium and background)
As the color distribution frequency of the medium drop A and the background portion B, the hue H and the saturation S given to each pixel of the image are considered to have a normal distribution. Therefore, in each pixel in the sampling region of the medium drop A and the background portion B, a two-dimensional normal distribution is estimated for each of the two components of hue H and saturation S. The density function f (x, y) of the two-dimensional normal distribution has an average value of each distribution of hue H and saturation S as μH and μS, standard deviations as σH and σS, offset component as W, variance covariance matrix as When Σ, it is expressed as follows.
When the density function f (x, y) of this two-dimensional normal distribution is identified by A for the medium drop and B for the background part,
Medium of HS _{distribution: f A (H, S |} μ H A, μ S A, Σ A)
Background HS distribution: f _B (H, S | μ _H ^B , μ _S ^B , Σ _B )
Can be estimated.

(Distinction between medium drop and background part)
Next, using the normal distribution of the medium drop A and the background portion B estimated above, the point corresponding to the pixel is determined from the H value and S value of each pixel in the observation image (H image and S image). It is probabilistically estimated whether it belongs to the medium drop A or the background. However, here, it is not necessary to calculate all of f _A (H, S) and f _B (H, S), and it is only necessary to pay attention to the parentheses of the exponential function exp in the above formula (2).
D _A and D _B shown in the above formula (5) are generally called Mahalanobis distances, and represent distances in consideration of the variance in the distance from the normal distribution. The Mahalanobis distances D _A and D _B are calculated for each pixel by substituting the H value and S value of each pixel into the above equation. If the point corresponding to the target pixel is D _A > D _B , it belongs to the medium drop A, If D _A <D _B , it is determined to belong to the background.

FIG. 8 shows a graph showing a distribution state of the H value and S value in the sampling region (a scatter diagram in which the vertical axis indicates the H value and the horizontal axis indicates the S value), and is a line that is shown in the upper right direction in the drawing. Indicates a boundary line (discriminant curve) based on the Mahalanobis distance such that D _A = D _B. This boundary line is a straight line if the variance (dispersion covariance matrix) between the H value and the S value is the same, and a quadratic curve if the variance is different as in this embodiment. In FIG. 8, the sampling distribution of the medium drop A is distributed on the left side and the sampling distribution of the background portion B is distributed on the right side with respect to the discriminant curve. Therefore, if the point (H, S) of the target pixel is plotted on the left side with respect to the boundary line on the HS plane in the figure, D _A > D _B , and the point corresponding to this target pixel is the area of the medium drop A It is determined that it belongs to. On the other hand, if the point (H, S) of the target pixel is plotted on the right side with respect to the boundary line, D _A <D _{B is established} , and the point corresponding to this target pixel is determined to belong to the background portion B. In this way, the Mahalanobis distances D _A and D _B are calculated for the H value and S value of each pixel on the observation image, and each point corresponding to each pixel is a region having a smaller Mahalanobis distance (medium drop A or background portion). B area).

In addition to the discrimination based on the Mahalanobis distance, as shown in FIG. 8, the H value and S belonging to, for example, μ _A ± 3σ _A (σ _A : standard deviation) as the spread width of the normal distribution sampled in the HS plane. By adding a pixel region having a value as a determination condition, a point (or region) deviating from the normal distribution range of the medium drop A is rejected from the medium drop A region.

  As described above, the two-dimensional normal distribution of the hue H and the saturation S is estimated from the HSV color space information of the observation image, and the conditions are determined by using the Mahalanobis distance and the spread width of the normal distribution for each pixel. A satisfactory pixel region is determined to be a region corresponding to the medium drop A on the image.

(Post-processing)
By the way, about the area | region of the culture medium drop A detected in this way, there exists a possibility that an excess area | region may be contained by noise etc. Therefore, in order to eliminate the influence of noise and the like, the size and the outer shape of the standard medium drop A are greatly deviated based on the size (area) and outer shape (circularity) of the medium drop A. It is good also as a structure which rejects the excess area | region which exists. In that case, first, binarization processing is performed as a contour extraction method on the entire observation image (original image). In the binarization processing, based on the result of the above discriminant analysis, for example, whether or not the Mahalanobis coefficient satisfies D _A > D _B and the normal distribution spread width K-σ, in units of pixels. Judgment is made and the original image is binarized with pixel data satisfying this as “1” and other pixel data as “0”. Next, a labeling process for assigning a unique label is performed on each object thus segmented. In this labeling process, the area of the object (the region surrounded by the extracted contour) is calculated together with labeling for identifying the object. Since the size of the medium drop A (diameter of about 7 mm with a medium drop of about 20 ml) imprinted on the observation image is almost determined by the image acquisition conditions (observation magnification, etc.), it can be a target of medium drop A discrimination. An upper limit value and a lower limit value of the area of the object are set in advance, and an object outside the setting range is excluded from the labeling target.

  In addition, since the medium drop A is generally formed in a circular shape, the circularity of each labeled object is obtained, and a threshold value or the like is set in advance for an object with a low circularity. Aside from the area of medium drop A. Here, the circularity is a scale for determining the circularity of an object. The circularity is determined, for example, by determining the center of gravity of each object in the image plane and extracting the contour of each object, and obtaining the difference between the maximum value and the minimum value of the distance from its center of gravity to the contour for each object, It is determined that the larger the difference is, the greater the deviation from the circle. Further, as another index of circularity, an eccentricity is used as a second moment. The eccentricity is represented by the ratio between the focal length and the long axis radius when the object region is approximated by an ellipse.

  In this way, unnecessary objects, noise components, etc. having the same color (hue H and saturation S) by chance can be accurately removed from the area of the medium drop A extracted by the color distribution estimation. .

  According to the culture processing image processing method as described above, even when the medium drop A to be detected changes in color due to the passage of time or the environment, it is based on the sampling region of the medium drop A and the background portion B. By estimating the color distribution of the medium drop and discriminating the area between the medium drop A and the background part B by a statistical method such as Mahalanobis distance, the medium drop A can be reliably detected from the culture vessel 10. It becomes possible.

(application)
Next, a specific application of image analysis executed in the image processing apparatus 100 of the culture observation system BS will be described with reference to FIGS. Here, FIG. 1 is a flowchart showing an outline of processing in the culture observation image processing program GP, and FIG. 9 is a block diagram showing an outline configuration of an image processing apparatus 100 that executes culture observation image processing.

  In the image processing apparatus 100, the culture vessel 10 to be observed is photographed by the imaging device 54c (the image storage unit 110 that acquires and stores the macro image, and converts the acquired macro image from the RGB color space to the HSV color space. The color space conversion unit 120, the sampling region specified by the user, the color distribution storage unit 130 that stores the color distribution state of the sampling region, and the HSV converted image are analyzed to determine the color distribution of the medium drop A. And an image analysis unit 140 that discriminates the region between the medium drop A and the background portion B, and an output unit 150 that outputs the determination result analyzed by the image analysis unit 140 to the outside. For example, the detection result (identification result) of the medium drop A determined by the above is output on the display panel 72 and displayed. The image processing apparatus 100 is configured by a preset stored image processing program GP in ROM62 is read into CPU 61, processing based on the image processing program GP by CPU 61 are sequentially executed.

  As described, in the culture observation system BS, observation in the culture vessel 10 designated every predetermined time is performed according to the observation conditions set in the observation program. Specifically, the CPU 61 operates each stage of the transport unit 4 to transport the culture vessel 10 to be observed from the stocker 3 to the observation unit 5 (in this embodiment, disposed on the optical axis of the macro observation system 54). A macro image is taken by the imaging device 54c.

  The image processing apparatus 100 first acquires a macro image taken by the imaging device 54c in step S1, and stores the acquired image together with index data such as the code number, observation position, and observation time of the culture vessel 10 as an image storage. Stored in the unit 110.

  In step S2, the macro image photographed by the imaging device 54c is displayed in the “observation image” frame 720 on the display panel 72 by the user interface, and the observer displays the macro image in the “observation image” frame 720. The mouse is operated on the image to designate at least one sampling region for the medium drop A and the background portion B. Each sampling region (position and size) designated by the observer is stored in the color distribution storage unit 130 in a state identified as the medium drop A and the background portion B.

  In step S3, the color space conversion unit 120 converts the color information of the macro image from the RGB color space to the HSV color space, and the macro image is separated into H, S, and V images. In step S3, for the noise reduction process, the H, S, V image is subjected to a smoothing process in which a Gaussian filter is applied in the image analysis unit 140. In addition, the color space conversion unit 120 performs an offset process for shifting the average value to the center value with respect to the hue H that is a circulation value.

  Next, in step S4, a two-dimensional normal distribution of the H value and S value in the sampling region of the medium drop A and the background portion B is estimated. FIG. 8 illustrates the distribution state (HS plane) of hue H and saturation S of each pixel in each sampling region. For example, each sampling region designated by the observer in step S2 is 10 × 10 pixels. When it is a pixel block, 100 points (for 100 pixels) are plotted on the HS plane. The distribution state of the H value and the S value in this sampling area is recorded in the color distribution storage unit 130.

In step S5, using the normal distribution of H value and S value of the medium drops A and the background part B estimated in step S4, the Mahalanobis distance D _A for each pixel of the determination target (outside sampling region), the D _B sequentially calculate. The Mahalanobis distances D _A and D _B are calculated as distances from points (H, S) indicating the H value and S value of each pixel to the center (center of gravity) of each sampling distribution region.

Subsequently, in step S6, the region of the medium drop A and the background portion B is determined from the culture vessel 10 using a statistical method such as Mahalanobis distance. That is, in step S61, for each pixel, by comparing the Mahalanobis distance D _A for the medium drops A, the Mahalanobis distance D _B to the background portion B, D _A <media if D _B drops A, D _A> D If it is _B , it is probabilistically estimated whether the point corresponding to the pixel of interest on the observed image belongs to the medium drop A or the background part B based on the judgment criterion that it belongs to the background part B, and the area of the medium drop A To decide. Further, in step S62, it is added to the condition for detecting the medium drop A that the normal distribution of the H value and S value in the sampling region of the medium drop A belongs to the range of the spread width k-σ (for example, 3σ _A ). Thus, the region deviating from this range is rejected from the region of the medium drop A. Thus, by estimating the distribution of hue H and saturation S in the sampling region of the medium drop A, the region of the medium drop A in the culture vessel 10 (dish 10a) is determined (step S7), and finally the detected medium A detection result indicating the area of drop A is output from the output unit 150 (step S8).

  The detection result output from the output unit 150 is displayed on the display panel 72 of the operation panel 7 in the user interface. For example, the outline indicating the region of the medium drop A in the “observation image” frame 720 on the display panel 72. , A symbol “A” indicating that the medium drop A is displayed, and the outline of the medium drop A are displayed in a hue and brightness different from those of other regions. FIG. 10 shows the detection result of the culture medium drop A displayed in the “observation image” frame 720 of the display panel 72. In this embodiment, the center dish 10a and the lower right dish 10a in FIG. The case where the culture medium drop A is formed one by one is illustrated.

  The detection data output from the output unit 150 is transmitted to a computer or the like externally connected via the communication unit 65 to display the same image, or the color change (pH change) of the medium drop A or fertilization It can be configured to be used as basic data for observing the growth state of the egg J.

  Thus, the observer can immediately recognize the medium drop A by referring to the image displayed on the display panel 72 and the image displayed on a monitor such as an externally connected computer. By acquiring a microscopic observation image by the microscopic observation system 55 only for the region (avoid unnecessary shooting of only the background), the target fertilized egg J can be efficiently detected from the medium drop A.

  As described above, according to the image processing program GP of the present embodiment, the culture processing image processing method and the image processing apparatus 100 configured by executing the image processing program GP, the medium drop A is Even when the color changes with time, the color drop of the medium drop A is estimated from the sampling of the medium drop and the background to determine the medium drop A and the background. It is possible to provide an image processing means that can stably detect the image with high accuracy.

  In this embodiment, when observing a time-series change in the growth state of the fertilized egg J or the like, if the inside of the culture container 10 is observed at a predetermined time interval, the medium drop A responds to the passage of time or environmental changes. Along with this, a color change is caused by a pH change. At that time, the observer does not need to designate the sampling area of the medium drop A and the background part B each time, and the sampling area initially designated by the observer is continuously stored in the color distribution storage unit 130. Thus, it is possible to efficiently detect the region of the medium drop A in time series by estimating the color distribution based on the sampling within the region.

  In the above-described embodiment, the processing method for estimating the color distribution based on the HSV color space (hue H, saturation S, and brightness V) is exemplified as the color space of the observation image. However, the present invention is not limited to this embodiment. The same effect can be obtained even when applied to any color space such as RGB color space, L * a * b color space, YUV color space, etc. Can do.

In the above-described embodiment, the estimation of the two-dimensional normal distribution based on the two components of the hue H and the saturation S in the HSV color space is exemplified, but the present invention is not limited to this, and the remaining brightness V is added. The same effect can be obtained by applying estimation of a three-dimensional normal distribution based on three components. This three-dimensional offset component W and the variance-covariance matrix Σ are
The probability density function f (H, S, V) of the three-dimensional normal distribution is
And can be determined based on the Mahalanobis distance.

  Furthermore, in the above-described embodiment, the discrimination method based on the Mahalanobis distance is exemplified as the statistical method. However, the statistical method is not limited to this, and other statistical methods such as Euclidean distance, discriminant analysis, maximum likelihood estimation method, etc. The same effect can be obtained when the method is applied to the estimation of the color distribution of the medium drop A.

BS culture observation system GP image processing program A medium drop B background portion J fertilized egg 5 observation unit 6 control unit 7 operation panel 10 culture vessel 10a dish 54 macro observation system 54c imaging device 61 CPU 62 ROM
63 RAM 100 Image processing device 120 Color space conversion unit 130 Color distribution analysis unit 140 Image analysis unit 150 Output unit

Claims (12)

For a culture container having a medium used for culturing the culture inside, obtain an observation image obtained by photographing the medium in the culture container located in the observation field of view with an imaging device,
From the areas of the culture medium in the culture container and the background portion excluding the culture medium, which are copied in the observation image, sampling regions designated by the observer as one area of the culture medium and the background portion are previously stored. Set,
Based on the color space information in the sampling region, by estimating the color distribution of the medium region in the observation image by statistical processing, the region of the medium and the background portion is determined, A culture observation image processing method characterized by identifying a region.   The culture processing image processing method according to claim 1, wherein any one of Mahalanobis distance, discriminant analysis, and maximum likelihood analysis is used for the statistical processing. The observation image is composed of an HSV color space expressed by hue, saturation and brightness,
The culture processing image processing method according to claim 1, wherein the color distribution is created based on the hue and the saturation that do not depend on the brightness.   When observing the inside of the culture vessel in time series at a predetermined time interval, the designated sampling area is fixed by an observer, and the observation is performed under the same sampling area at each observation time. The culture processing image processing method according to claim 1, wherein the plurality of culture media are identified by estimating a color distribution in the entire observation image. An image processing program for causing a computer to function as an image processing apparatus that is readable by a computer, acquires an image captured by an imaging device, and performs image processing,
Acquiring an observation image obtained by photographing the culture medium in the culture container located in the observation visual field with an imaging device for a culture container having a culture medium used for culturing the culture inside;
Record the sampling area designated by the observer as one area of the culture medium and the background part from the area of the culture medium in the culture container and the background part excluding the culture medium, which are copied in the observation image. And steps to
Based on the color space information in the sampling region, by estimating the color distribution of the medium region in the observation image by statistical processing, the region of the medium and the background portion is determined, Identifying a region;
An image processing program for observing a culture, which causes the computer to realize a step of outputting an identification result relating to the culture medium.   6. The culture processing image processing program according to claim 5, wherein any one of Mahalanobis distance, discriminant analysis, and maximum likelihood analysis is used for the statistical processing. The observation image is composed of an HSV color space expressed by hue, saturation and brightness,
The culture processing image processing program according to claim 5 or 6, wherein the color distribution is created based on the hue and the saturation that do not depend on the lightness.   When observing the inside of the culture vessel in time series at a predetermined time interval, the designated sampling area is fixed by an observer, and the observation is performed under the same sampling area at each observation time. The image processing program for culture observation according to any one of claims 5 to 7, wherein the plurality of culture media are identified by estimating a color distribution in the entire observation image. An imaging device for photographing a culture container having a medium used for culture of the culture inside;
Record the sampling area designated by the observer as one area of the culture medium and the background part from the area of the culture medium in the culture container and the background part excluding the culture medium, which are copied in the observation image. A storage unit to
Based on the color space information in the sampling region, by estimating the color distribution of the medium region in the observation image by statistical processing, the region of the medium and the background portion is determined, An image analysis unit for identifying an area;
An image processing apparatus for observing a culture, comprising: an output unit configured to output an identification result relating to the medium determined by the image analysis unit to the outside.   The culture processing image processing apparatus according to claim 9, wherein any one of Mahalanobis distance, discriminant analysis, and maximum likelihood analysis is used for the statistical processing. The observation image is composed of an HSV color space expressed by hue, saturation and brightness,
The image processing apparatus for culture observation according to claim 9 or 10, wherein the color distribution is created based on the hue and the saturation that do not depend on the brightness.   When observing the inside of the culture vessel in time series at a predetermined time interval, the designated sampling area is fixed by an observer, and the observation is performed under the same sampling area at each observation time. The culture processing image processing apparatus according to claim 9, wherein the plurality of culture media are identified by estimating a color distribution in an entire observation image.