JP2015191362A - Image data generation apparatus and image data generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a new technology of generating image data suitable for observation or diagnosis of a subject, by means of image processing, from original image data obtained by capturing the subject.SOLUTION: Image data generation apparatus includes: feature image data generation means which generates feature image data corresponding to image data which is formed by extracting or enhancing a difference between multiple pieces of point-of-view image data different in line-of-sight direction with respect to a subject, by use of original image data obtained by capturing the subject; and correction means which performs correction processing for reducing luminance on pixels corresponding to a low-luminance area in the original image data at least, of the pixels in the feature image data, to generate correction image data.

Description

  The present invention relates to an image data generation apparatus and an image data generation method for generating image data suitable for observation and diagnosis from image data obtained by photographing a subject.

In the field of pathology, as an alternative to an optical microscope, which is a tool for pathological diagnosis, there is a virtual slide system that enables imaging of a test sample placed on a slide and digitizes it to enable pathological diagnosis on a display. By digitizing pathological diagnosis images using a virtual slide system, conventional optical microscope images of specimens can be handled as digital data. As a result, advantages such as rapid remote diagnosis, explanation to patients using digital image data, sharing of rare cases, efficiency of education and practical training, etc. can be obtained.
Various image processing is possible for digital data, and various diagnosis support functions for supporting diagnosis by a pathologist for image data captured by a virtual slide system have been proposed.

Conventionally, the following proposal has been made as an example of a diagnosis support function.
Non-Patent Document 1 aims to calculate the N / C ratio (ratio of the nucleus to the cytoplasm), which is an important finding in diagnosing cancer, and uses a digital image processing technique to determine the pathology of the liver. A method for extracting a cell membrane from tissue specimen image data is disclosed. Non-Patent Document 1 improves the extraction accuracy of cell membranes by combining the color information of three types of observation images, bright field, dark field, and phase difference, compared to the case of a bright field observation image alone.

Moreover, not only the cell membrane but also the cell boundary (the cell boundary between cells contains intercellular substances other than the cell membrane) and the boundary between the cell and the tube or cavity are diagnosed. There is a big meaning above. The clear boundary makes it easier for a doctor to infer a complicated three-dimensional structure of the liver from the specimen, so that more accurate diagnosis can be realized from limited information.
The boundary between the cell and the tube or cavity is useful information for calculating the N / C ratio with high accuracy. For example, the pathological tissue specimen of the liver is roughly divided into a cell region composed of a nucleus and a cytoplasm, and a sinusoidal region that is a blood vessel that supplies a substance to the liver cell. It is necessary to exclude the sinusoidal area that does not exist correctly.

JP 2007-128209 A

Namiko Torisawa, Masanobu Takahashi, Masayuki Nakano, “Examination of the use of multi-imaging in the extraction of cell membranes in liver pathological tissue specimen images”, IEICE General Conference, D-16-9, 2009/3 Kazuya Kodama, Akira Kubota, “Generation of various blurs from a single lens system”, Journal of the Institute of Image Information and Television Engineers 65 (3), pp. March 372-381, 2111 Kazuya Kodama, Akira Kubota, “Scene Refocusing based on linear combination on frequency domain”, Video Media Processing Symposium (IMPS2012), I-3.02, pp. 45-46, October 2012 Kazuya KODAMA, Akira KUBOTA, "Efficient Reconstruction of All-in-Focus Images Through Shifted Pinholes from Multi-Focus Images for Dense Light Field Synthesis and Rendering, IEEE Trans. Image Processing, Vol.22, Issue 11, 15pages (2013-11 )

However, the conventional techniques described above have the following problems.
In Non-Patent Document 1, in order to acquire bright field, dark field, and phase difference observation images, a bright field microscope is equipped with a phase difference objective lens and a shared capacitor, and these are switched and photographed. For this reason, there are a cost problem that an additional part is required for the optical microscope for bright field observation, and a problem that the trouble of changing the optical system and the exposure condition occurs at the time of photographing, and the photographing time increases. In addition, the above-described effort creates a new problem in a virtual slide system that divides and captures a wide field of view for each local field of view. When shooting a wide field of view by switching the mechanical mechanism for each shooting of a local field of view, not only an increase in shooting time but also a problem of durability of the holding mechanism occurs because of shooting by switching the optical system. On the other hand, when shooting each wide-field area without changing the bright-field, dark-field, and phase-contrast observations, when moving the field of view due to differences in focusing accuracy or stage control in each image Accumulated errors in shooting positions are likely to occur. Therefore, there is a problem that misalignment is likely to occur between the image data, and it is difficult to compare the image data at the same pixel position.

  The present invention has been made in view of such problems, and for generating image data suitable for observation and diagnosis of a subject by image processing from original image data obtained by photographing the subject. The purpose is to provide new technology.

  The first aspect of the present invention corresponds to image data obtained by extracting or emphasizing a difference between a plurality of viewpoint image data having different line-of-sight directions with respect to the subject using original image data obtained by photographing the subject. A feature image data generating unit configured to generate feature image data, and performing correction processing for reducing brightness on at least a pixel corresponding to a low brightness region in the original image data among the pixels of the feature image data. And an image data generation device comprising correction means for generating corrected image data.

  According to a second aspect of the present invention, image data in which a computer extracts or emphasizes differences between a plurality of viewpoint image data having different line-of-sight directions with respect to the subject using original image data obtained by photographing the subject. Corresponding to the step of generating feature image data, and the computer performs a correction process for lowering the luminance of at least a pixel corresponding to the low luminance region in the original image data among the pixels of the feature image data Thus, a method for generating corrected image data is provided.

  A third aspect of the present invention is a program that causes a computer to execute each step of the image data generation method according to the present invention.

  According to the present invention, image data suitable for observation and diagnosis of a subject can be generated by image processing from original image data obtained by photographing the subject.

1 is a configuration diagram of an image data generation and display system according to an embodiment of the present invention. Display example to explain the functions of the image display application The figure which shows the internal structure of an image data generation apparatus A diagram showing a preparation that is an example of a subject The figure which shows typically the structure of the imaging device which image | photographs a to-be-photographed object. Diagram explaining the reason why contrast is emphasized in viewpoint image data The figure which shows the relationship between the angle (observation angle) which the deflection angle of a viewpoint, a gaze direction, and an optical axis make The figure which shows the unevenness which exists on the surface of the pathological specimen in the preparation The figure which shows the intensity | strength of the scattered light in observation angle (phi) in each surface of Fig.8 (a). The figure which shows Z stack image data in case the Z position of an object differs Diagram showing viewpoint image data and defocus when the Z position of an object is different Figure showing the defocus of scattered light extracted image data and scattered light weighted image data The figure which shows the example of GUI of Example 1. Flowchart of scattered image data generation process of embodiment 1 The figure explaining the transmitted light component suppression mask of Example 1, and its production | generation process. Flowchart of scattered light extraction image data generation processing of Embodiment 1 The figure explaining the transmitted light component suppression process of Example 1. FIG. The flowchart which shows the N / C ratio calculation processing of Example 1 The figure explaining the scattered image data calculation process of Example 2. Flowchart of scattered light weighted image data generation process of embodiment 2 The figure explaining the scattered light weighted | enhanced image data generation process of Example 2. The figure explaining the scattered light weighted | enhanced image data generation process of Example 3. Flowchart of scattered light extraction image data generation processing Flowchart of scattered light extraction image data generation processing Example of viewpoint weight function and viewpoint weight function for extracting scattered light information The figure which shows a mode that the surface unevenness | corrugation of a sample is observed from the viewpoint from which a declination differs 180 degrees Example of viewpoint position, argument selection function, argument selection viewpoint weight function of embodiment 6 Flowchart of processing in embodiment 6

(overall structure)
FIG. 1 shows the configuration of an image data generation and display system according to an embodiment of the present invention.
The image data generation apparatus (host computer) 100 is connected to an operation input device 110 that receives input from the user and a display 120 for presenting image data output from the image data generation apparatus 100 to the user. As the operation input device 110, a keyboard 111, a mouse 112, a dedicated controller 113 (for example, a trackball or a touch pad) for improving user operability can be used. The image data generating apparatus 100 is connected to a storage device 130 such as a hard disk, an optical drive, or a flash memory, and another computer system 140 that can be accessed through a network I / F. In FIG. 1, the storage device 130 exists outside the image data generation device 100, but may be built in the image data generation device 100.

  The image data generation device 100 acquires image data from the storage device 130 in accordance with a user control signal input from the operation input device 110, and generates image data for observation suitable for observation by applying image processing. Extract information necessary for diagnosis.

The image display application and the image data generation program (both not shown) are computer programs executed by the image data generation apparatus 100. These programs are stored in an internal storage device (not shown) or the storage device 130 in the image data generation device 100. Functions related to image data generation to be described later are provided by an image data generation program, and each function of the image data generation program can be called (utilized) via an image display application. The processing result of the image generation program (for example, generated image data for observation) is presented to the user via the image display application.

(Display screen)
FIG. 2 shows an example in which image data of a pathological specimen imaged in advance is displayed on the display 120 through an image display application.

  FIG. 2 shows the basic configuration of the screen layout of the image display application. In the entire window 201 of the display screen, an information area 202 indicating the status of display and operation and information on various image data, a thumbnail image 203 of the pathological specimen to be observed, a display area 205 for detailed observation of the pathological specimen image, a display area A display magnification 206 of 205 is arranged. A frame line 204 drawn on the thumbnail image 203 indicates the position and size of the area that is enlarged and displayed in the display area 205 for detailed observation. With the thumbnail image 203 and the frame line 204, the user can easily grasp which part of the entire pathological specimen image is being observed.

  The image displayed in the detailed observation display area 205 can be set and updated by a moving operation or an enlargement / reduction operation by the input operation device 110. For example, the movement can be realized by dragging the mouse on the screen, and the enlargement / reduction can be realized by rotating the mouse wheel or the like (for example, the forward rotation of the wheel is enlarged and the backward rotation is assigned to reduction). Further, switching to an image having a different focal position, that is, an image having a different depth position can be realized by rotating the mouse wheel while pressing a predetermined key (for example, the Ctrl key). For example, the forward rotation of the wheel is assigned to movement to an image having a deep depth, and the backward rotation is assigned to movement to an image having a shallow depth. The display area 205, the display magnification 206, and the frame line 204 in the thumbnail image 203 are updated in accordance with the change operation of the display image by the user as described above. In this way, the user can observe an image with a desired in-plane position, depth position, and magnification.

(Image data generator)
FIG. 3 is a diagram illustrating an internal configuration of the image data generation apparatus 100.
The CPU 301 controls the entire image data generation apparatus using programs and data stored in the main memory 302. In addition, the CPU 301 performs various arithmetic processing and data processing described in the following embodiments, for example, scattered image data generation processing, transmitted light component suppression mask generation processing, scattered light extraction image data generation processing, transmitted light component suppression processing, scattered light. Performs enhanced image data generation processing and the like.

  The main memory 302 has an area for temporarily storing programs and data loaded from the storage device 130 and programs and data downloaded from other computer systems 140 via the network I / F (interface) 304. The main memory 302 includes a work area necessary for the CPU 301 to perform various processes.

  The operation input device 110 is configured by devices that can input various instructions to the CPU 301 such as the keyboard 102, the mouse 103, and the dedicated controller 113. The user inputs information for controlling the operation of the image data generation apparatus 100 through the operation input device 110. Reference numeral 305 denotes an I / O (input / output) for notifying the CPU 301 of various instructions input via the operation input device 110.

  The storage device 130 is a large-capacity information storage device such as a hard disk, and stores an OS (Operating System), a program for causing the CPU 301 to execute processing described in the following embodiments, image data, and the like. Writing information to the storage device 130 and reading information from the storage device 130 are performed via the I / O 306.

  The display control device 307 performs control processing for causing the display 120 to display image data, characters, and the like. The display 120 displays a screen for prompting the user to input, and displays image data acquired from the storage device 130 or another computer system 140 and processed by the CPU 301.

  The arithmetic processing board 303 includes a processor and a buffer memory (not shown) in which specific arithmetic functions such as image processing are enhanced. In the following description, the CPU 301 is used for various arithmetic processing and data processing, and the main memory 302 is used as a memory area. However, a processor or a buffer memory in the arithmetic processing board can also be used. To do.

(subject)
FIG. 4 shows a preparation (also called a slide) of a pathological specimen that is an example of a subject. In the preparation of the pathological specimen, the pathological specimen 400 placed on the slide glass 410 is enclosed by an encapsulating agent (not shown) and a cover glass 411 placed thereon. The size and thickness of the pathological specimen 400 vary depending on the pathological specimen. Further, on the slide glass 410, there is a label area 412 in which information on the pathological specimen is recorded. Information may be recorded in the label area 412 by pen entry or printing of a barcode or a two-dimensional code. A storage medium that can store information by an electrical, magnetic, or optical method may be provided in the label area 412. In the following embodiments, the preparation of the pathological specimen shown in FIG. 4 will be described as an example of the subject.

(Imaging device)
FIG. 5 schematically illustrates a part of the configuration of an imaging apparatus that captures a subject and acquires a digital image. As shown in FIG. 5, in this embodiment, the x axis and the y axis are taken in parallel to the surface of the pathological specimen 400, and the z axis is taken in the depth direction of the pathological specimen 400 (the optical axis direction of the optical system).

  A preparation (pathological specimen 400) is placed on the stage 502, and light is emitted from the illumination unit 501. The light transmitted through the pathological specimen 400 is magnified by the imaging optical system 503 and forms an image on the light receiving surface of the imaging sensor 504. The imaging sensor 504 is a line sensor (one-dimensional sensor) or an area sensor (two-dimensional sensor) having a plurality of photoelectric conversion elements. The optical image of the pathological specimen 400 is converted into an electrical signal by the imaging sensor 504 and output as digital data.

  If the image data of the entire pathological specimen cannot be acquired by a single imaging, multiple division imaging is performed while moving the stage 502 in the x direction and / or the y direction, and the obtained plurality of division image data are synthesized. The image data of the entire pathological specimen is generated by (joining). Further, a plurality of image data (referred to as layer image data) having different focal positions in the optical axis direction (depth direction) are acquired by performing the imaging a plurality of times while moving the stage 502 in the z direction. In this specification, an image data group including a plurality of layer image data having different focal positions in the optical axis direction (depth direction) is referred to as “Z stack image data”. Further, layer image data and Z stack image data, which are image data acquired by photographing a subject, are referred to as “original image data”.

  The magnification value displayed on the display magnification 206 in FIG. 2 is a value obtained by multiplying the magnification of the imaging optical system 503 by the enlargement / reduction ratio on the image display application. Note that the magnification of the imaging optical system 503 may be fixed, or may be variable by exchanging the objective lens.

(Description of technology for generating viewpoint image data)
The image data generation apparatus 100 does not require an observation / imaging method for changing the optical system such as dark field observation or phase difference observation. Instead, intermediate image data (viewpoint image data) is generated from the Z stack image data by image processing, and image data for observation suitable for observation and diagnosis is generated using the intermediate image data. First, a technique that can be used for processing for generating viewpoint image data as intermediate image data from Z stack image data will be described.

It is known that viewpoint image data (arbitrary viewpoint image data) observed from an arbitrary direction can be generated based on a plurality of pieces of image data (Z stack image data) captured by changing the focal position in the optical axis direction. Here, the viewpoint image data represents image data obtained by observing a subject from a predetermined observation direction (that is, viewpoint).
For example, Japanese Patent Laid-Open No. 2007-128009 (hereinafter referred to as Patent Document 1) discloses a method for generating image data of an arbitrary viewpoint or an arbitrary blur from a defocused image data group captured by changing the focal position. Has been. In this method, three-dimensional defocusing (hereinafter referred to as three-dimensional defocusing), which is a collection of two-dimensional defocusing that occurs before and after the focal position, becomes invariant at the XYZ position with respect to the defocused image data group. The coordinate conversion process is performed as described above. And the image data which changed the viewpoint and blur is obtained by applying a three-dimensional filter process in the obtained orthogonal coordinate system (XYZ).
Non-Patent Document 2 discloses an improvement of the method of Patent Document 1. In Non-Patent Document 2, the line-of-sight direction is obtained from the viewpoint, and integrated image data is generated by integrating the Z stack image data in the line-of-sight direction, and similarly, integrated image data in the line-of-sight direction of three-dimensional defocusing is also generated. . Thereafter, the cumulative image data of the three-dimensional defocus is inversely filtered with respect to the integrated image data of the Z stack image data, thereby suppressing the influence due to the restriction in the Z direction (number of layer image data), and a high-quality viewpoint image. Data can be generated.
Non-Patent Document 3 discloses a method for speeding up the calculation of Non-Patent Document 2. In the method of Non-Patent Document 3, arbitrary viewpoint image data on the frequency domain is obtained by linear combination of a filter determined in advance without depending on a subject (scene) and Fourier transform image data at each Z position of the defocused image data group. Arbitrary defocus image data can be calculated efficiently.
Non-Patent Document 4 is a document describing the method of Non-Patent Document 3 in more detail.

In the present embodiment, viewpoint image data (arbitrary viewpoint image data) observed from an arbitrary direction is generated based on a plurality of pieces of image data (Z stack image data) captured by changing the focal position in the optical axis direction. Generate image data with arbitrary defocus. In the following description, these methods are collectively referred to as an MFI (multi-focus imaging) arbitrary viewpoint / defocus image generation method.
It should be noted that the three-dimensional focal blur is invariant at the xyz position in the Z stack image data photographed by changing the focal position with a microscope having a bilateral telecentric optical system. Therefore, when the MFI arbitrary viewpoint / defocused image generation method is applied to the Z stack image data photographed by the bilateral telecentric optical system, the coordinate conversion process and the image data enlargement / reduction process associated with the coordinate conversion process are unnecessary.

  2. Description of the Related Art An imaging device is known that can acquire image data in which four-dimensional information called light field (XY two-dimensional image data plus the degree of freedom of the viewpoint position) is recorded by one shooting. Such an imaging apparatus is called a light field camera or a light field microscope. In these apparatuses, a lens array is originally arranged at a position that is an image plane, and a light field is photographed by an image sensor behind the lens array. From the original image data in which the light field is recorded, image data at an arbitrary focal position and viewpoint image data (arbitrary viewpoint image data) observed from an arbitrary direction can be generated using a known technique.

In this embodiment, image data in an arbitrary observation direction generated by digital image processing based on captured image data such as Z-stack image data or a light field without physically changing the direction of the imaging device with respect to the subject. This is called “viewpoint image data”. This viewpoint image data is image data simulating an optical image formed on an imaging surface by a light beam having a principal ray as a principal ray and passing through an imaging optical system used for photographing a subject. The direction of the chief ray corresponds to the viewing direction. The direction of the chief ray can be set arbitrarily. Also, the size (NA) of the light beam can be set arbitrarily. When the purpose is image diagnosis or the like, it is desirable that the depth of field of the viewpoint image data is deep. Therefore, the NA of the luminous flux corresponding to the viewpoint image data is desirably 0.1 or less.

  Note that the viewpoint image data generated (calculated) by digital image processing is the image data captured by physically changing the imaging conditions (aperture position / size), optical axis direction, lens, etc. of the imaging optical system. Does not necessarily match. However, even if the image data does not match the actual captured image data, if the image data has the same characteristics as if the subject was observed by changing the viewpoint (that is, the same effect as changing the observation direction is digital) If it can be applied by image processing), it is useful for image observation and image diagnosis. Therefore, although it does not exactly match image data actually taken by changing the optical axis direction or the like, image data that has been subjected to digital image processing so that the same characteristics as such image data appear is also in this embodiment. Included in the viewpoint image data.

  From the defocused image group subjected to coordinate transformation, the viewpoint (x, y, 0) from the origin O (x, y, z) = (0, 0, 0) on the lens surface (corresponding to the pupil plane) in real space. It is possible to generate viewpoint image data observed through a pinhole at a position shifted by z) = (s, t, 0). In the MFI arbitrary viewpoint / defocused image generation method, the observation direction for observing the subject, that is, the line-of-sight direction can be changed by changing the position of the viewpoint on the lens surface.

  The line-of-sight direction is the slope of a straight line passing through the position (x, y, z) = (s, t, 0) of the viewpoint on the lens surface in a light beam emitted from a predetermined position of the subject corresponding to the image formed. Can be defined. The line-of-sight direction can be expressed in various ways. For example, it may be expressed by a three-dimensional vector indicating the traveling direction of a straight line, or an angle (observation angle) formed by the aforementioned three-dimensional vector with the optical axis and a vector projected onto a plane perpendicular to the optical axis may be An expression by an angle (deflection angle) with the X axis may be used.

When the imaging optical system is not bilateral telecentric, the three-dimensional defocus on the imaging surface changes depending on the spatial position of the focused subject (position in the xyz coordinates), and the viewpoint position (x, y on the lens surface). , Z) = (s, t, 0), the slope of the straight line is not constant. In that case, the line-of-sight direction should be defined on the orthogonal coordinate system (XYZ) after coordinate transformation described in Patent Document 1, and the line-of-sight direction is (X, Y, Z) = (− s, −t, 1). It can be expressed as a vector. Hereinafter, how to obtain the line-of-sight direction after coordinate conversion will be described.
In Patent Document 1, an arbitrary position where an imaging optical system is in focus and the same viewpoint position (x, y, z) = (s, t, 0) on the lens surface (corresponding to the pupil plane) of the imaging apparatus are disclosed. It is described that all of the light rays that connect () are parallel to each other in the orthogonal coordinate system (XYZ) after coordinate conversion. (See FIGS. 1 to 3 of Patent Document 1 and the description thereof)

数１の直線は座標変換後の直交座標系（ＸＹＺ）では以下の式で表される。

数２にＺ＝０（ｚ＝ｆ）、Ｚ＝１（ｚ＝∞）を代入し、３次元座標を求めると、それぞれ（Ｘ，Ｙ，Ｚ）＝（ｐ＋ｓ，ｑ＋ｔ，０）、（Ｘ，Ｙ，Ｚ）＝（ｐ，ｑ，１）となることから、直交座標系（Ｘ，Ｙ，Ｚ）での直線の傾きは（−ｓ，−ｔ，１）で表される。
従って、座標変換後の直交座標系での視線方向を表すベクトルは（Ｘ，Ｙ，Ｚ）＝（−ｓ，−ｔ，１）となる。 Light emitted from a certain point of the subject in the perspective coordinate system (real space before coordinate transformation) passes through (p + s, q + t, f) (f is the focal length), and the viewpoint position (x, y, z) = Refraction occurs at (s, t, 0). This straight line is represented by the following equation.

The straight line of Formula 1 is represented by the following expression in the orthogonal coordinate system (XYZ) after coordinate conversion.

Substituting Z = 0 (z = f) and Z = 1 (z = ∞) into Equation 2 to obtain three-dimensional coordinates, (X, Y, Z) = (p + s, q + t, 0), (X , Y, Z) = (p, q, 1), the slope of the straight line in the orthogonal coordinate system (X, Y, Z) is represented by (−s, −t, 1).
Therefore, the vector representing the line-of-sight direction in the orthogonal coordinate system after coordinate conversion is (X, Y, Z) = (− s, −t, 1).

When the imaging optical system is bilateral telecentric, three-dimensional defocusing in a plurality of pieces of image data (Z stack image data) photographed with the focus changed in the depth direction is not changed regardless of the Z position.
Therefore, coordinate conversion is not required to make the three-dimensional defocusing invariant regardless of the spatial position. A predetermined position (x, y, z) = (0, 0, za) of the subject in focus in the real space and a viewpoint position (x, y, z) = (s, t, 0) on the lens surface The slope (−s, −t, za) of the straight line connecting the lines may be regarded as the line-of-sight direction as it is.

(Correspondence between the angle of view and the observation angle φ when the specimen is actually observed)
FIG. 7A is a schematic diagram showing the position (x, y, z) = (s, t, 0) of the viewpoint in the real space, and FIG. 7B shows the viewpoint position in the orthogonal coordinate system (XYZ). It is a schematic diagram showing the light ray which passes a position (x, y, z) = (s, t, 0).

ただし、θはｔ，ｓの符号に応じて−１８０〜＋１８０度の範囲に収まるように調整する。 A dotted circle shown in FIG. 7A represents a range through which light rays can pass on the lens surface (z = 0). The angle at which the position (x, y, z) = (s, t, 0) of the viewpoint on the lens surface is the x axis on the lens surface (z = 0) or the line of sight (−s, −t) , 1) is defined as an angle formed by a straight line projected onto the xy plane and the x axis, the declination angle θ can be obtained by the following equation.

However, θ is adjusted to fall within the range of −180 to +180 degrees according to the signs of t and s.

なお、図７（ｂ）の２本の点線はレンズ面上の最も端を通る光線を示している。座標変換前の透視座標系（ｘｙｚ）でのレンズの絞り半径をｒ_ｍとすると、視点の位置（ｘ，ｙ
，ｚ）＝（ｓ，ｔ，０）が半径ｒ_ｍの内部にある場合のみ視点画像データは計算できる。 Next, a description will be given of the relationship viewing angle phi _T on conversion coordinates sight with reference to FIG. 7 (b).
In FIG. 7 (b), the straight line shown by Formula 2 and the straight line when the points p = 0 and q = 0 on the optical axis are substituted into Formula 2 are shown by bold arrows.
According to Patent Document 1, Z = 0 in the orthogonal coordinate system (XYZ) corresponds to z = f (or z = −∞) in the perspective coordinate system (xyz), and Z = 1 corresponds to z = ∞ (or z = -F). For this reason, FIG. 7B shows that the light beam from infinity (Z = 1) has a spread on the front focal plane (Z = 0) on the lens surface in the orthogonal coordinate system XYZ. (See FIG. 3 of Patent Document 1 and its description)
Here, the observation angle phi _T on conversion coordinates, line-of-sight (-s, -t, 1) and the optical axis (Z axis) is defined as the angle, the line of sight as is apparent from FIG. 7 (b) It does not depend on the position of the object, viewing angle phi _T is calculated by the following equation.

Note that the two dotted lines in FIG. 7B indicate the light rays that pass through the end most on the lens surface. When the aperture radius of the lens in the coordinate before conversion of the perspective coordinate system (xyz) and r _m, the position of the viewpoint (x, y
, Z) = (s, t , 0) viewpoint image data only if internal radius _{r m} can be calculated.

なお、撮像光学系が両側テレセントリックでない場合には、数５でΔｘとΔｚの代わりに、直交座標系（ＸＹＺ）におけるＸ方向のセンサ画素ピッチΔＸとＺ方向の移動間隔ΔＺを用いれば観察角φが求められる。
以上で、実際に標本を観察したときの視線に対応する偏角θ、観察角φについて説明した。
以降の説明では、レンズ面上の視点の位置（ｘ，ｙ，ｚ）＝（ｓ，ｔ，０）を視点（ｓ，ｔ）と略して記載する。また、直交座標系（ＸＹＺ）上での画像処理を前提として説明するため、視点の位置（ｓ，ｔ）に言及する場合のみ透視座標系（座標変換前の実空間）での位置を表すものとし、その他は特に断りが無い限りは、直交座標系（ＸＹＺ）での位置を表すとする。 Next, the deflection angle θ and the observation angle φ corresponding to the line of sight when the specimen is actually observed, not on the converted coordinates, will be described. According to Snell's law, when a light beam is incident on a boundary having a different refractive index, the product of the incident angle of the light beam and the refractive index of the medium on the incident side is equal to the product of the refraction angle of the light beam and the refractive index of the medium on the refractive side. Since the refractive index of the sample is larger than the refractive index of air, the observation angle in the sample is smaller than the observation angle in air. Therefore, the three-dimensional defocus in the sample, which is composed of refracted rays, is smaller than the three-dimensional defocus in air. However, in this embodiment, since the position of the viewpoint is calculated based on the three-dimensional imaging relationship due to the three-dimensional defocusing in the sample, there is no need to consider the influence of the refractive index of the sample, and the angle θ The observation angle φ directly represents the observation direction in the sample.
When the imaging optical system is bilateral telecentric, coordinate conversion is not required. Therefore, if the sensor pixel pitch in the x direction and the y direction are the same, the observation angle φ is the sensor pixel pitch Δx in the x direction and z It can be expressed by the following equation using a direction movement interval Δz (unit: μm).

If the imaging optical system is not bilateral telecentric, the observation angle φ can be obtained by using the sensor pixel pitch ΔX in the X direction and the movement interval ΔZ in the Z direction in the orthogonal coordinate system (XYZ) instead of Δx and Δz in Equation 5. Is required.
As described above, the declination angle θ and the observation angle φ corresponding to the line of sight when actually observing the sample have been described.
In the following description, the viewpoint position (x, y, z) = (s, t, 0) on the lens surface is abbreviated as viewpoint (s, t). In addition, in order to explain on the premise of image processing on an orthogonal coordinate system (XYZ), only when referring to the position (s, t) of the viewpoint, the position in the perspective coordinate system (real space before coordinate transformation) is represented. Unless otherwise specified, the position in the Cartesian coordinate system (XYZ) is expressed unless otherwise specified.

When the MFI arbitrary viewpoint / defocused image generation method is applied to the Z stack image data acquired by the imaging apparatus of FIG. 5, viewpoint image data in which the position of the viewpoint, that is, the observation direction is changed, can be generated.
The viewpoint image data calculated by the MFI arbitrary viewpoint / defocused image generation method has mainly two characteristics. One is that the viewpoint image data has a very deep (infinite) depth of field, and the boundaries of substances in specimens with different transmittances can be clearly seen. The other is that the viewpoint image data is close to the oblique illumination observation image obtained by illuminating the specimen from a partial area of the illumination, and the unevenness changing along the line-of-sight direction on the XY plane is emphasized, and the specimen is stereoscopic. It is a point that can be seen. In the viewpoint image data, as with the oblique illumination image, as the viewing direction tilts with respect to the optical axis, that is, as the viewing angle φ of the viewing line increases, the contrast of the unevenness of the specimen surface increases and the specimen surface becomes three-dimensional. appear.
(However, physically, the oblique illumination image and the viewpoint image data are different. The oblique illumination image has an optical blur due to the change of the focal position, but the viewpoint image data is not The difference is that the depth of field remains very deep regardless of the change, and the viewpoint image data changes depending on the Z position Zf of the Z stack image data to be focused, but the change is parallel movement in the XY directions. Represented by

Next, the reason why the boundary between substances having different transmittances, which is the first feature, can be clearly seen will be described.
FIG. 6A is a diagram illustrating the three-dimensional defocus of the optical system in the orthogonal coordinate system (XYZ). Reference numeral 600 denotes a three-dimensional defocused shape. The defocus is slightly at the focal position (the apexes of the two cones), but the defocus increases as the Z position moves away from the focal position. Using the MFI arbitrary viewpoint / defocused image generation method, viewpoint image data composed of light rays in an arbitrary line-of-sight direction (for example, straight line 610) passing through the inside of the cone 600 can be generated from the Z stack image data.

FIG. 6B shows a pathological sample in the orthogonal coordinate system (XYZ) viewed from different directions. An oblique cavity 630 exists inside the specimen 620 in FIG.
When viewed from the direction 631, a portion other than the cavity 630 can be seen through, so the contrast of the wall surface of the cavity 630 is not clear. Similarly, when viewed from the direction 632, the contrast of the cavity 630 is unclear. However, when observing from the direction 633 along the wall surface of the cavity 630, the contrast of the wall surface of the cavity 630 becomes clear because it is not affected by other parts. Even if the line-of-sight direction is slightly different from the direction of the hollow wall surface, the contrast can be kept relatively high.

On the other hand, the three-dimensional defocus of the Z-stack image data of the sample 620 is obtained by integrating rays in various gaze directions. Therefore, in the layer image data at any Z position (focal position), the image is blurred due to the influence of multidirectional light beams including the light beams in the directions 631 to 633, and the contrast of the cavity wall surface is compared with the observation image from the direction 633. It will not be clear. This phenomenon is not limited to cavities, but also in nuclei, cell membranes, fibers, and the like.
In the above, the phenomenon has been described in which the boundary between substances having different transmittances in the specimen can be clearly seen in the viewpoint image data.

A pathological specimen is a translucent object, and scattered light is present in addition to transmitted light. The presence of the scattered light gives rise to the second feature of the viewpoint image data.
Next, the reason why the contrast of the unevenness on the sample surface increases as the viewing angle φ of the line of sight increases due to scattered light from the sample will be described.

8A is a schematic diagram showing irregularities present on the surface of the pathological specimen in the preparation. It is assumed that the unevenness on the xz plane shown in FIG. 8A continues in the y direction which is the depth direction.
A pathological specimen for histological diagnosis is fixed with paraffin, sliced to a uniform thickness with a microtome, and then stained. However, pathological specimens are not completely uniform, and there are irregularities due to the structure of tissues and the composition of substances at the boundaries between cells and tubes and cavities, and between the nucleus and cytoplasm. There is an uneven structure as shown in FIG.
FIG. 8A is a simple model, and there are few sharp points as shown in FIG. In addition to the convex structure as shown in FIG. 8A, there is a concave structure inside the specimen. In addition, even if the surface is flat, if there is a substance with a different refractive index inside, the optical distance changes, so the discontinuity of the refractive index inside the sample can be regarded as surface irregularities.
In an actual preparation, a transparent encapsulant is present between the cover glass and the specimen. However, since the difference between the refractive index of the encapsulant and the refractive index of the sample is slight and has little influence, the following description will be made assuming that both refractive indexes are the same.

In FIG. 8A, reference numeral 811 denotes a surface without unevenness, 812 denotes a slope that rises to the right, and 813 denotes a slope that falls to the right. The inclination angles formed by the inclined surfaces 812 and 813 and the x axis are α (α> 0), respectively.
FIGS. 9A to 9C are schematic diagrams showing the intensity of scattered light at the observation angle φ on the surfaces 811 to 813 in FIG. 8A. FIG. 9A, FIG. 9B, and FIG. 9C show light scattering on the plane 811 and the inclined surfaces 812 and 813, respectively. Circles in contact with the respective surfaces indicate the intensity of the scattered light depending on the scattering direction when the light diffusion characteristics on the sample surface are assumed to be a completely diffuse transmission surface. The length of the thick arrow line in the circle indicates the intensity of scattered light when observed from an angle inclined by φ from the optical axis (z-axis). (In actuality, the specimen surface is not a completely diffuse transmission surface, but has an intensity dependency depending on the incident direction and the observation direction of light. However, in order to simplify the explanation, the description will be made assuming that the sample surface is a complete diffuse transmission surface.)

In a perfectly diffusive transmission surface, if the intensity of light in the normal direction orthogonal to the surface is I ₀ and the angle between the observation direction and the normal of the surface is δ, the intensity I (δ) of scattered light in the δ direction is I ( δ) = I ₀ cos δ
In FIGS. 9A, 9B, and 9C, the angles δ formed by the normal direction of the observation direction and the surface can be expressed by φ, φ + α, and φ−α, respectively. ,
I ₀ cosφ, I ₀ cos (φ + α), I ₀ cos (φ−α)
It becomes.
In addition, if the slope angle α is positive on the slope where the value of z increases from the observation direction (uphill slope) and the slope angle α is negative on the slope where the value of z decreases (downhill slope), either The intensity of scattered light can also be expressed as I ₀ cos (φ−α).

When a value obtained by dividing the intensity of scattered light in the observation angle φ direction of the inclined surfaces 812 and 813 by the intensity of scattered light in the observation angle φ direction of the plane 811 is defined as contrast C (φ, α), the contrast is represented by the following expression. .

表１より、観察角φが小さいときは、傾斜角αが大きくても斜面８１２、８１３の間のコントラストは低いため観察しづらく、観察角φが大きくなるに従い、傾斜角αが小さくてもコントラストは大きくなり、観察しやすくなることが分かる。
以上、標本での散乱光により、視線の観察角φが大きくなるほど、標本表面の凹凸のコントラストが高くなる理由について説明した。 Table 1 shows the values of contrast C (φ, α) with φ and α changed.

According to Table 1, when the observation angle φ is small, it is difficult to observe because the contrast between the inclined surfaces 812 and 813 is low even if the inclination angle α is large, and as the observation angle φ increases, the contrast becomes small. It becomes large and it becomes easy to observe.
As described above, the reason why the contrast of the unevenness on the sample surface increases as the viewing angle φ of the line of sight increases due to the scattered light in the sample has been described.

で求まる。数７から、見かけ上の傾斜角α’はαより小さくなり、凹凸方向角βと偏角θの差｜θ−β｜によって、コントラストＣが低下することが分かる。
ここで、｜θ−β｜が９０度を境に、傾斜角αおよびα’の符号は正から負、あるいは負から正に切り替わる点に注意する。これは、視点の偏角を変えた観察方向によって上りの斜面と下りの斜面が変わることに対応する。傾斜角α’は−α〜＋αの範囲を取り、例えば、斜面８１３では視点の偏角が｜θ−β｜＝０のときα’＝α（観察方向から見て上りの斜面）、｜θ−β｜＝πのときα’＝−α（下りの斜面）となる。 (Changes in scattered light intensity when the viewpoint is changed)
Next, changes in the scattered light intensity on the sample surface when the viewpoint is changed will be described.
In FIG. 8A, the edge direction of the surface unevenness is perpendicular to the x-axis in the xy plane, and the direction of elevation of the surface unevenness (direction perpendicular to the edge) is the angle of deviation θ (θ = 0) of the viewpoint. FIG. On the other hand, FIG. 8B is a diagram in the case where the direction of elevation change of the surface unevenness does not coincide with the x-axis. As shown in FIG. 8B, when the angle between the direction (821) perpendicular to the edge of the surface unevenness (820) and the x-axis is the unevenness direction angle β, the unevenness direction angle β and the deviation angle θ do not match. In this case, the surface irregularities 820 are observed from an oblique direction. At this time, the apparent inclination angle α ′ of the inclined surface 813 and the inclined surface 812 in the opposite direction viewed from the observation direction having the angle of deviation θ−β is

It is obtained by From Equation 7, it can be seen that the apparent inclination angle α ′ is smaller than α, and the contrast C decreases due to the difference | θ−β | between the concave / convex direction angle β and the deviation angle θ.
Note that the sign of the inclination angles α and α ′ is switched from positive to negative or from negative to positive with | θ−β | as 90 degrees. This corresponds to the fact that the ascending slope and the descending slope change according to the observation direction in which the deflection angle of the viewpoint is changed. The inclination angle α ′ takes a range of −α to + α. For example, on the inclined surface 813, when the angle of deviation of the viewpoint is | θ−β | = 0, α ′ = α (upward inclined surface when viewed from the observation direction), | θ When −β | = π, α ′ = − α (downhill slope).

病理標本では表面凹凸の傾斜角αは十分に小さいと推定できるため、ｃｏｓα＝１、ｓｉｎα＝αの近似式を用いて、数８は以下のように近似できる。

Next, consider the scattered light normalized intensity V (φ, α) obtained by normalizing the scattered light intensity of the inclined surface 813 observed from the direction of the angle θ−β of the viewpoint with the intensity of light observed on the plane 811. . The intensity of light observed on the plane 811 is expressed as I _ts (φ), and is a function of the observation angle φ that is constituted by the sum of the intensity of transmitted light and the intensity of scattered light. From the above, the scattered light normalized intensity V (φ, α) can be expressed by the following equation.

Since it can be estimated that the inclination angle α of the surface irregularities is sufficiently small in the pathological specimen, Equation 8 can be approximated as follows using an approximate expression of cos α = 1 and sin α = α.

Next, the values taken by A (φ) and B (φ) will be described.
Since the intensity of light I _{ts (φ)} have a tendency to decrease with general viewing angle phi from the characteristics of the illumination optical system, a transmitted light intensity when viewing angle phi = 0 and _{I 1, I ts (φ)} = I _When approximated as ₁ cos φ, A (φ) = I ₀ / I ₁ is obtained. If the scattered light is smaller than the transmitted light, A (φ) can be regarded as a relatively small constant. On the other hand, B (φ) becomes B (φ) = 0 from sinφ = 0 when φ = 0. Since sin φ increases and the light intensity I _ts (φ) decreases as the observation angle φ increases, B (φ) is an increasing function. Assuming I _ts (φ) = I ₁ cos φ, B (φ) = I ₀ / I ₁ × tan φ.

In the viewpoint image data calculated by the MFI arbitrary viewpoint / defocused image generation method, the average luminance of the image data does not change regardless of the viewpoint by the frequency filter processing that maintains the average. Therefore, it can be considered that a value obtained by removing the transmitted light intensity affected by the transmittance of the stained region from the luminance of the slope 813 in the viewpoint image data is substantially equal to the normalized intensity V (φ, α).

(Change in transmitted light intensity when the viewpoint is changed)
Next, a change in transmitted light intensity in viewpoint image data with different viewpoints will be considered.
In the specimen to be observed, in a region where the luminance difference with the adjacent substance is small (cytoplasm or cell boundary), the luminance difference is small even if the line-of-sight direction is different, and the intensity of transmitted light is small. In the case of a comparatively thin specimen having a thickness of about 4 μm, such as a histological pathological specimen, the position of the object inside the specimen is not greatly changed in the viewpoint image data calculated by focusing on the specimen surface. (In the viewpoint image data calculated by the MFI arbitrary viewpoint / in-focus image generation method, the object existing in the vicinity of the Z position Zf of the focused Z stack image data is the XY of substantially the same viewpoint image data regardless of the viewpoint. (It can be understood from the fact that there is no difference in the XY position in the image data obtained by integrating the viewpoint image data of a plurality of viewpoints, so that the blur is reduced.)
Therefore, the difference in transmitted light intensity between the viewpoint image data obtained by changing the viewpoint position can be regarded as slight, and the difference between the scattered light normalized intensity and the difference between the viewpoint image data should be considered to be almost equal. Can do.

（ただし、｜θ−β｜＝πのときα’＝−α） (Extraction of scattered light information on specimen surface from viewpoint image data with different viewpoint angle θ)
Next, it is considered to extract information on scattered light on the sample surface by calculation between viewpoint image data in which the deflection angle θ is changed.
In the predetermined surface asperity indicated by 813 in FIG. 8, when the difference | θ−β | between the concavo-convex direction angle β and the declination angle θ is 0 degree from Equation 7, the inclination angle is α, and the scattered light The normalized strength V (φ, α) is maximized. On the other hand, when | θ−β | is 180 degrees, the apparent inclination angle α ′ is −α, and the scattered light normalized intensity V (φ, α ′) is minimum.
Therefore, if subtraction is performed between the viewpoint image data in which | θ−β | becomes 0 and the viewpoint image data in which | θ−β | ) Can be extracted. The formula is shown below.

(However, when | θ−β | = π, α ′ = − α)

Since the inclination angle α and the concavo-convex direction angle β of the surface irregularity of the specimen are various, the difference is obtained between the viewpoint image data of the viewpoints where the deflection angle θ is different by 180 degrees at various viewpoints where the deflection angle θ is changed. If collected, it can be seen that information on scattered light on the sample surface can be extracted.
Note that it can be seen from Equation 10 that the scattered light information can be extracted even when the calculation is performed between the viewpoint image data of viewpoints with different deflection angles θ of 180 degrees. For example, when | θ−β | = π / 2 (90 degrees), α ′ = 0, and α × B (φ) can be extracted from the difference between the viewpoint image data.
In addition, not only subtraction but also division can extract information on scattered light on the sample surface. That is, since the value of V (φ, α) / V (φ, α ′) also changes according to the inclination angle α, it can be said that it is an index representing the information of scattered light.

(Extraction of scattered light information on specimen surface from viewpoint image data with different observation angle φ)
Subsequently, as in the case of the above-described declination angle θ, it is considered that information on scattered light on the sample surface is extracted by calculation between viewpoint image data with different observation angles φ.
The subtraction between the two viewpoint image data whose observation angles are φ and φ ′ can be expressed by the following equation. (However, approximating A (φ) = A (φ ′))

As already described, since B (φ) is a function that increases as φ increases, information on scattered light on the sample surface (by the calculation between two viewpoint image data with different observation angles φ) ( It can be seen that a part of () can be extracted. Since B (φ) is 0 when φ = 0, subtraction between the viewpoint image data at the observation angle φ and the viewpoint image data at the viewpoint at which the observation angle φ ′ is 0 is most effective.
As in the case of the above-mentioned declination angle θ, the uneven direction angle β in the sample is various, so not only the observation angle φ but also the difference between the viewpoint image data at various viewpoints with the declination angle θ changed. By collecting the results, it can be seen that the information on the scattered light on the sample surface can be extracted.

で表現できる。
ＤＩＶ（φ，φ’）は、αが０のとき１となり、αが０でないときはαの大きさに応じて強度が変わる。よって、除算ＤＩＶ（φ，φ’）によっても、傾斜角αの情報が抽出できることになる。
減算の場合と同様に、観察角φだけでなく、偏角θを変えた様々な視点において視点画像データ間で除算をし、その結果を集めれば、標本表面の散乱光の情報が抽出できる。 In addition, not only subtraction but also division can extract information on scattered light on the sample surface. From the above description, since the luminance of the viewpoint image data at the observation angle φ can be approximated by α × B (φ) + D (D is a constant), the luminance division between the viewpoint image data with the observation angle φ changed is

Can be expressed as
DIV (φ, φ ′) is 1 when α is 0, and when α is not 0, the intensity changes according to the magnitude of α. Therefore, the information of the inclination angle α can be extracted also by the division DIV (φ, φ ′).
As in the case of subtraction, by dividing the viewpoint image data from various viewpoints with different not only the observation angle φ but also the deflection angle θ, and collecting the results, information on the scattered light on the sample surface can be extracted.

Image data (microscope image data) taken with a transmission microscope as in the present embodiment includes a transmitted light component due to light transmitted through the specimen and a scattered light component due to light scattered on the surface of the specimen. . Since the intensity of the transmitted light component depends on the light transmittance of the sample, it represents a color difference or a refractive index difference in the sample. On the other hand, as described above, the intensity of the scattered light component mainly depends on the unevenness (surface shape) of the sample surface. In the microscope image data as the original image data, the scattered light component is hardly recognized because the transmitted light component is dominant. However, as described above, by extracting or emphasizing the difference between the plurality of viewpoint image data with different observation directions. Scattered light components can be extracted or emphasized. In other words, the “difference (difference)” and “ratio” between two viewpoint image data with different observation directions can be said to be feature amounts representing scattered light components (scattered light information) included in the original image data.
Note that “emphasis” is an operation that makes a part stand out (than the original state), and “extraction” is an operation that takes out only a part, but the same operation in that it focuses on a part Therefore, in this specification, the two terms are sometimes used without distinction. In addition, the operation of enhancing the scattered light component includes not only an operation of increasing the intensity of the scattered light component in the image data, but also reducing the intensity of the scattered light component relatively by reducing the intensity of the transmitted light component in the image data. The operation to raise is also applicable. Hereinafter, image data obtained by extracting or enhancing scattered light components (scattered light information) included in image data is referred to as scattered image data or feature image data. An operation for generating scattered image data from original image data is referred to as scattered image data generation (feature image data generation).
In the present invention, among scattered image data, image data obtained by extracting scattered light from a subject is referred to as scattered light extracted image data, and image data obtained by enhancing scattered light from the subject is referred to as scattered light emphasized image data.
Note that subtraction and division are exemplified as operations for extracting or enhancing the difference in viewpoint image data, but any operation may be used as long as the difference in image data can be enhanced.

The scattered image data is expected to be useful as observation image data suitable for observation of unevenness (surface shape) on the sample surface and diagnosis based thereon. For example, even if the surface of a certain area in the sample is uneven, if the transmittance in that area is almost uniform, the luminance or color of that area will be uniform in the original image data. Unevenness cannot be recognized. The scattered image data is particularly effective for visualizing such surface unevenness (surface unevenness that does not appear as a change in luminance or color). Further, since the cell boundary and the boundary between the cell and the sinusoid are also uneven, the scattered image data has an effect of clarifying these boundaries. Note that edge extraction (emphasis) is one of the processes for extracting or enhancing image features, but surface irregularities that do not appear as changes in image brightness or color cannot be visualized by edge extraction (emphasis). Therefore, the scattered image data extraction and the edge extraction may be used properly depending on what structure or feature of the specimen is desired to be observed.

  Although the case where scattered light is mainly generated on the sample surface has been described so far, the observation target is not necessarily limited to the sample surface. Even when there is a substance with a different refractive index from the surroundings that causes scattering at the focused position (Z = Zf), it can be considered in the same way as the scattering phenomenon on the sample surface, and observation is performed using scattered image data. it can.

(Three-dimensional imaging relationship in MFI arbitrary viewpoint / focal image generation method)
So far, we have described the characteristics of scattered light on the surface of the subject (the focused surface). However, not only scattered light but also transmitted light exists in the subject. In describing the influence of transmitted light on viewpoint image data, first, the three-dimensional imaging relationship in the MFI arbitrary viewpoint / defocused image generation method will be described.

（ただし、＊＊＊は３次元コンボリューションを表す。） In the MFI arbitrary viewpoint / defocused image generation method, a relationship in which a defocused image group (Z stack image data) subjected to coordinate transformation is represented by a convolution of a three-dimensional subject and three-dimensional defocus is established. . F (X, Y, Z) for a three-dimensional subject, h (X, Y, Z) for three-dimensional defocus, and g (X, Y, Z) for a defocused image group (Z stack image data) subjected to coordinate transformation ), The following formula is established.

(However, *** represents a three-dimensional convolution.)

FIG. 10A to FIG. 10F are schematic diagrams showing differences in Z stack image data when an object exists at a different Z position of a three-dimensional subject. The three-dimensional subject 1001 has an object at the position of Z = Zf, and the three-dimensional subject 1011 has an object at the position of Z = Zo.
The Z stack image data 1003 and 1013 are obtained by photographing the three-dimensional subjects 1001 and 1011 while changing the focal position in the Z direction using an optical system having a three-dimensional focal blur 1002, respectively. (Note that the three-dimensional defocus h (X, Y, Z) is the same because it is converted to shift invariant by coordinate conversion.) Obviously, in the Z stack image data 1003, the Z stack is at the position of Z = Zf. With the image data 1013, image data with the least defocus is obtained at the position of Z = Zo.

(Calculation of arbitrary viewpoint image data in the MFI arbitrary viewpoint / defocused image generation method)
As an MFI arbitrary viewpoint / defocused image generation method, methods disclosed in Patent Document 1 and Non-Patent Documents 2, 3, and 4 are known. As an example, an arbitrary viewpoint image data generation method according to the method of Non-Patent Document 2 will be described.
The viewpoint image data observed from the line-of-sight direction corresponding to the viewpoint (s, t) is assumed to be as _{, t} (X, Y, Zf). a _{s, t} (X, Y, Zf) is the integrated image data b _{s, t} (X, Y, Zf) of the Z stack image data in the same line-of-sight direction, and the three-dimensional imaging optical system in the same line-of-sight direction. It is obtained by deconvolution with the integrated value c _{s, t} (X, Y, Zf) of defocus. (The relationship described in FIG. 10, that is, the relationship in which g (X, Y, Z) is obtained by convolving f (X, Y, Z) and h (X, Y, Z) is in the direction of the line of sight. This is true regardless of the integral of.)

ただし、Ａ_ｓ，ｔ（ｕ，ｖ），Ｂ_ｓ，ｔ（ｕ，ｖ），Ｃ_ｓ，ｔ（ｕ，ｖ）はそれぞれａ_ｓ，ｔ（Ｘ，Ｙ，Ｚｆ）、ｂ_ｓ，ｔ（Ｘ，Ｙ，Ｚｆ）、ｃ_ｓ，ｔ（Ｘ，Ｙ，Ｚｆ）のフーリエ変換である。ｕ，ｖはそれぞれＸ，Ｙ方向の変化に対応する周波数座標である。
ａ_ｓ，ｔ（Ｘ，Ｙ，Ｚｆ）は以下の式で求められる。

ただし、Ｆ^−１｛｝はフーリエ逆変換を表す。 When expressed in terms of an expression, the following relationship holds.

However, A _{s, t} (u, v), B _{s, t} (u, v), C _{s, t} (u, v) are a _{s, t} (X, Y, Zf), b _{s, t} ( X, Y, Zf), cs _{, t} (X, Y, Zf). u and v are frequency coordinates corresponding to changes in the X and Y directions, respectively.
a _{s, t} (X, Y, Zf) is obtained by the following equation.

However, F < ^-1 > {} represents a Fourier inverse transform.

Next, the relationship between arbitrary viewpoint image data and arbitrary defocus image data in the MFI arbitrary viewpoint / defocus image generation method will be described.
11A to 11H show the relationship between arbitrary viewpoint image data and arbitrary defocused image data obtained by the MFI arbitrary viewpoint / defocused image generation method when an object exists at a different Z position of a three-dimensional subject. It is a schematic diagram which shows.
Reference numeral 1100 in FIG. 11A and reference numeral 1110 in FIG. 11E denote three-dimensional objects, which are equal to 1001 in FIG. 10A and 1011 in FIG. Light rays represented by solid lines in the three-dimensional subjects 1100 and 1110 represent light rays that pass through an object in the subject and pass through a predetermined viewpoint corresponding to each on the lens surface of the optical system. In addition, the broken line in FIG.11 (e) respond | corresponds to the continuous line in Fig.11 (a).
Rays 1100a and 1110a are rays passing through a certain viewpoint a on the lens surface, and the viewpoint image data observed so that the position of Z = Zf is the focal position from the line-of-sight direction corresponding to the viewpoint is shown in FIG. 1101 of FIG. 11 and 1111 of FIG. Similarly, the light rays 1100b and 1110b are light rays passing through a certain viewpoint b on the lens surface, and the viewpoint image data observed so that the position of each Z = Zf becomes the focal position is 1102 in FIG. 11 (g) 1112.

In the viewpoint image data 1101 and 1102, the image of the object appears at the same position. On the other hand, in the viewpoint image data 1111, 1112, the position of the object image does not appear at the same position, and is shifted by an amount determined by the distance dZ (= Zo−Zf) between Zf and Zo and the line-of-sight direction. appear.
In the MFI arbitrary viewpoint / defocus image generation method, the layer image data g (X, Y, Zf) at the position of Z = Zf in the Z stack image data can be expressed by the following expression.

ｋ（ｓ，ｔ）は以下の式に示すように、レンズ面上に存在する全ての視点（ｓ，ｔ）での総和が１になるように規格化されているものとする。

Here, k (s, t) is a function that indirectly represents the three-dimensional defocus of the imaging optical system, and represents the relative intensity distribution of light rays passing through each viewpoint (s, t) on the lens surface. The following relationship is established between k (s, t) and h (X, Y, Z).

It is assumed that k (s, t) is standardized so that the sum at all viewpoints (s, t) existing on the lens surface is 1, as shown in the following equation.

Further, a _{s, t} (X, Y, Zf) in Expression 16 represents viewpoint image data at Z = Zf.
From Expression 16, a plurality of viewpoint image data at a certain Z position (Z = Zf) is weighted and combined to obtain image data having an arbitrary defocus at the Z position (Z = Zf), that is, arbitrary defocus image data. Can be reconstructed.

In the following description, a function that defines a weight for each viewpoint (s, t) (for each line-of-sight direction) when integrating a plurality of image data, such as k (s, t), is referred to as a “viewpoint weight function”. . It can be said that the viewpoint weighting function represents the relative intensity distribution of light rays that originate from a certain point on the subject and pass through the viewpoint (s, t). The above k (s, t) is a viewpoint weighting function having characteristics corresponding to the three-dimensional defocusing of the imaging optical system used for photographing the subject. Any function may be used as the viewpoint weight function. For example, k _a (s, t) corresponding to arbitrary three-dimensional defocusing for generating arbitrary defocused image data by the MFI arbitrary viewpoint / defocused defocus image generation method can also be used as the viewpoint weight function. In addition, a function having an arbitrary shape such as a cylindrical shape exemplified in an example described later can be used as the viewpoint weight function.
In addition, although the expression of Formula 16 is expressed by an integral 意味 which means integration of continuous values, in actual image processing, it is an operation for a plurality of discrete (finite) viewpoints (s, t). Because it is, it is correct to use Sigma Σ. However, since integral ∫ can be generalized, the following description will be made using integral ∫. In the case viewpoint are discrete (finite) number 18 normalized such that the sum of 1 of all viewpoints weighting function k (s, t) used in calculating or k _{a (s,} t) Represents what to do.

ただし、関数δはディラックのデルタ関数である。ａは点物体の強度（画素値）であり、ｂは背景の強度である。 (Blurred image at Z = Zf)
Next, layer image data at a position of Z = Zf in the Z stack image data is obtained from the viewpoint image data at Z = Zf by using the MFI arbitrary viewpoint / defocused image generation method.
Here, in the three-dimensional subjects 1001 and 1011, when a point object exists on the optical axis of the optical system and viewpoint image data (omnifocal image data) viewed from a viewpoint passing through the optical axis is expressed by the following expression. To do.

However, the function δ is a Dirac delta function. a is the intensity (pixel value) of the point object, and b is the intensity of the background.

  Hereinafter, the layer image data at the position of Z = Zf in the Z stack image data when the three-dimensional subjects 1001 and 1011 are photographed is obtained using Equation 16.

(Layer image data of Z = Zf of the three-dimensional subject 1001)
If the thickness of the object can be ignored, the viewpoint image data a _{s, t} (X, Y, Zf) of Expression 16 can be expressed by I (X, Y, Zf) regardless of the position (s, t) of the viewpoint. Therefore, by substituting Equation 19 into Equation 16 and transforming it, layer image data represented by the following equation is obtained.

Ｚスタック画像データｇ（Ｘ，Ｙ，Ｚｆ）には被写体と焦点位置のずれｄＺによる焦点ぼけが含まれることがわかる。 (Layer image data of Z = Zf of the three-dimensional subject 1011)
Similarly, if the thickness of the object is negligible, the viewpoint image data a _{s, t} (X, Y
, Zf) can be expressed by translation of I (X, Y, Zf). Accordingly, the following equation is obtained when Equation 19 is translated according to the distance dZ between the viewpoint (s, t), the subject and the focal position, and is substituted into Equation 16 and transformed using Equations 17 and 18.

It can be seen that the Z stack image data g (X, Y, Zf) includes a defocus due to a deviation dZ between the subject and the focus position.

The defocused image data (layer image data) 1103 and 1113 in FIGS. 11D and 11H are obtained from a plurality of viewpoint image data corresponding to a large number of viewpoints set on the lens surface, Z = Zf. It is a schematic diagram of layer image data in. The layer image data 1103 represents image data corresponding to Expression 20, and 1113 represents image data corresponding to Expression 21.
In the layer image data 1103, the viewpoint image data whose position of the object image is not shifted is multiplied by the viewpoint weight function and integrated by the number of viewpoints, so that an image without blur is obtained. On the other hand, in the layer image data 1113, the viewpoint image data in which the position of the object image is shifted is multiplied by the viewpoint weight function and integrated by the number of viewpoints, so that an image including the blur of the optical system is obtained.
When there is a deviation dZ from the focal position of the subject, a shift in the position of the object image occurs in each viewpoint image data, and the luminance difference from the surroundings of the object becomes (ab), so that dZ and It can be seen that the larger the luminance difference (ab) is, the larger the difference between the viewpoint image data is.

(Method of extracting scattered image data using viewpoint image data)
Next, a method for extracting scattered image data using viewpoint image data will be described.
Assuming that the deviation dZ from the focal position of the subject is 0, as already described in Equation 10, the angle θ is 180 degrees at the same observation angle φ as the viewpoint image data of the predetermined viewpoint (s, t). When subtraction is performed between the viewpoint image data of different viewpoints, the scattered image data can be extracted.
Hereinafter, description will be made using equations.

(Calculation between viewpoint image data with different declination θ)
In the coordinate system shown in FIG. 7A, when the coordinates of the viewpoint P0 to be processed are (x, y) = (s _p , t _p ), the coordinates of the viewpoint P1 rotated 180 degrees are (x, y) = (−s _p , −t _p ). (The scattered image data can be extracted even when the rotation angle of the declination is other than 180 degrees, but here, the case of the 180 degree rotation that is the most effective will be described.)

数２２において１／２の係数を掛ける理由は、視点Ｐ０および視点Ｐ１のそれぞれに対応する視線方向から観察した場合に大きくなる散乱光が絶対値によって同時に抽出され、倍の強度となるのを防ぐためである。数２３では視点Ｐ１と比較して視点Ｐ０の視点画像データの強度の大きな画素を散乱光が強く発生している画素とみなし、差分が正になる画素の値だけを用いる。 I _P0 (X, Y, Zf) is the viewpoint image data observed so that the position of Z = Zf is the focal position from the viewing direction of the viewpoint P0, and the position of Z = Zf is focused from the viewing direction of the declination rotation viewpoint P1. Let the viewpoint image data observed so as to be in position be I _P1 (X, Y, Zf). Further, let the viewpoint scattered image data, which is the scattered image data obtained from the viewpoint image data of the viewpoint P0, be S _P0 (X, Y, Zf).
In that case, the scattered light information on the sample surface can be extracted by the calculation shown in the following Equation 22 or 23.

The reason for multiplying by a factor of 1/2 in Equation 22 is that scattered light that becomes large when observed from the line-of-sight direction corresponding to each of the viewpoints P0 and P1 is simultaneously extracted by the absolute value to prevent double intensity. Because. In Equation 23, a pixel whose intensity of the viewpoint image data of the viewpoint P0 is higher than that of the viewpoint P1 is regarded as a pixel in which scattered light is strongly generated, and only a pixel value having a positive difference is used.

  Hereinafter, the reason why the information on the scattered light on the specimen surface can be extracted by the arithmetic expressions of Expressions 22 and 23 will be described. As described above, the intensity change of the transmitted light of the viewpoint image data when the viewpoint is changed is small, while the luminance change due to the scattered light on the sample surface changes greatly according to the deviation angle θ. Therefore, the transmitted light component in the image data is canceled by subtraction between the viewpoint image data with different viewpoints, and the scattered light component (scattered light information) can be extracted. The transmitted light component may not be completely cancelled. If the intensity of the transmitted light component in the image data is weakened and image data in which the scattered light component in the image data is relatively enhanced can be obtained, such an operation can also be referred to as generation of scattered image data.

When the declination rotation angle is 180 degrees, the scattered light of the surface unevenness having the unevenness direction angle at which the scattered light is maximized at the declination angle of the viewpoint P0 is minimum at the declination angle of the viewpoint P1. Similarly, the scattered light of the surface unevenness having the unevenness direction angle at which the scattered light becomes the maximum at the deflection angle of the viewpoint P1 becomes the minimum at the deflection angle of the viewpoint P0.
Accordingly, if the difference D ₁ (X, Y, Zf) between the viewpoint image data of the viewpoint P0 and the viewpoint image data of the viewpoint P1 is obtained, the surface having the concavo-convex direction angle that maximizes the scattered light at the deviation angles of the viewpoints P0 and P1. Information on uneven scattered light can be extracted as image data.

ただし、Ｎは視点の数、Ｄ（ｉ）は視点ｉにおける視点画像データと偏角回転視点画像データの差分画像データ、Ｓ（）は関数であり、数２２、数２３のＳ_Ｐ０（Ｘ，Ｙ，Ｚｆ）を求める式に対応する。 In the equations 22 and 23, the reason why the absolute value or the positive value of the difference D ₁ (X, Y, Zf) is used in the calculation of the viewpoint scattered image data S _P0 (X, Y, Zf) is based on the viewpoint image data. This is because nonlinearity is required in the process of integrating a plurality of obtained viewpoint scattered image data. If the absolute value is not taken, the operation of calculating the difference between the viewpoint image data from various viewpoints and then calculating the sum of the differences is equivalent to the operation of calculating the difference after calculating the sum of the viewpoint image data of various viewpoints. Become. In that case, the viewpoint image data cancel each other, and the resultant integrated image is substantially zero. Therefore, in the scattered viewpoint image data S _P0 (X, Y, Zf), a non-linear function is used to prevent the cancellation.
As shown in the following equation, a nonlinear function is a result of adding a plurality of functions after applying the function to predetermined image data (viewpoint image data) and a plurality of predetermined image data (viewpoint image data). After that, the result of applying the function is assumed to be a function having a property that does not match.

Here, N is the number of viewpoints, D (i) is the difference image data of the viewpoint image data and the declination rotation viewpoint image data at the viewpoint i, S () is a function, and S _P0 (X, This corresponds to the equation for obtaining Y, Zf).

ただし、ｋ（ｓ，ｔ）は任意の視点重み関数であるとする。 Subsequently, image data obtained by integrating the viewpoint scattered image data S _P0 (X, Y, Zf) of Expressions 22 and 23, that is, scattered image data DS (X, Y, Zf) is considered for various viewpoints. . An expression for obtaining the scattered image data DS (X, Y, Zf) is expressed as follows.

However, k (s, t) is an arbitrary viewpoint weight function.

If integration of Formula 25 is performed for various viewpoints (s, t), scattered light components in various directions can be collected.
In the operation of equation 22, the viewpoint scatter image data _I P0 viewpoint P0 (X, Y, Zf) and the viewpoint P1 viewpoint scatter image data _{I P1 (X, Y, Zf} ) are equal. Accordingly, the viewpoint scattered image data is obtained for half of the viewpoints (s, t) to be integrated, and the weighted integration is doubled to obtain the same result as Expression 25. That is, in the case of Equation 22, there is a merit that the amount of calculation of integration is about ½. On the other hand, Equation 23 is effective in grasping the relationship between the observation direction and the scattered light intensity because the position where the scattered light is extracted also includes information on the observation direction. However, in the case where the calculation of Equation 23 is performed for a large number of viewpoints with different deflection angles θ and observation angles φ within the possible range on the lens surface, the result is the same regardless of which of Equations 22 and 23 is used. It becomes.

(Calculation between viewpoint image data with different observation angles φ)
As already described in Equation 11, the subtraction between the viewpoint image data of a certain viewpoint (s, t) and the viewpoint image data of the viewpoint or the viewpoint (0, 0) having the same declination angle θ and different observation angles φ. Can also extract scattered image data.
Even between the viewpoint image data with different observation angles φ, the observation angle change viewpoint P1 is obtained from the viewpoint P0, and after performing the calculation shown in Expression 22 or 23, scattered image data can be generated by executing Expression 25. However, when the observation angle φ is changed, the scattered image data can be generated by another calculation. This will be described below.

偏角θを回転する場合とは異なり、必ずしも非線型な演算を用いる必要はない。なぜなら、視点Ｐ０と視点Ｐ１の観察角φが異なる場合には、偏角θが異なる視点（すなわち、図７（ａ）で動径ｒ_ｐが等しくなる視点）で積分を計算した結果は等しくならないため、数２６の計算で０になることはないためである。 In the coordinate system shown in FIG. 7A, when the coordinates of the viewpoint P0 to be processed are (x, y) = (s _p , t _p ), the coordinates of the viewpoint P1 at which the observation angle φ is 0 are (x, y). y) = (0, 0). (The scattered image data can be extracted even if the angle of observation is changed to a value other than 0 degrees, but here, the case where the viewpoint P1 at which the most effective observation angle is 0 degrees is used will be described.)
When the viewpoint P0 and the viewpoint P1 are in the above relationship, scattered light can be extracted by the following formula. S _P0 (X, Y, Zf) is the viewpoint scattered image data of the viewpoint P0.

Unlike the case of rotating the deflection angle θ, it is not always necessary to use a non-linear calculation. This is because, when the viewing angle of the viewpoint P0 and the viewpoint P1 phi is different, viewpoints deflection angle θ are different (i.e., FIGS. 7 (a) viewpoint moving radius r _p is equal with) not equal the result of calculating the integral in Therefore, the calculation of Equation 26 does not become 0.

動径がｒとなる視点位置の積分は、動径がｒの視点にのみ値を持つ視点重み関数で表現できる。そのため、数２７を変形すると以下の式で変形できる。

ただし、ｋ_ｅｘ（ｓ，ｔ）は以下となる。

なお、視点重み関数ｋ_ａ１（ｓ，ｔ）、ｋ_ａ２（ｓ，ｔ）の全視点に対する積分はそれぞれ１となることから、ｋ_ｅｘ（ｓ，ｔ）の全視点に対する積分は０となる。

Assuming that the radius r of the viewpoint P0 is r1 and the radius r of the viewpoint P1 is r2, scattered image data obtained by executing integration with only the deflection angle θ changed is expressed by the following equation.

The integration of the viewpoint position where the radius is r can be expressed by a viewpoint weight function having a value only at the viewpoint where the radius is r. Therefore, if Equation 27 is transformed, it can be transformed by the following equation.

However, k _ex (s, t) is as follows.

Note that the integrals for all viewpoints of the viewpoint weight functions k _a1 (s, t) and k _a2 (s, t) are 1, respectively, and the integral for all viewpoints of k _ex (s, t) is 0.

ただし、ｒ_ｔｈは、ｒ_ｔｈ≦ｒ_ｍを満たす所定の値とする。ｒ_ｍは、散乱画像データの計算に用いる視点のうちレンズ面上で最も外側の視点、の原点（０，０）からの距離である。
数３０および数３１の条件を満たせば、ｋ_ｅｘ（ｓ，ｔ）は自由に設計できる。 As already mentioned several 11, in order to extract the information of the scattered light in _{difference, I P0 (X, Y,} Zf) viewing angle φ is _I P1 of (X, Y, Zf) from φ observation angle Also need to be big.
Therefore, when generalized, the condition of k _ex (s, t) for extracting scattered light information is a region having a radius greater than r _th (ie, an observation angle) with a predetermined radius r _th as a threshold. In a region where φ is large), it has a positive value, and in a region below r _th , it has a negative value. Here, the intensity of the scattered image data generated increases as the threshold value r _th decreases. An example of an effective condition is r _th = 0. The condition of k _ex (s, t) can be expressed by the following equation.

_{However, r th is} a predetermined value satisfying _{r th} ≦ _{r m.} r _m is the outermost point of view on the lens surface of the viewpoint used to calculate the scattering image data, which is the distance from the origin (0,0).
If the conditions of Equations 30 and 31 are satisfied, k _ex (s, t) can be designed freely.

From Equation 28, it can be seen that the scattered image data can be calculated by the difference between two arbitrary defocus image data having different defocus generated using k _a1 (s, t) and k _a2 (s, t). Further, it can be seen that the calculation can be performed also with arbitrary defocus image data generated using k _ex (s, t) that satisfies the conditions of Equations 30 and 31. Hereinafter, the viewpoint weight function calculated by Expression 29 or the viewpoint weight function k _ex (s, t) satisfying the conditions of Expression 30 and Expression 31 is referred to as a viewpoint weight function for extracting scattered light information. .

(Enhancing scattered light in scattered image data and reducing defocus)
As already described, when the subject is located away from the focal position as shown in FIG. 10D, the position of the subject in the viewpoint image data changes. Therefore, the scattered image data obtained by the calculation of Equation 25 or Equation 28 also includes the influence of the defocus of the subject away from the focal position.

ただし、Ｃ_Ｐ０（Ｘ，Ｙ，Ｚｆ）は以下の式で表わされる画像データ、つまり、視点Ｐ０の視点画像データに同じ視点Ｐ０の散乱画像データを加算した画像データである。以降、Ｃ_Ｐ０（Ｘ，Ｙ，Ｚｆ）を視点Ｐ０の散乱光強調画像データと呼ぶ。

なお、数３３のＳ_Ｐ０（Ｘ，Ｙ，Ｚｆ）は数２２または数２３で表される視点散乱画像データである。 As a method for reducing the influence of defocus while enhancing the scattered light in the scattered image data, there is an equation described below.

However, C _P0 (X, Y, Zf) is image data represented by the following expression, that is, image data obtained by adding scattered image data of the same viewpoint P0 to viewpoint image data of the viewpoint P0. Hereinafter, C _P0 (X, Y, Zf) is referred to as scattered light weighted image data of the viewpoint P0.

Note that S _P0 (X, Y, Zf) in Expression 33 is viewpoint scattered image data expressed by Expression 22 or Expression 23.

即ち、数３２は散乱光強調画像データの間の差分に相当する。 Equation 32 can be expressed by the following equation when transformed.

That is, Equation 32 corresponds to the difference between the scattered light weighted image data.

Next, the reason why the scattered light can be enhanced and the defocus can be reduced by the difference between the scattered light weighted image data of Equation 34 will be described with reference to FIG.
The stage of FIG. 12A shows (1) omnifocal image data and (2) arbitrary defocused image data when the subject is a point image at a position away from the focal position and the luminance value is lower than the surroundings. , (2) scattered light extraction image data, and (4) scattered light weighted image data. The stage (b) shows (1) omnifocal image data, (2) arbitrary defocused image data, and (3) when the subject is at a position away from the focal position and has an edge with a luminance difference. The scattered light extraction image data and (4) scattered light weighted image data are shown.
Note that the scattered light extraction image data is calculated using Equation 20, and the scattered light weighted image data is an addition value of the arbitrary defocused image data and the scattered light extraction image data expressed by the following equation. However, it is assumed that b>a> 0 holds in FIG.

ただし、ｈ（Ｘ，ｄＺ）はＸ方向とＺ方向についてのみ考慮した２次元の焦点ぼけとする。

散乱光強調画像データ１２０４では、全焦点画像データ１２０１で高輝度領域（輝度がｂの領域）において焦点ぼけが抑制できることが分かる。 The image data (omnifocal image data 1201, arbitrary defocused image data 1202, scattered light extraction image data 1203, scattered light weighted image data 1204) in the stage of FIG. Become.

However, h (X, dZ) is a two-dimensional defocus that takes into account only the X and Z directions.

In the scattered light weighted image data 1204, it can be seen that the omnifocal image data 1201 can suppress defocusing in a high-luminance region (a region where the luminance is b).

On the other hand, the image data (omnifocal image data 1211, arbitrary defocused image data 1212, scattered light extraction image data 1213, scattered light weighted image data 1214) in the stage of FIG. It becomes like this.

Similarly, the scattered light enhancement image data 1214 and the omnifocal image data 1211 can suppress defocusing in a high luminance region (a region where the luminance is b), but it is understood that the defocusing is not canceled in the low luminance region. . However, the scattered light component present in the arbitrary defocused image data 1212 and the scattered light component present in the scattered light extracted image data 1213 are added to each other, so that the information of the scattered light can be further enhanced. . Since only the transmitted light component is shown in FIG. 12, the scattered light component is not shown.
As already described, Expression 34 represents a difference between scattered light weighted image data having different viewpoint weight functions k _a1 (s, t) and k _a2 (s, t), and the viewpoint weight function is changed. This means that scattered light can be extracted by the difference between the calculated scattered light weighted image data. Accordingly, as in _Equation 28, the intensity of k _a1 (s, t) is relatively large at a position where the observation angle φ is large, and the intensity of k _a2 (s, t) is relatively large at a position where the observation angle φ is small. In this case, it can be seen that the scattered light component can be extracted efficiently.

ただし、Ｓ_Ｐ０（Ｘ，Ｙ，Ｚｆ）は数２２または数２３で表わされる視点散乱画像データである。 Subsequently, the relationship between the scattered image data (Equation 25) obtained by changing the deflection angle and the scattered image data (Equation 32) in which scattered light is emphasized is analyzed. Consider the following equation in which the viewpoint weight function k (s, t) in Expression 25 is changed to the viewpoint weight function k _ex (s, t) for extracting scattered light information.

However, S _P0 (X, Y, Zf) is the viewpoint scattered image data expressed by Equation 22 or Equation 23.

As described in Expression 10, when the observation angle φ is small, the scattered light information included in the viewpoint scattered image data S _P0 (X, Y, Zf) calculated in Expression 22 or Expression 23 is small. Therefore, if k _ex (s, t) has a negative intensity only at a small observation angle φ, the difference between the scattered light extracted image data calculated by Equation 25 and the scattered light extracted image data calculated by Equation 44 is small. In the case where the intensity is negative only when the observation angle φ is 0, the difference between the scattered light extracted image data calculated by Expression 25 and Expression 44 is 0.
It can be seen that Equation 32 is the sum of Equation 28 representing scattered light extracted image data with the observation angle φ changed and Equation 44 representing viewpoint scattered image data with the deflection angle θ changed. That is, it can be seen that Equation 32 means composite scattered light image extraction data obtained by changing the deflection angle θ and the observation angle φ. As described above, it has been described that scattered image data can be generated using viewpoint image data having a different deflection angle θ and observation angle φ.

(Influence of defocusing of subject in pathological specimen image data)
Next, the influence of the subject's defocus on the scattered image data generated from the pathological specimen will be described.
As shown at 1213 in FIG. 12B, when the subject is at a position away from the focal position and the subject has a large luminance difference (for example, an edge), the scattered light extracted image data has a luminance higher than that of the surroundings. Artifacts different from the scattered light component (defocus component), such as blurring blurring out in a low region, occur. Even when the scattered light extraction image data is obtained from the difference between the two scattered light weighted image data having different defocuss using Equation 32, it appears in the low luminance region 1214 (X is a negative region) in FIG. Since the defocus is different from each other, the same phenomenon occurs in the difference. Even when the difference between two out-of-focus image data with different defocus is obtained in Equation 28, the defocus difference remains in the same manner.

The above phenomenon tends to be particularly prominent in the low-luminance region of the edge where the luminance difference is large. For example, in a microscopic image of a HE-stained pathological specimen, the nucleus and the inside of the nucleus are stained with dark blue and the transmittance is low, and the cytoplasm portion is stained with light pink and the transmittance is high.
Therefore, since the difference in transmittance is relatively small inside the cytoplasm (in FIG. 12, the difference in luminance between a and b is small), the influence of defocusing is small, and the influence of defocusing in the low-luminance region is little. It doesn't matter.
However, since the difference in transmittance is large at the boundary between the nucleus and the cytoplasm (in the case of FIG. 12, the luminance difference between a and b is large), the defocus of the subject is in a low luminance region (in the case of 1214, the region where X is negative) Bleeding and artifacts may be observed. This phenomenon is observed in both scattered light extraction image data and scattered light weighted image data.

  In the embodiment described below, by focusing on the phenomenon that the scattered light tends to be small in the low luminance region and the scattered light is likely to be large in the high luminance region, the influence of the defocus of the subject appearing in the low luminance region is suppressed, and the scattered image is displayed. We propose a method to make the scattered light information in the data more reliable.

[Example 1]
Hereinafter, a specific example of the image data generation device 100 will be described.
(Scattering image data generation setting screen)
FIGS. 13A and 13B are examples of a setting screen for the scattered image data generation function in the first embodiment. After the area 207 in the display image is selected with the mouse in the image display application of FIG. 2, the item “scattered image data generation” (not shown) is selected from the function expansion menu 208 displayed by right-clicking the mouse. Correspondingly, a new window 1300 (FIG. 13A) showing images before and after the scattered image data generation process and a scattered image data generation function setting screen 1303 (FIG. 13B) are displayed. In the left area 1301 of the window 1300, the image data in the area 207 is displayed, and in the right area 1302, the image data of the scattered image data generation result is displayed.

  When changing the setting of the scattered image data generation function, the setting screen 1303 is operated. When the user presses the viewpoint setting button 1304 with the mouse, a setting screen for determining the line-of-sight direction (three-dimensional observation direction) used for generating scattered image data is displayed. Note that the number of viewpoints may be one or more. Details will be described later. When the user presses the scattered image data calculation setting button 1305, a scattered image data calculation setting screen for setting a method and parameters for calculating scattered image data is displayed. Various methods described above can be selected as a method for generating scattered image data, and details will be described later. If necessary, it is also possible to optionally set a noise removal parameter for the scattered image data. Details will be described later. When the user presses the transmitted light component suppression setting button 1306, a setting screen for suppressing artifacts (defocus components) in the low luminance region in the scattered image data is displayed. Details will be described later.

  An overlay display 1307 is a check box, and when this setting is validated, the image data in the selection area 207 and the scattered image data are displayed in a superimposed manner in the right area 1302. When the user makes the above settings as necessary and presses the execute button 1308, the scattered image data is generated and the processing result is displayed. Details will be described later.

  Reference numeral 1310 denotes a function expansion menu that can be called by right-clicking in the window 1300. In the function expansion menu 1310, items of image analysis such as N / C ratio calculation (not shown) are arranged. When an item is selected, an image analysis processing setting screen (not shown) is displayed, analysis processing is executed on the selected area or the entire window, and the processing result is displayed. Details will be described later.

(Scattered image data generation processing)
FIG. 14A shows a flow of scattered image data generation processing executed when the execution button 1308 is pressed. This process is realized by an image display application and an image data generation program called from the image display application.

In the Z stack image data acquisition step S1501, the necessary range of data is acquired from the Z stack image data stored in the main memory 302 or the storage device 130 based on the coordinates of the selection area 207 of the image being displayed by the image display application. To do. When the Z stack image data exists in another computer system 140, the network I / F 30
4 is acquired and stored in the main memory 302.

  Subsequently, in scattered image data calculation processing step S1502, viewpoint image data for a plurality of viewpoints is generated from the Z stack image data based on the viewpoint information for determining the viewing direction (observation direction) with respect to the subject. Then, scattered image data is generated using each viewpoint image data. Details will be described later.

  Subsequently, in contour extraction processing step S1503, contour extracted image data obtained by extracting the contour from the scattered image data is generated. Note that the processing in step S1503 is not essential, and application / non-application can be changed according to a setting (not shown). Details will be described later.

  In the image display processing step S1504, the contour extraction image data, the viewpoint scattered image data, or the scattered image data (scattered light weighted image data or scattered light extracted image data) is enlarged / reduced according to the display magnification of the image display application, and the right side. Displayed in area 1302. When the overlay display 1307 is valid, the contour extraction image data, the viewpoint scattered image data, or the scattered image data is displayed superimposed on the image data in the selection area 207. At this time, an image obtained by combining (adding or subtracting) the viewpoint scattered image data or scattered image data at the corresponding position with the image data in the selection region 207 may be displayed in the right region 1302. Further, image data obtained by tone-correcting the synthesized image data so that the brightness is close to that of the image data in the selection area 207 may be displayed in the right area 1302. You may perform the animation display which switches several viewpoint scattered image data by a fixed time interval. At this time, the contour extraction image data, the viewpoint scattered image data, or the scattered image data may be displayed with a color for each channel (RGB), or may be changed to a different color that does not overlap the sample color. . The image data used for display (contour extraction image data, viewpoint scattered image data, scattered image data, or image data obtained by synthesizing them with original image data) are all image observation and image diagnosis (including automatic diagnosis). For observation).

(Scattering image data calculation processing step S1502)
Next, the internal processing of the scattered image data calculation processing step S1502 corresponding to the main processing of the scattered image data generation processing executed by pressing the execution button 1308 will be described.
FIG. 14B is a flowchart showing the internal processing of the scattered image data calculation processing step S1502 of this embodiment, and FIG. 14C is a block diagram for realizing the function of the scattered image data calculation processing. The transmitted light component suppression mask generation unit 1701, the scattered light extraction image data generation unit 1702, and the transmitted light component suppression processing unit 1703 in FIG. 14C realize the functions of steps S1601, S1602, and S1603 in FIG. 14B, respectively. It is a block to do.

Hereinafter, a method for generating scattered image data will be described with reference to FIGS. 14B and 14C.
First, in the transmitted light component suppression mask generation step S1601, the transmitted light component suppression mask generation unit 1701 generates a transmitted light component suppression mask using the input Z stack image data, and transmits the transmitted light component suppression processing unit in the subsequent stage. 1703 is output.
The transmitted light component suppression mask is mask data having the same size in the XY direction as the image data of the focal position Z = Zf, and a coefficient assigned to each pixel. Details will be described later.
Subsequently, in the scattered light extraction image data generation step S1602, the scattered light extraction image data generation unit 1702 generates scattered light extraction image data by using the input Z stack image data, and performs transmitted light component suppression processing in the subsequent stage. Output to the unit 1703. Details will be described later.
Finally, in the transmitted light component suppression processing step S1603, the transmitted light component suppression processing unit 1703 uses the transmitted light component suppression mask and the scattered light extracted image data to generate corrected scattered light extracted image data in which the transmitted light component is suppressed. Generate. Details will be described later.

(Transmission light component suppression mask generation step S1601)
Subsequently, an internal process of the transmitted light component suppression mask generation step S1601 will be described. FIG. 15A is a flowchart showing the internal processing of step S1601.
In the transmitted light component suppression mask generation step S1601, a transmitted light component suppression mask is generated based on the transmitted light component suppression setting (1403). The process will be described with settings.
A transmitted light component suppression setting screen 1403 in FIG. 13E is an example of a setting screen displayed when the transmitted light component suppression setting button 1306 is pressed. Here, information for suppressing the defocus component of the subject in the low luminance region is set.
On the setting screen 1403, a “method” for specifying a transmitted light component suppression method, a “reference image” for specifying reference image data to be referred to when generating a transmitted light component suppression mask, and a gradation correction to be applied to the reference image data. The “correction characteristics” for selecting the type can be selected from the drop-down list.
In step S1601, a transmitted light component suppression mask is generated based on the setting information.

  First, in reference image data generation step S1801, reference image data for generating a transmitted light component suppression mask is generated from input Z stack image data. As the reference image data, image data of various types or characteristics can be used as long as the image data represents the luminance characteristics of the Z stack image data that is the original image data. A desired type or characteristic of reference image data can be set in the “reference image” on the setting screen 1403. Typically, viewpoint image data generated from Z stack image data and viewed from a viewpoint passing through the optical axis, that is, omnifocal image data, may be used as reference image data. Alternatively, arbitrary defocused image data generated from the Z stack image data (that is, image data having a defocus different from the original layer image data) may be used. For example, image data with a larger depth of field than the original layer image data may be used. Alternatively, any one of the layer image data selected from the Z stack image data may be used as the reference image data. Further, in the case of Z stack image data acquired with a microscope having a bilateral telecentric optical system, sensor fixed pattern noise tends to appear in the omnifocal image data. In such a case, noise reduction processing such as a median filter may be performed on the omnifocal image data. Further, the omnifocal image data may be generated by extracting a region having high contrast in the XY directions from each of the plurality of layer image data at the Z position in the Z stack image data and combining them. Since the method for generating arbitrary viewpoint image data and arbitrary defocused image data has already been described, description thereof will be omitted.

Next, in luminance conversion step S1802, the omnifocal image data generated in step 1801 is converted into luminance, and reference image data for mask reference is generated. Note that the omnifocal image data may be generated after the order of steps S1801 and S1802 is changed and the luminance conversion is first performed on the Z stack image data. Here, luminance conversion processing will be described. When the Z stack image data is monochrome image data, the luminance value is left as it is. When the Z stack image data is an RGB color image, the luminance value (Y) is obtained from the pixel values (R, G, B, respectively) of each color by a mathematical formula. For example, in the case of an NTSC signal used in analog television broadcasting, the conversion formula from RGB values to luminance values (Y) is
Y = 0.299 × R + 0.587 × G + 0.114 × B
It becomes. Since the conversion formula from RGB values to luminance values differs depending on the color space (sRGB, AdobeRGB, etc.) representing the signal, it is preferable to use a conversion formula corresponding to each color space.

Next, in tone correction step S1803, tone correction is performed on the reference image data. Here, the gradation correction set in the “correction characteristics” on the setting screen 1403 is applied. Examples of tone correction include “correction using tone curve”, “binarization”, and “adaptive binarization”. Binarization is binarization using a predetermined (fixed) threshold, and adaptive binarization is binarization that dynamically selects an appropriate threshold according to the luminance distribution of image data. The threshold value for binarization may be set directly by the user. When applied to a HE-stained pathological specimen, it is preferable to set a threshold that can separate the nucleus and cytoplasm regions. When the threshold value for identifying the nucleus and cytoplasm region is determined using color components, luminance conversion may not be performed in step S1802, and specific color components such as R and B may be input in step S1803. In binarization, a region with a higher luminance than the threshold is set to the maximum luminance (255), and a region with a lower luminance is set to the lowest luminance (0). Note that gradation correction step S1803 is not an essential process (when “No gradation correction” is selected on the setting screen 1403, step S1803 is skipped).

なお、Ｉ_ｒｅｆ（Ｘ，Ｙ，Ｚｆ）は参照画像データを表し、Ｉ_ｍａｘは最大輝度値（８ビット画像データの場合は２５５）を表す。すなわち、透過光成分抑制マスクは、参照画像データ（又は諧調補正した参照画像データ）の各画素の輝度値を最大輝度値で正規化した画像データである。 Finally, in mask coefficient calculation processing step S1804, a transmitted light component suppression mask M (X, Y, Zf) is generated from the reference image data subjected to gradation correction using the following equation.

I _ref (X, Y, Zf) represents reference image data, and I _max represents a maximum luminance value (255 in the case of 8-bit image data). That is, the transmitted light component suppression mask is image data obtained by normalizing the luminance value of each pixel of the reference image data (or the reference image data subjected to gradation correction) with the maximum luminance value.

FIG. 15B is a graph showing the relationship between the output coefficient (vertical axis) of the transmitted light component suppression mask and the input luminance (horizontal axis) of the reference image data. Reference numeral 1901 indicates a relationship when no gradation correction is performed, 1902 indicates a gradation correction by a tone curve, and 1903 indicates a relationship when binarization processing is performed. In any case, the coefficient of each pixel of the transmitted light component suppression mask has a value range of 0 to 1, a low coefficient in the low luminance region, and a high coefficient in the high luminance region. That is, the coefficient value of each pixel of the transmitted light component suppression mask is set to have a positive correlation with the luminance of each pixel of the reference image data.
By using the transmitted light component suppression mask as shown in FIG. 15B, it is possible to suppress the defocus component of the subject in the low luminance region of the scattered light extraction image data generated in step S1602. Details will be described later.

(Scattered light extraction image data generation step S1602)
FIG. 16A is a flowchart showing the internal processing of the scattered light extraction image data generation step S1602.
First, in the scattered light extraction image data generation method determination processing step S2001, the scattered light extraction image data generation method is determined based on the “method” selected on the setting screen 1402. As described above, if the method is different, the calculation method in the subsequent scattered image data generation execution processing step S2002 is different. In the setting screen 1402, “deflection angle change”, “observation angle change”, “composite change”, and the like can be selected. Subsequently, in scattered image data generation execution processing step S2002, scattered image data is generated based on the parameters set on the setting screens 1401 and 1402.

(Scattering image data generation execution processing step S2002: declination change or compound change)
FIG. 16B is a flowchart showing the internal processing of the scattered image data generation execution processing step S2002 when “deflection change” or “composite change” is selected as the “method” on the setting screen 1402. When “change declination” is selected, an operation corresponding to Equation 44 is performed, and when “composite change” is selected, an operation corresponding to Equation 32 is performed.
First, in viewpoint acquisition processing step S2101, the position information of the viewpoint necessary for generating viewpoint image data is acquired in step S2102 in the subsequent stage. In step S2101, position information of a predetermined viewpoint may be acquired from the main memory 302, the storage device 130, or another computer system 140. In step S2101, the position information of the viewpoint may be calculated and obtained based on information set on the image display application. Details will be described later.

  Subsequently, in viewpoint image data generation step S2102, viewpoint image data corresponding to the viewpoint obtained in step S2101 is generated based on the Z stack image data in the selection area 207. This function executed by the image data generation device 100 (CPU 301) is referred to as viewpoint image data generation means. Note that methods for generating arbitrary viewpoint image data from the Z stack image data (MFI arbitrary viewpoint / in-focus image generation method) include the methods described in Patent Document 1, Non-Patent Documents 2, 3, 4, and the like. Any method may be used.

  Subsequently, in viewpoint scattered image data generation processing step S2103, scattered image data generation processing is performed on the generated viewpoint image data based on the scattered image data calculation setting (1305). When there are a plurality of viewpoints, viewpoint scattered image data generation processing is executed for the number of viewpoints. Subsequently, in the viewpoint scattered image data integration processing step S2104, based on the scattered image data calculation setting (1305), the plurality of viewpoint scattered image data generated in step S2103 are integrated to generate scattered image data. The function of steps S2103 to S2104 executed by the image generation apparatus 100 (CPU 301) is called scattered image data generation means (feature image data generation means). However, when the scattered light of the subject is extracted, the scattered light extraction image data generation unit is called, and when the scattered light of the subject is emphasized, it is called the scattered light weighted image data generation unit. Details will be described later.

  Details of the viewpoint acquisition processing step S2101, the viewpoint scattered image data generation processing step S2103, and the viewpoint scattered image data integration processing step S2104 will be described below.

(Viewpoint acquisition processing step S2101)
Hereinafter, based on the viewpoint setting (1304), the process of calculating the position information of the viewpoint in the viewpoint acquisition process step S2101 will be described.
A viewpoint setting screen 1401 in FIG. 13C is an example of a setting screen displayed when the viewpoint setting button 1304 in FIG. 13B is pressed. Here, the viewpoint position of the viewpoint image data used to generate the scattered image data is set.
On the viewpoint setting screen 1401, “direct setting” and “mesh setting” can be selected as viewpoint setting methods. In the direct setting, the number of viewpoints and the position (s, t) of viewpoints are directly specified by the user. On the other hand, in the mesh setting, the user is allowed to specify the outer diameter, inner diameter (center shielding), and discretization step, and the position of each viewpoint is calculated from these specified values.

Specify the maximum deviation of the viewpoint to be calculated (the distance from the origin of the viewpoint when the position where the optical axis and the lens surface intersect) as the origin, and the "inner diameter (center shielding)" The minimum deviation amount of the viewpoint to be calculated (that is, the maximum deviation amount of the viewpoint that is not calculated) is designated. Here, the values of the outer diameter and inner diameter (center shielding) are set according to the distance (radius) centered on the origin on the lens surface. It should be noted that the outer diameter value exceeding the radius r _m of the optical system on the lens surface can not be set. In the “discretization step”, the position of the viewpoint for generating the viewpoint image data is discretely set in a donut-shaped region obtained by removing the circle specified by the “inner diameter” from the circle specified by the “outer diameter”. It is a step interval for. The finer the discretization step, the more viewpoints to calculate.

  Various shapes can be set in addition to the circles described above. For example, a plurality of concentric circles having different radii and straight lines extending from the center to the radiation can be set. In the case of concentric circle setting, a discretization step (for example, setting of angular intervals) for determining the radius of each circle and the density of viewpoints on each circle can be set. Further, in the case of a straight line extending from the center to the radiation, a discretization step for determining the line spacing (for example, setting of the angular spacing) and the density of the viewpoint on the radiation can be set.

(Viewpoint scattered image data generation processing step S2103)
A scattered image data calculation setting screen 1402 in FIG. 13D is an example of a setting screen displayed when the scattered image data calculation setting button 1305 in FIG. 13B is pressed. Here, parameters such as the type of generation of scattered light enhanced image data or scattered light extracted image data, its calculation method, and viewpoint weight function are set.
In the “type” of the setting screen 1402, either “scattered light emphasized image” or “scattered light extracted image” is selected from the group box under “type” according to the purpose of use of the user.
From the drop-down list under “Method”, the calculation method of the scattered image data used in the viewpoint scattered image data generation processing step S2103 can be selected. As described above, in this embodiment, the calculation method can be selected from “change of declination”, “change of observation angle”, and “composite change”.

From the drop-down list below “viewpoint weight function 1” and “viewpoint weight function 2”, the viewpoint weight function used in the viewpoint scattered image data integration processing step S2104 can be selected. The drop-down list displays multiple parameter settings that combine the shape of the focus blur, such as a Gaussian blur expressed by a Gaussian function or a cylindrical blur expressed by a cylinder, and its radius, and the user can select from these parameter settings. . In the viewpoint scattered image data generation processing step S2103, a viewpoint weight function for extracting scattered light information is generated using the viewpoint weight functions selected from the drop-down lists of “viewpoint weight function 1” and “viewpoint weight function 2”. . ("Viewpoint weight function 1" corresponds to k _a1 (s, t) in formula 29, and "view weight function 2" corresponds to k _a2 (s, t) in formula 29. Therefore, for extracting scattered light information The viewpoint weight function k _ex (s, t) can be calculated by the difference between the two.)

  Also, noise removal setting can be made on the viewpoint scattered image data calculation setting screen 1402. As the noise removal setting, binarization using a threshold, a median filter, a bilateral filter capable of removing noise while maintaining an edge, or the like can be applied. By this process, scattered image data having a clear contrast can be generated, and the N / C ratio can be detected more easily.

また、図２５（ｂ）の視点重み関数ｋ_ａ（ｓ，ｔ）は以下の式で表現できる。

σはガウス関数の標準偏差を表す。σによりぼけの広がりを制御できる。
数４６および数４７に示すｋ_ａ（ｓ，ｔ）の積分は１であり、数１８の条件を満たす。 (Example of viewpoint weight functions that can be set)
FIG. 25A and FIG. 25C are schematic diagrams illustrating examples of viewpoint weight functions, and FIG. 25C is a schematic diagram illustrating examples of viewpoint weight functions for extracting scattered light information. FIG. 25A shows a viewpoint weight function represented by a cylindrical shape, and FIG. 25B shows a viewpoint weight function represented by a Gaussian function. r _m is the distance from the origin (0,0) of any focal among viewpoint used to generate blurred image, the outermost point of view on the lens surface.
The viewpoint weight function k _a (s, t) in FIG. 25A can be expressed by the following equation.

Further, the viewpoint weight function k _a (s, t) in FIG. 25B can be expressed by the following expression.

σ represents the standard deviation of the Gaussian function. The spread of blur can be controlled by σ.
Integration of _k a (s, t) shown in the equation 46 and the number 47 is 1, the number 18 satisfy the.

The three-dimensional focal blur of the imaging optical system cannot be expressed strictly by the viewpoint weight function due to the influence of wave optical blur and various aberrations, but it is relatively well approximated by the Gaussian viewpoint weight function shown in Equation 47. it can. The viewpoint weight function in FIG. 25 (c) is obtained by subtracting the viewpoint weight function in FIG. 25 (b) from the viewpoint weight function in FIG. 25 (a), and satisfies the conditions of Expressions 30 and 31. That is, in the viewpoint weight function of FIG. 25 (c), the integral of all viewpoints is 0, and the integral of the area where the viewpoint radius r is equal to or smaller than the predetermined threshold value r _th is negative, and from the predetermined threshold value r _th The integration in the large area is positive.

Next, a method for generating viewpoint scattered image data in the viewpoint scattered image data generation processing step S2103 will be described.
As already described in Equation 10, subtraction between the viewpoint image data of the predetermined viewpoint (s, t) and the viewpoint image data of the viewpoint with the same observation angle φ and the deviation angle θ different by 180 degrees gives The shading information due to the difference in transmittance is canceled out, and scattered image data can be generated efficiently.

FIG. 16C is a flowchart showing the internal processing of the viewpoint scattered image data generation processing S2103 in the present embodiment. Hereinafter, a method for generating scattered image data for each viewpoint will be described with reference to FIG.
First, in a declination rotation viewpoint calculation step S2301, a declination rotation viewpoint in which the declination angle θ is at a position rotated by a predetermined angle with respect to the viewpoint to be processed is calculated. Since the rotation angle is most preferably 180 degrees, the case of 180 degrees will be described below.
In the coordinate system shown in FIG. 7A, when the coordinates of the viewpoint P0 to be processed are (x, y) = (s _p , t _p ), the coordinates of the declination rotating viewpoint P1 rotated by 180 degrees are (x, y). y) = (− s _p , −t _p ).

Next, in a declination rotation viewpoint image data generation step S2302, viewpoint image data observed from the declination rotation viewpoint P1 calculated in step S2301 is generated. Since the viewpoint image data generation method has already been described in the viewpoint image data generation step S2102, it will be omitted. If the viewpoint image data of the declination rotation viewpoint P1 has already been calculated in the viewpoint image data generation step S2102, it is not calculated again and exists in the storage device 130 and the main memory 303 of the image data generation apparatus 100. Read and use data.
Then the viewpoint scatter image data generating operation step S2303, the viewpoint image data _I P0 of the processing target viewpoint P0 (X, Y, Zf) and viewpoint image data _I P1 of the polarization angle spinning viewpoint P1 (X, Y, Zf) and From this, viewpoint scattered image data is generated and output. Specifically, when “change declination” is selected as the method on the setting screen 1402, the calculation shown in Expression 22 or 23 is performed, and when “Compound change” is selected, the calculation of Expression 33 is performed. Is called.

(Viewpoint scattered image data integration processing step S2104)
In the viewpoint scattered image data integration processing step S2104, the plurality of viewpoint scattered image data are integrated to generate scattered image data (scattered light weighted image data or scattered light extracted image data).
Specifically, when “changing declination” is selected as the method on the setting screen 1402, the calculation of Formula 44 is performed, and when “composite change” is selected, the calculation of Formula 32 is performed.
At this time, k _ex (s, t) in _Expression 44 and Expression 32 is obtained from the parameters set in “Viewpoint Weight Function 1” and “Viewpoint Weight Function 2” on the setting screen 1402, k _a1 (s, t), k After calculating _a2 (s, t), calculation is performed using Equation 29.

In the flowchart of FIG. 16B, the scattered image data storage buffer is initialized with 0 before entering the viewpoint loop processing, and the calculation in the integration of Equation 44 or Equation 32 is executed in Step S2103. The result may be added to the scattered image data storage buffer. In that case, the calculation in step S2104 is not necessary.
Further, the scattered light extraction image data is obtained by using the expression 25 instead of the expression 44 and using an arbitrary viewpoint weight function k (s, t) instead of the viewpoint weight function k _ex (s, t) for extracting scattered light information. Can also be generated and is within the scope of the present invention.
In the viewpoint scattered image data integration processing step S2104, a process for removing noise included in the scattered image data may be performed as in the viewpoint scattered image data generation processing step S2103. In that case, noise removal setting is performed on the scattered image data calculation setting screen 1402. A numerical value less than 0 in the generated scattered image data is set to 0. This is because there is almost no scattered light component in the region less than 0, and there are many defocus components (this process also has an effect of noise removal).

(Scattering image data generation execution processing step S2002: change of observation angle)
FIG. 16D is a flowchart showing the internal processing of the scattered image data generation execution processing step S2002 when “change observation angle” is selected as the “method” on the setting screen 1402.
The processing of the viewpoint acquisition processing step S2201 is the same as that of step S2101 in FIG.
Next, in viewpoint image data generation step S2202, viewpoint image data corresponding to the viewpoint obtained in step S2201 is generated based on the Z stack image data. This function executed by the image data generation device 100 (CPU 301) is referred to as viewpoint image data generation means. As a method for generating arbitrary viewpoint image data from the Z stack image data (MFI arbitrary viewpoint / in-focus image generation method), any method including the method described in Patent Document 1, Non-Patent Document 2, 3, 4 and the like described above is applicable. A method may be used.

When the process of step S2202 is completed for all viewpoints obtained in step S2201, the process advances to viewpoint image data integration process step S2203.
In the viewpoint image data integration processing, step S2202 is performed using the viewpoint weight function for extracting scattered light information calculated using Equation 29 based on “viewpoint weight function 1” and “viewpoint weight function 2” selected on the setting screen 1402. Integrate the viewpoint image data obtained in step 1. In the integration of the viewpoint image data, a calculation corresponding to Equation 28 is performed. Thereby, scattered image data is obtained.
In the viewpoint image data integration processing step S2203 as well, noise removal may be performed according to the noise removal setting on the setting screen 1402 as in step S2104 of FIG. In addition, a numerical value less than 0 in the generated scattered image data is set to 0.

(Transmission light component suppression processing step S1603)
Then, the process of the transmitted light component suppression process step S1603 of FIG.14 (b) is demonstrated.
In step S1603, processing is performed to suppress artifacts (defocus components) included in the scattered image data generated in step S1602. As described above, the artifact which is a problem in this case appears remarkably in a low-luminance region in the Z stack image data (original image data obtained by photographing) in the scattered image data (feature image data). Therefore, in the present embodiment, the artifact is reduced by an operation of performing a correction process for reducing the luminance on pixels corresponding to at least the low luminance region in the original image data in the scattered image data. The scattered image data (feature image data) after luminance correction is referred to as corrected image data. FIG. 17A is a flowchart showing the internal processing of step S1603. The process will be described with settings.
First, in the suppression method determination processing step S2401, a subsequent step is performed based on the transmitted light component suppression processing method (eg, “multiplication”) selected in the “method” of the transmitted light component suppression setting screen 1403 in FIG. In S2402, the method to be executed is determined.
Next, in the suppression execution processing step S2402, artifacts (defocus components) in the scattered light extracted image data are suppressed based on the method determined in step S2401.

In the transmitted light component suppression processing unit 1703 of this embodiment, the transmitted light component suppression mask M (X, Y, Zf) and scattered light extraction image data are based on “multiplication” which is the “method” selected on the setting screen 1403. Multiply between DS (X, Y, Zf). Thereby, corrected scattered light extraction image data (corrected image data) MDS (X, Y, Zf) in which artifacts (defocus components) in the low luminance region are suppressed is generated. The multiplication between the transmitted light component suppression mask M (X, Y, Zf) and the scattered light extracted image data DS (X, Y, Zf) is the transmitted light with respect to the pixel value of each pixel of the scattered light extracted image data. This is an operation of multiplying the coefficient of the corresponding pixel of the component suppression mask. In the formula:

FIG. 17B is a one-dimensional schematic diagram showing processing and effects in the suppression execution processing step S2402 when “multiplication” is selected as the method. Details will be described with reference to FIG.
Reference numeral 2501 in FIG. 17B represents reference image data of a subject that has an edge with a large luminance difference and is located away from the focal position. Reference numeral 2502 denotes a transmitted light component calculated by the transmitted light component suppression mask generation unit 1701. The suppression mask M (X, Y, Zf) is represented. In the mask 2502, the coefficient in the low luminance region (region where X is negative) is small. The coefficient of the mask 2502 has a value of 0-1. Note that as the reference image data 2501, as described above, omnifocal image data, arbitrary defocused image data having arbitrary defocusing, or the like can be used.

Reference numeral 2503 denotes scattered light extraction image data DS (X, Y, Zf) calculated by the scattered light extraction image data generation unit 1702. Artifacts (defocus components) occur in the low luminance region (region where X is negative) of the edge. Note that although the scattered light component is also included in 2503, it is not shown in the figure.
2504 is the calculation result of Formula 48. In the transmitted light component suppression mask 2502, the coefficient of the low luminance region (X is a negative region) is relatively smaller than the coefficient of the high luminance region (X is a positive region). Therefore, in the image data 2504 obtained by multiplying the scattered light extracted image data 2503 by a coefficient, artifacts (defocus components) in the low luminance region (region where X is negative) are reduced.

Note that the setting screens shown in FIGS. 13C to 13E are merely examples. It is desirable to provide a default setting or a function for automatically setting an optimum value so that a pathologist as a user can quickly observe and diagnose without being troubled by the setting.
The scattering image data calculation process (S1502 in FIG. 14A) in the present embodiment has been described above.

(Outline extraction processing)
Next, an example of the contour extraction process (S1503 in FIG. 14A) will be described.
In the scattered image data, although the scattered light component in the subject is emphasized, there are noise and signal strength. Therefore, contour extraction processing is performed to make the contour easier to see. For example, the scattered image data is binarized (the threshold value for binarization may be a predetermined value or may be dynamically determined), and then the expansion / reduction process is repeated to thereby define the contour. Can be extracted. In addition, there are various known techniques for the contour extraction method, and any method can be applied here. Further, by adding a thinning process, it is possible to improve the position accuracy where the contour exists. As a result of the processing, contour extraction image data is obtained from the scattered image data.

(Display and analysis of image data)
Subsequently, through the image display processing S1504, the viewpoint scattered image data, scattered image data, or contour extraction image data is displayed on the image display application, so that the cell boundary between cells, the boundary between cells and sinusoids, etc. Easy to understand. This makes it easier for the pathologist to image the three-dimensional structure of the affected tissue.
Further, by right-clicking the mouse in the window 1300, the function expansion menu 1310 is called, and an image analysis can be performed by selecting an item such as N / C ratio (nucleus / cytoplasm ratio) calculation.

FIG. 18 is an example of a processing flow for calculating the N / C ratio.
In calculating the N / C ratio, it is assumed that the image data of the selection area 207 in the left area 1301 and the image data of the contour extraction image data are used. Hereinafter, the nucleus portion in the image data is called a nucleus region, the cytoplasm portion surrounding the nucleus is called a cytoplasm region, and the whole of the nucleus region and the cytoplasm region is called a cell region.

  First, in the nuclear region determination processing step S2601, the nuclear region is determined. Examples include the following methods. In HE staining, the inside of the nucleus is dyed dark blue, so whether or not the pixels in the selection area 207 located in the corresponding closed area in the contour extraction image belong to a predetermined range of color gamut at a certain ratio or more. It can be determined whether or not it is a nuclear region. The ratio and color gamut used for discrimination may be learned in advance using a plurality of samples.

  Subsequently, in the cytoplasm region determination processing step S2602, the cytoplasm region is determined. In HE staining, the cytoplasm is stained pink. Therefore, as in the nuclear region determination process, whether or not the pixels in the selection region 207 located in the corresponding closed region in the contour extraction image data belong to a predetermined range of color gamut at a certain ratio or more, It can be determined whether or not. Thereafter, the cytoplasm region is specified by excluding the closed region regarded as the nucleus region in step S2601 from the cell region. The ratio and color gamut used for the discrimination here may be learned using a plurality of samples in advance.

If automatic processing does not provide sufficient accuracy, the user may intervene (help) to determine the area. In that case, after step S2602, a setting screen that allows the user to modify the contour, nucleus region, and cell region is displayed on the GUI.
Finally, in the N / C ratio calculation processing step S2603, the area of the nucleus region obtained as described above is divided by the area of the cytoplasm region to obtain the N / C ratio.
The N / C ratio calculation flow described above is merely an example, and various other modifications and improvements are possible.

(Advantages of this embodiment)
As described above, in the present embodiment, the low-luminance region artifacts (defocusing component) are obtained by using the property of low scattered light in the low-luminance region and multiplying the scattered light extraction image data by the mask calculated from the subject. Can be reduced. This enables users to observe surface irregularities (scattered light) at the cytoplasm and cell boundaries without worrying about the effects of low-luminance artifacts (focal components) present in the nucleus of pathological specimens. Can also respond.
Note that any of the methods described in the present embodiment can extract sample scattered image data from Z stack image data. Therefore, only by image processing (post-processing by a computer) (that is, without changing the optical system and exposure conditions at the time of imaging), cell membranes and cell boundaries, cells and tubes and cavities that are useful for observing specimens The boundary can be made clear. Furthermore, surface unevenness that hardly appears as a change in luminance or color can be visualized with an increased contrast.
Thereby, diagnosis support functions such as presentation of images useful for diagnosis and calculation of N / C ratio can be realized.

  In this embodiment, the scattered image data generation process is executed when the execution button 1308 is pressed. However, the setting parameters shown in FIGS. 13B and 13C to 13E are changed. You may make it perform a scattered image data generation process, whenever it changes. If it does so, a processing result will be displayed in real time synchronizing with the change of a setting parameter. In the case of this configuration, the setting items shown in FIGS. 13B and 13C to 13E may be deployed and arranged in one setting screen. Such mounting forms are also within the scope of the present invention.

[Example 2]
In the first embodiment described above, the processing for the scattered light extracted image data is described as an example of the scattered image data. On the other hand, in the second embodiment, a method for suppressing artifacts (focal blur components) in the low luminance region of the scattered light weighted image data will be described. As already described, the scattered light weighted image data is image data obtained by adding scattered light extracted image data to arbitrary defocused image data.

(Scattering image data calculation setting screen 1402)
In this embodiment, “scattered light weighted image” is selected as the “type” setting on the scattered image data calculation setting screen 1402. As in the first embodiment, the “method” can be selected from “change in declination”, “change in observation angle”, and “combined change”. The “viewpoint weighting function 1” and “viewpoint weighting function 2” are the same as in the first embodiment. From the drop-down list, the shape of the focal blur such as a Gaussian blur represented by a Gaussian function or a cylindrical blur represented by a cylindrical shape and its radius. Parameter settings combined with can be selected. Note that the viewpoint setting screen 1401 and the transmitted light component suppression setting screen 1403 are the same as those in the first embodiment, and thus description thereof is omitted.

FIG. 19A is a flowchart showing the internal processing of scattered image data calculation processing step S1502 in this embodiment. FIG. 19B is a block diagram for realizing the scattered image data calculation processing function. Hereinafter, a method of generating scattered image data (scattered light weighted image data) will be described with reference to FIGS. 19A and 19B.
The difference between the flowchart of this embodiment (FIG. 19A) and the flowchart of FIG. 1 (FIG. 14B) is that scattered light weighted image data generation processing step S2704 exists. The difference between the functional block of this embodiment (FIG. 19B) and the functional block of Embodiment 1 (FIG. 14C) is that a scattered light weighted image data generation unit 2804 exists. Steps S2701 to S2703 in FIG. 19A are the same as steps S1601 to S1603 in FIG. Further, blocks 2801 to 2803 in FIG. 19B have the same functions as blocks 1701 to 1703 in FIG. Hereinafter, the scattered light weighted image data generation processing step S2704 and the scattered light weighted image data generation unit 2804 will be described in detail.

(Scattered light weighted image data generation unit 2804)
Input / output of the scattered light weighted image data generation unit 2804 in FIG. 19B will be described. The scattered light weighted image data generation unit 2804 receives the Z stack image data and the corrected scattered light extraction image data MDS (X, Y, Zf) calculated from the information on the setting screens 1401 to 1403. The corrected scattered light extraction image data MDS (X, Y, Zf) is obtained by, for example, the calculation of Formula 48.
The scattered light weighted image data generation unit 2804 executes the process of the scattered light weighted image data generation processing step S2704. From the Z stack image data and the corrected scattered light extraction image data MDS (X, Y, Zf), corrected scattered light weighted image data MComp (X, Y, Zf) in which artifacts (defocus components) in the low luminance region are suppressed is obtained. Generated and output.

なお、αは散乱光の強調の度合いを決める合成係数であり、正の値を持つ。αの設定は設定画面１４０３でユーザにより変更可能である（不図示）。以降の説明ではα＝１の場合について説明する。 FIG. 20 is a flowchart showing internal processing of scattered light weighted image data generation processing step S2704. Hereinafter, the process of FIG. 20 will be described.
In the arbitrary defocus image data generation processing step S2901, first, arbitrary defocus image data a (X, Y, Zf) is generated from the Z stack image data. Any method can be used to generate the arbitrary defocused image data a (X, Y, Zf) from the Z stack image data, including the methods described in Patent Document 1, Non-Patent Documents 2, 3, 4, and the like. Good. The arbitrary defocus image data includes image data having a defocus different from the layer image data constituting the Z stack image data, and image data having the same defocus as the layer image data. Any image data can be generated by the method of Patent Document 1, Non-Patent Document 2, 3, 4 or the like. For the latter image data, any one of the layer image data selected from the Z stack image data is used as it is. Also good.
Next, in the corrected scattered light enhanced image data generation processing step S2902, the corrected scattered light extracted image data MDS (X, Y, Zf) generated in step S2703 is scaled by the coefficient α, and the arbitrary defocused image data a (X , Y, Zf). Thereby, corrected scattered light weighted image data MComp (X, Y, Zf) is generated. Expressed as a formula,

Α is a synthesis coefficient that determines the degree of enhancement of scattered light, and has a positive value. α can be changed by the user on the setting screen 1403 (not shown). In the following description, the case of α = 1 will be described.

When “deflection change” is set in the “method” on the setting screen 1402, the “viewpoint weighting function 1” set on the setting screen 1402 is used for generating arbitrary defocused image data and scattered light weighted image data. The corrected scattered light weighted image data generated in step S2902 is expressed as follows.

Next, the processing and effects of the scattered light weighted image data generation processing step S2704 will be described with reference to FIGS. 21A and 21B.
FIG. 21A is a schematic diagram showing the process performed by the transmitted light component suppression processing unit 2803 in one dimension. FIG. 21B is a schematic diagram showing the process performed by the scattered light weighted image data generation unit 2804 in one dimension.
In FIG. 21A, 3001 represents scattered light extraction image data DS (X, Y, Zf), 3002 represents transmitted light component suppression mask M (X, Y, Zf), and 3003 represents corrected scattered light extraction image data. MDS (X, Y, Zf) is represented. As already described in the first embodiment, in the corrected scattered light extraction image data MDS (X, Y, Zf), artifacts (defocus components) in the low luminance region (region where X is negative) of the edge are suppressed. .
Reference numeral 3004 in FIG. 21B represents arbitrary defocus image data a (X, Y, Zf) calculated from the Z stack image data. When the corrected scattered light extraction image data 3003 is added to the arbitrary defocused image data 3004, corrected scattered light weighted image data MComp (X, Y, Zf) indicated by 3005 can be generated.

  In the corrected scattered light weighted image data 3005, compared to the scattered light weighted image data 1214 shown in FIG. 12, artifacts (defocus components) in the low luminance region (region where X is negative) are suppressed. On the other hand, in the high-luminance region (region where X is positive), the defocus is suppressed similarly to the scattered light weighted image data 1214. In addition, since the scattered light component (not shown) is included in the high luminance region of the scattered light extraction image data 3003, the scattered light component is emphasized in the corrected scattered light weighted image data 3005.

In the flowchart shown in FIG. 19A, the integration of a plurality of viewpoints is executed in each of steps S2702 to S2704. However, as shown in Formula 50, it is also possible to perform integration after performing the calculations in steps S2702 to S2704 for each viewpoint, and this method is also included in the scope of the present invention.
When “observation angle change” or “composite change” is set in the “method” on the setting screen 1402, “viewpoint weighting function 1” set on the setting screen 1402 is used to generate arbitrary defocused image data. Further, in the generation of scattered light weighted image data, the viewpoint weight function for extracting scattered light information calculated from the setting information of “viewpoint weight function 1” and “viewpoint weight function 2” set on the setting screen 1402 is used. Even in such a case, since the corrected scattered light extraction image data MDS (X, Y, Zf) is used, the scattered light can be enhanced while suppressing artifacts (defocus components) in the low luminance region.

(Advantages of this embodiment)
According to the method of the present embodiment, it is possible to reduce the influence of defocusing in the low luminance region in the scattered light weighted image data. This allows users to observe surface irregularities at the cytoplasm and cell boundaries and scattered light in the specimen without worrying about the effects of artifacts (defocusing components) in low-luminance areas such as the nucleus of pathological specimens. I can respond.

[Example 3]
In this embodiment, a description will be given of a method for further suppressing artifacts (defocus components) in a low luminance region by changing the value range of the transmitted light component suppression mask M (X, Y, Zf).

In this embodiment, threshold processing is performed in step S1803 in FIG. 15A, and the high luminance region is set to the maximum luminance (255), and the low luminance region is set to the negative maximum luminance (−255) instead of 0. Thereby, in step S1804, a transmitted light component suppression mask M (X, Y, Zf) having a range of −1 to +1 can be generated.
By using this mask, it is possible to further reduce the defocus in the low luminance region, and to enhance the scattered light in the high luminance region. In this embodiment, “scattered light image enhancement” is selected for “type” of the scattered image data calculation setting 1402, and “change observation angle” is selected for “method”. In addition, “Multiply” is selected in “Method” on the transmitted light component suppression setting screen 1403.

Next, the effect when using the transmitted light component suppression mask M (X, Y, Zf) will be described. Since the flow of the scattered image data generation process is the same as that of the second embodiment, the block diagram of FIG. 19B is also referred to in the following description.
22A to 22D show the insides of the transmitted light component suppression mask generation unit 2801, the scattered light extraction image data generation unit 2802, the transmitted light component suppression processing unit 2803, and the scattered light weighted image data generation unit 2804, respectively. It is the schematic diagram which demonstrated the process in one dimension.

In FIG. 22A, a transmitted light component suppression mask 3102 having a value range of −1 to +1 is generated from omnifocal image data (reference image data) 3101 of the edge of a subject having a luminance difference.
In FIG. 22B, a viewpoint weight function for extracting scattered light information is obtained using the parameters of “viewpoint weight function 1” and “viewpoint weight function 2” selected on the setting screen 1402, and processing corresponding to Equation 28 is performed. Run. Reference numeral 3111 denotes arbitrary defocused image data when the parameter selected by the “viewpoint weight function 1” is set. Reference numeral 3101 denotes omnifocal image data generated from the parameter selected by “viewpoint weighting function 2”. Reference numeral 3112 denotes scattered light extraction image data obtained by taking the difference between the arbitrary defocus image data 3111 and the omnifocal image data 3101 and taking only a positive value.

FIG. 22C shows a state in which corrected scattered light extraction image data 3121 whose sign is inverted in the low luminance region is generated by multiplying the scattered light extraction image data 3112 and the transmitted light component suppression mask 3102.
In FIG. 22D, corrected scattered light weighted image data 3131 in which artifacts (focal blur components) in the low luminance region are suppressed is generated by adding the arbitrary defocused image data 3111 and the corrected scattered light extracted image data 3121. It represents the situation.

In the corrected scattered light weighted image data 3131, compared with the corrected scattered light weighted image data 3005 of the second embodiment (see FIG. 21), artifacts in the low luminance region (region where X is negative) can be further suppressed. In the high brightness region (region where X is positive) of the corrected scattered light weighted image data 3131, the defocus is not reduced, but the scattered light component (not shown) is enhanced by addition.
When the method of the present embodiment is applied to HE-stained pathological specimen image data, the influence of artifacts (defocus components) can be suppressed in the low-luminance region where the nucleus is present, and the cytoplasm is present. It is possible to enhance the scattered light in the high luminance region.

(Advantages of this embodiment)
According to the method of the present embodiment, it is possible to generate scattered light weighted image data in which artifacts (focal blur components) in the low luminance region are further reduced as compared with the second embodiment. This allows users to observe surface irregularities at the cytoplasm and cell boundaries and scattered light in the specimen without worrying about the effects of artifacts (defocusing components) in low-luminance areas such as the nucleus of pathological specimens. I can respond.

[Example 4]
In the fourth embodiment, a technique for speeding up the processing of the scattered light extraction image data generation step S1602 when “change observation angle” is selected as the method on the setting screen 1402 of the first embodiment will be described.
The arithmetic expression of Expression 28 calculated in the flowchart shown in step S2203 in FIG. 16D of the first embodiment is an arbitrary expression having a defocus corresponding to the viewpoint weight function k _ex (s, t) from the Z stack image data. This can be regarded as a process for generating defocused image data.
In the method based on Non-Patent Document 2, multiplication on the Fourier transform shown in Equation 14 and inverse Fourier transform shown in Equation 15 are required for each calculation of viewpoint image data. Therefore, when the number of viewpoints is increased in order to increase the calculation accuracy, there is a problem that it takes a calculation time in proportion to the number of viewpoints.
Therefore, it is possible to increase the speed by using the method of Patent Document 1 or Non-Patent Documents 3 and 4 that does not require Fourier transform and Fourier inverse transform for each viewpoint.

Ｆ（ｕ，ｖ，ｗ）、Ｈ（ｕ，ｖ，ｗ）、Ｇ（ｕ，ｖ，ｗ）はそれぞれ３次元被写体ｆ（Ｘ，Ｙ，Ｚ）、撮像光学系の３次元焦点ぼけｈ（Ｘ，Ｙ，Ｚ）、Ｚスタック画像データｇ（Ｘ，Ｙ，Ｚ）の３次元フーリエ変換を表す。 (Calculation by the method of Patent Document 1)
First, an outline of the calculation of the method described in Patent Document 1, which is one of the MFI arbitrary viewpoint / defocused image generation methods, will be described.
When the relationship of Equation 13 is established between the three-dimensional subject f (X, Y, Z), the three-dimensional defocus h (X, Y, Z), and the Z stack image data g (X, Y, Z), the space Since the above convolution is represented by a product over frequency, the following relationship holds. (When the imaging optical system is not bilateral telecentric, coordinate conversion processing is required for the Z stack image data, but the description is omitted because it has already been described.)

F (u, v, w), H (u, v, w), and G (u, v, w) are a three-dimensional subject f (X, Y, Z), respectively, and a three-dimensional defocus h (( X, Y, Z) and three-dimensional Fourier transform of the Z stack image data g (X, Y, Z).

ただし、Ｃ（ｕ，ｖ，ｗ）は３次元周波数フィルタ（単に「３次元フィルタ」とも呼ぶ）であり、以下で表わされる。

Let H _a (u, v, w) be the three-dimensional Fourier transform of the desired three-dimensional defocus h _a (X, Y, Z). A three-dimensional Fourier transform G _a (u, v, w) of the Z stack image data g _a (X, Y, Z) having a defocus expressed by h _a (X, Y, Z) is obtained as follows.

However, C (u, v, w) is a three-dimensional frequency filter (also simply referred to as “three-dimensional filter”), and is expressed below.

即ち、３次元被写体ｆ（Ｘ，Ｙ，Ｚ）の情報を用いずに、Ｚスタック画像データｇ（Ｘ，Ｙ，Ｚ）と撮像光学系の３次元焦点ぼけｈ（Ｘ，Ｙ，Ｚ）と所望の３次元焦点ぼけｈ_ａ（Ｘ，Ｙ，Ｚ）の情報から、所望の焦点ぼけ画像データを生成できる。
ｇ_ａ（Ｘ，Ｙ，Ｚ）を安定的に求めるためには、数５３で表わされる３次元周波数フィルタが安定である必要があり、３次元焦点ぼけｈ_ａ（Ｘ，Ｙ，Ｚ）は撮像光学系内を通る光束の範囲内の光線によって表現される焦点ぼけである必要がある。 From Equation 52, Z stack image data having _a desired three-dimensional defocus h _a (X, Y, Z) can be calculated by the following equation.

That is, without using the information of the three-dimensional subject f (X, Y, Z), the Z stack image data g (X, Y, Z) and the three-dimensional defocus h (X, Y, Z) of the imaging optical system. Desired defocus image data can be generated from information on the desired three-dimensional defocus h _a (X, Y, Z).
In order to stably obtain g _a (X, Y, Z), the three-dimensional frequency filter expressed by Formula 53 needs to be stable, and the three-dimensional defocus h _a (X, Y, Z) is imaged. The defocus needs to be expressed by rays in the range of the light beam passing through the optical system.

  As already described in Expression 17, the three-dimensional defocus h (X, Y, Z) of the imaging optical system is generated from the viewpoint weight function k (s, t) corresponding to the imaging optical system. As shown in FIGS. 10B and 10E, the ideal three-dimensional defocus is enlarged as the distance from the focal position increases. Therefore, when the viewpoint weighting function k (s, t) is represented by a function or image data, the two-dimensional defocusing at each Z position of the three-dimensional defocusing h (X, Y, Z) is a function or image data. It is obtained by scaling according to the distance from the focal position.

Since the three-dimensional defocus h (X, Y, Z) and the arbitrary three-dimensional defocus h _a (X, Y, Z) of the imaging optical system do not depend on the photographed image of the subject, they can be calculated in advance. . In this embodiment, each Fourier of h _a1 (X, Y, Z) and h _a2 (X, Y, Z) corresponding to the viewpoint weight functions that can be selected by “viewpoint weight function 1” and “viewpoint weight function 2”. The transformations H _a1 (u, v, w) and H _a2 (u, v, w) are calculated in advance. These Fourier transforms are stored in advance in the storage device 130 or the main memory 302 in the image data generation device 100. Similarly, the Fourier transform H (u, v, w) of the three-dimensional defocus h (X, Y, Z) of the imaging optical system is similarly calculated in advance and stored in the storage device 130 or the main memory 302.

Ｈ（ｕ，ｖ，ｗ）と数５５で得られるＨ_ａ（ｕ，ｖ，ｗ）を数５３に代入することで、散乱光情報抽出用の視点重み関数ｋ_ｅｘ（ｓ，ｔ）に対応する３次元周波数フィルタＣ（ｕ，ｖ，ｗ）が生成できる。 (Scattered light extraction image data generation step S1602 using the method of Patent Document 1)
FIG. 23 is a flowchart showing internal processing of scattered light extraction image data generation step S1602 when the method described in Patent Document 1 is used. Hereinafter, the processing will be described in order.
First, in the three-dimensional frequency filter generation processing step S3201, a three-dimensional frequency filter C (u, v, w) is generated. A method for generating the three-dimensional frequency filter C (u, v, w) will be described below.
The Fourier transform H (u, v, w) of the three-dimensional defocus of the imaging system is read from the storage device 130 or the like. Further, using the setting information on the setting screen 1402 as an index, the Fourier transform H _a1 (u, v, w) of the three-dimensional defocus corresponding to the viewpoint weight function selected by “viewpoint weight function 1” and “viewpoint weight function 2”. , H _a2 (u, v, w) are read from the storage device 130 or the like.
The Fourier transform of the three-dimensional defocus corresponding to the viewpoint weight function k _ex (s, t) for extracting scattered light information obtained from the “viewpoint weight function 1” and “viewpoint weight function 2” is a property of the linearity of the Fourier transform. Is obtained by the following equation.

By substituting H (u, v, w) and H _a (u, v, w) obtained in Equation 55 into Equation 53, it corresponds to the viewpoint weight function k _ex (s, t) for extracting scattered light information. A three-dimensional frequency filter C (u, v, w) can be generated.

In addition, after _obtaining k _ex (s, t) by the calculation of _Equation 29, h _a (X, Y, Z) is obtained by scaling the function or image data according to the distance from the focal position, and finally, h _a H _a (u, v, w) may be obtained by Fourier transform of (X, Y, Z).
As in the first embodiment, k _ex (s, t) is a viewpoint weight function that satisfies the conditions of Equations 30 and 31.

Next, in the three-dimensional Fourier transform processing step S3202, the Z stack image data g (X, Y, Z) is three-dimensional Fourier transformed to generate a three-dimensional Fourier transform G (u, v, w) of the Z stack image data. .
Next, in the three-dimensional filter application processing step S3203, C (u, v, w) calculated in step S3201 and G (u, v, w) calculated in step S3202 are substituted into Equation 52, and G _a (u , V, w).
Next, in a three-dimensional inverse Fourier transform processing step S3204, a three-dimensional inverse Fourier transform is applied to G _a (u, v, w) to obtain a desired three-dimensional defocus h _a (X, Y, Z). The generated Z stack image data g _a (X, Y, Z) is generated.

Finally, in layer image data acquisition processing step S3205, the layer image data g _a (X, Y, Zf) at the focal position (Z = Zf) is acquired from the calculated g _a (X, Y, Z), and scattered. Output as light extraction image data DS (X, Y, Zf).
Through the above processing, the scattered light extraction image data is generated from the original Z stack image data using the viewpoint weight function k _ex (s, t) for extracting scattered light information without obtaining the viewpoint image data individually. Can do.

数５６のｎはレイヤー画像データの番号であり０から始まりＮ−１で終わる。ｎ_ｆはＺ＝Ｚｆに対応する位置のレイヤー画像データの番号を表す。またＧ^（ｎ）（ｕ，ｖ）はＺスタック画像データのｎ枚目のレイヤー画像データｇ^（ｎ）（Ｘ，Ｙ）のフーリエ変換であり、以下の式で表わされる。

なお、Ｆ｛｝はフーリエ変換を表す。 (Calculation by the method of Non-Patent Documents 3 and 4)
Next, an outline of the calculation of the methods described in Non-Patent Documents 3 and 4, which is one of the MFI arbitrary viewpoint / defocused image generation methods, will be described.
In the methods described in Non-Patent Documents 3 and 4, the Fourier transform A (u, v) of the arbitrary defocus image data a (X, Y, Zf) at the Z position Z = Zf can be generated by the following equation.

N in Expression 56 is the number of the layer image data, which starts from 0 and ends with N-1. n _f represents the number of layer image data at a position corresponding to Z = Zf. G ⁽ⁿ⁾ (u, v) is a Fourier transform of the ^nth layer image data g ⁽ⁿ⁾ (X, Y) of the Z stack image data, and is expressed by the following equation.

F {} represents a Fourier transform.

なお、ｋ_ａ（ｓ，ｔ）は所望の３次元焦点ぼけを生成するための視点重み関数を表わす。また、Ｃ_ｓ，ｔ（ｕ，ｖ）は撮像光学系の３次元焦点ぼけｈ（Ｘ，Ｙ，Ｚ）の視線方向への積分値ｃ_ｓ，ｔ（Ｘ，Ｙ，Ｚｆ）のフーリエ変換であり、Ｃ_ｓ，ｔ（ｕ，ｖ）^−１はその逆数である。またｅ^{−２πｉ（ｓｕ＋ｔｖ）ｎ}は周波数領域で平行移動を行うフィルタである。 Further, H ⁽ⁿ⁾ (u, v) in Expression 56 represents a two-dimensional frequency filter (also simply referred to as “two-dimensional filter”) for each layer image data, and is represented by the following expression.

Note that k _a (s, t) represents a viewpoint weight function for generating a desired three-dimensional defocus. C _{s, t} (u, v) is the Fourier transform of the integral value c _{s, t} (X, Y, Zf) in the line-of-sight direction of the three-dimensional defocus h (X, Y, Z) of the imaging optical system. Yes, C _{s, t} (u, v) ⁻¹ is its reciprocal. Further, e ^{−2πi (su + tv) n} is a filter that performs translation in the frequency domain.

Equation 58 does not depend on the photographed image G ⁽ⁿ⁾ (u, v) of the subject. Therefore, by calculating H ⁽ⁿ⁾ (u, v) in advance and storing it in advance in the storage device 130 or the main memory 302 in the image data generation device 100, the operation of Formula 56 is greatly speeded up. It becomes possible.
In the case of the present embodiment, for the viewpoint weight functions that can be selected by “viewpoint weight function 1” and “viewpoint weight function 2” on the setting screen 1402, H ⁽ⁿ⁾ (u, v ) And stored in the storage device 130 or the main memory 302 in advance.

(Scattered light extraction image data generation step S1602 using the methods of Non-Patent Documents 3 and 4)
FIG. 24 is a flowchart showing the internal processing of scattered light extraction image data generation step S1602 when the methods described in Non-Patent Documents 3 and 4 are used. Hereinafter, the processing will be described in order.
Steps S3301 to S3303 exist in a loop of the number of layers of Z stack image data, and processing is performed for each layer image data. Note that the layer number loop is changed in order from 0 to N-1.

In layer image data Fourier transform processing step S3301, layer image data g ⁽ⁿ⁾ (X, Y) of the layer number n to be processed is acquired from the Z stack image data, and Fourier transform is performed as shown in Equation 57. , G ⁽ⁿ⁾ (u, v).

In the layer filter read processing step S3302, the frequency filter for each layer image data corresponding to each viewpoint weight function from the storage device 130 or the main memory 302 based on the selection information of “viewpoint weight function 1” and “viewpoint weight function 2”. Is read.
Subsequently, a frequency filter for each layer image data corresponding to a viewpoint weight function for extracting scattered light information is generated using a frequency filter for each layer image data corresponding to each of the viewpoint weight function 1 and the viewpoint weight function 2.

散乱光情報抽出用の視点重み関数ｋ_ｅｘ（ｓ，ｔ）は実施例１と同様、数３０および数３１の条件を満たす視点重み関数である。
なお、予め、視点重み関数１および視点重み関数２が選択された場合の散乱光情報抽出用の視点重み関数に相当するレイヤー画像データごとの周波数フィルタを生成し、記憶装置１３０やメインメモリ３０２に格納しても良い。その場合、レイヤーフィルタ読み出し処理ステップＳ３３０２では視点重み関数１および視点重み関数２の情報に基づき、散乱光情報抽出用の視点重み関数ｋ_ｅｘ（ｓ，ｔ）に対応するＨ^（ｎ）（ｕ，ｖ）を読み出す
。 For example, the viewpoint weight function 1 is k _a1 (s, t), the viewpoint weight function 2 is k _a2 (s, t), and the frequency filter for each layer image data is H _a1 ⁽ⁿ⁾ (u, v), Let H _a2 ⁽ⁿ⁾ (u, v). In this case, the frequency filter H ⁽ⁿ⁾ (u, v ⁾ corresponding to the viewpoint weight function k _ex (s, t) (= k _a1 (s, t) −k _a2 (s, t)) for extracting scattered light information. ) Can be calculated by the following formula.

The viewpoint weight function k _ex (s, t) for extracting scattered light information is a viewpoint weight function that satisfies the conditions of Equations 30 and 31 as in the first embodiment.
A frequency filter for each layer image data corresponding to the viewpoint weight function for extracting scattered light information when the viewpoint weight function 1 and the viewpoint weight function 2 are selected is generated in advance and stored in the storage device 130 or the main memory 302. It may be stored. In that case, in layer filter readout processing step S3302, H ⁽ⁿ⁾ (u, u, corresponding to the viewpoint weight function k _ex (s, t) for extracting scattered light information is based on the information of the viewpoint weight function 1 and the viewpoint weight function 2. v) is read.

Next, in layer filter application processing step S3303, multiplication is executed between the frequency filters H ^(n−nf) (u, v) and G ⁽ⁿ⁾ (u, v) for each read layer image data. As a result, the frequency filter application result H ^(n−nf) (u, v) × G ⁽ⁿ⁾ (u, v) is generated for each layer image data.
If there is unprocessed layer image data, the layer number is incremented and the process returns to step S3301. On the other hand, when the processing of steps S3301 to S3303 is completed for all layer image data from 0 to N-1, the process proceeds to step S3304.

In the filter result integration processing step S3304, the frequency filter application results for all the layer image data are integrated. Specifically, processing corresponding to Formula 56 is performed to obtain A (u, v).
Finally, in step S3305, inverse Fourier transform is performed on A (u, v) to generate arbitrary defocus image data a (X, Y, Zf). As described in Expression 28, the arbitrary defocus image data a (X, Y, Zf) generated using the viewpoint weight function k _ex (s, t) for extracting scattered light information is scattered image data (scattered light extracted image). Data) DS (X, Y, Zf). Therefore, in step S3305, the calculated image data is output as scattered light extraction image data DS (X, Y, Zf).
As described above, even when the methods of Non-Patent Documents 3 and 4 are used, the original viewpoint weight function k _ex (s, t) for extracting scattered light information is used without obtaining viewpoint image data for each viewpoint. Scattered light extraction image data can be generated from the Z stack image data.

(Advantages of this embodiment)
According to the method of this embodiment, it is not necessary to generate viewpoint image data corresponding to the number of viewpoints with a large calculation load as compared with the processing flow described in FIG. (Scattered light extraction image data) can be generated. As a result, it is possible to respond to the user's desire to quickly observe the surface irregularities in the cytoplasm and cell boundaries and the scattered light in the specimen.

[Example 5]
In the fifth embodiment, a method of generating scattered light weighted image data in which scattered light is more emphasized when “change observation angle” is selected in the “method” of the scattered image data calculation setting 1402 will be described.
In this embodiment, the scattered light weighted image data can be calculated using the flowchart shown in FIG. 16D, FIG. 23, or FIG. However, the applied frequency filter is different from that of the above-described embodiment.

なお、ｋ_ａ（ｓ，ｔ）の積分値は数１８より１であり、ｋ_ｅｘ（ｓ，ｔ）の積分値は数３０より０であることから、ｋ_ｂ（ｓ，ｔ）の積分値は１となる。

The viewpoint weight function corresponding to the scattered light weighted image data is the sum of the viewpoint weight function k _a (s, t) of desired arbitrary defocused image data and the viewpoint weight function k _ex (s, t) for extracting scattered light information. Is required. Accordingly, in step S3201 of FIG. 23 or step 3302 of FIG. 24, the function k _b (s, t) of the following equation is used instead of the viewpoint weight function k _ex (s, t) for extracting scattered light information. A filter may be generated or read out. k _b (s, t) is referred to as a viewpoint weight function for enhancing scattered light information.

Since the integral value of k _a (s, t) is 1 from Equation 18, and the integral value of k _ex (s, t) is 0 from Equation 30, the integral value of k _b (s, t). Becomes 1.

In addition, when k _b (s, t) satisfies the following relationship, it is possible to generate scattering-weighted image data in which the scattered light information is more emphasized.

In Expression 62, the integration of the viewpoint weight function equal to or smaller than the predetermined threshold value r _th has a negative value means that the effect of extracting scattered light information using k _ex (s, t) is not canceled. Further, that the integral of the viewpoint weight function larger than the threshold r _th is larger than 1 means that viewpoint image data having an observation angle φ larger than the predetermined threshold and having a large contrast of the scattered light component is used.
The arbitrary-focus-blurred image data generated by the viewpoint weight function that satisfies the conditions of Equations 61 and 62 is more emphasized in the scattered light information than the arbitrary-focus-blurred image data generated by the viewpoint weight function that satisfies only the condition of Equation 61. Will be.

  In this embodiment, when the scattered light weighted image data is generated, “scattered light extraction image” is selected as the “type” in the scattered image data calculation setting 1402 and “observation angle change” is selected as the “method”. . In addition, “no suppression processing” is selected in “method” of the transmitted light component suppression setting 1403. When “no suppression processing” is selected, it is assumed that only the processing of step S2702 is executed without executing the processing of step S2701, step S2703, and step S2704 in FIG. 19A of the second embodiment. In FIG. 19B, it is assumed that processing is not executed in the blocks 2801, 2803, and 2804, and only the scattered light extraction image data generation unit 2802 is executed and the result is output.

In the present embodiment, in the scattered light extraction image data generation processing step S2702, a viewpoint weight function k _b (s, t) for enhancing scattered light information is generated using the following equation.

In order to enhance the scattered light information, it is preferable that the viewpoint weight function for enhancing the scattered light information satisfies the conditions of Equations 61 and 62. Therefore, even if there is an auxiliary setting for displaying a combination of “viewpoint weight function 1” and “viewpoint weight function 2” that can generate a viewpoint weight function for enhancing scattered light information that satisfies the conditions of Equations 61 and 62, good. For example, as “viewpoint weight function 1”, “cylindrical blur with radius rm” having the same weight is selected for all viewpoints of radius rm or less, and as “viewpoint weight function 2”, only the viewpoint (0, 0) is weighted. "Pinhole (0, 0
) ”. In this case, since the viewpoint weight function k _b (s, t) for enhancing scattered light information calculated in Expression 63 satisfies the conditions of Expression 61 and Expression 62, “Pinhole (0, 0)” in the drop-down list The color may be changed and displayed.

For the internal processing in step S2702, any of FIGS. 16D, 23, and 24 described in the first embodiment may be used.
When the processing of FIG. 16D is used, in step S2203, the viewpoint weight function for enhancing scattered light information calculated using the formula 63 based on “viewpoint weight function 1” and “viewpoint weight function 2”. Is used to integrate the viewpoint image data obtained in step S2202. This processing corresponds to the method of Non-Patent Document 2.

そして、数５３を用いて、散乱光情報強調用の視点重み関数ｋ_ｂ（ｓ，ｔ）に対応する３次元周波数フィルタＣ（ｕ，ｖ，ｗ）が生成できる。 Next, the case where the flowchart of FIG. 23 corresponding to the system of Patent Document 1 is used will be described.
In step S3201, the three-dimensional defocused Fourier transform H _a (u, v) corresponding to the viewpoint weight function for enhancing scattered light information can be calculated by the following equation. H _a1 (u, v, w) and H _a2 (u, v, w) are respectively read based on the information of “viewpoint weight function 1” and “viewpoint weight function 2” selected on the setting screen 1402. It is a Fourier transform with dimensional defocus.

Then, using Equation 53, a three-dimensional frequency filter C (u, v, w) corresponding to the viewpoint weight function k _b (s, t) for enhancing scattered light information can be generated.

Next, the case where the flowchart of FIG. 24 corresponding to the system of nonpatent literatures 3 and 4 is used is demonstrated.
The frequency filter H ⁽ⁿ⁾ (u, v) for each layer image corresponding to the viewpoint weight function for enhancing scattered light information can be calculated by the following equation. H _a1 ⁽ⁿ⁾ (u, v) and H _a2 ⁽ⁿ⁾ (u, v) are read based on the information of “viewpoint weight function 1” and “viewpoint weight function 2” selected on the setting screen 1402, respectively. This is a frequency filter for each layer image.

As described above, in the case where “observation angle change” is selected in “method” of the scattered image data calculation setting 1402, the conditions (Equation 61 and Equation 62) for further enhancing scattered light and the method of generating the scattered light enhanced image data Said.
In this embodiment, the case where the processes of the transmitted light component suppression mask generation processing step S2701, the transmitted light component suppression processing step S2703, and the scattered light weighted image data generation processing step S2704 are not executed has been described with reference to FIG. However, similar to the second embodiment, it can be executed, and that case also falls within the scope of the present invention.

(Advantages of this embodiment)
By using the method of the present embodiment, it is possible to generate scattered light weighted image data in which the scattered light information is more emphasized. Further, by using the processing flows of FIGS. 23 and 24, it is possible to generate scattered image data (scattered light weighted image data) at high speed. As a result, it is possible to respond to the user's desire to quickly observe the surface irregularities in the cytoplasm and cell boundaries and the scattered light in the specimen.

[Example 6]
In the sixth embodiment, a high-speed generation method of scattered light extraction image data or scattered light weighted image data when “changing declination” or “composite change” is selected in the “method” of the scattered image data calculation setting 1402 will be described.
In the fourth embodiment, a method is described in which the scattered image data is not calculated after calculating the viewpoint image data having different observation angles φ but using the filter. In the present embodiment, a method of calculating collectively using a filter even when the deflection angle θ is different will be described.

FIG. 26 is a schematic diagram showing a state in which the surface irregular slope 813 of the specimen shown in FIG. 8 is observed from the viewpoint where the deviation angle θ is 180 degrees different at the same observation angle φ. A straight line 3501 is a straight line parallel to the optical axis, and the observation angle φ is zero. The deflection angle theta of the straight line 3511 and the line 3512 are respectively theta _i and _{θ i + π ([rad]} ), observation angle phi is ₁ both phi. The deflection angle theta of the straight line 3521 and the line 3522 are respectively theta _i and _{θ i + π ([rad]} ), observation angle phi is ₂ both phi.

As already described in Equation 10, the information on the scattered light at the predetermined position (X, Y, Zf) where the slope exists can be extracted by the difference between the viewpoint image data having the same observation angle and different declination angles. As described in Expression 10, when the difference between the viewpoint image data whose line-of-sight direction is parallel to the straight line 3511 and the viewpoint image data whose line-of-sight direction is parallel to the straight line 3512 is obtained, the inclination angles α and B (φ ₁ ) of the inclined surface 813 are obtained. Information on proportionally scattered light of 2α × B (φ ₁ ) can be extracted. Similarly, when the difference between the viewpoint image data whose line-of-sight direction is parallel to the straight line 3521 and the viewpoint image data whose line-of-sight direction is parallel to the straight line 3522 is obtained, information on scattered light of 2α × B (φ ₂ ) can be extracted.

  In the present embodiment, a case where information of scattered light is extracted using a difference between viewpoint image data corresponding to viewpoints having different deflection angles θ of 180 degrees (π [rad]) from each other will be described. The difference is not limited to 180 degrees. For example, even when the difference in declination is 45 degrees, the apparent change of the inclination angle α ′ is about ½ from Equation 7, and α−α ′ = α / 2 in Equation 10, and the difference in declination A scattered image can be extracted with an intensity of about ¼ when the angle is 180 degrees. Although depending on the size of the observation angle φ, if the difference between the declination angles of the two viewpoints is 45 degrees or more, sufficient intensity can be obtained for extracting scattered light information.

When the imaging optical system is bilateral telecentric, there is a relationship of Equation 5 between the viewing angle φ of the line of sight and the moving radius r of the viewpoint (= √ (s ² + t ² )). (A relationship of Equation 4 is provided between the viewing angle phi _T and radius r Given on the transformed coordinate even if it is not bilateral telecentric.) Therefore, by using the radius r of the viewpoint instead of the line of sight of the viewing angle phi It can also be expressed.
The viewpoint image data in which the radius of the viewpoint is r (the observation angle of the line of sight is φ) and the deflection angle is θ is represented as I _{r, θ} (X, Y, Zf). The absolute value of the difference between the viewpoint image data having the same viewpoint radial radius r (gaze observation angle φ) and declination angles θ and θ + π ([rad]) is added at M different radial radius r. The operation to do is represented by the following equation.

When the apparent inclination angle of each position of the sample observed from the direction of the declination angle θ is expressed by α _θ (X, Y, Zf), the equation 66 can be modified as follows.

Since the apparent tilt angle does not change unless the declination angle θ _i is changed, α _θi (X, Y, Zf) is constant. B (φ) is 0 when φ = 0, and is an increasing function that increases as φ increases. (From Equation 5, B (φ) can be regarded as a function of r, so B (φ) is 0 when r = 0, and can be said to be an increasing function that increases as r increases.)
Therefore, as long as the deflection angle θ _i is not changed, the positive and negative signs in the absolute value of Expression 67 do not change even if the moving radius r _j is changed. Therefore, the absolute value of the right side of Equation 67 and the order of Σ can be interchanged and can be modified as follows.

即ち、同一の動径ｒ_ｊで偏角θが異なる２つの視点画像データの差分の絶対値を、複数の動径について集めた散乱光の情報は、同一の動径ｒ_ｊで偏角θが異なる２つの視点画像データの差分を複数の動径について集めた後で絶対値を取った結果と等しい。 Therefore, the right side of Formula 66 can be modified as follows.

That is, the information on the scattered light obtained by collecting the absolute values of the differences between the two viewpoint image data having the same radius r _j and different angles θ, with respect to the plurality of radiuses, has the angle θ of the same radius r _j. This is equivalent to the result of taking the absolute value after collecting the differences between two different viewpoint image data for a plurality of radius vectors.

（ただし、数７０の視点重み関数は光軸対称であり、動径ｒ_ｊ，偏角θ_ｉ＋πの視点に対応する視点画像データの視点重み関数の値はｋ（ｒ_ｊ，θ_ｉ）に等しい。） It should be noted that even when the right side of the equation 66 is multiplied by a viewpoint weight function k (r _j , θ _i ) having a positive value corresponding to the viewpoint image data of the viewpoint radius r _j and the deflection angle θ _i , the absolute value is obtained. The internal sign does not change. Therefore, the viewpoint weight function k (r _j , θ _i ) can also be included in the absolute value, and the following relationship is also established.

(However, the viewpoint weight function of Formula 70 is symmetric with respect to the optical axis, and the value of the viewpoint weight function of the viewpoint image data corresponding to the viewpoint of the radius r _j and the declination angle θ _i + π is k (r _j , θ _i ). equal.)

Subsequently, in order to modify and speed up the operation having nonlinearity of S _P0 (X, Y, Zf) in Expression 25, Expression 70 is compared with Expression 25. In the following, an expression in the case where Expression 22 is used for calculating S _P0 (X, Y, Zf) of Expression 25 is shown.

なお、視点Ｐ０および視点Ｐ１の座標は極座標表現でＰ０（ｒ_ｊ，θ_ｉ）、Ｐ１（ｒ_ｊ，θ_ｉ＋π）とする。 The integration of s and t is performed only under the condition that the argument P0 has a deflection angle θ _i , the viewpoint P1 has a deflection angle θ _i + π, and the viewpoints P0 and P1 have the same radius r _j (observation angle). If this is the case, as in Equation 70, the absolute value of the right side of Equation 71 and the order of integration can be changed.
That is, the integral in Formula 71 can be expressed by the following equation.

Note that the coordinates of the viewpoint P0 and the viewpoint P1 are P0 (r _j , θ _i ) and P1 (r _j , θ _i + π) in polar coordinate representation.

(Effect of changing the order of absolute value and integration)
Next, the effect of changing the order of absolute values and integration will be described.
When the integration order can be changed as shown in Equation 72, the expression in the absolute value of Equation 72 can be expressed by filtering as in the fourth embodiment. Therefore, it is not necessary to obtain viewpoint image data for a large number of viewpoints, and the calculation can be speeded up. Details will be described later.
Also, when extracting scattered light information using the equation on the left side of Equation 72, if there is noise in each viewpoint image data, the noise is converted to a positive value by taking the absolute value of the difference. Accumulated. On the other hand, when the scattered light information is extracted using the expression on the right side of Formula 72, even if there is noise in the viewpoint image data, it is averaged by adding the differences between the plurality of viewpoint image data, and then takes an absolute value. , Noise is difficult to accumulate. Therefore, extraction of scattered light with less noise and higher image quality can be obtained by using the expression on the right side of Formula 72.

なお、動径ｒ_ｊおよび偏角θ_ｉの存在する範囲はそれぞれ０≦ｒ_ｊ≦ｒ_ｍおよび０≦θ_ｉ＜２πである。図２７（ａ）では動径方向のサンプリング数はＭ＝５、偏角方向のサンプリング数は２Ｎ＝１６（Ｎ＝８）であり、動径ｒ_ｊおよび偏角θ_ｉは以下で表わされている。

(High-speed scattered image data generation method using the viewpoint expressed in polar coordinates)
FIG. 27A is a diagram illustrating an example of a viewpoint position in the present embodiment. The position of each viewpoint is expressed in polar coordinates using elements of M radiuses r _j and 2N declination angles θ _i as shown below.

Incidentally, the range of existence of the radius vector _{r j} and deviation angle theta _i respectively is 0 ≦ _{r j} ≦ _{r m} and 0 ≦ θ _i <2π. In FIG. 27A, the sampling number in the radial direction is M = 5, the sampling number in the declination direction is 2N = 16 (N = 8), and the radial r _j and the declination θ _i are expressed as follows. ing.

なお、図２７（ａ）に示すような動径方向と偏角方向による視点位置の指定は視点設定１４０１の設定（不図示）で行うことができる。 In FIG. 27A, as shown below, there are viewpoints in which the deflection angle is rotated by 180 degrees (π [rad]) with respect to the deflection angle θ _{i at} all viewpoints.

Note that designation of the viewpoint position based on the radial direction and the declination direction as shown in FIG. 27A can be performed by setting the viewpoint setting 1401 (not shown).

(Extraction of scattered light information)
Subsequently, the absolute value of the difference between the two viewpoint image data of the predetermined declination angles θ _i and θ _i + π is added for a plurality of radiuses r _j , and the result is added for various declination angles θ _i. Now, consider extracting scattered light information at a number of viewpoint positions shown in FIG. Expressed by mathematical formulas, the following formulas 76 and 77 are calculated.

When the viewpoint P0 (r _j , θ _i ) and the viewpoint P1 (r _j , θ _i + π) are interchanged, the values of the viewpoint weight function and the results within the absolute value are equal. Can be transformed into a formula. In Equation 79, the number of additions (Σ) can be reduced from 2N to N.

Hereinafter, DS _θi (X, Y, Zf) represented by _Expression 76 or 78 is referred to as declination angle selective scattering image data. In addition, DS (X, Y, Zf) represented by Formula 77 or Formula 79 is called scattered image data or scattered light extracted image data as before.

続いて、数８０の視点の位置（ｒ，θ）を（ｓ，ｔ）で表わし、シグマ（Σ）をインテグラル（∬）で表現すると、以下の数８１〜数８４のように表現できる。

ここでＬ_θｉ（ｓ，ｔ）は以下の式で表わされる。

また、ｌ_θｉ（ｓ，ｔ）は以下の式で表わされる。

即ち、ｌ_θｉ（ｓ，ｔ）は、視点（ｓ，ｔ）の偏角θがθ_ｉであるときは１を取り、それ以外では０となる関数である。ただし、視点（ｓ，ｔ）＝（０，０）の場合は、ｌ_θｉ（，０）＝０、Ｌ_θｉ（０，０）＝０とする。 Here, the right side of Formula 78 can be transformed into the following formula using Formula 70.

Subsequently, when the position (r, θ) of the viewpoint of Expression 80 is expressed by (s, t) and sigma (Σ) is expressed by integral (∬), it can be expressed as Expressions 81 to 84 below.

Here, L _θi (s, t) is expressed by the following equation.

L _θi (s, t) is expressed by the following equation.

That is, l _θi (s, t) is a function that takes 1 when the declination angle θ of the viewpoint (s, t) is θ _i and 0 otherwise. However, when the viewpoint (s, t) = (0, 0), l _θi (, 0) = 0 and L _θi (0, 0) = 0.

Hereinafter, k (s, t) L _θi (s, t) is referred to as an argument selection viewpoint weight function, and L _θi (s, t) is referred to as an argument selection function.
The calculation for obtaining S _θi (X, Y, Zf) in Equation 82 is for calculating arbitrary out-of-focus image data whose viewpoint weight function is an argument selection viewpoint weight function k (s, t) L _θi (s, t) _. Is equal to

(About declination selection viewpoint weight function)
Next, a method for generating the argument selection viewpoint weight function k (s, t) L _θi (s, t) will be described with reference to the drawings.
FIG. 27B is a schematic diagram illustrating an example of the argument selection function L _θi (s, t). FIG. 27C illustrates an argument selection viewpoint weight function k (s, t) L _θi (s, t) _. ) Is a schematic diagram showing an example.
FIG. 27A is a schematic diagram of the viewpoint weighting function k (s, t), and the sampling position is determined by a combination of a plurality of radiuses r _j and declinations θ _i as already described. FIG. 27B is a schematic diagram showing a declination selection function L _θ1 (s, t) of θ _i = θ ₁ . It has a value of 1 on the line of declination θ ₁ (3611), has a value of −1 on the line of declination θ ₁ + π (3612), and is 0 in other regions. Note that the origin (3613) is zero.

FIG. 27C shows a deviation obtained by multiplying the viewpoint weighting function k (s, t) shown in FIG. 27A by the argument selection function L _θ1 (s, t) shown in FIG. It is a schematic diagram which shows angle selection viewpoint weight function k (s, t) L ( _theta ) 1 (s, t). The viewpoint (excluding the origin) where the declination is θ ₁ has a positive value, the viewpoint (excluding the origin) where the declination is θ ₁ + π has a negative value, and is zero otherwise. That is, the numerical formula 82 on the right side to perform the subtraction of the viewpoint image data addition viewpoint image data corresponding to a plurality of viewpoints polarization angle becomes theta _i, the declination corresponding to a plurality of viewpoints to be theta _{i +} [pi Means. In FIG. 27 (c), there are four viewpoints with the deflection angle θ _i and the deflection angle θ _i + π, respectively, so that processing with a filter corresponding to the deflection angle selection viewpoint weight function generates a total of eight viewpoint image data. Differences between corresponding viewpoint image data can be processed together.

(Generation flow of scattered light extraction image data)
FIG. 28A shows scattered light extraction in this embodiment when “scattering light extraction image” is selected as “type” and “change declination” is selected as “method” on the scattered image data calculation setting screen 1402. It is a flowchart which shows the internal process of image data generation step S1602. Hereinafter, the process will be described with reference to FIG. (The scattered light extraction image data generation method shown in FIG. 28A corresponds to the calculation of Equation 25.)

Declination selective scattering image data generation processing step of FIG. 28 (a) S3701 is within the loop of the polarization angle theta _i of the viewpoint, generates a deflection angle selective scattering image data corresponding to the deviation angle theta _i set by the loop To do. For example, in the case of the viewpoint position shown in FIG. 27A, θ _i is changed from θ ₀ to θ _N−1 and N times of loop processing is performed.
In addition, Formula 81 and Formula 82 are used for calculation of the deflection angle selective scattering image data DS _θi (X, Y, Zf). After obtaining S _θi (X, Y, Zf) by generating arbitrary defocused image data corresponding to the argument selection viewpoint weight function k (s, t) L _θi (s, t), S The absolute value of _θi (X, Y, Zf) is obtained. Details will be described later.

When the process of step S3701 is completed for all the preset declination angles θ _i , the process proceeds to declination-selection scattered image data integration process step S3702.
In declination selective scattering image data integration processing step S3702, the number 79 indicates performs arithmetic, step declination selective scattering image data DS _.theta.i calculated in S3701 (X, Y, Zf) from the scattered light extracted image data DS (X, Y, Zf) is generated.

Next, details of the deflection angle selective scattered image data generation processing step S3701 will be described. FIG. 28B is a flowchart showing an internal process of the declination-selection scattered image data generation process step S3701.
In the argument selection viewpoint weight function generation processing step S3801, an argument selection viewpoint weight function k (s, t) L _θi (s, t) is generated. Specifically, first, k _a1 (s, t) (k _a1 (r, θ) in polar coordinate display) is generated based on the selection information of “viewpoint weight function 1” on the scattered image data calculation setting screen 1402. Next, the argument selection function L _θi (s, t) corresponding to the argument θ _i to be processed is read from the storage device 130 or the main memory 302 in the image data generation device 100. Then, by multiplying the argument selection function L _θi (s, t) by k _a1 (s, t) corresponding to the viewpoint weight function 1, the argument selection viewpoint weight function k (s, t) L _θi ( s, t).
In step S3801, a plurality of two-dimensional filters and three-dimensional filters corresponding to the argument selection viewpoint weight function are not generated without generating the argument selection viewpoint weight function k (s, t) L _θi (s, t). An index for reading from the main memory 302 or the like may be generated. For example, the number d1 of the viewpoint weight function and the number d2 of the argument θ _i are determined as indexes.

Next, in an arbitrary defocus image generation processing step S3802, arbitrary defocus image data is generated based on the argument selection viewpoint weight function k (s, t) L _θi (s, t), and S _θi (X, Y). , Zf). When an index for reading the argument selection viewpoint weight function is created in step S3801, a plurality of two-dimensional filters and three-dimensional filters corresponding to the argument selection viewpoint weight function are read from the numbers d1 and d2, and an arbitrary focus Generate blurred image data.
Any method shown in the processing flow of FIG. 23 (Patent Document 1) and the processing flow of FIG. 24 (Non-Patent Documents 3 and 4) may be used as a method for generating arbitrary defocused image data. Details will be described later. Note that the method shown in the processing flow of FIG. 16D (non-patent document 2) may be used for the purpose of reducing noise, not speeding up.

Finally, in step S3803 for selecting the selected angle of declination image data, an operation for obtaining the absolute value shown in Equation 81 is performed on S _θi (X, Y, Zf) output from step S3802, and the declination selection selected image data DS. _θi (X, Y, Zf) is generated.

  Next, details of the arbitrary defocused image generation processing step S3802 will be described.

(When the processing flow in FIG. 23 is used to generate arbitrary defocus image data)
Processing in each step when using the processing flow of FIG. 23 (Patent Document 1) will be described.
In step S3201, the three-dimensional focal point Fourier transform H _a (u, v, w) corresponding to the argument selection viewpoint weighting function k (s, t) L _θi (s, t) and the three-dimensional focal point of the imaging optical system. The blurred Fourier transform H (u, v, w) is read from the main memory 302 or the like. These Fourier transforms are calculated in advance and stored in the main memory 302 or the storage device 130. The three-dimensional defocused Fourier transform H _a (u, v, w) corresponding to the argument selection viewpoint weight function can be read using the index determined in step S3801.
Then, a three-dimensional frequency filter C (u, v, w) is generated by the calculation shown in Formula 53.

In the three-dimensional Fourier transform processing step S3202, the three-dimensional Fourier transform of the Z stack image data is not executed for each deflection angle θ _{i to be} processed, and the Fourier transform result calculated once is stored in the main memory 302 or the storage device 130. After that, read and process.
Since the subsequent processes of the three-dimensional filter application processing step S3203, the three-dimensional inverse Fourier transform process S3204, and the layer image data acquisition process S3205 are the same as those described in the fourth embodiment, the description thereof will be omitted.

(When the processing flow of FIG. 24 is used for generating arbitrary defocused image data)
Next, processing in each step when using the processing flow of FIG. 24 (Non-Patent Documents 3 and 4) will be described.
In step S3301, the Fourier transform of the layer image data is not executed for each deflection angle θ _{i to be} processed, the Fourier transform result calculated once is stored in the main memory 302 or the storage device 130, and the subsequent processing is read and processed.
Subsequently, in step S3302, the frequency filter data H ^(n−nf) (u, v) for each layer image data corresponding to the argument selection viewpoint weighting function k (s, t) L _θi (s, t) is main. Read from the memory 302 or the like. It is assumed that these frequency filter data are calculated in advance and stored in the main memory 302 or the storage device 130.

The frequency filter data H ^(n−nf) (u, v) for each layer image data can be read using the index determined in step S3801. Alternatively, using the argument selection viewpoint weight function k (s, t) L _θi (s, t) calculated in step S3801, the frequency filter data H ^(n−nf) (u, v) may be calculated. In this case, C _{s, t} (u, v) ⁻¹ and e ^{−2πi (su + tv) n} may be calculated in advance and stored in the main memory 302 or the storage device 130. If they are read out and used for calculation, Formula 58 can be calculated at high speed.
Since the subsequent processes of the layer filter application processing step S3303, the filter result integration processing step S3304, and the inverse Fourier transform step S3305 are the same as those described in the fourth embodiment, description thereof will be omitted.

In the foregoing, the method for generating scattered light extraction image data when “change declination” is selected as the “method” of the scattered image data calculation setting screen 1402 has been described.
The calculation of scattered light extraction image data can be speeded up by using the processing flow of FIG. 28A and FIG. 28B inside the scattered light extraction image data generation step S1602. Further, the scattered light extracted image data has an effect of suppressing noise by averaging, and noise of the generated scattered light extracted image data can be reduced.

(Generation method of scattered light weighted image data when "Change declination" is selected)
Next, when obtaining the scattered light weighted image data from the scattered light extraction image data described in the present embodiment, that is, selecting “scattered light weighted image” as “type” in the scattered image data calculation setting screen 1402, ”Will be described for selecting“ change declination ”. Similar to the second embodiment, the description will be made with reference to FIG.
In order to simplify the description, a case will be described in the present embodiment where “no suppression processing” is selected in “method” of the transmitted light component suppression setting screen 1403 as in the fifth embodiment. Therefore, in FIG. 19A of the second embodiment, it is assumed that the processes in steps S2701 and S2704 are performed without executing the processes in steps S2701 and S2703. In FIG. 19B, processing is not executed in the blocks 2801, 2803, and 2804, and only 2802 is executed and the result is output. Although not described in the present embodiment, as in the second embodiment, the case where “multiplication” is selected in “method” on the transmitted light component suppression setting screen 1403 is also included in the scope of the present invention.

First, in the scattered light extraction image data generation processing step S2702, the viewpoint weight function k _a1 (s, t) corresponding to the setting information of “viewpoint weight function 1” is generated, and the scattered light extraction image data DS (X, Y, Zf) is generated. The processing flow shown in FIGS. 28A and 28B is performed in step S2702 of the present embodiment. Since it has already been described, details are omitted.
Next, in the scattered light weighted image data generation processing step S2704, arbitrary defocused image data a (X, Y, Zf) is generated from the Z stack image data, and is synthesized with the scattered light extracted image data DS (X, Y, Zf). Process. The internal processing in step S2704 will be described using FIG.

In the arbitrary defocus image data generation processing step S2901, first, arbitrary defocus image data a (X, Y, Zf) is generated from the Z stack image data. Any method shown in the processing flow of FIG. 23 (Patent Document 1) and the processing flow of FIG. 24 (Non-Patent Documents 3 and 4) may be used as a method for generating arbitrary defocused image data from Z stack image data. .
In this embodiment, “No suppression process” is selected in “Method” on the setting screen 1403. Therefore, in the corrected scattered light enhanced image data generation processing step S2902, the scattered light extracted image data DS (X, Y, Zf) generated in step S2702 is used as the corrected scattered light extracted image data MDS (X, Y, Zf). . As shown in Formula 49, the MDS (X, Y, Zf) on the right side is scaled and added to the arbitrarily defocused image data a (X, Y, Zf) to obtain the scattered light weighted image data.

In the foregoing, the method of generating scattered light weighted image data when “change declination” is selected as the “method” of the scattered image data calculation setting screen 1402 has been described.
By using the processing flow of FIG. 28A and FIG. 28B in step S2702, the calculation of the scattered light extracted image data can be speeded up, and the scattered light weighted image data can be obtained at high speed. In addition, since the scattered light extracted image data obtained in step S2702 has an effect of averaging and suppressing noise, the scattered light weighted image data can also reduce noise.

(Generation method of scattered light extraction image data when "Composite change" is selected)
Next, a method of generating scattered light extraction image data when “composite change” is selected in “method” on the setting screen 1402 will be described. In the above setting, as shown in Equation 32, using the viewpoint weight function for extracting scattered light information, when the scattered light extraction image data when “change observation angle” is selected and “change declination” are selected. The addition of the scattered light extraction image data is obtained.

When obtaining scattered light extraction image data when “change declination” is selected using Equation 44, the viewpoint for extracting scattered light information obtained from the setting information of “viewpoint weight function 1” and “viewpoint weight function 2” It was able to be calculated by using the weight function k _ex (s, t). However, when the scattered light extraction image data is obtained by the calculations shown in Equations 79 and 80 described so far in the present embodiment, since the viewpoint weight function exists inside the absolute value, linearity is not established. Therefore, the scattered light extraction image data cannot be calculated by replacing the viewpoint weight function k (r _j , θ _i ) in Expression 80 with k _ex (r _j , θ _i ). Accordingly, as shown in the equation 34, the scattered light weighted image data of the viewpoint weight function k _a1 (s, t) and the viewpoint weight function k _a2 (s, t) are obtained, and the difference is taken to obtain the scattered light extracted image. Ask for data.

即ち、実施例５で述べた散乱光抽出画像データに、視点重み関数ｋ_ａ１（ｓ，ｔ）を用いて求めた散乱光強調画像データと視点重み関数ｋ_ａ２（ｓ，ｔ）を用いて求めた散乱光強調画像データの差を加算することと等しい。既に述べたように実施例４および本実施例でこれまで述べた散乱光抽出画像データの生成フローは、複数の視点画像データを求めることなく、任意焦点ぼけ画像データの生成として処理するため、高速な計算が可能である。 Expressions for obtaining the scattered light extraction image data when “composite change” is selected in the “method” of the setting screen 1402 in the present embodiment, which is obtained by modifying Equation 34, are shown below.

In other words, the scattered light extraction image data described in the fifth embodiment is obtained using the scattered light weighted image data obtained using the viewpoint weight function k _a1 (s, t) and the viewpoint weight function k _a2 (s, t). It is equivalent to adding the difference of the scattered light weighted image data. As already described, the scattered light extraction image data generation flow described so far in the fourth embodiment and the present embodiment is processed as generation of arbitrary defocused image data without obtaining a plurality of viewpoint image data. Calculation is possible.

FIG. 28C shows a scattered light extraction image data generation step S1602 in this embodiment when “scattering light extraction image” is selected for “type” and “composite change” is selected for “method” on the setting screen 1402. It is a flowchart which shows the internal processing.
In the observation angle change scattered light extraction image data generation step S3901, as described in the fourth embodiment, the scattered light information extraction is performed from the setting information of “viewpoint weight function 1” and “viewpoint weight function 2” set on the setting screen 1402. A viewpoint weight function k _ex (s, t) is generated. Then, arbitrary defocused image data corresponding to the viewpoint weight function for extracting scattered light information is generated. Note that any method of the processing flow in FIG. 23 (Patent Document 1) and the processing flow in FIG. 24 (Non-Patent Documents 3 and 4) may be used as a method for generating arbitrary defocused image data. The scattered light extracted image data calculated in step S3901 is referred to as observation angle changed scattered light extracted image data.

Next, in the first declination-change scattered light extraction image data generation step S3902, based on the setting information of “viewpoint weight function 1”, k _a1 (s, t) or its index number is obtained, and Equations 81 and 79 are obtained. Is used to generate scattered light extraction image data DS (X, Y, Zf). Since the processing flow has been described with reference to FIGS. The scattered light extracted image data calculated in step S3902 is referred to as first deviation angle changed scattered light extracted image data.
Similarly, in the second deflection angle modified scattered light extraction image data generation step S3903, k _a2 (s, t) or its index number is obtained based on the setting information of “viewpoint weighting function 2”. The scattered light extraction image data DS (X, Y, Zf) is generated by using this. The processing content is the same as that in step S3902 except that the viewpoint weight function is different. The scattered light extracted image data calculated in step S3903 is referred to as second deflection angle changed scattered light extracted image data.

Finally, in the composite modified scattered light extraction image data generation step S3904, the calculation shown in Expression 85 is performed. That is, first, the second deflection angle changed scattered light extraction image data is subtracted from the first deflection angle changed scattered light extraction image data to calculate the deflection angle changed scattered light extraction image data. Next, the obtained deviation angle-modified scattered light extraction image data is added to the observation angle-change scattered light extraction image data to generate composite modified scattering light extraction image data.
When the composite modified scattered light extracted image data is displayed as an image, the data less than 0 contains less scattered light information and can be regarded as noise.

In the foregoing, the method for generating scattered light extraction image data when “composite change” is selected as the “method” of the setting screen 1402 has been described. By using the calculation method described in the fourth embodiment and the present embodiment, the scattered light extraction image data can be calculated at high speed. Further, as already described, since noise is suppressed in the deflection angle-extracted scattered light extracted image data, the generated scattered light extracted image data also has low noise.
In the setting screen 1402, when “scattered light extraction image” is selected as the “type” and “change declination” is selected as the “method”, the first declination change scattered light is used as the scattered light extraction image data. A difference between the extracted image data and the second declination-change scattered light extracted image data may be output. The scattered light extracted image data in this case corresponds to the calculation of Equation 44.

(Advantages of this embodiment)
If the method of the present embodiment is used, the difference between viewpoint image data having the same observation angle φ and different viewpoint deviation angles θ is calculated, and even when these are collected by addition, the corresponding viewpoint image data is not calculated. And can be calculated together by filtering. Therefore, the generation of scattered light extraction image data and scattered light weighted image data can be speeded up. As a result, it is possible to respond to the user's desire to quickly observe the surface irregularities in the cytoplasm and cell boundaries and the scattered light in the specimen. In addition, in the scattered light extracted image data and scattered light weighted image data obtained by the method of the present embodiment, the noise component is suppressed because the subtraction and the absolute value are obtained after the addition is performed on the plurality of viewpoint image data. Therefore, even a weak scattered light component can be observed without being buried in a noise component.

The preferred embodiments of the present invention have been described above, but the configuration of the present invention is not limited to these embodiments.
For example, in the above embodiment, the case where Z stack image data taken with a bright field microscope is used as original image data has been described. However, the present invention can also be applied to original image data taken with an epi-illumination microscope, a light field camera, a light field microscope, or the like.

  In the above embodiment, a pathological specimen has been described as an example of the subject, but the subject is not limited thereto. It may be a reflective object such as a metal that is an observation target of an epi-illumination microscope. Further, it may be a transparent biological specimen that is an observation target of a transmission observation type microscope. In any case, by using the technique disclosed in Patent Document 1 or the like, arbitrary viewpoint image data can be generated from a plurality of layer image data groups photographed by changing the focal position of the subject in the depth direction. Applicable. When original image data obtained by photographing a reflective object is used, the original image data includes an image of reflected light (specular reflection) component and an image of scattered light component. However, scattered objects such as paper do not emit scattered light. Become dominant. In that case, the scattered light component can be extracted or enhanced by performing the same processing as in the above embodiment.

  Moreover, you may combine the structure demonstrated in each Example mutually. For example, when the viewpoint scattered image data is generated in step S2103, the viewpoint scattered image data is obtained using the processing target viewpoint, the declination rotation viewpoint, and the observation angle change viewpoint, and addition is performed between the two. The strength and reliability of the viewpoint scattered image data may be increased.

In addition, the various image data generation processes described in the above embodiments may be performed in the real space or in the frequency space. In that case, the calculation may be performed in the real space or in the frequency space. That is, in this specification, the term image data is a concept that includes both real space image data and frequency space image data.
In each of the above embodiments, the calculation of the image data is expressed by a mathematical expression, but it is not always necessary to perform the calculation according to the mathematical expression in actual processing. A specific process or algorithm may be designed in any way as long as image data corresponding to the calculation result expressed by the mathematical formula can be obtained.

100: Image generation apparatus

Claims (17)

A feature that generates feature image data corresponding to image data obtained by extracting or emphasizing differences between a plurality of viewpoint image data having different line-of-sight directions with respect to the subject using original image data obtained by photographing the subject Image data generating means;
Correction means for generating corrected image data by performing a correction process for reducing the luminance of at least the pixels corresponding to the low-luminance region in the original image data among the pixels of the feature image data;
An image data generation device comprising: The original image data is Z stack image data including a plurality of layer image data obtained by photographing a plurality of layers having different positions in the optical axis direction in the subject. Image data generation device. The correction means corrects the feature image data using mask data in which a coefficient for a pixel corresponding to a low luminance region in the original image data is set to a smaller value than a coefficient for a pixel corresponding to a high luminance region. The image data generation apparatus according to claim 2, wherein the image data generation apparatus performs processing. Mask generation means for generating the mask data in which a coefficient for each pixel is set so as to have a positive correlation with the luminance of each pixel of the reference image data, using reference image data representing the luminance characteristics of the original image data; The image data generation apparatus according to claim 3, further comprising: 5. The mask generation unit according to claim 4, wherein the mask generation unit uses, as the mask data, data obtained by normalizing a luminance value of each pixel of the reference image data or the gradation-corrected reference image data with a maximum luminance value. Image data generation device.   The image data generation apparatus according to claim 5, wherein a type of gradation correction to be applied to the reference image data can be changed by a user. The mask data is data in which a coefficient for a pixel corresponding to an area having a luminance lower than a threshold value in the reference image data is set to 0, and a coefficient for other pixels is set to 1. The image data generation device according to any one of the above. The correction unit generates the corrected image data by multiplying each pixel of the feature image data by a coefficient of a corresponding pixel of the mask data. The image data generation device according to item 1. The reference image data is
Image data generated from the Z stack image data and having a defocus different from the layer image data;
Image data generated from the Z stack image data and having a greater depth of field than the layer image data;
Viewpoint image data having a line-of-sight direction parallel to the optical axis direction, generated from the Z-stack image data, or
The image data generation apparatus according to claim 2, wherein the image data generation apparatus is any one of the plurality of layer image data. The corrected image data is synthesized with image data having a defocus different from the layer image data generated from the Z stack image data or any one of the plurality of layer image data. The image data generation device according to claim 2, further comprising enhanced image data generation means for generating enhanced image data. The emphasized image data generation means adds image data having a defocus different from the layer image data, or the corrected image data multiplied by a synthesis coefficient to the layer image data, thereby adding the emphasized image data. The image data generation apparatus according to claim 10, wherein the image data generation apparatus generates the image data. The image data generation apparatus according to claim 11, wherein the synthesis coefficient can be changed by a user. The image data generation apparatus according to claim 1, wherein the original image data is microscope image data obtained by photographing the subject with a microscope. The image data generation apparatus according to claim 1, wherein the feature image data is image data obtained by extracting or enhancing a feature of a scattered light component included in the original image data. . The image according to any one of claims 1 to 13, wherein the feature image data is image data in which the contrast of the unevenness on the surface of the subject is increased compared to the original image data. Data generator. Feature image data corresponding to image data obtained by extracting or emphasizing differences between a plurality of viewpoint image data with different line-of-sight directions with respect to the subject using original image data obtained by photographing the subject Generating step;
A step of generating corrected image data by performing a correction process for lowering the luminance of at least a pixel corresponding to a low luminance region in the original image data among the pixels of the feature image data;
A method of generating image data, comprising:   A program causing a computer to execute each step of the image data generation method according to claim 16.

Images