[go: up one dir, main page]

WO2008157843A1 - Système et procédé pour la détection, la caractérisation, la visualisation et la classification d'objets dans des données d'image - Google Patents

Système et procédé pour la détection, la caractérisation, la visualisation et la classification d'objets dans des données d'image Download PDF

Info

Publication number
WO2008157843A1
WO2008157843A1 PCT/US2008/067949 US2008067949W WO2008157843A1 WO 2008157843 A1 WO2008157843 A1 WO 2008157843A1 US 2008067949 W US2008067949 W US 2008067949W WO 2008157843 A1 WO2008157843 A1 WO 2008157843A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
objects
color
images
interest
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2008/067949
Other languages
English (en)
Inventor
Thomas E. Ramsay
Eugene B. Ramsay
Gerard Felteau
Victor F. Kriporotov
Oleksandr Andrushchenko
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guardian Technologies International Inc
Original Assignee
Guardian Technologies International Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US12/014,043 external-priority patent/US7840048B2/en
Priority claimed from US12/014,028 external-priority patent/US7817833B2/en
Priority claimed from US12/014,034 external-priority patent/US7907762B2/en
Application filed by Guardian Technologies International Inc filed Critical Guardian Technologies International Inc
Publication of WO2008157843A1 publication Critical patent/WO2008157843A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/05—Recognition of patterns representing particular kinds of hidden objects, e.g. weapons, explosives, drugs

Definitions

  • This invention relates to image analysis and, more specifically, to a system and method for detection, characterization, visualization and classification of objects in image data. This includes, but is not limited to, a methodology for accomplishing image segmentation, clarification, visualization, feature extraction, classification, and identification.
  • Computer-based image recognition systems rely solely on the pixel content contained in a two-dimensional image.
  • the image analysis relies entirely on pixel luminance or color, and/ or spatial relationship of pixels to one another.
  • image recognition systems utilize analysis methodologies that often assume that distinctive characteristics of objects exist and can be differentiated.
  • PDF Probability Density Functions
  • grayscale images contain only a single luminance value for each pixel. In these images, neither the type of the PDF nor the statistical features of the original image provide features sufficient for accurate pattern recognition.
  • CAD Computer Aided Detection
  • CAD Computer Aided Detection
  • four classifiers were created to identify suspicious areas. They included the presence of calcifications, spiculation, roughness, and shape. Classification of calcifications uses variances in histogram analysis. Spiculations are classified by looking at lines that were tangential to suspected tumors. Roughness measurements examine variances along vertical lines in the image and, shape is determined by calculating a perimeter to area ratio. Reliance on these classifiers to determine the presence of breast cancer led to very high false positives and negatives. The sensitivity and selectivity of the approach was too low to be used with confidence.
  • An object of the invention is to solve at least the above problems and/or disadvantages and to provide at least the advantages described hereinafter. [0013] Therefore, an object of the present invention is to provide a system capable of detecting objects of interest in image data with a high degree of confidence and accuracy.
  • Another object of the present invention is to provide a system and method that does not directly rely on predetermined knowledge of an objects shape, volume, texture or density to be able to locate and identify a specific object or object type in an image.
  • Another object of the present invention is to provide a system and method of identifying objects of interest in image data that is effective at analyzing images in both two- and three-dimensional representational space using either pixels or voxels.
  • Another object of the present invention is to provide a system and method of distinguishing a class of known objects from objects of similar color and texture whether or not they have been previously explicitly observed by the system.
  • Another object of the present invention is to provide a system and method of identifying objects of interest in image data that works with very difficult to distinguish/classify image object types, such as: (i) apparent random data; ( ⁇ ) unstructured data; and (iii) different object types in original images.
  • Another object of the present invention is to provide a system and method of identifying objects of interest in image data that can cause either convergence or divergence (clusterization) of explicit or implicit image object characteristics that can be useful in creating discriminating features/characteristics.
  • Another object of the present invention is to provide a system and method of identifying objects of interest in image data that can preserve object self-similarity during transformations.
  • Another object of the present invention is to provide a system and method of identifying objects of interest in image data that is stable and repeatable in its behavior.
  • a method of identifying an object of interest in image data comprising receiving the image data and applying at least two predetermined transforms to the image data to effect divergence of the object of interest from other objects, wherein an output of a first of the at least two predetermined transforms is input into a subsequent predetermined transform.
  • a method of identifying an object of interest in image data comprising receiving the image data, replicating the image data to form at least two image layers, wherein each of the at least two image layers comprises the image data, and applying at least two predetermined transforms to each of the image layers to effect divergence of the object of interest from other objects, wherein an output of a first of the at least two predetermined transforms is input into a subsequent predetermined transform.
  • Fig. 1 is a bifurcation diagram
  • FIG. 2 is a diagram illustrating how three complementary paradigms are used to obtain intelligent image informatics, in accordance with one embodiment of the present invention
  • FIG. 3 is a block diagram of a system for identifying an object of interest in image data, in accordance with one embodiment of the present invention.
  • Figs. 4A-5C are transfer functions applied to the pixel color of the image, in accordance with the present invention.
  • Fig. 6A is an input x-ray image of a suitcase, in accordance with the present invention.
  • Fig. 6B is the x-ray image of Fig. 6a after application of the image transformation divergence process of the present invention
  • Fig. 7 is a block diagram of an image transformation divergence system and method, in accordance with one embodiment of the present invention.
  • Figs. 8A-8M are x-ray images of a suitcase at different stages in the image transformation recognition process of the present invention.
  • Fig. 8N is an example of a divergence transformation applied to an x-ray image during the image transformation divergence process of the present invention
  • Fig. 9 is an original input medical image of normal and cancerous cells
  • Fig. 10 is the image of Fig. 9 after application of the image transformation recognition process of the present invention.
  • Fig. 11 is an original input ophthalmology image of a retina
  • Fig. 12 is the image of Fig. 11 after application of the image transformation recognition process of the present invention.
  • FIG. 13 is a flowchart of a method of creating a Support Vector Machine model, in accordance with one embodiment of the present invention.
  • FIG. 14 is a flowchart of a method of performing a Support Vector Machine operation, in accordance with one embodiment of the present invention.
  • Figs. 15A-15C are medical x-ray images
  • Figs 16A and 16B are x-ray images from a Smith Detection (Smith) x-ray scanner and a Rapiscan x-ray scanner, respectively;
  • Fig. 17 is a schematic diagram of an x-ray scanner
  • Fig. 18 is a schematic diagram of an x-ray source used in the x-ray scanner of Fig. 17;
  • Figs. 19A and 19B are X-ray images from a Smith scanner and a Rapiscan scanner, respectively, which illustrate geometric distortions with colors;
  • Fig. 20 is a schematic diagram of an x-ray scanner
  • Fig. 22 is a plot showing a 3D view of (P, C) space with const,
  • Fig. 24A is a plot showing an RGB_DNA 3x2D view for a Smith HiScan 604Oi scanner
  • Fig. 24B is a plot showing an RGB_DNA 3x2D view for a Rapiscan 515 scanner
  • Fig. 25A is a plot showing an RGB_DNA 3D view for a Smith HiScan 604Oi scanner
  • Fig. 25B is a plot showing an RGB_DNA 3D view for a Rapiscan 515 scanner
  • Fig. 26 are plots showing the modeling of 2D (P, C) space on the left and 3D RGB_DNA on the right for a Smith scanner;
  • Fig. 27 are plots showing the sequence of (P 5 C) 2D elastic transformation to RGB_DNA (and back);
  • Fig. 28 is a plot of a 2D (P,C) representation of a Smith RGB_DNA set of unique colors
  • Fig. 30 is a schematic diagram of an x-ray scanner with an object to be scanned that consists of multiple layers of materials;
  • Fig. 31 is a plot showing 2D (P, C) space with vector addition
  • Fig. 32 is a plot showing a color algebra example for a Smith calibration bag consisting of overlapped materials
  • Fig. 33 are examples of images with their 3D RGB_DNA views
  • Fig. 34 are plots showing incorrect RGB_DNA as a result of accidental conversion from 24-bit bmp to 16-bit bmp and back to 24 bit bmp;
  • Fig. 36 is a plot showing the z-lines shown in Fig. 35 from the point in RGB space lying on the prolongation of the major diagonal of RGB cube
  • Fig. 37 are plots showing examples of extracted z-lines and theirs colors in 3x2D RGB_DNA view;
  • Fig. 38 are plots showing extracted z-lines number 1, 7 and 25 and theirs colors in 3D RGB_DNA view;
  • Fig. 39 is a plot showing the fragment of typical 25 bin's z-metrics for 1 st nine z-lines;
  • Fig 40 are organic only, normal and metal only images and their respective 3D RGB_DNA; [0069] Fig. 41 shows an original image and its RGB_DNA with no filters applied;
  • Fig. 42 shows the image of Fig. 41 with a z-filter applied to keep light organic s
  • Fig. 43 shows the image of Fig. 41 with a z-filter applied to keep heavy organics
  • Fig. 44 shows the image of Fig. 41 with a z-filter applied to keep heavy organics and metal;
  • Fig. 45 shows the image of Fig. 41 with a z-filter applied to keep light organics and metal;
  • Fig. 46 are images of containers containing water and liquid explosive before and after signature mapping
  • Fig 47 is a block diagram of one example of a signature mapping transformation sequence
  • Fig. 48 are images of a mammogram having undergone two signature mapping sequences
  • Figs. 49a and 49b are images of a mammogram before and after undergoing a signature mapping process, respectively;
  • Figs. 50a and 50b are images of two mammograms before and after undergoing a signature mapping process
  • Fig. 51 are images illustrating original mammograms taken over four years and the same mammograms after signature mapping processing; [0080] Figs. 52a and 52b ate mammograms before and after signature mapping, respectively, of a breast with microcalcification;
  • Figs. 53a and 53b are a hyperspectral image before and after signature mapping, respectively;
  • Fig. 54 is a block diagram showing steps in one preferred signature mapping method.
  • Fig. 55 are images showing: (a) a mammogram of a breast with cancerous tissue showing the cancerous region after signature mapping; (b) a mammogram of a breast with no cancerous regions (benign); and (c) an RGB representation of the cancerous and benign regions of images (a) and (b).
  • Point operation is a mapping of a plurality of data from one space to another space which, for example, can be a point-to-point mapping from one coordinate system to a different coordinate system.
  • Such data can be represented, for example, by coordinates such as (x, y) and mapped to different coordinates ( ⁇ , ⁇ ) values of pixels in an image.
  • Z effective Is the effective atomic number for a mixture/compound of elements. It is an atomic number of a hypothetical uniform material of a single element with an attenuation coefficient equal to the coefficient of the mixture/compound. Z effective can be a fractional number and depends not only on the content of the mixture/compound, but also on the energy spectrum of the x-rays.
  • Hyperspectral data is data that is obtained from a plurality of sensors at a plurality of wavelengths or energies.
  • a single pixel or hyperspectral datum can have hundreds or more values, one for each energy or wavelength.
  • Hyperspectral data can include one pixel, a plurality of pixels, or a segment of an image of pixels, etc., with said content.
  • hyperspectral data can be treated in a manner analogous to the manner in which data resulting from a divergence transformation is treated throughout this application for systems and methods for threat or object recognition, identification, image normalization and all other processes and systems discussed herein.
  • a divergence transformation can be applied to hyperspectral data in order to extract information from the hyperspectral data that would not otherwise have been apparent.
  • Divergence transformations can be applied to a plurality of pixels at a single wavelength of hyperspectral data or multiple wavelengths of one or more pixels of hyperspectral data in order to observe information that would otherwise not have been apparent.
  • Nodal point A nodal point is a point in an image transformation or series of image transformations where similar pixel values exhibit a significantly distinguishable change in value. Pixels are a unitary value within a 2D or multi-dimensional space (such as a voxel).
  • Object An object can be a person, place or thing.
  • An object of interest is a class or type of object such as explosives, guns, tumors, metals, knives, camouflage, etc.
  • An object of interest can also be a region with a particular type of rocks, vegetation, etc.
  • Threat A threat is a type of object of interest which typically but not necessarily could be dangerous.
  • Image receiver can include a process, a processor, software, firmware and/or hardware that receives image data.
  • Image mapping unit can be a processor, a process, software, firmware and/or hardware that maps image data to predetermined coordinate systems or spaces.
  • a comparing unit can be hardware, firmware, software, a process and/or processor that can compare data to determine whether there is a difference in the data.
  • Color space is a space in which data can be arranged or mapped.
  • One example is a space associated with red, green and blue (RGB). However, it can be associated with any number and types of colors or color representations in any number of dimensions.
  • HSI color space A color space where data is arranged or mapped by Hue, Saturation and Intensity.
  • Predetermined color space is a space that is designed to represent data in a manner that is useful and that could, for example, cause information that may not have otherwise been apparent to present itself or become obtainable or more apparent.
  • RGB DNA refers to a representation in a predetermined color space of most or all possible values of colors which can be produced from a given image source.
  • the values of colors again are not limited to visual colors but are representations of values, energies, etc., that can be produced by the image system.
  • Signature is a representation of an object of interest or a feature of interest in a predetermined space and a predetermined color space. This applies to both hyperspectral data and/ or image data.
  • a template is part or all of an RGB DNA and corresponds to an image source or that corresponds to a feature or object of interest for part or all of a mapping to a predetermined color space.
  • Algorithms From time to time, transforms and/or divergence transformations are referred to herein as algorithms.
  • Modality any of the various types of equipment or probes used to acquire images. Radiography, CT, ultrasound and magnetic resonance imaging are examples for modalities in this context.
  • the analysis capabilities of the present invention can apply to a multiplicity of input devices created from different electromagnetic and sound emanating sources such as ultraviolet, visual light, infra-red, gamma particles, alpha particles, etc.
  • Image Transformation Divergence System and Method - General Overview
  • the present invention identifies objects of interest in image data utilizing image conditioning and data analysis in a process herein termed “Image Transformation” (ITR) or, equivalently, “Image Transformation Divergence” (ITD).
  • ITR Image Transformation
  • ITD Image Transformation Divergence
  • the ITD process can cause different yet almost identical objects in a single image to diverge in their measurable properties.
  • An aspect of the present invention is the discovery that objects in images, when subjected to special transformations, will exhibit radically different responses based on the pixel values of the imaged objects. Using the system and methods of the present invention, certain objects that appear almost indistinguishable from other objects to the eye or computer recognition systems, or are otherwise identical, generate radically different and significant differences that can be measured.
  • Another aspect of the present invention is the discovery that objects in images can be driven to a point of non-linearity by certain transformation functions.
  • the transformation functions can be applied singly or in a sequence, so that the behavior of the system progresses from one state through a series of changes to a point of rapid departure from stability called the "point of divergence.”
  • Figure 1 is an example of a bifurcation diagram illustrating iterative uses of divergence transforms, where each node represents an iteration or application of another divergence transform.
  • a single image is represented as a simple point on the left of the diagram.
  • There are several branches in the diagram at lines A, B and C) as the line progresses from the original image representation on the left, indicating node points where bifurcation occurs ("points of bifurcation").
  • three divergence transforms were used in series at points A, B and C.
  • each divergence transform results in a bifurcation of the image objects or data.
  • MLAs Machine Learning Algorithms
  • Another aspect of the present invention is that one can apply the "principle of divergence" to the apparent stability of fixed points or pixels in an image and, by altering one or more parameter values, give rise to a set of new, distinct and clearly divergent image objects. Because each original object captured in an image responds uniquely at its point of divergence, the methods of the present invention can be used in an image recognition system to distinguish and measure objects. It is particularly useful in separating and identifying objects that have almost identical color, density and volume.
  • special transformations are applied to images in an iterative "filter chain" sequence.
  • the nature of the sequence of transforms causes objects in the image to exhibit radically different responses based on their pixel value(s) such as color (that are related to the physical properties inherent in the original objects in the image).
  • pixel value(s) such as color (that are related to the physical properties inherent in the original objects in the image).
  • certain objects that appear almost indistinguishable to the eye or computer recognition systems from other objects generate radically different and significant differences that can be easily measured.
  • the ITD process works with an apparently stable set of fixed points or pixels in an image and, by altering one or more parameter values, giving rise to a set of new, distinct, and clearly divergent image objects. Commonly used and understood transforms work within the domain where images maintain equilibrium.
  • the ITD method starts by first segmenting the image into objects of interest, then applying different filter sequences to the same original pixels in the identified objects of interest using the process.
  • the process is not limited to a linear sequence of filter processing.
  • an explosive inside of a metal container can be located by first locating all containers, remapping the original pixel data with known coordinates in the image and then examining the remapped original pixels in the identified object(s) in the image for threats with additional filter sequences.
  • transforms can be tuned to optimize the distinction of the object of interest of the images.
  • the process works for both image segmentation and feature generation through an iterative process of applying image transforms. As discussed above, it is defined mathematically as a reaching a Repellor Point.
  • An aspect of the present invention is the use of three complementary paradigms to extract information out of images that would otherwise not be readily available. This process is herein referred to as "Intelligent Image Informatics". As illustrated in Figure 2, the three complementary paradigms include: (1) Image Processing; (2) Pattern Classification (Contextual Imagery with Machine Learning); and (3) ⁇ - Physics.
  • Imaging can take place in the spatial domain, spectral domain, RGB_DNA space and/or feature space.
  • the Feature Extraction Process can use the image's describers/qualifiers/characteristics from the above mentioned domains. These feature can be analyzed by many pattern classification techniques, also called Machine Learning Algorithms such as Support Vector Machines (SVM), decision trees/graphs, ⁇ - Physics refers to the physics that governs the image source, such as dual energy scanning systems, the z-effective exhibited by different materials and the RGB_DNA that characterizes the image source. All of these methodologies and concepts will be explained in more detail below.
  • SVM Support Vector Machines
  • the ITD methodologies of the present invention reveal signatures in radiographic image objects that have been previously invisible to the human eye.
  • the application of specific non-linear functions to a grey-scale or color radiographic images is the basis of ITD. Due to the Compton and photoelectric effects, objects in the image exhibit unique, invariant responses to the ITD algorithms based on their physical interactions with the electromagnetic beam.
  • By applying a combination of complementary functions in an iterative fashion objects of very similar grey-scale or color content in the original image significantly diverge at a point of non-linearity. This divergence causes almost statistically equivalent objects in the original image to display significant density, color and pattern differences.
  • Different algorithms are used for distinguishing objects that exhibit different ranges of effective atomic numbers (Z e ff). The algorithms are tuned to be optimal within certain fractional ranges of resultant electromagnetic Compton/photoelectric combinations.
  • the hypercube now contains spectral bands for each object that are the result of the object's response to each ITD iteration. This is quite similar to the creation of hyperspectral data that is collected by sensors from the reflectance of objects.
  • the hypercube data contains both spatial and spectral components that can be used for effective pattern classification rule generation.
  • FIG. 3 is a block diagram of a system 100 for identifying an object of interest in image data, in accordance with one embodiment of the present invention.
  • the system 100 comprises an input channel 110 for inputting image data 120 from an image source (not shown) and an image analysis system 130.
  • the image analysis system 130 generates transformed image data utilizing ITD, in which the object of interest is distinguishable from other objects in the image data.
  • the object of interest can be any type of object.
  • the object of interest can be a medical object of interest, in which case the image data can be computer tomography (CT) image data, x-ray image data, or any other type of medical image data.
  • CT computer tomography
  • the object of interest can be a threat object, such as weapons, explosives, biological agents, etc., that may be hidden in luggage.
  • the image data is typically x-ray image data from luggage screening machines.
  • At least one divergence transformation preferably a point operation, is preferably utilized in the image analysis system 130.
  • a point operation converts a single input image into a single output image. Each output pixel's value depends only on the value(s) of its corresponding pixel in the input image. Input pixel coordinates correlate to output pixel coordinates such that Xi, Yi — » X 0 , Y 0 - A point operation does not change the spatial relationships within an image. This is quite different from local operations where the value of neighboring pixels determines the value of the output pixel.
  • Point operations can correlate both gray levels and individual color channels in images.
  • One example of a point operation is shown in the transfer function of Figure 4A.
  • Fig. 4A 8 bit (256 shades of gray) input levels are shown on the horizontal axis and output levels are shown on the vertical axis. If one were to apply the point operation of Fig. 4A to an input image, there would be a 1 to 1 correlation between the input and the output (transformed) image. Thus, input and output images would be the same.
  • Point operations are predictable in how they modify the histogram of an image. Point operations are typically used to optimize images by adjusting the contrast or brightness of an image. This process is known as contrast enhancing. They are typically used as a copying technique, except that the pixel values are modified according to the specified transfer function. Point operations are also typically used for photometric calibration, contrast enhancement, monitor display calibration, thresholding and clipping to limit the number of levels of gray in an image.
  • the point operation is specified by the transformation function f and can be defined as: where A is an input image and B is an output image.
  • the at least one divergence transformation used in the image analysis system 130 can be either linear or non-linear point operations, or both.
  • Non-linear point operations are used for changing the brightness/contrast of a particular part of an image relative to the rest of the image. This can allow the midpoints of an image to be brightened or darkened while maintaining blacks and white in the picture.
  • Figure 4B is a linear transfer function
  • Figures 4C-4E illustrate transformations of some non-linear point operations.
  • An aspect of the present invention is the discovery that the transfer function can be used to bring an images to a point where two initially close colors become radically different after the application of the transfer function. This typically requires a radical change in the output slope of the resultant transfer function of Figure 5A.
  • the present invention preferably utilizes radical luminance (grayscale), color channel or a combination of luminance and color channel transfer functions to achieve image object differentiation for purposes of image analysis and pattern recognition of objects.
  • the placement of the nodal points in the transfer function(s) is one key parameter. An example of nodal point placements are shown in the transfer function example illustrated in the Figure 5B.
  • the nodal points in the transfer function used in the present invention are preferably placed so as to frequently create radical differences in color or luminance between image objects that otherwise are almost identical.
  • Figure 6A shows an input image
  • Figure 6B shows the changes made to the input image (the transformed image obtained) as a result of applying the transfer function of Fig. 5C.
  • the input image is an x-ray image of a suitcase taken by a luggage scanner.
  • the objects of interest are shoes 300 and a bar of explosives 310 on the left side of the suitcase.
  • the orange background in the image makes a radical departure from the orange objects of interest (300 and 310) and other objects that are almost identical to the objects of interest.
  • the use of different nodal points in the transfer function will cause the objects of interest to exhibit a different color from other objects.
  • Data points connecting the nodes can be calculated using several established methods.
  • a common method of mathematically calculating the data points between nodes is through the use of cubic splines.
  • Additional imaging processes are preferably applied in the process of object recognition to accomplish specific tasks. Convolutions such as median and dilate algorithms cause neighboring pixels to behave in similar ways under the transfer function, and may be applied to assure the objects' integrity during the transformation process.
  • Figure 7 is a block diagram of one preferred embodiment of the image analysis system 130 of Fig. 3, along with a flowchart of a method for identifying an object of interest in image data using the image analysis system 130.
  • the image analysis system 130 includes an image conditioner 2000 and a data anaryzer 3000.
  • Figures 8A-8M are x-ray images of a suitcase at different stages in the image analysis process. These images are just one example of the types of images that can be analyzed with the present invention. Other types of images, e.g., medical images from X-ray machines or CT scanners, or quantized photographic images can also be analyzed with the system and methods of the present invention.
  • the method starts at step 400, where image may optionally be normalized.
  • the normalization process preferably comprises the following processes: (1) referencing; (2) benchmarking; (3) conformity process; and (4) correction process.
  • the referencing process is used to get a reference image containing an object of interest for a given type of X-ray machine.
  • This process consists of passing a container containing one or more objects of interest into a reference X-ray machine to get a reference image.
  • the referencing process is preferably performed once for each X-ray machine model/ type/manufacturer.
  • the benchmarking process is used to get a transfer function used to adjust the colors of the reference image taken by a given X-ray machine that is not the reference X-ray machine. This process consists of passing a reference container into any given X-ray machine to get the image of this reference container, which is herein referred to as the "current image.” Then, the current image obtained for this X-ray machine is compared with the reference image. The difference between the current image and the reference container is made to create a transfer function. [00149] As a transformation of the image's colors of a container, the benchmarking process determines the transfer function that maps all the colors of the current image color scheme ("current color scheme",) to the corresponding colors that are present in the reference color scheme of the reference image. The transfer function applied to the current image transforms it into the reference image.
  • the adjustment of the colors of X-ray machines of a different type/model/manufacturer requires a distinct and specific calibration process. All X-ray machines are preferably also put through a normalization process. X-ray machines of a same type/model/manufacturer are preferably normalized using the same calibration process. AU X-ray machines of different types are preferably calibrated and all the machines, no matter their type, are preferably normalized.
  • the conformity process is preferably used to correct the image color representation of any objects that pass through a given X-ray machine.
  • the conformity process corrects the machine's image color representation (color scheme) in such a way that the color scheme of a reference image will fit the reference color scheme of the reference container.
  • the conformity process preferably consists of applying the transfer function to each bag that passes into an X-ray machine to "normalize" the color output of the machine. This process is specific to every X-ray machine because of the machine's specific transfer function. Each time a container passes through the X-ray machine, the conformity process is preferably applied.
  • the correction process is preferably used to correct the images from the X-ray machine. It preferably minimizes image distortions and artifacts. X-ray machine manufacturers use detector topologies and algorithms that could have negative effects on the image geometry and colors. Geometric distortions, artifacts and color changes made by the manufacturer have negative impacts on images that are supposed to rigorously represent the physical aspects and nature of the objects that are passed through the machine.
  • the correction process is preferably the same for all X-ray machines of a given model/ type/manufacturer.
  • image processing is performed on the image.
  • image processing techniques including, but not limited to, ITD, spatial and spectral transformations, convolutions, histogram equalization and gamma adjustments, color replacement, band-pass filtering, image sharpening and blurring, region growing, hyperspectral image processing, color space conversion, etc.
  • ITD is used for the image processing step 410, and as such the image is segmented by applying a color determining transform that effect specifically those objects that match a certain color/density/effective atomic number characteristics. Objects of interest are isolated and identified by their responses to the sequence of filters. Image segmentation is preferably performed using a series of sub-steps.
  • Figs. 8B-8H show the image after each segmentation sub-step.
  • the resulting areas of green in Fig. 8G are analyzed to see if they meet a minimum size requirement. This removes the small green pixels.
  • the remaining objects of interest are then re-mapped to a new white background, resulting in the image of Fig. 8H. Most of the background, organic substances, and metal objects are eliminated in this step, leaving the water bottle 500, fruit 510, peanut butter 520 and object of interest 530.
  • step 420 features are extracted by the data analyzer 3000 subjecting the original pixels of the areas of interest identified in step 410 to at least one feature extraction process. It is at this step that at least one divergence transformation is applied to the original pixels of the areas of interest identified in step 410.
  • step 430 data conditioning is performed by the data analyzer 3000, in which the data is mathematically transformed to enhance its efficiency for the MLA to be applied at step 440.
  • meta data is created (new metrics from the metrics created in the feature extraction step 420 such as the generation of hypercubes.
  • This metadata can consist of any feature that is derived from the initial features generated from the spatial domain. Meta data are frequently features of the spectral domain, Fourier space, RGB_DNA, and z-effective among others.
  • Machine Learning Algorithms are capable of automatic pattern classification. Pattern classification techniques automatically determine extremely complex and reliable relationships between the image characteristics also called features. These characteristics are use by the Rules-base that exploits the relationships to automatically detect object into the images.
  • MLAs machine learning algorithms
  • the feature extraction process of step 420 is applied in order to represent the images with numbers.
  • the MLAs applied at step 440 are responsible for generating the detection system that determines if an object of interest is present.
  • MLAs need structured data types, such as numbers and qualitative /categorical data as inputs.
  • the Feature Extraction Process is applied to transform the image or segments of an image into numbers. Each number is a metric that represents a characteristic of the image.
  • Each image is associated with a collection of the metrics that represents it.
  • the collection of the metrics related to an image is herein referred to as a vector.
  • MLAs analyze the vector of the metrics for all the images and find the metrics' relationships that make up a "rules-base.”
  • the metrics created by the feature extraction process 420 are used to reflect the image content are, but not limited to, mean, median, standard deviation, rotation cosine measures, kurtosis, Skewness of colors and, spectral histogram, co-occurance measures, gabor wavelet measures, unique color histograms, percent response, and arithmetic entropy measures.
  • the objects are classified by the data analyzer 3000 based upon the rules-base that classify images into objects of interest and objects not of interest according to the values of their metrics, which were extracted at step 420.
  • the object of interest 530 is measured in this process for its orange content.
  • the peanut butter jar 520 shows green as its primary value, and is therefore rejected.
  • the detected objects of interest 530 are thus distinguished from all other objects (non-detected objects 470). Steps 410-450 may be repeated as many times as desired on the non-detected objects 470 in an iterative fashion in order to improve the detection performance.
  • Determination of distinguishing features between objects of interest and other possible objects is done by the rule-base as a result of the analysis of the vectors of the metrics by the MLAs applied at step 440.
  • MLAs There are hundreds of different MLAs that can be used including, but not limited to, decision trees, neural networks, support vector machines
  • the rules-base is therefore preferably entered into code and preferably
  • TAL A sample of TAL is shown below.
  • Additional metrics can be created by applying spectrally-based processes, such as Fourier, to the previously modified objects of interest or by analyzing eigenvalue produced from a Principal Components Analysis to reduce the dimension space of the vectors and remove outliers and non-representative data (metrics/images).
  • a color replacement technique is used to further emphasize tendencies of color changes. For example, objects that contain a value on the red channel > 100, can be remapped to a level of 255 red so all bright red colors are made pure red. This is used to help identify metal objects that have varying densities..
  • the system and methods of the present invention are based on a methodology that is not restricted to a specific image type or imaging modality. It is capable of identifying and distinguishing a broad range of object types across a broad range of imaging applications. It works equally as well in applications such as CT scans, MRI, PET scans, mammography, cancer cell detection, geographic information systems, and remote sensing. It can identify and distinguish metal objects as well.
  • the present invention is capable of, for example, distinguishing cancer cell growth in blood samples and is being tested with both mammograms and x-rays of lungs.
  • Fig. 9 shows an original input image with normal and cancerous cells.
  • Fig. 10 shows the image after the ITD process of the present invention has been applied, with only cancer cells showing up in green.
  • FIG. 11 shows an original ophthalmology image of the retina.
  • Fig. 12 shows the image after the ITD process of the present invention have been applied, with the area of interest defined in red.
  • the analytical processing provided by the present invention can be extended to integrate data from a patient's familial history, blood tests, x-rays, CT, PET (Positron Emission Tomography), and MRI scans into a single integrated analysis for radiologists, oncologists and the patient's personal physician. It can also assist drug companies in reducing costs by minimizing testing time for new drug certification.
  • MLA Machine Learning Algorithms
  • Contextual imagery not only focuses on the segmented imaged, but on the entire image as well. Context often carries relevant and discriminative information that could determine if an object of interest is present or not in the scene.
  • MLAs analyze the vectors of metrics taken from the images.
  • the choice of metrics is important. Therefore, the feature extraction process preferably includes "data conditioning" to statistically improve the dataset analyzed by the MLA.
  • Image conditioning is preferably carried out as part of the data conditioning. Image conditioning is one of the first steps performed by the image processing function. It initially consists of the removal of obvious or almost obvious objects that are not one of the objects of interest from the image. By applying image processing functions to the image, some important observations can also be made. For example, some unobvious portions of the object of interest may be distinguished from other elements that are not part of the object of interest upon the application of certain types of image processing. These aspects of image conditioning leverage the MLA's detection capability
  • Image normalization is preferably the first process applied to the image. This consists of the removal of certain image characteristics, such as the artificial image enhancement (artifacts) that is sometimes applied the system that created the image. Image normalization could also include removing image distortions created by the acquisition system, as well as removal of intentional and unintentional artifacts created by the software that constructed the image.
  • image normalization could also include removing image distortions created by the acquisition system, as well as removal of intentional and unintentional artifacts created by the software that constructed the image.
  • SVMs Support Vector Machines
  • a nonlinear separating surface between the classes can be drawn with the SVM technique.
  • the separating surface is drawn by the SVM technique in an optimal way, maximizing the margin between the classes. In general, this provides a high probability that, with proper implementation, no other separating surface will provide better generalization performance within this framework.
  • the SVM technique is robust to small perturbations and noise in data.
  • FIG. 13 is a flowchart of a method of creating an SVM model, in accordance with one embodiment of the present invention.
  • the method starts at step 600, where a nonlinear transformation type and its parameters are chosen.
  • the transformation is performed by the use of specific "kernels", which are mathematical functions. Sigmoid, Gaussian or Polynomial kernels are preferably used.
  • step 610 a quadratic programming optimization problem for the soft margin is solved efficiently. This requires a proper choice of the optimization procedure parameters as well.
  • FIG. 14 is a flowchart of a method of performing an SVM operation, in accordance with one embodiment of the present invention.
  • a feature generation technique is applied at step 700 to yield a vector of the generated features that is used for the analysis.
  • a specified kernel transformation is applied to each of all possible couples of the analyzed vector and a Support Vector.
  • the received values are weighted according to the respective weight coefficients and added all together with the free term.
  • the result of the kernel transformation is used to classify the image.
  • the image is classified as falling in a first class (e.g., a threat) if the final result is larger than or equal to zero, and is otherwise classified as belonging to a second class (e.g., non-threat).
  • a first class e.g., a threat
  • a second class e.g., non-threat
  • RGB-DNA is one of the image processing techniques that can be used in normalization step 400 and the image processing step 410 (Fig. 7).
  • RGB-DNA refers to a representation, in a predetermined color space, of most or all possible values of colors which can be produced from a given image source.
  • values of colors is not limited to visual colors, but refers to representations of values, energies, etc. that can be produced by the imaging system. The use of RGB DNA for image analysis will be described in detail in this section.
  • Figures 16A and 16B are x-ray images from a Smith Detection (Smith)
  • Figure 17 is a schematic diagram of a typical x-ray scanner. The scanner
  • the X-ray source 840 is typically implemented with an
  • X-ray tubes that has a rotating anode 900, which is used for generating an uninterrupted flow of X-ray photons 910.
  • the spectrum 920 of the x-ray radiation is polychromatic, with a couple of peaks of characteristic lines.
  • the spectrum covers a range from approximately 160 KeV to approximately 25 KeV.
  • the X-ray photons 910 of the beam 860 penetrate the materials in the item being scanned 830, thereby experiencing attenuation of different natures (scattering, absorption etc.). Then, the x-ray beam 860 goes into the L-shaped detector array 810 to be measured.
  • the array 810 is typically a set of pre-assembled groups of detectors (16, 32 or 64 detectors) positioned perpendicular to the x-ray beams 860.
  • Each individual detector is responsible for one row of pixels on the x-ray image.
  • two detectors per pixel row are used, i.e., the high-energy detector is placed on the top of the low energy one. They are typically separated by a copper filter (typically ⁇ 0.5mm thick) installed for energy discrimination. This filter is a crucial element of this technique. This paves a path for calculating the Z e jf (effective atomic number) and d (integral density of the material) of the scanned object 830.
  • the moving belt 820 in the scanner 800 works as a slicing mechanism.
  • One slice is one column of pixels.
  • the speed of the belt should be synchronized with timing of the system to avoid distortion in lengthwise dimensions of the images.
  • An L-shaped detector array 810 causes clearly visible geometric distortions in shapes. These distortions are the results of the projection-detection scheme of a particular scanner design, which can be understood by simple geometrical constructions.
  • Figures 19A and 19B are X-ray images from a Smith scanner and a Rapiscan scanner, respectively, which illustrate geometric distortions with colors. The distortions are particularly apparent in the shapes of the frames 1000 and wheels 1010.
  • the RGB 3D color schemes of different vendors can be mapped into a single universal 2D (Z e ⁇ d) space of physical parameters of Z e ff and d.
  • the possibility of such mapping can be shown by looking at a mathematical description of the dual energy technique, and by looking at the depth of proprietary color schemes of two well known scanner vendors — Smith Detection and Rapiscan.
  • photons vP> and fe y?) measured by low energy and high energy detectors, respectively, for the geometry shown in Figure 20.
  • Equations (1) and (2) are as follows:
  • Any color image we see on the computer screen of a dual energy scanner is a 2D array of pixels with colors represented by (R,G,B) triplets.
  • the number of unique colors needed to maintain an acceptable visual quality of a dual energy color image can be quite large and approaches at least the number of colors of a medium class digital camera ⁇ 1500000. Nevertheless, it was discovered that the number of unique colors in an average baggage color image is approximately 7,000 colors for a Smith HiScan 604Oi scanner and less than 100,000 for a Rapiscan 515 scanner.
  • An aspect of the present invention is the development of tools to visualize the set of unique colors, both as 3x2D projections to RG, GB and BR planes of the RGB cube, as shown in Figures 24A and 24B, and also as a 3D rotating view based on an OpenGL open source prototype, as shown in Figures 25A and 25B.
  • Figs. 24A and 24B are RGB_DNA 3x2D views for a Smith HiScan 604Oi scanner and a Rapiscan 515 scanner, respectively.
  • Figs. 25A and 25B are 3D rotating views for a Smith HiScan 604Oi scanner and a Rapiscan 515 scanner, respectively.
  • the phrase "RGB_DN was assigned to the discovered color schemes, where term "DNA” was used because of the fact that all images, at least from the scanners of a particular model, will inherit this unique set of RGB colors.
  • FIG. 26 include plots of 2D (P,C) space (left plot) and 3D RGB_DNA (right plot) for a Smith scanner. It is clear that the point of origin (0,0) of (P 5 C) reflects the RGB point of (255,255,255) on a 3D view of RGB_DNA. These points are responsible for the case of zero attenuation.
  • the next logical step consists of finding the relations between Black Pole (0,0,0) of the RGB_DNA and the Black Zone boundary of the (P,C) space. This point and the boundary are responsible for the scenario of the maximum possible measured attenuation. Beyond this point, the penetration is so weak that detectors "can not see it at all”. In 3D RGB_DNA we have a single point-wise Black Pole, and in 2D (P,C) we have a stretched boundary.
  • the Black Zone boundary in (P,C) can be compressed/tightened to a single Black Pole or, what is more practical and convenient, the Black Pole of 3D RGB_DNA can be expanded and transformed to the curve, and together with unbent (piecewise-linear in our case) color curves of RGB_DNA, this 3D surface can be transformed to the 2D area similar to the 2D (P,C) space.
  • the next step that needs to be performed is noting that the color curves on the Smiths 3D RGB_DNA surface and the straight lines of the Z e ff— const on (P, C) plane are actually the same entities. They are the two-dimensional manifolds which are topologically equivalent, and can be mutually mapped by a one-to-one relationship.
  • This mapping for Smiths scanner is shown in Figure 28.
  • the Rapiscan scanner color scheme can be mapped to the (P, C) space in the same manner as continuous elastic deformation,.
  • Equations (11) and (12) above express the effective atomic number Z and density d as functions of P and C for a single uniform layer of a material.
  • the formulas for Z and d can be derived from
  • Figure 33 shows examples of images with their 3D RGB_DNA views. Only the image on the far left is in correct original RGB_DNA colors. The other two images are visually undistinguished from first one. Nevertheless, they are in fact bmp images converted back from gif and jpeg conversions of an original bmp image.
  • FIG. 34 shows accidental conversion from 24- bit bmp to 16-bit and back.
  • Figs. 34 and 32 can be compared to see the difference between the incorrect RGB_DNA and the correct one.
  • RGB_DNA colors in color images of dual energy x- ray scanners is fixed for each model and are much less than 16,777,216 RGB triplets in 24-bit bmp, it is possible to make automated inspection of incoming images without the actual visual review of their RGB_DNA (3x2D or 3D). This process can detect the presence of images not created with that x-ray scanner or the scanner is out of calibration.
  • RGB_DNA itself as a limited subset of the entire 24-bit RGB set makes it possible.
  • the component designed and implemented for this purpose performs a fast search through already collected RGB_DNA sets for each pixel of an incoming image, and assures that the system will not be confused.
  • the color scheme of the Smith scanner is comprised of 29 color curves that are stretched from white RGB pole (255,255,255) to black pole (0,0,0). There is one more line of gray colors used for edge enhancement, but these colors do not represent any materials.
  • the color component can determine that the RGB color of a pixel belongs to the RGB_DNA whole set of colors, but it can not determine which one of the 29 curves this color is a part.
  • each color curve represents the line in (P, C) space with Zeff ⁇ const. They are referred to herein as "z-lines.” If the colors of each line are known, it is possible to exploit this fact for at least two very useful applications. The first application is the physics-based feature vectors computation in pattern classification algorithms, which will be discussed in more detail below.
  • the second application uses z-lines for removing or keeping selected materials from an image. This is a much more flexible image filtering tool than so called "organic and metal stripping" provided by x-ray scanner manufacturers, as will be discussed in more detail below.
  • the angular coordinate ⁇ is an invariant for all points of the same z-line. This means that HSI color space is more suitable, or more natural, for z-lines than RGB, and extraction of z-line's colors is a straightforward operation universal, not only for the Smith color scheme, but for the Rapiscan color scheme as well.
  • Figure 37 shows z-line numbers 3 and 15 together with their respective colors.
  • Figure 38 shows a 3D view for extracted z-line numbers 1, 7 and 25 with respective colors.
  • Saturation S is thus far unemployed. It can be an unemployed free parameter (and is for Smith and Rapiscan scanners) responsible for carrying the proprietary "look and feel" of the color scheme. Colors of the same objects can appear differently on Smith and Rapiscan scanners having the same or close H and I, but different S.
  • Results of feature extraction for color images depends on the colors of an image, the color scheme (RGB, HIS or other) and the algorithm of the features computation itself. Mapping z-lines and their ordered colors to (P 5 C) space opens up an opportunity to exclude color from the feature extraction process. Instead of using three variables of a particular color space, such as R, G, and B in RGB, to feed the feature extraction algorithm, two variables of (P 5 C) can be used.
  • Objects are represented in images as groups of associated pixel intensities that generally vary within a defined pattern for that object and imaging modality. Variation of intensities among an object's pixels/voxels provides observable and quantifiable textures that correspond to properties of the original object.
  • signature mapping causes these objects to form separated neighborhood groups and differentiate themselves from other pixels/voxels objects in the image. The resulting differentiated pixel/voxel groups are each expressed as a unique "signature”.
  • signature mapping delineates an object's density gradients and highlights its boundaries based on the rate of neighboring pixel intensity variation.
  • Signature mapping is achieved by applying high order derivatives to the image data.
  • SMO signature mapping object
  • Figure 47 shows an example of a transformational sequence.
  • the original image 1500 is an 8-bit grayscale image and is replicated three times to form layers 1510A, 1510Band 1510C.
  • Each of the newly-created images or layers together form an image stack 1510.
  • Each layer 1510A, 1510B and 1510C in the stack 1510 may subsequently processed with one or more transforms.
  • layer 1 (1510A) is processed with a wavelet algorithm while layers 2 (1510b) and 3 (1510C) are processed with spatially-based non-linear transforms (SBTs) to yield three layers 1520A, 1520B and 1520C with new pixel values.
  • SBTs spatially-based non-linear transforms
  • the resultant pixel data for each layer is then processed a second time with SBTs. The output of this process is then mapped into 3 different channels (RGB color space in this example).
  • the output of one transform sequence is used as the input for a subsequent sequence.
  • individual layers may be combined with other layers through a mathematical function, such as multiplying or adding the pixel values of one layer with another.
  • the output of this procedure may, for example, be further combined with other layers or even the original image itself.
  • any image, layer or set of layers can have specific pixels removed and the remaining pixels remapped to the original pixel values.
  • the resultant image(s) can then be saved to a file and/or displayed on a monitor for viewing.
  • the RGB image can then be analyzed, relevant SMO features can be extracted, and signatures can be analyzed to determine the presence of certain types of objects.
  • Color images can be also processed using signature mapping in three ways: (1) the color images can be processed with their color values; (2) separate color channels in a color image can be processed separately; or (3) converted first to grayscale.
  • the grayscale image can then be processed using signature mapping as described in connection with Fig. 47. This has the effect of minimizing the obscuring effects of minimal color differences that make distinguishing the object of interest difficult.
  • the resultant frequency and/or spatially- processed signature mapped images may be combined, and the signature mapped image(s) may be combined with the original color image.
  • Any type of image can be processed using signature mapping, including, but not limited to images from a wide range of color spaces, such as HSI (HSV/HSL), RGB, CMYK.
  • signature mapping dynamically generates pixel neighborhood differentiation; b) the total number of signature mapping stages (consecutive image transformations) in the dynamic process is not limited; c) signature mapping has an iterative design where objects of interest become more and more visually and quantifiably discernible as sequences are added to the process; and d) additional sequences can be added to optimize the system's sensitivity and selectivity and thus remove false positives.
  • the signature mapping process typically begins with the use of various imaging software tools (e.g., Adobe ® Photoshop ® ) to generate linear and/or nonlinear transformational filters.
  • the transformations are preferably designed to reveal, enhance and visually expose the objects of interest and surrounding areas in image data. Sequences are preferably combined so that signature mapping becomes an iterative process in which every iteration (image transformation) gives rise to better discrimination for objects of interest.
  • the output of signature mapping can be used to delineate signatures that emphasize an object's texture, morphology, and boundaries.
  • the signatures may be designed to correlate to the morphological features of various tissues and to known pathology of diseases.
  • Figure 48 illustrates the results of two out of many possible signature mapping sequences.
  • the original image 1600 on the left is a mammogram.
  • the mammogram contains a tumor (malignant formation) 1610 that can be seen as a circular-shaped bright spot.
  • the two images on the right 1620 and 1630 are the result of processing the mammogram with two different signature mapping sequences.
  • the grayscale image 1620 clearly defines the texture (structure) inside the tumor 1610 including projections known as spiculations.
  • the image on the lower right 1630 displays a color set (signature) that is characteristic for this type of cancerous lesion.
  • the malignant formation consists of 2-3 approximately circular, sometimes nearly concentric, ragged regions 1640 encompassing a central core 1650. Both the circular regions 1640 and the core 1650 have visually different colors against images of the non cancerous breast tissue, even when there is no such visually noticeable difference in the original image 1600.
  • Applying the signature mapping process to "normal" mammograms without the cancerous lesions/tumors 1610 does not result in the appearance of the picture described above. By creating color gradients, characteristics signatures for a given tumor type remains constant over a broad range of image densities. As a result, signature mapping can accurately characterize tissues, even when the tumors are of very different sizes, shapes, and densities and/or they are located in greatly varying densities of breast tissue.
  • the result of applying signature mapping to a particular mammogram has many benefits.
  • One of these is that the visual characteristics of the signatures can be used effectively by radiologists in their clinical practice.
  • Another benefit is that the transformed images produced at different visually- derived stages of signature mapping provides valuable information for application in CAD systems. It is much easier to develop algorithms of malignant formation detection/localization by taking into account the mentioned visual malignant formation features. In a rare case when signature mapping does not lead directly to a visually distinctive malignant formation, it is still possible to use the transformed images together with more traditional sources of pertinent clinical information and the original image 1600. Since the transformed images are typically derived nonlinearly to enhance the visual malignant formation characteristics, informational features extracted from these images are a valuable addition to the traditionally used features.
  • Figures 49a and 49b show another example of the use of signature mapping on a mammogram.
  • Figure 49a shows the original image 1700
  • Fig. 49b shows the processed image 1710 after signature mapping.
  • the tumor can be seen in the original image 1700
  • a comparison of Figs. 49a and 49b illustrates how signature mapping distinguishes the features and boundaries of the tumor 1720 that allow for the improvement of CAD performance.
  • the signature mapped image 1710 reveals other patterns that are not discernable in the original image 1700.
  • Figures 50a and 50b show two mammograms from the database, with Fig. 50a showing the original mammograms 1800A and 1800B, and Fig. 50b showing the same mammograms after processing with the signature mapping software (1810A and 1810B, respectively).
  • the breast shown in image 1800A has a tumor.
  • the breast in image 1800B is normal.
  • the mammograms 1800A and 1800B were classified the most difficult to read with a "level 1" subtlety.
  • the resulting color signature in image 1810A accurately indicates the presence and exact location of the biopsy-confirmed tumor 1820.
  • Image 1810B does not show a signature for a tumor.
  • Signature mapping sequences can be employed solely to visualize the presence of tumors.
  • the tumors can be easily observed even when they are very small, occur in dense or very fatty breast tissue, or occur along the chest wall or nipple areas of the breast.
  • Using signature mapping as a visualization tool can greatly enhance the diagnostic accuracy of radiologists, even without the use of CAD.
  • object discrimination can be even more accurate.
  • signature mapping is applied to an image
  • SMO feature vectors are extracted from the resultant image data
  • edge detection operators i.e. Canny, Marr-Hildreth or Petrou
  • flexible shape extractors snakes
  • image segmentation Hardalick-Shapiro and graph partitioning
  • region descriptors shape characteristics, color histograms and, Zernike moments
  • the data for the analysis is extracted from the following sources: a. SMOs generated at various stages of the SM process, b. original (unfiltered) mammogram images, c. all the available clinical information regarding a particular patient including, but no limited to age, the related hereditary diseases, the number of children, etc.
  • a discriminative predictive model can be developed using a classification approach (e.g., Neural Nets, Support Vector Machines or/and Random Forests).
  • a classification approach e.g., Neural Nets, Support Vector Machines or/and Random Forests.
  • One aspect to this classification approach is the extraction of feature vectors and the features chosen during the previous step.
  • the model is tuned regarding a better feature selection in combination and a model parameter adjustment.
  • Figure 51 illustrates a longitudinal study from an existing mammogram collection. It illustrates the analytical possibilities using signature mapping in creating a predictive model. The original mammograms are shown on the top row 1900 and the signature mapped images are shown in the bottom row 1910 for four years starting in the year 1997.
  • FIG. 52a and 52b A section of a mammogram is shown in Figures 52a and 52b, with Fig. 52a showing an original mammogram and Fig. 52b showing the mammogram after signature mapping.
  • the breast shown in the mammograms contains numerous microcalcifications. Radiologists and CAD software currently access these calcifications by determining if they are located in a cluster.
  • Figure 52b (after signature mapping) shows the actual structure inside of the microcalcifications. The differences in SMO features between the microcalcifications on the upper left (1800) and those in the lower right (1810) provide information as to the growth of the calcifications into tumors.
  • Hyperspectral [00269] Signature mapping can be used to process hyperspectral and color images.
  • each layer of the image can be treated as one of the transform layers of the signature mapping image stack.
  • the output of signature mapping can be combined with typical methods used for hyperspectral analysis, such as Principle Component Analysis and Hidden Markov Models.
  • Combinations of hyperspectral signatures and signature mapping signatures can be integrated for modeling and analysis including the creation of entirely new approaches for analysis using feature space mapping.
  • the signature mapping sequencing may be optimized by first performing analysis on representative hyperspectral data then mapping relevant frequencies into a feature space.
  • the hyperspectral images for these frequencies can be selected and then further processed using signature mapping. This can greatly reduce the number of "slices" needed to be sent from a airplane or satellite, thus reducing network requirements and processing time.
  • Figures 53a and 53b show a color representation of 3 layers from a hyperspectral image, with Fig. 53a showing the original hyperspectral image and Fig. 53b showing the hyperspectral image after signature mapping.
  • the red boxes 2000 indicates the location of vehicles.
  • Fig. 53b shows an orange-colored signature where vehicles were present. Only 3 layers of the hyperspectral image were needed to identify the objects using SM compared with 64 using standard analytical methods used for analyzing hyperspectral data.
  • the image analysis system 130 can be implemented with a general purpose computer. However, it can also be implemented with a special purpose computer, programmed microprocessor or microcontroller and peripheral integrated circuit elements, ASICs or other integrated circuits, hardwired electronic or logic circuits such as discrete element circuits, programmable logic devices such as FPGA, PLD, PLA or PAL or the like.
  • the signature mapping algorithm can be used as either one or combination of
  • the signature mapping process preferably consists of five major
  • Feature vectors are extracted (2530) from the data generated by the characterizations and analyzed by a rules-based model for each area of interest candidate.
  • a classification (2540) result is determined for each area of interest candidate (disease or non-disease, threat or non-threat) and a degree of confidence is pronounced or quantification details presented, if required.
  • the input image data is converted into
  • the image pre-processing step 2500 can also apply image
  • these areas can be lung field or breast region.
  • This phase assures that the follow-up processing phases is only limited inside the delineated area thus detection outputs will not be shown outside this area.
  • the algorithms can use the contrast between background and body, as well as certain anatomic parameters and/or signature mapping processes to determine the delineated area.
  • a pre-processed image 2505 in one or more channels is generated after this phase.
  • the original image 2400 (grayscale in this example) is first converted into a multi channel (may be represented as a color image in RGB or CMYK color space or with any number of channels) in step 2500. Then, at step 2510, a series of image transformation processes are then applied to each channel of the image iteratively and consecutively, as described in connection with Fig. 47 described above to yield multiple transformation images 2515.
  • the image transformation step 2510 is further defined as follows:
  • signature mapping employs a synthesis-based methodology that expands the original pixel data.
  • the original image contains only one channel, as is the case in a grayscale image, the original image is replicated one or more times to form additional identical channels.
  • the transforms are then applied to these newly created channels.
  • the transformational sequence(s) may include frequency-based, spatially-based, or a combination of both types of algorithmic processes that collectively generate additional image planes to form a multi-spectral-like hypercube.
  • the hypercube has pixel dimensions P n where n is the total number of outputs from all iterations of the transformations applied to all channels of the original image plus the original image itself.
  • the combined output of all channels in the newly created hypercube generates a high order derivative of the original image in which objects of similar physical makeup form signature-mapped objects (SMO) that are consistent for its type of material and distinguishes itself from other objects in the image.
  • SMO signature-mapped objects
  • step 2530 areas of interest candidates 2525 are differentiated from most other material areas of interest that could be false positives through additional transformations that characterize the areas of interest.
  • one or more segregation scenarios are applied to the AOIs to generate data and corresponding hypercubes.
  • Pixel-based features individual pixel values from each pixel in an AOI — these are derived using color histograms from transformed color channels including normalized color values, color compositions, unique color representations, hue/luminance/saturation (HLS) values. These color histograms directly capture the property (new pixel values) created from image transformation.
  • HLS hue/luminance/saturation
  • Region-based features derived from the neighborhood relationships over an area including elements of co-occurrence matrix, coefficients of Gabor and log Gabor wavelet transformation. These features describe the texture inside and outside the object.
  • Object-based features including statistical moments, frequency, and other feature parameters describing the shape, size, perimeters, etc.
  • AU of the features are preferably expressed as a digital number and some of them are preferably based on clinical information of disease and normal structures or differences between different material characteristics.
  • a pixel-based feature can have over 2 millions digital numbers, region-based feature over 30,000 numbers, and object- based features over 50 numbers.
  • the most highly probable 450 features could be selected such that they best represent diseases from non-diseases. These 450 features can form two obvious clusters (representing disease vs. non-disease) in hyperspace separated by hyperplane. Although over 2 millions features have been calculated initially, approximately 450 optimal feature vectors 2535 would then be used for the follow-up classification step 2540 in the signature mapping process. Pending the types of material, diseases and imaging modalities, 30-90 feature vectors would then typically provide both the robust and accurate classification results.
  • the metrics obtained during this phase of image analysis are determined through testing and selected based on a particular object's signature and the properties that optimally differentiate its signature from that of other materials.
  • the following list is provided as an illustration of only a few of the many thousands of sampling protocols that can be utilized during the contextual imagery analysis phase:
  • the classification step 2540 a methodology is utilized to recognize each candidate preferably based on an average of 500 optimal feature values.
  • the classification step 2540 generates a degree of confidence for its classification.
  • the signature mapping process can use different classification models (e.g., decision tree, voting, etc.).
  • classification models e.g., decision tree, voting, etc.
  • multi-class or multi- target models in which detection would mean not only general separation of abnormalities or threatening conditions, but also distinguishing their specific types.
  • SVM Support Vector Machine
  • SVMs also have proven themselves as efficient tools and we prefer to use the most practical case of single target models aimed at producing simple "normal-abnormal" (or “threat-non-threat”) classifications.
  • a Support Vector Machine is utilized to perform classification by constructing an iV-dimensional hyperplane that optimally separates the data into two categories, i.e., disease and non-disease ones, where, N is the number of features used to train the SVM.
  • the goal of SVM modeling is to find the optimal hyperplane that separates clusters of vector in such a way that cases with one category of the target variable (e.g., disease one) are on one side of the plane and cases with the other category (e.g., non-disease one) are on the other size of the plane.
  • Such a separation is achieved via a 'training' process using sets of optimal feature vectors associated grouped into disease and non-disease ones.
  • Training of SVM is to obtain the best model (i.e., mathematical description) of the hyperplane.
  • the trained classification simply uses the feature vector of a selected candidate (consisting, for example, of 3 color AOIs and 1 luminescence AOI) to determine on which side of the decision boundary for an input test pattern lies and assign the corresponding class label of disease or non-diseases.
  • the classification phase also generates an approximate degree of confidence by calculating the distance from the centroid of feature vector to hyperplane.
  • Modules of Assembly Line for an SVM Model [00288] Currently, the assembly line for generating a SVM model includes 7 processing modules which cover its functionality, creating the specific flow for an assembly line. The descriptions of particular modules and their functionality are provided below.
  • Rough Filtering Module The Development of an SVM Model begins from filtering & squeezing the initial set of potentially useful features, which often, at the beginning, may number in millions, in order to better describe the Data Set of Threat & Non-Threat Cases.
  • Separability is an analytic tool for creating a list of separability measurements for the features left after the initial rough squeezing. It ranks features according to their ability to separate Threat/Non-Threat cases.
  • Filter Separability Module This module creates a restricted set of the highest ranking features. It helps in improving the data and boosting the detection rates.
  • SVM Training Module Training of the resulting SVM Model implements a learning process through development of a set of rules aimed at distinguishing between Threat and Non- threat Cases.
  • SVM Test Module Testing of the resulting SVM Model allows making predictions regarding the Threats and Non-Threats within the Test Set. In this way the Test Module is able to calculate the expected Detection rate for the final SVM Model. Note that Threat and Non-threat cases are completely identifiable and presented within a Test Set in a non-biased and equal way.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

L'invention propose un système et un procédé pour la détection, la caractérisation, la visualisation et la classification d'objets dans des données d'image. La présente invention utilise les principes de divergence de transformation itérative et de mise en correspondance de signature, dans lesquels des objets dans des images, une fois soumis à des transformations spéciales, afficheront des réponses radicalement différentes sur la base des propriétés physiques, chimiques ou numériques de l'objet ou de sa représentation (telle que des images), combinées à des capacités d'apprentissage machine. En utilisant le système et les procédés de la présente invention, certains objets qui semblent non distinguables d'autres objets à l'œil ou pour des systèmes de reconnaissance informatiques, ou qui sont autrement presque identiques, génèrent des différences radicalement différentes et statistiquement significatives dans les descripteurs (mesures) d'image qui peuvent être facilement mesurés.
PCT/US2008/067949 2007-06-21 2008-06-23 Système et procédé pour la détection, la caractérisation, la visualisation et la classification d'objets dans des données d'image Ceased WO2008157843A1 (fr)

Applications Claiming Priority (12)

Application Number Priority Date Filing Date Title
US92931307P 2007-06-21 2007-06-21
US60/929,313 2007-06-21
US12/014,043 US7840048B2 (en) 2004-05-26 2008-01-14 System and method for determining whether there is an anomaly in data
US12/014,028 US7817833B2 (en) 2004-05-26 2008-01-14 System and method for identifying feature of interest in hyperspectral data
US12/014,043 2008-01-14
US12/014,034 US7907762B2 (en) 2004-05-26 2008-01-14 Method of creating a divergence transform for identifying a feature of interest in hyperspectral data
US12/014,034 2008-01-14
US12/014,028 2008-01-14
US1029808A 2008-01-23 2008-01-23
US12/010,298 2008-01-23
US5748608P 2008-05-30 2008-05-30
US61/057,486 2008-05-30

Publications (1)

Publication Number Publication Date
WO2008157843A1 true WO2008157843A1 (fr) 2008-12-24

Family

ID=40156723

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2008/067949 Ceased WO2008157843A1 (fr) 2007-06-21 2008-06-23 Système et procédé pour la détection, la caractérisation, la visualisation et la classification d'objets dans des données d'image

Country Status (1)

Country Link
WO (1) WO2008157843A1 (fr)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015051463A1 (fr) * 2013-10-09 2015-04-16 Voti Inc. Techniques d'imagerie d'un objet balayé
US9406146B2 (en) 2009-06-30 2016-08-02 Koninklijke Philips N.V. Quantitative perfusion analysis
WO2019032558A1 (fr) * 2017-08-07 2019-02-14 Imago Systems, Inc. Système et procédé de visualisation et de caractérisation d'objets dans des images
CN110412548A (zh) * 2019-07-20 2019-11-05 中国船舶重工集团公司第七二四研究所 基于高分辨一维距离像的雷达多目标识别方法
CN111160364A (zh) * 2019-12-05 2020-05-15 湖南大学 一种基于不同域下残差特征的多种操作链取证检测方法
CN111681249A (zh) * 2020-05-14 2020-09-18 中山艾尚智同信息科技有限公司 基于Grabcut的砂石颗粒的改进分割算法研究
US10873681B2 (en) 2016-02-08 2020-12-22 Imago Systems, Inc. System and method for the visualization and characterization of objects in images
US11308619B2 (en) 2020-07-17 2022-04-19 International Business Machines Corporation Evaluating a mammogram using a plurality of prior mammograms and deep learning algorithms
CN114677303A (zh) * 2022-03-28 2022-06-28 中国人民解放军国防科技大学 一种基于集成边缘数据增强的亚像素边缘检测方法
US11734819B2 (en) 2020-07-21 2023-08-22 Merative Us L.P. Deep learning modeling using health screening images
WO2023232191A1 (fr) * 2022-06-02 2023-12-07 minrocon GmbH Procédé d'analyse de la densité de grains individuels d'un flux de matière en vrac de manière orientée
US11885752B2 (en) 2021-06-30 2024-01-30 Rapiscan Holdings, Inc. Calibration method and device therefor
CN117975452A (zh) * 2024-01-31 2024-05-03 西南石油大学 基于显微图像处理和gbdt决策树砂岩研磨性预测方法
US11977037B2 (en) 2018-10-22 2024-05-07 Rapiscan Holdings, Inc. Insert for screening tray
US12019035B2 (en) 2021-07-16 2024-06-25 Rapiscan Holdings, Inc. Material detection in x-ray security screening
CN118898757A (zh) * 2024-07-03 2024-11-05 中国人民解放军陆军勤务学院 一种基于遥感图像时空谱特征关联融合的样本数据迁移增强方法及系统
US12181422B2 (en) 2019-09-16 2024-12-31 Rapiscan Holdings, Inc. Probabilistic image analysis
WO2025078936A1 (fr) * 2023-10-08 2025-04-17 Optina Diagnostics, Inc. Procédé et système de mappage de structures biologiques indiquant un état médical, à partir d'une image rétinienne multispectrale
US12361671B2 (en) 2021-09-07 2025-07-15 Rapiscan Systems, Inc. Methods and systems for accurate visual layer separation in the displays of scanning systems

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030012441A1 (en) * 2001-07-02 2003-01-16 Jasc Software, Inc. Detection of lines in images
US20050074154A1 (en) * 2003-10-02 2005-04-07 Siemens Corporate Research Inc. System and method for local deformable motion analysis
US20060104513A1 (en) * 2004-11-15 2006-05-18 Shmuel Aharon GPU accelerated multi-label digital photo and video editing

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030012441A1 (en) * 2001-07-02 2003-01-16 Jasc Software, Inc. Detection of lines in images
US20050074154A1 (en) * 2003-10-02 2005-04-07 Siemens Corporate Research Inc. System and method for local deformable motion analysis
US20060104513A1 (en) * 2004-11-15 2006-05-18 Shmuel Aharon GPU accelerated multi-label digital photo and video editing

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9406146B2 (en) 2009-06-30 2016-08-02 Koninklijke Philips N.V. Quantitative perfusion analysis
US10089956B2 (en) 2013-10-09 2018-10-02 Voti Inc. Techniques for imaging a scanned object
WO2015051463A1 (fr) * 2013-10-09 2015-04-16 Voti Inc. Techniques d'imagerie d'un objet balayé
US10873681B2 (en) 2016-02-08 2020-12-22 Imago Systems, Inc. System and method for the visualization and characterization of objects in images
US11734911B2 (en) 2016-02-08 2023-08-22 Imago Systems, Inc. System and method for the visualization and characterization of objects in images
WO2019032558A1 (fr) * 2017-08-07 2019-02-14 Imago Systems, Inc. Système et procédé de visualisation et de caractérisation d'objets dans des images
US11935215B2 (en) 2017-08-07 2024-03-19 Imago Systems, Inc. System and method for the visualization and characterization of objects in images
US11977037B2 (en) 2018-10-22 2024-05-07 Rapiscan Holdings, Inc. Insert for screening tray
US12467886B2 (en) 2018-10-22 2025-11-11 Rapiscan Holdings, Inc. Tray insert for screening tray
CN110412548A (zh) * 2019-07-20 2019-11-05 中国船舶重工集团公司第七二四研究所 基于高分辨一维距离像的雷达多目标识别方法
US12181422B2 (en) 2019-09-16 2024-12-31 Rapiscan Holdings, Inc. Probabilistic image analysis
CN111160364A (zh) * 2019-12-05 2020-05-15 湖南大学 一种基于不同域下残差特征的多种操作链取证检测方法
CN111160364B (zh) * 2019-12-05 2023-06-02 湖南大学 一种基于不同域下残差特征的多种操作链取证检测方法
CN111681249A (zh) * 2020-05-14 2020-09-18 中山艾尚智同信息科技有限公司 基于Grabcut的砂石颗粒的改进分割算法研究
US11308619B2 (en) 2020-07-17 2022-04-19 International Business Machines Corporation Evaluating a mammogram using a plurality of prior mammograms and deep learning algorithms
US11734819B2 (en) 2020-07-21 2023-08-22 Merative Us L.P. Deep learning modeling using health screening images
US11885752B2 (en) 2021-06-30 2024-01-30 Rapiscan Holdings, Inc. Calibration method and device therefor
US12019035B2 (en) 2021-07-16 2024-06-25 Rapiscan Holdings, Inc. Material detection in x-ray security screening
US12270772B2 (en) 2021-07-16 2025-04-08 Rapiscan Holdings, Inc. Material detection in X-ray security screening
US12361671B2 (en) 2021-09-07 2025-07-15 Rapiscan Systems, Inc. Methods and systems for accurate visual layer separation in the displays of scanning systems
CN114677303B (zh) * 2022-03-28 2024-02-02 中国人民解放军国防科技大学 一种基于集成边缘数据增强的亚像素边缘检测方法
CN114677303A (zh) * 2022-03-28 2022-06-28 中国人民解放军国防科技大学 一种基于集成边缘数据增强的亚像素边缘检测方法
WO2023232191A1 (fr) * 2022-06-02 2023-12-07 minrocon GmbH Procédé d'analyse de la densité de grains individuels d'un flux de matière en vrac de manière orientée
WO2025078936A1 (fr) * 2023-10-08 2025-04-17 Optina Diagnostics, Inc. Procédé et système de mappage de structures biologiques indiquant un état médical, à partir d'une image rétinienne multispectrale
CN117975452A (zh) * 2024-01-31 2024-05-03 西南石油大学 基于显微图像处理和gbdt决策树砂岩研磨性预测方法
CN118898757A (zh) * 2024-07-03 2024-11-05 中国人民解放军陆军勤务学院 一种基于遥感图像时空谱特征关联融合的样本数据迁移增强方法及系统

Similar Documents

Publication Publication Date Title
WO2008157843A1 (fr) Système et procédé pour la détection, la caractérisation, la visualisation et la classification d'objets dans des données d'image
US7817833B2 (en) System and method for identifying feature of interest in hyperspectral data
US8045805B2 (en) Method for determining whether a feature of interest or an anomaly is present in an image
US7907762B2 (en) Method of creating a divergence transform for identifying a feature of interest in hyperspectral data
Khan et al. Lungs nodule detection framework from computed tomography images using support vector machine
US7492937B2 (en) System and method for identifying objects of interest in image data
US20100266179A1 (en) System and method for texture visualization and image analysis to differentiate between malignant and benign lesions
US8488863B2 (en) Combinational pixel-by-pixel and object-level classifying, segmenting, and agglomerating in performing quantitative image analysis that distinguishes between healthy non-cancerous and cancerous cell nuclei and delineates nuclear, cytoplasm, and stromal material objects from stained biological tissue materials
EP2391887B1 (fr) Évaluation des risques à base d'images - indicateur pronostique de survie et résultats d'une histopathologie numérique
JP3947109B2 (ja) コンピュータ利用画像分析
US7840048B2 (en) System and method for determining whether there is an anomaly in data
US20060269135A1 (en) System and method for identifying objects of interest in image data
Sinha et al. Medical image processing
WO2013049153A2 (fr) Systèmes et procédés pour le criblage et le pronostic automatisés du cancer à partir d'images de biopsie sur lamelle entière
Swiderski et al. Novel methods of image description and ensemble of classifiers in application to mammogram analysis
US6674880B1 (en) Convolution filtering of similarity data for visual display of enhanced image
WO2010063010A2 (fr) Système et procédé de visualisation de texture et d'analyse d'image pour différencier des lésions malignes et des lésions bénignes
GB2457022A (en) Creating a fuzzy inference model for medical image analysis
Safdari et al. Image patch-based method for automated classification and detection of focal liver lesions on CT
US20060269161A1 (en) Method of creating a divergence transform for a class of objects
US12211206B1 (en) Multistage region-of-interest identification in medical images
Kehl et al. Multi-spectral imaging via computed tomography (music)-comparing unsupervised spectral segmentations for material differentiation
Boccignone et al. Using Renyi's information and wavelets for target detection: an application to mammograms
Mercan et al. Automated Diagnosis of Breast Cancer and Pre-invasive Lesions on Digital Whole Slide Images.
Mousa et al. Enhancing Diagnostic Decision-Making with Image Mining Techniques: A Proposed Framework for Medical Images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 08771770

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 08771770

Country of ref document: EP

Kind code of ref document: A1