JP2006292385A - Visual information processing system and visual information processing method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a visual information processing system capable of new visual information processing, which has not been acquired before, by combining an optical spatial modulation element and an imager. <P>SOLUTION: The visual information processing system is provided with both an optical pattern generating part having the optical spatial modulation element for modulating radiant light from a light source on the basis of a prescribed optical pattern for irradiating an object to be imaged with radiant light modulated on the basis of the optical pattern and an image characteristics extraction part for imaging the object to be imaged on which the optical pattern is superposed and extracting the amount of characteristics of the object to be imaged. The optical pattern comprises a plurality of optical patterns capable of acquiring three-dimensional information of the object to be imaged on the basis of the principle of triangulation and optical patterns in which at least either the type or number of the optical patterns is adaptively controlled on the basis of the amount of characteristics extracted by the image characteristics extraction part. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a visual information processing system and a visual information processing method thereof, and more particularly to a visual information processing system using a spatial light modulation element and a high-speed imaging element and a visual information processing method thereof.

  Today, imaging devices and cameras using solid-state imaging devices such as CMOS image sensors are spreading their application fields in various forms regardless of consumer use or industrial use.

  In particular, in the field of industrial imaging devices, application forms in which physical fluctuations of an object are extracted from image data and the extracted information is used as input values and feedback values for various control systems have become widespread.

  For example, a system that extracts a physical quantity such as the position of the center of gravity of a moving object and the posture angle of the moving object by imaging a moving object, and performs various controls, for example, control of a robot arm, based on the physical quantity or its change amount is there. In an imaging device used in such a control system, although it depends on the moving speed of an object, in general, it is possible to realize a much higher resolution and a higher frame rate in real time than a consumer application. It becomes necessary.

  The imaging camera device disclosed in Patent Document 1 discloses a technology that achieves both high spatial resolution and high-speed real-time performance. FIG. 15 illustrates a configuration of an imaging camera device 200 disclosed in Patent Document 1, and includes an imager (solid-state imaging device) 20 having a high spatial resolution, a calculation unit 30, and a coordinate conversion unit 40. .

  Image data (image data at the current frame time) of the partial image (m × n) specified by the coordinate conversion unit 40 is cut out from the entire image (M × N) of the imager 20 to the arithmetic unit 30. Is input. The calculation unit 30 performs a calculation for extracting the image feature amount ξ from the partial image data. The image feature amount ξ is, for example, the size, position, posture angle, etc. of the object included in the partial image data, or the amount of change thereof. The coordinate conversion unit 40 calculates the coordinates (m ′ × n ′) of the partial image to be cut out at the next frame time based on the extracted image feature amount ξ. For example, when the object is moving in parallel or rotating, the coordinates (m × n) of the partial image of the previous frame are subjected to coordinate conversion for parallel movement or rotation by affine transformation or the like, and the next frame The coordinates (m ′ × n ′) of the partial image are calculated.

  The calculated coordinates (m ′ × n ′) of the next frame are sequentially read from all the images of the imager, and image data of a new partial image (m ′ × n ′) is input to the arithmetic unit 30.

  In the imaging camera device 200 disclosed in Patent Document 1, the image feature amount ξ calculated by the calculation unit 30 is fed back to read out a necessary partial image instead of reading out all image data of the imager. This makes it possible to achieve high-speed real-time performance while maintaining high spatial resolution.

On the other hand, a technique for imaging an imaging object not only as a two-dimensional image but also as a three-dimensional image is being developed today. For example, Non-Patent Document 1 discloses a method for generating a three-dimensional image based on the triangulation principle.
JP 2003-319262 A “Outline of Image Processing / Experiment”, [online], [Search April 1, 2005], Internet <URL: http://www.image.esys.tsukuba.ac.jp/range_finder/gaiyo.html>

  By the way, recently, a device called a light spatial modulation element has been developed and put into practical use, and has been widely used in projector apparatuses and the like. There are two types of spatial light modulators: a transmissive type such as a liquid crystal light valve, and a reflective type called DMD (Digital Micro-mirror Device).

  In a projector apparatus using a transmissive light spatial modulation element, an image signal is applied to a liquid crystal light valve, and light from a light source is transmitted through the liquid crystal light valve to project an image on a front screen or the like.

  On the other hand, a projector apparatus using a reflective spatial modulation element projects an image on a front screen or the like after reflecting light from a light source to a DMD. The DMD has a large number of minute mirrors corresponding to the number of pixels. By changing the angle of the mirror for each pixel, the amount of light projected on the screen is turned on and off for each pixel, and a two-dimensional image is displayed. To form. Although the change in the mirror angle changes in a binary manner depending on the on / off signal, the continuous light intensity is expressed for the human eye by setting the on / off period by pulse width modulation. Can do. Further, by changing the duty ratio of the pulse width modulation for each pixel and each frame, a continuous moving image can be formed both spatially and temporally.

  Conventionally, light spatial modulation elements such as liquid crystal light valves and DMDs are often used exclusively for projector apparatuses. With the development of high-brightness and high-speed system technology, it is not only used as a projector for presentations, but also as a large TV or a projector for theatrical movies. Is used as an image forming element of a projector apparatus.

  The visual information processing system according to the present invention does not use these light spatial modulation elements as conventional image forming elements.

  Image formation (imaging) itself is performed by a solid-state image sensor such as a CMOS image sensor. At the same time, spatial and temporal modulation is performed on the light from the object incident on the solid-state imaging device by using an optical spatial modulation device such as a liquid crystal light valve or DMD.

  An object of the visual information processing system according to the present invention is to realize new visual information processing that has not been obtained in the past by combining a light spatial modulation element and a solid-state imaging element.

  In particular, an image feature amount is extracted from image data picked up by an imager (solid-state image pickup device) as in the technique disclosed in Patent Document 1, and an optical spatial modulation device is provided in a feedback loop based on the image feature amount. Thus, the object is to realize new visual information processing adapted to the movement and change of the object.

  Specifically, one of the objects of the present invention is to combine a spatial light modulation element and an imager (solid-state image sensor), thereby feeding back a three-dimensional image of an imaging object including a moving object based on an image feature amount. An object of the present invention is to provide a visual information processing system and a visual information processing method thereof that can be generated at high speed and with high efficiency by control.

  Another object of the present invention is to combine a light spatial modulation element and an imager so that the difference between brightness and darkness is very large, and it is also suitable for an object whose difference changes temporally and spatially. An object of the present invention is to provide a visual information processing system and a visual information processing method thereof that can ensure a dynamic range of brightness.

  Another object of the present invention is to provide a specific fixed or moving object while providing high-sensitivity image data by sacrificing resolution to some extent as a whole image by combining a spatial light modulator and an imager. Is to provide a visual information processing system and its visual information processing method capable of realizing high resolution.

  Another object of the present invention is to provide a visual information processing system capable of extracting a predetermined luminance variation for each pixel and generating a luminance variation spectrum image by combining a spatial light modulator and an imager, and the visual information It is to provide an information processing method.

  Furthermore, another object of the present invention is to provide visual information that can perform spatial filter processing such as edge enhancement on an entire image or a specific fixed or moving object by combining a spatial light modulator and an imager. To provide a processing system and a visual information processing method thereof.

  In order to solve the above-described problem, a visual information processing system according to the present invention includes an optical spatial modulation element that modulates emitted light from a light source based on a predetermined light pattern, as described in claim 1, An image for extracting the feature quantity of the imaging object by imaging the imaging object on which the light pattern is superimposed, and a light pattern generation unit that irradiates the imaging object with the emitted light modulated based on the light pattern A feature extraction unit, wherein the light pattern is a plurality of light patterns capable of obtaining three-dimensional information of the imaging object based on a triangulation principle, and the feature amount extracted by the image feature extraction unit Based on the above, at least one of the type and number of light patterns is an adaptively controlled light pattern. According to the visual information processing system having this configuration, it is possible to generate a three-dimensional image of an imaging object including a moving object at high speed and with high efficiency by feedback control based on an image feature amount.

  In addition, the visual information processing system according to the present invention includes a light spatial modulation element having a predetermined pixel region and controlling an exposure time of incident light from the imaging target for each pixel. An image feature extraction unit that captures an image of the imaging object with reflected or transmitted light from the spatial light modulation element in which an exposure time of incident light is controlled for each pixel, and extracts a feature amount of the imaging object; And the feature amount extracted by the image feature extraction unit is a feature amount including luminance for each pixel of the imaging object, and the spatial light modulator adapts an exposure time based on the luminance for each pixel. It is characterized by controlling. According to the visual information processing system according to this configuration, it is possible to secure a dynamic range of appropriate brightness even for an object in which the difference in brightness is very large and the difference changes temporally and spatially.

  According to the visual information processing system of the present invention, as described in claim 3, the pixels in the designated area are grouped into a plurality of pixels, and the plurality of pixel signals in each group are sequentially switched from the object to be imaged. An optical spatial modulation element that reflects or transmits the incident light and outputs the incident light from the imaging object output from the optical spatial modulation element with an imager having a resolution of the same number of pixels as the number of groups And an image feature extraction unit that extracts a feature amount of the imaging target, wherein the image feature extraction unit designates a region based on the feature amount, and is sequentially switched and input to the imager in the region. The signals of the plurality of pixels are combined to increase the resolution of the imager to the same resolution as the spatial light modulator. According to the visual information processing system of this configuration, high resolution can be realized for a specific fixed or moving object while providing high-sensitivity image data at the expense of resolution to some extent for the entire image.

  According to another aspect of the visual information processing system of the present invention, the pixels in the designated area are grouped into a plurality of pixels, and the plurality of pixel signals in each group are turned on / off at different frequencies. An optical spatial modulation element that reflects or transmits incident light from the imaging object, and the incident light from the imaging object that is output from the optical spatial modulation element, and the resolution of the optical spatial modulation element An image feature extraction unit that captures an image with the same resolution and extracts a feature amount of the object to be imaged, wherein the image feature extraction unit designates a region based on the feature amount, and the imaging is performed in the region. The luminance variation of the object is frequency-resolved corresponding to the different frequencies, and a spectrum image corresponding to the different frequencies is generated. According to the visual information processing system of this configuration, it is possible to extract a predetermined luminance variation for each pixel and generate a luminance variation spectrum image.

  According to the visual information processing system of the present invention, as described in claim 5, the pixels in the designated region are grouped into a plurality of pixels, and the plurality of pixel signals in each group are converted into a predetermined plurality of spatial patterns. An optical spatial modulation element that sequentially switches based on the incident light from the imaging object to be reflected or transmitted, and an incident light from the imaging object that is output from the optical spatial modulation element is equal to the number of groups. An image feature extraction unit that captures an image with an imager having a number of pixels and extracts a feature amount of the imaging target, and the image feature extraction unit specifies a region based on the feature amount, Calculating a plurality of signals sequentially output from the imager based on a plurality of spatial patterns, and performing spatial filtering on a plurality of pixels in the group as processing targets; That. According to the visual information processing system according to this configuration, it is possible to perform spatial filter processing such as edge enhancement on the entire image or a specific fixed or moving object.

  According to the visual information processing method of the present invention, as described in claim 8, the light spatial light modulating element modulates the emitted light from the light source based on a predetermined light pattern, and is modulated based on the light pattern. Irradiating the imaging object with the emitted light; and imaging the imaging object on which the light pattern is superimposed, and extracting a feature quantity of the imaging object, wherein the light pattern is triangular A plurality of light patterns capable of obtaining three-dimensional information of the imaging object based on a surveying principle, and at least one of the type and number of light patterns based on the feature values extracted by the image feature extraction unit The light pattern is adaptively controlled. According to the visual information processing method according to this method, a three-dimensional image of an imaging object including a moving object can be generated at high speed and with high efficiency by feedback control based on an image feature amount.

  The visual information processing method according to the present invention controls the exposure time of the incident light from the imaging target for each pixel by the light spatial modulation element having a predetermined pixel area. And extracting the feature quantity of the imaging object by imaging the imaging object with reflected light or transmitted light from the spatial light modulator whose exposure time is controlled for each pixel. Further, the feature amount is a feature amount including luminance for each pixel of the imaging object, and the step of controlling for each pixel adaptively controls an exposure time based on the luminance for each pixel. . According to the visual information processing method according to the present method, it is possible to ensure an appropriate brightness dynamic range even for an object in which the difference in brightness is very large and the difference changes temporally and spatially.

  According to the visual information processing method of the present invention, as described in claim 10, a plurality of pixel signals in the group are sequentially received by a spatial light modulation element in which pixels in a specified region are grouped into a plurality. The step of switching and reflecting or transmitting the incident light from the imaging object, and the incident light from the imaging object output from the light spatial modulation element has a resolution of the same number of pixels as the number of groups Capturing an image with an imager, and extracting a feature amount of the object to be imaged, wherein the step of extracting the feature amount designates a region based on the feature amount, and sequentially switches in the region to the imager. The method includes combining the signals of the plurality of input pixels and increasing the resolution of the imager to the same resolution as that of the spatial light modulator. According to the visual information processing method according to the present method, high resolution can be realized for a specific fixed or moving object while providing high-sensitivity image data at the expense of resolution to some extent for the entire image.

  According to the visual information processing method of the present invention, as described in claim 11, each of the plurality of pixel signals in the group is obtained by the light spatial modulation element in which the pixels in the specified region are grouped into a plurality of groups. A step of reflecting or transmitting incident light from the imaging object by turning on and off at different frequencies and outputting the incident light from the imaging object output from the spatial light modulator; Capturing an image with an imager having the same resolution as that of the image, and extracting a feature amount of the object to be imaged, wherein the step of extracting the feature amount specifies a region based on the feature amount, and in the region, Including a process of generating a spectrum image corresponding to the different frequencies by performing frequency decomposition on luminance variations of the imaging object corresponding to the different frequencies.According to the visual information processing method according to the present method, it is possible to extract a predetermined luminance variation for each pixel and generate a luminance variation spectrum image.

  According to the visual information processing method of the present invention, as described in claim 12, a plurality of pixel signals in the group are predetermined by a light spatial modulation element in which pixels in a specified region are grouped into a plurality. The step of sequentially switching based on the plurality of spatial patterns and reflecting or transmitting the incident light from the imaging object, and the incident light from the imaging object output from the light spatial modulation element, Imaging with an imager having the same number of pixels as the number of pixels, and extracting the feature amount of the object to be imaged, wherein the step of extracting the feature amount specifies a region based on the feature amount, and in the region A plurality of signals sequentially output from the imager based on the plurality of spatial patterns, and a spatial filter for processing a plurality of pixels in the group. Characterized by comprising a process of performing. According to the visual information processing method of the present method, it is possible to perform spatial filter processing such as edge enhancement on the entire image or a specific fixed or moving object.

  According to the visual information processing system and the visual information processing method according to the present invention, by combining the light spatial modulation element and the imager, it is possible to realize new visual information processing that has not been obtained conventionally, In particular, by providing an optical spatial modulation element in the feedback loop based on the image feature amount, it is possible to realize new visual information processing adapted to the movement and change of the object.

  Embodiments of a visual information processing system and a visual information processing method according to the present invention will be described with reference to the accompanying drawings.

(1) 1st Embodiment FIG. 1: is a figure which shows the system configuration example of the visual information processing system 1 which concerns on the 1st Embodiment of this invention. The visual information processing system 1 according to the first embodiment generates a three-dimensional image of an imaging object at high speed and with high efficiency.

  The visual information processing system 1 includes an image feature extraction unit 2 and a light pattern irradiation unit 3.

  Among these, the image feature extraction unit 2 has the same configuration as that of the imaging camera device 200 shown in FIG. 15, and includes an imager (solid-state imaging device) 20 having a high spatial resolution, a calculation unit 30, and coordinate conversion. Part 40. The imager 20 is a CMOS image sensor having a predetermined number of pixels, for example.

  Image data (image data at the current frame time) of the partial image (m × n) designated by the coordinate conversion unit 40 is extracted from the entire image (M × N) of the imager 20 in the calculation unit 30. Is input. The calculation unit 30 performs a calculation for extracting the image feature amount ξ from the partial image data. The image feature amount ξ is, for example, the size, position, posture angle, etc. of the object included in the partial image data, or the amount of change thereof. The coordinate conversion unit 40 calculates the coordinates (m ′ × n ′) of the partial image to be cut out at the next frame time based on the extracted image feature amount ξ. For example, when the object is moving in parallel or rotating, the coordinates (m × n) of the partial image of the previous frame are subjected to coordinate conversion for parallel movement or rotation by affine transformation or the like, and the next frame The coordinates (m ′ × n ′) of the partial image are calculated.

  The calculated coordinates (m ′ × n ′) of the next frame are sequentially read from all the images of the imager, and image data of a new partial image (m ′ × n ′) is input to the arithmetic unit 30.

  In the imaging camera device 200 disclosed in Patent Document 1, the image feature amount ξ calculated by the calculation unit 30 is fed back to read out a necessary partial image instead of reading out all image data of the imager. This makes it possible to achieve high-speed real-time performance while maintaining high spatial resolution.

  On the other hand, the light pattern irradiation unit 3 includes a light spatial modulation element 10, a light source 12, and a light pattern generation unit 11.

  The light pattern generation unit 11 generates various light pattern signals based on the image feature quantity ξ fed back from the image feature extraction unit 2, and outputs the light pattern signal to the light spatial modulation element 10.

  The spatial light modulator 10 generates various light patterns by modulating the radiated light from the light source 12 based on the light pattern signal. The spatial light modulator 10 is roughly divided into a reflective spatial light modulator 10a and a transmissive spatial light modulator 10b. Examples of the reflective spatial light modulator 10a include a digital micromirror device (hereinafter referred to as DMD). Further, as the transmissive light spatial modulation element 10b, for example, there is a transmissive liquid crystal light valve.

  The spatial light modulation element 10 of either the reflection type or the transmission type has a large number of pixels on the reflection surface or transmission surface, and the intensity of the reflection light or transmission light can be controlled for each pixel. Also, any of the spatial light modulators 10 can change the light intensity control for each pixel at a high speed in terms of time. In this sense, the spatial light modulator 10 can also be called a spatiotemporal modulator.

  An operation of generating a three-dimensional image of the visual information processing system 1 configured as described above will be described.

  FIG. 2 is a first diagram illustrating the operation concept of the three-dimensional image generation of the visual information processing system 1. First, the emitted light from the light source 12 is spatially modulated by the light spatial modulation element 10 to generate a predetermined light pattern, and the imaging object 100 is irradiated with this light pattern via the optical system.

  When the light spatial modulation element 10 is a reflection type, the light source 12 may be disposed on the front side of the light spatial modulation element 10, and when the light spatial modulation element 10 is a transmission type, the light source 12 may be disposed on the rear side of the light spatial modulation element 10. .

  The light pattern illustrated in FIG. 2 shows a state in which the imaging object 100 is irradiated with a vertical stripe pattern composed of a plurality of vertical stripes.

  An image from the imaging object 100 is incident on the imager 20 via the optical system. At this time, an image in which the imaging object 100 and the light pattern are superimposed is incident as incident light. Image data captured by the imager 20 is input to the calculation unit 30 of the image feature extraction unit 2.

  The computing unit 30 generates a three-dimensional image from a plurality of image data on which light patterns are superimposed. At this time, the image data extracted from the imager 20 may be the entire pixel area of the imager 20 or image data obtained by extracting a partial area of the imager 20 as described above. When extracting image data of a partial area, it is possible to perform processing at a higher speed (high frame rate) than when extracting image data of all pixel areas.

  In the calculation unit 30, an image feature amount ξ is further extracted. The image feature amount ξ is, for example, the position (center of gravity position), size, posture angle, etc. of the imaging object. In addition, these are the amount of change (speed).

  By feeding back the image feature quantity ξ and adaptively controlling the type and number of light patterns, a three-dimensional image can be generated at a higher speed and with higher accuracy than the conventional three-dimensional image generation method.

  FIG. 3 is a diagram for explaining a conventional method for generating a three-dimensional image. The details are described in, for example, Non-Patent Document 1 and the like, so only a brief description is given here.

  First, in the example of FIG. 3, k types and k different light patterns 300 are prepared as one set, and these light patterns are sequentially irradiated onto the imaging object 100. The imager 20 acquires k pieces of image data in which the light pattern and the imaging object are superimposed. One k-dimensional image (not shown) is obtained by processing the k pieces of image data.

  In the case of continuous imaging, the above process is repeated. In the conventional method, the same k types and k light patterns are irradiated to the imaging target for each frame.

  The types and number of light patterns depend on the size and resolution (spatial resolution) of the imaging object. For example, the set of light patterns shown in FIG. 3 is a striped pattern having the two most vertical stripes, which is the coarsest interval (this is called the coarsest interval), and the most dense interval (this is called the closest interval). In this pattern, the vertical stripes are successively halved.

  In this case, the coarsest interval needs to substantially match the size of the object to be imaged. On the other hand, the closest interval substantially matches the spatial resolution. Therefore, when imaging an imaging object of the same size, if a high spatial resolution is to be obtained, a light pattern with a small close interval is required, resulting in an increase in the types and number of light patterns. Even in the case of the same resolution, the larger the imaging target or the wider the imaging target range, the larger the coarsest interval needs to be increased. In this case also, the types and number of light patterns increase. become.

  In the conventional 3D image generation method, a uniform light pattern is generated. Therefore, when the information of the imaging target is unknown, or when the imaging target is changed, it is optimal. It was difficult to obtain an optical pattern.

  In contrast, the visual information processing system 1 according to the present embodiment is configured to be able to extract the image feature quantity ξ as shown in FIG. It is possible to adaptively generate (adaptive control) a light pattern that optimizes the evaluation function.

  FIG. 4 schematically shows the concept of light pattern generation by the visual information processing system 1 according to the present embodiment. For example, the imager 20 generates a plurality of image data 301 based on the light pattern 300 of the Nth frame, and generates a three-dimensional image of the Nth frame based on this.

  Based on this three-dimensional image, the image feature quantity ξ is extracted by the image feature extraction unit 2. A light pattern that optimizes the evaluation function can be generated using the image feature amount ξ.

  For example, when the processing time is set as the evaluation function and the processing time is minimized, the type and number of light patterns may be minimized within a range in which a predetermined resolution is maintained. At this time, by feeding back the position and size of the imaging object as the image feature amount ξ, it becomes possible to reduce the coarsest interval of the light pattern from a wide range to the size range of the imaging object, As a result, the type and number of light patterns are reduced, and the processing time is shortened.

  Further, in the conventional three-dimensional image generation method, since there are many types and numbers of light patterns, it is difficult to generate an accurate three-dimensional image due to changes in the frame when the imaging target moves. On the other hand, in the visual information processing system 1 according to the present embodiment, as described above, the type and number of light patterns are reduced and the frame time is shortened. A three-dimensional image can be generated or three-dimensional information can be extracted.

  Furthermore, in the visual information processing system 1 according to the present embodiment, it is possible to detect the presence / absence of movement of the imaging target, the moving speed, etc. as the image feature amount ξ. Flexible processing is possible.

  FIG. 5 shows an example of processing for a dynamic imaging object. FIG. 5A shows a case where there is no movement in the entire image. In this case, even if the frame time is somewhat long, there is little error due to the movement of the object, so that it is possible to use a light pattern with high spatial resolution in which the closest interval is reduced.

  On the other hand, FIG. 5B shows a situation where the moving object 400 appears in a part of the imaging range. The image feature extraction unit 2 can extract the position, size, speed, and the like of the moving object 400 as the image feature amount ξ. By feeding back these image feature amounts ξ to the light pattern generation unit 3, it is possible to generate a light pattern having a different pattern only in the region 401 (region with motion) around the moving object 400. For example, the light pattern is changed to a light pattern in which the density of vertical stripes in an area with movement is coarser than that in the surrounding area.

  In the light pattern before the change, since the interval between the vertical stripes is too large compared to the movement amount, it is difficult to continuously generate a three-dimensional image of the moving object 400. On the other hand, in the light pattern after the change, the spatial resolution is slightly reduced by roughening the interval between the vertical stripes in the region 401 around the moving object 400, but three-dimensional measurement in continuous frames is possible. In addition, the high spatial resolution can be maintained for the peripheral area where there is no movement, as before the change.

  In generation of a light pattern using the light spatial modulation element 10, a light pattern having an arbitrary shape can be formed in principle on a two-dimensional pixel region. Therefore, it is possible to form not only a vertical stripe pattern that is uniform in the entire area as in the prior art but also a pattern of one or a plurality of partial areas in a form different from the surrounding pattern. It is also possible to follow the movement and move it.

(2) Second Embodiment FIG. 6 is a diagram showing a relationship between an optical spatial modulation element and an imager commonly used in visual information processing systems 1a to 1d according to second to fifth embodiments of the present invention. It is. The spatial light modulator in the first embodiment is used as a form for spatially modulating the emitted light from the light source. On the other hand, the spatial light modulators in the second to fifth embodiments are configured to spatially modulate incident light from the imaging object as shown in FIG. Also in this case, as in the first embodiment, two forms of a reflection type and a transmission type can be taken. FIG. 6A shows a form using a reflective spatial light modulator 10a, for example, DMD. FIG. 6B shows a form using a transmissive spatial light modulator 10b, for example, a transmissive liquid crystal light valve. In any form, incident light from the imaging object 100 is modulated (spatial modulation) for each pixel by the spatial light modulation elements 10a and 10b, and then output to the imager 20.

  In the following description, a transmissive spatial light modulator is used in the drawings, but this is for avoiding duplication of description, and does not exclude a reflective spatial light modulator.

  FIG. 7 is a diagram illustrating a configuration example of the visual information processing system 1a according to the second embodiment. The visual information processing system 1a according to the second embodiment is configured to be able to control the exposure time of incident light from the reflecting object 100 for each pixel.

  The visual information processing system 1a includes a light spatial modulation element 10, an image feature extraction unit 2a, and an exposure time control unit 15.

  The spatial light modulator 10 controls the exposure time for each pixel based on a control signal from the exposure time control unit 15, and outputs reflected light or transmitted light to the imager 20 of the image feature extraction unit 2a. In the case of the reflective spatial light modulator 10a, for example, DMD, the exposure time is controlled for each pixel by controlling the on time, that is, the time for which the reflection surface of the micromirror is directed toward the imager 20 for each pixel. can do. In the case of the transmission type spatial light modulator 10b, the exposure time can be equivalently controlled for each pixel by controlling the amount of transmission for each pixel by controlling the voltage applied to the pixel.

  The basic configuration of the image feature extraction unit 2a is the same as that of the image feature extraction unit 2 according to the first embodiment. However, in the second embodiment, it is assumed that the image feature amount ξ obtained by the calculation unit 30 includes luminance.

  FIG. 8 is a diagram showing an operation concept of exposure time control by the visual information processing system 1a. The imager 20 alone has a limit on the dynamic range with respect to brightness due to problems such as element accuracy, and it is generally difficult to capture an object with extremely high luminance and an object with very low luminance simultaneously with a clear image. . Although it is possible in principle to provide a sensitivity control circuit or the like for each pixel of the imager 20, the number of pixels and the pixel area are reduced due to mounting restrictions of the sensitivity control circuit and the like. The performance of the imager is sacrificed.

  On the other hand, in the visual information processing system 1a according to the present embodiment, the exposure time is controlled by the light spatial modulation element 10, and reflected light or transmitted light whose exposure time is controlled is output to the conventional imager 20. It is. The brightness of the image data captured by the imager 20 is extracted as an image feature amount ξ for each pixel and fed back to the light spatial modulation element 10 via the exposure time control unit 15. If the brightness of a specific area of the image data captured by the imager 20 is too high, the exposure time of the pixels in that area is decreased. Conversely, if the brightness of another specific area is too low, the image exposure in that area is exposed. Increase time. As described above, by adaptively controlling the luminance of the image observed by the imager 20 in units of pixels, it is possible to realize a substantially wide dynamic range.

  In this embodiment, since the imager 20 itself can be used as it is, the resolution and sensitivity inherent in the imager 20 are not sacrificed.

  As a modification of the visual information processing system 1a according to the second embodiment, a visual information processing system 1a having an automatic spatial sensitivity correction function can be used.

  In general, the imager 20 does not necessarily have completely uniform sensitivity over the entire range of the pixel region, and there is a spatial difference in sensitivity. Similarly, color filter arrays such as RGB and CMY used for color imaging also have spatial sensitivity (transmittance) non-uniformity.

  In the visual information processing system 1a according to this embodiment, since the exposure time for each pixel can be controlled by the light spatial modulation element 10, the spatial nonuniformity of the imager 20 and the color filter array can be easily corrected. can do.

  Further, as another modification example of the visual information processing system 1a according to the second embodiment, a form for correcting luminance non-uniformity due to a deviation in the position of illumination light, which occurs in the case of indoor imaging or the like, is adopted. You can also. When the luminance gradually decreases from a region close to the illumination light toward a region far from the illumination light, it is necessary to correct the luminance to be uniform especially when generating a binary image. In the visual information processing system 1a according to this embodiment, the luminance information of the imager 20 is monitored, and the exposure time of the light spatial modulation element 10 is adaptively controlled, so that the entire pixel has a uniform luminance regardless of the position of the illumination light. Image data can be generated.

(3) Third Embodiment FIG. 9 is a diagram illustrating a configuration example of a visual information processing system 1b according to a third embodiment. The visual information processing system 1b according to the third embodiment is configured to perform processing for improving the resolution of the imager 20 by combining the imager 20 and the spatial light modulator 10.

  The visual information processing system 1b includes a light spatial modulation element 10 and an image feature extraction unit 2b.

  The spatial light modulator 10 according to the present embodiment sequentially switches pixels that reflect or transmit incident light within a predetermined group to the imager 20 based on a control signal from the pixel switching unit 16 of the image feature extraction unit 2b. Output.

  The basic configuration of the image feature extraction unit 2b is the same as that of the image feature extraction unit 2 according to the first embodiment. However, in the third embodiment, it is assumed that the image feature amount ξ obtained by the calculation unit 30 includes area information of a specific imaging target.

  FIG. 10 is a diagram illustrating an operation concept of high-resolution processing by the visual information processing system 1b.

  In the visual information processing system 1b according to the third embodiment, the resolution of the spatial light modulator 10 is configured to be higher than the resolution of the imager 20. In the example of FIG. 10, the spatial light modulator 10 has 16 × 16 (256) pixels, the imager 20 has 4 × 4 (16) pixels, and the spatial light modulator 10 has a resolution 16 times that of the imager 20. is doing.

  For example, when imaging an object installed in a dark place without illumination or the like, an imager 20 with higher sensitivity is required. In general, increasing the pixel area is the most effective for improving the sensitivity of the imager 20, and many high-sensitivity imagers have a larger pixel area than a normal imager. For this reason, in general, high-sensitivity imagers are limited in the number of pixels in a predetermined area, and the resolution is often lower than that of a normal imager.

  In the visual information processing system 1b according to the third embodiment, even such a high-sensitivity and low-resolution imager is combined with the light spatial modulation element 10 to achieve high resolution.

  The resolution of the imager 20 illustrated in FIG. 10 is 1/16 compared to the resolution of the spatial light modulator 10. That is, in this example, 16 pixels of the spatial light modulator 10 correspond to one pixel of the imager 20.

  First, the pixels of the spatial light modulation element 10 are grouped every 16 (4 × 4) elements. Then, only one pixel in the group is sequentially turned on. In the example of FIG. 10, the pixel is sequentially switched (scanned) and turned on from the upper left pixel of the (4 × 4) element group. For other groups, the pixels are turned on in the same switching sequence.

  With such a pixel switching method, the number of pixels corresponding to the number of groups is output from each group every time one pixel is switched. In the example of FIG. 10, one pixel from each of the 16 groups of the spatial light modulators 10 is turned on, and a total of 16 pixels are output to the imager 20. The imager 20 first generates one piece of low-resolution image data using these 16 pixels. In the spatial light modulator 10, the pixels in the group are sequentially switched on, and the imager 20 sequentially generates low-resolution image data. When switching of all the pixels in the group is completed, the imager 20 generates 16 low-resolution images.

  Next, by combining these 16 pieces of low resolution image data, one piece of high resolution image data is generated. Since each of the 16 pieces of low-resolution image data and the pixel positions in the group have a one-to-one correspondence, it is possible to generate one piece of high-resolution image data by combining the 16 pieces of low-resolution image data. . In this example, the resolution can be increased to 16 times the resolution of the imager 20.

  In the example of FIG. 10, a mode is shown in which 16 pixels in the group are turned on one by one. On the other hand, for example, 4 (2 × 2) pixels of 16 pixels may be turned on. In this case, the imager 20 generates four pieces of low-resolution image data. By combining these four pieces, it is possible to generate image data with a resolution four times higher. Compared to the previous example, the resolution of the image data after synthesis is 1/4, but the number of image data to be synthesized is also 1/4, and the processing time is shortened.

  Thus, in the visual information processing system 1b according to this embodiment, the resolution of the imager 20 can be improved, and the resolution to be improved can be easily changed as described above.

  As shown in FIG. 9, in the visual information processing system 1b according to the present embodiment, the image feature extraction unit 2b extracts a specific region of the imaging target. Therefore, by feeding back this specific area, the above-described high resolution processing can be performed only for the range of the predetermined pixel area.

  For example, the area of the moving object is extracted as the image feature amount ξ by the image feature extraction unit 2b. Then, the high resolution processing is performed on the moving object region, and the other regions remain at low resolution. By such adaptive processing, it is possible to achieve a visual information processing system 1b that realizes high resolution for a moving object to be watched and suppresses a decrease in sensitivity and an increase in processing time as an entire image. It becomes.

(4) Fourth Embodiment FIG. 11 is a diagram illustrating a configuration example of a visual information processing system 1c according to a fourth embodiment.

The visual information processing system 1c according to the fourth embodiment combines the imager 20 and the spatial light modulation element 10 to extract a spectrum image obtained by extracting frequency components such as luminance fluctuations of the imaging target for each pixel and imaging the spectrum image. It is configured so that it can be generated.

  The visual information processing system 1c includes a light spatial modulation element 10 and an image feature extraction unit 2c.

  The spatial light modulation element 10 according to the present embodiment is turned on / off by a predetermined periodic signal for each pixel based on a control signal from the pixel on / off control unit 17 of the image feature extraction unit 2c, and reflects incident light. The light is transmitted and output to the imager 20.

  The basic configuration of the image feature extraction unit 2c is the same as that of the image feature extraction unit 2b according to the third embodiment. The image feature extraction unit 2c includes region information of a specific imaging target as an image feature amount ξ obtained by the calculation unit 30. It is supposed to be.

  FIG. 12 is a diagram showing an operation concept of spectrum image generation processing by the visual information processing system 1c.

  In the fourth embodiment, the spatial light modulator 10 and the imager 20 are basically configured with the same number of pixels.

  The spatial light modulator 10 is divided into a plurality of groups having a plurality of pixels. In the example of FIG. 12, grouping is performed every 4 (2 × 2) pixels.

  An on / off control signal having a different frequency is applied to each of the four pixels in the group. For example, a control signal for switching on / off at frequencies 1 to 4 is applied to the pixels 1 to 4, respectively.

  When the luminance of the imaging object fluctuates at a predetermined frequency, if this fluctuation frequency matches the frequency of the on / off control signal applied to the spatial light modulator 10, the imager 20 holds the luminance of the corresponding pixel. Is done. On the other hand, when the frequency of the luminance fluctuation of the imaging object and the frequency of the on / off control signal applied to the spatial light modulator 10 do not match, the imager 20 suppresses the luminance of the corresponding pixel. That is, since the light spatial modulation element 10 functions as a correlator and the integrator is configured by the capacitance component included in the imager 20, it is possible to extract the fluctuation frequency of the imaging object for each pixel.

  For example, when a fluorescent lamp is lit with a commercial power supply of 50 Hz in a complicated background, in the visual information processing system 1c according to the present embodiment, by matching one of the on / off control frequencies to 50 Hz, A spectrum image in which only the fluorescent lamp portion is emphasized can be generated.

  The on / off control frequency within each group can be arbitrarily set, and can be set as appropriate according to the physical condition of the imaging object and the frequency to be measured.

  For example, when detecting a crack or the like of a structure, when it is known in advance that the crack or the like fluctuates at a specific resonance frequency, the crack or the like is obtained by matching one of the on / off control frequencies with the resonance frequency. It is possible to generate a spectrum image corresponding to the position. As a result, even a minute crack or the like that is difficult to visually recognize can be visualized by extracting the resonance frequency and imaging it.

  Conversely, even when the resonance frequency of the observation object is unknown, if the frequency of a specific part in the image is different from the frequency in the surrounding range, an image (extracted from the specific part) (Spectrum image) can also be generated.

  For example, a printed wiring board is vibrated at a known frequency, and a part having a frequency component different from the vibration frequency is extracted as an image, so that it is possible to detect a soldering failure portion of a mounted component as image information. .

  Further, by feeding back specific area information and specific frequency components as the image feature amount ξ from the image feature extraction unit 2c, a spectrum image centered on the specific frequency components is obtained for a part of all the pixel areas. Can be generated. In this form, even if the region where the spectrum image is to be generated and the center frequency thereof are continuously changed, it is possible to track the change by adaptive control.

(5) Fifth Embodiment FIG. 13 is a diagram illustrating a configuration example of a visual information processing system 1d according to a fifth embodiment. The visual information processing system 1d according to the fifth embodiment is configured such that spatial filtering processing such as edge enhancement can be realized by simple arithmetic processing by combining the imager 20 and the spatial light modulator 10. .

  The visual information processing system 1d includes a light spatial modulation element 10 and an image feature extraction unit 2d.

  The spatial light modulator 10 according to the present embodiment is turned on / off for each pixel based on a control signal from the spatial pattern generation control unit 18 of the image feature extraction unit 2d, and reflects or transmits incident light to the imager 20. Output.

  The image feature extraction unit 2d has a basic configuration similar to that of the image feature extraction unit 2b according to the third embodiment, and includes region information of a specific imaging target as an image feature amount ξ obtained by the calculation unit 30. It is supposed to be.

  FIG. 14 is a diagram illustrating an operation concept of spatial filtering processing by the visual information processing system 1d.

  In the fifth embodiment, the number of pixels of the spatial light modulator 10 is configured to be larger than the number of pixels of the imager 20. In the example of FIG. 14, the number of pixels of the spatial light modulator 10 is configured to be four times the number of pixels of the imager 20.

  The spatial light modulator 10 is divided into a plurality of groups having a plurality of pixels. In the example of FIG. 14, grouping is performed every 4 (2 × 2) pixels.

  For the four pixels in the group, an on / off control signal is applied to the corresponding pixel based on a predetermined spatial pattern. For example, as illustrated in FIG. 14, in the spatial pattern 1, all four pixels in the group are turned on. In the space pattern 2, two pixels in the right half are turned on and two pixels in the left half are turned off. On the other hand, in the space pattern 3, the two pixels in the left half are turned on and the two pixels in the right half are turned off.

  In the visual information processing system 1d according to the present embodiment, edges in the X direction and the Y direction are obtained only by subtraction from the image data of the imager 20 obtained when the spatial light modulator 10 is set to the spatial patterns 1, 2, and 3. Spatial filtering processing for emphasis processing can be realized.

  Specifically, as shown in the equation (1) in FIG. 14, the image of the imager 20 obtained in the case of the space pattern 2, I (x, y, t + Δt), is obtained in the case of the space pattern 1. By subtracting 1/2 of I (x, y, t) of the image of the imager 20 to be generated, image data in which the edge in the X direction is emphasized can be generated. Strictly speaking, the image data obtained from the spatial pattern 1 and the image data obtained from the spatial pattern 2 are different in time, but by switching the pixels of the spatial light modulator 10 at high speed, the equation (1) in FIG. The arithmetic expression on the right side can be approximated on the left side.

  Similarly, as shown in Expression (2) of FIG. 14, it is obtained for the spatial pattern 1 from the image of the imager 20, I (x, y, t + 2Δt) obtained for the spatial pattern 3. By subtracting 1/2 of I (x, y, t) of the image of the imager 20, image data in which the edge in the Y direction is emphasized can be generated.

  According to the visual information processing system 1d according to the fifth embodiment, a plurality of spatial patterns are formed using the light spatial modulation element 10, and the image data of the imager 20 obtained according to each spatial pattern is subtracted. The spatial filtering process can be performed by a simple operation such as the above.

  Also, as shown in FIG. 13, it is possible to perform spatial filtering processing only on a specific region by feeding back an image feature amount ξ indicating a predetermined region. Furthermore, different types of spatial filter processing can be applied to a plurality of different regions.

  In addition, a spatial matching process or the like can be performed by generating an image that matches a wider area, such as a template image, as a spatial pattern by the light spatial modulation element 10. Also, not only an image prepared in advance but also a modulation pattern based on image information of the previous frame is prepared, so that processing such as block matching can be performed.

  According to the visual information processing system and the visual information processing according to each of the above-described embodiments, a new visual information processing that has not been obtained in the past can be performed by combining a spatial light modulation element and an imager (solid-state imaging element). Can be realized.

  In particular, by extracting image features from image data captured by an imager and providing a spatial light modulation element in the feedback loop based on these image features, new visual information processing adapted to the movement and change of the object Can be realized.

  Note that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined.

The figure which shows the system configuration example of 1st Embodiment of the visual information processing system which concerns on this invention. The 1st figure explaining operation | movement of the three-dimensional image generation in 1st Embodiment of the visual information processing system which concerns on this invention. FIG. 7 is a second diagram illustrating the operation of generating a three-dimensional image in the first embodiment of the visual information processing system according to the present invention. The 3rd figure explaining operation | movement of the three-dimensional image generation in 1st Embodiment of the visual information processing system which concerns on this invention. FIG. 9 is a fourth diagram illustrating the operation of generating a three-dimensional image in the first embodiment of the visual information processing system according to the present invention. The figure which shows the relationship between the light spatial modulation element and imager in 2nd thru | or 5th embodiment of the visual information processing system which concerns on this invention. The figure which shows the system configuration example of 2nd Embodiment of the visual information processing system which concerns on this invention. The figure explaining the operation | movement concept of exposure time control in 2nd Embodiment of the visual information processing system which concerns on this invention. The figure which shows the system configuration example of 3rd Embodiment of the visual information processing system which concerns on this invention. The figure explaining the operation | movement concept of the high resolution process in 3rd Embodiment of the visual information processing system which concerns on this invention. The figure which shows the system configuration example of 4th Embodiment of the visual information processing system which concerns on this invention. The figure explaining the operation | movement concept of the spectrum image generation process in 4th Embodiment of the visual information processing system which concerns on this invention. The figure which shows the system configuration example of 5th Embodiment of the visual information processing system which concerns on this invention. The figure explaining the operation | movement concept of the spectrum image generation process in 5th Embodiment of the visual information processing system which concerns on this invention. The figure which shows the structural example of the conventional imaging camera apparatus which performs extraction of an image feature-value, and a feedback process.

Explanation of symbols

1, 1a, 1b, 1c, 1d Visual information processing system 2, 2a, 2b, 2c, 2d Image feature extraction unit 3 Light pattern irradiation unit 10 Light spatial modulation element 10a Light spatial modulation element (reflection type)
10b Light spatial modulation element (transmission type)
DESCRIPTION OF SYMBOLS 11 Light pattern generation part 12 Light source 15 Exposure time control part 16 Pixel switching part 17 Pixel on / off control part 18 Spatial pattern generation part 20 Imager 30 Calculation part 30
40 Coordinate converter 50 High-resolution processor 60 Spectrum image generator 70 Spatial filtering processor 100 Imaging object ξ Image feature ρ Coordinate transformation parameter

Claims (14)

A light pattern generation unit that irradiates the imaging object with the radiation light modulated based on the light pattern, and having a light spatial modulation element that modulates the light emitted from the light source based on a predetermined light pattern;
An image feature extraction unit that images the imaging object on which the light pattern is superimposed and extracts a feature amount of the imaging object;
With
The light pattern is a plurality of light patterns capable of obtaining three-dimensional information of the imaging object based on the triangulation principle, and the type of the light pattern based on the feature amount extracted by the image feature extraction unit. And at least one of the numbers is a light pattern that is adaptively controlled. A spatial light modulation element that has a predetermined pixel region and controls the exposure time of incident light from the imaging target for each pixel;
An image feature extraction unit that captures an image of the imaging object with reflected or transmitted light from the spatial light modulation element in which an exposure time of incident light is controlled for each pixel, and extracts a feature amount of the imaging object;
With
The feature amount extracted by the image feature extraction unit is a feature amount including luminance for each pixel of the imaging target, and the spatial light modulator adaptively controls an exposure time based on the luminance for each pixel. A visual information processing system characterized by this. A spatial light modulation element that groups pixels in a designated region into a plurality of pixels, sequentially switches a plurality of pixel signals in each group, and reflects or transmits incident light from an imaging target; and
An image feature extraction unit that captures incident light from the imaging target object output from the light spatial modulation element with an imager having a resolution of the same number of pixels as the number of groups, and extracts a feature amount of the imaging target object; ,
With
The image feature extraction unit designates a region based on the feature amount, sequentially switches the region, combines the signals of the plurality of pixels input to the imager, and changes the resolution of the imager to the spatial light modulation. A visual information processing system characterized by increasing to the same resolution as the element. A spatial light modulation element that groups pixels in a specified area into a plurality of pixels, and turns on / off a plurality of pixel signals in each group at different frequencies to reflect or output incident light from an imaging object. When,
An image feature extraction unit for capturing incident light from the imaging object output from the light spatial modulation element with an imager having the same resolution as the resolution of the light spatial modulation element, and extracting a feature quantity of the imaging object; ,
With
The image feature extraction unit designates a region based on the feature amount, and in that region, the luminance variation of the imaging object is frequency-resolved corresponding to the different frequencies, and spectrum images corresponding to the different frequencies are obtained. A visual information processing system characterized by generating. Light that is output by reflecting or transmitting incident light from an imaging object by grouping pixels in a specified area into a plurality of pixels and sequentially switching a plurality of pixel signals in each group based on a plurality of predetermined spatial patterns. A spatial modulation element;
An image feature extraction unit that captures incident light from the imaging object output from the light spatial modulation element with an imager having the same number of pixels as the number of groups, and extracts a feature amount of the imaging object;
With
The image feature extraction unit designates a region based on the feature amount, calculates a plurality of signals sequentially output from the imager based on the plurality of spatial patterns in the region, and a plurality of pixels in the group A visual information processing system characterized by performing spatial filtering for processing. The image feature extraction unit
An imager unit having a pixel area of a predetermined size and outputting image data of a partial area of the pixel area for each frame according to pixel position information;
The image data of the partial area is input, an image feature amount is calculated based on the input image data and a previously calculated coordinate conversion parameter, and the calculated image feature amount and the coordinate conversion parameter are A calculation unit for generating a coordinate conversion parameter for the next frame based on the
A coordinate conversion unit that converts the pixel position of the partial area based on the coordinate conversion parameters for the next frame, and generates pixel position information of the partial area of the next frame;
The visual information processing system according to claim 1, comprising: a visual information processing system. The visual information processing system according to claim 1, wherein the light spatial modulation element is a digital micromirror device (DMD) or a liquid crystal light valve. Modulating light emitted from a light source by a light spatial modulation element based on a predetermined light pattern, and irradiating the object to be imaged with the emitted light modulated based on the light pattern;
Imaging the imaging object on which the light pattern is superimposed, and extracting a feature quantity of the imaging object;
With
The light pattern is a plurality of light patterns capable of obtaining three-dimensional information of the imaging object based on the triangulation principle, and the type of the light pattern based on the feature amount extracted by the image feature extraction unit. And a visual information processing method characterized in that at least one of the numbers is a light pattern that is adaptively controlled. Controlling the exposure time of incident light from the imaging target for each pixel by means of a spatial light modulator having a predetermined pixel area;
Imaging the imaging object with reflected or transmitted light from the light spatial modulation element whose exposure time is controlled for each pixel, and extracting a feature amount of the imaging object;
With
The extracted feature amount is a feature amount including luminance for each pixel of the imaging object, and the step of controlling for each pixel adaptively controls an exposure time based on the luminance for each pixel. Visual information processing method. A step of sequentially switching a plurality of pixel signals in the group and reflecting or transmitting incident light from the imaging target by a light spatial modulation element in which pixels in a specified region are grouped into a plurality of groups;
Capturing incident light from the imaging object output from the light spatial modulation element with an imager having a resolution of the same number of pixels as the number of groups, and extracting a feature amount of the imaging object;
With
The step of extracting the feature amount specifies a region based on the feature amount, and sequentially switches the region to synthesize signals of the plurality of pixels input to the imager, and sets the resolution of the imager to the light A visual information processing method comprising a process for increasing the resolution to the same as that of a spatial modulation element. A spatial light modulation element in which pixels in a specified area are grouped into a plurality of pixels, and the plurality of pixel signals in the group are turned on / off at different frequencies to reflect or transmit incident light from the object to be imaged. Output step;
Capturing incident light from the imaging object output from the spatial light modulation element with an imager having the same resolution as the resolution of the spatial light modulation element, and extracting a feature amount of the imaging object;
With
The step of extracting the feature amount designates a region based on the feature amount, and in the region, the luminance variation of the imaging object is frequency-resolved corresponding to the different frequency, and the spectrum corresponding to the different frequency. A visual information processing method comprising a process of generating an image. A plurality of pixel signals in the group are sequentially switched based on a predetermined plurality of spatial patterns to reflect incident light from the object to be imaged or reflected by a spatial light modulation element in which a plurality of pixels in a specified region are grouped. A step of transmitting and outputting;
Capturing incident light from the imaging object output from the light spatial modulation element with an imager having the same number of pixels as the number of groups, and extracting a feature amount of the imaging object;
With
The step of extracting the feature amount designates a region based on the feature amount, calculates a plurality of signals sequentially output from the imager based on the plurality of spatial patterns in the region, and extracts a plurality of features in the group The visual information processing system characterized by including the process which performs the spatial filtering which makes this pixel the process object. The step of extracting the feature amount includes:
Outputting image data of a partial region of the pixel region for each frame from an imager having a pixel region of a predetermined size according to pixel position information;
The image data of the partial area is input, an image feature amount is calculated based on the input image data and a previously calculated coordinate conversion parameter, and the calculated image feature amount and the coordinate conversion parameter are Generating a coordinate transformation parameter for the next frame based on,
Transforming the pixel position of the partial area based on the coordinate conversion parameters for the next frame to generate pixel position information of the partial area of the next frame;
The visual information processing method according to claim 8, further comprising: 13. The visual information processing method according to claim 8, wherein the light spatial modulation element is a digital micromirror device (DMD) or a liquid crystal light valve.

Images