JP2002008031A - Pattern detection apparatus and method, image processing apparatus and method - Google Patents

Abstract

(57) [Problem] To detect a predetermined pattern from an input pattern while stably and efficiently setting and changing a gaze position and a gaze area. An input layer for inputting a pattern to a pattern detection device for detecting a predetermined pattern from an input pattern.
101, corresponding to each point obtained by sampling the input pattern by a predetermined method, a feature detection layer 102 provided with a plurality of feature detection modules 106 for detecting a plurality of features in a plurality of layers, respectively. And a gaze area setting control layer 108 for controlling the setting of the gaze area 109 regarding the output of the lower layer based on the distribution of the feedback signal from the upper layer.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a pattern detecting apparatus and method for performing pattern recognition, specific object detection, and the like, and an image processing apparatus and method.

[0002]

2. Description of the Related Art Conventionally, in the field of image recognition and voice recognition, a type in which a recognition processing algorithm specialized for a specific recognition target is sequentially calculated and executed as computer software, or a dedicated parallel image processor (SIM) is used.
D, MIMD machine, etc.).

In the image recognition algorithm, it is considered important to reduce the calculation cost (load) required for the recognition process by performing the recognition while selectively shifting the gaze area, etc., based on an analogy with the biological processing. Have been.

For example, in a hierarchical information processing method according to Japanese Patent Publication No. 6-34236, a hierarchical information processing method includes a feature extraction element layer and a feature integration layer that performs an output according to an output from a feature extraction element corresponding to the same feature. Between the lower layers, a plurality of downward signal paths from the upper layer to the lower layer are provided corresponding to a plurality of upward signal paths from the lower layer to the upper layer, and according to the downward signal from the uppermost layer, The processing area (gaze area for recognition) is set by performing the segmentation by controlling the transmission of the upward signal and selectively extracting the specific pattern and the self-associative associative ability.

In US Pat. No. 4,876,731, the upward signal path and the downward signal path as described above are connected to an output layer (uppermost layer).
Is controlled based on contextual information (rule database, probabilistic weighting processing) from.

In Japanese Patent No. 2856702, correction of recognition is called gaze, and a gaze degree judging section for selecting a gaze degree for each partial region when a pattern cannot be specified for accurate recognition, and a gaze degree determination section for selecting the gaze degree. And a gaze degree control unit for performing recognition in consideration of the gaze degree.

[0007] A method for controlling the setting of a gaze area by selective routing proposed by Koch and Ullman (1985) (Human N
eurobiology, vol. 4, pp. 219-227) is a saliency map extraction mechanism based on feature extraction and a selective mapping mechanism, a so-called Winner-take-all neural network (Japanese Patent Laid-Open No. 5-242069). No., US5049758, US5059
The gaze control is performed by using a gaze position selection mechanism according to Japanese Patent Application Laid-Open No. 814 and US Pat. No. 5,146,106, and a mechanism for suppressing neural elements at the selected position.

A method of obtaining and expressing a scale and a position invariant of a center of an object by a dynamic routing network (And
erson, et al. 1995, Routing Networks in Visual Cor
tex, in Handbook of Brain Theory and Neural Network
s (M. Arbib, Ed.), MIT Press, pp. 823-826, Olhausen
et al. 1995, A Multiscale Dynamic Routing Circuit
for Forming Size- and Position-Invariant Object R
epresentations, J. Computational Neuroscience, vol.
2 pp.45-62.), By routing information through control neurons that have the function of dynamically setting connection weights, control of the gaze area and observer-centered features of the recognition target Performs conversion from expression to object-centric expression.

The method by selective tuning (Culhane &
Tsotsos, 1992, An Attentional Prototype for Early
Vision. Proceedings of Second European Conference
onComputer Vision, (G.Sadini Ed.), Springer-Verla
g, pp. 551-560.), the overall winner position in the uppermost layer is searched by the mechanism that activates only the winner from the WTA circuit in the uppermost layer to the lower layer WTA circuit. Is determined in the lowest layer (the layer that directly receives input data). Selection of position and feature selection in gaze control
This is achieved by suppressing the coupling irrespective of the position of the object and suppressing elements that detect features not related to the object.

[0010]

However, the method of performing the selective gaze as described above has the following problems.

First, the method according to Japanese Patent Publication No. 6-34236 presupposes that there is a downward signal path paired with an upward signal path in a hierarchical neural circuit configuration. A neural circuit having the same size as the neural circuit corresponding to the upward signal path is required as a circuit forming the downward signal path, and there has been a problem that the circuit scale becomes very large.

Further, in this method, there is no mechanism for sequentially changing and controlling the gaze position, so that there is a problem in that the operation at the time of setting and changing the gaze area lacks stability due to the influence of noise or the like. That is, there is an interaction between the elements of the intermediate layer of the upward path and the elements of the intermediate layer of the downward path over all layers, and the gaze position is ultimately determined through the whole of the interactions. When there are a plurality of objects belonging to the same category, the gazing point position is not controlled so as to make a stable sequential transition between the objects,
There is a problem that the gaze position vibrates only between specific objects (or in the vicinity thereof).

Furthermore, when a plurality of objects of the same category as the detection (recognition) object exist in the input data, processing is performed on the plurality of objects simultaneously (processing that is not substantially watched).
Occurs or in order to update the gaze position appropriately,
Each time, there is a problem that it is necessary to finely adjust the network parameters.

In the selective routing method, since the gaze position is controlled only by a mechanism for suppressing the selected area, efficient control of the gaze position is not easy, and the gaze position is not easily controlled. Control may be biased toward a specific target or a specific site.

Further, in the method based on the dynamic routing network, the interlayer connection is reconfigured via many control neurons capable of dynamically changing the synaptic connection weight, so that the circuit configuration becomes complicated, Since the action is a bottom-up processing process based on the feature saliency of the lower layer, it has been difficult to efficiently control the gaze position when a plurality of the same category objects exist.

Further, in the method based on selective tuning, a selection similar to a so-called pruning in which a connection which is not involved in a selected object is simply removed is performed hierarchically and dynamically. There is a problem that it is not easy to efficiently control the gazing point position.

Further, there are the following problems common to the above-mentioned conventional systems.

First, since there is no mechanism capable of coping with the difference in the size of the recognition target, when there are objects of different sizes at a plurality of positions at the same time, or for a plurality of scenes with different sizes of the recognition target, It was necessary to tune network parameters.

Further, when a plurality of objects belonging to the same category exist at a plurality of positions in the input data, the gaze position cannot be transitioned (updated) uniformly and efficiently between them.

[0020]

According to the present invention, there is provided, in accordance with the present invention, an input means for inputting a pattern to a pattern detecting device, and a sampling of the pattern input from the input means by a predetermined method. Corresponding to each point obtained by providing a plurality of feature detecting elements for detecting a plurality of features on a plurality of layers, respectively, detecting processing means for detecting a predetermined pattern, and from a higher layer of the plurality of layers, Gaze area setting control means for controlling the setting of a gaze area for lower layer outputs of the plurality of layers based on the distribution of the feedback signal.

According to another aspect of the present invention, an input means for inputting a pattern to the pattern detection device, and a pattern corresponding to each point obtained by sampling the pattern input from the input means by a predetermined method. A plurality of feature detection elements for detecting a plurality of features, a saliency detection element for detecting feature saliency, and coupling means for coupling between the elements and transmitting a signal. Forming a plurality of element layers related to the following features, comprising a detection processing means for detecting a predetermined pattern, and a gaze area setting control means,
The coupling means includes feedback coupling means for transmitting a signal from an element layer relating to a higher-order characteristic to an element layer relating to a lower-order characteristic, and the gaze area setting control means includes the characteristic saliency and the feedback coupling. The setting of a gaze area for low-order feature data or input data is controlled based on the signal transmission amount obtained from the means.

According to another aspect of the present invention, an input means for inputting a pattern to a pattern detection device, a feature detection layer for extracting a plurality of features for each of a plurality of resolutions or scale levels, and the feature detection layer And at least one feature integration layer that integrates and outputs the output from
A plurality of elements having a hierarchical network structure having a plurality of layers, comprising: a detection processing unit for detecting a predetermined pattern; and a gaze area setting control unit, wherein the feature detection layer and the feature integration layer are connected to each other by a predetermined connection unit. Wherein the feature detection layer or the feature integration layer has a saliency detection element for detecting feature saliency, and the coupling means performs feedback for transmitting a signal from an upper layer to a lower layer of the hierarchical network structure. The apparatus further includes a coupling unit, and the gaze area setting control unit generates a control signal related to the gaze area based on the signal from the feedback coupling unit and the feature saliency.

According to another aspect of the present invention, an input means for inputting a pattern, a gaze area setting control means, each of which corresponds to a different resolution or scale level, and has a hierarchical structure. And a plurality of feature detecting elements for detecting a plurality of features, respectively, corresponding to each point obtained by sampling the input data by a predetermined method. And multiplex processing means for combining a plurality of feature detection element outputs of the processing means corresponding to different resolutions or scale levels, and coupling means for performing signal transmission between different elements, the coupling means, It has a feedback coupling means for transmitting a signal from an upper layer to a lower layer, and detects a predetermined pattern.

According to another aspect of the present invention, a storage means for storing model data of an object to be photographed in the image processing apparatus, and the model data satisfying a predetermined standard while sequentially updating the gaze area. The apparatus includes setting means for searching for and setting a gaze area, and determination means for determining a shooting condition based on the gaze area set by the gaze area setting means.

According to another aspect of the present invention, a gaze position detecting means for detecting a gaze position from a user's gaze in the image processing apparatus, and a gaze for searching and setting a gaze region based on the gaze position. The image processing apparatus includes an area setting unit and a determining unit that determines a photographing condition based on the gaze area set by the gaze area setting unit.

According to another aspect of the present invention, in a pattern detection method, an input step of inputting a pattern, and corresponding to each point obtained by sampling the input pattern by a predetermined method, A detection process step of performing a feature detection process of detecting a plurality of features in a hierarchical manner to detect a predetermined pattern, and a distribution of a feedback signal from an upper layer of the detection process step; A gaze area setting control step of controlling setting of a gaze area relating to lower layer output.

According to another aspect of the present invention, in the pattern detection method, an input step of inputting a pattern, a detection processing step of detecting a predetermined pattern from the input pattern, and a gaze area setting control step Wherein the detection processing step comprises detecting a plurality of features relating to features from low order to high order, corresponding to each point obtained by sampling the input pattern by a predetermined method. Number of feature detection steps, comprising a saliency detection step of detecting feature saliency, the gaze area setting control step, low-order feature data based on the feature saliency and the feedback signal about the higher-order features or The position and size of the gaze area related to the input data are set and controlled.

According to another aspect of the present invention, in the pattern detection method, an input step of inputting a pattern, a detection processing step of detecting a predetermined pattern from the input pattern, and a gaze area setting control step Wherein the detection processing step includes: a feature detection step of extracting a plurality of features for each of a plurality of resolutions or scale levels; and a feature integration step of integrating and outputting an output of the feature detection step by a predetermined method. Each having at least one hierarchical processing structure, wherein the feature detection step or the feature integration step includes a saliency detection step of detecting a feature saliency; It is characterized in that a control signal relating to the attention area is generated based on the high-order feedback signal and the feature saliency. To.

According to another aspect of the present invention, in the pattern detection method, an input step of inputting a pattern, a gaze area setting control step, and a predetermined pattern based on the input pattern are hierarchically configured. A detection processing step of performing detection, wherein the detection processing step detects a plurality of features corresponding to each point obtained by sampling the input data by a predetermined method. And a multiplexing step of a resolution or scale level for combining a plurality of outputs of the data processing step for different resolution or scale levels.

According to another aspect of the present invention, there is provided a method for searching for and setting a gaze area that meets a predetermined criterion with respect to model data of an object to be photographed while sequentially updating the gaze area in the image processing method. And a step of determining a photographing condition based on the gaze area set in the setting step.

According to another aspect of the present invention, a gaze position detecting step of detecting a gaze position from a user's line of sight, and a setting for searching and setting a gaze region based on the gaze position are provided in the image processing method. And a determining step of determining an imaging condition based on the gaze area set in the setting step.

[0032]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS <First Embodiment> An embodiment of the present invention will be described below in detail with reference to the drawings.

The overall configuration Overview FIG. 1 is a diagram illustrating an overall configuration diagram of a pattern detection and recognition apparatus of this embodiment. Here, the pattern information is What
Processing is performed by a route (upper part in the figure) and a Where path (lower part in the figure). What path is the recognition of objects or geometric features
The Where route deals mainly with information relating to (detection), and the Where route mainly deals with information relating to the position (arrangement) of an object or feature.

The What path is a so-called Convolutional network structure (LeCun, Y. and Bengio, Y., 1995, "Co
nvolutional Networks for Images Speech, and Time S
eries "in Handbook of Brain Theory and Neural Netw
orks (M. Arbib, Ed.), MIT Press, pp. 255-258). However, it differs from the conventional one in that the interlayer connection in the same path can locally form mutual connection (described later). The final output of the What path corresponds to a recognition result, that is, a recognized target category. The final output of the Where path indicates a location corresponding to the recognition result.

The data input layer 101 is a photoelectric conversion element such as a CMOS sensor or a CCD element for detecting and recognizing an image, and is a voice input sensor for detecting and recognizing voice. Also, the analysis result of the predetermined data analysis unit (for example, principal component analysis, vector quantization, etc.)
May be inputted. The data input layer 101 performs data input common to the two paths.

Although the details of the application to speech recognition in the present embodiment will not be described, the selective gaze referred to in the following description is, for example, a method of extracting only a specific band or a phoneme, or a technique such as independent component analysis. This corresponds to selectively processing only the “features” extracted by the above, or extracting and processing the voice (features) of a specific speaker depending on the context.

Hereinafter, the case of inputting an image will be described. The What path includes the feature detection layer 102 ((1, 0),
(1, 1), ..., (1, N)) and the feature integration layer 103 ((2, 0),
(2, 1),..., (2, N)) and a gaze area setting control layer 108.

The first feature detection layer (1,0) is Gabor wavel
By multi-resolution processing by et conversion or the like, local low-order features (which may include color component features in addition to geometric features) of the image pattern are transferred to each position on the entire screen (or a predetermined sampling point over the entire screen). At the same location at a plurality of scale levels or resolutions, the number of feature categories, and the types of feature amounts (for example, when a line segment in a predetermined direction is extracted as a geometric feature, the Of the receptive field according to the geometrical structure (slope of the line segment), and is composed of neuron elements that generate pulse trains according to the degree.

The attention area setting control layer 108 includes a feature saliency map from a feature integration layer (2,0) described later and a feedback connection from an upper layer, for example, a feature position detection layer (3, k), a Where path described later. And controls the setting and updating of the position and size of the gaze area based on the feedback signal. The details of the operation mechanism will be described later.

The feature detection layer (1, k) as a whole forms a processing channel at a plurality of resolutions (or scale levels). That is, in the case where the Gabor wavelet transform is performed in the feature detection layer (1, 0), as shown in FIG. 12, a feature having a Gabor filter kernel having the same scale level and different direction selectivity in the receptive field structure, as shown in FIG. The set of detection cells forms the same processing channel in the feature detection layer (1,0), and in the subsequent layer (1,1), the feature detection cells (higher-order) receive the output from the feature detection cells. ) Belongs to the same channel as the processing channel. In the subsequent layer (1, k) (where k> 1),
Similarly, feature detection cells receiving outputs from a plurality of feature-integrated cells forming the same channel in the (2, k-1) layer are configured to belong to the channel. In each processing channel, processing at the same scale level (or resolution) proceeds, and detection and recognition from low-order features to high-order features are performed by hierarchical parallel processing.

The feature integration layer (2,0) on the What path is
It has a predetermined receptive field structure (hereinafter, the receptive field means the coupling range with the output element of the immediately preceding layer, and the receptive field structure means the distribution of the coupling load), and is composed of neuron elements that generate a pulse train. The integration of outputs of a plurality of neuron elements in the same receptive field from the feature detection layer (1, 0) (operations such as subsampling by local averaging and the like and processing for combining processing results at different scale levels) are performed.

Each receptive field of a neuron in the feature integration layer has a common structure among neurons in the same layer. Each feature detection layer (1, 1), (1, 2), ..., (1, N)) and each feature integration layer ((2, 1), (2, 2), ..., (2, N)) Is
Each has a predetermined receptive field structure acquired by learning,
Like the above-described layers, the former ((1, 1),...) Detects a plurality of different features in each feature detection module, and the latter ((2, 1),. The integration of the detection results related to multiple features is performed.

However, the former feature detection layer is connected (wired) so as to receive the cell element output of the preceding feature integration layer belonging to the same channel. The feature integration layer performs two types of processing. The first sub-sampling is for averaging the output from a local region (local receptive field of the feature integrated layer neuron) from the feature detection cell population of the same feature category and the same scale level. In the second process, which is a process of combining processing results at different scale levels, a linear combination (or non-linear combination) of outputs of a plurality of feature detection cell populations in the same feature category and over a plurality of different scale levels is performed.

The Where path has a feature position detection layer ((3,0),..., (3, k)), and a predetermined (not necessarily all) feature detection layer on the What path. And the output of low-order, middle-order, and higher-order features is involved, and the Where path is feedback-coupled from a higher-order feature integration layer or feature detection layer ((1, N) → ( 3, k) → A watching area setting control layer 108) is formed.

That is, this feedback connection is performed by learning in advance to the gaze control neuron 1801 (FIG. 18) of the gaze area setting control layer 108 relating to a specific lower-order feature constituting the higher-order feature pattern detected in the upper layer. It shall be formed as a bond. This learning is based on self-organization in which connections are strengthened or formed (other connections are suppressed or eliminated) by simultaneous firing of upper layer feature detection neurons and lower layer feature detection neurons within a predetermined time width. It involves a chemical process.

The output of each feature position detection layer 107 holds a relatively high resolution component of the output of the corresponding feature detection layer 102 (in FIG. 1, each arrow from the feature detection layer to the feature position detection layer indicates this). Is shown schematically) or the feature integration layer output is temporarily stored to represent a saliency map for each feature category. Here, the saliency map of a certain layer indicates the spatial distribution of the detection levels (or existence probabilities on input data) of a plurality of feature categories (feature elements) having the same level of complexity.
The processing in the feature position detection layer of the Where route will be described later.

The mechanism for connecting the neurons between the layers is as follows.
As shown in FIG. 2A, a signal transmission unit 203 (wiring or delay line) corresponding to axons or dendrites of nerve cells,
And a synapse connection circuit S202.

In FIG. 2A, certain features are detected (integrated).
FIG. 4 shows the configuration of a coupling mechanism involved in an output (input as viewed from the cell) from a neuron group (n _i ) of a feature-integrated (detected) cell forming a receptive field for the cell. Portions indicated by bold lines as signal transmission units constitute a common bus line, on which pulse signals from a plurality of neurons are transmitted in time series. The same configuration is adopted when receiving an input from the output destination cell. In this case, the input signal and the output signal may be divided and processed on the time axis in the same configuration,
The two systems (dendritic side) and output (axon side) may be provided with the same configuration as in FIG. 2A for processing.

The synapse circuit S202 has an interlayer connection (a connection between a neuron on the feature detection layer 102 and a neuron on the feature integration layer 103, and each layer has a connection to a subsequent layer and a preceding layer. May be involved)
Some are involved in connections between neurons in the same layer. The latter is mainly used, if necessary, mainly for coupling a pacemaker neuron described later with a feature detection or feature integration neuron.

The so-called excitatory connection is caused by the synaptic circuit S20.
In 2, amplification of the pulse signal is performed, and the suppressive coupling is to provide attenuation in reverse. When transmitting information by a pulse signal, amplification and attenuation can be realized by any of amplitude modulation, pulse width modulation, phase modulation, and frequency modulation of the pulse signal.

In the present embodiment, the synapse coupling circuit S202 is mainly used as a pulse phase modulation element, and the amplification of the signal is substantially advancing the pulse arrival time as an amount unique to the feature, and the attenuation is substantially the same. It is converted as a long delay. That is, the synaptic connection gives the arrival position (phase) on the time axis peculiar to the feature of the output destination neuron as described later, and qualitatively, the excitatory connection gives the phase of the arrival pulse with respect to a certain reference phase. In the case of inhibitory coupling.

In FIG. 2A, each neuron element n
_j outputs a pulse signal (spike train) and uses a so-called integral-and-fire type neuron element as described later. Note that, as shown in FIG. 2C, a circuit block may be configured by integrating the synapse connection circuit and the neuron element.

Each of the feature position detection layers in the Where path shown in FIG. 1 receives the output of the What path feature detection layer and the like, retains the positional relationship on the data input layer, and stores each point on the coarsely sampled grid points. Therefore, only neurons corresponding to components useful for recognition (pre-registered from the pattern of the recognition category) out of the feature extraction results on the What path respond by filtering or the like.

For example, in the uppermost layer in the Where path, neurons corresponding to the category of the recognition target are arranged on a lattice, and the position where the corresponding target exists is expressed. Further, the neurons in the hidden layer in the Where path receive a top-down input from the upper layer (in the Where path) (or an input by a route of the upper path of the What path → the upper layer of the Where path → the intermediate layer of the Where path). Thus, the sensitivity adjustment or the like can be performed so as to respond only when a feature that can be arranged with the corresponding recognition target existing position as the center is detected.

When performing hierarchical feature detection in which the positional relationship (or positional information) between the detected features is retained along the Where path, the receptive field structure is locally (eg, elliptical).
If the configuration is such that the size gradually increases in the upper layer (or the size is larger than one pixel on the sensor surface from the middle layer to the upper layer and is constant), the characteristic elements (graphic elements, graphic elements, The positional relationship between the graphic patterns can be such that each feature element (graphic element) is detected in each layer while preserving the positional relationship on the sensor surface to some extent.

As another form of the Where path, the receptive field size becomes larger in the upper layer hierarchically, and the maximum value is output from the neurons corresponding to the detected symmetric category in the uppermost layer. May be a neural network configured to fire only. In such a system,
Information on the arrangement relationship (spatial phase) in the data input layer is also stored to some extent in the uppermost layer (and each intermediate layer).

As another network configuration, FIG.
In the network configuration shown in FIG. 7 (the same as FIG. 1 except for the receptive field size except for the receptive field size), on the contrary, the feature integration layer stores the spatial arrangement relation of the feature category up to the higher-order features, so that the receptive field is also stored in the upper layer. The size is kept below a certain level. Therefore, feedback coupling from the feature integration layer to the gaze area setting control layer 108 is performed without setting the feature position detection layer.

[0058] will be described neuron element then neurons constituting each layer. Each neuron element is an extended model based on the so-called integral-and-fire neuron, and fires when the result of spatiotemporal linear addition of the input signal (pulse train corresponding to the action potential) exceeds the threshold, and the pulse shape It is the same as a so-called integral-and-fire neuron in that it outputs a signal.

FIG. 2B shows an example of a basic configuration representing the operation principle of a pulse generation circuit (CMOS circuit) as a neuron element, and a known circuit (IEEE Trans. On Neural).
Networks Vol. 10, pp. 540). Here, it is configured to receive excitatory and inhibitory inputs as inputs.

The operation principle of the circuit will be described below.
Excitatory input side capacitor C₁And resistance R₁Circuit time constant
Is the capacitor C_TwoAnd resistance R_TwoSmaller than circuit time constant
In the steady state, the transistor T₁, T_Two, T_ThreeIs cut off
I have. Note that the resistor is actually a transistor that is an active load.
Data. C₁Increases the potential of C_TwoTiger than that of
Transistor T₁Above the threshold of₁Is active
And the transistor T _Two, T_ThreeActivate G
Lanista T_2,T_ThreeConstitutes a current mirror circuit, and FIG.
(B) is output by an output circuit (not shown).
Pasita C₁Output from the side. Capacitor C_TwoCharge accumulation
When the amount becomes maximum, the transistor T₁Is shut off and the
As a result transistor T_TwoAnd T_ThreeIs also cut off and positive
The feedback is configured to be zero.

[0061] a so-called capacitor C ₂ is the refractory period is discharged, as long as the potential of the C ₁ does not become larger than the threshold amount of T ₁ than the potential of the C _2, neurons do not respond. Capacitor C ₁ , C
_A periodic pulse is output by repeating the alternate charging and discharging of ₂ , and its frequency generally depends on the level of the excitatory input. However, depending on the presence of the refractory period, it is possible to limit the maximum value or to output a constant frequency.

The potential of the capacitor, that is, the charge storage amount, is temporally controlled by a reference voltage control circuit (time window weight function generation circuit). Reflecting this control characteristic is weighted addition to an input pulse within a later-described time window (see FIG. 7). This reference voltage control circuit generates a reference voltage signal (corresponding to a weighting function of FIG. 7B) based on an input timing from a pacemaker neuron described later (or a mutual coupling input with a neuron in a subsequent layer). .

In some cases, the input of suppressiveness is not necessarily required in the present embodiment. However, the divergence (saturation) of the output can be prevented by making the input from the pacemaker neuron to the feature detection layer neuron to be suppressive. Can be.

In general, the relationship between the sum of the input signals and the output level (pulse phase, pulse frequency, pulse width, etc.) changes depending on the sensitivity characteristics of the neuron.
The sensitivity characteristic can be changed by a top-down input from an upper layer. In the following, for convenience of explanation, it is assumed that the circuit parameters are set so that the frequency of the pulse output corresponding to the total value of the input signal rises sharply (thus, almost in binary in the frequency domain), and the output level is determined by pulse phase modulation. (Such as the timing at which phase modulation is applied).

As a pulse phase modulating means, a circuit as shown in FIG. 5 to be described later may be added and used.
As a result, the phase of the pulse output from the neuron changes as a result of the reference voltage being controlled by the weight function within the time window, and this phase can be used as the output level of the neuron.

The time τ _w1 corresponding to the maximum value of the weighting function as shown in FIG. 7B that gives the time integration characteristic (reception sensitivity characteristic) of the pulse subjected to the pulse phase modulation by the synaptic connection is generally The time is set earlier than the estimated arrival time τ _{s1 of the} pulse unique to the feature given by the synaptic connection. As a result, a pulse arriving earlier within a certain range from the estimated arrival time (in the example of FIG. 7B, a pulse that arrives too early is attenuated) is a pulse signal having a high output level in the neuron receiving it. Is integrated over time. The shape of the weight function is not limited to a symmetric shape such as Gaussian, but may be an asymmetric shape. It should be noted that the center of each weighting function in FIG. 7B is not the expected pulse arrival time for the above-described purpose.

The phase of the neuron output (before synapse) is based on the beginning of the time window as described later, and the delay (phase) from the reference time is the time when a reference pulse (by a pacemaker output or the like) is received. It has output characteristics determined by the amount of charge storage. The details of the circuit configuration for providing such output characteristics will not be described because they are not the focus of the present invention. The post-synaptic pulse phase is obtained by adding the pre-synaptic phase to the unique phase modulation amount given by the synapse.

When the sum of inputs obtained by using a window function or the like exceeds a threshold value, a known circuit configuration that outputs an oscillation output with a predetermined timing delay may be used.

The configuration of the neuron element is a neuron belonging to the feature detection layer 102 or the feature integration layer 103. If the firing pattern is controlled based on the output timing of a pacemaker neuron described later, a pulse from the pacemaker neuron is used. After receiving the output, the circuit configuration may be such that the neuron outputs a pulse with a phase delay according to the input level (simple or weighted sum of the above input) received from the receptive field of the preceding layer. . In this case, before the pulse signal is input from the pacemaker neuron, there is a transitional transition state in which the neurons output pulses at random phases with respect to each other according to the input level.

Further, when a pacemaker neuron is not used as described later, the above-mentioned synchronizing and firing signals generated by the mutual connection between neurons (between the feature detection layer and the feature integration layer) and network dynamics are used as a reference. The circuit configuration may be such that the firing timing of the output pulse of the feature detection neuron is controlled according to such an input level.

The neurons of the feature detection layer 102 have a receptive field structure according to the feature category as described above, and input pulse signals (current values) from the neurons of the preceding layer (the input layer 101 or the feature integration layer 103). Or a potential), a non-decreasing and non-linear function that asymptotically saturates to a certain level, such as a sigmoid function, according to the sum when the weighted sum (described later) of the time window function exceeds a threshold value. That is, pulse output is performed at an output level that takes a so-called squashing function value (in this case, the output is given by a phase change, but may be a change based on frequency, amplitude, and pulse width).

Processing in the feature detection layer (1,0) (Gabor wavelet
( Lower-order feature extraction by transformation, etc.) The feature detection layer (1, 0) has a pattern structure (lower-order feature) having a predetermined spatial frequency in a local area of a certain size and having a vertical directional component. Neuron N1 to detect features)
If a corresponding structure exists in the receptive field of the neuron N1 on the data input layer 101, a pulse is output at a phase corresponding to the contrast. Such a function can be realized by a Gabor filter. Less than,
The feature detection filter function performed by each neuron of the feature detection layer (1, 0) will be described.

In the feature detection layer (1, 0), multi-scale
It is assumed that Gabor wavelet transform represented by a filter set of multi-directional components is performed, and each neuron (or each group including a plurality of neurons) in a layer has a predetermined G value.
It has an abor filter function.

In the feature detection layer 102, a plurality of neuron groups each having a receptive field structure corresponding to a convolution operation kernel of a plurality of Gabor functions having a constant scale level (resolution) and different direction selectivity are grouped together. Form one channel. At this time, as shown in FIG. 13, neuron groups forming the same channel have different direction selectivity, and neuron groups having the same size selectivity may be arranged at positions close to each other, as shown in FIG. The neuron groups belonging to the same feature category and belonging to different processing channels may be arranged close to each other.

This is because the arrangement configuration shown in each of the above figures is easier to realize in terms of the circuit configuration, for the sake of the later-described combining process in collective coding. FIG. 12, 1
The details of the circuit configuration of No. 3 will also be described later.

The details of the method of performing the Gabor wavelet conversion in the neural network are described in the literature by Daugman (1988) (IEEE Trans. On Acoustics, Speech, and Signal Pro
cessing, vol. 36, pp. 1169-1179).

The Gabor wavelet is given by the following equation (1)
Has a shape obtained by modulating a sine wave having a constant directional component and a spatial frequency with a Gaussian function, as given by
It is specified by the index m of the scaling level and the index n of the direction component. As a wavelet, this set of filters has similar functional shapes and different main directions and sizes. This wavelet is that the function form is localized in the spatial frequency domain and the real space domain, the simultaneous uncertainty regarding the position and the spatial frequency is minimized, and it is the function that is most localized in the real space and the frequency space Known (J, G. Daugman (1985), Unce
rtainty relation for resolution in space, spatial
frequency, and orientation optimized by two-dimens
ional visual cortical filters, Journal of Optical
Society of America A, vol.2, pp. 1160-1169).

[0078]

[Outside 1]

Here, (x, y) is a position in the image, a is a scaling factor, θ _n is a directional component of the filter, and W
Is the fundamental spatial frequency, σ _x and σ _y are the x directions of the filter function,
It is a parameter that gives the size of the spread in the y direction. In the present embodiment, θ _n is 0 degree, 30 degrees, 60 degrees, 9 in six directions.
Taking values of 0 degrees, 120 degrees, and 150 degrees, a is set to 2 and m is given as an integer taking a value of 1 to 3.

The parameters σ _x , which determine the characteristics of the filter,
σ _y and a are desirably set such that there is no bias (sensitivity) to a specific spatial frequency and direction by appropriately and homogeneously overlapping each other in the Fourier domain. Therefore, for example, if the half-value levels for the amplitude maximum value after the Fourier transform are designed to be in contact with each other in the Fourier domain,

[0081]

[Outside 2] Becomes Here, U _H and U _L are the maximum and minimum values of the spatial frequency band covered by the wavelet transform, and M gives the number of scaling levels in that range.

The structure of the receptive field of the feature detecting cell given by equation (1) has scale selectivity and direction selectivity of a predetermined width determined by σ _x and σ _y . That is, since the Fourier transform of the equation (1) has a Gaussian function shape, a peak tuning (sensitivity) characteristic is given to a specific spatial frequency and direction. Gabor filters with different scale indices have different size selectivities, because the size (spread) of the Gabor filter kernel changes according to the scale index m. In collective coding described later, outputs from a plurality of feature detection cells whose sensitivity characteristics mainly overlap with each other with respect to size selectivity are integrated.

Each of the filters g _mn (x, y) and the input grayscale image
The Gabor wavelet transform is performed by performing a dimensional convolution operation. That is,

[0084]

[Outside 3]

Here, I is an input image, and W _mn is a Gabor wavelet transform coefficient. A set of W _mn (m = 1, 2, 3; n = 1,..., 6) is obtained at each point as a feature vector. ' ^* ' ^Indicates complex conjugate.

Each neuron of the feature detection layer (1, 0) is represented by g _mn
Has a receptive field structure corresponding to G _mn having the same scale index m has the same size receptive field, and the corresponding kernel g _mn size has a size corresponding to the scale index in operation. here,
30 × 30, 1 on the input image in order from the coarsest scale
The size was 5 × 15, 7 × 7.

Each neuron outputs a non-linear squashing function of a wavelet transform coefficient value obtained by performing a product-sum input of a distribution weighting coefficient and image data (here, a phase reference is used; , A pulse width reference may be used). As a result,
As an output of the entire layer (1, 0), Gaborwa of Expression (4) is used.
This means that velet conversion has been performed.

Processing at the feature detection layer (medium order, higher order feature extraction
Out), while the subsequent feature detection layer ((1,1), (1,2), ...,
Unlike the feature detection layer (1,0), each neuron of (1, N) forms a receptive field structure for detecting a feature unique to the pattern to be recognized by a so-called Hebb learning rule or the like. The size of the local region where the feature detection is performed gradually becomes closer to the size of the entire recognition target in a later layer, and a medium-order or higher-order feature is geometrically detected.

For example, when performing face detection / recognition, the middle-order (or higher-order) features represent features at the level of a figure element such as an eye, a nose, and a mouth that constitute the face. If different processing channels have the same hierarchical level (the complexity of detected features is the same level), the detected features differ in the same category but are detected on different scales. It is in. For example, "eye" as a secondary feature
In different processing channels, detection is performed as "eyes" having different sizes. That is, detection is attempted in a plurality of processing channels with different scale level selectivity for a given size "eye" in the image.

Note that the feature detection layer neuron is generally
A mechanism may be provided to receive the coupling of the shunting inhibition (shunting inhibition) based on the output of the preceding layer in order to stabilize the output (independent of low-order and higher-order feature extraction). .

Selective Gaze Processing The structure and operation of the gaze area setting control layer 108 will be described below. In the initial state, the size of the gaze area set in the same layer is set to be the entire screen for the processing channel having the largest size among the plurality of previously set scale levels.

More specifically, as shown in FIG. 18, gaze control neurons 1801 corresponding to a plurality of predetermined positions of input data (positions at which feature detection is performed) are arranged in the gaze area control layer. In FIG. 18, an area indicated by an ellipse is a set point of gaze and a corresponding area on each layer (for convenience of explanation, only one is shown for each layer, but in general, a plurality of feature categories , There may be a plurality of corresponding regions in each layer; the size of the region is approximately the same as the size of the receptive field in each layer).

In FIG. 18, a thick arrow pointing from the feature detection layer to the feature position detection layer schematically indicates that each neuron of the feature position detection layer receives an output of the corresponding feature detection layer neuron. Each feature position detection layer neuron
In general, it has a receptive field smaller in size than the feature integration layer neuron, and performs sub-sampling in the same way as the feature integration layer to better preserve spatial arrangement information between feature categories than feature integration Can be.

Each gaze control neuron 1801 has a feedback connection from the Where path from the upper feature position detection layer (3, 2) or (3, 1), or in the case of FIG. , 2), is input via a feedback connection from the neuron for the position corresponding to the gaze control neuron. The gaze control neuron may receive an output from the feature integration layer (2, 0) in parallel. Linear addition by predetermined weighting of these two or three types of inputs
As a result of a gaze area selection control process (described later) based on an input from either (or non-linear addition) or one of them, the selected specific gaze control neuron is activated, and the gaze area is determined.

The position of the gaze area is mainly input by feedback inputting the position information of the object of the maximum level detected by the uppermost feature integration layer (2, N) or the uppermost feature position detection layer (3, M). Is set. In the former case (that is, the configuration in FIG. 17), only the case where the position information is held in the feature integration layer is limited. There are mainly the following three methods for selecting the gaze position. (1) The gaze control neuron with the largest linear sum (nonlinear sum) of the feedback amount from the upper layer (or the sum of the feedback amounts from a plurality of upper layers) and the output of the feature detection layer (the saliency of low-order features) Give the viewpoint position. (2) A gaze control neuron that receives a maximum (maximum) value of a low-order feature saliency map gives a gaze point position. (3) A gaze control neuron whose feedback amount from the upper layer (sum of a plurality of feedback paths in a specific gaze control neuron) is maximum (maximum) gives a gaze point position.

Here, the feedback amount is the output level of the upper layer feature position detection layer (or feature integration layer) neuron for a particular processing channel and a particular feature category.
(Or an amount proportional thereto). In the present invention, (1)
Or, it corresponds to the case of (3). The above (2) is a known method, and has the above-mentioned drawbacks. As described above, the gaze position is determined by the feedback amount from the upper layer making a certain contribution. As a result, the object recognition process includes a bottom-up process by signal processing from the lower layer to the upper layer and a lower-layer process from the upper layer to the lower layer. It starts only after the setting of the gaze area (activation of the gaze control neuron) brought about by the cooperative action between the upper layer neuron and the lower layer neuron by the top-down process to.

For this reason, in the initial process (the process until the upper-layer neuron outputs a signal in the bottom-up process first), a so-called “recognized / detected” state is not reached. A latent state in which a so-called saliency map relating to higher-order features is output), based on the result of a gaze control signal generated due to the involvement of a top-down process, detection of higher-order features in an upper layer through a bottom-up process again. At the time when the integrated neuron outputs (at this stage, neurons with a high output level are localized in the upper layer and sparsely present), a series of recognition /
The operation related to detection is completed. The upper layer is not necessarily limited to the (2, N) layer in FIG. 1 or FIG. 17 or the (3, k) layer in FIG. 1, but may be a lower intermediate layer or a higher layer (not shown). Good.

The number of gaze control neurons 1801 does not necessarily have to match the number of neurons on the feature integration layer (2,0) (that is, the number N _F0 of feature detection positions). It may be the same as the number of neurons (N _Fp ) in the position detection layer. As a precondition, when N _Fp <N _F0 , the number of sampling points for performing gaze control is smaller than the number of low-order feature detection positions, but the lower limit of the size of the detection target is νS ₀ (here, 0 Assuming that <ν <1, S ₀ is the entire area of the screen), practically, there is almost no problem unless N _Fp << νN _F0 . Note that the low-order feature saliency map received by the gaze control neuron may be an output from a feature detection layer (for example, a (1,0) layer).

On the other hand, the size of the gaze area set by updating the gaze position is such that the uppermost layer neuron that gives the maximum amount of feedback to the gaze control neuron 1801 belongs (or the feature integration layer corresponding to the gaze control neuron). The same applies to the channel to which the neuron belongs) The processing channel (in the case of (3) above), or the feature integration corresponding to the gaze control neuron in which the linear sum of the input from the upper layer and the low-order feature saliency is the maximum (maximum) It has a predetermined size determined by the processing channel to which the layer neuron belongs. In the present embodiment, the size of the gaze area is the kernel size of the Gabor filter belonging to the processing channel of the feature integration layer (2, 0).

As a specific configuration of the gaze area setting control layer, in addition to the gaze control neuron 1801,
Alternatively, in the channel processing structure shown in FIG. 13, a configuration like the gating circuit in FIG. 16 and a gaze position update control circuit 1901 shown in FIG.

This gating circuit is used for masking low-order features (propagating only data of a specific partial area to the upper layer), and the gating circuit is used for the feature integration layer.
It has a function of selectively extracting an area to be watched in the selected specific channel from the output from (2, 0) and transmitting the signal to the subsequent channel. In this way, by setting the low-order feature signal only from the specific channel and the specific area on the input data to propagate, the gaze area can be set.

As another configuration, a certain gaze control neuron is operated (for example, by a feedback signal from an upper layer).
By being selected and activated, a pulse for promoting firing is transmitted to the feature integration layer (2,0) neuron at the corresponding position via a synapse, and as a result, the feature integration layer neuron is turned ON (ignition enabled) State: A state in which an output of a pacemaker neuron or the like can be received, that is, a transitional transition state or a state in which the transition state is allowed) may be used.

The gaze area setting control layer 108 and the feature integration layer (2,
20) and 21 show examples of signal propagation paths between the feature detection layer (1, 1). 20 and 21, both are set together with a so-called digital circuit common bus, and have a gate circuit 2001 having a switch function and ON / OFF of the switch function.
A switch signal control circuit 2002 that controls OFF at high speed is used. In FIG. 20, the gaze control neuron 18
01 denotes a plurality of neurons in the feature integration layer (2,0), that is, neurons belonging to an area 109 centered on a neuron on the feature integration layer corresponding to the gaze position (hereinafter, referred to as a “gaze center neuron”). The region size is specific to the processing channel to which the gaze-centered neuron on the integration layer belongs) and makes connections with neurons within a predetermined size region specific to the processing channel.

In the configuration of FIG. 20, the gaze control neuron 18
01 has a connection from the feature integration layer to the feature integration layer, and the gaze position is controlled based only on the feedback amount from the feature position detection layer on the Where path. A configuration may be adopted in which the gaze position is controlled based on the feedback amount and the feature integration layer output). On the other hand, in the configuration of FIG. 21, the gaze control neuron 1801 has a similar connection to the gating layer 2101 corresponding to the gating circuit of FIG. 16, and changes from the feature integration layer (2,0) to the feature detection layer (1, The output to 1) is made through the gating layer 2101.

In the configuration shown in FIG. 20, an area of a specific size (generally a circular or rectangular area) is activated in the processing channel centered on the feature integration layer neuron at the set gaze position and corresponds to the area. The output of the feature integration layer (2, 0) propagates to the feature detection layer (1, 1). On the other hand, in the configuration shown in FIG. 21, only the gate of the region of a specific size is opened in the processing channel centered on the set gaze position in the gating layer, and the feature integration layer (2,0) corresponding to the region is opened.
Is propagated to the feature detection layer (1, 1).

This gating layer is composed of an FPGA or the like, and the entire logic circuit corresponding to each position of the gaze control neuron is composed of the feature integration layer (2,0) and the feature detection layer (1,2).
1), the gaze control neuron is activated, and only the logic circuit unit at the corresponding position in the gating layer is turned ON, and the signal is transmitted to the subsequent layer. Is transmitted to

The size of the gaze area to be set is determined according to the processing channel to which the feature integration layer neuron corresponding to the activated gaze control neuron belongs as described above. For example, if the corresponding feature integration layer neuron belongs to the processing channel 1 of FIG. 12 or 13, it will be the largest of the predetermined sizes.

Next, a mechanism for updating the gaze position will be described. In the case of the above-mentioned gaze control mechanism (3),
The gaze point position after the first setting is basically determined by the magnitude of the feedback amount received by each gaze control neuron from the upper layer in a region within a certain range of the latest gaze position (next to the maximum value of the feedback amount). Are updated in the order of). In the case of the above-mentioned gaze control mechanism (1),
Position in the order of the evaluation value G _f as shown below are updated.

G _f = η ₁ S _f + η ₂ S _{FB, f} (5)

Here, G _f is an evaluation value for the feature f, S _f is the saliency of the feature f from the lower feature integration layer, and S _{FB, f} is the detection _value for the top or middle feature detection layer. The feedback amounts to the gaze control neuron 1801 relating to the obtained feature f, η ₁ and η ₂ are positive constants. η ₁ and η ₂ are
The nature of the task of recognition or detection, for example, when performing detailed feature comparison such as identification between similar objects, or average (rough) detection of objects belonging to a specific category such as a human face
It can be changed adaptively, for example, when performing feature comparison.

Specifically, in the former problem, the ratio of η _{2 to} η ₁ is reduced, and conversely, in the latter case, it is increased. Such control of η ₁ and η ₂ is performed by changing the corresponding synaptic connection to the gaze control neuron.

FIG. 22 shows the gaze control neuron 1801 and the input distribution control circuit 22 for the gaze control neuron when controlling η ₁ and η ₂ associated with the synapse of the gaze control neuron.
An example of the configuration centered on 02 is shown.

Here, the input distribution control circuit 2202 performs the modulation of the synapse 2201 involved in the modulation of the detection level of the input of the saliency map signal of the lower-order feature from the lower layer and the modulation of the detection level of the feedback signal from the upper layer. Involved synapses 2201
Η ₁ according to a control signal from the outside (for example, a signal indicating a target detection mode or a detailed identification mode as a signal relating to a problem from a higher layer not shown in FIG. 1). , Η ₂ , and the like, for controlling the amount of phase delay at the synapse.

The size of the neighborhood area is set to be substantially the same as the size of the gaze area on the input data (for example, a value not exceeding 10 times the gaze area size). The reason for limiting the search to the vicinity area is to maintain the high speed of the processing. If the processing speed does not significantly deteriorate, the entire screen may be used as the search range.

Next, a gaze position update control circuit 1901 that performs sequential update control of the gaze position will be described. This circuit is
As shown in FIG. 19, the primary storage unit 19 of the feedback amount to the gaze control neuron 1801 corresponding to the latest gaze position
02, each feedback amount input unit 1903 in the neighboring area, detects the next (next candidate) feedback amount following the latest gaze position feedback amount in the neighboring area, detects and activates the gaze control neuron receiving the feedback And an update position determining unit 1904 to be performed.

The update position determining means 1904 is provided in the primary storage unit.
1902 and the input from the feedback input unit 1903 in the vicinity area, and compares the feedback amount in the latest gaze control neuron with the feedback amount in the vicinity area.
07, a secondary feedback signal amplifying unit 1906, which will be described later, and a next candidate feedback amount detecting unit 1905, and when the next candidate feedback amount detecting unit 1905 detects a feedback amount according to the latest feedback amount, the secondary feedback The signal amplifier 1906 outputs a pulse to the corresponding gaze control neuron.

According to the latest feedback amount, specifically, the feedback amount is randomly input in the vicinity area, and the two feedback amounts are compared by the above-described method. This indicates that the feedback amount to be compared is larger than the range or the latest feedback amount (possible if the search range is updated), and by detecting this, the next update position is effectively detected.

Next, transmission of signals between the gaze control neuron 1801 and the gaze position update control circuit 1901 in the structure schematically shown in FIGS. 20 and 21 will be described. As will be described later, the specific gaze control neuron 1801 and the gaze position update control circuit 1901 temporarily form a mutual connection, and the gaze control neuron corresponding to the update position selected by the method described above is updated from the update position determination unit 1904. (A feedback signal having a signal level proportional to the amount of feedback from the upper layer to the gaze control neuron) is activated (as a result, the gaze region specific to the neuron is opened, thereby Only a signal corresponding to the data in the corresponding area on the data is transmitted to the subsequent layer).

[0119] The above-mentioned temporary mutual coupling is defined as another specific area on the neighborhood of the gaze control neuron that is activated at a certain time (ie, the signal transmission control to the subsequent layer is performed). Is temporarily connected to a gaze control neuron (signal propagation) through a local common bus.
Specifically, among other gaze control neurons in the vicinity area, only one neuron (at a random position) within a certain time range (for example, on the order of tens of milliseconds) spontaneously fires and outputs, As a result of the spike pulse reaching the activated gaze control neuron through the common bus, a communication channel with the already activated gaze position update control circuit 1901 is established within the time range.

For this purpose, access to the common bus is obtained only when, for example, a neuron other than the latest gaze control neuron in the neighboring area is in a firing state. Specifically, a variable resistance array and its control circuit are used in which the pulse signal from the firing neuron propagates on the wiring connected to the common bus, thereby temporarily reducing the resistance of the wiring. 20 (not shown), or between each gaze control neuron and the common bus.
A gate circuit 2001 (switch circuit) and a switch signal control circuit 2002 schematically shown in FIG. 1 are set, and the switch signal control circuit 2002 is one of the gate circuits (FIG. 20) using a firing pulse signal from the gaze control neuron as a trigger. , 21 of 2001
In (c), a switch ON signal is sent to temporarily open the channel (connection to the common bus is in the ON state).

After the gaze position of the next candidate is selected, the secondary feedback signal is amplified by the amplifying means 84 so that the feedback signal from the upper layer to the gaze control neuron is amplified by a predetermined amount (in the case of phase modulation, the advance of the pulse phase, etc.). Is returned to the gaze control neuron.

Here, the number of gaze control neurons activated at the same time (therefore, the gaze area) is limited to one for each feature category. For this reason, when the gaze control neuron that has been activated at a certain time transitions from the activated gaze control neuron to a different gaze control neuron, the activity of the original activated neuron is assumed to be suppressed for a certain period of time. This is achieved by the gaze control neurons (nearest neighbors) forming inhibitory connections (dotted lines from gaze control neurons 1801 in FIGS. 20 and 21 represent inhibitory connections).

In the above, the range of the feedback amount related to the gaze control has a preset lower limit. After the control is shifted to the gaze position corresponding to the lower limit, the control returns to the first gaze position. Settings shall be made.

The above-described mechanism of directly controlling the gaze area by the gaze control layer using the feedback signal from the upper layer provides the following unconventional effects.

Compared to the case where the gaze area control is performed using only the saliency map relating to the low-order features, the search can be performed more efficiently (thus, faster). A meaningless visual search that continues to vibrate can be avoided.

In a target to be watched, only a characteristic portion or pattern (for example, eyes in the case of a face or the like) to be watched in the target can be selectively and efficiently searched. In this case, the feedback from the intermediate layer that is involved in the detection of a characteristic portion in the target may be used instead of using the feedback from the uppermost layer.

By searching for the next update position in the region near the latest gaze position, gaze position update control can be performed at high speed.

Even when there are a plurality of targets belonging to the same category, by appropriately setting the size of the neighboring area as the search range, the range of the transition of the gazing point position is not fixed to the specific target, and It is possible to control such that the transitions are made sequentially and stably.

Processing in the Feature Integration Layer The neurons in the feature integration layer ((2,0), (2,1),...) Will be described. As shown in FIG. 1, a feature detection layer (for example,
The connection from (1,0)) to the feature integration layer (eg, (2,0)) is
It is configured to receive an excitatory connection input from a neuron of the same feature element (type) of the preceding feature detection layer in the receptive field of the feature integration neuron, and the neurons of the integration layer are assigned to each feature category as described above. Subsampling by local averaging (calculation of average value, representative value, maximum value, etc. of input from neurons that form the receptive field of feature detection neuron) (subsampling neuron) and a different scale (processing channel ), There is a type (collective coded neurons) that combines outputs related to features in the same category.

According to the former, a plurality of pulses of the same type of characteristic are inputted, and they are integrated and averaged in a local area (receptive field) (or a representative value such as a maximum value in the receptive field). ), The position fluctuation of the feature,
Deformation can be reliably detected. For this reason, the receptive field structure of the feature integration layer neuron is uniform regardless of the feature category (for example, each is a rectangular region of a predetermined size, and the sensitivity or the weight coefficient is uniformly distributed therein). You may comprise so that it may become.

The collective coding process relating to the scale level The mechanism of the latter collective coding (population coding) will be described in detail. In a collective coded neuron, normalization of the output from multiple sub-sampling neurons at the same hierarchical level (with the same complexity of figure features) but belonging to different processing channels and within the same feature integration layer Integrate by taking a linear combination. For example,
In the feature integration layer (2,0) receiving the output of the feature detection layer (1,0) that performs Gabor wavelet conversion, a set of Gabor filters {g _mn } belonging to different processing channels and having equal direction selectivity
The outputs corresponding to (constant n, m = 1, 2,...) are integrated by a linear combination or the like. Specifically, p _ij (t) is determined by selecting the direction component selectivity i
The output of a sub-sampling neuron whose scale selectivity becomes j, and q _ij (t) is a population code (population code) having the same selectivity, represents the linear combination of the normalized output of the sub-sampling neuron. Expression (6) and Expression (7) representing the normalization method thereof are given. In addition,
Equations (6) and (7) express the output state transitions of the sub-sampling neurons and the collective encoding neurons as discrete time transitions for convenience of explanation.

[0132]

[Outside 4]

Here, w _{ij, ab} represents a plurality of neurons (or a group of neurons) having different selectivities (sensitivity characteristics).
From the subsampling neuron output of the feature category, i.e., the index of the direction component selectivity is a, the index of the scale level selectivity is b, the index of the direction component selectivity is i, and the index of the scale level selectivity is j Coupling coefficients representing contributions (to the encoding neuron).

W _{ij, ab} indicates a filter function (selectivity) centered on the directional component index i and the scale level index j, and is typically a function shape of | ia | and | jb | (w
_{ij, ab} = f (| ia |, | jb |)). As described later, this w
Collective coding by linear combination via _{ij, ab} takes into account q _ij considering the detection level of other selective neurons.
Provide the existence probabilities for the feature categories (directional components) and scale levels.

C is a normalization constant, and λ and β are constants (β is typically 1 or 2). C is a constant for preventing p _ij from diverging even when the sum of the collective codes for a certain feature category is almost zero. Note that q _ij (0) = p _ij (0) in the initial state when the system is started. In Equations (6) and (7) corresponding to FIG. 12, addition is performed only for the scale level selectivity index. As a result, the collective coded neuron outputs the existence probability (amount proportional to) of each feature belonging to the same feature category and belonging to a different scale level (processing channel).

On the other hand, as in the case of FIG. 13, the addition of the direction component selectivity index is generally further performed, so that a system for collectively encoding the intermediate levels of a predetermined number of direction components is also performed. Can be assembled. In this case, the parameters (β in equations (8) and (9),
And w _{ij, lk} ) appropriately, in the configuration shown in FIG. 13, each collective coded neuron outputs the probability of existence of a feature (an amount proportional to) for each scale level and each feature category. can do.

As shown in equation (6), the collective code q_ij(t)
Is related to the output of neurons with different sensitivity characteristics
Obtained by normalized linear combination. Has reached a steady state
Q_{i j}(t) is properly normalized (e.g., q_ijSum value for
If the value is between 0 and 1 then q
_ijIs the probability that the direction component is i and the scale level is j.
Will give. Therefore, the size of the target in the input data
Explicitly determine the scale level corresponding to the size
Has q_ijFind the curve that fits
Estimate and find the corresponding scale level
No. The scale level determined in this way is generally
Indicates an intermediate value of a preset scale level.

FIG. 23 shows an example of collective encoding of the scale level. The horizontal axis represents the scale level, and the vertical axis represents the cell output. The output is equivalent to the pulse phase, and a neuron having a peak sensitivity at a specific scale has a lower output level for a feature having a size deviated from that scale as compared to a feature having a size corresponding to the specific scale. That is,
A phase delay will occur. The figure shows a sensitivity curve (so-called tuning curve) relating to scale selectivity of each feature detection cell, each cell output, and a collective code integrated output obtained by integrating them (moment of each cell output with respect to the scale level, ie, Linear sum). The position on the horizontal axis of the integrated output of the collective code reflects the estimated value of the scale (size) of the recognition target.

In this embodiment, actually, the scale level is not explicitly obtained, and the output from the feature integration layer to the feature detection layer is q _ij (or may be normalized q _ij ). That is, FIG.
In both of the cases 2 and 13, the output from the feature integration layer to the feature detection layer is not the output from the sub-sampling neuron, but the output of the collective encoding neuron.
Finally, like q _ij after the above-described normalization, it is collectively represented as a detection probability of a specific target over a plurality of scale levels (resolutions).

In the circuit configuration of the feature integration layer shown in FIG.
First, the sub-sampling neuron circuit 1201 receives, in the local receptive field of the sub-sampling neuron, a neuron output having the same size selectivity as each feature category among the neurons in the preceding stage of the feature detection layer, and performs local averaging. . Each sub-sampling neuron output is sent to the joint processing circuit 1203. At this time, as described later, a pulse signal from each neuron is converted by a synapse circuit (not shown) into a predetermined phase amount (for example, when β in Expression (7) is 2, the square of the output level of the feature detection neuron). (Proportional amount) and propagated over the local common bus. However, wiring between neurons may be physically independent wiring without using a common bus.

In the combination processing circuit 1203, the equations (6) and (7)
And collectively encodes information having the same feature category but different size selectivity (across multiple processing channels).

Further, in FIG. 12, collective encoding is performed on sub-sampled neuron outputs having the same feature category (directional component selectivity), whereas in the circuit configuration shown in FIG. The processing as shown in equations (8) and (9) is performed by the coupling processing circuit that performs the entire processing.

[0143]

[Outside 5]

On the other hand, P _a for each channel in the (2,0) layer
Signal amplification / attenuation (pulse phase advance / delay)
A channel activity control circuit can be set up, such as for the output from each collective encoding neuron.
FIG. 15 is a schematic diagram of a configuration in which such a channel activity control circuit 1502 is set. This channel activity control circuit is set between the collective coding neuron 1202 in FIGS. 12 and 13 and the next layer, the feature detection layer.

In the final layer, the existence probability of a recognition target as a higher-order feature over a plurality of channels is determined by the activity level of a neuron (ie, firing frequency, firing spike phase, etc.).
Is expressed as In the Where processing path (or when the position information of the detection / recognition target is also detected in the final layer), the existence probability of the target corresponding to the position (place) in the input data in the final layer ) Is detected as the activity level of each neuron. The collective coding may be obtained by a linear combination without normalization, but may be easily affected by noise, and it is preferable to perform the normalization.

The normalization shown in the equation (7) or (9) is performed at the neural network level by so-called shunting in suppression.
(6) or (8), the linear connection as shown in equation (6) or (8)
Can be realized by:

FIG. 14 shows an example of a normalization circuit when β is 2. This normalization circuit includes a sum-of-squares calculation circuit 1403 for calculating the sum of squares of the outputs of the feature detection cells n _ij belonging to different processing channels, and a shunt type suppression circuit 1404 for mainly normalizing Expression (6). , And a linear sum circuit 1405 for obtaining and outputting a linear sum of Expression (5).

In the square sum calculation circuit 1403, an interneuron (in pool) holds (pools) the square value of each feature detection cell.
ter-neuron) element 1406 is present and the interneuron 140
Each synapse coupling element 1402 that provides a coupling to 6 provides a pulse phase delay (or pulse width modulation, pulse frequency modulation) corresponding to the square of the output of the feature detection cell 1401.

The shunt-type suppression circuit 1404 outputs a variable resistance element, a capacitor, and an output of the characteristic detection cell 1401 which are proportional to the reciprocal of a value obtained by multiplying the output of the interneuron 1406 by a predetermined coefficient (λ / C). Pulse phase modulation circuit giving a square
(Or a pulse width modulation circuit or a pulse frequency modulation circuit).

Next, a modification of the channel processing will be described. A configuration in which collective coding is performed for each processing channel as described above, and each processing channel output is transmitted to a subsequent layer (that is, a configuration in which the configuration of FIG. 12 or 13 is cascaded to the subsequent layer) ), To improve processing efficiency and reduce power consumption, the feature integration layer (2,
The output of the collective coded neuron may be propagated only to the feature detection cells (in the next layer) belonging to the same channel as the processing channel that gives the maximum response level in 0).

In this case, in addition to the configuration shown in FIGS. 12 and 13, a maximum input detection circuit, a so-called winner-take-off circuit, is used as a processing channel selection circuit that receives the output of the collective encoding neuron circuit and gives the maximum response level. All circuit (W
TA circuit) is set to exist between the output of the feature integration layer (2,0) and the next feature detection layer (1,1). The processing channel selection circuit may be set for each position of the feature integration layer, or may be set as a circuit for calculating the maximum response level for each processing channel for the entire input data regardless of the location, one for the layer. Is also good.

As the WTA circuit, for example, Japanese Patent Application Laid-Open No. 08-321
Known structures described in Japanese Patent No. 747, USP5059814, USP5146106 and others can be used. FIG. 16A schematically shows a configuration in which the output of only the processing channel exhibiting the maximum response of the feature integration layer is propagated to the next layer, the feature detection layer, by the WTA circuit in the feature integration layer. This is shown in FIG.
5 channel activity control circuit 1502 to gating circuit 16
Replaced with 02.

The gating circuit 1602 is shown in FIG.
And a channel selection circuit for transmitting the output of each neuron from the processing channel exhibiting the maximum average output level to the same channel in the next layer, as shown in FIG. 1604.

In the subsequent feature integration layer (2, k) (k is 1 or more), such a processing channel selection circuit is not necessarily required. For example, the output of the feature integration layer after higher-order feature detection is performed. The processing channel selection in the integrated layer of the low-order or medium-order features may be performed by feedback via the processing channel selection circuit. This concludes the description of the modification of the channel processing. Note that the flow of the sub-sampling, combination processing, and collective encoding as shown in FIGS. 12 and 13 is not limited to the configuration in which the flow is performed in the feature integration layer. For example, a separate layer for the combination processing and collective encoding is provided. Needless to say, this may be done.

It should be noted that even when objects having substantially the same size exist close to each other or partially overlap each other, a plurality of partial features due to a local receptive field structure and a sub-sampling structure are used. Needless to say, the recognition and detection performance of the object is maintained by the integrated detection mechanism.

Pulse signal processing in feature integration layer The above-described coupling circuit outputs a pulse corresponding to the collective coding level obtained by the equation (5) or (7) to each collective coding neuron. Each integrated cell with number (2, k)
The above-described collectively coded neurons as (n ₁ , n ₂ , n ₃ ) receive a pulse input from the pacemaker neuron of the layer of the layer number (1, k + 1) and perform the preceding feature detection layer or sensor. When the output of the above-described coupling circuit is at a sufficient level due to the input from the input layer (layer number (1, k)) (for example, the average number of input pulses in a certain time range or time window is larger than a threshold, or In the case where the phase is advanced, the output is performed based on the falling time of the pulse from the pacemaker.

The above-described sub-sampling neurons do not receive control from any pacemaker neurons, and have an average (independent phase independent for each sub-sampling neuron) from the preceding (1, k) layer feature detection cells. The sub-sampling process is performed based on the output level within a time window having Also, the pulse output timing control from the sub-sampling neuron to the connection processing neuron,
It is performed without the intervention of the pacemaker neuron, and the pulse output from the joint processing circuit to the collective encoding neuron is the same.

As described above, in the present embodiment, the feature-integrated cells (sub-sampling neurons, collective encoding neurons, etc.) are controlled by the timing control from the pacemaker neuron on the feature detection layer of the preceding layer number (1, k). Not configured to receive. This is because in a feature-integrated cell, the phase (frequency, pulse width, or amplitude) determined by the input level (such as the sum of the input pulses over time) within a certain time range is not the arrival time pattern of the input pulse. This is because the pulse is output in the present embodiment, but the timing at which the time window is generated is not so important. Note that this is not intended to exclude a configuration in which the feature-integrated cells receive the timing control from the pacemaker neuron in the feature detection layer in the preceding layer, and it goes without saying that such a configuration is also possible.

Processing in the feature position detection layer As shown in FIG. 1, the feature position detection layer forms a Where path separated from the What path, and has a feature detection layer (1,
k) and a feedback connection to the attention area setting control layer 108. The arrangement and function of the neurons of the feature position detection layer 107 are such that the collective coding in the feature integration layer 103 is not performed, and the size of the receptive field between the upper layer and the lower layer is large so that the information of the feature arrangement relation is not lost. Except that it does not change, subsampling is performed by arranging each neuron for each feature category and combining with the feature detection layer neuron, similarly to the subsampling neuron 1201 of the feature integration layer 103. as a result,
In the top-most feature position detection layer, a distribution of firing states of neurons representing a rough spatial distribution (arrangement) of the recognition / detection target is obtained.

Pulse Signal Processing in the Feature Position Detection Layer This is the same as the sub-sampling neuron 1201 in the feature integration layer 103. That is, the neurons in the (3, k) layer are not controlled by any of the pacemaker neurons, and
k) Sub-sampling processing is performed based on an average output level (within a time window having an independent phase for each sub-sampling neuron) from the feature detection cells in the layer.

[0161] The operating principle of the pattern detection Next, pulse code of two-dimensional figure pattern detection method will be described. FIG. 3 is a diagram schematically showing a state of propagation of a pulse signal from the feature integration layer 103 to the feature detection layer 102 (for example, from the layer (2,0) to the layer (1,1) in FIG. 1). It is.

Each neuron n _i (n _{1 to} n) on the feature integration layer 103 side
₄ ) correspond to different feature amounts (or feature elements), respectively, and the neuron n ′ _j on the feature detection layer 102 side uses higher-order features (graphics) obtained by combining the features in the same receptive field. Element).

The connection between the neurons is determined when the pulse is propagated.
Between and neuron n_iFrom neuron n ' _jSynaptic connection to
(S_ij) Delay due to time delays etc.
Delay, and as a result, via the common bus line 301
The neuron n '_jPulse train P arriving at_iIs the feature integration layer 1
Learning as long as pulse output is made from each neuron in 03
The amount of delay in synaptic connections determined by
(In FIG. 3A,
P_Four, P_Three, P_Two, P₁In order of arrival).

FIG. 3B shows a case where the time window synchronization control is performed using the timing signal from the pacemaker neuron, and the feature integrated cells n ₁ , n ₂ ,
Pulse propagation from n ₂ , n ₃ (representing different types of features) to a certain feature detection cell (n ' _j ) (which performs higher-order feature detection) on layer number (1, k + 1) The timing etc. are shown.

FIG. 6 is a diagram showing a network configuration when the feature detection layer neuron has an input from a pacemaker neuron. In FIG. 6, pacemaker neurons 603 (n _p ) form feature receptive neurons 602 (n _j , _nk etc.) that form the same receptive field and detect different types of features.
, And form the same receptive field as those, and receive excitatory connections from the neurons 601 on the feature integration layer (or input layer). Then, at a predetermined timing (or frequency) determined by the sum of the inputs (or to control the state of the receptive field as a whole, such as an average activity level of the entire receptive field). Pulse output is performed to the feature detection neuron 602 and the feature integration neuron.

Each of the feature detection neurons 602 is configured so that the time windows are phase-locked with each other using the input as a trigger signal. However, as described above, before the pacemaker neuron input is present, the phase is not locked. Each neuron outputs a pulse with a random phase. In the feature detection neuron 602, a pacemaker neuron 603 is used.
Before the input from, the time window integration described later is not performed.
The same integration is performed using a pulse input from the pacemaker neuron 603 as a trigger.

Here, the time window is determined for each feature detection cell (n ′ _i ), and is common to each neuron in the feature integration layer forming the same receptive field for the cell and the pacemaker neuron 603; Gives the time range for time window integration.

The pacemaker neuron 603 at the layer number (1, k) (k is a natural number) outputs the pulse output to the layer number (2, k-1).
Each integrated cell and its pacemaker neuron 60
By outputting to the feature detection cell (layer number (1, k)) to which 3 belongs, a timing signal for generating a time window when the feature detection cell temporally adds inputs is given. The start time of this time window is a reference time for determining the arrival time of the pulse output from each integrated cell. That is, the pacemaker neuron 603 gives a pulse output time from the feature integration cell and a reference pulse for time window integration in the feature detection cell.

Each pulse is given a predetermined amount of phase delay when passing through the synapse circuit, and further, reaches a feature detecting cell through a signal transmission line such as a common bus. The arrangement of the pulses on the time axis at this time is indicated by the pulses (P ₁ , P ₂ , P ₃ ) indicated by the dotted lines on the time axis of the feature detection cells.

Each pulse (P ₁ ,
P ₂ , P ₃ ) time window integration (usually one integration; however, charge accumulation by multiple time window integrations or averaging of multiple time window integrations may be performed. If the result is larger than the threshold value, a pulse output (P _d ) is made based on the end time of the time window. The learning time window shown in FIG. 3B is referred to when a learning rule described later is executed.

[0171] Synaptic circuits, etc. Figure 4 is a diagram showing the structure of the synapse circuit S _i. FIG. 4A shows a case where the synapse circuit 202 (S _i ) has a neuron n
_This shows that the small circuits 401 that give the synaptic connection strength (phase delay) to each neuron n ′ _j that is the connection destination of _i are arranged in a matrix. This way,
Wiring from the synapse circuit to the connected neuron can be performed on the same line (local common bus 301) corresponding to each receptive field (virtual wiring between neurons can be performed), which is a problem from the past. Is reduced (removed).

When a plurality of pulse inputs from the same receptive field are received, the neuron of the connection destination determines which neuron has emitted each pulse from the arrival time of the pulse on the basis of a time window (when the characteristic detection cell Corresponding to the feature to be detected,
It can be identified on the time axis by the phase delay inherent to the low-order features that constitute it.

As shown in FIG. 4B, each synapse coupling small circuit 401 includes a learning circuit 402 and a phase delay circuit 403. The learning circuit 402 adjusts the delay amount by changing the characteristics of the phase delay circuit 403, and also stores the characteristic value (or its control value) on the floating gate element or on a capacitor coupled to the floating gate element. It is something to memorize.

FIG. 5 is a diagram showing a detailed configuration of a synapse connection small circuit. The phase delay circuit 403 is a pulse phase modulation circuit. For example, as shown in FIG. 5A, monostable multivibrators 506 and 507, resistors 501 and 504, capacitors
503, 505 and a transistor 502 can be used. FIG. 5B shows a square wave P1 ([1] in FIG. 5B) input to the monostable multivibrator 506, and a square wave P2 (the same [2]) output from the monostable multivibrator 506. ), The respective timings of the square wave P3 (the same [3]) output from the monostable multivibrator 507.

Although the details of the operation mechanism of the phase delay circuit 403 are omitted, the pulse width of P1 is determined by the time until the voltage of the capacitor 503 due to the charging current reaches a predetermined threshold, and the width of P2 is determined. Is determined by the time constant of the resistor 504 and the capacitor 505. When the pulse width of P2 is
If the falling point shifts later, the rising point of P3 also shifts by the same amount, but the pulse width of P3 does not change. As a result, only the phase of the input pulse is changed. Is modulated and output.

Refresh circuit using control voltage Ec as reference voltage
The pulse phase (delay amount) can be controlled by changing the learning circuit 402 that controls the amount of charge stored in the capacitor 508 that applies the connection weight to the capacitor 509. In order to maintain the connection weight for a long period of time, after the learning operation, as a charge of a floating gate element (not shown) added to the outside of the circuit of FIG. The coupling load may be stored. Other configurations designed to reduce the circuit scale (for example, see Japanese Patent Application Laid-Open No. 5-37317)
A publicly known circuit configuration such as that described in Japanese Patent Application Laid-Open No. H10-327054 can be used.

In the case where the network adopts a configuration in which the connection weight has a covalent connection form (particularly, a case where a plurality of synapse connections are represented by the same weight coefficient), the delay amount at each synapse (described below). Unlike the case of FIG. 3, Pij) in equation (9) may be uniform within the same receptive field.
In particular, the connection from the feature detection layer to the feature integration layer involves sub-sampling by local averaging of the output of the feature detection layer, which is the preceding layer, and the like, and therefore does not depend on the detection target (that is, the problem Regardless), it can be configured in this manner.

In this case, as shown in FIG. 4B, each small circuit shown in FIG. 4A may be a single circuit _{Sk, i} connected by the local common bus line 401. It becomes an economical circuit configuration. On the other hand, the feature integration layer 103 (or the sensor input layer 1
When the connection from (01) to the feature detection layer 102 is made like this, the feature detection neuron detects an event called simultaneous arrival (or almost simultaneous arrival) of pulses representing a plurality of different feature elements. is there.

When the connections have symmetry, the connections giving the same weight (phase delay) are represented by the same small synapse connection circuit, so that a considerable number of synapse connections are represented by a small number of circuits. It can be configured to be. Especially in the detection of geometric features, since the distribution of the connection load in the receptive field often has symmetry,
The number of synapse connection circuits can be reduced, and the circuit scale can be significantly reduced.

FIG. 5 shows an example of a learning circuit at a synapse that achieves simultaneous arrival of pulses or a predetermined amount of phase modulation.
What has a circuit element as shown in FIG. That is, the learning circuit 402 is connected to the pulse propagation time measurement circuit 510.
(Here, the propagation time is the time difference between the pulse output time at the pre-synapse of a neuron in a certain layer and the arrival time of the pulse at the output destination neuron on the next layer.
(B) shows the sum of the synapse delay and the time required for propagation), the time window generation circuit 511, and the pulse phase for adjusting the pulse phase modulation amount in the synapse so that the propagation time becomes a constant value. A modulation amount adjustment circuit 512 can be used.

As the propagation time measuring circuit, a clock pulse from a pacemaker neuron that forms the same local receptive field as described later is input, and within a predetermined time width (time window: see FIG. 3B). A configuration in which the propagation time is obtained based on the output of the clock pulse from the counter circuit is used. By setting the time window based on the firing time of the output destination neuron,
The extended Hebb's learning rule as shown below is applied.

Learning Rule Also, the learning circuit 402 may make the width of the time window narrower as the frequency with which objects of the same category are presented increases. In this way, the more the pattern is familiar (that is, the number of presentations and the number of learnings is large), the detection of simultaneous arrival of multiple pulses (coincid
ence detection) mode. By doing so, the time required for feature detection can be reduced (the operation of instantaneous detection becomes possible) .However, it is possible to perform detailed comparative analysis of the spatial arrangement of feature elements, identification between similar patterns, etc. Becomes unsuitable.

The learning process of the delay amount is extended to, for example, a complex number domain, so that the neuron ni of the feature detection layer is obtained.
Connection weight C _ij between the feature integration layer neuron n _j
Is given as C _ij = S _ij exp (iP _ij ) (10) Here, S _ij is the coupling strength, P _ij is the phase, and i before it is a pure imaginary number, and is a phase corresponding to the time delay of the pulse signal output from the neuron j to the neuron i at a predetermined frequency. S _ij reflects the receptive field structure of neuron i and generally has a different structure depending on the target to be recognized and detected. This is formed separately by learning (supervised learning or self-organization) or is formed as a predetermined structure.

On the other hand, the learning rule for self-organization regarding the delay amount is as follows:

[0185]

[Outside 6] Given by However,

[0186]

[Outside 7] Is the time derivative of C, τ _ij is the above time delay (preset amount), and β (β1) is a constant.

[0187] Solving the above equation, C _ij converges to βexp (-2πiτ _ij), hence, P _ij converges to-tau _ij. An example of the application of the learning rule will be described with reference to the time window at the time of learning shown in FIG. 3 (B). The anterior neuron (n1, n2, n
The connection weight is updated according to equation (11) only when 3) and the posterior neuron (feature detection cell) are firing together in the time range of the learning time window. In addition,
In FIG. 3B, the feature detection cells are fired after the elapse of the time window, but may be fired before the elapse of the time window in FIG.

The processing mainly performed in the feature detection layer (at the time of learning and recognition) will be described below.

In each of the feature detection layers 102, as described above, a pulse signal relating to a plurality of different features from the same receptive field is input into a processing channel set for each scale level, A total (load sum) calculation and threshold processing are performed. The pulse corresponding to each feature amount is determined by the delay amount (phase) determined in advance by learning.
Arrives at predetermined time intervals.

The learning control of the pulse arrival time pattern is not the focus of the present application and will not be described in detail. For example, the more the characteristic elements constituting a certain figure pattern are the most contributing to the detection of the figure, the earlier. Introducing competitive learning is such that, as they arrive, pulse elements having substantially the same pulse arrival times arrive at a certain amount of time apart from each other. Alternatively, a predetermined feature element
It may be designed to arrive at different time intervals between (feature elements constituting the recognition target, which are considered to be particularly important: for example, a feature having a large average curvature, a feature having a high linearity, etc.).

In this embodiment, neurons corresponding to each lower-order feature element in the same receptive field on the feature integration layer, which is the preceding layer, are synchronously fired (pulse output) at predetermined phases.
Will do.

In general, there is a connection to a feature detection neuron that detects the same higher-order feature at a different position, which is a neuron in the feature integration layer. Having the same characteristics of the combination).
At this time, it is needless to say that synchronous firing occurs with these feature detection neurons. However, the output level (here, the phase is used as a reference, but the frequency, the amplitude, and the pulse width may be used as the reference) may be determined by summing (or averaging) contributions from a plurality of pacemaker neurons provided for each receptive field of the feature detection neuron. Etc.). In each neuron on the feature detection layer 102, the calculation of the spatiotemporally weighted sum of the input pulses (sum of weights) is performed only in a time window of a predetermined width for the pulse train that has arrived at the neuron. The mechanism for implementing the weighted addition within the time window is not limited to the neuron element circuit shown in FIG. 2, and it goes without saying that the mechanism may be implemented by another method.

This time window corresponds to the refractory period of the actual neuron.
(refractory period) It corresponds to the time zone to some extent. In other words, there is no output from the neuron regardless of any input during the refractory period (time range other than the time window), but the firing according to the input level occurs in the time window other than that time range. Is similar to the neuron.

The refractory period shown in FIG. 3B is a time period from immediately after the firing of the feature detection cells to the start of the next time window. Needless to say, the length of the refractory period and the width of the time window can be arbitrarily set, and the refractory period does not have to be shorter than the time window as shown in FIG. Even without using a pacemaker neuron, the start time of the time window is determined by the mechanism of synchronous firing between the neurons in the feature detection layer and the feature integration layer due to the weak mutual coupling between the neurons and the predetermined connection conditions (EMIzhikevich, 1999 'Weakly Pulse- Coupled Oscil
lation, FM Interactions, Synchronization, and Osci
llatory Associative Memory 'IEEE Trans.on Neural
Networks, vol.10. Pp.508-526.), These neurons are identical. It is known that the synchronous firing is generally caused by a mutual coupling and a pull-in phenomenon between neurons.

Therefore, also in the present embodiment, by configuring so as to satisfy the weak mutual connection between neurons and a predetermined synaptic connection condition, the pacemaker neuron can be eliminated.
Such an effect can be obtained.

In the present embodiment, as schematically shown in FIG. 6, as a mechanism already described, for example, a pacemaker neuron (for a fixed frequency) which receives an input from the same receptive field for each feature detection layer neuron By inputting timing information (clock pulse) based on (pulse output), the above-described common start timing may be provided.

In the case of such a configuration, it is not necessary to perform the synchronization control of the time window (if it is necessary) over the entire network, and there is a fluctuation and fluctuation of the clock pulse as described above. Is also uniformly affected by the output from the local receptive field (the fluctuation of the position of the window function on the time axis is the same between neurons forming the same receptive field). The reliability of detection does not degrade. Since a highly reliable synchronous operation is enabled by such local circuit control, the tolerance of variation regarding circuit element parameters is also increased.

Hereinafter, a feature detecting neuron that detects a triangle as a feature will be described for simplicity. The feature integration layer 103 at the preceding stage has L-shaped patterns (f ₁₁ , f ₁₂ ,...) Having various orientations as shown in FIG. 7C, and continuity (connectivity) with the L-shaped pattern. Those that respond to graphical features (feature elements) such as a combination pattern (f ₂₁ , f ₂₂ ,...) Of the line segments, and a combination (f ₃₁ ,. I do.

Also, f ₄₁ , f ₄₂ , and f _{43 in} the figure are features constituting triangles having different directions, and indicate features corresponding to f ₁₁ , f ₁₂ , and f ₁₃ . As a result of setting a specific delay amount between neurons forming interlayer connection by learning, in a triangular feature detection neuron, each sub time window (time slot) (w ₁ , w ₂ , …)
The setting is made in advance so that the pulses corresponding to the main and different features constituting the triangle arrive.

For example, w ₁ , w ₂ ,
, Wn, as shown in FIG. 7A, a pulse corresponding to a combination of a set of features that constitutes a triangle as a whole arrives first. Here, the L-shaped pattern (f ₁₁ ,
f ₁₂ , f ₁₃ ) arrive within w ₁ , w ₂ , w ₃ respectively, and the feature element
pulse corresponding to _{(f 21, f 22, f} 23) , respectively w _1, w _2, w ₃
The amount of delay is set by learning so as to arrive within.

The pulses corresponding to the characteristic elements (f ₃₁ , f ₃₂ , f ₃₃ ) arrive in the same order. In the case of FIG. 7A, a pulse corresponding to one characteristic element arrives in one sub time window (time slot). The meaning of dividing into sub-time windows is that, by performing pulse detection (detection of characteristic elements) corresponding to different characteristic elements developed and expressed on the time axis in each sub-time window individually and reliably, those characteristics are obtained. The purpose of the present invention is to increase the changeability and adaptability of the processing mode, such as whether the conditions of all feature elements are to be detected or a certain percentage of features are to be detected. .

For example, a situation in which the recognition (detection) target is a face, and the search (detection) of the eyes, which is a part of the face, is important (the eye pattern detection priority should be set high in the visual search) Case), by introducing feedback coupling from higher order feature detection layers,
It is possible to selectively enhance the reaction selectivity (detection sensitivity of a specific feature) corresponding to the feature element pattern constituting the eye. By doing so, it is possible to give higher importance to lower-order feature elements constituting higher-order feature elements (patterns) and detect them.

Further, assuming that a pulse is set in advance so that a more important feature arrives in a sub time window earlier, the weight function value in the sub time window is set to be larger than the value in the other sub time windows. Thereby, the feature having a higher importance can be more easily detected. This importance (detection priority between features) can be obtained by learning or can be defined in advance.

Therefore, if it is only necessary to detect a certain percentage of characteristic elements, the division into sub-time windows is almost meaningless, and may be performed in one time window.

Incidentally, pulses corresponding to a plurality of (three) different characteristic elements may be respectively arrived and added (see FIG. 7D). That is, one sub time window
(Time slot) includes a plurality of characteristic elements ((D) in FIG. 7),
Alternatively, it may be assumed that pulses corresponding to an arbitrary number of characteristic elements are input. In this case, in FIG. 7 (D), in the first sub-time window, pulses corresponding to the other feature elements f ₂₁ and f ₂₃ that support the detection of the apex portion f ₁₁ of the triangle arrive, and similarly, 2 In the second sub-time window, the vertex angle part f ₁₂
The pulses of other feature elements f ₂₂ and f ₃₁ have arrived that support the detection of.

The number of divisions into sub-time windows (time slots), the width of each sub-time window (time slot), the class of the feature, and the allocation of the pulse time interval corresponding to the feature are not limited to those described above. Needless to say, it can be changed. For example, a sub time window corresponding to a feature element such as 'X' or '+' may be set in addition to the above-described feature element. Such feature elements can be said to be redundant (or unnecessary) in the detection of a figure of a triangle, but conversely, the detection accuracy of a figure pattern of a triangle can be increased by detecting that they do not exist.

Further, even when a deformation not represented by the combination of these feature elements is applied (for example, when rotation within a certain range is given), the output of the neuron of the feature integration layer representing the above-mentioned feature element is obtained. The pulse responds with a continuous phase delay (delay amount: a range in which the pulse arrives in a predetermined sub-time window (time slot)) according to the degree of deviation from the ideal pattern (so-called “gra”).
For this reason, the output is stabilized so that the allowable range for the deformation of the detected graphic feature becomes a certain level or more. For example, the triangle formed by the features corresponding to the triangle (Q1) which is formed by the features corresponding to the features f _11, f _12, f ₁₃ shown in (C) of FIG. _{7, f 41, f 42,} f 43 In (Q2), at least the directions should be different from each other.

In this case, detection (integration) corresponding to each feature
When cells are present, for a triangle (Q3) corresponding to an intermediate orientation of the two _{triangles, f 11, f 12, f} 13 corresponding to the detection (integration) cells and f _41, f _42, f ₄₃ The detection (cells) corresponding to the above are all lower than the maximum response output, and are directly at the output level according to the convolution operation value with the filter kernel as the receptive field structure determined according to the type of feature. If the vector quantity as an output from the cell is integrated as being unique to the intermediate figure, it is possible to detect an intermediate figure (when rotated) in the state of two triangles.

For example, qualitatively, the rotation angle is small,
F closer to Q1₁₁, F₁₂, F₁₃From cells corresponding to
The output is relatively large, and conversely, the closer it is to Q2, the more f₄₁,
f₄₂, F ₄₃The output from the cell corresponding to is increased.

Spatial and temporal integration of pulse output and network
Click properties Next calculation of spatial weighted sum when the input pulses (weighted sum) will be described. As shown in FIG. 7B, in each neuron, the weighted sum of the input pulses is calculated by a predetermined weighting function (for example, Gaussian) for each of the sub-time windows (time slots), and the sum of the weighted sums is a threshold value. Be compared. τ _j represents the center position of the weight function of the sub-time window j, and is represented by the start time reference of the time window (the elapsed time from the start time). The weight function is generally a function of a distance (shift on a time axis) from a predetermined center position (representing a pulse arrival time when a feature to be detected is detected).

Therefore, assuming that the peak position τ of the weight function of each sub-time window (time slot) of a neuron is a time delay after learning between neurons, a spatio-temporal weighted sum of input pulses (sum of weights) is performed. Neural networks are a kind of time-domain radial basis function network (Radial Bass Network).
is Function Network; hereinafter abbreviated as RBF). The time window F _Ti of the neuron n _i using the weight function of the Gaussian function is expressed as σ and the coefficient factor (equivalent to the synaptic connection weight value) of each sub-time window as b _ij .

[0212]

[Outside 8]

Note that the weight function may take a negative value. For example, when a neuron of a certain feature detection layer is to finally detect a triangle, a feature (F _faulse ) that is clearly not a component of the figure pattern (for example, “X”, '+' Etc.)
Is detected, the detection output of the triangle is not finally made even if the contribution from other feature elements is large,
In the input sum calculation processing, the characteristic (F _faulse )
Can be given a weighting function that gives a negative contribution and a coupling from the feature detection (integrated) cells.

[0214] space sum when the input signal to the neuron n _i of the feature detection layer X _i (t) is

[0215]

[Outside 9] Can be expressed as Here, ε _j is the initial phase of the output pulse from the neuron n _j , and converges to 0 by synchronous firing with the neuron n _i , or changes the phase of the time window by the timing pulse input from the pacemaker neuron. When forcibly synchronizing to 0, ε _j may always be 0. When the pulse input of FIG. 7A and the weighted sum by the weight function shown in FIG. 7B are executed, a temporal transition of the weighted sum value as shown in FIG. 7E is obtained. The feature detection neuron outputs a pulse when the weighted sum reaches a threshold value (Vt).

[0216] The output pulse signal from the neuron n _i, as described above, with the spatio-temporal sum squashing nonlinear function to become an output level and time delay given by learning (the so-called total input sum) of the input signal (phase) Output to neurons in the upper layer (pulse output is fixed frequency (binary), and phase modulation amount which becomes a squashing nonlinear function for spatio-temporal sum of input signal is added to phase corresponding to fixed delay amount determined by learning. Output).

[0217] Processing Flow Overview Figure 8 is a flowchart illustrating a processing procedure of each layer described above. The flow of processing from low-order feature detection to high-order feature detection is summarized as shown in FIG. First, in step S801, low-order feature detection (for example, Gabor wa at each position)
velet conversion coefficients). Next, step S
In step 802, a low-order feature integration process for performing local averaging of those features and the like is performed. Further, detection and integration of middle-order features are performed in steps S803 to S804, and detection and integration of higher-order features are performed in steps S805 to S806. Then, in step S807, the presence or absence of a recognition (detection) target or the detection position output is performed as the output of the final layer. The number of layers allocated to steps S803 to 804 and S805 to 806 can be arbitrarily set or changed according to the task (recognition target or the like).

FIG. 9 is a flowchart showing the processing procedure of each feature detection neuron 602. First, step S9
In 01, a pulse corresponding to a plurality of feature categories is applied to the same receptive field 10 in the input layer 101 or the feature integration layer 103 as the preceding layer.
Step S9: receiving input from the neuron 601 forming 5
In step 02, a time window and a weighting function are generated based on a local synchronization signal input from the pacemaker neuron 603 (or obtained by interaction with a preceding layer neuron). The sum of the weights by the function is obtained, and it is determined in step S904 whether or not the threshold has been reached.
At 05, pulse output is performed. Although steps S902 and S903 are shown in time series, they are actually performed almost simultaneously.

The processing procedure of each feature integration neuron is as shown in the flowchart of FIG. That is, in step S1001, a pulse is input from the feature detection processing module 104 in the same category, which is a feature detection neuron that forms a local receptive field specific to the neuron. Input pulses are added in a time range other than the initial period.
In step S1003, a determination is made as to whether the total value of the input pulses (measured on the basis of potential, for example) has reached a threshold value. If the threshold value has been reached, in step S1004, a pulse is output at a phase corresponding to the total value. do.

FIG. 27 shows an outline of the flow of main processing related to gaze position setting control. First, in step S2701, the gaze position is set at the center of the screen, and the gaze area size is set to the entire screen. Next, in step S2702,
Executes the process of the What path up to the highest layer. At this stage, as described above, a so-called object is not perceived, that is, a recognition state has not been reached.

Then, in step S2703, the maximum feedback amount or a predetermined maximum evaluation value (formula (3)) is obtained based on the feedback amount via the Where path from the output of the feature position detection layer (3, M) corresponding to the uppermost layer. 5) The neuron on the feature integration layer corresponding to the gaze control neuron to be received is set as a new gaze position, and the size of the gaze region specific to the processing channel to which the feature integration layer neuron belongs is set.
In S2704, recognition processing in the What path is performed. Next, in step S2705, a gaze position update determination is performed, and in the case of updating, a search process for a next candidate gaze control neuron (step S2706) is performed as an update control of the gaze region in a region near the latest gaze position. .

Here, the update determination of the gaze area is performed, for example, by determining whether or not there is a gaze control neuron (referred to as “next candidate neuron”) having a sufficient feedback amount in the neighboring area. It is determined whether or not there is no next candidate neuron in the neighboring area and whether or not an unset gaze position exists in the screen (in this case, a determination circuit for the presence or absence of an unset gaze position not shown) Or the determination of the presence or absence of an input of a control signal from the outside.

In the first case, one of the next candidate neurons is selected by the above-described method, and the next gaze position and the like are set.
In the second case, any unset position outside the latest neighborhood
(Or any unset position adjacent to the neighboring area) sets the next gaze position. The third case is a case where, for example, when the operation signal of pressing the shutter button by the user is not detected as a control signal, the user gives an instruction to prompt update, and the signal of the update instruction is detected as the control signal.

In step S2707, as a result of the update, the gaze control neuron receiving the above-described sub-maximum feedback amount is activated, and the neuron is moved from the neuron to the feature integration layer (FIG.
In this case, or to the gating layer (in the case of FIG. 21), a setting signal for the next gaze area is output. Step S2
At 708, by opening a part of the specific processing channel, a signal is propagated from the feature integration layer neuron corresponding to the updated gaze area to the feature detection layer (1, 1). On the other hand, as a result of the gaze update determination, no update is performed
In the three cases (see the above three cases of determination), the update control operation of the watch area is ended.

Network and Other Modifications of the Configuration The input pulse has a feature (or
(Spatial arrangement relationship of characteristic elements), so that a spatio-temporal RBF can be configured.

Specifically, each neuron output value is further weighted and added to obtain a sufficient number of predetermined feature element sets (feature detection cells) and a sufficient number of sub-time values. A spatio-temporal function of a pulse pattern corresponding to an arbitrary figure pattern can be expressed by calculating a weighted sum (load sum) in a window (time slot). If the change of the recognition target category and its shape is limited to some extent, necessary feature detection cells and sub-time windows
(Time slots) can be reduced.

In this embodiment, the common bus is a local bus line that is assigned to the same receptive field. However, the present invention is not limited to this. As in the case of the line, the pulse phase delay amount may be divided and set on the time axis. In addition, a common bus line may be used between adjacent receptive fields having a relatively large overlapping ratio.

It should be noted that processing is performed so that the result of the weighted product-sum operation within each sub-time window (time slot) becomes a non-linear squashing function value without using the above-mentioned spatio-temporal RBF (or threshold processing). ) And take the product of them. For example, a threshold processing result is obtained by a circuit configuration (not shown).
(Binary) may be obtained for each sub time window and stored in the temporary storage means, and the logical product of the threshold processing results sequentially obtained may be obtained in a time-series manner.

When the product is obtained by performing the threshold processing, it goes without saying that the tolerance of feature detection under a pattern loss or low contrast condition is reduced.

The above-described processing (detection of a graphic pattern by spatiotemporal RBF) can be realized as an operation similar to the recall process of associative memory. That is, even if a defect of a low-order (or medium-order) feature element to be detected in a certain local area (or entire area) occurs, some other feature elements are detected and the sum value (expression (13) If)) exceeds the threshold value, it is possible to detect a medium-order (or higher-order) feature element (fire the corresponding neuron) in the entire spatiotemporal RBF network.

The configuration of the network need not be limited to that shown in FIG. 1, but may be MLP or any other configuration as long as it includes a layer for detecting a predetermined geometric characteristic element. Needless to say.

In this embodiment, Ga is used for low-order feature extraction.
Although the bor wavelet transform is used, it goes without saying that other multi-scale features (for example, a local autocorrelation coefficient obtained with a size proportional to the scale) may be used.

In the network configuration shown in FIG. 1, recognition of a figure pattern or the like is performed by a network composed of synapse elements for performing pulse width (analog value) modulation operation and integral-and-fire neurons. May go. In this case, modulation by synaptic each pulse width and the pulse width of the postsynaptic the presynaptic signal, W _b, is given by W _a = S _ij W _b when the W _a. Here, S _ij has the same meaning as the coupling strength (Equation (10)). In order to increase the dynamic range of the modulation, it is necessary to make the basic pulse width of the pulse signal sufficiently smaller than the period (basic pulse interval).

The firing (pulse output) of the neuron occurs when the potential exceeds a predetermined threshold value due to accumulation of electric charges accompanying the inflow of a plurality of pulse currents representing predetermined characteristic elements.
In the case of pulse width modulation or the like, weighted addition of arriving pulses for each sub time window is not particularly required, but integration is performed in a time window having a predetermined width. In this case, the feature element (figure pattern) to be detected is a temporal sum of signals input to the feature detection layer neurons (sum of pulse current values).
Only depends on. The width of the input pulse corresponds to the value of the weight function.

In addition, the configuration in which the scale-invariant feature expression is obtained first makes it possible to maintain the scale-invariant recognition performance without using a configuration having a plurality of processing channels up to the middle and higher orders, and to improve the efficiency of the selective gaze processing. The following configuration may be adopted as a configuration that improves the circuit configuration, simplifies the circuit configuration, reduces the size, and further reduces the power consumption.

That is, in the basic configuration described above, the attention area setting control layer is performed after the collective encoding of the low-order features, and the collective encoding is performed at each layer level. And the collective coding as described above is performed only for the lower-order features, and as described later, a scale-invariant feature expression is obtained by phase modulation of the pulse for each feature (scale-invariant pulse information conversion). Then, the above-mentioned gaze area setting control is performed, and the medium-order and high-order feature detection may be performed in this scale-invariant feature expression domain. The gaze area setting control may be performed before the collective encoding, but the processing efficiency is reduced as compared with the case where the control is performed after the collective coding.

In order to perform the above-described conversion into scale-invariant pulse information, features having the same feature category but different scale levels (sizes) may be represented by the same pulse interval. A conversion may be performed so that the phase offset amount of the arrival pattern of a plurality of pulses to the feature detection neuron for detecting the pulse becomes a constant value regardless of the pulse from any processing channel. Even when information is expressed by pulse width modulation, the same processing may be performed on the expansion or contraction or offset amount of the pulse width.

Further, in the feature detecting neurons belonging to different scale levels (processing channels), the intervals between the arrival times of the pulses (or the time patterns of the pulse arrivals) corresponding to the graphic features of the same category (for example, L-shaped patterns) ) May be determined depending on the scale level and time-divided.

The collective encoding process is performed by linear combination or the like by weighted addition over the entire time-division pulse signal. Selective gaze processing selectively extracts output data from the feature integration layer (2, 0) corresponding to a specific partial region as a gaze region before collective coding, and collective coding processing is selected. The output data is performed on the time axis. In the layers after the feature detection layer (1, 1), the multi-scale processing can be performed by the same circuit without providing a different circuit for each processing channel, resulting in an economical circuit configuration. That is,
From the (1,1) layer onwards, the difference in processing channel can be physically distinguished from the viewpoint of circuit configuration.

The output from the feature detection layer (1, 1) to the feature integration layer (2, 1) (and the same applies to the subsequent layers) is output at each processing channel output (for each scale level). This is done in splits.

Application Example Installed in Image Pickup Apparatus Next, by mounting the pattern detection (recognition) apparatus according to the configuration of the present embodiment in the image pickup apparatus, focusing on a specific subject, color correction of the specific subject, and exposure control can be performed. The case of performing the operation will be described with reference to FIG. FIG.
1 is a diagram illustrating a configuration of an example in which a pattern detection (recognition) device according to an embodiment is used in an imaging device.

An image pickup apparatus 1101 shown in FIG. 11 includes an image forming optical system 1102 including a photographing lens and a drive control mechanism for zoom photographing, and CC
D or CMOS image sensor 1103, imaging parameter measurement unit 1104, video signal processing circuit 1105, storage unit 1106, control signal generation unit 1107 that generates control signals such as control of imaging operation, control of imaging conditions, and viewfinders such as EVF A display 1108, a strobe light emitting unit 1109, a recording medium 1110, and the like.
Prepare as 11.

The image pickup apparatus 1101 performs detection (recognition of the position and size) or recognition of a face image of a person registered in advance, for example, from a finder image before photographing by a subject detection (recognition) apparatus 1111. Then, when the position and size information of the person is input from the subject detection (recognition) device 1111 to the control signal generation unit 1107, the control signal generation unit 1107 outputs the person based on the output from the imaging parameter measurement unit 1104. A control signal for optimally performing focus control (AF), exposure condition control (AE), white balance control (AW), and the like is generated.

FIG. 28 shows an outline of a processing flow in the case of performing imaging while performing selective gaze in the imaging apparatus.

First, in step S2801, in order to specify an object to be detected / recognized in advance, model data of an object to be photographed (for example, a face of the object person when photographing a person) is stored in the storage unit 1106 or The data is loaded into a temporary storage unit (not shown) via a communication unit (not shown) built in the apparatus. Next, in step S2802, the gaze control process is initialized (for example, the gaze area is set to the center position of the screen, the entire screen size, etc.) in the shooting standby state (shutter half-pressed state, etc.).

Subsequently, in step S2803, the gaze area updating process as described above is started, and a predetermined reference (for example,
Set a gaze area that matches the `` detection '' of the shooting target based on the top layer output), temporarily stop the gaze area update process in step S2804, and center the selected gaze area in step S2805. Control (A
F, AE, AW and zooming). At this time, in step S2806, a marker such as a cross mark is displayed on the finder display so that the user can confirm the photographing target.

Next, as an erroneous detection determination process (step S2807), for example, it is determined whether or not the shutter button has been pressed within a certain time range, or whether or not the user has issued an instruction to search for another object (for example, shutter halfway). (E.g., when the pressed state is released). Therefore, if it is determined that an erroneous detection has occurred, the process returns to step S2803, and starts the attention area update process again. If it is determined that there is no erroneous detection, shooting is performed under the shooting conditions at that time.

As a result of using the pattern detection (recognition) apparatus according to the above-described embodiment in an image pickup apparatus, when a plurality of subjects are present in an input image, the position of the subject and the size in the screen are determined in advance. Even if there is no such object, the subject can be detected (recognized) reliably and efficiently, such a function can be realized with low power consumption and high speed (real time), and detection of a person and the like and optimal control of photographing based on the detection. (A
F, AE, etc.).

<Second Embodiment> In the present embodiment, the priorities relating to places (positions in the screen) where selective gaze is to be performed in advance, etc.
By calculating (priority), the selective gaze updating process itself is performed at high speed. As an effect unique to the present embodiment,
By setting priorities in advance (before starting a detection operation or the like) by the priority setting unit 2401 defined later, processing efficiency (speed of gazing point update) is greatly improved. Further, by setting an allowable range of the priority in order to limit the updateable position, and updating the gaze position only to the gaze control neuron of the priority within the range, the processing speed is further increased.

FIG. 24 shows a configuration of a main part of this embodiment centered on the attention area setting control layer 108. A priority setting unit 2401 is provided for setting a priority order for selecting a gaze control neuron based on a saliency map of lower-order features (output of the feature integration layer (2, 0)) and a feedback amount from an upper layer. This is the gaze area setting control layer together with the gaze position update control circuit 1901
It is preferable to set it close to 108 (may be set on the gaze area setting control layer 108). Note that the priority setting unit 2401 and the gaze position update control circuit 1901 are digital signal processing circuits (or logic gate arrays) in the present embodiment. As in the previous embodiment, the gaze area setting control layer 108 includes gaze control neurons (the number of processing channels x the number of feature categories represented by ellipses in FIG. 8) corresponding to each position where low-order feature detection is performed on input data. Existing at the same position).

Here, the quantity indicating the priority corresponds to the linear sum (or the rank) of the output of the feature integration layer (value of saliency) and the feedback amount from the upper layer as shown in equation (5).
A control signal for activating the gaze control neurons corresponding to the descending order of priority is sent from the gaze position update control circuit 1901 to a specific (updated position) gaze control neuron. The gaze position update control circuit 1901 sequentially accesses each gaze control neuron by a predetermined method to calculate a priority and performs sorting in order to obtain a position (address) of a gaze control neuron with a high priority. Then, the address information of the neuron is stored in the primary storage unit (not shown) in order of the priority. However, only the address of the gaze control neuron having the priority within the preset priority allowable range is stored.

As a method of accessing the gaze control neuron for calculating the priority, for example, a spiral sampling is performed from an initial gaze position (the center of the screen as in the previous embodiment). In addition, the processing channel to be searched is determined based on the photographing conditions (subject distance, magnification, etc.). If the subject size is known in advance, the size of the target in the screen according to the photographing conditions is also known. However, only the corresponding gaze control neuron group may be searched.

The gaze position update control circuit 1901 does not sequentially select all the registered gaze control neuron addresses, but selects gaze control neurons within a preset priority range. For example, in the initial stage of performing the visual search (selective gaze control), the allowable range of the priority is set high, and every time the circuit makes a round in the range (this is counted as the number of gaze position search updates, The allowable range of the priority may be changed (according to the number of times). For example, the priority allowable range may be reduced or the allowable range may be expanded each time the number of times increases.

<Third Embodiment> FIGS. 20 and 21 show the structure of the present embodiment centered on the attention area setting control layer 108. FIG. In the present embodiment, the gaze control neuron 1801 is an upper layer (for example, the feature position detection layer (3, 2) or the feature integration layer (2, 2)) that outputs information on the position and existence probability of a target belonging to the recognition target category. And the position and existence probability of the secondary features of the target of the recognition category
(See the description of the collective encoding process) or an intermediate layer that outputs information on the detection level (for example, the feature position detection layer (3,
1) or the feedback integration from the feature integration layer (2, 1)), and when searching for the target by a control signal from the outside (for example, a layer higher than the top layer shown in FIG. 1) ( In the detection mode), the feedback input from the upper layer is prioritized, and when the target is recognized (recognition mode), the feedback input from the intermediate layer is prioritized.

Here, the search for an object in the detection mode is as follows.
Recognition simply means detecting a target in the same category as the recognition target (e.g., detecting a face) .Recognition means that the gaze position is controlled based on more detailed characteristics, and the target is Is determined (for example, whether or not the face is a specific person's face). At the time of the recognition of the latter, the so-called gaze degree is higher than at the time of the search of the former.

The priority method based on the weighting of the feedback combination is set by feedback amount modulation section 2601 shown in FIG. FIG. 25 shows the flow of processing via the feedback amount modulation section 2601. The feedback amount modulation section 2601 first inputs the feedback amount from the upper layer and the feedback amount from the intermediate layer to each gaze control neuron (2501).

A mode control signal of the detection mode or the recognition mode is input (2502). In the case of the detection mode, only the feedback amount from the upper layer or a linear sum (αF ₁₎ obtained by appropriately weighting both feedback amounts is used. + βF _2: F ₁ is calculated from the higher layer, F ₂ is the feedback amount) take, as the contribution regarding the amount of feedback from the upper layer is large (α> β ≧ 0) coupling feedback amount from an intermediate layer And performs modulation (amplification) such that the feedback amount becomes an output (2503). On the other hand, in the case of the recognition mode, only the feedback amount from the hidden layer or the combined feedback amount (0 ≦ α <β) is calculated such that the feedback amount contribution from the hidden layer by the linear sum of both feedback amounts is large. And give a similar modulation (2504).

In the present embodiment, the gaze position setting control can be performed in the same manner as in the first embodiment, aside from which feedback combination is prioritized. However, in addition to the method according to the first embodiment, the gaze point position is controlled. This may be performed while giving a random temporal variation regarding By giving a fluctuation to the gaze position in this way, there is a case where the search target can be detected more efficiently. That is, in the first embodiment,
When the gaze position is updated, the next candidate search process in the vicinity area is always required (when there is no gaze control neuron as the next candidate in the vicinity area of the latest gaze position, outside or outside of the vicinity area). Although the next gaze position is set at the adjacent position), the following specific effects can be obtained by giving the fluctuation of the gaze position.

(1) Even if the size of a nearby area to be searched is made smaller (than when no fluctuation is applied), the processing load required for searching in a nearby area is reduced while keeping the time required for searching equal. be able to. This is because the random fluctuation of the gaze position can be performed without performing a comparison process such as a feedback amount to the gaze control neuron in the vicinity area.
This is because an effect equivalent to performing the search for the attention position is provided.

(2) When the width of the fluctuation is small, the recognition (detection) rate can be improved by temporally averaging the output of the feature detection layer centered on a predetermined point of interest.

When the fluctuation of the gaze position is given, the fluctuation range can be changed according to the degree of gaze. Here, the degree of fixation is a degree of emphasizing the amount of feedback from a layer that detects a specific pattern constituting the entire pattern of the recognition / detection target or detects an arrangement between the specific patterns serving as such components. For example, the detection level of the feedback amount in the gaze control neuron is determined by the layer (upper layer) for detecting the entire pattern (higher-order features).
Detection level value determined based on the feedback input from, i.e. is expressed by the ratio of the feedback amount F ₂ from the intermediate layer (F ₂ / F ₁₎ for the feedback amount F1 from the upper layer.

When the degree of gaze is high, the fluctuation range of the gaze position is controlled in the manner of reducing the fluctuation range. The gaze level is a monotonically increasing function of the sum of feedback amounts from the upper layer, or a feedback amount that increases the gaze degree (feedback amount to the neuron) as the difference in feedback amount among a plurality of gaze control neurons is smaller. It may be determined by performing amplification. The latter amplification of the feedback amount is performed by the watch area setting control layer 108 as shown in FIG.
This is performed for each corresponding gaze control neuron 1801 by the feedback amount modulation section 2601 shown in FIG.

As a method for giving a temporal change to the gazing point position, for example, the first embodiment centering on the latest gazing point position
In the vicinity area similar to the above, based on the feedback amount after the above-described modulation, the updated gazing point is set in the same manner as in the first embodiment. The gaze control neuron (gaze position) is set. At this time, the width of the random fluctuation may be controlled as described above according to the degree of gaze.

<Fourth Embodiment> In the present embodiment, FIG.
As shown in FIG. 9, an assist information detection unit 2902 (hereinafter, referred to as an image input device (camera, video camera, etc.)) detects information such as the position and size of an image input (photographing) target intended by a user (photographer) inside the image input device (camera, video camera, etc.). In the present embodiment, the line of sight will be described as an example), and a subject detection (recognition) device 1111 including the gaze area setting control unit 2901 according to the above-described embodiment is provided.

The assist information (line of sight) detection unit 2902 and the attention area setting control unit 2901 perform the gaze position setting control in conjunction with each other, so that it is suitable for a specific target on the screen and reflects the intention of the user. Perform high-speed automatic shooting that can. The assist information may be explicitly set by the user in addition to the line of sight. Hereinafter, the assist information detecting unit 2902
Is described as a gaze detection unit 2902.

In FIG. 29, an imaging device 1101 is an imaging optical system including a photographing lens and a drive control mechanism for zoom photographing.
1102, CCD or CMOS image sensor 1103, measurement unit 1104 of imaging parameter, video signal processing circuit 1105, storage unit
1106, a control signal generating unit 1107 for generating control signals for controlling an imaging operation, controlling an imaging condition, etc., a display 1108 also serving as a finder such as an EVF, a strobe light emitting unit 110
9, a recording medium 1110 and the like.

Further, there is provided a subject detection (recognition) means 1111 including the above-described gaze detection unit 2901, an eyepiece optical system 2903 for gaze detection, and a gaze area setting control unit 2901. The eyepiece optical system 2903 includes an eyepiece, a light splitter such as a half mirror, a condenser lens, an illumination light source such as an LED that emits infrared light, and the like. , A penta- roof prism, a photoelectric converter, a signal processing unit, and the like, and outputs a line-of-sight position detection signal to the imaging parameter measuring unit 1104.

The configuration of the line-of-sight detection unit 2901 is disclosed in Japanese Patent Publication No.
No. 63296, Japanese Patent No. 2941847, and the like can be used, and the description is omitted.

As a specific procedure of gaze control, first, a place (area) to which the user pays attention is extracted in advance by the gaze detection unit 2902, and the information is stored in the primary storage unit. Next, the gaze area setting control unit 2901 is activated, and preferentially searches for a pre-stored gaze candidate position (plurality). Here, to preferentially search, as in the case where the gaze position is set using the priority order described in the second embodiment, for example, the gaze position is set in the order of the position where the user gazes. Each time, this means that a search for a nearby area as described in the first embodiment or the like is performed within a fixed time range, and thereafter, updating to a user's gaze position and searching for a nearby area are alternately repeated. The reason will be described later.

On the other hand, a signal from the line-of-sight detection unit 2902 may be input at regular time intervals during the operation of the gaze area setting control unit 2901 to preferentially search around the position of the user's gaze obtained at that time. Specifically, a gaze control neuron closest to the gaze position in the screen specified by the signal from the gaze detection unit 2902 is selected, and the control neuron is activated (or the gating layer in FIG. 21). By sending a gate open signal to a certain range of gates in 2101), recognition (detection) processing reflecting the gaze position of the user is performed.

As described above, the user gazes (does)
The main reason for performing the gaze area setting control by combining the information regarding the position and the gaze position information automatically updated by the gaze position setting control process is as follows.

[0272] 1. The position that the user gazes at does not always accurately represent (useful for detecting) the imaging target.

[0273] 2. The search range can be narrowed by using the position information of the imaging target intended by the user in an auxiliary manner as compared with the processing of the gaze position setting control alone as shown in the first, second, and third embodiments, and a more efficient search can be performed. It can be performed.

[0274] 3. Error detection of the gaze position set by the gaze area setting control unit is facilitated.

As described above, by using the result of detecting the gaze position of the user, the subject can be detected and recognized reliably and at high speed.

According to the embodiment described above, low-order features (for example, edges that have no meaning in detecting the target).
The setting control of the area to be watched can be performed with high efficiency by a small-scale circuit without being affected by the disturbance. In particular, by incorporating the multiple resolution processing and collective coding mechanisms into the gaze control involving the feedback amount from the upper layer, objects belonging to the same category can be located at arbitrary positions with different sizes. Even in such a case, by providing a mechanism for performing feedback control using the detection level of the higher-order feature, it is possible to efficiently search among a plurality of targets.

Further, by performing a threshold processing of a weighted weighted sum within a time window on a pulse signal which is a feature detection signal, there are a plurality of symmetries to be detected (recognized) in a complicated and diverse background, and their arrangement is performed. Even if the relationship is not known in advance, and even if its deformation (including position fluctuation, rotation, etc.), especially loss of feature detection due to size change, illumination, noise, etc., occurs, a desired pattern can be reliably formed. It can be detected efficiently. This effect can be realized regardless of the specific network structure.

Furthermore, by updating the setting of the gaze area by using the assist information (gaze direction and the like) from the user and the gaze area setting control processing, the gaze area can be updated quickly and reliably.

In addition, it is possible to efficiently search for a gaze area in accordance with the detection level of higher-order features, and to detect and recognize a target of a predetermined category with a high-speed and compact configuration.

Further, by setting the gaze area only for the low-order features or input data, the low-order features and the high-order features as in the prior art (selective tuning method and Japanese Patent Publication No. 6-34236) are performed. This eliminates the need for setting control up to the next, thereby improving processing efficiency and speeding up.

Further, in a recognition mode in which a detailed pattern is analyzed by using both the characteristic saliency of a low order or the like and the feedback signal in a mixed manner, a pattern as an element constituting a recognition target or a local pattern is analyzed. In a detection mode that preferentially utilizes information related to various feature arrangements and simply detects a pattern belonging to a certain feature category, the overall features (higher-order features or feedback signals from upper layers) are preferentially handled. Adaptive processing can be performed.

Further, since the gaze position is determined in advance and the gaze position is controlled based on the result, the search control of the gaze area can be performed at a very high speed.

Further, by changing the allowable range of the priority regarding the gaze position in accordance with the number of gaze position searches, the search process of the recognition target circulates (almost the same search position is repeatedly searched). Can change the substantial priority by changing the allowable priority.

[0284] Further, the gaze positions are set in descending order of priority from the priority distribution, and the size of the gaze area is controlled based on the processing channel to which the feature selected based on the priority belongs. Even when the size is not known in advance, it is possible to automatically set the gaze area according to the size of the target.

Also, by setting and controlling the gaze area as an active receptive field of the feature detection elements belonging to the lower-order feature detection layer, the gaze area can be set without setting a special gating element for setting the gaze area. Settings can be made.

By extracting a plurality of features for each of a plurality of resolutions or scale levels, it is possible to efficiently search for an object of an arbitrary size.

Also, by giving priority to the feedback input from the upper layer when searching for an object and giving priority to the feedback input from the intermediate layer when recognizing the object, a higher order is used when only an object belonging to a predetermined category is detected. Search based on the features of the pattern (overall features of the pattern), and in the case of identifying and recognizing similar objects, medium-level features (such as patterns that form part of the whole and information on the arrangement relationship of the features) , Processing can be performed in detail.

When the predetermined degree of gaze is large, the temporal fluctuation of the center position of the gaze area is reduced, so that the fluctuation of the gaze position at the time of visual search can be varied to improve the search efficiency according to the degree of gaze. Achieve shorter search time.

Also, the size of the gaze area is set based on the detected scale level of the pattern belonging to the category to be recognized. As a result, even if the size of the target is not known in advance, the size previously associated with the processing channel can be used as the estimated size, and the gaze area size can be set with high efficiency.

Since the degree of fixation is a monotonically increasing function of the magnitude of the level of the feedback signal from the upper layer, it is assumed that the degree of fixation increases as the detection level of the higher-order feature increases. Automatic processing can be performed.

Further, by controlling the operation based on the output signal from the above-described pattern detection device, it becomes possible to input an image targeting a specific subject at low power consumption and at an arbitrary subject distance. As the image input device, the present invention can be applied to a photographing device for so-called still images, moving images, and other three-dimensional images, and a device including a copier, a facsimile, a printer, and other image input units.

[0292] Further, by setting a gazing area that matches a predetermined criterion and performing optimization control of photographic conditions centering on the set gazing area, a photographic subject can be searched with high efficiency.
Optimal automatic shooting according to the detected target can be performed.

Furthermore, by updating or setting the gaze area based on the detected assist information from the user such as the line of sight, the gaze area setting control can be performed quickly and reliably.

[0294]

As described above, according to the present invention,
The detection of a predetermined pattern can be performed at high speed while setting the gaze area with high efficiency.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a network configuration according to an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of a synapse section and a neuron element section.

FIG. 3 is a diagram showing how multiple pulses propagate from a feature integration layer or an input layer to a feature detection layer neuron in the first embodiment.

FIG. 4 is a diagram showing a configuration diagram of a synapse circuit.

FIG. 5 is a diagram showing a configuration of a synapse coupling small circuit and a configuration of a pulse phase delay circuit used in the first embodiment.

FIG. 6 is a diagram showing a network configuration in a case where an input from a pacemaker neuron is provided to a feature detection layer neuron.

FIG. 7 shows a configuration of a time window when processing a plurality of pulses corresponding to different feature elements input to the feature detection neuron,
It is a figure which shows the example of a weight function distribution, and the example of a characteristic element.

FIG. 8 is a flowchart showing a processing procedure of each layer.

FIG. 9 is a flowchart illustrating a processing procedure of each feature detection neuron.

FIG. 10 is a flowchart illustrating a processing procedure of each feature integration neuron.

FIG. 11 is a diagram illustrating a configuration of an example in which the pattern detection (recognition) device according to the embodiment is used for an imaging device.

FIG. 12 is a diagram showing a circuit configuration of a feature integration layer.

FIG. 13 is a diagram showing a circuit configuration of a feature integration layer.

FIG. 14 is a diagram illustrating a configuration of a normalization circuit.

FIG. 15 is a diagram showing a configuration of a channel activity control circuit.

FIG. 16 is a diagram illustrating a configuration of a gating circuit.

FIG. 17 is a block diagram illustrating a network configuration according to the first embodiment.

FIG. 18 is a schematic connection diagram centered on a gaze control neuron in a gaze control layer.

FIG. 19 is a diagram illustrating a configuration of a gaze position control circuit.

FIG. 20 is a diagram illustrating a network configuration centered on a gaze area setting control layer.

FIG. 21 is a diagram showing a network configuration centering on a gaze area setting control layer.

FIG. 22 is a diagram showing a distribution control circuit for lower-layer input and upper-layer feedback input to the gaze control neuron.

FIG. 23 is a diagram illustrating an output example when outputs of multiple processing channels having different scale selectivities are integrated by collective encoding.

FIG. 24 is a diagram illustrating a network configuration used in the second embodiment.

FIG. 25 is a diagram showing a flow of processing in a feedback amount modulation circuit.

FIG. 26 is a diagram illustrating a configuration centering on a gaze control layer according to the third embodiment.

FIG. 27 is a flowchart showing the flow of processing related to gaze position setting control.

FIG. 28 is a flowchart showing a control flow when the selective gaze mechanism is incorporated in the photographing apparatus.

FIG. 29 is a diagram illustrating an image input device including a subject recognition mechanism.

Claims (30)

[Claims] 1. An input means for inputting a pattern, and a plurality of feature detection means for detecting a plurality of features corresponding to each point obtained by sampling the pattern input from the input means by a predetermined method. A detection processing unit that includes an element in a plurality of layers and detects a predetermined pattern; and controls a setting of a gaze area with respect to an output of a lower layer of the plurality of layers based on a distribution of a feedback signal from an upper layer of the plurality of layers. A pattern detection device, comprising: 2. The pattern detection apparatus according to claim 1, wherein the gaze area setting unit updates the setting of the position and size of the gaze area. 3. An input means for inputting a pattern, and a plurality of feature detection means for detecting a plurality of features corresponding to each point obtained by sampling the pattern input from the input means by a predetermined method. An element, a saliency detection element for detecting feature saliency, and coupling means for coupling between the elements and transmitting a signal, forming a plurality of element layers relating to characteristics from low to high, forming a predetermined pattern Detection processing means for performing detection of a target, and gaze area setting control means, wherein the coupling means is a feedback coupling means for transmitting a signal from an element layer relating to a higher-order characteristic to an element layer relating to a lower-order characteristic. The gaze area setting control means includes a function for controlling low-order feature data or input data based on the feature saliency and a signal transmission amount obtained from the feedback combining means. Pattern detecting apparatus characterized by controlling the setting of the fixation region that. 4. The pattern detecting apparatus according to claim 3, wherein the gaze area setting unit updates the setting of the position and size of the gaze area. 5. The gaze area setting control means determines a gaze position priority based on a signal transmission amount from the feedback coupling means and a saliency of a low-order feature based on a position of a gaze area and a sampling point position on each input data. 4. The pattern detection apparatus according to claim 3, further comprising: a priority calculation unit obtained in step (1), and a gaze position setting unit that sets a gaze position in descending order of priority from the priority distribution. 6. The gaze area setting control means counts a gaze position search count, and a priority allowable range in which the gaze position can be set by the gaze position setting means according to the gaze position search count. 6. A pattern detecting apparatus according to claim 5, further comprising control means for controlling the pattern detection. 7. The detection unit has a plurality of processing channels corresponding to a plurality of scale levels or resolutions, and the gaze area setting control unit determines a processing channel to which a feature selected based on the priority belongs. The pattern detection apparatus according to claim 5, wherein the size of the gaze area is controlled based on the gaze area. 8. The pattern detection apparatus according to claim 3, wherein the gaze area setting control means sets and controls the gaze area as an active receptive field of a feature detection element belonging to a low-order feature detection layer. 9. The gaze area setting control means includes a feedback connection from an upper layer that outputs information on the position and existence probability of a target belonging to a recognition target category, and the position and presence of a secondary feature included in the target of the recognition category. Receiving feedback from an intermediate layer that outputs information about the probability, giving priority to the feedback input from the upper layer when searching for the target, and giving priority to the feedback input from the intermediate layer when recognizing the target. Item 3
3. The pattern detection device according to 1. 10. The pattern detecting apparatus according to claim 3, wherein said gaze area setting control means reduces a temporal variation of a center position of the gaze area when a predetermined degree of gaze is large. 11. The pattern detection apparatus according to claim 9, wherein the degree of gaze is a monotonically increasing function value of the magnitude of a feedback signal level from the upper layer. 12. An input means for inputting a pattern, a feature detection layer for extracting a plurality of features for each of a plurality of resolutions or scale levels, and a feature integration for integrating and outputting the outputs from the feature detection layer by a predetermined method. A hierarchical network structure having at least one layer and a detection processing unit for detecting a predetermined pattern; and a gaze area setting control unit. Wherein the feature detection layer or the feature integration layer has a saliency detection element for detecting feature saliency, and wherein the coupling means comprises an upper layer to a lower layer of the hierarchical network structure. A feedback coupling unit that transmits a signal to the gaze area setting control unit, wherein the attention area setting control unit includes a signal from the feedback coupling unit and the characteristic saliency. A pattern detection apparatus for generating a control signal for a gaze area based on the pattern detection apparatus. 13. The pattern detection apparatus according to claim 12, wherein the gaze area setting control unit sets the size of the gaze area based on a detected scale level of the pattern belonging to the recognition target category. . 14. An input means for inputting a pattern, a gaze area setting control means, and a plurality of processing means each corresponding to a different resolution or scale level and having a hierarchical structure, Corresponds to each point obtained by sampling the input data by a predetermined method, a plurality of feature detecting elements for detecting a plurality of features, respectively, and the processing means of the processing means corresponding to different resolutions or scale levels. Multiplexing processing means for combining a plurality of feature detection element outputs, and coupling means for performing signal transmission between different elements, wherein the coupling means includes feedback coupling means for performing signal transmission from an upper layer to a lower layer. A pattern detection device, comprising: detecting a predetermined pattern. 15. The pattern detection apparatus according to claim 14, wherein the gaze area setting control means sets the size of the gaze area based on a detected scale level of the pattern belonging to the recognition target category. . 16. The pattern detection apparatus according to claim 14, wherein said gaze area setting control means controls setting of a gaze area relating to feature data or input data. 17. The pattern detecting apparatus according to claim 16, wherein the gaze area setting control means sets the size of the gaze area based on a scale level detected for a pattern belonging to a recognition target category. 18. An image processing apparatus for controlling an operation based on an output signal from the pattern detection apparatus according to claim 1. Description: 19. a storage means for storing model data of an imaging target; a gaze area setting means for searching for and setting a gaze area that meets a predetermined standard with respect to the model data while sequentially updating the gaze area; An image processing apparatus comprising: a determination unit configured to determine a photographing condition based on the gaze area set by the gaze area setting unit. 20. An apparatus according to claim 19, further comprising initialization means for initializing the gaze area in a photographing standby state.
An image processing apparatus according to claim 1. 21. The image processing apparatus according to claim 19, further comprising display means for displaying a photographing target corresponding to the gazing area set by said gazing area setting means. 22. A shutter driving signal detecting means,
20. The image processing apparatus according to claim 19, wherein the gaze area setting unit restarts the search for the gaze area when the detection unit does not detect the shutter drive signal within a predetermined time after setting the gaze area. 23. A gaze position detecting means for detecting a gaze position from a user's gaze; a gaze area setting means for searching and setting a gaze area based on the gaze position; and a gaze set by the gaze area setting means. Determining means for determining a photographing condition based on the region. 24. The image processing apparatus according to claim 23, wherein the gaze area setting unit searches for the vicinity of the gaze position. 25. An input step of inputting a pattern, and a feature detection process of detecting a plurality of features corresponding to each point obtained by sampling the input pattern by a predetermined method is performed in a hierarchical manner. A detection processing step of detecting a predetermined pattern; and a gaze area setting control for controlling setting of a gaze area related to an output of a lower layer of the detection processing step based on a distribution of a feedback signal from an upper layer of the detection processing step. And a pattern detecting method. 26. An input step of inputting a pattern, a detection processing step of detecting a predetermined pattern from the input pattern, and a gaze area setting control step, wherein the detection processing step A plurality of feature detecting steps for detecting a plurality of features relating to features from low order to high order, corresponding to each point obtained by sampling the pattern by a predetermined method, and a saliency detecting a feature saliency Detecting step, the gaze area setting control step, based on the feature saliency and the feedback signal related to the higher-order features, to set and control the position and size of the gaze area related to low-order feature data or input data. Characteristic pattern detection method. 27. An input step of inputting a pattern, a detection processing step of detecting a predetermined pattern from the input pattern, and a gaze area setting control step, wherein the detection processing step includes a plurality of resolutions or A hierarchical processing structure having at least one feature detecting step of extracting a plurality of features for each scale level and at least one feature integrating step of integrating and outputting the outputs of the feature detecting steps by a predetermined method; The detection step or the feature integration step includes a saliency level detection step of detecting a feature saliency level, and the gaze area setting control step relates to a gaze area based on the upper feedback signal of the hierarchical processing structure and the feature saliency level. A pattern detection method characterized by generating a control signal. 28. An input step of inputting a pattern, a gaze area setting control step, a detection processing step configured to detect a predetermined pattern from the input pattern, the detection processing step comprising: A plurality of feature detection steps for respectively detecting a plurality of features corresponding to each point obtained by sampling the input data by a predetermined method; and a plurality of the data processing steps related to mutually different resolutions or scale levels. And a resolution or scale level multiplexing step of combining the outputs of the patterns. 29. A setting step of searching for and setting a gaze area that matches a predetermined criterion with respect to model data of an imaging target while sequentially updating the gaze area, based on the gaze area set in the setting step. Determining a photographing condition. 30. A gaze position detecting step of detecting a gaze position from a user's line of sight, a setting step of searching for and setting a gaze area based on the gaze position, and based on the gaze area set in the setting step. Determining a photographing condition.