[go: up one dir, main page]

WO2010091358A1 - Annotation d'image rapide par décodage d'état cérébral et exploitation de motif visuel - Google Patents

Annotation d'image rapide par décodage d'état cérébral et exploitation de motif visuel Download PDF

Info

Publication number
WO2010091358A1
WO2010091358A1 PCT/US2010/023494 US2010023494W WO2010091358A1 WO 2010091358 A1 WO2010091358 A1 WO 2010091358A1 US 2010023494 W US2010023494 W US 2010023494W WO 2010091358 A1 WO2010091358 A1 WO 2010091358A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
multimedia
label
label data
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2010/023494
Other languages
English (en)
Inventor
Shih-Fu Chang
Jun Wang
Paul Sajda
Eric Pohlmeyer
Barbara Hanna
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Columbia University in the City of New York
Original Assignee
Columbia University in the City of New York
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from PCT/US2009/069237 external-priority patent/WO2010075408A1/fr
Application filed by Columbia University in the City of New York filed Critical Columbia University in the City of New York
Publication of WO2010091358A1 publication Critical patent/WO2010091358A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2323—Non-hierarchical techniques based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/35—Categorising the entire scene, e.g. birthday party or wedding scene

Definitions

  • the human brain is an exceptionally powerful visual information processing system. Humans can recognize objects at a glance, under varying poses, illuminations and scales, and are able to rapidly learn and recognize new configurations of objects and exploit relevant context even in highly cluttered scenes. While human visual systems can recognize a wide range of targets under challenging conditions, they generally have limited throughput. Human visual information processing happens with neurons which are extremely slow relative to state-of-the-art digital electronics-i.e. the frequency of a neuron's firing is measured in Hertz whereas modern digital computers have transistors which switch at Gigahertz speeds.
  • Certain embodiments of the disclosed subject matter use brain signals measured by EEG to detect and classify generic objects of interest in multimedia data. Certain embodiments of the disclosed subject matter are designed to facilitate rapid retrieval and exploration of image and video collections.
  • the disclosed subject matter incorporates graph-based label propagation methods and intuitive graphic user interfaces ("GUIs") that allow users to quickly browse and annotate a small set of multimedia data, and then in real or near-real time provide refined labels for all remaining unlabeled data in the collection. Using such refined labels, additional positive results matching a user's interest can be identified.
  • GUIs graphic user interfaces
  • Such a system can be used, e.g., as a bootstrapping system for developing additional target recognition tools needed in critical image application domains such as in intelligence, surveillance, consumer applications, biomedical applications, and in Internet applications.
  • certain disclosed systems and methods can be implemented to propagate the initial labels to the remaining data and predict the most likely labels (or scores) for each data point on the graph.
  • the propagation process is optimized with respect to several criteria.
  • the system can be implemented to consider factors such as: how well the predictions fit the already-known labels; the regularity of the predictions over data in the graph; the balance of labels from different classes; if the results are sensitive to quality of the initial labels and specific ways the labeled data are selected.
  • the processes providing the initial labels to label propagation systems can come from various sources, such as other classifiers using different modalities (for example, text, visual, or metadata), models (for example, supervised computer vision models or a brain computer interface), rank information regarding the data from other search engines, or even other manual annotation tools, hi some systems and methods, when dealing with labels/scores from imperfect sources (e.g., search engines), additional processes can be implemented to filter the initial labels and assess their reliability before using them as inputs for the propagation process.
  • sources such as other classifiers using different modalities (for example, text, visual, or metadata), models (for example, supervised computer vision models or a brain computer interface), rank information regarding the data from other search engines, or even other manual annotation tools, hi some systems and methods, when dealing with labels/scores from imperfect sources (e.g., search engines), additional processes can be implemented to filter the initial labels and assess their reliability before using them as inputs for the propagation process.
  • Certain embodiments of the disclosed subject matter use the output of the brain signal analysis as an input to a label propagation system which propagates the initial brain-signal based labels to the remaining data and predict the most likely labels (or scores) for each data point on the graph or novel data not included in the graph.
  • the output of certain disclosed system embodiments can include refined or predicted labels (or scores indicating likelihood of positive detection) of some or all the images in the collection. These outputs can be used to identify additional positive samples matching targets of interest, which in turn can be used for a variety of functions, such as to train more robust classifiers, arrange the best presentation order for image browsing, or rearrange image presentations.
  • Certain embodiments of the disclosed subject matter use a refined or predicted label set to modify the initial set of data to be presented to a user in a brain signal based target detection system.
  • a partially labeled multimedia data set is received and an iterative graph-based optimization method is employed resulting in improved label propagation results and an updated data set with refined labels.
  • Embodiments of the disclosed systems and methods are able to handle label sets of unbalanced class size and weigh labeled samples based on their degrees of connectivity or other importance measures.
  • the predicted labels of all the nodes of the graph can be used to determine the best order of presenting the results to the user.
  • the images can be ranked in the database in a descending order of likelihood so that user can quickly find additional relevant images.
  • the most informative samples can be displayed to the user to obtain the user's feedback, so that the feedback and labels can be collected for those critical samples.
  • the graph propagation process can also be applied to predict labels for new data that is not yet included in the graph. Such processes can be based, for example, on nearest neighbor voting or some form of extrapolation from an existing graph to external nodes.
  • the graph based label propagation can use a graph superposition method to incrementally update the label propagation results, without needing to repeat computations associated with previously labeled samples.
  • Fig. 1 is a diagram illustrating exemplary aspects of computer vision system modes in accordance with the presently disclosed subject matter
  • FIG. 2 is diagram illustrating an exemplary graphic user interface (GUI) portion of a computer vision module in accordance with the presently disclosed subject matter;
  • GUI graphic user interface
  • Fig. 3 is a flow chart illustrating an exemplary computer vision labeling propagation and refining method in accordance with the presently disclosed subject matter
  • Fig. 4 is a diagram illustrating a fraction of a computer vision constructed graph and computation of a computer vision node regularizer method in accordance with the presently disclosed subject matter
  • Fig. 5 is a flow chart illustrating an exemplary computer vision labeling diagnosis method in accordance with the presently disclosed subject matter.
  • Fig. 6 illustrates hardware and functional components of an exemplary system for brain signal acquisition and processing
  • Fig. 7 is a diagram illustrating exemplary aspects of a combined brain- computer multimedia processing system in accordance with the presently disclosed subject matter
  • Fig. 8 is a flow chart illustrating an exemplary method according to the presently disclosed subject matter.
  • Systems and methods as disclosed herein can be used to overcome the labeling and classification deficiencies of prior systems and methods described above by coupling both computer vision and human vision components in various configurations.
  • Computer vision components will first be described in accordance with the present disclosure.
  • Figure 1 illustrates a system and various exemplary usage modes in accordance with the presently disclosed subject matter.
  • the exemplary computer vision components of Figure 1 can be used to build an affinity graph to capture the relationship among individual images, video, or other multimedia data.
  • One exemplary computer vision system can be a transductive annotation by graph (TAG) data processing system.
  • TAG transductive annotation by graph
  • the affinity between multimedia files can be represented graphically in various ways, for example: a continuous valued similarity measurement or logic associations (e.g., relevance or irrelevance) to a query target, or other constraints (e.g., images taken at the same location).
  • the graph can also be used to propagate information from labeled data to unlabeled data in the same collection. As illustrated in Figure 1, each node in the graph 150 can represent a basic entity (data sample) for retrieval and annotation.
  • nodes in the graph 150 can be associated with either a binary label (e.g., positive vs. negative) or a continuous- valued score approximating the likelihood of detecting a given target.
  • the represented entity can be, for example, an image, a video clip, a multimedia document, or an object contained in an image or video.
  • each data sample can first be pre-processed 120 (e.g., using operations such as scaling, partitioning, noise reduction, smoothing, quality enhancement, and other operations as are known in the art).
  • Pre-filters can also be used to filter likely candidates of interest (e.g., images that are likely to contain targets of interest). After pre-processing and filtering, features can be extracted from each sample 130.
  • TAG systems and methods in accordance with the disclosed subject matter do not necessarily require usage of any specific features.
  • feature sets preferred by practical applications can be used.
  • feature sets can be global (e.g., color, texture, edge), local (e.g., local interest points), temporal (e.g. motion), and/or spatial (e.g., layout).
  • multiple types and modalities of features can be aggregated or combined.
  • affinity or similarity between each pair of samples is computed 140. No specific metrics are required by TAG, though judicious choices of features and similarity metrics can significantly impact the quality of the final label prediction results.
  • the pair- wise affinity values can then be assigned and used as weights of the corresponding edges in the graph 150.
  • a subset of default data can be displayed in the browsing window of the GUI based on, for example, certain metadata (e.g., time, ID, etc.) or a random sampling of the data collection.
  • certain metadata e.g., time, ID, etc.
  • a user can view an image of interest and then provide feedback about relevance of the result (e.g., marking the image as "relevant” or “irrelevant” or with multi-grade relevance labels). Such feedback can then be used to encode labels which are assigned to the corresponding nodes in the graph.
  • the initial labels of a subset of nodes in the graph can be provided by external filters, classifiers, or ranking systems.
  • an external classifier using image features and computer vision classification models can be used to predict whether the target is present in an image and assign the image to the most likely class (positive vs. negative or one of multiple classes).
  • the target of interest is a product image search for web-based images
  • external web image search engines can be used to retrieve most likely image results using a keyword search.
  • the rank information of each returned image can then be used to estimate the likelihood of detecting the target in the image and approximate the class scores which can be assigned to the corresponding node in the graph.
  • An initial label set can also be generated based on the initial output of the human vision components of Figure 1.
  • the TAG system hardware configuration can include an audio- visual (AV) terminal, which can be used to form, present or display audio-visual content.
  • AV audio- visual
  • Such terminals can include (but are not limited to) end- user terminals equipped with a monitor screen and speakers, as well as server and mainframe computer facilities in which audio-visual information is processed.
  • desired functionality can be achieved using any combination of hardware, firmware or software, as would be understood by one of ordinary skill in the art.
  • the system can also include input circuitry for receiving information to be processed. Information to be processed can be furnished to the terminal from a remote information source via a telecommunications channel, or it can be retrieved from a local archive, for example.
  • the system further can include processor circuitry capable of processing the multimedia and related data and performing computational algorithms.
  • the processor circuitry may be a microprocessor, such as those manufactured by Intel, or any other processing unit capable of performing the processing described herein.
  • the disclosed system can include computer memory comprising RAM, ROM, hard disk, cache memory, buffer memory, tape drive, or any other computer memory media capable of storing electronic data.
  • the memory chosen in connection with an implementation of the claimed subject matter can be a single memory or multiple memories, and can be comprised of a single computer-readable medium or multiple different computer-readable media, as would be understood by one of ordinary skill in the art.
  • Figure 2 shows an exemplary system GUI that can optionally be implemented in accordance with the presently disclosed subject matter.
  • the disclosed GUI can include a variety of components.
  • image browsing area 210 as shown in the upper left corner of the GUI, can be provided to allow users to browse and label images and provide feedback about displayed images.
  • the image browsing area can present the top ranked images from left to right and from top to bottom, or in any other fashion as would be advantageous depending on the particulars of the application.
  • System status bar 220 can be used to display information about the prediction model used, the status of current propagation process and other helpful information.
  • the system processing status as illustrated in Figure 2 can provide system status descriptions such as, for example, 'Ready', ' Updating' or 'Re-ranking.
  • the top right area 230 of the GUI can be implemented to indicate the name of current target class, e.g., "statue of liberty" as shown in Figure 3.
  • this field can be left blank or can be populated with general default text such as "target of interest.”
  • Annotation function area 240 can be provided below the target name area 230.
  • a user can choose from labels such as 'Positive', 'Negative', and 'Unlabeled.' Also, statistical information, such as the number of positive, negative and unlabeled samples can be shown.
  • the function button in this embodiment includes labels 'Next Page', 'Previous Page', 'Model Update', "Clear Annotation', and 'System Info.
  • image browsing functions can be implemented in connection with such a system and method. After reviewing the current ranking results or the initial ranking, in this embodiment, such functionality can be implemented to allow a user to browse additional images by clicking the buttons 'Next Page' and 'Previous Page.' Additionally, a user can also use the sliding bar to move through more pages at once.
  • Manual annotation functions can also optionally be implemented.
  • the user can annotate specific images by clicking on them.
  • positive images can be marked with a check mark
  • negative images can be marked with a cross mark ' x '
  • unlabeled images can be marked with a circle O ' .
  • Automatic propagation functions can also be implemented in connection with certain embodiments. After a user inputs some labels, clicking the button 'Model Update' can trigger the label propagation process and the system will thereafter automatically infer the labels and generate a refined ranking score for each image. A user can reset the system to its initial status by clicking the button labeled ' Clear Annotation. 7 A user can also click the button labeled 'System Info' to generate system information, and output the ranking results in various formats that would be useful to one of ordinary skill in the art, such as, for example, a MATLAB-compatible format.
  • two auxiliary functions are provided which are controlled by checking boxes 'Instant Update' and 'Hide Labels.'
  • 'Instant Update When a user selects 'Instant Update,' the shown system will respond to each individual labeling operation and instantly update the ranking list. The user can also hide the labeled images and only show the ranking results of unlabeled images by checking 'Hide Labels.'
  • FIG. 3 is a chart illustrating a TAG labeling propagation method in accordance with an exemplary implementation of the presently disclosed subject matter.
  • the similarity or association relations between data samples are computed or acquired to construct an affinity graph.
  • some graph quantities, including a propagation matrix and gradient coefficient matrix are computed based on the affinity graph.
  • an initial label or score set over a subset of graph data is acquired. In various embodiments, this can be done via either interactive or automatic mode, or by some other mode implemented in connection with the disclosed subject matter.
  • Procedure 350 is optional, where one or more unreliable labels are selected and removed from the existing label set.
  • Procedure 360 cleaned label set are obtained and a node regularization matrix is updated to handle the unbalanced class size problem of label data set. Procedures 340, 350, and 360 can be repeated until a certain number of iterations or some stop criteria are met.
  • step 370 the final classification function and prediction scores over the data samples are computed.
  • the corresponding labels for the labeled data set can be denoted as where y e ⁇ /, • ••, ⁇ ? ⁇ and c is the number of classes.
  • Each sample X 1 - can be treated as the node on the graph and the weight of edge e i f can be represented as Wj j .
  • Y y 0 otherwise (means data x ; is unlabeled).
  • a data sample can belong to multiple classes simultaneously and thus multiple elements in the same row of Y can be equal to 1.
  • Figure 4 shows a fraction of a representative constructed graph with weight matrix W , node degree matrix D , and label matrix Y .
  • a classification function F can then be estimated on the graph to minimize a cost function.
  • the cost function typically enforces a tradeoff between the smoothness of the function over the graph and the accuracy of the function at fitting the label information for the labeled nodes.
  • Embodiments of the disclosed TAG systems and methods can provide improved quality of label propagation results.
  • disclosed embodiments can include: superposition law based incremental label propagation; a node regularizer for balancing label imbalance and weighting label importance; alternating minimization based label propagation; and label diagnosis through self tuning.
  • Embodiments of the disclosed TAG systems and methods can also include an incremental learning method that allows for efficient addition of newly labeled samples. Results can be quickly updated using a superposition process without repeating the computation associated with the labeled samples already used in the previous iterations of propagation. Contributions from the new labels can be easily added to update the final prediction results. Such incremental learning capabilities can be useful for achieving real-time responses to a user's interaction. Since the optimal prediction can be decomposed into a series of parallel problems, and the prediction score for individual class can be formulated as component terms that only depend on individual columns of a classification matrix F : where a e (0,1) is a constant parameter.
  • each column of F encodes the label information of each individual class
  • decomposition reveals that biases can arise if the input labels are disproportionately imbalanced.
  • Prior propagation algorithms often failed in this unbalanced case, as the results tended to be biased towards the dominant class.
  • embodiments disclosed herein can apply a graph regularization method to effectively address the class imbalance issue.
  • each class can be assigned an equal amount of weight and each member of a class can be assigned a weight (termed as node regularizer) proportional to its connection density and inversely proportional to the number of samples sharing the same class.
  • V j1 - the value of V j1 - is computed as:
  • A I degree of the labeled nodes in class j .
  • Figure 5 illustrates the calculation of node regularizer on a fraction of an exemplary constructed graph.
  • the node weighting mechanism described above allows labeled nodes with a high degree to contribute more during the graph diffusion and label propagation process. However, the total diffusion of each class can be kept equal and normalized to be one. Therefore the influence of different classes can be balanced even if the given class labels are unbalanced.
  • class proportion information is known beforehand, it can be integrated into particular systems and methods by scaling the diffusion with the prior class proportion. Because of the nature of graph transduction and unknown class prior knowledge, however, equal class balancing leads to generally more reliable solutions than label proportional weighting.
  • F 7 is the j th column of the classification matrix
  • P s is the j th column of the propagation matrix P (The propagation matrix will be defined later).
  • Certain embodiments of the disclosed systems and methods make modifications to the cost function used in previously used systems and methods. For example, in certain systems and methods, the optimization is explicitly shown over both the classification function F and the binary label matrix Y:
  • Z VY
  • alternating minimization procedure to solve the above optimization problem can also contribute to improvements over prior methods and systems, as disclosed herein.
  • the cost function discussed above includes two variables that can be optimized. While simultaneously recovering both solutions can be difficult due to the mixed integer programming problem over binary Y and continuous F , a greedy alternating minimization approach can be used instead.
  • dF dF
  • Y e B" xc is a binary matrix and subject to certain linear constraints
  • another part of another embodiment of the disclosed alternating minimization requires solving a linearly constrained max cut problem which is NP . Due to the alternating minimization outer loop, investigating guaranteed approximation schemes to solve a constrained max cut problem for Y can be unjustified due to the solution's dependence on the dynamically varying classification function F during an alternating minimization procedure. Instead, embodiments of the currently disclosed methods and systems can use a greedy gradient-based approach to incrementally update Y while keeping the classification function F at the corresponding optimal setting. Moreover, because the node regularizer term V normalizes the labeled data, updates of V can be interleaved based on the revised Y .
  • the classification function, F e R" xc is continuous and its loss terms are convex, which allows its minimum to be recovered by zeroing the partial derivative:
  • ⁇ i*,j *) arg min ⁇ >1 ⁇ y - ⁇ c V Z ..Q
  • the updated Y in accordance with certain disclosed embodiments is greedy and could therefore oscillate and backtrack from predicted labeling in previous iterations without convergence guarantees. To guarantee convergence and avoid backtracking, inconsistency or unstable oscillation in the greedy propagation of labels, in preferred embodiments, once an unlabeled point has been labeled, its labeling can no longer be changed.
  • the most recently labeled point (i* 5 j*) is removed from future consideration and the algorithm only searches for the minimal gradient entries corresponding to the remaining unlabeled samples.
  • the new labeled node x z - can be removed from X M and added to X ⁇ .
  • V 1 -,- k ⁇
  • embodiments of the disclosed methods and systems can be extended to formulate a graph transduction procedure with the ability to handle mislabeled instances.
  • a bidirectional greedy search approach can be used to simultaneously drive wrong label correction and new label inferencing. This mechanism can allow for automatic pruning of incorrect labels and maintain a set of consistent and informative labels.
  • Modified embodiments of the systems and methods disclosed earlier can be equipped to more effectively deal with mislabeled samples and develop new "Label Diagnosis through Self Tuning" (LDST) systems and methods.
  • LDST Label Diagnosis through Self Tuning
  • a set of initial labels is acquired. They can be acquired, for example, either by user annotation or from another resource, such as text-based multimedia search results.
  • the gradient of the cost function with respect to label variable is computed based on the current label set is computed, and a label is added from said unlabeled data set based on the greedy search, i.e., finding the unlabeled sample with minimum gradient value.
  • a label is then removed from said label set based on the greedy search, i.e., finding the labeled sample with maximum gradient value.
  • the last two can be performed in reverse order without losing generalization, and can be executed a variable number of times (e.g., several new labels can be added after removing an existing label).
  • Certain embodiments of the disclosed systems and methods update the computed gradients based on the new label set and repeat the last two parts of the procedure to retrieve a refined label set.
  • Embodiments of the disclosed LDST systems and methods can be used to improve the results of text based image search results, hi one embodiment, top- ranked images can be truncated to create a set of pseudo-positive labels, while lower- ranked images can be treated as unlabeled samples. LDST systems and methods can then be applied to tune the imperfect labels and further refine the rank list. Additional embodiments can be used on a variety of data set types, including text classification on webpages and to correctly identify handwritten data samples. The human vision components of the presently disclosed subject matter (including the brain-computer interface) are now described.
  • EEG EEG
  • P300 a perceptual orienting response or shift of attention which can be driven by the content of the sensory input stream.
  • Certain systems and methods in accordance with the disclosed subject matter distinguish between two distinct brain states: (+) positive states in which the subject sees something of interest in an image, versus (-) negative states for which the image contains nothing of particular interest. This distinction is not to deduce from the brain signal what the exact content is or what the subject sees in the image, but instead, to utilize the high temporal resolution of EEG to detect individual recognition events from just a short segment of EEG data.
  • exemplary embodiments of the disclosed systems and methods can detect the brain signals elicited by positive examples, and distinguish them from the brain activity generated by negative example images.
  • a task for exemplary disclosed analysis systems and methods is to classify the signal between two possible alternatives.
  • a fast image sequences is presented to a user via a process known as rapid serial visual presentation (RSVP).
  • RSVP rapid serial visual presentation
  • images can be presented very rapidly, for example, at rates of 5 to 10 images per second.
  • certain exemplary methods analyze Is of data, recorded with multiple surface electrodes, following the presentation of an image.
  • Systems and methods in accordance with the disclosed subject matter can be used to measure linear variations in EEG measurement signals. By averaging over EEG measurements with appropriate coefficients (positive or negative with magnitudes corresponding to how discriminant each electrode is) it is possible to obtain a weighted average of the electrical potentials that can be used to differentiate positive from negative examples as represented below:
  • xu represents the electrical potential measured at time t for electrode i on the scalp surface
  • w,- represents the spatial weights which have to be chosen appropriately.
  • a goal of this summation is to combine voltages linearly such that the sum y is maximally different between two conditions. This can be thought of as computing a neuronal current source y t that differs most between times samples t+ following positive examples and the times t- following negative examples, y t+ > y t - Label '+' indicates that the expression is evaluated with a signal X 1/ recorded following positive examples and label '-' indicates the same for negative examples.
  • optimal weight vectors, wi ⁇ are calculated for a time window following the presentation of the data (index k labels the time window):
  • FLD can be used to train coefficients w,* within each window of time, i.e., w,* is trained such that y kt+ > y kt -.
  • the coefficients V k can be used learned using PLR after all exemplars have been observed such that y + > y ⁇ . Because of the two-part process of first combining activity in space, and then again in time, this algorithm can be referred to as a "Hierarchical Discriminant Component Analysis"
  • each time sample provides a separate exemplar that is used when training the FLD. For instance, a system with 50 training exemplars and 10 samples per window results in 500 training samples for a classification algorithm that can need to find 64 spatial weighting coefficients w ⁇ for the kth window. These multiple samples within a time window will correspond to a single examplar image and are therefore not independent.
  • noise-statistic variations in the signal within the time window are assumed to reflect non-discriminant "noise.”
  • the number of free parameters Wj t in this general form is the full set of dimensions - 6,400 for certain embodiments, for example - with only a handful of positive exemplars to choose their values.
  • the matrix w; t can be of lower rank, e.g., K.
  • the linear summation can then be written as: where is a low-rank bilinear representation of the full parameter space.
  • This bilinear model assumes that discriminant current sources are static in space with their magnitude (and possibly polarity) changing in time.
  • the model allows for K such components with their spatial distribution captured by M,* and their temporal trajectory integrated with weights V & .
  • a goal is to find coefficient u ⁇ , V t k such that the bilinear projection is larger for positive examples than for negative examples, i.e., y + > y ⁇ .
  • the x values referenced above need not be a time- domain signal, but could also be in the frequency domain.
  • the linear integration could be performed in either domain.
  • ERP event related potentials
  • ERP refers to activity that is evoked in a fixed temporal relationship to an external event, that is, positive and negative deflections occur at the same time relative to the event - for example, the time of image presentation.
  • the EEG often shows variations in the strength of oscillatory activity. Observable events can change the magnitude of ongoing oscillatory activity or can induce oscillations in the EEG.
  • the value ⁇ can be referred to as an interest level for discretizing the EEG scores
  • TAG computer vision systems
  • EEG-based brain-computer interfaces
  • TAG-based processing can operate on a dataset so as to eliminate regions of very low target probability and also provide an initial ordering of regions having high target probability, hi addition, TAG can center image chips of potential regions of interest (ROIs) in a large image or set of images, which improves detection since potential targets are foveated when images are presented to subject.
  • ROIs regions of interest
  • the top M images of the reordered dataset in which sensitivity is high but specificity can be low, can be sampled and presented to the subject for EEG-RSVP analysis.
  • the brain-computer interface processing can be tuned to produce high sensitivity and low specificity, with the EEG-RSVP mode can be used to increase specificity while maintaining sensitivity.
  • FIG. 6 illustrates the hardware components of a particular embodiment of a subsystem for brain data acquisition in accordance with the disclosed subject matter.
  • a subsystem can include EEG electrodes 610, which may be passive or active electrodes.
  • the electrodes are connected to an EEG amplifier 620, which processes and amplifies the EEG signals for further analysis.
  • An analog-to-digital converter 630 is then used to input the data received from the amplifier into a computer 650.
  • Interface 640 between the A-D converter 630 and the computer 650 may be a wire interface connected via USB or other standard, or a wireless connection via Bluetooth or other standard, or any other known mechanism for data transfer.
  • the system hardware implementation can use multiple computers 650, such as three personal computers (laptop, desktop, handheld, or any other personal computing device), two used for the RSVP and EEG recording and classification, and one for image processing, or the functionality of all modules could be performed from a single computer.
  • computers 650 such as three personal computers (laptop, desktop, handheld, or any other personal computing device), two used for the RSVP and EEG recording and classification, and one for image processing, or the functionality of all modules could be performed from a single computer.
  • computers 650 such as three personal computers (laptop, desktop, handheld, or any other personal computing device), two used for the RSVP and EEG recording and classification, and one for image processing, or the functionality of all modules could be performed from a single computer.
  • One of ordinary skill in the art would understand a variety of different configurations of such a system, including a general purpose personal computer programmed with software sufficient to enable the methods of the disclosed subject matter described herein.
  • the analysis system utilizes a 64 electrode EEG recording system in a standard montage.
  • EEG can be recorded at, for example, a 1 kHz sampling rate.
  • the RSVP display module can uses a dedicated interface to display blocks of images at the specified frame rate, hi certain embodiments, blocks are typically 100 images long.
  • the frame rate can be set to 5 or 1 OHz depending on the target/imagery types, and the human observer's response to preliminary presentations.
  • the interface draws from a pool of potential target chips and a pool of "distracters.”
  • One role of the distracters is to achieve a desired prevalence of target chips, that will maintain the human observer engaged in the presentation: if the prevalence is too low or too high, the observer can not keep an adequate focus and can more easily miss detections.
  • the computer vision outputs include some false positives, the number of distracters used depends in fact on the expected number of true target chips from the computer vision module.
  • the exemplary EEG analysis module can receive a list of image chips and detection details from the computer vision module, that includes pixel locations and detection confidence scores, and uses this input to generate the RSVP image sequences that will be used for presentation to the subject and analysis. It then performs several tasks: it acquires and records the EEG signals, using for example the hardware identified in Figure 6, orchestrates the RSVP, matches the EEG recordings with the presented images, trains an underlying classifier using training sequences, and uses the classifier with new generated image sequences.
  • a classification module relies on a hierarchical discriminant component analysis algorithm. At the first level, the classifier can use multiple temporal linear discriminators, each trained on a different time window relative to the image onset, to estimate EEG signatures of target detection.
  • the classifier can estimate a set of spatial coefficients that will optimally combine the outputs of the temporal discriminators to yield the final classification outcomes.
  • the classification module is used in two different stages: training and actual usage with new imagery.
  • the training can include a presentation of blocks with a set number of known targets in each block.
  • the training sequences need not be related to the test sequences, in terms of their content, as the premise of the approach is that it detects objects of interest, but is not sensitive to the signatures of specific objects, and can therefore maintain its detection performance from one type of imagery to another.
  • an exemplary human imaging module can generate a list of images or image chips and their associated classification confidences, which can be used to prioritize the visualization of the corresponding images or image locations.
  • the visualization interface permits the visualization of the prioritized locations in an adequate software environment for the task or user at hand. For example, for image analysts, certain embodiments use an interface to Remote View, an imagery exploitation software application often used in the Geolntelligence community. The interface provides a play control like toolbar that lets the analyst jump from one prioritized location to the next, while the analyst still retains access to all of RemoteView's functionality.
  • EEG-RSVP analysis systems and methods followed by computer vision such as TAG
  • computer vision such as TAG
  • the EEG-RSVP is first run on samples of D 1 , which can result in an image reordering in which images are ranked based on how they attracted the human subject's attention.
  • This reordering can be used to generate labels for a computer vision based learning system which, given a partial labeling of D 1 , propagates these labels and re-orders the database.
  • EEG scores are numbers with more positive scores indicating that the subject was interested in or strongly attending to the presented multimedia data.
  • the scores are sorted and the multimedia data associated with the top N scores are considered positives and labeled as such (given label +1) and used as training data for the TAG system.
  • N can be chosen to be fixed (e.g., top 20 scores) or can be selected based on the requirements of a certain precision (e.g., the N scores where at least X% are true positives and 100-X% are false positives).
  • the real-number values of the EEG scores are used to weight the strength of the training examples. For instance a EEG score of 0.3 for a EEG labeled image would result in a training label that is three times stronger than an EEG labeled image with a score of 0.1.
  • the EEG-RSVP is designed to identify a small number "interesting" images which are then used by a semi- supervised computer vision system to reorder the entire image database.
  • Exemplary systems in which computer vision analysis follows EEG analysis can be similar to the alternative computer vision followed by EEG systems, in that they can use the same type of components, such as a computer vision module, an EEG analysis module, and a visualization/review module.
  • the EEG analysis module precedes the computer vision TAG components.
  • the number of examples provided by the EEG analysis can be insufficient to train conventional supervised learning algorithms, and there can be inaccuracies in the EEG outputs due to typically lower sensitivity of the approach. Therefore, a computer vision TAG module underpinned by a semi-supervised learning algorithm can be used to improve the EEG output.
  • the outputs of the EEG systems and methods are a set of positive and negative examples (as determined by a suitable EEG confidence threshold), that can serve as labeled inputs to a graph-based classifier to predict the labels of remaining unlabeled examples in a database.
  • TAG can then incorporate its automated graph-based label propagation methods and in real or near-real time generate refined labels for all remaining unlabeled data in the collection. Further, as previously mentioned, it is also possible to have tightly coupled EEG systems and computer vision systems and methods in accordance with the disclosed subject matter.
  • both EEG analysis and computer visions are run in parallel and coupled either at the feature space level, or at the output (or confidence measure) level, leading to a single combined confidence measure that can serve as a priority indicator.
  • This mode can require prior information on the target type.
  • These modes can also potentially include feedback or multiple iterations within a closed-loop system.
  • FIG. 7 is a diagram illustrating exemplary aspects of a combined human-computer/multimedia-processing system in accordance with the presently disclosed subject matter, hi this particular embodiment, EEG-based generic interest detector 710 includes an interest object detector 710 which performs calculations resulting in an initial label (annotation) set 730. This set can be used as an input to a computer vision system 740 including, among other things, a visual similarity graph 750 and a label refinement module 760 which can be used to generate a refined label/annotation set 770.
  • a computer vision system 740 including, among other things, a visual similarity graph 750 and a label refinement module 760 which can be used to generate a refined label/annotation set 770.
  • FIG. 8 is a flow chart illustrating a combined EEG-based interest detection method coupled to a TAG labeling propagation method in accordance with an exemplary implementation of the presently disclosed subject matter.
  • a system presents multimedia data to a user.
  • the system receives user response data based on the user's brain signal response to the presented data.
  • the system determines user interest in the presented data based on the EEG response data.
  • 820 and 830 involve using hardware such as the system of Figure 6 to, more specifically, receive EEG signals and amplify, decode, and process them in real time as the human subject is viewing a rapid succession of images being presented on a display.
  • the system can extract relevant features from image data and generates an initial label set.
  • the similarity or association relations between data samples are computed or acquired to construct an affinity graph
  • some graph quantities including a propagation matrix and gradient coefficient matrix, are computed based on the affinity graph.
  • an initial label or score set over a subset of graph data is acquired from the system component which generated the initial label set based on user response data.
  • 850 and 860 can be performed before or after 870.
  • one or more new labels are selected and added to the label set.
  • 880 is optional, wherein one or more unreliable labels are selected and removed from the existing label set.
  • cleaned label set are obtained and a node regularization matrix is updated to handle the unbalanced class size problem of label data set.
  • 875, 880 and 890 can be repeated, if necessary, until a certain number of iterations or some stop criteria are met.
  • the final classification function and prediction scores over the data samples are computed.
  • the output of 895 can be used to select and arrange certain presentations of images to users as input to 810 to complete a looped integration system.
  • Certain embodiments of the disclosed systems and methods can also be used for web search improvements. Images on such web sharing sites often are already associated with textual tags, assigned by users who upload the images. However, it is well known to those skilled in the art that such manually assigned tags are erratic and inaccurate. Discrepancies can be due, for example, to the ambiguity of labels or lack of control of the labeling process. Embodiments of the disclosed systems and methods can be used to efficiently refine the accuracy of the labels and improve the overall usefulness of search results from these types of internet websites, and more generally, to improve the usefulness and accuracy of internet multimedia searches overall.
  • a system and method for collaborative search can be employed, hi such an embodiment, multiple EEG scores can be received from multiple human observers simultaneously, with the human observers each being presented the same multimedia data. The multiple scores can be processed and used to construct labels for the displayed multimedia data.
  • computer vision systems and methods for use with brain- computer interfaces and methods as described herein are not limited to TAG or LDST systems and methods.
  • Other back-end systems that may be used to process the brain- computer interface data may include any type of graphical probabilistic/generative models, including clustering, support vector machines, belief networks, and kernel- based systems and methods. Any computer vision label propagation component may be utilized.
  • the described brain- computer interface can be used in conjunction with a number of different computer analysis systems to achieve the principles of the disclosed subject matter.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Library & Information Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Discrete Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Image Analysis (AREA)

Abstract

Une perception visuelle humaine est capable de reconnaître une large plage de cibles mais a un débit de traitement limité. Une vision par machine peut traiter des images à une vitesse élevée mais souffre d'une précision de reconnaissance inadéquate de classes de cibles générales. L'invention porte sur des systèmes et des procédés qui combinent les forces des deux systèmes et améliorent les systèmes et procédés de traitement multimédia existants pour fournir un marquage, une catégorisation et une recherche multimédia améliorés.
PCT/US2010/023494 2009-02-09 2010-02-08 Annotation d'image rapide par décodage d'état cérébral et exploitation de motif visuel Ceased WO2010091358A1 (fr)

Applications Claiming Priority (10)

Application Number Priority Date Filing Date Title
US15112409P 2009-02-09 2009-02-09
US61/151,124 2009-02-09
US17178909P 2009-04-22 2009-04-22
US61/171,789 2009-04-22
US23332509P 2009-08-12 2009-08-12
US61/233,325 2009-08-12
US23367509P 2009-08-31 2009-08-31
US61/233,675 2009-08-31
USPCT/US09/069237 2009-12-22
PCT/US2009/069237 WO2010075408A1 (fr) 2008-12-22 2009-12-22 Système et procédé d'annotation et de recherche de contenus multimédias

Publications (1)

Publication Number Publication Date
WO2010091358A1 true WO2010091358A1 (fr) 2010-08-12

Family

ID=42542417

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2010/023494 Ceased WO2010091358A1 (fr) 2009-02-09 2010-02-08 Annotation d'image rapide par décodage d'état cérébral et exploitation de motif visuel

Country Status (1)

Country Link
WO (1) WO2010091358A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110664400A (zh) * 2019-09-20 2020-01-10 清华大学 基于度信息的脑电特征电位溯源方法
CN114004751A (zh) * 2020-07-27 2022-02-01 华为技术有限公司 一种图像处理方法及其相关设备

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030046018A1 (en) * 2001-04-20 2003-03-06 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandeten Forschung E.V Method for segmentation and identification of nonstationary time series
US20060293921A1 (en) * 2000-10-19 2006-12-28 Mccarthy John Input device for web content manager responsive to browser viewers' psychological preferences, behavioral responses and physiological stress indicators
US20080222670A1 (en) * 2007-03-07 2008-09-11 Lee Hans C Method and system for using coherence of biological responses as a measure of performance of a media

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060293921A1 (en) * 2000-10-19 2006-12-28 Mccarthy John Input device for web content manager responsive to browser viewers' psychological preferences, behavioral responses and physiological stress indicators
US20030046018A1 (en) * 2001-04-20 2003-03-06 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandeten Forschung E.V Method for segmentation and identification of nonstationary time series
US20080222670A1 (en) * 2007-03-07 2008-09-11 Lee Hans C Method and system for using coherence of biological responses as a measure of performance of a media

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
SAJDA ET AL.: "In a blink of an eye and a switch of a transistor: Cortically-coupled computer vision'", JOURNAL OF LATEX CLASS FILES, vol. 6, no. 1, January 2007 (2007-01-01) *
WANG ET AL.: "Columbia TAG System - Transductive Annotation by Graph Version 1.0'", COLUMBIA UNIVERSITY ADVENT TECHNICAL REPORT #225-2008-3, 15 October 2008 (2008-10-15) *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110664400A (zh) * 2019-09-20 2020-01-10 清华大学 基于度信息的脑电特征电位溯源方法
CN114004751A (zh) * 2020-07-27 2022-02-01 华为技术有限公司 一种图像处理方法及其相关设备

Similar Documents

Publication Publication Date Title
US8671069B2 (en) Rapid image annotation via brain state decoding and visual pattern mining
Li et al. A novel transferability attention neural network model for EEG emotion recognition
US20110314367A1 (en) System And Method For Annotating And Searching Media
US10664719B2 (en) Accurate tag relevance prediction for image search
EP2192500B1 (fr) Système et procédé pour fournir une identification robuste de sujets dans des indexages sociaux
US20170236055A1 (en) Accurate tag relevance prediction for image search
Hamreras et al. Content based image retrieval by ensembles of deep learning object classifiers
Wu et al. Hierarchical attention for part-aware face detection
Akila et al. Parkinson classification neural network with mass algorithm for processing speech signals
Zhu et al. Identification of cashmere and wool based on LBP and GLCM texture feature selection
Alayed Machine Learning and Deep Learning Approaches for Arabic Sign Language Recognition: A Decade Systematic Literature Review
Agarwal et al. A mathematical model based on modified ID3 algorithm for healthcare diagnostics model
Rani et al. MobileNet for human activity recognition in smart surveillance using transfer learning
Yu et al. LIME-explained small-scale tabular transformer used for improving the classification performance of multi-category causes of death in colorectal cancer
Li et al. An implicit relevance feedback method for CBIR with real-time eye tracking
WO2010091358A1 (fr) Annotation d'image rapide par décodage d'état cérébral et exploitation de motif visuel
Yang et al. mPadal: a joint local-and-global multi-view feature selection method for activity recognition
Rabaev et al. Automated gender classification from handwriting: a systematic survey
Tian et al. Automatic image annotation with real-world community contributed data set
Eliwa et al. Particle swarm optimization framework for Parkinson’s disease prediction
Bhandari et al. Modeling eye gaze velocity trajectories using GANs with spectral loss for enhanced fidelity
Chen et al. Multi-modal multi-layered topic classification model for social event analysis
Kunhoth et al. Multimodal Ensemble with Conditional Feature Fusion for Dysgraphia Diagnosis in Children from Handwriting Samples
Sree et al. An Analysis of Heart Disease Prediction using Machine Learning
Xue et al. Optimization of parallel SVM algorithm for big data

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10739234

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 10739234

Country of ref document: EP

Kind code of ref document: A1