HK1139262A - System and method for integrating video analytics and data analytics/mining - Google Patents

Description

System and method for integrated video analysis and data analysis/mining

Technical Field

The present invention relates generally to systems and methods for analyzing video, and more particularly to systems and methods for integrating video analysis and data analysis/data mining to leverage the effectiveness of both video analysis and data analysis.

Background

The use of video surveillance and analysis in deterring shoplifting and theft in retail stores has become commonplace. However, in retail and other environments, there is often too much data and video collected from security and business operations to be effectively and efficiently managed by a person. With tighter budgets and pressure to limit the total number of staff, the burden is even greater. Stores need tools to filter and mine data so that they can determine anomalies, patterns, and/or anomalous behavior. In addition, there are more complex collusion threats, ranging from clerks "sweethearting" transactions for their own or consumer benefit (bypassing the scanner), to organized crime groups that work together across multiple incidences and multiple locations.

Some have attempted to solve and manage these problems from a business management standpoint, employing solutions based on analyzing data available from store systems (e.g., points of sale) to identify patterns of anomalous behavior that show areas of interest. Improvements to these solutions include having these modes trigger video clips from the video surveillance system that provide visual verification of the condition. Other improvements have explored this problem from a security standpoint, using computer algorithms to analyze video from video surveillance systems so that certain levels of anomalous behavior can be visually detected independent of other triggers and used to implement policies in enterprise operations.

This approach of separate business data has failed for several reasons. Because the store system may be bypassed, data describing the condition characteristics may not be available. Furthermore, data systems tend to be post-mining, which limits their ability to handle real-time/time sensitive alerts and notifications. This improvement also fails in view of the limitations of the data systems described above and the reliance on data triggers for playback of video segments. The isolated analysis of video is problematic because it may be prone to false alarms or to a level of accuracy that is insufficient to make it reliable. Furthermore, these arrangements often require event configuration/rule definitions to detect anomalies, and these patterns may not be known in advance.

Therefore, there is a need for a system and method for integrating video analytics and data analytics/data mining that leverages the power of both video analytics and data analytics to compensate for the limitations of the aforementioned solutions. What is also needed is integrated software that can provide business intelligence and operational intelligence for facility entry/exit points, point of sale and service points, and for the entire interior and exterior.

Disclosure of Invention

The present invention advantageously provides a method and system that integrates video analytics technology with data analytics technology to more accurately identify potentially suspicious behaviors and events that require attention by management personnel. In general, the present invention provides methods and systems for monitoring a facility, such as a retail store or warehouse, using data collected at a point of sale register (point of sale register) to more accurately identify objects and events simultaneously detected by a video monitoring system.

One aspect of the invention includes a method for detecting potential suspicious behavior in a monitored facility. Video content of an activity occurring in a monitored facility and transaction data relating to a transaction processed at a point of transaction terminal (point of transaction terminal) are collected. The video content is associated with the transactional data to produce associated data. A set of user-defined rules is applied to the associated data. The transaction is determined to be potentially suspicious in response to identifying a match between the associated data and at least one rule of the set of user-defined rules.

Another aspect of the invention includes a method of automatically identifying activities occurring at a monitored facility. Video content of activities occurring in the monitored facility is collected. Video content is analyzed using object recognition techniques by applying a set of video analysis rules to the collected video information. Transaction data relating to one or more transactions processed through at least one point-of-transaction terminal in a sales facility is also collected. In response to determining that the video content is consistent with at least one video analytics rule of the set of video analytics rules, the video content is associated with the transactional data to provide associated transactional data.

In accordance with another aspect of the present invention, a system for analyzing activities occurring at a monitored facility includes a video analysis system, a data analysis system, and an integrated server. The integrated server is communicatively coupled to the video analytics system and the data analytics system. The monitored facility includes at least one point-of-sale register. A video analysis system collects video content of activities occurring in a monitored facility. A data analysis system collects transaction data relating to one or more transactions processed by at least one point-of-transaction terminal. The integration server associates the video content with the transaction data to produce associated data. The integration server also applies a set of user-defined rules to the associated data and identifies a match between the associated data and at least one rule of the set of user-defined rules.

Drawings

A more complete appreciation of the invention and the attendant advantages and features thereof will be readily understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

FIG. 1 is a block diagram of an exemplary video and data analysis system constructed in accordance with the principles of the present invention;

FIG. 2 is a block diagram of an exemplary video and data monitoring site constructed in accordance with the principles of the present invention;

FIG. 3 is a flow diagram of an exemplary return transaction process conducted in accordance with the principles of the present invention;

FIG. 4 is a flow chart of an exemplary cash voiding transaction process conducted in accordance with the principles of the present invention;

FIG. 5 is a flow chart of an exemplary consumer counting process conducted in accordance with the principles of the present invention;

FIG. 6 is a flow diagram of an exemplary process for automatically linking transactional exceptions to indexed video, conducted in accordance with the principles of the present invention;

FIG. 7 is a flow chart of an exemplary queue duration (linkage) measurement process, conducted in accordance with the principles of the present invention;

FIG. 8 is a flowchart of an exemplary process for detecting a cash drawer open (cash drawer opening) by video analytics without detecting a transaction, conducted in accordance with the principles of the present invention;

FIG. 9 is a flow chart of an exemplary process for setting up point of sale ("POS") rules and generating exceptions, conducted in accordance with the principles of the present invention;

FIG. 10 is a flow chart of an exemplary process for setting up user definable video rules and generating alerts in accordance with the principles of the present invention;

FIG. 11 is a flow chart of an exemplary process for setting up a combination of user definable store data rules and video rules, conducted in accordance with the principles of the present invention; and

fig. 12 is a flow chart of an exemplary reporting process, conducted in accordance with the principles of the present invention.

Detailed Description

Before describing in detail exemplary embodiments that are in accordance with the present invention, it should be observed that the embodiments reside primarily in combinations of apparatus components and processing steps related to implementing a system and method for analyzing video to determine the presence of an alarm condition by integrating video analysis with data analysis/data mining techniques. Accordingly, by describing the system and method components in general terms in the figures, where appropriate, only those specific details that are pertinent to understanding the embodiments of the present invention are shown so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

As used herein, relational terms, such as "first" and "second," "top" and "bottom," and the like, may be used solely to distinguish one entity or element from another entity or element without necessarily requiring or implying any physical or logical relationship or order between such entities or elements.

One embodiment of the present invention advantageously provides a method and system for analyzing video using video analysis and data analysis/data mining techniques. In one embodiment, the present invention may include software consisting of a user interface (e.g., client/browser), an administration and analysis component, and reporting capabilities. Video systems with embedded analytics at the edge, video storage on a digital video recorder ("DVR") or other storage device, and retail transaction data devices may also be included.

In another embodiment, the user interface allows the user to define configurations and rules, predefined events (pre-events), and to mine data and video after the fact. The video and data systems may be networked together and communicate via database transmission (database transmission) and queries and application program interfaces. The video and data analysis node may have the capability to: their analysis is performed in an embedded, distributed manner and the processed metadata is transferred to the system database.

An extremely general embodiment of the present invention enables the addition of new pre-packaged and customer-defined rules and the measurement of operational metrics called key performance indicators ("KPIs"). By understanding the problems and opportunities of the consumer, the system can be used to define use cases that are the basis for generating enabling rules (enablingrule) and KPIs.

The system may be programmable to trigger alarms in real time, and to mine patterns of data and behavior after the fact, and combine the information sources of both to enhance the ability to address more complex and broader use cases. The system may also be programmable to combine triggers for video analysis and data analysis in a wide variety of combinations: data analysis trigger-video verification, video analysis trigger-data verification, data analysis trigger-video analysis verification, video analysis trigger-data analysis verification.

Referring now to the drawings in which like reference designators refer to like elements, there is shown in FIG. 1 an exemplary business intelligence system 10 for integrated video analytics and data analytics/data mining that leverages the capabilities of both video analytics and data analytics constructed in accordance with the principles of the present invention. The business intelligence system 10 can be constructed to support enterprise-wide video solutions and broader uses in retail business.

The business intelligence system 10 incorporates a video analysis subsystem 12 and a data analysis subsystem 14 to analyze and detect suspicious activity and store/warehouse management events. The video analysis subsystem 12 may include one or more cameras 16 (one shown), a video recorder 18, a video engine 20, a video controller 22, and a video system interface 24. The camera 16 captures images of activity in the local field of view and transmits the images to the video recorder 18 and/or the video engine 20. The video recorder 18 can time stamp the captured image and store it for later recall. The video engine 20 performs a target recognition/detection function on the captured images to determine whether the images captured by the camera 16 meet conditions determined according to preset rules. Note that the functionality of the video engine 20 may be embedded in the camera 16 or other edge device to allow processing of live video in addition to allowing storage of video in the video recorder 18. The time stamping may also be performed by the camera 16 or some other intermediate device. The video controller 22 controls the basic configuration of the video system, such as which camera 16 is active; pan, tilt, pan angle and focus settings for each camera 16; playback of the requested video segment, etc. The video system interface 24 allows the user to set rules and conditions for the video analytics server 20, as well as to select particular video segments for playback.

Each component of the video analysis subsystem 12 may be directly connected to other components in the video analysis subsystem 12 at a local level. Alternatively and/or additionally, each component of the video analytics subsystem 12 may be connected to other components in the video analytics subsystem 12, the data analytics subsystem 14, the network client 26, and/or other locations via a local area network ("LAN") (not shown) or a wide area network ("WAN") 28. In addition, components of the video analysis subsystem 12 may be co-located or embedded in other components of the system 10. For example, the video system interface 24 may be implemented on the web client 26 as a web browser or as a plug-in to an existing data analysis and/or video software application.

The data analysis subsystem 14 includes a point of transaction terminal 30 for collecting information relating to transactions in the monitored facility. The point of transaction terminal 30 may be a point of sale ("POS") register for collecting information relating to sales transactions conducted at checkout. The point of transaction terminal 30 may include a communication interface for communicating data with the data engine 32. The data engine 32 receives data from one or more POS loggers 30 relating to completed, initiated or invalid transactions. The data analysis server 32 analyzes the transaction data to determine whether any transaction or set of transactions meets conditions determined according to preset rules and post-event mining. The data analysis system interface 24 allows a user to set rules and conditions for the data engine 32, as well as generate and view reports.

The integration server 36 combines the elements of the video engine 20 and the data engine 32 to correlate transaction events occurring at the point of transaction terminal 30 with object identifications detected by the video engine 20. The integration server 36 may include the video engine 20 and/or the data engine 32. In addition, the data analysis system interface 24 and the video system interface 24 may be combined into a single user interface (i.e., a dashboard) located at the network client 26. Using the dashboard, a user may combine one or more rules from the video analytics system 12 with one or more rules from the data analytics system 14 to create a set of rules for the integration server 36 to accurately determine when a very specific event occurs. In addition, the system 10 may include a dashboard for each user type to allow access to only views and reports that are important to their operational needs.

The integration server 36 may be stand-alone or can be located on any application server. The integration server 36 for real-time events can also be located at the corporate headquarters level (centralcoporate level), on the same hardware server as the data engine 32 or on a dedicated application server. The business intelligence system 10 should be able to time synchronize all components.

The business intelligence system 10 may be implemented at a local store/area, at a corporate headquarters office, or a combination thereof connected via a wide area network 28. The wide area network 16 may include the internet, an intranet, or other communication network. Although depicted in fig. 1 as a WAN, the principles of the present invention may also be applied to other forms of communication networks, such as personal area networks ("PANs"), local area networks ("LANs"), campus area networks ("CANs"), metropolitan area networks ("MANs"), etc.

While the system 10 as a whole may be quite complex, everyday use is quite user friendly and intuitive. The system 10 advantageously provides an easy to use video system interface 24, data analysis system interface 34, and report generation package (reporting package) to analyze data and view live and stored video that supports alarms and modes.

Referring now to FIG. 2, there is shown a layout of an exemplary local commercial establishment 38 detailing possible video surveillance locations and data collection sites in accordance with the principles of the present invention. Although FIG. 2 shows a retail facility, the invention is not so limited. It is contemplated that any monitored facility can be implemented and supported by the present invention, such as a warehouse or a location where goods or property enter or leave. The system 10 is programmable and can provide business and operational intelligence to operating facility entry/exit points 40, sales (i.e., transaction) points such as check-out lines (check-out lines) 42 or customer service ports 44, service points 46, and pick-out points 48, both internal and external to the overall monitored facility.

In FIG. 3, an exemplary operational flow diagram is provided that describes steps performed in determining that a return transaction has occurred in the absence of an actual consumer. In one embodiment, the process allows store managers or loss prevention ("LP") professionals to monitor in real time when returns occur but no customers are present in front of the POS station. At least one camera 16 should monitor the area around a given POS register 30. When a return transaction is processed at the POS register 30 (step S100), the data engine 32 receives POS data regarding the return transaction (step S102). This data may include, for example, an identifier of the POS register, the type of transaction, the time of the transaction, the name or other identifier of the employee performing the transaction, the amount of the transaction, etc. The data engine 32 requests visual verification from the video engine 20 (step S104).

The video engine 20 attempts to count the number of consumers present in front of the POS register (step S106). If the video engine 20 is unable to count the consumers, the transaction is marked as "number of consumers unknown" (step S108). For example, certain environmental conditions may render the video engine 20 unable to determine an accurate number of consumers, such as sudden lighting changes, very dark lighting, poor video quality, strong glare in the image, camera movement. All transactions marked as "consumer quantity unknown" may constitute suspicious activity, and details of the transaction may be included in a report for further review at a later time.

If the video engine 20 returns a number of consumers that is not equal to zero (step S110), indicating that at least one consumer is present at the checkout counter, the transaction is deemed normal (step S112) and no further action is taken. However, if the video engine 20 returns a number of consumers equal to zero (step S110), indicating that no consumer is present at the checkout counter, an "return fraud" alarm is generated (step S114) and the return transaction is flagged. The alerts may be displayed on the dashboard, saved in a database, sent to the video recorder 18, and/or sent to an event handler of the video analytics system 12. If the user wishes to play back the corresponding video, he/she simply selects the alert indicator from the dashboard and the video is then played back and marked as "viewed". All tagged transactions are available for post event mining.

Referring now to FIG. 4, an exemplary operational flow diagram is provided that describes steps performed in determining that a cash transaction is to be voided in the absence of an actual consumer. As in the case described above, at least one camera 16 should monitor the area surrounding a given POS register 30. When the cash transaction at the POS register 30 is invalidated (step S120), the data engine 32 receives POS data regarding the cash transaction (step S122). The data engine 32 requests visual verification from the video engine 20 (step S124). The video engine 20 attempts to count the number of consumers present in front of the POS register (step S126). If the video engine 20 is unable to count the customers, the cash void transaction is flagged as "customer count unknown" (step S128). All transactions marked as "consumer quantity unknown" may constitute suspicious activity and details of the transaction may be included in a report for further review at a later time.

If the video engine 20 returns a number of consumers not equal to zero (step S130), indicating that at least one consumer is present at the checkout counter, the transaction is deemed normal (step S132) and no further action is taken. However, if the video engine 20 returns a number of consumers equal to zero (step S130), indicating that no consumer is present at the checkout counter, an alarm of "cash posting (cash post) invalid fraud" is generated (step S134) and the cash invalid transaction is flagged. As in the case of return fraud, the alert may be displayed on the dashboard, saved in a database, sent to the video recorder 18, and/or sent to an event handler of the video analytics system 12. If the user wishes to play back the corresponding video, he/she simply selects the alert indicator from the dashboard and the video is then played back and marked as "viewed". All tagged transactions are available for post event mining.

Referring now to FIG. 5, an exemplary operational flow diagram is provided that describes steps performed in counting and detecting high traffic in (high traffic) periods or high net occupancy (high net occupancy) for people entering and leaving a store over a period of time. In one embodiment, this information is combined with data from sales and employee management systems to determine peaks and troughs for store employee and sales conversion calculations. At least one camera 16 should monitor each entry and/or exit location in the store.

Using the dashboard, the user requests to start the people counting feature and specifies the time period of the count. The integration server 36 receives the request for people counting (step S140) and notifies the video engine 20 to count people that are photographed entering and/or leaving the store during a predetermined period of time (step S142). The data engine 20 determines the number of transactions and the total number of transactions that occurred during the predetermined time period (step S144). A report of the results is generated (step S146) and a visual representation of the report is displayed on the dashboard.

Referring now to fig. 6, an exemplary operational flow diagram is provided that describes steps performed in playing back recorded video that pertains to a transaction anomaly (i.e., an event that has been flagged as potentially containing suspicious activity). The integration server 36 receives a request for video on transaction exceptions (step S148). Integration server 36 retrieves the corresponding video from video recording system 18 (step S150) and plays the requested video at network client interface 26 using, for example, a dashboard (step S152).

FIG. 7 provides an exemplary operational flow diagram that describes steps performed in determining checkout queue duration. In one embodiment, the present invention allows a store keeper or other corporate manager to identify situations where the check-out wait queue is longer than a predetermined limit or where the wait time is longer than a predetermined limit and retrieve the corresponding POS data. This feature allows the user to investigate the underlying factors that cause the delay, such as someone making a large purchase, insufficient check-out register opening, etc.

Using the object recognition algorithm, the video engine 20 determines that the checkout queue or the duration of time used in the checkout queue is longer than a predetermined limit (step S154). The alert may be sent to the network client interface 26 and the video recorder 18 (step S156). The alert may be displayed, for example, on an event handler of the network client interface 26 or an alert list in the video controller 22. The integration server 36 receives an alarm information request requesting transaction data occurring at the time of the alarm (step S158). The alert information request may be initiated, for example, by the user clicking on an alert displayed on the network client interface 26. The data engine 32 outputs a list of transactions that occurred during the alert (step S160). The list may be displayed at the network client interface 26 or may be printed as a physical copy.

Referring now to FIG. 8, an exemplary operational flow diagram is provided that describes steps performed to determine whether a cash register drawer may have been improperly opened. The video engine 20 detects that the cash drawer is open (step S162). The integration server 36 sends a query to the data engine 32 and/or the point of transaction terminal 30 to verify if any transaction occurred (step S164). If the transaction does occur (step S166), no alarm is required (step S168) and the process ends. However, if no transaction has occurred (step S168), an alert is generated (step S170), which may be displayed on the dashboard, saved in a database, and/or sent to the video recorder 18 and the network client interface 26. The integration server 36 receives an alarm information request requesting a video recorded at the time of alarm (step S172). The alert information request may be initiated, for example, by the user clicking on an alert displayed on the network client interface 26. The corresponding video is then played back (step S174), for example, using the dashboard, and the corresponding video is marked as "viewed" (step S176).

FIG. 9 provides an exemplary operational flow diagram that describes steps performed to set up POS rules and generate exception reports. In one embodiment, a retail store keeper or other corporate manager can use the dashboard to define POS data rules and key performance indicators ("KPIs") (step S178). For example, the rules may be as simple as editing a list of all returns that occur in a store or company, or just for a particular recorder and/or a particular employee and/or a particular product and/or a particular time. This provides the ability to perform complex data mining on any data type captured by the system. The data engine 32 queries the database of the point of transaction terminal 30 for the rule/KPI (step S180) and generates a KPI report (step S182) listing any anomalies for the rule/KPI.

FIG. 10 provides an exemplary operational flow diagram that describes steps performed to set up user definable video rules and generate alerts identifying violations. In a manner similar to that described above in relation to defining POS data rules as detailed in fig. 9, embodiments of the present invention also provide a means for setting up video analytics rules. The video analytics rules can be defined by a retail store manager, loss prevention professional, or other corporate administrator using the dashboard (step S184) and may include rules for alerting when any particular visual pattern, behavior, or content is detected. The video analytics rules are sent to the video engine 20 and any embedded edge devices (step S186). Video analysis alerts are generated each time the video engine 20 determines that at least one video analysis rule is violated (step S188).

FIG. 11 provides an exemplary operational flow diagram that describes steps performed by combining POS data rules with video analytics rules to accurately define specific alarm events. In this way, data intelligence and video intelligence are integrated to determine when a particular event occurs as defined according to the needs of the user. POS data rules are defined using a user interface such as a dashboard (step S190). The video analysis rules may also be defined using a dashboard (step S192). The applicable POS and video rules are selected (step S194) and combined using logical operations (e.g., and, or, not, if false, true, etc.) to produce user-defined conditions (step S196). The user-defined conditions are then run to generate real-time events or to perform post-hoc searches (step 198).

Referring now to FIG. 12, an exemplary operational flow diagram is provided that describes steps performed to generate reports for all rules/KPIs, alarms, and events. The desired rules, KPIs, events, and/or conditions are selected (step S200) and the duration and report format are specified (step 202). The integration server 36 selects POS data and video recordings corresponding to the selected rules, KPIs, events, and or conditions that occur within a specified duration to generate a report in a specified format (step S204). The report may be used to further investigate and identify suspicious activity and/or to improve overall store management.

From a security standpoint, software solutions may support auto-authenticated connections, such as integrated Windows authentication ("IWA"), also known as NT authentication. The security feature may restrict local application specific user IDs. Passwords should be used to access the system 10. While a LAN ID based license may be used, additional security features may also be used. Members in one or more active directory groups may be used. With active directory support, the user does not need to provide any additional authentication when launching the application. Security should be based on the identity of the currently logged-on workstation user, while the authorization verification is done automatically in the background. The application itself may have strong database security standards, with multiple levels of security applied to the database system as a whole, as well as to individual tables in the database.

The software provides the central server with an automatic run log and remote program failure (bug)/bug/problem reports. Program failures are automatically collected by software. End users can submit their own program failures via a website or through the application itself. All databases and records can be backed up and archived. The installation process of any application in the system of the present invention can be a silent, automatic installation on servers and workstations. The software deployment may be in accordance with standard scripting tools (e.g., SMS) and without user interaction. Remote configuration may also be used. The update may also be done remotely. The configuration process is user friendly, including but not limited to automatically detecting video recording devices in the LAN, and providing a graphical user interface for any configuration of all devices and components. The integration server 36 may be compatible with common enterprise server environments including, but not limited to, enterprise web servers, enterprise application servers, and enterprise database servers.

Other features of a system that may be embodied in the present invention include store gateways (store gateways) for: collecting video analytics alerts and count data, transmitting data to companies for transfer to databases in databases and server methods utilizing file transfer protocol ("FTP"), presenting video analytics alerts and confirmations at the store level, configuring video alerts via rules management tools or integrated interfaces, and presenting anomaly reports/data mining/trend analysis of POS data with video analytics and video verification.

The system 10 may also include artificial intelligence to distinguish alarms from exception reporting channels. Different examples illustrating video analysis include, but are not limited to, moving to unauthorized delivery areas, restricted stock areas, hiding merchandise, stopping or loitering for too long a time to indicate potential suspicious activity or need help, and crowds indicating potential suspicious activity. Different examples of exception reporting/trend analysis illustrating data and video analysis with a POS center include, but are not limited to, transactions that are invalid due to the absence of a customer, transactions that are invalid due to the absence of a manager, queuing, and people counting.

The system 10 may be programmable to allow for the definition and configuration of company-wide video analytics during initial installation at the store level. The system 10 may also include a store level solution that is programmable for managing addressing, database modification, transmission, and other store level video management functions. Data input may be taken from video surveillance and video analysis and integrated with mapping information (e.g., mapping between cameras and recorders/aisles).

Aspects of the database for the system may include using data transfer packets from video surveillance and video analytics, and mapping data. Some possible, expected data segments include, but are not limited to, count, date/time, rule ID, camera ID, and rule type (occupancy, etc.). The data mapping may include: store ID, organization ID, reference #, reference type (recorder, aisle, etc.), and activity type (consumer occupancy, item scan, etc.).

A time synchronization mechanism may be used to link POS data with video information, which may be similar to how a recorder synchronizes the time of POS data. System 10 may be constructed to allow video analytics rules to be managed (change control) at an enterprise-wide level, and not only at a store or specific area level. Rule management methods may be included which will facilitate initial configuration and future updates. One approach is to set up areas at the store level and apply rules at the company/enterprise level. In the field of transmission, the data may be located in a flat file or structured database located in a folder at the store level and collected over a network and transmitted to another location (e.g., a POS) with other data. The data can then be made available for database transfer. An alternative approach is to use an FTP based transfer mechanism.

The present invention advantageously provides a high degree of sensitivity/detectability with respect to revealing problem areas. Users can solve problems with employees and consumers more quickly through disciplinary chores, improvements in consumer services, or even training improvements (training improvements). By incorporating data sources and analysis sources into the automatable system of the invention, the output will be more reliable and accurate, and false alarms will be minimized or eliminated. False alarms can undermine confidence in the solution and limit its success.

The present invention can be realized in hardware, software, or a combination of hardware and software. Any kind of computing system, or other apparatus adapted for carrying out the methods described herein, is suited to perform the functions described herein.

A typical combination of hardware and software could be a specialized or general purpose computer system having one or more processing elements and a computer program stored on a storage medium that, when loaded and executed, controls the computer system such that it carries out the methods described herein. The present invention can also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which-when loaded in a computing system-is able to carry out these methods. Storage medium refers to any volatile or non-volatile storage device.

Computer program or application in the context of this document is any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following: a) conversion to another language, code or notation; b) replicate in different material forms.

Moreover, unless mention was made above to the contrary, it should be noted that all of the accompanying drawings are not to scale. Significantly, this invention can be embodied in other specific forms without departing from the spirit or essential attributes thereof, and accordingly, reference should be had to the following claims, rather than to the foregoing specification, as indicating the scope of the invention.

Claims (20)

1. A method for monitoring potential suspicious activity in a monitored facility, the method comprising:

collecting video content of an activity occurring in a monitored facility;

collecting transaction data relating to transactions processed at a point-of-transaction terminal;

associating the video content with the transaction data to produce associated data; and

applying a set of user-defined rules to the associated data; and

determining that the transaction is potentially suspicious in response to identifying a match between the associated data and at least one rule of the set of user-defined rules.

2. The method of claim 1, wherein the set of user-defined rules comprises a combination of one or more video analytics rules and one or more data analytics rules.

3. The method of claim 2, further comprising:

tagging the video content with a first timestamp indicating a time at which the activity occurred; and

tagging the transaction data with a second timestamp indicating a time at which the transaction was processed,

wherein the video content is associated with the transaction data by matching the first timestamp with the second timestamp.

4. The method of claim 3, wherein:

the data analysis rules include rules that determine that a return transaction has occurred; and

the video analysis rules include rules that determine that there are no consumers at the point-of-sale recorder.

5. The method of claim 3, wherein:

the data analysis rules include rules that determine that a cash transaction is invalid; and

the video analysis rules include rules that determine that there are no consumers at the point-of-sale recorder.

6. The method of claim 3, wherein:

the data analysis rules include rules that determine that no transactions have occurred; and

the video analysis rules include rules that determine that a drawer of a point-of-sale recorder is open.

7. A method of automatically identifying activities occurring at a monitored facility, the method comprising:

collecting video content of an activity occurring in the monitored facility;

analyzing the video content using a target recognition technique by applying a set of video analysis rules to the collected video content;

collecting transaction data relating to one or more transactions processed by at least one point-of-transaction terminal in the sales facility; and

in response to determining that the video content is consistent with at least one video analytics rule of the set of video analytics rules, associating the video content with the transactional data to provide associated transactional data.

8. The method of claim 7, further comprising:

tagging the video content with a first timestamp indicating a time at which the activity occurred; and

tagging the transaction data with a second timestamp indicating a time at which the transaction was processed,

wherein the video content is associated with the transaction data by matching the first timestamp with the second timestamp.

9. The method of claim 8, further comprising: generating an alert in response to determining that the video content is consistent with at least one video analytics rule of the set of video analytics rules.

10. The method of claim 9, further comprising: using the transaction data to determine why the video content is consistent with at least one video analytics rule of the set of video analytics rules.

11. The method of claim 9, wherein the at least one video analysis rule comprises a rule that determines that a number of consumers standing in a checkout queue exceeds a predetermined limit.

12. The method of claim 9, wherein the at least one video analysis rule comprises a rule that determines that a duration of time a consumer spends standing in a checkout queue exceeds a predetermined limit.

13. The method of claim 8, further comprising:

in response to determining that the video content is in accordance with at least one video analytics rule of the set of video analytics rules, generating a report detailing transactions that occurred when the video content is in accordance with at least one video analytics rule.

14. The method of claim 13, wherein the at least one video analytics rule comprises a rule that determines a number of consumers entering and leaving the sales facility.

15. A system for analyzing activity occurring in a monitored facility, the monitored facility including at least one point of transaction terminal, the system comprising:

a video analysis system operable to collect video content of activities occurring in the monitored facility;

a data analysis system operable to collect transaction data relating to one or more transactions processed by the at least one point of transaction terminal; and

an integration server communicatively coupled to the video analytics system and the data analytics system, the integration server operable to:

associating the video content with the transaction data to produce associated data;

applying a set of user-defined rules to the associated data; and

a match between the associated data and at least one rule of the set of user-defined rules is identified.

16. The system of claim 15, wherein the integration server is further operable to determine that the one or more transactions are potentially suspicious and generate an alert, the system further comprising a client interface communicatively connected to the integration server, the client interface operable to indicate the alert.

17. The system of claim 16, wherein the client interface is further operable to receive the set of user-defined rules comprising a combination of one or more video analytics rules and one or more data analytics rules.

18. The system of claim 17, wherein the video content includes a first timestamp indicating a time at which the activity occurred and the transaction data includes a second timestamp indicating a time at which the transaction was processed, the integration server further operable to associate the video content with the transaction data by matching the first timestamp with the second timestamp.

19. The system of claim 18, wherein:

the data analysis rules include rules that determine that a return transaction has occurred; and

the video analytics rules include rules that determine that there are no consumers at the point-of-transaction terminal.

20. The system of claim 18, wherein:

the data analysis rules include rules that determine that a cash transaction has been invalidated; and

the video analytics rules include rules that determine that there are no consumers at the point-of-transaction terminal.