GB2553123A

GB2553123A - Data collector

Info

Publication number: GB2553123A
Application number: GB1614425.5A
Authority: GB
Inventors: Albornoz Jose
Original assignee: Fujitsu Ltd
Current assignee: Fujitsu Ltd
Priority date: 2016-08-24
Filing date: 2016-08-24
Publication date: 2018-02-28
Also published as: GB201614425D0

Abstract

Data collector 10b is configured to receive images from an image inputter (10a; Fig 1), identify and track subjects in the images 202, determine physical properties of the subjects 203, classify the identified subjects 204 and detect incongruous situations 205. An activation signal may be sent to an alarm 206 in the event that an unexpected or suspicious situation is detected. There may be a de-identifier that allows people to be de-identified once their physical properties have been determined. There can be multiple image inputters (10a), such as video cameras. Data collector 10b may receive inputs from various sensors. The physical properties that are determined can include the age and gender of the person being tracked. The apparatus and method may be used to track individual customers in a shop environment, prisoners within a prison environment or members of public at a local event, such as a concert, convention or sporting event.

Description

(54) Title of the Invention: Data collector

Abstract Title: System for detecting incongruous situations (57) Data collector 10b is configured to receive images from an image inputter (10a; Fig 1), identify and track subjects in the images 202, determine physical properties of the subjects 203, classify the identified subjects 204 and detect incongruous situations 205. An activation signal may be sent to an alarm 206 in the event that an unexpected or suspicious situation is detected. There may be a de-identifier that allows people to be de-identified once their physical properties have been determined. There can be multiple image inputters (10a), such as video cameras. Data collector 10b may receive inputs from various sensors. The physical properties that are determined can include the age and gender of the person being tracked. The apparatus and method may be used to track individual customers in a shop environment, prisoners within a prison environment or members of public at a local event, such as a concert, convention or sporting event.

10c lOd image

Processor

Figure 2

10b

1/11

Figure 1

ι ·» ' ¹

2/11

Figure 2

3/11 f\l o

ΓΜ

Π3 (J

E ,2

Figure 3

CM

CO

CM

4/11 <Ό <Ζ) >.

-C ο_ >· <υ α_ ο

u.

Q.

<υ ιο

2>· (ϋ

C <

Figure 4

CM

Ο

5/11

X .5!
φ	Μ—
4-i C	'(Λ CO
Ο	TO
u	U

Figure 5

6/11

co ο

ID

fO		in
o	o	o
<D	o	o
ω	ω	ω

7/11

8/11

Figure 8

9/11 o

ω

Figure 9

10/11

ΓΟ ο

σ» ω

ο

Ch ω

Figure 10

ο cn ω

ο

Οϊ ω

<QN3

11/11

Figure 11

ΓΝ

ΟΊ cn

Data Collector [001], The invention relates to a data collector, an image analysis method, a computer-readable medium comprising a computer program, and an image analysis system [002]. The current growth of online shopping has impacted on the profitability of many companies primarily based around physical (brick-and-mortar) shops. Online shops have clear advantages in terms of the convenience for the user; allowing shopping to be performed at any time from almost any location. However, another advantage that online shops have over physical shops results from the inherent technical requirements of online commerce.

[003], The use of cookies, the tracking of IP addresses and the use of customerspecific accounts allows online stores to gather and retain a significant amount of information about users, with specific reference to their shopping habits. It is common for users to complete profile information (either directly or through a link to a social network platform) when using an online store. Online shops typically also record information on each item a user views, as well as information on the items actually purchased. This information allows an online stores to construct detailed profiles of users, which may be used to provide tailored recommendations of products or services a user may be interested in purchasing.

[004], Typically, companies which operate physical shops do not have access to the same amount of detailed information about their customers. Some companies operating physical shops have introduced loyalty cards in an attempt to track the purchasing behaviour of customers. However, while loyalty cards may provide a record of, for example, what a customer buys, the cards require active input from the customers, and it is common for transactions to not be recorded on a loyalty card either because the customer does not possess a loyalty card, because the customer does not have a loyalty card currently in their possession, or because a customer chooses not to use a loyalty card. Also, loyalty card based systems cannot provide information on what products a customer may have viewed but not purchased.

[005], Companies operating physical shops therefore face problems when attempting to obtain information about their customers, particularly real time information regarding customer ages and genders, or the social context of a customer visit. This information may be useful to physical shops both in terms of long-term inventory selection or shop design, and also in terms of short term sales. As an example of this, a sales person may be directed to suggest different products to a couple, an individual, a group of friends, and so on.

[006], Further, the amount of custom a physical shop may expect on a particular day may be determined by various factors that are not within the control of the company that owns the shop. Examples of such factors include the local weather, large gatherings such as concerts or sports events in the vicinity of the store, and so on. These factors would generally be expected to have less of an influence on the custom levels of online shops, which do not require a customer to actually visit a particular location in order to make a purchase.

[007], Physical shops also face different problems to online shops in terms of abnormal customer behaviour. Physical shops may have to deal with issues such as lost children, thefts, physical violence or other crimes, for example. It may be difficult for the owners of physical shops to detect instances of any of the above issues in sufficient time to address the issue in a positive way (for example, catching a thief or reuniting a lost child with a responsible adult).

[008], Other physical spaces (in addition to shops) may also encounter the same or similar problems in terms of abnormal behaviour, and also in terms of tracking and monitoring the movements and behaviours of persons in the physical spaces. Examples of physical spaces that may encounter problems in this regard include public spaces in towns and cities, schools, prisons, offices, and so on. The recent rise in acts of violence in public spaces (for example, terrorist attacks) have made detecting incongruous situations and analysing or (ideally) preventing abnormal behaviour a more pressing issue for those responsible for overseeing the safety and security of public spaces.

[009], Currently, shops and other physical spaces may attempt to collect information on the motivations and profiles of persons entering the shop or other physical space by having employees present the persons in the shop or other physical space with questionnaires, or by having the employees survey behaviour and movements. However, this has implications for the privacy of the persons in the shop or physical space, cannot provide rapid analysis and results, may irritate persons in the shop or other physical space and is an inefficient use of staff resources.

[0010]. In all of the physical spaces listed above, including shops, it is not uncommon for video cameras or still image cameras to be present. However, the mere presence of cameras is not sufficient to provide the necessary information; it is also necessary to monitor and analyse the images captured by the cameras to extract useful information. This monitoring and analysis may be used after an event has occurred (for example, when preparing an analysis of customer behaviour over the course of a day or when examining footage for evidence of a crime), or may be an ongoing process with the aim of detecting an incongruous situation and addressing the situation before negative consequences occur (for example, identifying a lost child before the child has left the monitored area). This monitoring is usually performed by human operators, however it may be difficult for human operators to simultaneously monitor a large number of cameras (potentially each of which is capturing images of a large number of people). Also, and particularly in security monitoring applications, it may be difficult for human operators to maintain concentration for extended periods of time.

[0011]. US 2009/0299814 A1 discloses a system for assessing the personality and mood of a customer on the basis of a captured image of the customer. The system disclosed therein analyses customer gait and facial expression in an attempt to discern customer emotional state, and uses this information to suggest a strategy to sales people.

[0012], It is desirable to address some of the above issues.

[0013], According to an aspect of an embodiment of the invention, there is provided a data collector for receiving images from an image inputter, the data collector comprising an image processor, the image processor comprising an image tracker configured to identify subjects in the received images and track the identified subjects between the received images, wherein the image tracker is configured to output the results of the identifying and tracking as identification and tracking information to a physical property analyser, the physical property analyser being configured: to receive the identification and tracking information and the received images; to analyse the identification and tracking information and the received images to determine physical properties of the identified subjects; and to transmit the received images, the identification and tracking information and the determined physical properties to a context classifier, and the context classifier being configured: to receive the received images, the identification and tracking information and the determined physical properties; to compare the received images, the identification and tracking information and the determined physical properties for the identified subjects against predetermined classification information in order to classify the identified subjects; and to output the received images, the identification and tracking information, the determined physical properties and the generated classification information for the identified subjects to a situation analyser, and the situation analyser being configured; to perform a situation analysis by comparing the outputs from the image tracker, physical property analyser and/or context classifier to predetermined situation types, wherein the situation analyser is further configured to receive sensor inputs from one or more sensors and/or data from databases, and to incorporate the sensor inputs and/or data into the situation analysis; and to detect an incongruous situation on the basis of the situation analysis. The apparatus provides an efficient mechanism for collecting and analysing data on subjects in a physical area without requiring conscious cooperation from the subjects, and for detecting incongruous situations rapidly, accurately and without the requirement for user input.

[0014]. A further aspect of an embodiment of the invention provides a data collector, wherein the situation analyser is further configured to send an activation signal to an alarm if the situation analysis indicates that an incongruous situation has been detected. This aspect of an embodiment of the invention provides a mechanism for raising an alert in the event that an incongruous situation is detected, thereby helping to prevent negative outcomes.

[0015], A further aspect of an embodiment of the invention provides a data collector, wherein the image processor utilises convolutional neural networks. The use of neural networks in this aspect of an embodiment of the invention provides a rapid and accurate analysis mechanism.

[0016], A further aspect of an embodiment of the invention provides a data collector, wherein the physical property analyser includes a de-identifier configured to deidentify the identified subjects once the physical properties of the identified subjects have been determined. This aspect of an embodiment of the invention helps to ensure the privacy of identified subjects, and may also address potential legal issues in some jurisdictions.

[0017], A further aspect of an embodiment of the invention provides a data collector, wherein the de-identifier may be deactivated by an authorised user, so that the identified subjects are not de-identified. This aspect of an embodiment of the invention allows the privacy of identified subjects to be provided in most situations, while also allowing the desire for privacy to be overridden in exceptional situations or where there are pressing needs to do so. This results in a versatile system that is suitable for a wide range of situations.

[0018], A further aspect of an embodiment of the invention provides a data collector, wherein the context classifier is configured to detect subjects in an abnormal group type. This aspect of an embodiment of the invention provides an economical and simple way to implement basic safety provisions in the data collector.

[0019]. A further aspect of an embodiment of the invention provides a data collector, wherein the context classifier is configured to send an activation signal to an alarm if the context classifier detects an abnormal group type. This aspect of an embodiment of the invention provides a mechanism for rapidly raising an alert in the event that an abnormal group type is detected, thereby helping to prevent negative outcomes.

[0020]. A further aspect of an embodiment of the invention provides a data collector, wherein the data collector is configured to receive images from a plurality of image inputters, each image inputter being a video camera, and wherein the data collector is configured to receive the images from each image inputter in the form of a video stream. This aspect of an embodiment of the invention allows a larger physical space to be monitored accurately by the system, and also improves the tracking of individual subjects.

[0021], A further aspect of an embodiment of the invention provides a data collector, the data collector being further configured to transmit the determined physical properties and/or the classification information to a data storage unit for incorporation into a subject data set. This aspect of an embodiment of the invention ensures that collected data is subsequently available for analysis as required.

[0022]. A further aspect of an embodiment of the invention provides a data collector, wherein the subject data set is used by a data processor to generate periodic reports. This aspect of an embodiment of the invention ensures that employees working in relation to a given physical space are kept informed of the status of that physical space.

[0023], According to further aspects of embodiments of the present invention there are provided methods for implementing the functions described above in the context of the aspects of the embodiments of the data collector. The methods provide the same advantages as described above with reference to the aspects of the embodiments of the data collector.

[0024], According to an aspect of an embodiment of the present invention, there is provided a computer-readable medium comprising a computer program which, when executed by a computing apparatus, causes the computing apparatus to execute a method as described above. This provides the advantage of an easy and efficient way to implement a method of the present invention.

[0025]. The invention will now be further described, by way of example only, with reference to the following figures, in which;

[0026], Figure 1 is a schematic representation of an image collection and analysis system comprising a data collector in accordance with an aspect of an embodiment of the invention.

[0027J. Figure 2 is a schematic representation of the data collector in accordance with an aspect of an embodiment of the invention.

[0028], Figure 3 is a schematic representation of the image tracker in accordance with an aspect of an embodiment of the invention.

[0029], Figure 4 is a schematic representation of the physical property analyser in accordance with an aspect of an embodiment of the invention.

[0030], Figure 5 is a schematic representation of the context classifier in accordance with an aspect of an embodiment of the invention.

[0031]. Figure 6 is a flowchart illustrating the processing performed by the image tracker, physical property analyser and context classifier in accordance with an aspect of an embodiment of the invention.

[0032], Figure 7 is a schematic representation of the situation analyser in accordance with an aspect of an embodiment of the invention.

[0033]. Figure 8 is a flowchart illustrating the processing performed by the situation analyser in accordance with an aspect of an embodiment of the invention.

[0034], Figure 9 is a flowchart illustrating the processing performed by the data merger in accordance with an aspect of an embodiment of the invention.

[0035]. Figure 10is a flowchart illustrating the processing performed by the data processor in accordance with an aspect of an embodiment of the invention.

[0036], Figure 11 is a schematic representation of a computing device that may be used to implement aspects of embodiments of the present invention.

[0037], The schematic representation in Figure 1 illustrates an image collection and analysis system that incorporates a data collector 10b of an aspect of an embodiment of the present invention. The image collection and analysis system shown in Figure 1 also includes an image inputter 10a; in the example shown in Figure 1 the image inputter 10a is a video camera, however the data collector may also receive images from still image cameras. The data collector 10b may also receive inputs from various types of sensors, including microphones, thermometers, hygrometers, or other useful sensor types as may currently exist or as may be introduced as technology advances. Some aspects of embodiments of the invention also receive external data updates from an external database or data store 10c in relation to local events, including weather updates, information relating to gatherings such as concerts, conventions, sports events, and so on. The external data may be provided by an internet connection. The image collection and analysis system may also incorporate a user interface 10d.

[0038]. The data collector 10b is shown in greater detail in Figure 2. Although the data collector 10b may function using only a single image inputter 10a, the data collector 10b is more effective when used to track the progress of subjects through a physical space (such as a shop, a public square, a prison, a school, or another physical space) using a plurality of image inputters 10a. This is because it can be difficult to obtain sufficient information on a subject using the feed from a single image inputter 10a, particularly in the event that the image inputter 10a is a still image camera or is not capable of tracking a subject. It is easier to obtain sufficient information for the system to generate high quality results if the image inputter 10a is a video camera, and easier still if a plurality of video cameras covering a physical space from a plurality of angles are used.

[0039], Stationary mounted video cameras that are attached to walls, pillars, posts, ceilings, etc. are typically used as image inputters 10a. However, other types of cameras can also be used as image inputters 10a. In particular, the data collector 10b can receive images from body mounted cameras (as often employed by security personnel), or from other portable cameras such as mobile phone cameras. When body mounted cameras are used as image inputters 10a, this can be particularly useful if the system is being used to detect a developing incongruous situation such as a person suspected of having an explosive device, as the feed from the body mounted cameras can be useful when directing security personnel. When mobile phone cameras are used as image inputters 10a, this is typically for use in an analysis after an event has occurred. For example, where a crime has taken place, mobile phone footage from witnesses can be inputted into the system for use in the analysis.

[0040]. The data collector shown in Figure 2 is configured to receive input images from a plurality of video cameras, the input images being provided in the form of a series of video streams in MPEG-4 format. Any suitable video file format may be used, and any suitable image file format may be used for still image inputs.

Examples of suitable video file formats include Flash Video files, Windows Media Video files, MPEG-4 files, and so on. Examples of suitable image file formats include JPEG, TIFF, PNG, GIF and BMP files, but other image file formats may also be used.

[0041], The images from the image inputters 10a are fed into an image processor 200 in the data collector 10b, which is responsible for the initial analysis of the input images. The image processor 200 comprises an image tracker 202, a physical property analyser 203 a context classifier 204, and a situation analyser 205.

[0042]. Figure 3 shows a schematic of the image tracker 202, comprising the image combiner 21, subject recogniser 22, subject labeller 23 and subject tracker 24. The image tracker 202 receives the image from the image inputters 10a. Where several image inputters 10a are used in conjunction (as in the present embodiment), the image tracker 202 analyses the images in conjunction. In order to allow the images to be analysed in conjunction, an image combiner 21 is used to form a combined image data set. The images are typically arranged in the combined image data set in chronological order. In the present embodiment, the image inputters 10a are a plurality of video cameras, both stationary mounted cameras which feed data to the data collector 10b using a wired connection and body mounted cameras which feed data to the data collector 10b using a wireless connection. Accordingly, the input images are in the form of a series of video streams. The image combiner 21 combines the video feeds such that the stream from each camera for a given moment in time may be analysed simultaneously. When the system is configured to use a single image inputter 10a, the image combiner 21 is not required to combine the input images.

[0043]. Individual subjects, such as customers in a shop, pedestrians in a public space, prisoners in a prison, schoolchildren in a school, and so on, are recognised by the image tracker 202. The recognition is performed by the subject recogniser 22, which is configured to recognise a person (individual subject), using standard image processing techniques such as vector analysis, in the input images that are received from the image combiner 21. The subject recogniser 22 recognises the presence of a generic individual subject (that is, a person), but is not configured to specifically identify the individual subject.

[0044], Once one or more individual subjects has been recognised by the subject recogniser 22, the individual subject(s) are each assigned a numerical identification. This numerical identification is unique to an individual subject (in the context of the system) and is used to refer to the individual subjects. The numerical identifications are assigned by the subject labeller 23 in the image tracker 202, typically sequentially in the order that the subject recogniser 22 recognises the individual subjects, although random numbers may also be used. The numerical identifications are not limited to numbers, and may also contain other characters, such as letters or punctuation. In the present embodiment, an alphanumeric code is used.

[0045]. In some aspects of embodiments the individual subject identifications are linked with other records (tor example, store card records, national identification information, passport records, police records, and so on). In the event that another identification is established for an individual subject as a result of this linking, the numerical identification may be replaced in the system with this other identification, or the identifications may be linked and maintained in parallel. Further, even in the event that a subject is identified using other records, it is possible for the system to de-identify subjects (as discussed below), in which instance the numerical identification may be retained as a suitably anonymous way of referring to a subject.

In embodiments of the invention that use linking, this process is performed by the physical property analyser 203, however, the present embodiment does not use linking.

[0046]. Once an individual subject has been assigned a numerical identification by the subject labeller 23, the subject tracker 24 of the image tracker 202 analyses the input images and tracks the movements of the individual subject between images. In the present embodiment where a plurality of video streams are provided and combined by the image combiner 21, the individual subject is tracked by the subject tracker 24 across the combined video image streams. A video stream in this context is treated as a series of temporally separated captured images (frames of a video). Any images (frames) containing a tracked individual subject are labelled with identification and tracking information in such a way as to indicate the presence of that individual subject. In the present embodiment the labelling is performed by the subject tracker 24 by including the numerical identifier of the individual subject in the metadata of the images, however other labelling methods may also be used. For example, the images may be edited to display the relevant numerical identifier or identifiers in the relevant area of the image where the individual subject has been located, or in an additional section of the image that is reserved for this purpose (such as the border of the image).

[0047], For each individual subject, the images identified as showing the individual subject and the identification and tracking information are then passed from the image tracker 202 to the physical property analyser 203. Although the system may be configured to cache images showing an individual subject (for example, in the image tracker 202 or a linked memory) and related identification and tracking information before sending the images together to the physical property analyser 203, the present embodiment is configured to transmit images and identification and tracking information to the physical property analyser 203 as soon as the individual subjects have been identified and tracked in the images by the image tracker 202.

[0048], The physical property analyser 203 receives the images and information from the image tracker 202. The physical property analyser 203 is capable of identifying the time and location at which an image has been captured using the metadata of the image. This allows the physical property analyser to reconstruct the movements of an individual subject. The physical property analyser 203 then analyses the images and estimates the physical properties of the individual subject.

[0049], The physical properties include the age and gender of the subject. In the present embodiment, the age of the individual subject is estimated by determining a 10 year band in which the individual subject age is believed to lie. The subject gender is typically determined to be male or female, however the physical property analyser

203 may also be configured to return an other/indeterminate result in cases where the individual subject gender cannot easily be determined. The physical property analyser 203 may also be configured to provide a confidence level, that indicates how reliable an estimate is, when estimating the physical properties of an individual subject.

[0050]. A schematic of the physical property analyser 203 is shown in Figure 4. The determination of the gender of the individual subject is performed by a gender inferer 301 within the physical property analyser 203. Similarly, the age group of the individual subject is determined by an age group inferer 302. The embodiment of the physical property analyser 203 shown schematically in Figure 4 also includes the deidentifier 303; an optional feature which is discussed in greater detail below.

[0051]. In order to estimate the age and gender of the individual subject, the physical property analyser 203 takes into consideration various observed characteristics of the individual subject. These characteristics may include facial features, silhouette, gait, hairstyle, attire, size, and so on. The physical property analyser 203 may also take into consideration, when determining the age and gender of a given individual subject, determined properties of further individual subjects that are in the same image as the given individual subject. In the event that one or more of the characteristics cannot be analysed (for example, the facial features cannot be analysed because a clear image of the face of the individual subject has not been obtained), the analysis proceeds using the available characteristics.

[0052]. The physical property analyser 203 of the present embodiment utilises a convolutional neural network to perform the analysis. The use of neural networks is discussed in greater detail below.

[0053], As illustrated in Figure 4, the physical property analyser 203 may include a de-identifier 303. The de-identifier 303 is configured such that, once the age and gender of an individual subject have been estimated as described above, the identity of the individual subject is obscured. The de-identifier 303 may therefore be useful if it is desired to maintain the privacy of individual subjects, both for the comfort of the individual subjects and because this may be legally required for some applications of the system in some jurisdictions.

[0054]. The primary identifying feature of a person that is typically captured using image capturing systems is the facial appearance of the person. Accordingly, the deidentifier 303 is configured to obscure the facial features of the individual subjects. The facial features of an individual subject may be obscured by pixelating or blurring the area of each image that shows the face of the individual subject. Alternatively or additionally, a mask may be applied to each image.

[0055]. Although the facial appearance is typically the primary visual identifying feature of an individual subject, other characteristics may also be used to identify a subject. Examples of other distinguishing features may include distinctive hairstyles and clothing. If it is desired to provide a greater degree of certainty that the identity of an individual subject has been obscured, then further de-identifying measures may also be applied by the de-identifier 303. The head or entire body of an individual subject may be obscured using pixilation, blurring or masking. Each image may be modified to display an outline silhouette of an individual subject or, for a still greater degree of anonymity, an area of the image surrounding an individual subject may be masked using a simple geometric shape (such as a rectangle or oval) so that the outline silhouette of the individual subject is also obscured. Each image in which an individual subject is included may be marked by the numerical identifier, either on the image itself or in the image metadata.

[0056]. The de-identifier 303 is intended to ensure the privacy of individual subjects. However, in some circumstances, the desire to allow individual subjects privacy is overridden by other requirements. For example, it may be necessary to definitively identify individual subjects in situations involving criminal matters, such as thefts from a shop. Further, the system may be installed in a location wherein it is not desired to obscure the identity of individual subjects at all, such as a prison. Therefore, the deidentifier 303 may be configured such that a user having authorisation may deactivate the de-identifier 303, such that individual subjects may more easily be identified. The deactivation may be performed in advance (so that the deidentification is not applied to the images), or alternatively may be retroactively applied (so that images that have previously been modified by the de-identifier 303 revert to their original form in which the individual subjects may more easily be identified). The deactivation may be applied selectively, such that only a particular individual subject or group of individual subjects are not de-identified. The deactivation may also be globally applied, such that none of the individual subjects are de-identified.

[0057], As shown in Figure 2, the images identified as showing an individual subject, the identification and tracking information and the determined physical properties are then passed from the physical property analyser 203 to the context classifier 204. The system may be configured to cache the images identified as showing an individual subject, the identification and tracking information and the determined physical properties relating to an individual subject (for example, in the physical property analyser or a linked memory) before sending the images together to the context classifier 204. However, in the present embodiment transmission to the context classifier 204 occurs as soon as the physical properties have been determined (and the images have been processed by the de-identifier 303, if this processing is to be implemented).

[0058], A schematic of the context classifier of the present embodiment is shown in Figure 5. The context classifier 204 receives the images identified as showing an individual subject, the identification and tracking information and the determined physical properties and analyses the same. The analysis may be performed using a neural network, which may be the same neural network as is used by the physical property analyser 203 if the physical property analyser 203 uses a neural network, or a different neural network.

[0059]. As understood in the art, an artificial neural network is a computational model that may be used to perform analysis and classification. The neural network may therefore act in one or more ofthe physical property analyser 203, the context classifier 204 and the situation analyser 205. The neural network may be implemented using a series of hardware units, but the present embodiment utilises a software implemented neural network.

[0060], The neural network analyses the provided information as a whole, rather than dividing the information for separate consideration. The neural network is intended to capture key characteristics of the information, without giving undue weight to small differences. This is particularly relevant in the case of the context classifier 204 and the situation analyser 205, wherein the disposition of one or a group of individual subjects may be compared en masse to predetermined classification information (in the case of the context classifier 204) or predetermined situation types (in the case of the situation analyser 205).

[0061]. Alternatively, the necessary analysis may be performed using feature detection techniques (that is, edge detection, corner detection, and so on) such as Sobel filter techniques or LoG techniques, as are common in the art.

[0062]. The context classifier 204 first identifies groupings of individual subjects using a group detector 401. The groupings of individual subjects are identified by analysing the proximity of individual subjects to one another, whether tracking of individual subjects indicates that the subjects are following the same or a similar path around a space, whether tracking ofthe eye position ofthe individual subjects indicates that the individual subjects look at each other repeatedly, and so on. Other non-visual information may also be used if available, such as audio information that indicates individual subjects are talking to one another.

[0063], Once groupings have been identified by the group detector 401, the grouping information is passed to the group classifier 402, as shown in Figure 5. The group classifier 402 comprises a database of predetermined classification information. The identified groupings are compared against this predetermined classification information and the nature of the identified groupings are determined. Examples of potential groupings include individual persons, couples, groups of friends, parents with children, and so on. The group classifier uses the age and gender estimates provided by the physical property analyser 203 as the primary determining factor when identifying the nature of the groupings. However, the other information transmitted from the physical property analyser 203 may also be used in the determination.

[0064]. Once a classification for a group has been determined, the group is given an appropriate label by a group labeller 403. The group labeller 403 is shown in Figure 5. The group labels are used to identify the general type of group that has been identified. A group type may encompass several different group compositions. For example, a group identified as consisting of a female aged between 30 and 40 years, a male child of between 5 and 10 years and a female child of between 10 and 15 could be identified as a family group type. However, a group consisting of a male between 40 and 50 years, a female of between 40 and 45 years and a female of between 10 and 15 years could also be labelled as a family group type. The group types that the group labeller 403 seeks to identify are determined by the situation in which the data collector 10b is being used. For example, if an embodiment of the system is used in the context of a prison, the group labeller 403 may be configured not to search for family groups.

[0065]. An overview flowchart of the processing performed by the image tracker 202, physical property analyser 203 and context classifier 204 of the present embodiment is shown in Figure 6. In step S601, a plurality of video feeds are received from the image inputters 10a at the image tracker 202 of the image processor 200. The video feeds are combined by the image combiner 21 in step S602. In step S603, the subject recogniser 22 recognises the individual subjects in the combined image data. The recognised subjects are assigned a numerical identifier by the subject labeller 23, and then tracked between the images by the subject tracker 24, in step S604. The images and the identification and tracking data are then sent to the physical property analyser 203.

[0066], The physical property analyser 203 receives the images and the identification and tracking data, and the gender inferer 301 and age group inferer 302 estimates the age and gender of the individual subjects, in step S605. Then, in the present embodiment, the status of the de-identifier 303 is checked at step S606. If the deidentifier is 303 is active (YES in step S606), the individual subjects are then de14 identified in step S607, before the images are passed to the context classifier 204 in step S608. If the de-identifier 303 is not active (NO in step S606), step S607 is skipped and the images are passed directly to the context classifier 204 in step S608 without undergoing de-identification.

[0067]. In step S608, the group detector 401 of the context classifier 204 identifies groupings of individual subjects. Then, in step S609, the group classifier 402 identifies the nature of the groupings. In step S610 the group is labelled by the group labeller 403. The images, identification and tracking information, determined physical properties and classification information is then passed on by the image processor 200 to a data store 207 in step S611, where the data is stored for further analysis in step S612. The processes performed from step S612 are discussed with reference to Figure 9 below.

[0068]. The present embodiment includes a situation analyser 205, as shown in Figure 2. A schematic diagram of the situation analyser 205 is shown in Figure 7, and a flowchart of the processes performed by the situation analyser 205 is shown in Figure 8.

[0069]. The situation analyser 205 receives at least the images identified as showing an individual subject and the identification and tracking information from the image tracker 202. The situation analyser 205 may also receive the determined physical properties from the physical property analyser 203 and/or the group label from the context classifier 204. The information received by the situation analyser 205 may be transmitted directly from the image tracker 202, physical property analyser 203 and/or context classifier 204, or may be transmitted via one of the other components. That is, the situation analyser 205 may receive (for example) the physical properties from the context classifier 204, wherein the context classifier initially received the physical properties from the physical property analyser 203. The information from the image tracker 202, physical property analyser 203 and/or context classifier 204 is received by the situation analyser 205 at step S701 in Figure 8.

[0070]. The situation analyser 205 may also receive inputs from other sensors 503, such as microphones, thermometers, hygrometers, or other useful sensor types as may currently exist or as may be introduced as technology advances. The sensor inputs are received by the situation analyser at step S704 in Figure 8. The embodiment of the situation analyser 205 shown in Figure 7 has an incorporated thermometer, however the sensors 503 need not be incorporated in the situation analyser 205 and separate sensors 503 may alternatively or additionally be used. In particular, in aspects of embodiments of the invention wherein the functions of the situation analyser 205 (and potentially all of the other components of the image processor 200) are executed by a computer program, the situation analyser 205 does not incorporate sensors 503. The situation analyser 205 may also receive external data updates from an external database or data store 10c in relation to local events, including weather updates, information relating to gatherings such as concerts, conventions, sports events, and so on. The external data may be provided by an internet connection, either a wired or a wireless connection.

[0071], As shown in Figure 7, the situation analyser 205 may also incorporate a convolutional neural network 501, as are known in the art. A non-limiting example of a convolutional neural network used in image analysis and suitable for use in the present invention is discussed in “ImageNet Classification with Deep Convolution Neural Networks” by Krizhevsky, A. et al., which is available at http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf as of 12 August 2016.

[0072], The convolutional neural network 501 used in the situation analyser 205 may be the same neural network as is used by the physical property analyser 203 and/or the context classifier 204, or may be a separate neural network. In the embodiment shown in Figure 7, the convolutional neural network 501 is used exclusively by the situation analyser 205, and is incorporated within the situation analyser 205.

[0073]. The information provided by the image tracker 202, physical property analyser 203 and/or context classifier 204 and received by the situation analyser 205 is analysed to identify incongruous situations. In the present embodiment, the analysis is performed by the convolutional neural network 501 at step S702 as shown in Figure 8. Incongruous situations in the context of the present invention are any situations which are not in keeping with what would be expected for the environment in which the data collector 10b is operating and, by definition, will depend to some extent on what the operating environment is. That is, an incongruous situation can be defined as a conjunction of factors (location, people, weather, etc.) which would occur infrequently and which could result in a negative outcome (such as damage or loss to persons or property). Examples of incongruous situations which could be detected by the situation analyser 205 include lost children (that is, children below a predetermined age threshold that are identified as individuals by the context classifier), thefts, terrorist attacks, and so on. Of particular relevance in the present social and geopolitical climate are security applications of the data collector, whereby the system may be used to detect individuals behaving or dressed in an unusual fashion, e.g., in light of the surrounding environment, who may be planning an attack on a civilian population.

[0074], The incongruous situations that the situation analyser 205 detects are dependent on the specific combination of sensor read outs and information that is available to the situation analyser 205, and also on the context in which the data collector 10b is being used. For example, a data collector 10b that is operating in a shopping centre is quite likely to detect a lost child at some point, and will be configured to detect this situation. However, a data collector 10b that is operating in a prison is not likely to detect a lost child, and may not be configured to detect this situation.

[0075], Some situations detected by the situation analyser 205 may be detected purely using the information provided by the image tracker 202, physical property analyser 203 and/or context classifier 204. However, other situations that the situation analyser 205 may be configured to detect require other information to be detected. For this reason the embodiment of the situation analyser 205 shown in Figure 7 also comprises a subject timer 502. The subject timer 502 receives the same information as the convolutional neural network 501. The subject timer 502 is used to time how long an individual subject or group of subjects remains in a particular location or environment, as shown in step S703 in Figure 8. This timing information may be useful for detecting some situations, for example, a lone individual that is identified as remaining immediately outside a door to a non-public area in a shop for a significant period of time may indicate that the individual is attempting to pick a lock on the door, or is loitering with the intent of passing through the doorway when the door is opened by a member of staff.

[0076]. Detection of some incongruous situations requires a plurality of pieces of information to be present, to analyse the behaviours/characteristics in the context of external factors such as the surrounding environments (location, weather, time of occurrence, etc.); for example, as discussed above, the situation of a lone individual remaining immediately outside a door to a non-public area in a shop for a significant period of time. This situation requires the identification of the lone individual and the location of the individual (by the convolutional neural network 501) and the duration of time the individual remains in that location (that is identified using the subject timer 502). The detection of incongruous situations from plural inputs is performed by a combiner 504 and a classifier 505, as illustrated by step S705 in Figure 8.

[0077], In the present embodiment, the combiner 504 receives inputs from the convolutional neural network 501, the subject timer 502 and the sensor 503 (in this embodiment, a thermometer). The combiner 504 combines these inputs and transmits the combined information to the classifier 505 at step S706. The classifier 505 then classifies the situation at step S707. If the situation is identified as incongruous (YES at step S707), further action is taken. In the present embodiment, the situation analyser 205 is connected to an alarm 206, and is configured to send an alarm activation signal to this alarm 206 in the event that an incongruous situation is detected, as shown in step S708 of Figure 8. The situation analyser 205 may also be configured, in the event that an incongruous situation is detected, to record the information received, to create a log entry, to send a notification by emails or telephone to a designated staff member, and so on. After the alarm and/or other actions have been activated, the process ends, as shown in step S709 of Figure 8. Alternatively, if no incongruous situation is detected (NO at step S707), then the process ends as shown in step S709 without the alarm and/or other actions being activated.

[0078]. To further explain how the situation analyser 205 operates, an example of an incongruous situation in a public space, specifically a public square in a city, detected utilising the convolutional neural network 501, subject timer 502, sensor 503 (in this instance, a thermometer) and external data source 10c (not shown in figure 7) of the present embodiment will now be described. The data collector 10b is being used to monitor a series of image inputters 10a that are video cameras and that are installed in the public square. The convolutional neural network 501 identifies a lone male of 20-30 years in age using the determined physical properties from the physical property analyser 203 and the group label from the context classifier 204, and further identifies that the lone male is wearing bulky clothing and walking with an unusual gait using the images and information from the image tracker 202. The subject timer 502 indicates that the subject has been in the same position for 15 minutes . The thermometer 503 indicates that the current temperature is 27°C. The external data source 10c provides the forecast and current weather for the city, both of which indicate warm and sunny weather. The information is combined at the combiner 504. The classifier 505 identifies an incongruous situation of a potential terrorist carrying explosives, because the identified subject fits a likely demographic profile for terrorists, has been loitering in the public square for some time, is wearing clothing that is not suitable for the predicted or actual weather, and is walking with a gait suggesting that he is encumbered (potentially due to explosives concealed within the bulky clothing). A message is sent to the computer screen of a security guard located in a nearby non-public area; the message contains an image of the identified subject and brief details of the incongruous situation. An activation signal is also sent to an alarm 206 in the non-public area where the security guard is located.

[0079]. In addition to the alarm activation signal that may be sent by the situation analyser 205, aspects of embodiments of the invention may also be configured such that the context classifier 204 is connected to an alarm 206 and may send an alarm activation signal. The alarm may be the same alarm that the situation analyser 205 is connected to or a different alarm. The context classifier 204 may be configured to trigger the alarm 206 and/or send another type of message or alert (as discussed above in the context of the situation analyser 205) when certain abnormal types of group are identified. An abnormal group type may be defined as any type of group which are not normally present in the environment in which the system is operating. As an example of this, the context classifier 204 may be configured to send an alert when a lost child group type (wherein the individual subject is a child between 0 and 10 years old that is not with any other subjects) is detected. The context classifier 204 may make this detection without requiring access to any sensor feeds or data from an external data source 10c (as may be used by the situation analyser 205), and therefore may provide a quick and simple mechanism for detecting abnormal group types.

[0080], The images identified as showing an individual subject and the identification and tracking information from the image tracker 202, the determined physical properties from the physical property analyser 203 and/or the group label from the context classifier 204 may be transmitted to the data store 207 and stored therein. In the present embodiment, the images and tracking information are not retained in the data store 207; the physical properties and the group label are stored. The storage is indicated by step S611 in Figure 6. The data store 207 is shown in Figure 2. The data stored in the data store 207 is retained for later analysis or consideration by staff members.

[0081], In the embodiment of the invention shown in Figure 2, external data is provided from an external data source 10c and is transmitted via a wireless or wired connection to the data collector 10b. In addition to being used by the situation analyser 205, the external data is also used in a data combining process in the embodiment of the invention shown in Figure 2. The data combining process is illustrated by flowchart in Figure 9. The data merger 208 retrieves the stored data from the data store 207 at step S801. When the external data arrives at the data collector 10b, it is sent to a data merger 208 at step S802. The stored data is combined with the external data from the external data source to create a merged dataset 209 at step S803. The merged dataset 209 is then stored at step S804. At step S805, the merged dataset 209 is analysed, as discussed below.

[0082]. The data in the data store 207 may be used by a data processor 210 to generate periodic reports 211, without requiring any external data. The periodic reports 211 may indicate, for example, the number of customers in a shop on a given day having a particular group label, and may also be used to determine if certain areas of a shop are more attractive to customers than other areas. The periodicity of the reports may be set depending on the specific use of the reports. As an example of potential periodicity settings, the reports may be set to be generated annually if a general overview of the performance of a shop is desired, or weekly if a shop has reorganised a display and the owner wishes to evaluate the attractiveness of the new display to customers. In addition to generating periodic reports 211, the data processor 210 may also generate reports on demand.

[0083]. The frequency with which the external data is retrieved from the external data store 10c depends on the nature of the external data. Typically the external data may be retrieved with the same periodicity as the periodic reports 211 are generated, however the external data may also be retrieved at a set frequency, for example, daily. Once retrieved, the external data may be stored in the data merger 208 until it is merged with the data from the data store 207. Alternatively, the external data may be combined with the data from the data store 207 as the external data is retrieved, thereby forming the merged dataset 209, and the merged dataset 209 is retained for use by the data processor 210 in generating a report. Although the data processor 210 may generate the periodic reports 211 using only the information that is available in the data store 207, most aspects of embodiments use merged data, comprising the stored data, when preparing the periodic reports 211.

[0084], The flowchart of Figure 10 illustrates the process by which the data processor 210 generates the periodic reports 211 in the present embodiment. The data processor 210 receives the merged data set 209 from the data merger 208 at step S805. At step S901, the data processor 210 generates a statistical digest of the merged dataset 209. Then, at step S902, the statistical digest is used to generate graphical visualisations, which are easier for users to understand and interpret than raw numerical data. Using predictive models, the merged dataset 209 is also used to generate predictions of future trends in step S903. The predictions may be incorporated into the graphical visualisations, or presented separately.

[0085]. At step S904, the data processor 210 generates a report including the graphical representations and the predictions of future trends, and typically also including the statistical digest. Finally, at step S905, the report generation process ends. The report may then be outputted for the user via the user interface 10d, as shown in Figure 1. The user interface may be connected to the data collector 10b directly, or the user may operate the data collector remotely via the internet or another network.

[0086], Various different types of external data may be used, and the exact nature of the external data source 10c and the type of external data used is dependent upon the use for the periodic reports. Examples of the types of external data that may be relied upon are weather forecasts or reports, lists of upcoming events in a particular area, staff lists, and so on. In a specific example of the use of the external data, a shop in which the system is installed may use a website which publishes weather reports as the external data source 10c. The weather reports may be combined in the data merger 208 with the stored data from the data store 207 that covers the same time period as the weather reports relate to, thereby creating a merged dataset 209. This operation may be conducted on a daily basis, using the data collected for the previous day. The periodic reports 211 may then be used to check for a correlation between particular types of customers and the weather on a given day.

For example, the periodic reports 211 could indicate if families were more likely to enter the shop on wet days. Using this information, weather forecasts may be used to predict future customer behaviour.

[0087]. The reports may also be generated on demand. This is particularly applicable to aspects of embodiments of the invention used in a security orientated context. For example, if a crime is known or suspected to have taken place in a public space in which a system comprising a data collector 10b is operating, a report may be generated in order to assist in the identification of the perpetrators or to inform the design of security measures to prevent future crimes of a similar nature.

[0088]. Figure 11 is a block diagram of a computing device, such as a personal computer, which embodies the present invention, and which may be used to implement an embodiment of the method for producing a summary image from a series of images, and may also be used to implement an embodiment of the image analysis method. The computing device comprises a processor 993, and memory 994. Optionally, the computing device also includes a network interface 997 for communication with other computing devices, for example with other computing devices of invention embodiments, or for computing with remote databases (such as an external data source 10c) or remote user interfaces 10d.

[0089]. For example, an aspect of an embodiment of the invention may be composed of a network of such computing devices, such that components of the data collector 10b are split across a plurality of computing devices. In particular, where the physical property analyser 203, context classifier 204 and/or situation analyser 205 utilise a neural network, this neural network may be implemented by a plurality of interconnected computing devices. Optionally, the computing device also includes one or more input mechanisms such as keyboard and mouse or touchscreen interface 996 and a display unit such as one or more monitors 995, all of which are examples of user interfaces 10d. The components are connectable to one another via a bus 992.

[0090]. The memory 994 may include a computer readable medium, which term may refer to a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) configured to carry computer-executable instructions or have data structures stored thereon. Computer-executable instructions may include, for example, instructions and data accessible by and causing a general purpose computer, special purpose computer, or special purpose processing device (e.g., one or more processors) to perform one or more functions or operations. Thus, the term “computer-readable storage medium” may also include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methods of the present disclosure. The term “computer-readable storage medium” may accordingly be taken to include, but not be limited to, solid-state memories, optical media and magnetic media. By way of example, and not limitation, such computerreadable media may include non-transitory computer-readable storage media, including Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid state memory devices).

[0091], The processor 993 is configured to control the computing device and execute processing operations, for example executing code stored in the memory to implement the various different functions of the image processor 200, image tracker 202, image combiner 21, subject recogniser 22, subject labeller 23, subject tracker 24, physical property analyser 203, gender inferer 301, age group inferer 302, de-identifier 303, context classifier 204, group detector 401, group classifier 402, group labeller 403, data store 207, data merger 208 and data processor 210 described here and in the claims. The memory 994 stores data being read and written by the processor 993. As referred to herein, a processor may include one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. The processor may include a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processor may also include one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. In one or more embodiments, a processor is configured to execute instructions for performing the operations and steps discussed herein.

[0092], The display unit 997 may display a representation of data stored by the computing device and may also display a cursor and dialog boxes and screens enabling interaction between a user and the programs and data stored on the computing device. The display unit may also comprise a touchscreen interface. Local input mechanisms 996 are an example of the user interface 10d, and may enable a user to input data and instructions to the computing device.

[0093]. The network interface (network l/F) 997 may be connected to a network, such as the Internet, and is connectable to other such computing devices via the network. The network l/F 997 may control data input/output from/to other apparatus via the network.

[0094]. Other peripheral devices such as microphone, speakers, printer, power supply unit, fan, case, scanner, trackerball etc. may be included in the computing device. [0095], The image processor 200 of Figure 2 may be a processor 993 (or plurality thereof) executing processing instructions (a program) stored on a memory 994 and exchanging data via a network l/F 997 or bus 992. In particular, the processor 993 may execute processing instructions to receive, via the network l/F 997 or bus 992, input image files from one or more image inputters 10a, such as video cameras, and process the images as discussed above. Furthermore, the processor 993 may execute processing instructions to store the results of the processing and/or the input images on a connected storage unit (such as the data store 207) and/or to transmit, via the network l/F 997 or bus 992, the results and/or the input images to a remote data store 207 if said data store 207 is located remotely from the image processor 200.

[0096], The image tracker 202 of the image processor 200, as shown in Figures 2 and 3 may be a processor 993 (or plurality thereof) executing processing instructions (a program) stored on a memory 994 and exchanging data via a network l/F 997 or bus 992. In particular, the processor 993 may execute processing instructions to receive, via the network l/F 997 or bus 992, input image files from one or more image inputters 10a, such as video cameras, and identify and track subjects in the images as discussed above. Furthermore, the processor 993 may execute processing instructions to store the results of the identification and tracking and/or the input images on a connected storage unit (such as the data store 207) and/or to transmit, via the network l/F 997 or bus 992, the results and/or the input images to the physical property analyser 203.

[0097], The physical property analyser 203 of the image processor 200, as shown in Figures 2 and 4 may be a processor 993 (or plurality thereof) executing processing instructions (a program) stored on a memory 994 and exchanging data via a network l/F 997 or bus 992. In particular, the processor 993 may execute processing instructions to receive, via the network l/F 997 or bus 992, input image files and/or identification and tracking information from the image tracker 202, and determine physical properties of the subjects in the images as discussed above. Furthermore, the processor 993 may execute processing instructions to store the input image files, identification and tracking information and/or determined physical properties on a connected storage unit (such as the data store 207) and/or to transmit, via the network l/F 997 or bus 992, input image files, identification and tracking information and/or determined physical properties to the context classifier 204.

[0098], The context classifier 204 of the image processor 200, as shown in Figures 2 and 5 may be a processor 993 (or plurality thereof) executing processing instructions (a program) stored on a memory 994 and exchanging data via a network l/F 997 or bus 992. In particular, the processor 993 may execute processing instructions to receive, via the network l/F 997 or bus 992, input image files, identification and tracking information and/or determined physical properties from the physical property analyser 203, and generate classification information for the identified subjects as discussed above. Furthermore, the processor 993 may execute processing instructions to store the input image files, identification and tracking information, determined physical properties and/or generated classification information on a connected storage unit (such as the data store 207) and/or to transmit, via the network l/F 997 or bus 992, the input image files, identification and tracking information, determined physical properties and/or generated classification information to a remote data store 207.

[0099]. The situation analyser 205, as shown in Figures 2 and 6 may be a processor 993 (or plurality thereof) executing processing instructions (a program) stored on a memory 994 and exchanging data via a network l/F 997 or bus 992. In particular, the processor 993 may execute processing instructions to receive, via the network l/F 997 or bus 992, input image files, identification and tracking information, determined physical properties and/or generated classification information from the image tracker 202, physical property analyser 203 and/or context classifier 204 and perform a situation analysis as discussed above. Furthermore, the processor 993 may execute processing instructions to store the input image files, identification and tracking information, determined physical properties and/or generated classification information on a connected storage unit (such as the data store 207) and/or to transmit, via the network l/F 997 or bus 992, the input image files, identification and tracking information, determined physical properties and/or generated classification information to a remote data store 207. The processor 993 may further transmit an alarm signal to an alarm 206, and/or a further message, in the event that an incongruous situation is detected as discussed above.

[00100]. The data store 207, as shown in Figure 2 may be a processor 993 (or plurality thereof) executing processing instructions (a program) stored on a memory 994 and exchanging data via a network l/F 997 or bus 992. In particular, the processor 993 may execute processing instructions to receive, via the network l/F 997 or bus 992, input image files, identification and tracking information, determined physical properties and/or generated classification information from the context classifier 204, and store or transmit the received information as discussed above. The data store 207 may be connected to, incorporated within or remote from the image processor 200.

[00101], The data merger 208, as shown in Figure 2 may be a processor 993 (or plurality thereof) executing processing instructions (a program) stored on a memory 994 and exchanging data via a network l/F 997 or bus 992. In particular, the processor 993 may execute processing instructions to receive, via the network l/F 997 or bus 992, stored data from the data store 207 and external data from the external data source 20c and merge the data to create a merged data set 209 as discussed above. Furthermore, the processor 993 may execute processing instructions to store the merged data set 209 and/or to transmit, via the network l/F 997 or bus 992, the merged data set to the data processor 210.

[00102], The data processor 210, as shown in Figure 2 may be a processor 993 (or plurality thereof) executing processing instructions (a program) stored on a memory 994 and exchanging data via a network l/F 997 or bus 992. In particular, the processor 993 may execute processing instructions to receive, via the network l/F 997 or bus 992, stored data from the data store 207 or a merged data set 209 from the data merger 208 as discussed above. Furthermore, the processor 993 may execute processing instructions to process the stored data or merged data set 209 and generate periodic reports 211 as discussed above, and the processor 993 may execute further instructions to store the periodic reports 211 or transmit the periodic reports to a user interface via the network l/F or bus 992 as discussed above.

[00103]. Methods embodying the present invention may be carried out on a computing device such as that illustrated in Figure 11. Such a computing device need not have every component illustrated in Figure 11, and may be composed of a subset of those components. A method embodying the present invention may be carried out by a single computing device in communication with one or more data storage servers via a network, as discussed above.

[00104], As the present invention is configured to allow data to be collected and analysed without requiring the conscious participation of the subjects, it is applicable to a range of physical spaces as discussed above. Potentially negative situations may be detected in an entirely automated fashion, which may both reduce the staffing requirements for an entity using the system and also increase the effectiveness of negative situation detection (because the system is not impacted by loss of concentration as may occur with human monitoring). Further, the generation of the periodic reports and the optional inclusion of external data (via a merged dataset) allow rapid evaluation and decision making and predictions of future behaviour to be provided as and when required.

[00105], For the avoidance of doubt, the scope of the invention is defined by the claims.

Claims

1. A data collector for receiving images from an image inputter, the data collector comprising an image processor, the image processor comprising an image tracker configured to identify one or more subjects in the received images and track the identified subjects between the received images, wherein the image tracker is configured to output the results of the identifying and tracking as identification and tracking information to a physical property analyser, the physical property analyser being configured:

to receive the identification and tracking information and the received images;

to analyse the identification and tracking information and the received images to determine physical properties of the identified subjects; and to transmit the received images, the identification and tracking information and the determined physical properties to a context classifier, the context classifier being configured:

to receive the received images, the identification and tracking information and the determined physical properties;

to compare the received images, the identification and tracking information and the determined physical properties for the identified subjects against predetermined classification information in order to classify the identified subjects; and to output the received images, the identification and tracking information, the determined physical properties and the generated classification information for the identified subjects to a situation analyser, and the situation analyser being configured:

to perform a situation analysis by comparing the outputs from the image tracker, physical property analyser and/or context classifier to predetermined situation types, wherein the situation analyser is further configured to receive sensor inputs from one or more sensors and/or data from databases, and to incorporate the sensor inputs and/or data into the situation analysis; and to detect an incongruous situation on the basis of the situation analysis.

2. The data collector of claim 1, wherein the situation analyser is further configured to send an activation signal to an alarm if the situation analysis indicates that an incongruous situation has been detected.

3. The data collector of any preceding claim, wherein the image processor utilises convolutional neural networks.

4. The data collector of any preceding claim, wherein the physical property analyser includes a de-identifier configured to de-identify the identified subjects once the physical properties of the identified subjects have been determined.

5. The data collector of claim 4, wherein the de-identifier can be deactivated by an authorised user, so that the identified subjects are not de-identified.

6. The data collector of any preceding claim, wherein the context classifier is configured to detect subjects in an abnormal group type.

7. The data collector of claim 6, wherein the context classifier is configured to send an activation signal to an alarm if the context classifier detects an abnormal group type.

8. The data collector of any preceding claim, wherein the data collector is configured to receive images from a plurality of image inputters, each image inputter being a video camera, and wherein the data collector is configured to receive the images from each image inputter in the form of a video stream.

9. The data collector of any preceding claim, the data collector being further configured to transmit the received images, identification and tracking information, determined physical properties and/or the classification information to a data storage unit for incorporation into a subject data set.

10. The data collector of claim 9, wherein the subject data set is used by a data processor to generate periodic reports.

11. An image analysis method, the method comprising:

receiving images at a data collector from an image inputter; identifying one or more subjects in the received images and tracking the identified subjects between received images;

analysing the identification and tracking information to determine physical properties of the identified subjects;

comparing the received images, the identification and tracking information and the determined physical properties for the identified subjects against predetermined classification information in order to classify the identified subjects;

outputting the received images, the identification and tracking information, the determined physical properties and the generated classification information for the identified subjects;

performing a situation analysis by comparing the identification and tracking information, physical properties of the identified subjects and/or the classification information to predetermined situation types, while incorporating sensor inputs from one or more sensors and/or data from databases; and detecting an incongruous situation on the basis of the situation analysis.

12. The image analysis method of claim 11 wherein, if the situation analysis indicates that an incongruous situation has been detected, an activation signal is sent to an alarm.

13. The image analysis method of any of claims 11 and 12, wherein convolutional neural networks are used in the analysis method.

14. The image analysis method of any of claims 11 to 13, where the analysing step comprises de-identifying the identified subjects once the physical properties of the identified subjects have been determined.

15. The image analysis method of claim 14, wherein the de-identifying can be deactivated by an authorised user.

16. The image analysis method of any of claims 11 to 15, wherein the comparing step comprises detecting subjects in an abnormal group type.

17. The image analysis method of claim 16 wherein, if an abnormal group type is detected, an activation signal is sent to an alarm.

18. The image analysis method of any of claims 11 to 17, wherein images are received from a plurality of image inputters, each of which is a video camera, wherein the images from each image inputter are received in the form of a video stream.

19. The image analysis method of any of claims 11 to 18, further comprising transmitting received images, identification and tracking information, determined physical properties and/or classification information to a data storage unit and incorporating the transmitted received images, identification and tracking information, determined physical properties and/or classification information into a subject data set.

20. The image analysis method of claim 19, further comprising generating periodic reports using the subject data set.

21. A computer-readable medium comprising a computer program which, when executed by a computer, causes the computer to perform the image analysis method of any of claims 11 to 20.

22. An image analysis system comprising:

a data collector of any of claims 1 to 10; and an image inputter.

23. A data collector as hereinbefore described and/or with reference to any of the accompanying figures.

24. An image analysis method as hereinbefore described and/or with reference to any of the accompanying figures.

25. A computer-readable medium comprising a computer program as hereinbefore described and/or with reference to any ofthe accompanying figures.

26. An image analysis system as hereinbefore described and/or with reference to any of the accompanying figures.

Intellectual

Property

Office

Application No: Claims searched:

GB1614425.5

1-26