GB2626940A - Video surveillance system - Google Patents

Abstract

A video management system (VMS) comprising a recording server and a processing unit. The recording server receives and stores a plurality of metadata streams associated with a plurality of video streams associated with a plurality of cameras. The processing unit is configured to receive GIS data associated with each camera, receive FOV data associated with each video stream, and lens/sensor information of each camera. The VMS then determines an adjusted FOV associated with each video stream, where the adjusted FOV indicates unobscured FOV and/or limited visibility FOV. Embodiments include a video surveillance system. Further embodiment incorporate a search function allowing a user to search for relevant objects at relevant times in the stored video streams. A user interface (UI) may be included which displays the position of cameras identified as a result of a search as well las the adjusted FOV of each camera.

Description

VIDEO SURVEILLANCE SYSTEM

TECHNICAL FIELD

The present disclosure generally relates to video surveillance systems and, optionally, computer-implemented video management methods for video surveillance systems.

BACKGROUND

Modern video surveillance systems have evolved into highly complex and often heterogeneous systems comprising a large number of different peripheral devices and computer hardware elements that are tied together via a networked infrastructure, and controlled by means of advanced management software. One important component of modern video surveillance systems is a video recording and processing system that allows video streams from one or more video cameras to be received, stored and processed.

A video management system (VMS), also known as video management software or a video management server, is a component or sub-system of a video surveillance system. The VMS typically provides various video management services, such as one or more of the following: collecting one or more video streams from one or more video cameras, storing the received one or more video streams to a storage device and providing an interface to view the received one or more live video streams and/or to access one or more stored video streams.

Further, a VMS may be configured to also handle other types of data besides video streams such as, but not limited to, audio streams, data from monitoring services such as motion detectors, fire alarms, etc. Moreover, it is generally desirable that surveillance systems and, in particular, VMSs are versatile and can be used in different types of applications which may impose different demands or requirements to processing and displaying received video streams supplied by the one or more video cameras. Moreover, the demands and requirements imposed in a surveillance system may change over time.

A particular challenge to video surveillance systems and the VMS subsystem is to handle video streams supplied by moving or movable, i.e. non-stationary, video cameras during normal operation of video surveillance system. The movable video camera or cameras may move or travel through a geographical surveillance area and/or facilities like office buildings etc. Another challenge for video surveillance VMS subsystems is the growing amount of data as devices recording data such as video cameras, microphones, and various detectors become ever more numerous.

SUMMARY

It is an object of at least some aspect described in the present disclosure to solve one or more of the problems identified above and/or other problems associated with existing video surveillance systems, or at least to provide an alternative to known systems.

In an aspect is disclosed a video management system (VMS) comprising: - a recording server configured for receiving and storing a plurality of metadata streams, the metadata streams being associated with a plurality of video streams, respectively, each video stream being supplied by, or associated with, respective ones of a plurality of video cameras; and - a processing unit configured to receive data via a data communication interface, and further configured to: -receive GIS data associated with the position of each camera, -receive FOV data associated with the plurality of video streams, the FOV data comprising: position and orientation of each camera, and image sensor and lens information of each camera, and -determine an adjusted FOV associated with each video stream, the determination being based on the FOV data and on the GIS data, where the adjusted FOV indicates unobscured FOV and/or limited-visibility FOV.

The video stream and the metadata stream associated with each video stream may each comprise timestamps, i.e. a digital record of time. This allows for time synchronization of the video streams and metadata streams received at the VMS.

Metadata may be obtained from different sources. Field-of-view (FOV) data is also metadata, i.e. data that describes and gives information about other data.

The FOV data used in the determination of an adjusted FOV may for example be obtained from a driver, or from a setting, such as a default setting, or as metadata in a metadata stream associated with a video stream. The adjusted FOV may for example be determined using a default setting of the lens and/or using image sensor information obtained from a driver. In some embodiments, the FOV data is obtained, entirely or partially, by the VMS as a metadata stream supplied by, or associated with, the movable video camera form which the respective video stream was acquired. The position of a respective camera may be obtained from a camera-associated GPS unit or device.

The determined adjusted FOV may be added to the metadata stream for the respective camera, and may be stored as part of the metadata stream.

The GIS data may be a dataset stored in a suitable manner in a component within the VMS, for example in a GIS repository, or the GIS data may be obtained from a database external to the VMS.

The plurality of video cameras may comprise at least one movable video camera and optionally one or more stationary video cameras. In some embodiments, the plurality of video cameras comprises a plurality of movable video cameras. The movable video cameras may each travel along a path or trail within an area via mounting to any suitable support structure of a vehicle, for example motorized vehicles like cars, trucks, busses, trains, motorcycles etc. The movable video cameras may be moved or transported along the path or trail of the area by way of mounting on, or worn by, a person via a suitable support like a belt etc. The one or more optional stationary video cameras may be mounted on, or fixed to, various kinds of stationary structures like factory or office buildings, train stations, support structures, etc. and may be arranged at traffic roads or junctions or the like.

Metadata on the position and orientation of a camera together with metadata on the image sensor and lens information of the camera allows for the calculation of a theoretical FOV of the respective camera, i.e. the extent of the area within which the camera can observe the world at any given moment.

The image sensor and lens characteristics will provide information on the width and depth covered by the camera. While the image sensor and some characteristics of the lens of the camera usually does not change over time, the lens information will comprise both physical, i.e. fixed, characteristics of the lens, and the current, i.e. dynamic, zoom level. Thus, these characteristics provide the shape of the FOV of the camera.

The position and orientation of the camera will provide the information needed to determine where the camera is viewing from at a given time and in which direction. In some embodiments, the metadata for the orientation of each camera comprises one or more of: compass direction, relative pan, and relative tilt.

In the determination of the adjusted FOV, structures described in the GIS data for the area covered by the FOV are used to remove obstructed parts of the FOV of the camera, or mark parts of the FOV of the camera as having limited visibility. That is, the part(s) of the FOV of the camera for which the line of sight is interrupted by one or more structures described in the GIS data can be accounted for by the FOV being adjusted to not include the areas beyond the interrupted line of sight or to mark one or more areas as having limited visibility. The determination of the adjusted FOV may be done using geometric intersection calculations to account for the structures described in the GIS for the area covered by the FOV.

A determination of the adjusted FOV may be made at each timestamp or whenever the position, lens characteristics, and/or orientation of the camera changes. For example, a determination of the adjusted FOV may be made whenever the position, lens characteristics, and/or orientation of the camera changes more than a minimum value of the position, lens characteristics, and/or orientation.

The determined adjusted FOV can advantageously be utilized in various ways, for example as a way to provide information on video coverage, e.g. on how much of an area is covered, or as a way to improve a search routine, e.g. by only searching in video streams from cameras that are able to possibly see an area of interest or by only listing camera with a relevant adjusted FOV for viewing of the video streams by a user. Thus, the system and method described herein facilitate finding a video stream of interest to the user.

In some embodiments, the recording server is further configured for receiving and storing the plurality of video streams with which the plurality of metadata streams is associated, respectively. The processing unit may be configured to store the plurality of video streams and respective metadata streams in a video data repository and in a metadata repository, respectively, and to retrieve the plurality of video streams and respective metadata streams from the video data repository and metadata repository, respectively. as discussed in further detail below with reference to the appended drawings.

In some embodiments, the VMS comprises means for receiving and storing the adjusted FOV for later retrieval. The determined adjusted FOV may be stored in the video data repository, the metadata repository, or in a repository separate from the video data and metadata repository. The processing unit may be configured to store the adjusted FOV stream in a suitable repository.

The processing unit may be configured to utilize one or more search criteria defined by a VMS operator, e.g. a user of the VMS, for a search of the plurality of video streams, and/or associated metadata streams and/or adjusted FOV. To initiate a search, the user may specify, for example, a certain time span and/or a certain geographical area. The user may additionally specify criteria related to identification within the plurality of video streams for identification of, for example, a target object, a target activity and/or a target incident.

In some embodiments, the processing unit is further configured to receive a search query comprising a target location or target area, and to search the adjusted FOV to identify a subset of the plurality of video streams for which the adjusted FOV comprises the target location or target area.

In some embodiments, the processing unit is further configured to receive a target time or a target time interval as part of the search query, and to perform the search with the further limitation that the target location or target area is comprised within the adjusted FOV at the target time or within the target time interval.

In some embodiments, the processing unit is further configured to receive at least one of: a target object, a target activity and a target incident as part of the search query, and to perform the search with the further limitation the target object and/or target activity and/or target incident is identified within the plurality of video streams and/or respective metadata.

Thus, in some embodiments, the search can provide a time, from the timestamps, at which a target object, and/or a target activity, and/or a target incident occurred at a position or within a geographic area. In some embodiments, the search can provide a position or an area at which a target object, and/or a target activity, and/or a target incident occurred at a given time or within a given time interval.

In some embodiments, the processing unit is further configured to perform the search in the identified subset of the plurality of video streams and/or respective associated metadata streams.

In some embodiments, the VMS further comprises a user interface (UI) client configured to provide a graphical user interface (GUI), and wherein the UI is configured to display to a user via the GUI: -a geo-map of a selected area comprising the target position or target area, and -search results of the identified subset of the plurality of video streams as an icon or thumbnail on the geo-map, wherein the icon or thumbnail position on the geo-map is representative of the camera position from which the respective video stream within the subset was recorded.

The icon or thumbnail on the geo-map may be displayed adjacent to the respective positions of the search results for example by an arrow or similar pointer indicating a structure at the geo-map position in question such as a specific position or location on a road, street or highway, a factory or office building, a train or bus stations, a traffic junction etc. Thus, in some embodiments, the VMS is capable of refining initial search results, either automatically or under user control, by applying at least one supplementary search criterion to reduce the number of remaining search results, while the visualization on the geo-map of representative icons or thumbnails provides the user with an intuitive and fast way of identifying relevant search results on the geo-map for further exploration and analysis as needed.

The skilled person will understand that numerous types of visually distinguishing attributes may be associated with or added to the icons or thumbnails for example coloured borders, coloured objects, textual tags, etc. In some embodiments, the UI client is further configured to display on the geomap the adjusted FOV of each camera in the subset at a timestamp associated with the search result.

In another aspect is disclosed a video surveillance system comprising: a plurality of video cameras arranged in a surveillance area and configured to generate respective video streams, wherein each camera of the plurality of video cameras comprise, or is associated with, a position detecting device and an orientation detecting device configured to add position metadata and orientation metadata, respectively, to each of the respective video streams; and a video management system (VMS) as disclosed herein.

The plurality of video cameras may comprise at least one movable video camera and optionally one or more stationary video cameras according to one embodiment of the video surveillance system. In some embodiments, the plurality of video cameras comprises a plurality of movable video cameras. The movable video cameras may each travel along a path or trail within the surveillance area via mounting to any suitable support structure of a vehicle for example motorized vehicles like cars, trucks, busses, trains, motorcycles etc. The movable video cameras may be moved or transported along the path or trail of the surveillance area by way of mounting on, or worn by, a person via a suitable support like a belt etc. The one or more stationary video cameras may be mounted on, or fixed to, various kinds of stationary structures like factory or office buildings, train stations, support structures, etc. and may be arranged at traffic roads or junctions or the like.

In another aspect is disclosed a computer-implemented video management method for a video management system (VMS), comprising steps: a) receive, at a video management system, a plurality of metadata streams and associated with a plurality of video streams, respectively, each video stream being supplied by, or associated with, respective ones of a plurality of video cameras; b) receive, at a processing unit comprised in the video management system, GIS data associated with the position of each camera, c) receive, by the processing unit, field-of-view (FOV) data associated with the plurality of video streams, d) determine, by the processing unit comprised in the video management system, an adjusted FOV associated with each video stream, the determination being based on the FOV data and on the GIS data, where the adjusted FOV indicates unobscured FOV and/or limited-visibility FOV.

The computer-implemented video management method may further comprise steps of: e) receive, by the video management system, a search query comprising a target location or target area, 0 search the adjusted FOV associated with each video stream to identify a subset of the plurality of video streams for which the adjusted FOV comprises the target location or target area.

The computer-implemented video management method may further comprise steps of: g) receive a target time or a target time interval as part of the search query, h) perform the search with the further limitation that the target location or target area is comprised within the adjusted FOV at the target time or within the target time interval.

The computer-implemented video management method may further comprise steps of: i) display a geo-map of a selected area comprising the target position or target area, j) display search results of the identified subset of the plurality of video streams as an icon or thumbnail on the geo-map, wherein the icon or thumbnail position on the geo-map is representative of the camera position from which the respective video stream within the subset was recorded.

The computer-implemented video management method may further comprise steps of: k) display, in the geo-map, the adjusted FOV of each camera in the subset at a timestamp associated with the search result.

In another aspect is disclosed a video management system (VMS) comprising a processing unit comprising microprocessor executable program instructions configured to carry out one or more of the method steps disclosed herein.

In the aspects disclosed herein, terms and features relate to the terms and features having the same name in the other aspects and therefore the descriptions and explanations of terms and features given in one aspect apply, with appropriate changes, to the other aspects. Additional aspects, embodiments, features and advantages will be made apparent from the following detailed description of embodiments and with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects will be apparent and elucidated from the embodiments described in the following with reference to the drawings in which: FIG. 1 is a schematic block diagram of an exemplary video surveillance system according to some embodiments; FIGS. 2A and 2B illustrate a moving camera recording video in an environment, wherein the FOV of the camera is being partially obstructed by structures in the environment; FIG. 3 illustrates a camera recording video in an environment, where the FOV of the camera is being partially obstructed by structures in the environment; FIG. 4 shows a flow diagram of a computer-implemented video management method according to some embodiments; FIGS. 5A and 5B show flow diagrams of a computer-implemented video management method according to some embodiments; FIGS. 6A and 6B show a swim lane diagrams of a computer-implemented video management method according to some embodiments; FIGS. 7A and 7B illustrate in schematic form an exemplary graphical user interface of a VMS according to some embodiments; FIG. 8 illustrates in schematic form an exemplary graphical user interface of a VMS according to some embodiments;

DETAILED DESCRIPTION

FIG. 1 is a schematic block diagram of an exemplary video surveillance system 10. The video surveillance system 10 comprises a plurality of video cameras 100a, 100b, 100c communicatively connected to a video management system (VMS) 300 via respective wired or wireless communication links or connections 200.

Some embodiments of the video surveillance system 10 may comprise a mix of movable video cameras and stationary video cameras for example at least one movable video camera 100c and one or more stationary video cameras 100a, 100b. Other embodiments may exclusively comprise one or more movable video camera(s) and no stationary video cameras while yet other embodiments exclusively comprise stationary video cameras. The stationary video cameras 100a, 100b are, when present, typically distributed across a predetermined area or space where surveillance is desired. The number and position/location of the stationary video cameras 100a, 100b of the video surveillance system 10 as well as the type of video camera comprised therein may be selected based on factors such as a level of surveillance desired, a size of the surveillance area or facility and/or the complexity of the layout of the surveillance area or facility. The movable video camera(s) 100c has a Field of view (FOV) and the stationary video cameras 100a, 100b have respective FOVs (not shown). The FOV is the open, observable area of the camera in question as schematically illustrated by a pie-shaped outline 110c. The skilled person will appreciate that different types of video cameras may have different FOVs for example caused by different optical properties of camera lenses.

In the present specification, the term "movable" as a property of a video camera means the camera can be moved, i.e. is geographically dynamic, while carrying out video recording and/or live video streaming. The video recording and/or live video streaming is often carried out during active operation of the video surveillance system 10. The movable video camera is for example displaced along a certain path or trail of the surveillance area. A stationary video camera is typically fixed to a stationary, like a building wall or a pole in the surveillance area.

The movable video camera 100c may travel along a path or trail of the surveillance area via mounting to any suitable support structure of various types of vehicles for example motorized vehicles like cars, trucks, busses, trains, motorcycles etc. The movable video camera 100c may be moved along the path or trail of the surveillance area by being mounted on, or worn by, a person via a suitable support like a belt etc. The person may for example be a police officer, bus driver, firefighter etc. In the latter situation, the movable video camera 100c travels through the surveillance area when the person walks or runs.

Alternatively, the movable video camera 100c may be transported or moved via the vehicle's travel when the person wearing the movable video camera 100c is a driver or passenger of the vehicle. The stationary video cameras 100a, 100b may be mounted on, or fixed to, various kinds of stationary structures like factory or office buildings, train stations, support structures arranged at traffic roads or junctions etc. The movable video camera(s) may be conventional portable video camera(s) known as such in the art of video surveillance. It will be appreciated that the video surveillance system 10 typically includes a plurality of movable video cameras of the same type and/or different types. Different types of movable video cameras of the video surveillance system 10 may for example be tailored to specific operation schemes and placements, e.g. fixed to a truck or on-person fixations. The movable video cameras of different types may be configured to supply video streams of different resolution, in different formats or outputting additional metadata associated with the video stream. Examples of functions of the movable video cameras may include one or more of the following: video streaming, in particular live streaming, and/or video recording and audio streaming and/or audio recording. The video streaming and/or video recording may be carried out in visible wavelength ranges and/or in infrared wavelength ranges, such as near-infrared wavelength ranges. The moveable video camera(s) and stationary video cameras may comprise various control functions such as pan or zoom, image processing capabilities, motion detection, etc. The respective video streams supplied by the stationary video cameras 100a, 100b as well as those of the one or more movable video cameras 100c are associated respective metadata streams. The metadata stream may be separate stream from the associated video stream but originating from either the same video camera or another device mounted on the same person or vehicle as the video camera. The metadata stream associated with each video stream preferably includes time stamps together with corresponding position data associated with the video camera in question. This property allows time synchronization of the video streams and metadata streams at the VMS. The respective geolocations of the stationary video cameras 100a, 100b and those of the one or more movable video cameras 100c may be derived from the position data supplied by a camera associated GPS unit or device. The associated GPS unit or device of a movable or stationary video camera may be built into the video camera as schematically illustrated by GPS device 102c of the movable video camera 100c, or may fixed to a vehicle or person carrying the movable video camera in question.

The stationary video cameras 100a, 100b as well as the one or more movable video cameras 100c are often communicatively connected to the video management system (VMS) 300 as mentioned above for example connected via a local area network 200 or in any other suitable manner, e.g. via point-to-point wired and/or wireless connections, or the like. For example, the stationary video cameras 100a, 100b may be connected to the VMS via an Ethernet connection. The one or more movable video cameras 100c may often be wirelessly connected to the VMS 300 for example through a wireless network like Wi-Fi, a 4G and/or 5G network. However, one or more movable video cameras 100c may alternatively be configured to record the video stream during active operation where the video camera moves in or through the surveillance area. In the latter scenario, the recorded video stream may be transferred to, or off-loaded at, a media repository 350 of the VMS 300 at the time of return to an associated station. In the latter use case, the video stream may be offloaded at regular time intervals for example when a camera user or cameral vehicle such as a bus driver or police officer returns to the station.

The skilled person will understand that some exemplary video surveillance systems may include additional sensors providing sensor signals and/or media streams different from video streams, such as audio signals, radar signals, Lidar signals, etc. The VMS 300 is preferably configured to store the received video streams in the media repository 350. The VMS 300 provides an interface 360 for accessing live video streams as well as the previously discussed added metadata, and to access video streams with respective metadata stored in the media repository 350. The interface 360 may implement different types of interfaces. For example, the interface may provide an application interface, e.g. in the form of a software development kit and/or one or more communication protocols, such as a suitable messaging protocol, e.g. SOAP, XML, etc. Accordingly, the interface may operate as a gateway to different types of systems. The VMS may be configured to implement various types of processing of received live video streams and/or recorded and retrieved video streams for example object detection, object recognition, motion detection etc. The media repository 350 may comprise a media database or other suitable storage device for storing media content. The VMS may include a user interface client (UI client) 400, for example configured to provide a graphical user interface, displayed on a suitable user screen or screens of the VMS 300. The graphical user interface enables users to view live video streams and/or stored video streams and/or to control operation of one or more of the stationary video cameras 100a, 100b and/or control operation of the one or more movable video cameras 100c. The content and structure of data items displayed through the user interface may be configurable by the operator via control buttons etc. The user interface comprises a map component integrated in VMS. The map component is utilized to build or provide a geo-map of at least a part of the surveillance area for presentation on the user screen. The map component may be configure to provide a geo-map overview of the respective positions of the plurality of video cameras.

The VMS 300 may be embodied as one or more software program(s) comprising respective computer executable instructions configured for execution on a suitable data processing system, e.g. by one or more server computers. The data processing system implementing the VMS is typically arranged remote from the one or more movable video cameras 100c as the latter often travel over a large geographical area for example through a route or trail comprising various streets, roads and facilities. The route or trail may cover a city neighbourhood or even an entire city. The video streams from the movable video camera(s) may be transmitted to the VMS 300 over wireless public or other wireless communications networks. Alternatively, the movable video camera(s) 100c of the video surveillance system 10 may move in relative proximity to a locally arranged on-site VMS 300 for example in a manufacturing facility, residential or office buildings, shopping centre etc..

The VMS 300 may comprise one or more camera drivers 310 for providing interfaces to respective types of stationary and movable video cameras. Different types of these video cameras may provide their respective video streams in different formats, e.g. using different encoding schemes and/or different network protocols. Similarly, different cameras may provide different interfaces for camera control such as zoom, or pan. Accordingly, the VMS 300 may include a plurality of different camera drivers 310 configured to cooperate with respective camera types. In particular, the camera drivers 310 may implement one or more suitable network protocols and/or other communications standards for transmitting data between movable and stationary video cameras and/or other peripheral devices and data processing systems. Examples of such protocols and standards include the Open Network Video Interface Forum (ONVIF) standard and the Real Time Streaming Protocol (RTSP).

The camera drivers 310 may further configured to add one time stamp to each frame of the received video streams 101 so as to ensure that the video streams, which are stored and subsequently supplied by the VMS 300, include a uniform time stamp. The added time stamp will also be referred to as a canonical time stamp. The canonical time stamp is indicative of the time of receipt, by the VMS 300, of the respective video streams from the respective stationary and movable video cameras. The camera drivers thus provide uniformly time-stamped input video streams 311, each time-stamped input video stream 311 corresponding to a respective one of the received video streams.

The VMS 300 comprises a recording server 320. The recording server may be embodied as a software program module executed by a suitable data processing system, e.g. by one or more server computers. The recording server receives the inputted video streams 311 originating from the respective stationary and movable video cameras through the corresponding camera drivers 310. The recording server stores the received inputted video streams in a suitable media storage device, such as a suitable media database. It will be appreciated that the media repository 350 may be part of the VMS 300 or it may be separate from, but communicatively coupled to the VMS. The media repository 350 may be implemented as any suitable mass storage device, such as one or more hard disks or the like. The storing of the received input video streams is also referred to as recording the received input video streams. The recording server may receive additional data such as the previously discussed metadata stream.

The VMS 300 may store the generated metadata in a suitable metadata repository 340, such as a suitable metadata database, which may be separate from or integrated into the media repository 350. To this end, the VMS 300 may include an index server 330. The index server may be embodied as a software program module executed by a suitable data processing system, e.g. by one or more server computers. The index server may receive metadata and store the received metadata in the metadata repository 340. The index server may further index the stored metadata so as to allow faster subsequent search and retrieval of stored metadata.

FIGS. 2A and 2B illustrate a moving camera 100c recording video in an environment, wherein the FOV 110c of the camera is being partially obstructed by structures 810, 811 in the environment.

The camera 100c may be e.g. a camera mounted in or on a vehicle or it may be worn or carried by e.g. a person. The camera 100c is moving along on a road structure 800 with buildings 810,811 interspersed beside the roads.

In FIG. 2A the FOV 110c of the camera is obstructed by a building 810 creating an obscured area 830 of FOV, where the camera is unable to observe.

FOV data, for example FOV data supplied by the camera, and possibly other devices associated with the camera, such as e.g. a GPS device, may be used to calculate the area covered by the FOV 110c of the camera. Additionally, FOV data may be obtained from other sources. FOV data used in such calculations are position and orientation of the camera, and image sensor and lens information of the camera. Combining this FOV data with relevant data from a Geographic Information System (GIS), an adjusted FOV may be calculated in which wholly or partially obstructing structures, such as buildings 810, are accounted for.

In FIG. 2B, the camera 100c is turned on the road 800 and the FOV 110c of the camera is now being obscured by a second building 811. With an unobstructed view, the camera would now be able to see the target object 840, a car. However, the second building 811 is interrupting the line-of-sight of the video camera 100c and hiding the car 840.

FIG. 3A illustrates a camera 100c recording video in an environment, where the FOV 110c of the camera is being partially obstructed by structures 850, 860 in the environment.

The camera 100c is pointing towards a target object 840, a car, which would be clearly visible to the camera were it not for the fully obscuring object 860 and the partially obscuring object 850. The fully obscuring object 860 completely blocks line-of-sight for the camera. Thus, the area of the FOV wherein the fully obscuring object 860 is as well as the "shadow" cast in the FOV 110c by the fully obscuring object 860, is obscured and the camera is not able to see this area.

Similarly, a partially obscuring object 850 partially blocks line-of-sight for the camera. Thus, the area of the FOV wherein the partially obscuring object 850 is as well as the "shadow" cast in the FOV 110c by the partially obscuring object 850, is an area of limited visibility for the camera.

FIGS. 3B and 3C illustrate the concepts of the FOV 110c and an adjusted FOV 880 of a camera. FIG. 3B shows an outline of the FOV 110c of a camera (not shown) with areas within the FOV, which are either obscured 830 or of limited visibility 870.

FIG. 3C shows an adjusted FOV 880, wherein both the obscured 830 or of limited visibility 870 areas of the FOV 110c have been removed leaving only the areas unobstructed by any structures described in the GIS data used in the determination of the adjusted FOV 880. In other embodiments, the adjusted FOV 880 may comprise the limited visibility area 870, possibly indicating the area of limited visibility.

FIG. 4 shows a flow diagram of a computer-implemented video management method according to some embodiments.

In step 505, a video management system receives a plurality of metadata streams, possibly bound to timestamps, and associated with a plurality of video streams, respectively. Each video stream is supplied by, or associated with, respective ones of a plurality of video cameras and each of the metadata streams may comprise FOV data. Additionally, FOV data may be obtained from other sources. The FOV data comprise: position and orientation of each camera, and image sensor and lens information of each camera.

Further, the video management system receives, or retrieves from a local repository, GIS data associated with the position of each camera.

In step 510, a processing unit comprised in the video management system determines an adjusted FOV associated with each video stream, the determination being based on the FOV data and on the GIS data, where the adjusted FOV indicates unobscured FOV and/or limited-visibility FOV.

In optional step 515, the adjusted FOV is stored, for example in a local repository.

FIGS. 5A and 5B show flow diagrams of a computer-implemented video management method according to some embodiments.

In step 520, a VMS receives a search query, for example via a User Interface (UI) client comprised in the VMS. The search query comprises a target location or target area.

In step 525, a processing unit comprised in the VMS performs a search in the data repository comprising determined adjusted FOV and identifies a subset of the plurality of video streams for which the adjusted FOV comprises the target location or target area.

Additional optional steps may comprise receiving a target time or a target time interval as part of the search query, and performing the additional limitation that the target location or target area is comprised within the adjusted FOV at the target time or within the target time interval.

In step 530, a user interface (UI) client comprised in the VMS and configured to provide a graphical user interface (GUI) provides GUI instructions for the display of the search results, see also description of FIG. 5B.

In FIG. 5B a flow diagram of the display of search results is shown.

In step 530, a user interface (UI) client comprised in the VMS and configured to provide a graphical user interface (GUI) provides GUI instructions for the display of the search results. The search results, which are a subset of the plurality of video streams, may be conveyed to a user via the GUI as the respective camera from which the video stream originated.

The search results may be provided via the GUI as a listing and/or as a visual display by overlaying the search results on a geographical map.

In step 532, the search results are listed in some form, for example as text strings or as icons, each identifying a camera from the search results.

In step 534, which may be performed instead of or together with step 532, a geomap of a selected area comprising the target position or target area is displayed in the GUI.

In step 536, the search results of the identified subset of the plurality of video streams are displayed on the geo-map, see also FIGS. 7A, 7B, and 8. The respective cameras in the search results are displayed on the geo-map as an icon or thumbnail and at a position on the geo-map that is representative of the camera position, where the video stream that lead to the search result was recorded.

Additionally, the adjusted FOV of each camera in the subset at a timestamp associated with the search result may be displayed in the geo-map, see also FIG: 8.

FIGS. 6A and 6B show swim lane diagrams of a computer-implemented video management method according to some embodiments. The diagrams illustrate the sharing of data to and from the VMS.

In FIG. 6A is illustrated how a plurality of movable cameras 100c and associated devices, e.g. a GPS device, generate respective video streams and metadata streams, possibly bound to timestamps. The video streams and metadata streams are provided to a VMS 300 as disclosed herein. Each camera of the plurality of video cameras comprise, or is associated with, a position detecting device and an orientation detecting device configured to add position metadata and orientation metadata, respectively, to each of the respective video streams.

The VMS 300 comprises a User Interface (UI) client, which provides the necessary instructions for display of a Graphical User Interface (GUI) on a display 450 receiving data from the UI client.

The VMS 300 may act as a subsystem within a video surveillance system.

In FIG. 6B is illustrated in detail how the VMS 300 receives and shares data. Each column shown corresponds to the elements shown and described in connection with FIG. 6A.

A plurality of movable cameras 100c record video streams. Together with metadata streams generated by components within each movable camera 100c or by one or more associated devices, the video streams are transferred to the VMS 300.

The VMS receives the video and metadata streams and determines an adjusted FOV, where the determination of the adjusted FOV is done separately from or in connection with a search. The VMS may receive a search query and perform a search based on the adjusted FOV.

The result of the search may be displayed in a GUI on a display 450 connected to the VMS 300, see also FIGS. 7A, 7B, and 8.

FIGS. 7A and 7B illustrate in schematic form an exemplary graphical user interface (GUI) 700 of a VMS according to some embodiments.

The GUI comprises a number of frames 710, 720, 730, wherein information is presented to a user.

A search query frame 710 allows a user to provide a search string, the content of which is transmitted to a VMS via a User Interface (UI) client in a suitable form.

The one or more search results provided by the VMS in response to the search query are transmitted in a suitable form to the display on which the GUI is presented. The search result(s) may be conveyed to a user as a list of cameras from each of which a video stream provided a search result. In the GUI 700 the list of cameras may then be provided in a GUI frame 720 for that purpose.

Additionally, the GUI presents the user with a frame showing a geo-map 730. In addition to, or instead of, presenting the user with a list of cameras the search results may be illustrated to a user in the geo-map 730. For example, an icon or a thumbnail 740, 745 may be shown in a position on the geo-map that is representative of the camera position at a timestamp, when the respective video stream that provided the search result was recorded. A thumbnail comprises a frame of a video stream, which may further facilitate a user's comprehension and/or interpretation of the search result. For example, if a user is searching a large number of video streams looking for a specific car in a geographical area, a frame from a video stream in the area may assist the user in the evaluation of the video streams. For illustration, video streams that may be of interest to the user are shown with a thumbnail of a car 745, while the video streams, which are not of interest to the user are shown with a thumbnail of an obstructed view 740.

The skilled person will appreciate that the direct display on the geo-map of the positions of video cameras, possibly as thumbnails, provide the user with a fast and intuitive overview of the area where the search was conducted. This display scheme improves the user's awareness and decision-making as to possible other areas should be investigated in relation to the search results thus reducing the cognitive load on the user.

In FIG. 7A is illustrated a result of a search in a plurality of video streams without the reduction in the searched set of video streams provided by taking the adjusted FOV into consideration. The outcome of the search is a multitude of search results shown as thumbnails 740, 745 in the geo-map 730. The user is searching for video recordings on a road 800 in the center of the geographic area shown in the geo-map 730. Without knowing which structures such as buildings 810 are obstructing the view of cameras in the area, the search provides a large number of search results. Many of the video streams in the search result will not be of interest to the user as the view is obstructed, as shown in the thumbnail of an obstructed view 740. The user will either have to go through the multitude of search results, which may be too numerous for a manual search to be feasible, or try to restrict the search result further which may have the unintended result that a video stream of interest is not provided as a result of the further restricted search. Another possibility, as an alternative or in addition, is that an automated search for a target within the video stream is performed using image recognition. However, the larger the set of video streams to be looked through or searched by an automated process are, the more time and resource-intensive it will be to perform an automated image recognition search.

Instead, both a manual and an automated search can be facilitated by a reduction in the set of video streams in which the manual or automated search is performed. In FIG. 7B is illustrated what may be a result of the same search as shown in FIG. 7A, but with the reduction in the searched set of video streams provided by taking the adjusted FOV into consideration. That is, instead of searching in a larger set of video streams, the search is done in a subset of the plurality of video streams and the result is fewer search results compared to the illustration in FIG. 7A. This lower number of search results makes facilitates both a manual and/or an automated search.

FIG. 8 illustrates in schematic form an exemplary graphical user interface of a VMS according to some embodiments. The GUI 700 shown in FIG. 8 is similar to that shown in FIGS. 7A and 7B except for the GUI frame 750 for displaying a video frame or video stream. Additionally, the geo-map 730 displays not only thumbnails 745, but also a graphical representation of the adjusted FOV 880 of each camera at a timestamp associated with the search result. The adjusted FOV 880 of each camera is shown in connection with the thumbnail 745 in a position on the geo-map 730 that is representative of the camera position at the timestamp.

The thumbnails or icons associated with each search result may be configured as or associated with one or more user-activatable buttons, where the activation of a button may provide the user with various functionalities in the GUI 700. For example, the activation of a button may select an associated camera and e.g. display a video frame or a video stream from the camera in the GUI frame 750 for that purpose. Alternatively, or additionally, the geo-map may be centred on the selected camera and/or the geo-map view may zoom to show the adjusted FOV of the selected camera in detail.

LIST OF REFERENCES

Surveillance system 100a, 100b Stationary video camera(s) 100c Movable video camera(s) 102c GPS device 110c FOV of video camera Wred or wireless communication links/connections 300 VMS 310 Camera drivers (CD) 311 Inputted video streams 320 Recording server (RS) 330 Index server (IS) 340 Metadata repository (metadb) 350 Media repository (mediadb) 360 Interface 400 UI Client 450 Display 500-590 Method steps 700 GUI 710 GUI frame with search query 720 GUI frame with listing of search results 730 GUI frame with geo-map 740 Icon or thumbnail search result 745 Icon or thumbnail search result showing target 750 GUI frame with video frame or video stream 800 Road 810/811 Building 830 Obscured area of FOV 840 Target object 850 Partially obscuring object 860 Fully obscuring object 870 Limited visibility area of FOV 880 Adjusted FOV

Claims (17)

1. CLAIMS1. A video management system (VMS) comprising: - a recording server configured for receiving and storing a plurality of metadata streams, the metadata streams being associated with a plurality of video streams, respectively, each video stream being supplied by, or associated with, respective ones of a plurality of video cameras; and - a processing unit configured to receive data via a data communication interface, and further configured to: -receive GIS data associated with the position of each camera, -receive field-of-view (FOV) data associated with the plurality of video streams, the FOV data comprising: position and orientation of each camera, and image sensor and lens information of each camera, and -determine an adjusted FOV associated with each video stream, the determination being based on the FOV data and on the GIS data, where the adjusted FOV indicates unobscured FOV and/or limited-visibility FOV.
2. 2. A video management system (VMS) according to claim 1, wherein the recording server is further configured for receiving and storing the plurality of video streams with which the plurality of metadata streams is associated, respectively.
3. 3. A video management system (VMS) according to any of the previous claims, wherein the VMS comprises means for receiving and storing the adjusted FOV for later retrieval.
4. 4. A video management system (VMS) according to any of the previous claims, wherein the metadata for the orientation of each camera comprises one or more of: compass direction, relative pan, and relative tilt.
5. 5. A video management system (VMS) according to any of the previous claims, wherein the processing unit is further configured to: - receive a search query comprising a target location or target area, and -search the adjusted FOV to identify a subset of the plurality of video streams for which the adjusted FOV comprises the target location or target area.
6. 6. A video management system (VMS) according to claim 5, wherein the processing unit is further configured to: -receive a target time or a target time interval as part of the search query, and - perform the search with the further limitation that the target location or target area is comprised within the adjusted FOV at the target time or within the target time interval.
7. 7. A video management system (VMS) according to any of claims 5 or 6, wherein the processing unit is further configured to: - receive at least one of: a target object, a target activity and a target incident as part of the search query, and - perform the search with the further limitation the target object and/or target activity and/or target incident is identified within the plurality of video streams and/or respective metadata.
8. 8. A video management system (VMS) according to claim 7, wherein the processing unit is further configured to perform the search in the identified subset of the plurality of video streams and/or respective associated metadata streams.
9. 9. A video management system (VMS) according to any of claims 5-7, wherein the VMS further comprises a user interface (UI) client configured to provide a graphical user interface (GUI), and wherein the UI client is configured to display to a user via the GUI: - a geo-map of a selected area comprising the target position or target area, and - search results of the identified subset of the plurality of video streams as an icon or thumbnail on the geo-map, wherein the icon or thumbnail position on the geo-map is representative of the camera position from which the respective video stream within the subset was recorded.
10. 10. A video management system (VMS) according to claim 9, wherein the UI client is further configured to display, on the geo-map, the adjusted FOV of each camera in the subset at a timestamp associated with the search result.
11. 11. A video surveillance system comprising: a plurality of video cameras arranged in a surveillance area and configured to generate respective video streams, wherein each camera of the plurality of video cameras comprise, or is associated with, a position detecting device and an orientation detecting device configured to add position metadata and orientation metadata, respectively, to each of the respective video streams; and a video management system (VMS) according to any of claims 1-9.
12. 12. A computer-implemented video management method for a video management system (VMS), comprising steps: a) receive, at a video management system, a plurality of metadata streams and associated with a plurality of video streams, respectively, each video stream being supplied by, or associated with, respective ones of a plurality of video cameras; b) receive, at a processing unit comprised in the video management system, GIS data associated with the position of each camera, c) receive, by the processing unit, field-of-view (FOV) data associated with the plurality of video streams, d) determine, by the processing unit, an adjusted FOV associated with each video stream, the determination being based on the FOV data and on the GIS data, where the adjusted FOV indicates unobscured FOV and/or limited-visibility FOV.
13. 13. A computer-implemented video management method according to claim 12, further comprising steps: e) receive, by the video management system, a search query comprising a target location or target area, f) search the adjusted FOV associated with each video stream to identify a subset of the plurality of video streams for which the adjusted FOV comprises the target location or target area.
14. 14. A computer-implemented video management method according to claim 13, further comprising steps: g) receive a target time or a target time interval as part of the search query, h) perform the search with the further limitation that the target location or target area is comprised within the adjusted FOV at the target time or within the target time interval.
15. 15. A computer-implemented video management method according to any of claims 13-14, further comprising steps: i) display a geo-map of a selected area comprising the target position or target area, j) display search results of the identified subset of the plurality of video streams as an icon or thumbnail on the geo-map, wherein the icon or thumbnail position on the geo-map is representative of the camera position from which the respective video stream within the subset was recorded.
16. 16. A computer-implemented video management method according to claim 15, further comprising step: k) display, in the geo-map, the adjusted FOV of each camera in the subset at a timestamp associated with the search result.
17. 17. A video management system (VMS) comprising a processing unit comprising microprocessor executable program instructions configured to carry out one or more of the method steps of claims 12-16.