[go: up one dir, main page]

CN106354816B - video image processing method and device - Google Patents

video image processing method and device Download PDF

Info

Publication number
CN106354816B
CN106354816B CN201610765659.5A CN201610765659A CN106354816B CN 106354816 B CN106354816 B CN 106354816B CN 201610765659 A CN201610765659 A CN 201610765659A CN 106354816 B CN106354816 B CN 106354816B
Authority
CN
China
Prior art keywords
image
video
target object
information
feature
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610765659.5A
Other languages
Chinese (zh)
Other versions
CN106354816A (en
Inventor
邹博
刘玉洁
周玲武
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Neusoft Corp
Original Assignee
Neusoft Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Neusoft Corp filed Critical Neusoft Corp
Priority to CN201610765659.5A priority Critical patent/CN106354816B/en
Publication of CN106354816A publication Critical patent/CN106354816A/en
Application granted granted Critical
Publication of CN106354816B publication Critical patent/CN106354816B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/73—Querying
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7837—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/48—Matching video sequences
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Library & Information Science (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Image Analysis (AREA)

Abstract

the application provides a video image processing method and a video image processing device, wherein the method comprises the following steps: acquiring a video image sequence; identifying a target object from video image frames in a sequence of video images; tracking a target object and determining a motion track of the target object; acquiring video structured information based on a target object and a motion track of the target object; and performing target object retrieval and/or performing video condensation on the video image sequence based on the video structural information. The video image processing method and the video image processing device can rapidly detect the target, namely, the target detection speed is increased, and further the case detection speed is increased.

Description

Video image processing method and device
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a video image processing method and apparatus.
background
with the improvement of video monitoring systems, video image investigation technology has become the fourth investigation and solution solving technology behind public security organ criminal investigation technology, action technology and network investigation technology. The conventional video image investigation technology mainly adopts a man-sea tactic, namely, a large number of investigators are required to investigate the target from each video image frame of the video, and the investigation mode consumes a large amount of manpower and a long time, namely, the conventional investigation mode is time-consuming and labor-consuming and has poor investigation effect.
Disclosure of Invention
in view of the above, the present invention provides a video image processing method and apparatus, so as to solve the problem that the case detection speed is slow due to time and labor consumption of the video image detection method in the prior art, and the technical scheme is as follows:
A method of video image processing, the method comprising:
acquiring a video image sequence;
Identifying a target object from image frames in the sequence of video images;
Tracking the target object and determining a motion track of the target object;
acquiring video structured information based on the target object and the motion trail of the target object;
performing target object retrieval and/or performing video condensation on the video image sequence based on the video structured information.
wherein the identifying a target object from each frame of video image in the sequence of video images comprises:
Identifying a target object from each video image frame in the sequence of video images based on a deep convolutional neural network.
wherein the tracking the target object comprises:
And tracking the target object by adopting a Lucas-Kanade optical flow method tracking algorithm based on the optical flow points extracted from the target object in the video image frame.
wherein the video structuring information comprises: text information and/or image feature information of a target object, wherein the text information of the target object comprises attribute information and motion information of the target object;
the retrieving of the target object based on the video structural information includes:
When a retrieval instruction of text information to be retrieved is received, retrieving in the text information of the target object based on the text information to be retrieved, or when a retrieval instruction of an image to be retrieved is received, retrieving in the image feature information of the target object based on the image to be retrieved, or when a retrieval instruction of event information to be retrieved is received, retrieving in the text information based on the event to be retrieved and a pre-established event model to obtain a retrieval result;
And outputting the target object information associated with the retrieval result.
wherein the image feature information of the target object comprises a depth convolution feature and a local feature;
The retrieving in the image feature information of the target object based on the image to be retrieved to obtain a retrieval result, including:
Matching the depth convolution characteristics of the image to be retrieved in the image characteristic information of the target object according to a first matching rule to obtain a candidate characteristic set;
and matching the depth convolution characteristics and the local characteristics of the image to be retrieved in the candidate characteristic set according to a second matching rule based on the depth convolution characteristics and the local characteristics of the image to be retrieved to obtain target image characteristics serving as the detection result.
matching the depth convolution characteristics based on the image to be retrieved in the image characteristic information of the target object according to a first matching rule to obtain a candidate characteristic set, wherein the matching comprises the following steps:
Acquiring the depth convolution characteristic and the local characteristic of the image to be retrieved, and performing binary coding on the depth convolution characteristic of the image to be retrieved to obtain a binary coding characteristic of the image to be retrieved;
Respectively matching the binary coding features of the image to be retrieved with binary coding features corresponding to each depth convolution feature in the image feature information of the target object, determining the binary coding features with the matching degree of the binary coding features of the image to be retrieved being greater than a first preset value as target binary coding features, and taking the target depth convolution features and the target local features corresponding to the target coding features as candidate feature sets;
Matching the depth convolution characteristic and the local characteristic based on the image to be retrieved in the candidate characteristic set according to a second matching rule to obtain a target image characteristic as the detection result, wherein the matching comprises the following steps:
Matching the depth convolution features of the image to be retrieved with each depth convolution feature in the candidate feature set, matching the local features of the image to be retrieved with each local feature in the candidate feature set, and taking the image features of which the comprehensive matching degree with the corresponding local features is greater than a second preset value as retrieval results;
The outputting the target object information associated with the retrieval result specifically includes:
and outputting a target object image associated with the image features of which the comprehensive matching degree of the depth convolution features and the corresponding local features is greater than a second preset value, wherein the target object image is an image of the target object extracted from a video image frame where the target object is located in advance.
Wherein the video structuring information comprises: image density of each video image frame in the sequence of video images, the image density being used to characterize a condition of a target object in the video image frame;
Acquiring the structured information based on the target object and the motion trajectory of the target object comprises:
Determining structure information of a monitored area in the video image sequence according to the position of the target object in a video image frame and the motion track of the target object, wherein the structure information of the monitored area comprises area information of the target object appearing in the monitored area;
Determining an image density of each video image frame in the sequence of video images based on the structural information of the monitored area.
Wherein the video-condensing the sequence of video images based on the video structuring information comprises:
Segmenting the video image sequence based on the image density of each video image frame, and determining a video segment to be condensed from each segment;
and performing video concentration on the video segment to be concentrated, and merging the video segment subjected to the video concentration with other video segments not subjected to the video concentration to obtain a concentrated video image sequence.
wherein the segmenting the sequence of video images based on the image density and determining a video segment to be condensed from each segment comprises:
Dividing the video image sequence into a plurality of video segments by using a preset image density threshold value according to the image density of each video image frame, wherein each video segment comprises a plurality of continuous video image frames;
and determining the video segments of which the image density of each video image frame is greater than the image density threshold value as the video segments to be condensed.
wherein the video concentration of the video segment to be concentrated includes:
Determining an optimal movement strategy for moving at least one target object in the video segment to be condensed in a time dimension and a space dimension through a space-time condensation model;
and carrying out image fusion based on the optimal movement strategy to obtain the concentrated video segment.
A video image processing apparatus, the apparatus comprising: the system comprises a video acquisition module, a target identification module, a target tracking module, a video structured information acquisition module and a processing module;
The video acquisition module is used for acquiring a video image sequence;
The target identification module is used for identifying a target object from video image frames in the video image sequence acquired by the video acquisition module;
The target tracking module is used for tracking the target object identified by the target identification module and determining the motion track of the target object;
the information acquisition module is used for acquiring video structured information based on the target object identified by the target identification module and the motion track of the target object determined by the target tracking module;
the processing module is used for carrying out target object retrieval and/or carrying out video concentration on the video image sequence based on the video structural information acquired by the information acquisition module.
the target identification module is specifically configured to identify a target object from each video image frame in the video image sequence based on a deep convolutional neural network.
the target tracking module is specifically configured to track the target object by using a Lucas-Kanade optical flow method tracking algorithm based on optical flow points extracted from the target object in the video image frame.
wherein the video structuring information comprises: text information and/or image feature information of a target object, wherein the text information of the target object comprises attribute information and motion information of the target object;
the processing module comprises: a retrieval module and an output module;
the retrieval module is used for retrieving in the text information of the target object based on the text information to be retrieved when a retrieval instruction of the text information to be retrieved is received, or retrieving in the image characteristic information of the target object based on the image to be retrieved when a retrieval instruction of the image to be retrieved is received, or retrieving in the text information based on the event to be retrieved and a pre-established event model when a retrieval instruction of the event information to be retrieved is received, so as to obtain a retrieval result;
The output module is used for outputting the target object information related to the retrieval result of the retrieval module.
wherein the image feature information of the target object comprises a depth convolution feature and a local feature;
the retrieval module comprises: a rough matching module and an accurate matching module;
the rough matching module is used for matching in the image feature information of the target object according to a first matching rule based on the depth convolution feature of the image to be retrieved to obtain a candidate feature set;
and the precise matching module is used for matching the depth convolution characteristics and the local characteristics of the image to be retrieved in the candidate characteristic set according to a second matching rule based on the depth convolution characteristics and the local characteristics of the image to be retrieved to obtain target image characteristics as the detection result.
Wherein the coarse matching module comprises: a feature acquisition and processing sub-module and a coarse matching sub-module;
the feature obtaining and processing submodule is used for obtaining depth convolution features and local features of the image to be retrieved, performing binary coding on the depth convolution features of the image to be retrieved to obtain binary coding features of the image to be retrieved, and performing binary coding on each depth convolution feature in the image feature information of the target object to obtain binary coding features corresponding to each depth convolution feature in the image feature information of the target object;
The rough matching submodule is used for respectively matching the binary coding features of the image to be retrieved with the binary coding features corresponding to the depth convolution features in the image feature information of the target object, determining the binary coding features of which the matching degree with the binary coding features of the image to be retrieved is greater than a first preset value as target binary coding features, and taking the target depth convolution features and the target local features corresponding to the target coding features as candidate feature sets;
The precise matching module is specifically configured to match the depth convolution features of the image to be retrieved with each depth convolution feature in the candidate feature set, match the local features of the image to be retrieved with each local feature in the candidate feature set, and take the image features of which the comprehensive matching degree between the depth convolution features and the corresponding local features is greater than a second preset value as a retrieval result;
the output module is specifically configured to output a target object image associated with an image feature whose comprehensive matching degree between the depth convolution feature and the corresponding local feature is greater than a second preset value, where the target object image is an image of the target object extracted in advance from a video image frame in which the target object is located.
wherein the video structuring information comprises: image density of each video image frame in the sequence of video images, the image density being used to characterize a condition of a target object in the video image frame;
The information acquisition module includes: a monitoring area structure determining module and an image density determining module;
the monitoring area structure determining submodule is used for determining the structure information of a monitoring area in the video image sequence according to the position of the target object in a video image frame and the motion track of the target object, wherein the structure information of the monitoring area comprises area information of the target object appearing in the monitoring area;
the image density determining submodule is used for determining the image density of each video image frame in the video image sequence based on the structural information of the monitoring area determined by the monitoring area structure determining submodule.
wherein the processing module comprises: the video processing device comprises a video preprocessing module and a video concentration module;
the video preprocessing module is used for segmenting the video image sequence based on the image density and determining a video segment to be concentrated from each segment;
The video concentration module is used for carrying out video concentration on the video segment to be concentrated and merging the video segment subjected to video concentration with other video segments not subjected to video concentration to obtain a concentrated video image sequence.
Wherein, the video preprocessing module comprises: the video segmentation submodule and the video segment to be concentrated determining submodule;
the video segmentation submodule is used for dividing the video image sequence into a plurality of video segments by utilizing a preset image density threshold value according to the image density of each video image frame;
The to-be-condensed video segment determining submodule is used for determining the video segments of which the image densities of all the video image frames are greater than the image density threshold value as the to-be-condensed video segments.
Wherein the video enrichment module comprises: an optimal concentration strategy determination submodule and an image fusion submodule;
the optimal concentration strategy determining submodule is used for determining an optimal moving strategy for moving at least one target object in the video segment to be concentrated on a time dimension and a space dimension through a space-time concentration model;
And the image fusion submodule is used for carrying out image fusion based on the optimal movement strategy to obtain a concentrated video segment.
the technical scheme has the following beneficial effects:
the video image processing method and the video image processing device provided by the invention can identify and track the target object in the video image sequence, further can acquire the video structural information based on the target object and the motion track of the target object, can search based on the video structural information after acquiring the video structural information, and can quickly detect the target by the mode. That is, how the user knows some information of the target object in advance, the information can be directly utilized to search so as to quickly detect the target, and if the user does not know the information of the target object, the user can directly browse the condensed video so as to quickly detect the target. The video image processing method and the video image processing device can rapidly detect the target, namely, the speed of detecting the target is increased, and the detection speed of the case is increased.
drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a schematic flowchart of a video image processing method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a video image processing method according to an embodiment of the present invention, in which target detection is performed to generate a series of target candidate frames in a video image frame;
fig. 3 is a schematic diagram of optical flow points extracted from a target object in a video image frame in a video image processing method according to an embodiment of the present invention;
fig. 4 is a schematic flowchart of a specific implementation manner of obtaining a retrieval result based on retrieval of an image to be retrieved in image feature information of a target object in the video image processing method according to the embodiment of the present invention;
Fig. 5 is a schematic flowchart of acquiring video structured information based on a target object and a motion trajectory of the target object in the video image processing method according to the embodiment of the present invention;
fig. 6 is a schematic flow chart of an implementation process of segmenting a video image sequence based on image density and determining a video segment to be condensed from each segment in the video image processing method according to the embodiment of the present invention;
Fig. 7 is a schematic flowchart of an implementation process of performing video compression on a video segment to be compressed in the video image processing method according to the embodiment of the present invention;
Fig. 8 is a schematic structural diagram of a video image processing apparatus according to an embodiment of the present invention.
Detailed Description
the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
An embodiment of the present invention provides a video image processing method, please refer to fig. 1, which shows a flow diagram of the video image processing method, and the method may include:
step S101: a sequence of video images is acquired.
Step S102: a target object is identified from video image frames in a sequence of video images.
step S103: and tracking the target object and determining the motion trail of the target object.
Step S104: and acquiring video structured information based on the target object and the motion trail of the target object.
Step S105: and performing target object retrieval and/or performing video condensation on the video image sequence based on the video structural information.
The video image processing method provided by the invention can identify and track the target object in the video image sequence, further can acquire video structural information based on the target object and the motion track of the target object, can search based on the video structural information after acquiring the video structural information, can quickly detect the target by the method, and can also perform video concentration based on the video structural information. That is, how the user knows the specific information of the target object in advance, the information can be directly utilized for retrieval, so that the target object is quickly detected, and if the user does not know the information of the target object, the user can directly browse the condensed video, so that the target object is quickly detected. The video image processing method provided by the embodiment of the invention can quickly detect the target object from the video, namely, the embodiment of the invention improves the speed of target detection, further improves the detection speed of the case, and has better user experience.
The background modeling method is mostly adopted in the conventional target identification method, namely, the background of the image is modeled firstly, the image is compared with the background model after the model is established, and the foreground target is determined according to the comparison result. However, the method has poor adaptability in environments with low contrast and light variation, and often generates many false identifications when identifying moving targets, and often generates missing detections for stationary targets. In view of the problems of the existing identification method, in order to improve the accuracy of subsequent retrieval, the invention provides an identification method for identifying a target object in a video image frame based on a deep convolutional neural network, that is, in the above embodiment, the target object is identified from the video image frame in a video image sequence
the process of identifying the target object from the video image frame based on the deep convolutional neural network comprises the following steps: firstly, target detection is carried out by using a target detection model based on a deep convolutional neural network, a series of target candidate frames are generated in a video image frame, as shown in fig. 2, then target identification is carried out by using a target classification model based on the deep convolutional neural network, and the target candidate frames are corrected.
it should be noted that the deep convolutional neural network needs to be trained before being identified by using it: in a possible implementation manner, aiming at the characteristics of the public security monitoring environment and target (vehicle/human), an AlexNet network structure can be selected for training, an ImageNet2012 data set is used for pre-training, and a public security monitoring sample is used for optimizing the network on the basis. Because the difference between different vehicle types is large, the samples are divided into 6 categories in total for cars, passenger cars, trucks, tricycles, non-motor vehicles (motorcycles/electric vehicles/bicycles) and pedestrians during training.
In addition, in order to improve the subsequent identification speed, the convolution feature sharing can be carried out on the target detection network model and the target classification network model.
And tracking the target object after the target object is identified. Considering that the optical flow points are usually extracted from the whole image when the existing optical flow tracking algorithm extracts the optical flow points, however, the target object is often concerned when the target tracking is performed, the optical flow points of other irrelevant areas interfere with the tracking of the target object, and in order to improve the tracking speed and accuracy, the improved Lucas-Kanade optical flow tracking algorithm is adopted to track the target object, namely, the Lucas-Kanade optical flow tracking algorithm is adopted to track the target object based on the optical flow points extracted from the target object (foreground image) in the video image frame.
specifically, optical flow points are extracted from the whole video image frame, then optical flow points are extracted from the foreground image (i.e. the target object in the video image frame), and finally optical flow points in the whole video image frame except the optical flow points in the foreground image are filtered out based on the optical flow points extracted from the foreground image, as shown in fig. 3. The foreground image is obtained by frame difference between two adjacent video images, a white part in the second image in fig. 3 is a foreground region, and a black part is a background region.
In the above embodiment, the video structural information obtained based on the target object and the motion trajectory of the target object may include: text information and/or image feature information of the target object. The text information of the target object may include attribute information and motion information of the target object.
For example, the target object is a vehicle, the attribute information of the target object may include a category of the vehicle, a color of the vehicle, a license plate number of the vehicle, a brand model of the vehicle, and the like, and the motion information of the target object may be a motion direction of the vehicle, a position of the vehicle in the video image frame, and the like.
after the video structural information is acquired, retrieval can be performed based on the video structural information, so that the target object can be rapidly detected. There are various ways to implement target object retrieval based on video structured information.
In a possible implementation manner, the retrieval may be performed based on a text, that is, when a retrieval instruction of the text information to be retrieved is received, the text information of the target object is retrieved based on the text information to be retrieved, a retrieval result is obtained, and the target object information associated with the retrieval result is output. It should be noted that, in this embodiment, the text information of all the target objects may be combined into a text information base, and when performing text search, the search is performed in the text information base.
preferably, the target object information is a target object image, the target object image is an image of a target object extracted in advance from a video image frame where the target object is located, and the target object image is associated with text information and/or image feature information in the video structured information.
Specifically, text information to be retrieved input by a user is acquired, target text information matched with the text information to be retrieved is searched in the text information of the target object based on the text information to be retrieved, and a target object image associated with the target text information is output.
for example, a user inputs a vehicle license plate number on a retrieval interface for retrieval, and if the target information of the target object contains the vehicle license plate number, an image of a vehicle associated with the vehicle license plate number can be directly displayed, so that the target is quickly detected.
in another possible implementation manner, the retrieval may be performed based on rules, that is, when a retrieval instruction of the event information to be retrieved is received, the retrieval is performed in the text information of the target object based on the event to be retrieved and the pre-established event model, so as to obtain a retrieval result, and the target object information associated with the retrieval result is output.
specifically, event information to be retrieved input by a user is acquired, text information related to the event information to be retrieved is searched in the text information based on the event information to be retrieved, and target text information is determined from the text information related to the event information to be retrieved, wherein the event information corresponding to the target text information is the event information to be retrieved; and outputting the target object image associated with the target text information.
The event information may be area intrusion, tripwire, loitering and the like, the text information related to the event information to be retrieved may be position information of the target object in the video image frame or a motion track of the target object, which event occurs in the target object may be determined based on a change in the position or the motion track of the target object, and if an event identical to the event to be retrieved occurs in a target object, the target object image is output.
in another possible implementation manner, the retrieval may be performed based on the image, and when a retrieval instruction of the image to be retrieved is received, the retrieval result is obtained by retrieving in the image feature information base of the target object based on the image to be retrieved, and the target object information associated with the retrieval result is output. It should be noted that, in this embodiment, the image feature information of all the target objects may be combined into an image feature information base, and when performing image retrieval, retrieval may be performed in the image feature information base based on the features of the image to be retrieved.
specifically, an image to be retrieved input by a user is obtained, image features of the image to be retrieved are extracted as image features to be retrieved, then target image features matched with the image features to be retrieved are searched in image feature information based on the image features to be retrieved, and a target object image associated with the target image features is output. For the user, when searching based on the image, the user only needs to input the image to be searched in the searching interface, and the target object image matched with the image to be clued can be obtained.
in a preferred implementation manner, the image features include depth convolution features and local features, and in this embodiment, matching may be performed in the image feature information of the target object according to a first matching rule based on the depth convolution features of the image to be retrieved to obtain a candidate feature set; and then matching the depth convolution characteristics and the local characteristics of the image to be retrieved in the candidate characteristic set according to a second matching rule to obtain target image characteristics as a detection result. I.e. the final output is information of the target object associated with the target image feature.
Referring to fig. 4, a schematic flow chart of a specific implementation manner for obtaining a retrieval result by retrieving in image feature information of a target object based on an image to be retrieved is shown, and may include:
step S401: and acquiring the depth convolution characteristics and the local characteristics of the image to be retrieved, and performing binary coding on the depth convolution characteristics of the image to be retrieved to acquire the binary coding characteristics of the image to be retrieved.
It should be noted that the deep convolution features are extracted from the high layer based on the CNN deep convolution neural network, and the local features concern the local attributes of the image and can be used as the assistance and supplement of the deep convolution features. Among them, the local features are preferably SURF features with high speed and high robustness, and are marked as FSURF。
In a preferred implementation, a PCA method may be used to perform dimension reduction on the depth convolution features, remove redundant features, use the depth convolution features from which the redundant features are removed as final depth convolution features for subsequent matching, and record the final depth convolution features as FCNN+PCA. Performing binary coding on the depth convolution characteristics by adopting an LSH (least squares h) method to generate binary coding characteristics, FCNNH。
step S402: and respectively matching the binary coding features of the image to be retrieved with the binary coding features corresponding to the depth convolution features in the image feature information of the target object, determining the binary coding features with the matching degree of the binary coding features of the image to be retrieved being greater than a first preset value as target binary coding features, and taking the target depth convolution features and the local features corresponding to the target coding features as candidate feature sets.
When the binary coding features are matched, the matching degree can be represented by the similarity, and the similarity can be obtained by calculating the hamming distance between the two binary coding features.
since the image feature is associated with the target image, specifying the candidate feature set corresponds to specifying the candidate image set { O }1，O2，……，ON}：
Wherein, OiRepresents the ith image to be matched, Sirepresenting the similarity, theta, of the image to be retrieved and the ith image to be matchedHis a similarity threshold.
The process of matching the binary coding features is a coarse matching process, and after the coarse matching is completed, the precise matching is further performed based on the depth convolution features and the local features.
Step S403: and matching the depth convolution features of the image to be retrieved with each depth convolution feature in the candidate feature set, matching the local features of the image to be retrieved with each local feature in the candidate feature set, and taking the image features of which the comprehensive matching degree with the corresponding local features is greater than a second preset value as retrieval results.
In the process of accurate matching, Euclidean distance is adopted for similarity calculation, and the similarity is specifically calculated according to the following formula:
S(k)＝α×SCNN+PCA(k)+β×SSURF(k)
wherein alpha and beta respectively represent the weight of similarity calculated by the depth convolution characteristic and the local characteristic, SCNN+PCA(k) Representing the similarity of the deep convolution feature calculation, SSURF(k) representing the similarity of the local feature calculations.
Outputting target object information associated with the retrieval result, specifically: and outputting the target object image associated with the image features of which the comprehensive matching degree of the depth convolution features and the corresponding local features is greater than a second preset value.
In outputting the target object images, if there are a plurality of target object images satisfying the condition, the respective target object images may be displayed in order of high to low similarity.
the above procedure gives an implementation way of increasing the speed of image investigation, i.e. retrieving to obtain the target based on the target information, and the precondition for obtaining the target by this way is that the keywords used for retrieval are known in advance, i.e. the user knows in advance part of the information of the object to be investigated, however, at some point, the investigator may not know anything about the object, i.e. without keywords for retrieval, in which case the video image sequence can only be browsed one by one, considering that a video image sequence usually contains a large number of video image frames, in order to increase the detection speed, the embodiment of the invention performs video concentration on the video image frame sequence based on the video structural information, so that the concentrated video image frame has fewer frames and contains a large amount of information.
in this embodiment, the video structuring information may include: image density of each video image frame in a sequence of video images. Wherein the image density is used to characterize the condition of the target object in the video image frame.
referring to fig. 5, a schematic flowchart illustrating a process of obtaining video structured information based on a target object and a motion trajectory of the target object in the foregoing embodiment may include:
Step S501: and determining the structural information of the monitored area in the video image sequence according to the position of the target object in the video image frame and the motion track of the target object.
The structure information of the monitored area comprises area information of the target object appearing in the monitored area.
Step S502: the image density of each video image frame in the video image sequence is determined based on the structural information of the monitored area.
After the video structuring information is obtained, video condensation can be performed on the video image sequence based on the video structuring information. In order to increase the video concentration speed, in the embodiment of the present invention, a target video image sequence is first segmented based on the image density of each video image frame, a video segment to be concentrated is determined from each segment, then the video segment to be concentrated is subjected to video concentration, and the video segment subjected to video concentration is merged with other video segments not subjected to video concentration, so as to obtain a concentrated video image sequence.
further, referring to fig. 6, a flowchart illustrating an implementation process of segmenting a video image sequence based on image density and determining a video segment to be condensed from each segment is shown, and the implementation process may include:
Step S601: the video image sequence is divided into a plurality of video segments by the image density of each video image frame by using a preset image density threshold value.
Step S602: and determining the video segments of which the image density of each video image frame is less than the image density threshold value as the video segments to be condensed.
Illustratively, 100 video image frames are included in the video image sequence, the image density of each image video frame in the first 30 video image frames is less than the set image density threshold, the image density of each image video frame in the 31 th to 70 th video image frames is greater than the set image density threshold, and the image density of each image video frame in the 71 th to 100 th video image frames is less than the set image density threshold, then the video image sequence can be divided into 3 video segments, 1-30 frames are the 1 st video segment, 31-70 frames are the 2 nd video segment, 71-100 frames are the 3 rd video segment, and since the image densities of 1-30 frames and 71-100 frames are less than the set image density threshold, the two video segments of 1-30 frames and 31-70 frames are determined as the video segments to be condensed.
After determining the video segment to be condensed, performing video condensation on the video segment to be condensed, referring to fig. 7, a flowchart illustrating an implementation process of performing video condensation on the video segment to be condensed is shown, and the implementation process may include:
step S701: and determining an optimal movement strategy for moving at least one target object in the video segment to be condensed in a time dimension and a space dimension through a space-time condensation model.
Step S702: and carrying out image fusion based on the optimal movement strategy to obtain the concentrated video segment.
the space-time concentration model provided by the embodiment of the invention can carry out maximum video concentration in two dimensions of time and space while ensuring the original time sequence of the target without losing any target, and the concentrated video has no collision and stroboflash and has good visual effect.
specifically, the energy function of the spatio-temporal enrichment model is characterized as follows:
E(M)＝min{ΣEa(b)+{αΣEc(b,b')+βΣEt(b,b')}},(b,b'∈B)
Where b is the image sequence with the first target object, b' is the image sequence with the second target object, Σ Ea(b) For the active energy loss function, if the object with large area is not mapped into the condensed video, the penalty represented by this term is larger, and vice versa, it is understood that the object with large area should be retained in the condensed video. Ec(b, b') collision penalty terms,is the inner product of two trajectory conflict periods. In the condensed video, the original target is moved on a time axis and spatial distribution, so that the situations of cross collision, shielding and the like between tracks are inevitably generated, and if two target sequences have a shared time interval and the tracks are crossed, a penalty item is the inner product operation of a corresponding overlapped area. Etthe term (b, b') is a timing penalty term, which means that the sequence of the original video activity events is kept as much as possible, for example, two people walk in front of each other or talk side by side in the original video, and the relative relationship is reasonably kept in the condensed video. EtAnd (b, b ') is exp (- (d (b, b ')/omega), d (b, b ') represents the Euclidean distance of the central pixel of the two track sharing time interval, and omega is a custom parameter, and the event timing is adjusted. The above-described optimal movement strategy is a strategy in which the corresponding target object moves in time and space when the value of the energy function of the spatio-temporal enrichment model is the minimum value.
Corresponding to the above method, an embodiment of the present invention further provides a video image processing apparatus, please refer to fig. 8, which shows a schematic structural diagram of the apparatus, and the apparatus may include: a video acquisition module 801, a target recognition module 802, a target tracking module 803, an information acquisition module 804, and a processing module 805.
a video obtaining module 801, configured to obtain a video image sequence.
a target identification module 802, configured to identify a target object from video image frames in the video image sequence acquired by the video acquisition module 801.
And a target tracking module 803, configured to track the target object identified by the target identification module 802 and determine a motion trajectory of the target object.
The information obtaining module 804 obtains video knotting information based on the target object identified by the target identifying module 802 and the motion trajectory of the target object determined by the target tracking module 803.
A processing module 805, configured to perform target object retrieval and/or perform video condensation on the video image sequence based on the video structural information acquired by the video structural information acquiring module 804.
The video image processing device provided by the invention can identify and track the target object in the video image sequence, further can acquire the video structural information based on the target object and the motion track of the target object, can search based on the video structural information after acquiring the video structural information, and can quickly detect the target by the mode. That is, how the user knows some information of the target object in advance, the information can be directly utilized to search so as to quickly detect the target object, and if the user does not know the information of the target object, the user can directly browse the condensed video so as to quickly detect the target object. The video image processing device provided by the embodiment of the invention can quickly detect the target object from the video, namely, the embodiment of the invention improves the speed of target detection, further improves the detection speed of the case, and has better user experience.
in the video image processing apparatus provided in the above embodiment, the target identification module 802 is specifically configured to identify a target object from each video image frame in a video image sequence based on a deep convolutional neural network.
In the video image processing apparatus provided in the above embodiment, the target tracking module 803 is specifically configured to track the target object by using a Lucas-Kanade optical flow tracking algorithm based on the optical flow points extracted from the target object in the video image frame.
In the video image processing apparatus provided in the foregoing embodiment, the video structured information obtained by the video structured information obtaining module 804 includes: text information and/or image feature information of the target object, the text information of the target object including attribute information and motion information of the target object.
The processing module 805 includes: the device comprises a retrieval module and an output module.
the retrieval module is used for retrieving in the text information of the target object based on the text information to be retrieved when a retrieval instruction of the text information to be retrieved is received, or retrieving in the image characteristic information of the target object based on the image to be retrieved when a retrieval instruction of the image to be retrieved is received, or retrieving in the text information based on the event to be retrieved and a pre-established event model when a retrieval instruction of the event information to be retrieved is received, so as to obtain a retrieval result;
And the output module is used for outputting the target object information related to the retrieval result of the retrieval module.
in the above-described embodiment, the image feature information of the target object includes the depth convolution feature and the local feature associated with the depth convolution feature.
The retrieval module may include: a coarse matching module and an accurate matching module.
and the rough matching module is used for matching in the image characteristic information of the target object according to a first matching rule based on the depth convolution characteristic of the image to be retrieved to obtain a candidate characteristic set.
And the precise matching module is used for matching the depth convolution characteristics and the local characteristics of the image to be retrieved in the candidate characteristic set according to a second matching rule to obtain the target image characteristics as a detection result.
Further, the coarse matching module comprises: a feature acquisition and processing sub-module and a coarse matching sub-module.
And the feature obtaining and processing submodule is used for obtaining the depth convolution features and the local features of the image to be retrieved, performing binary coding on the depth convolution features of the image to be retrieved to obtain binary coding features of the image to be retrieved, and also used for performing binary coding on each depth convolution feature in the image feature information of the target object to obtain binary coding features corresponding to each depth convolution feature in the image feature information of the target object.
and the rough matching submodule is used for respectively matching the binary coding features of the image to be retrieved with the binary coding features corresponding to the depth convolution features in the image feature information of the target object, determining the binary coding features with the matching degree of the binary coding features of the image to be retrieved being larger than a first preset value as target binary coding features, and taking the target depth convolution features and the target local features corresponding to the target coding features as candidate feature sets.
and the precise matching module is specifically used for matching the depth convolution features of the image to be retrieved with each depth convolution feature in the candidate feature set, matching the local features of the image to be retrieved with each local feature in the candidate feature set, and taking the image features of which the comprehensive matching degree with the corresponding local features is greater than a second preset value as the retrieval result.
the output module is specifically configured to output a target object image associated with an image feature whose comprehensive matching degree between the depth convolution feature and the corresponding local feature is greater than a second preset value, where the target object image is an image of a target object extracted in advance from a video image frame where the target object is located.
In the video image processing apparatus provided in the foregoing embodiment, the video structured information obtained by the video structured information obtaining module 804 includes: and the image density of each video image frame in the video image sequence is used for representing the condition of the target object in the video image frame.
the video structured information obtaining module comprises: a monitoring area structure determining module and an image density determining module.
And the monitoring area structure determining submodule is used for determining the structure information of a monitoring area in a video image sequence according to the position of a target object in a video image frame and the motion track of the target object, wherein the structure information of the monitoring area comprises the area information of the target object appearing in the monitoring area.
and the image density determining submodule is used for determining the image density of each video image frame in the video image sequence based on the structural information of the monitoring area determined by the monitoring area structure determining submodule.
In the video image processing apparatus provided in the foregoing embodiment, the processing module includes: the device comprises a video preprocessing module and a video concentration module.
And the video preprocessing module is used for segmenting the video image sequence based on the image density and determining a video segment to be condensed from each segment.
And the video concentration module is used for carrying out video concentration on the video segment to be concentrated and merging the video segment subjected to the video concentration with other video segments which are not subjected to the video concentration to obtain a concentrated video image sequence.
Further, the video preprocessing module comprises: the video segmentation submodule and the video segment to be concentrated determining submodule.
The video segmentation sub-module is used for dividing the video image sequence into a plurality of video segments by utilizing a preset image density threshold value according to the image density of each video image frame;
And the to-be-condensed video segment determining submodule is used for determining the video segments of which the image densities of all the video image frames are greater than the image density threshold value as the to-be-condensed video segments.
further, the video enrichment module comprises: and the optimal concentration strategy determination sub-module and the image fusion sub-module.
And the optimal concentration strategy determining submodule is used for determining an optimal movement strategy for moving at least one target object in the video segment to be concentrated on a time dimension and a space dimension through a space-time concentration model.
and the image fusion submodule is used for carrying out image fusion based on the optimal movement strategy to obtain the concentrated video segment.
the embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
in the several embodiments provided in the present application, it should be understood that the disclosed method, apparatus, and device may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (20)

1. A method for video image processing, the method comprising:
Acquiring a video image sequence;
Identifying a target object from image frames in the sequence of video images;
tracking the target object and determining a motion track of the target object;
acquiring video structured information based on the target object and the motion trail of the target object;
Performing target object retrieval and/or performing video condensation on the video image sequence based on the video structural information, wherein the video structural information comprises image feature information of the target object and/or image densities of video image frames in the video image sequence, the image feature information of the target object is used for target object retrieval, the image densities of the video image frames in the video image sequence are used for video condensation, the image feature information comprises convolution characteristics and local characteristics, and the image densities are used for representing the situation of the target object in the video image frames;
The target object retrieval is carried out based on the image characteristics of the target object, and the method comprises the following steps:
acquiring a binarization coding feature of an image to be retrieved, wherein the binarization coding feature is obtained by performing binarization coding on a convolution feature of the image to be retrieved; acquiring a candidate feature set from the image feature information of the target object based on the binarization coding features of the image to be retrieved, wherein the candidate feature set comprises at least one convolution feature and a local feature corresponding to each convolution feature; and determining a target image feature from the candidate feature set based on the convolution feature and the local feature of the image to be retrieved, and outputting a target object image associated with the target image feature.
2. The method of claim 1, wherein the identifying a target object from each frame of video image in the sequence of video images comprises:
Identifying a target object from each video image frame in the sequence of video images based on a deep convolutional neural network.
3. The video image processing method of claim 1, wherein the tracking the target object comprises:
And tracking the target object by adopting a Lucas-Kanade optical flow method tracking algorithm based on the optical flow points extracted from the target object in the video image frame.
4. The video image processing method of claim 1, wherein the video structuring information further comprises: text information of a target object, wherein the text information of the target object comprises attribute information and motion information of the target object;
Then, performing target object retrieval based on the text information of the target object, including:
when a retrieval instruction of text information to be retrieved is received, retrieving in the text information of the target object based on the text information to be retrieved, or when the retrieval instruction of event information to be retrieved is received, retrieving in the text information based on the event to be retrieved and a pre-established event model to obtain a retrieval result;
And outputting the target object information associated with the retrieval result.
5. The video image processing method according to claim 4, wherein said obtaining a candidate feature set from the image feature information of the target object based on the binarized coding feature of the image to be retrieved comprises:
matching the image characteristic information of the target object according to a first matching rule based on the binarization coding characteristics of the image to be retrieved to obtain a candidate characteristic set;
the determining the target image feature from the candidate feature set based on the convolution feature and the local feature of the image to be retrieved comprises:
and matching the depth convolution characteristics and the local characteristics of the image to be retrieved in the candidate characteristic set according to a second matching rule to obtain the target image characteristics.
6. the video image processing method according to claim 5, wherein the matching based on the binarization coding feature of the image to be retrieved in the image feature information of the target object according to a first matching rule to obtain a candidate feature set comprises:
respectively matching the binary coding features of the image to be retrieved with binary coding features corresponding to each depth convolution feature in the image feature information of the target object, determining the binary coding features with the matching degree of the binary coding features of the image to be retrieved being greater than a first preset value as target binary coding features, and taking the target depth convolution features and the target local features corresponding to the target coding features as candidate feature sets;
Matching the depth convolution characteristic and the local characteristic based on the image to be retrieved in the candidate characteristic set according to a second matching rule to obtain the target image characteristic, wherein the matching comprises the following steps:
Matching the depth convolution features of the image to be retrieved with each depth convolution feature in the candidate feature set, matching the local features of the image to be retrieved with each local feature in the candidate feature set, and taking the image features of which the comprehensive matching degree with the corresponding local features is greater than a second preset value as retrieval results;
the outputting of the target object image associated with the target image feature specifically includes:
and outputting a target object image associated with the image features of which the comprehensive matching degree of the depth convolution features and the corresponding local features is greater than a second preset value, wherein the target object image is an image of the target object extracted from a video image frame where the target object is located in advance.
7. The method of claim 1, wherein the obtaining the image density of each video image frame in the video image sequence based on the target object and the motion trail of the target object comprises:
determining structure information of a monitored area in the video image sequence according to the position of the target object in a video image frame and the motion track of the target object, wherein the structure information of the monitored area comprises area information of the target object appearing in the monitored area;
Determining an image density of each video image frame in the sequence of video images based on the structural information of the monitored area.
8. The method of claim 7, wherein the video-condensing the sequence of video images based on the video structuring information comprises:
Segmenting the video image sequence based on the image density of each video image frame, and determining a video segment to be condensed from each segment;
And performing video concentration on the video segment to be concentrated, and merging the video segment subjected to the video concentration with other video segments not subjected to the video concentration to obtain a concentrated video image sequence.
9. The method according to claim 8, wherein said segmenting the sequence of video images based on the image density and determining a video segment to be condensed from each segment comprises:
dividing the video image sequence into a plurality of video segments by using a preset image density threshold value according to the image density of each video image frame, wherein each video segment comprises a plurality of continuous video image frames;
And determining the video segments of which the image density of each video image frame is greater than the image density threshold value as the video segments to be condensed.
10. The video image processing method according to claim 8, wherein said video-condensing the video segment to be condensed comprises:
Determining an optimal movement strategy for moving at least one target object in the video segment to be condensed in a time dimension and a space dimension through a space-time condensation model;
and carrying out image fusion based on the optimal movement strategy to obtain the concentrated video segment.
11. A video image processing apparatus, characterized in that the apparatus comprises: the system comprises a video acquisition module, a target identification module, a target tracking module, a video structured information acquisition module and a processing module;
The video acquisition module is used for acquiring a video image sequence;
the target identification module is used for identifying a target object from video image frames in the video image sequence acquired by the video acquisition module;
the target tracking module is used for tracking the target object identified by the target identification module and determining the motion track of the target object;
The information acquisition module is used for acquiring video structured information based on the target object identified by the target identification module and the motion track of the target object determined by the target tracking module;
The processing module is used for carrying out target object retrieval and/or video concentration on the video image sequence based on the video structural information acquired by the information acquisition module; wherein the video structuring information comprises image feature information of the target object and/or image densities of video image frames in the video image sequence, the image feature information of the target object is used for target object retrieval, the image densities of the video image frames in the video image sequence are used for video condensation, the image feature information comprises convolution characteristics and local characteristics, and the image densities are used for representing the situation of the target object in the video image frames;
Wherein the processing module comprises a retrieval module and an output module;
the retrieval module is used for acquiring the binarization coding characteristics of an image to be retrieved, acquiring a candidate feature set from the image feature information of the target object based on the binarization coding characteristics of the image to be retrieved, and determining target image features from the candidate feature set based on the convolution features and the local features of the image to be retrieved; the binary coding feature is obtained by carrying out binary coding on the convolution feature of the image to be retrieved; the candidate feature set comprises at least one convolution feature and a local feature corresponding to each convolution feature;
The output module is used for outputting the target object image associated with the target image characteristic.
12. the video image processing apparatus according to claim 11, wherein the target identification module is specifically configured to identify a target object from each video image frame in the video image sequence based on a deep convolutional neural network.
13. the video image processing apparatus according to claim 11, wherein the target tracking module is specifically configured to track the target object by using a Lucas-Kanade optical flow tracking algorithm based on optical flow points extracted from the target object in the video image frame.
14. the video image processing apparatus of claim 11, wherein the video structuring information further comprises: text information of a target object, wherein the text information of the target object comprises attribute information and motion information of the target object;
The retrieval module is further used for retrieving in the text information of the target object based on the text information to be retrieved when a retrieval instruction of the text information to be retrieved is received, or retrieving in the text information based on the event to be retrieved and a pre-established event model when a retrieval instruction of the event information to be retrieved is received, so as to obtain a retrieval result;
The output module is further used for outputting the target object information associated with the retrieval result.
15. The video image processing apparatus of claim 14, wherein the retrieving module comprises: a rough matching module and an accurate matching module;
The rough matching module is used for acquiring the binarization coding characteristics of the image to be retrieved, and matching the binarization coding characteristics of the image to be retrieved in the image characteristic information of the target object according to a first matching rule based on the binarization coding characteristics of the image to be retrieved to acquire a candidate characteristic set;
and the precise matching module is used for matching the depth convolution characteristics and the local characteristics of the image to be retrieved in the candidate characteristic set according to a second matching rule based on the depth convolution characteristics and the local characteristics of the image to be retrieved to obtain target image characteristics as the retrieval result.
16. the video image processing apparatus of claim 15, wherein the coarse matching module comprises: a feature acquisition and processing sub-module and a coarse matching sub-module;
the feature obtaining and processing submodule is used for obtaining depth convolution features and local features of the image to be retrieved, performing binary coding on the depth convolution features of the image to be retrieved to obtain binary coding features of the image to be retrieved, and performing binary coding on each depth convolution feature in the image feature information of the target object to obtain binary coding features corresponding to each depth convolution feature in the image feature information of the target object;
The rough matching submodule is used for respectively matching the binary coding features of the image to be retrieved with the binary coding features corresponding to the depth convolution features in the image feature information of the target object, determining the binary coding features of which the matching degree with the binary coding features of the image to be retrieved is greater than a first preset value as target binary coding features, and taking the target depth convolution features and the target local features corresponding to the target coding features as candidate feature sets;
the precise matching module is specifically configured to match the depth convolution features of the image to be retrieved with each depth convolution feature in the candidate feature set, match the local features of the image to be retrieved with each local feature in the candidate feature set, and take the image features of which the comprehensive matching degree between the depth convolution features and the corresponding local features is greater than a second preset value as a retrieval result;
The output module is specifically configured to output a target object image associated with an image feature whose comprehensive matching degree between the depth convolution feature and the corresponding local feature is greater than a second preset value, where the target object image is an image of the target object extracted in advance from a video image frame in which the target object is located.
17. the video image processing apparatus according to claim 11, wherein the information acquisition module includes: a monitoring area structure determining module and an image density determining module;
the monitoring area structure determining submodule is used for determining the structure information of a monitoring area in the video image sequence according to the position of the target object in a video image frame and the motion track of the target object, wherein the structure information of the monitoring area comprises area information of the target object appearing in the monitoring area;
The image density determining submodule is used for determining the image density of each video image frame in the video image sequence based on the structural information of the monitoring area determined by the monitoring area structure determining submodule.
18. The video image processing apparatus of claim 17, wherein the processing module comprises: the video processing device comprises a video preprocessing module and a video concentration module;
The video preprocessing module is used for segmenting the video image sequence based on the image density and determining a video segment to be concentrated from each segment;
The video concentration module is used for carrying out video concentration on the video segment to be concentrated and merging the video segment subjected to video concentration with other video segments not subjected to video concentration to obtain a concentrated video image sequence.
19. The video image processing apparatus of claim 18, wherein the video pre-processing module comprises: the video segmentation submodule and the video segment to be concentrated determining submodule;
The video segmentation submodule is used for dividing the video image sequence into a plurality of video segments by utilizing a preset image density threshold value according to the image density of each video image frame;
The to-be-condensed video segment determining submodule is used for determining the video segments of which the image densities of all the video image frames are greater than the image density threshold value as the to-be-condensed video segments.
20. the video image processing apparatus of claim 18, wherein the video enrichment module comprises: an optimal concentration strategy determination submodule and an image fusion submodule;
the optimal concentration strategy determining submodule is used for determining an optimal moving strategy for moving at least one target object in the video segment to be concentrated on a time dimension and a space dimension through a space-time concentration model;
And the image fusion submodule is used for carrying out image fusion based on the optimal movement strategy to obtain a concentrated video segment.
CN201610765659.5A 2016-08-30 2016-08-30 video image processing method and device Active CN106354816B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610765659.5A CN106354816B (en) 2016-08-30 2016-08-30 video image processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610765659.5A CN106354816B (en) 2016-08-30 2016-08-30 video image processing method and device

Publications (2)

Publication Number Publication Date
CN106354816A CN106354816A (en) 2017-01-25
CN106354816B true CN106354816B (en) 2019-12-13

Family

ID=57856028

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610765659.5A Active CN106354816B (en) 2016-08-30 2016-08-30 video image processing method and device

Country Status (1)

Country Link
CN (1) CN106354816B (en)

Families Citing this family (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108664844A (en) * 2017-03-28 2018-10-16 爱唯秀股份有限公司 Image target semantic recognition and tracking of convolutional deep neural network
CN107038713A (en) * 2017-04-12 2017-08-11 南京航空航天大学 A kind of moving target method for catching for merging optical flow method and neutral net
CN107346415A (en) * 2017-06-08 2017-11-14 小草数语（北京）科技有限公司 Method of video image processing, device and monitoring device
CN109215055A (en) 2017-06-30 2019-01-15 杭州海康威视数字技术股份有限公司 A kind of target's feature-extraction method, apparatus and application system
CN107506370A (en) * 2017-07-07 2017-12-22 大圣科技股份有限公司 Multi-medium data depth method for digging, storage medium and electronic equipment
CN107633480A (en) * 2017-09-14 2018-01-26 光锐恒宇（北京）科技有限公司 A kind of image processing method and device
CN107730560A (en) * 2017-10-17 2018-02-23 张家港全智电子科技有限公司 A kind of target trajectory extracting method based on sequence of video images
CN107705324A (en) * 2017-10-20 2018-02-16 中山大学 A kind of video object detection method based on machine learning
CN109803067A (en) * 2017-11-16 2019-05-24 富士通株式会社 Video concentration method, video enrichment facility and electronic equipment
CN108875517B (en) * 2017-12-15 2022-07-08 北京旷视科技有限公司 Video processing method, device and system and storage medium
CN109993032B (en) * 2017-12-29 2021-09-17 杭州海康威视数字技术股份有限公司 Shared bicycle target identification method and device and camera
CN108304808B (en) * 2018-02-06 2021-08-17 广东顺德西安交通大学研究院 Monitoring video object detection method based on temporal-spatial information and deep network
CN110659384B (en) * 2018-06-13 2022-10-04 杭州海康威视数字技术股份有限公司 Video structured analysis method and device
CN110166851B (en) 2018-08-21 2022-01-04 腾讯科技（深圳）有限公司 Video abstract generation method and device and storage medium
CN109508408B (en) * 2018-10-25 2021-07-30 北京陌上花科技有限公司 A frame density-based video retrieval method and computer-readable storage medium
CN109657546B (en) * 2018-11-12 2024-08-02 平安科技（深圳）有限公司 Video behavior recognition method based on neural network and terminal equipment
CN111435370A (en) * 2019-01-11 2020-07-21 富士通株式会社 Information processing apparatus, method, and machine-readable storage medium
CN109977816B (en) * 2019-03-13 2021-05-18 联想(北京)有限公司 Information processing method, device, terminal and storage medium
CN110008859A (en) * 2019-03-20 2019-07-12 北京迈格威科技有限公司 The dog of view-based access control model only recognition methods and device again
CN110175263A (en) * 2019-04-08 2019-08-27 浙江大华技术股份有限公司 A kind of method of positioning video frame, the method and terminal device for saving video
CN110188617A (en) * 2019-05-05 2019-08-30 深圳供电局有限公司 Intelligent monitoring method and system for machine room
CN110264496A (en) * 2019-06-03 2019-09-20 深圳市恩钛控股有限公司 Video structural processing system and method
CN110225310A (en) * 2019-06-24 2019-09-10 浙江大华技术股份有限公司 Computer readable storage medium, the display methods of video and device
CN110363171A (en) * 2019-07-22 2019-10-22 北京百度网讯科技有限公司 Method for training sky region prediction model and method for identifying sky region
CN110751065B (en) * 2019-09-30 2023-04-28 北京旷视科技有限公司 Training data acquisition method and device
CN113469200A (en) * 2020-03-30 2021-10-01 阿里巴巴集团控股有限公司 Data processing method and system, storage medium and computing device
CN111696136B (en) * 2020-06-09 2023-06-16 电子科技大学 Target tracking method based on coding and decoding structure
CN111898416A (en) * 2020-06-17 2020-11-06 绍兴埃瓦科技有限公司 Video stream processing method and device, computer equipment and storage medium
CN112036306B (en) * 2020-08-31 2024-06-28 公安部第三研究所 System and method for realizing target tracking based on monitoring video analysis
CN112422898B (en) * 2020-10-27 2022-06-17 中电鸿信信息科技有限公司 Video concentration method introducing deep behavior understanding
CN113515649B (en) * 2020-11-19 2024-03-01 阿里巴巴集团控股有限公司 Data structuring method, system, device, equipment and storage medium
CN112579811B (en) * 2020-12-11 2024-06-25 公安部第三研究所 Target image retrieval and identification system, method, device, processor and computer readable storage medium for video investigation
CN114679564B (en) * 2020-12-24 2025-08-22 浙江宇视科技有限公司 Video summary processing method, device, electronic device and storage medium
CN113365104B (en) * 2021-06-04 2022-09-09 中国建设银行股份有限公司 Video concentration method and device
CN113641852A (en) * 2021-07-13 2021-11-12 彩虹无人机科技有限公司 Retrieval method, electronic device and medium for photoelectric video target of unmanned aerial vehicle
CN113949823A (en) * 2021-09-30 2022-01-18 广西中科曙光云计算有限公司 Video concentration method and device
CN113792150B (en) * 2021-11-15 2022-02-11 湖南科德信息咨询集团有限公司 Man-machine cooperative intelligent demand identification method and system
CN114547374B (en) * 2022-02-18 2025-11-21 武汉烽火凯卓科技有限公司 Method and system for extracting and storing real-time video structured information
CN114998810B (en) * 2022-07-11 2023-07-18 北京烽火万家科技有限公司 A Neural Network-Based AI Video Deep Learning System

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8582807B2 (en) * 2010-03-15 2013-11-12 Nec Laboratories America, Inc. Systems and methods for determining personal characteristics
CN102930061B (en) * 2012-11-28 2016-01-06 安徽水天信息科技有限公司 A kind of video summarization method based on moving object detection
CN105512684B (en) * 2015-12-09 2018-08-28 江苏航天大为科技股份有限公司 Logo automatic identifying method based on principal component analysis convolutional neural networks

Also Published As

Publication number Publication date
CN106354816A (en) 2017-01-25

Similar Documents

Publication Publication Date Title
CN106354816B (en) video image processing method and device
Elharrouss et al. A combined multiple action recognition and summarization for surveillance video sequences
Xu et al. Segment as points for efficient online multi-object tracking and segmentation
Lin et al. Helmet use detection of tracked motorcycles using cnn-based multi-task learning
Gawande et al. Pedestrian detection and tracking in video surveillance system: issues, comprehensive review, and challenges
Wang et al. An effective method for plate number recognition
US20130170696A1 (en) Clustering-based object classification
Yang et al. Traffic-informed multi-camera sensing (TIMS) system based on vehicle re-identification
Balia et al. A deep learning solution for integrated traffic control through automatic license plate recognition
CN113177518A (en) Vehicle weight identification method recommended by weak supervision area
Lin et al. Visual-attention-based background modeling for detecting infrequently moving objects
CN114549867B (en) Gate machine fare evasion detection method, device, computer equipment and storage medium
Qasim et al. Abandoned object detection and classification using deep embedded vision
Sharif et al. Deep crowd anomaly detection: state-of-the-art, challenges, and future research directions
CN119810752B (en) Passenger flow statistics analysis system and method based on reid assistance
Zhang et al. Appearance-based loop closure detection via locality-driven accurate motion field learning
Mishra et al. A Study on Classification for Static and Moving Object in Video Surveillance System.
Gu et al. Embedded and real-time vehicle detection system for challenging on-road scenes
Yang et al. Video anomaly detection for surveillance based on effective frame area
Abed et al. Deep learning-based few-shot person re-identification from top-view rgb and depth images
Krithika et al. MAFONN-EP: a minimal angular feature oriented neural network based emotion prediction system in image processing
Kumar et al. Traffic surveillance and speed limit violation detection system
Chen et al. Object tracking over a multiple-camera network
Frontoni et al. People counting in crowded environment and re-identification
Leon et al. Car detection in sequences of images of urban environments using mixture of deformable part models

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant