HK1220795B - Teaching data generating device, method, and program, and crowd state recognition device, method, and program - Google Patents

Description

Training data generation device, method, and program, as well as crowd state recognition device, method, and program

Technical Field

The present invention relates to a training data generating device, a training data generating method and a training data generating program for generating training data, and a crowd state recognition device, a crowd state recognition method and a crowd state recognition program for recognizing the state of a crowd in an image.

Background Art

Various techniques for recognizing the state of a crowd in an image (which will be denoted as a crowd state hereinafter) are proposed (see PTL 1 to PTL 3).

The human behavior determination device described in PTL 1 extracts a changed area from a video and calculates a feature quantity based on the changed area, where differences are caused by background differentiation, etc. The human behavior determination device then determines whether the changed area is a human area by using a human discriminator that performs machine learning on the feature quantities, thereby detecting the human area. In addition, the human behavior determination device associates the detected human areas between frames by taking into account distance or color histogram, and tracks the human areas over a predetermined number of frames. The human behavior determination device then calculates feature quantities of the human trajectory, such as average speed, tracking time, and direction of movement, based on the human trajectory obtained through tracking, and determines human behavior based on the feature quantities of the human trajectory.

The head counting device described in PTL 2 counts the number of people based on a video of a crowd captured in a video. The head counting device extracts the heads of people included in the image based on a head model. The head counting device then links the head positions of people identified as the same person between frames using features such as position information or color distribution, and counts the number of people based on the linking results (person tracking results).

The system described in PTL 3 detects states such as stable (e.g., the mainstream of a person) and unstable (e.g., opposed to the mainstream). The system aggregates the optical flow attributes of a determination block, which serves as a determination unit, and calculates an evaluation value used to assess the stability of the optical flow. The system then determines the state of the determination block based on the evaluation value.

Reference List

Patent Literature

PTL 1: Japanese Patent Application Laid-Open No. 2011-100175 (paragraphs 0028 to 0030)

PTL 2: Japanese Patent Application Laid-Open No. 2010-198566 (paragraphs 0046 to 0051)

PTL 3: Japanese Patent Application Laid-Open No. 2012-22370 (paragraph 0009)

Summary of the Invention

Technical issues

With the techniques described in PTL 1 to PTL 3, determination performance degrades for videos with a low frame rate. In particular, with the techniques described in PTL 1 to PTL 3, the state of a crowd in an image cannot be determined for a still image.

This is because the techniques described in PTL 1 to PTL 3 use each frame in the video and the state determination performance depends on the interval between frames. For example, using the technique described in PTL 1, the human region is associated between frames to obtain the human trajectory. In addition, using the technique described in PTL 2, the head position is linked between frames and the result is assumed to be the human tracking result. When such a trajectory or tracking result is obtained, the human region or head position needs to be associated between frames. At this time, the amount of human movement is greater at a low frame rate, and therefore the change in the human region or head position or the change in shape (posture) increases. In addition, the influence caused by interference from lighting, etc. also increases. Therefore, it is difficult to associate the human region or head position between frames. As a result, the accuracy of human trajectories, etc. is reduced and the accuracy of determining the state of the crowd in the image is reduced. In addition, also using the technique described in PTL 3, it is difficult to correctly find the optical flow at a low frame rate, and the accuracy of the resulting aggregated attributes is reduced and the state determination performance is reduced.

For example, assume a method of using a discriminator with a learned dictionary to identify the state of a crowd in an image. The dictionary is learned through training data such as images indicating the state of a crowd. However, a large amount of training data (learning data) for learning the dictionary needs to be collected. For example, the arrangement of people (overlap between people or deviation of people's positions), the direction and density of people (people per unit area) need to be defined in various states, and a large number of images in which the angle, background, lighting, clothes or posture of the people are photographed are changed differently in each state need to be collected. Machine learning is performed by using images so that the dictionary of the discriminator is acquired. However, the workload for collecting training data increases when a large amount of such training data is collected.

Therefore, an object of the present invention is to provide a training data generating device, a training data generating method, and a training data generating program that can easily generate a large amount of training data for machine learning of a dictionary of a discriminator for recognizing a crowd state.

Another object of the present invention is to provide a crowd state recognition device, a crowd state recognition method, and a crowd state recognition program that can preferably recognize the state of a crowd in an image regardless of the frame rate.

Solutions to the Problem

According to the present invention, a training data generating device includes: a background extraction device for selecting a background image from a plurality of pre-prepared background images, extracting an area in the background image, and enlarging or reducing the image corresponding to the extracted area to an image of a predetermined size; a human state determination device for determining the human state of a crowd based on a multi-person state control designation as designation information about the human state of multiple people and an individual person state control designation as designation information about the state of an individual person among multiple people; and a crowd state image synthesis device for generating a crowd state image, specifying a training label for the crowd state image, and outputting a pair of the crowd state image and the training label, wherein the crowd state image is an image in which a human image corresponding to the human state determined by the human state determination device is synthesized with an image of a predetermined size acquired by the background extraction device.

In addition, a crowd state recognition device according to the present invention includes: a rectangular area group storage device for storing a group of rectangular areas indicating parts on an image to be recognized for crowd state; a crowd state recognition dictionary storage device for storing a dictionary of discriminators obtained by machine learning using a plurality of pairs of crowd state images and training labels for the crowd state images, the crowd state image being an image of a person expressing a crowd state in a predetermined size and including a reference part expressed as being the same size as the reference part of a person defined for the predetermined size; and a crowd state recognition device for extracting an area indicated in the group of rectangular areas stored in the rectangular area group storage device from a given image, and recognizing the state of a crowd captured in the extracted image based on the dictionary.

In addition, a training data generation method according to the present invention includes: a background extraction step of selecting a background image from a plurality of pre-prepared background images, extracting an area in the background image, and enlarging or reducing the image corresponding to the extracted area to an image of a predetermined size; a person state determination step of determining the person state of a crowd based on a multi-person state control indication as indication information about the person state of multiple people and an individual person state control indication as indication information about the state of an individual person among multiple people; and a crowd state image synthesis step of generating a crowd state image, specifying a training label for the crowd state image, and outputting a pair of the crowd state image and the training label, wherein the crowd state image is an image in which a person image corresponding to the person state determined in the person state determination step is synthesized with an image of a predetermined size acquired in the background extraction step.

In addition, in a crowd state recognition method according to the present invention, a rectangular area group storage device stores a group of rectangular areas indicating parts on an image to be recognized for crowd state, and a crowd state recognition dictionary storage device stores a dictionary of discriminators obtained by machine learning using multiple pairings of crowd state images and training labels for the crowd state images, the crowd state image being an image of a person expressing a crowd state in a predetermined size and including a reference part expressed as the same size as the reference part of a person defined for the predetermined size, the method including a crowd state recognition step of extracting an area indicated in the group of rectangular areas stored in the rectangular area group storage device from a given image and recognizing the state of a crowd captured in the extracted image based on the dictionary.

In addition, a training data generation program according to the present invention enables a computer to execute: background extraction processing, selecting a background image from a plurality of pre-prepared background images, extracting an area in the background image, and enlarging or reducing the image corresponding to the extracted area to an image of a predetermined size; human state determination processing, determining the human state of a crowd based on a multi-person state control designation as designation information about the human state of multiple people and an individual person state control designation as designation information about the state of an individual person among multiple people; and crowd state image synthesis processing, generating a crowd state image, specifying a training label for the crowd state image, and outputting a pair of the crowd state image and the training label, wherein the crowd state image is an image in which a human image corresponding to the human state determined in the human state determination processing is synthesized with an image of a predetermined size acquired in the background extraction processing.

In addition, a crowd state recognition program according to the present invention enables a computer including a rectangular area group storage device and a crowd state recognition dictionary storage device to execute: crowd state recognition processing, extracting an area indicated in the group of rectangular areas stored in the rectangular area group storage device from a given image and recognizing the state of a crowd captured in the extracted image based on a dictionary, the rectangular area group storage device being used to store a group of rectangular areas indicating parts on the image to be recognized for the crowd state, and the crowd state recognition dictionary storage device being used to store a dictionary of discriminators obtained by performing machine learning using a plurality of pairings of crowd state images and training labels for the crowd state images, the crowd state image being an image of a person whose reference part is expressed as being the same size as the reference part of a person defined for the predetermined size.

Advantageous Effects of the Invention

By using the training data generating device, the training data generating method, and the training data generating program according to the present invention, a large amount of training data for machine learning of a dictionary of a discriminator for recognizing a crowd state can be easily generated.

With the crowd state recognition device, the crowd state recognition method, and the crowd state recognition program according to the present invention, it is possible to preferably recognize the crowd state in an image regardless of the frame rate.

BRIEF DESCRIPTION OF THE DRAWINGS

[ Fig. 1 ] It depicts a block diagram illustrating an exemplary structure of a training data generating apparatus according to the present invention.

[ Fig. 2 ] It depicts a schematic diagram illustrating exemplary information stored in the crowd state control specification storage device. [ Fig. 2 ] FIG.

[ Fig. 3 ] It depicts a schematic diagram illustrating exemplary information stored in a human status control specifying storage device.

[ Fig. 4 ] It depicts a diagram illustrating, by way of example, human images stored in a human image storage device and human region images corresponding to the human images. [ Fig. 4 ] FIG.

[ Fig. 5 ] It depicts a schematic diagram illustrating an example of conditions satisfying a background person state.

[ Fig. 6 ] It depicts a schematic diagram illustrating an example of conditions satisfying a foreground person state.

[ Fig. 7 ] It depicts a block diagram illustrating an exemplary structure of a crowd state recognition device according to the present invention.

[ Fig. 8 ] It depicts a schematic diagram illustrating how to recognize the degree of congestion (the number of people) by way of example.

[Fig. 9] It depicts a schematic diagram illustrating how to recognize the direction of a crowd by way of example.

[Fig. 10] It depicts a schematic diagram illustrating how to identify a non-abnormal person or an abnormal person by way of example.

[ Fig. 11 ] It depicts a schematic diagram illustrating how to identify a disordered state or an ordered state by way of example.

[ Fig. 12 ] It depicts a flowchart illustrating an exemplary processing progress of a training data generating device.

[ Fig. 13 ] It depicts a flowchart illustrating an exemplary processing progress of step S1 .

[ Fig. 14 ] It depicts a flowchart illustrating an exemplary processing progress of step S2 .

[ Fig. 15 ] It depicts a flowchart illustrating an exemplary processing progress of step S3.

[ Fig. 16 ] It depicts a flowchart illustrating an exemplary processing progress of step S4.

[ Fig. 17 ] It depicts a flowchart illustrating an exemplary processing progress of the crowd state recognition device.

[ Fig. 18 ] It depicts a block diagram illustrating a specific structure of a training data generating device according to the present invention by way of example.

[ Fig. 19 ] It depicts a block diagram illustrating a specific structure of a crowd state recognition device according to the present invention by way of example.

[ Fig. 20 ] It depicts a block diagram illustrating main parts in a training data generating apparatus according to the present invention.

[Fig. 21] It depicts a block diagram illustrating main parts in a crowd state recognition device according to the present invention.

DETAILED DESCRIPTION

Exemplary embodiments according to the present invention will be described below with reference to the accompanying drawings.

1 is a block diagram illustrating an exemplary structure of a training data generating device according to the present invention. The training data generating device 10 according to the present invention generates training data for machine learning of crowd states in an image. Specifically, the training data generating device 10 creates multiple pairs of partial images of crowd states and training labels corresponding to the partial images. Here, "local" indicates an area smaller than the image (the image acquired by the image acquisition device 3 described below (see FIG7 )) to be identified for the crowd state. The partial image of the crowd state then represents a set of basic parts (which will be represented as reference parts below) of the people configuring the crowd in such an area. According to this exemplary embodiment, it will be described assuming that the head is used as the reference part, but other parts besides the head can be used as the reference part. The partial image of the crowd state will be represented as a crowd patch. The crowd patch can indicate therein parts of the person other than the reference part (the head in this example).

The training data generating device 10 includes a data processing device 1 operating under program control, and a storage device 2 for storing information therein.

The storage device 2 includes a background image storage device 21, a learned local image information storage device 22, a crowd state control specification storage device 23, a human state control specification storage device 24, a human image storage device 25 and a human area image storage device 26.

The background image storage device 21 stores a plurality of background images (a set of background images) used as backgrounds in crowd patches. The background images do not include people. An image of the actual place where an image to be recognized for the crowd state was taken can be used as the background image. Background images generated using CG (computer graphics) or the like can also be used.

The learning local image information storage device 22 stores the size of the crowd patch (a local image of the crowd state used for machine learning) and the size of the reference part of the person used for the crowd patch. For example, the size of the crowd patch is assumed to be h pixels in height and w pixels in width. The height size of the reference part of the person constituting the crowd shot in the crowd patch (the head in this example) is assumed to be 1/α of the height size of the crowd patch or h/α pixels. In this case, a height of h pixels and a width of w pixels are stored in the learning local image information storage device 22 as the size of the crowd patch. A height of h/α pixels is stored in the learning local image information storage device 22 as the size of the reference part of the person. An example in which the height size is stored as the size of the reference part is assumed here, but the size of the reference part to be stored is not limited to the height size. For example, it is assumed that the width size of the reference part of the person is defined as 1/α times the width size of the crowd patch or w/α pixels. In this case, the height h pixels and width w pixels are stored in the learned local image information storage device 22 as the size of the crowd patch, and the width w/α pixels can be stored in the learned local image information storage device 22 as the size of the reference part of the person. In actual use, the size of the reference part of the person can be the height size or the width size. The relationship between the size of the crowd patch and the size of the reference part of the person only needs to be known, and the diagonal size, etc. can be used.

Here, the size of a person's reference part is the size used to identify a person as a human when their reference part is photographed as being the same size as the size in the crowd patch. For example, when a person's reference part is photographed as being significantly larger or significantly smaller than the crowd patch, the person forms part of the crowd but is only seen as background.

The crowd state control designation storage device 23 stores designation information about the human state for multiple people (which will be represented as multi-person state control designation below) when synthesizing multiple human images in a crowd patch. The multi-person state control designation is previously defined by the operator of the training data generating device 10 and is stored in the crowd state control designation storage device 23. The multi-person state control designation is defined according to items, such as the item "arrangement of people" for the arrangement relationship of multiple people such as overlapping people or position deviation when synthesizing multiple human images, the item "direction of people" about the orientation of people, or the item "number of people" for the number of people or density. The items with defined multi-person state control designations are not limited to these. Figure 2 is a schematic diagram illustrating exemplary information stored in the crowd state control designation storage device 23. Figure 2 illustrates the multi-person state control designations defined for "arrangement of people", "direction of people" and "number of people".

The forms specified by multi-person status control include "pre-booked status", "random" and "pre-booked rules".

The "predetermined state" is a form of designation used to indicate a specific state of a corresponding item. In the example shown in Figure 2, the "three people" defined for the "number of people" item corresponds to the "predetermined state." In this example, the "number of people" is specifically designated as "three people." Another example of a "predetermined state" is, for example, "all people facing right" for the "direction of people" item.

"Random" indicates that the state can be arbitrarily defined for the corresponding item. In the example shown in FIG2 , the multi-person state control indication "Random" is defined for "arrangement of people" and "direction of people".

"Predetermined rules" are a form of specification that indicates that the status of the corresponding item can be defined within the scope of satisfying the operator-specified rule. For example, if the rule "People are arranged with a 50% overlap" is defined for the item "Person Arrangement," the person's arrangement is specified to at least define the status of the person satisfying this rule. For example, if the rule "People's Direction" is defined to at least define the status of the person satisfying this rule, such as "People arranged to the right of the center of the crowd patch face right, and People arranged to the left of the center face left," the person's direction is specified to at least define the status of the person satisfying this rule.

The crowd state control indication storage device 23 stores the presence of a designated training label per item. In the example shown in FIG2 , “○” indicates information about the presence of a designated training label and “×” indicates information about the absence of a designated training label. This is the same in FIG3 described later.

The operator selects one or more items with specified training labels from among the items with defined multi-person status control designations. In addition, the operator defines the multi-person status control designation for each item regardless of whether the item will be designated with a training label. In the example shown in FIG2 , a multi-person status control designation (a random designation in this example) is defined for the items “arrangement of people” and “direction of people” that do not have a specified training label. The operator assumes a form of multi-person status control designation as a “predetermined state” for the item with a specified training label. In the example shown in FIG2 , a specific state of “three people” is designated for the item “number of people” with a specified training label. The crowd status control designation storage device 23 stores therein the multi-person status control designation and the existence of the designated training labels defined by the operator according to the item.

FIG2 illustrates the items “arrangement of people”, “direction of people”, and “number of people” by way of example, but the items for which the operator defines the existence of multi-person state control designation and the designated training label are not limited thereto. According to the present exemplary embodiment, description will be made assuming that the crowd state control designation storage device 23 stores therein the multi-person state control designation and the existence of the designated training label defined by the operator for at least the items “arrangement of people”, “direction of people”, and “number of people”.

The multi-person state control designation storage device 24 stores information used to designate the state of each person when synthesizing multiple person images within a crowd patch (hereinafter referred to as an individual person state control designation). While the "multi-person state control designation" designates the state of a person for multiple people, the "individual person state control designation" designates the state of an individual person belonging to a group of people. The individual person state control designation is previously defined by the operator of the training data generating device 10 and stored in the person state control designation storage device 24. The individual person state control designation is defined according to items such as "the shooting angle of the person," "the lighting of the person," "the posture of the person," "the clothes of the person," "the body shape of the person," "the hairstyle of the person," or "the size of the person when synthesized with the crowd patch" when synthesizing with the crowd patch. Items with defined individual person state control designations are not limited to these items. FIG3 is a schematic diagram illustrating exemplary information stored in the person state control designation storage device 24. FIG3 illustrates individual person state control designations defined for the items "the shooting angle of the person," "the lighting of the person," and "the posture of the person."

Similar to the multi-person status control indication, the forms of individual status control indication are "predetermined status", "random" and "predetermined rules".

As described with respect to multi-person status control designation, "predetermined status" is a form of designation used to indicate a specific status of a corresponding item. In the example shown in FIG3 , "walking" defined for the item "person's posture" corresponds to the "predetermined status." In this example, "person's posture" is specifically designated as a walking posture.

As described for the multi-person state control indication, the "random" indication can arbitrarily define a state for the corresponding item. In the example shown in FIG3 , the individual state control indication "random" is defined for "lighting for people."

As described for multi-person status control instructions, "predetermined rules" are a form of instruction for defining the status of a corresponding item within the range of satisfying the rules specified by the operator. In the example shown in FIG3 , a predetermined rule is defined for "shooting angle of a person." In this example, it is specified that the shooting angle of a person is calculated and the state of the person according to the shooting angle is defined by using an equation based on the camera parameters from the arrangement of the person at the time of synthesis. For example, when a rule is defined for "size of a person when synthesizing with a crowd patch" that "determines the size of a person at the time of synthesis based on the arrangement of the person at the time of synthesis and the size of the reference part stored in the learned local image information storage device 22," the size of the person will be defined to at least satisfy this rule.

The multi-person state control indication storage device 24 also stores the existence of the indicated training label on a project-by-project basis.

The operator can select one or more items with designated training labels not only for items with defined multi-person status control designations but also for items with defined individual person status control designations. Also, in this case, the operator defines the individual person status control designation for each item regardless of whether the item will be designated with a training label. In the example shown in FIG3 , the individual person status control designation is defined for the items “shooting angle of the person” and “lighting of the person” that do not have designated training labels. The operator assumes a form of individual person status control designation as the “predetermined state” of the item with a designated training label. In the example shown in FIG3 , the item “posture of the person” with a designated training label is specifically designated as the state of walking. The person status control designation storage device 24 stores therein the existence of the individual person status control designation and the designated training labels defined by the user according to the item.

The operator may not specify a training label for all items with a defined individual status control designation. As described above, the operator defines one or more items as items with a specified training label for items with a defined multi-person status control designation.

According to this exemplary embodiment, the existence of individual person state control indications defined by the operator and indications of at least one training label for the items "shooting angle of the person", "lighting of the person", "posture of the person", "clothing of the person", "body shape of the person", "hair style of the person" and "size of the person when synthesized with a crowd patch" will be stored in the assumed person state control indication storage device 24.

The content of the multi-person state control designation defined for an item having a designated training label is a training label corresponding to a crowd patch generated based on information stored in the crowd state control designation storage device 23. Similarly, the content of the individual person state control designation defined for an item having a designated training label is a training label corresponding to a crowd patch generated based on information stored in the person state control designation storage device 24. The training label based on the multi-person state control designation is a main training label, and the training label based on the individual person state control designation is a supplementary training label for the training label.

Specifically, the data processing device 1 (see FIG1 ) determines the state of people and generates a crowd patch in which people are synthesized based on the multi-person state control designation of each item stored in the crowd state control designation storage device 23 and the individual person state control designation of each item stored in the person state control designation storage device 24. The data processing device 1 defines the contents of the multi-person state control designation and the individual person state control designation defined for the items with the designated training labels as training labels for the crowd patch. For example, it is assumed that the data processing device 1 generates a crowd patch based on the multi-person state control designation and the individual person state control designation shown in FIG2 and FIG3 . In this case, three walking people are photographed in the crowd patch. The data processing device 1 defines the training label "three people, walking" as a training label for the crowd patch.

The item "person size when synthesized with a crowd patch" is stored in the person status control designation storage device 24. When a person identified as a human is synthesized in a crowd patch, for example, the size of the person's reference part stored in the learned partial image information storage device 22 may be specified, or a random individual person status control designation may be specified as "person size when synthesized with a crowd patch." If, as a result of the random designation, the person's status is temporarily determined to be a reference part size significantly different from the reference part size of the person stored in the learned partial image information storage device 22, the temporary determination of the person's status may be made again. When a person representing the background is synthesized in a crowd patch, for example, a size significantly different from the reference part size of the person stored in the learned partial image information storage device 22 may be specified, or a random individual person status control designation may be specified as "person size when synthesized with a crowd patch." If, as a result of the random designation, the status of a person not corresponding to the background is temporarily determined, the temporary determination of the person's status may be made again.

As described below, according to the present exemplary embodiment, the data processing device 1 determines the state of a person to be identified as a human being (which may be represented below as a foreground person) and determines the state of a background person. The multi-person state control designation and the individual person state control designation for determining the state of the foreground person and the multi-person state control designation and the individual person state control designation for determining the state of the background person may be defined separately by the operator. In this case, the crowd state control designation storage device 23 stores therein the multi-person state control designation for determining the state of the foreground person and the multi-person state control designation for determining the state of the background person. The person state control designation storage device 24 stores therein the individual person state control designation for determining the state of the foreground person and the individual person state control designation for determining the state of the background person. The multi-person state control designation and the individual person state control designation may not be separated for determining the state of the foreground person and for determining the state of the background person.

The human image storage device 25 stores a plurality of human images (a set of human images) to which information on human conditions, such as the direction of the person, the angle from which the person is photographed, the lighting of the person, the person's posture, the clothing of the person image, the body shape, and the hairstyle, is added. That is, the data processing device 1 can read a human image that matches a certain condition from the human image storage device 25.

The human area image storage device 26 stores a group of human area images corresponding to the group of human images stored in the human image storage device 25. The human area image is an image indicating the area of a person in the human image stored in the human image storage device 25. Figure 4 is a diagram illustrating, by way of example, human images stored in the human image storage device 25 and human area images corresponding to the human images. Figure 4 illustrates, by way of example, four pairs of human images and human area images. The human area image may be an image in which the area of a person captured in the human image is expressed in a single color (white in the example shown in Figure 4) and the area other than the person is expressed in another single color (black in the example shown in Figure 4). The human area image is not limited to this example. The human area image may be an image that can indicate the area of a person in the human image.

The person region image is used to crop only the person (or only the region of the person) from the corresponding person image.

It may be configured such that not a set of various human images is previously prepared and stored in the human image storage device 25 but the data processing apparatus 1 includes a human image generating device (not shown) for generating a human image matching a determined human state by CG or the like.

The data processing device 1 includes a background extraction device 11 , a human state determination device 15 , a crowd state image synthesis device 14 and a control device 16 .

The background extraction device 11 selects a background image from the set of background images stored in the background image storage device 21. The background extraction device 11 calculates the aspect ratio of the crowd patch size stored in the learned local image information storage device 22. The background extraction device 11 temporarily extracts a background image at an appropriate position and size from the selected background image to satisfy the aspect ratio. In addition, the background extraction device 11 enlarges or reduces the temporarily extracted background image to match the crowd patch size stored in the learned local image information storage device 22. In this way, enlarging or reducing the area extracted from the image to match the crowd patch size can be expressed as normalization.

When the background extraction device 11 temporarily extracts the background at an appropriate position and an appropriate size, an area at a random position and a random size can be extracted to satisfy the aspect ratio. Assuming that the size of the reference part of the person at each position in the image is known, the background extraction device 11 can find the size of the crowd patch that is enlarged or reduced at a magnification or reduction rate based on the size of the known reference part at each position in the image, and the size of the reference part of the person stored in the learned local image information storage device 22 is enlarged or reduced at the magnification or reduction rate. The background extraction device 11 can then extract an area with the size found for the position in the image. The method used by the background extraction device 11 to temporarily extract an area from the selected background image may be other methods.

The human state determination device 15 temporarily determines the human state based on the multi-person state control indication stored in the crowd state control indication storage device 23 and the individual person state control indication stored in the human state control indication storage device 24, while determining the final human state based on the conditions of the size of the human reference part of the crowd patch size and how the reference part is expressed.

Here, when a person's status is determined to satisfy both the multi-person status control specification and the individual person's status control specification, it is indicated that "random" is possible in the specification, and thus the appropriate person's status may not be obtained. In this case, the person's status that satisfies both the multi-person status control specification and the individual person's status control specification is re-determined. When the appropriate person's status is obtained, the person's status is finally determined. In this way, the person's status can be re-determined, and thus the expression "temporarily determined" can be used.

According to this exemplary embodiment, the human state determination device 15 determines the foreground human state and the background human state. When determining whether the provisionally determined foreground human state is appropriate, the human state determination device 15 makes a determination based on whether the size of a reference part comparable to the size of a reference part of a person of the crowd patch size is obtained, or how the reference part is expressed. When determining whether the provisionally determined background human state is appropriate, the human state determination device 15 makes a determination based on whether the size of a reference part significantly different from the size of a reference part of a person of the crowd patch size is obtained, or how the reference part is expressed.

The human state determining device 15 will be described in more detail below. The human state determining device 15 includes a background human state determining device 12 and a foreground human state determining device 13.

The background person state determination device 12 defines the arrangement of people, the direction of people, the number of people, the shooting angle of people, the lighting of people, the posture of people, the clothing of people, the body shape of people, the hairstyle of people, the size of people when synthesized with the crowd patch, etc., and temporarily determines the state of people corresponding to the background based on the multi-person state control designation stored in the crowd state control designation storage device 23 and the individual person state control designation stored in the person state control designation storage device 24. The background person state determination device 12 determines whether the temporarily determined person state meets the conditions of the background person state, and if the conditions of the background person state are not met, makes a temporary determination of the person state again. If the temporarily determined person state meets the conditions, the background person state determination device 12 finally determines the temporarily determined person state as the state of the person corresponding to the background.

The conditions for the background person status correspond to, for example, the fact that the person is arranged so that the person's reference part is not within the crowd patch, or the fact that the size of the person's reference part is significantly larger or smaller than the size of the reference part stored in the learned local image information storage device 22 when synthesized. Under these conditions, the status of the person corresponding to the background is ultimately determined based on the size of the person's reference part relative to the size of the crowd patch or how the reference part is expressed. The conditions listed here are exemplary, and other conditions can be used for the background person status conditions.

The fact that the reference part of a person is within the crowd patch indicates a state in which more than a predetermined ratio of the area expressing the reference part of the person is captured in the crowd patch. Conversely, the fact that the reference part of a person is not within the crowd patch indicates a state in which less than a predetermined ratio of the area expressing the reference part of the person is captured in the crowd patch. For example, assume that the predetermined ratio was previously defined as 80%. In this case, for example, if 85% of the area expressing the reference part is captured within the crowd patch, it can be said that the reference part of the person is within the crowd patch. For example, if only 20% of the area expressing the reference part is captured in the crowd patch, it can be said that the reference part of the person is not within the crowd patch. 80% is an exemplary ratio, and a value other than 80% may be defined as the predetermined ratio.

According to the present exemplary embodiment, a first threshold value indicating a size larger than the size of the reference part stored in the learned partial image information storage device 22 and a second threshold value indicating a size smaller than the size of the reference part stored in the learned partial image information storage device 22 are predefined. The fact that the size of the reference part of a person is as large as the size of the reference part stored in the learned partial image information storage device 22 when synthesized indicates that the size of the reference part of the person is equal to or larger than the second threshold value and equal to or smaller than the first threshold value when synthesized. The fact that the size of the reference part of the person is much larger than the size of the reference part stored in the learned partial image information storage device 22 when synthesized indicates that the size of the reference part of the person is larger than the first threshold value when synthesized. The fact that the size of the reference part of the person is much smaller than the size of the reference part stored in the learned partial image information storage device 22 when synthesized indicates that the size of the reference part of the person is smaller than the second threshold value when synthesized.

Figures 5(a) to 5(d) are schematic diagrams illustrating examples in which the condition of the background person state is satisfied. In this example, it is assumed that the height size of the reference part of the person (the head in this example) is stored in the learning local image information storage device 22 as 1/α (or h/α pixels) of the height size of h pixels of the crowd patch. The human state shown in Figures 5(a) and 5(b) is in an arranged state, in which the reference part of the person is not found in the crowd patch, and thus the condition of the background person state is satisfied. The human state shown in Figure 5(c) is that the size of the reference part is much smaller than the defined size of the reference part, and thus the condition of the background person state is satisfied. The human state shown in Figure 5(d) is that the size of the reference part is much larger than the defined size of the reference part, and thus the condition of the background person state is satisfied.

The foreground person state determination device 13 defines the arrangement of people, the direction of people, the number of people, the shooting angle of people, the lighting of people, the posture of people, the clothing of people, the body shape of people, the hairstyle of people, the size of people when synthesized with the crowd patch, etc., and temporarily determines the state of the person corresponding to the foreground based on the multi-person state control designation stored in the crowd state control designation storage device 23 and the individual person state control designation stored in the person state control designation storage device 24. The foreground person state determination device 13 then determines whether the temporarily determined person state meets the conditions of the foreground person state, and if the conditions of the foreground person state are not met, a temporary determination of the person state is made again. In addition, if the temporarily determined person state meets the conditions, the foreground person state determination device 13 finally determines the temporarily determined person state as the state of the person corresponding to the foreground.

The conditions for the foreground person status are, for example, that the person's reference part is arranged within the crowd patch and that the size of the person's reference part, when synthesized, is the same as the size of the reference part stored in the learned partial image information storage device 22. Under this condition, the status of the person corresponding to the foreground is ultimately determined based on the size of the person's reference part relative to the size of the crowd patch or how the reference part is expressed. The conditions listed here are exemplary, and other conditions can be used for the foreground person status conditions.

Figures 6(a) to 6(d) are schematic diagrams illustrating examples in which the conditions for the foreground person state are satisfied. As described with reference to Figures 5(a) to 5(d), the height size of the reference part of the person (the head in this example) is assumed to be 1/α (or h/α pixels) of the height size of the h pixels of the crowd patch in the learning local image information storage device 22. Any person state shown in Figures 6(a) to 6(d) is such that the reference part of the person is within the crowd patch and the size of the reference part is the same as the size of the reference part stored in the learning local image information storage device 22. Therefore, any person state shown in Figures 6(a) to 6(d) satisfies the conditions for the foreground person state.

As described above, the multi-person state control designation and individual person state control designation for determining the state of the foreground person and the multi-person state control designation and individual person state control designation for determining the state of the background person can be defined separately by the operator. In this case, the background person state determining device 12 can temporarily determine the state of the person based on the multi-person state control designation and individual person state control designation for determining the state of the background person. The foreground person state determining device 13 can then temporarily determine the state of the person based on the multi-person state control designation and individual person state control designation for determining the state of the foreground person. As described above, when the multi-person state control designation and individual person state control designation are defined for determining the state of the foreground person and for determining the state of the background person, respectively, the number of foreground people and the number of background people can be changed.

The crowd state image synthesis device 14 reads a person image that satisfies the person state (such as the person's direction, the number of people, the shooting angle of the person, the lighting of the person, the person's posture, the person's clothing, the person's body shape, and the person's hairstyle) finally determined by the background person state determination device 12 from the person image storage device 25, and further reads a person region image corresponding to the person image from the person region image storage device 26. The crowd state image synthesis device 14 then crops an image of only the person (or only the person region) from the person image by using the person region image. Similarly, the crowd state image synthesis device 14 reads a person image that satisfies the person state finally determined by the foreground person state determination device 13 from the person image storage device 25, and further reads a person region image corresponding to the person image from the person region image storage device 26. The crowd state image synthesis device 14 then crops an image of only the person from the person image by using the person region image.

The crowd state image synthesis device 14 synthesizes the above-mentioned cropped image of only people with the background image. At this time, the crowd state image synthesis device 14 synthesizes the image of only people, which was cropped based on the human state finally determined by the background human state determination device 12, with the background image based on the "human arrangement" and "human size when synthesized with the crowd patch" determined by the background human state determination device 12. Furthermore, the crowd state image synthesis device 14 synthesizes the image of only people, which was cropped based on the human state finally determined by the foreground human state determination device 13, with the background image based on the "human arrangement" and "human size when synthesized with the crowd patch" determined by the foreground human state determination device 13. Here, the background image is the image normalized by the background extraction device 11. The result of the synthesis is a crowd patch.

When combining images of only people with the background image, the crowd state image combining device 14 sequentially overlaps and combines images of people located at the positions furthest from the camera. For example, when the upper portion of an image is further from the camera, the crowd state image combining device 14 sequentially overlaps and combines images of people located at the upper portion of the screen. When information regarding camera calibration is provided, the crowd state image combining device 14 overlaps and combines images of people located at the farthest position from the camera, taking into account the 3D positions of the people's images.

The above example assumes that the crowd state image synthesis device 14 uses the human region image to crop an image containing only people from a human image and synthesizes the image containing only people with a background image. The crowd state image synthesis device 14 can divide the human image read from the human image storage device 25 into a human region and a non-human region based on the human region image corresponding to the human image, weight the human region and the non-human region, and then combine and synthesize the human image based on the weights. In this case, the human region is weighted more heavily than the non-human region. The weights can be varied between these regions.

As described above, the data processing device 1 may include a human image generating means (not shown) for generating a human image matching a specified human state by CG, etc. In this case, the human image generating means (not shown) generates a human image matching the human state determined by the background human state determining means 12 or the human state determined by the foreground human state determining means 13, and the crowd state image synthesizing means 14 may synthesize the human image to thereby generate a crowd patch.

When generating crowd patches, the crowd state image synthesis device 14 reads training labels from the crowd state control specification storage device 23 and the person state control specification storage device 24. Specifically, the crowd state image synthesis device 14 reads the contents of the multi-person state control specification for items with designated training labels from the crowd state control specification storage device 23, and reads the contents of the individual person state control specification for items with designated training labels from the person state control specification storage device 24. The crowd state image synthesis device 14 then outputs a pair of crowd patches and training labels. The crowd patches and training labels serve as training data for machine learning to identify crowd states in images.

The control means 16 causes the background extraction means 11, the human state determination means 15 (specifically, the background human state determination means 12 and the foreground human state determination means 13), and the crowd state image synthesis means 14 to repeatedly perform a series of processes. As a result, the data processing device 1 outputs a large number of pairs of crowd patches and training labels.

When changing the person status designation or training label, the operator resets the multi-person status control designation, individual person status control designation, and the designated training label so that the data processing device 1 outputs a large number of pairs of crowd patches and training labels according to the settings. Thus, the operator can obtain a large amount of desired training data.

FIG7 is a block diagram illustrating an exemplary structure of a crowd state recognition device according to the present invention. The crowd state recognition device 30 according to the present invention recognizes the state of a crowd in a given image. The crowd state recognition device 30 includes an image acquisition device 3, a data processing device 4 operating under program control, and a storage device 5 for storing information therein.

The image acquisition device 3 is a camera for acquiring images to be recognized for a crowd state.

The storage device 5 includes a search window storage means 51 and a crowd state recognition dictionary storage means 52 .

The search window storage device 51 stores a set of rectangular areas indicating the portion of the image to be identified for the crowd state. The rectangular area can be referred to as a search window. The set of rectangular areas can be set by defining the change size of the crowd patch according to the position on the image based on the camera parameters indicating the position, posture, focal length and lens distortion of the image acquisition device 3 and the size of the reference part corresponding to the crowd patch size (the size of the reference part stored in the learning local image information storage device 22). For example, the size of the reference part of the person photographed in the image can be derived from the camera parameters. When the size of the reference part of the person stored in the learning local image information storage device 22 is enlarged or reduced according to the size of the reference part, the size of the crowd patch is enlarged or reduced at a magnification or reduction rate, thereby setting the size of the rectangular area. The set of rectangular areas can be set to cover the position on the image. The set of rectangular areas can be freely set without being limited to this method. In addition, the set of rectangular areas can be set to overlap.

The crowd state recognition dictionary storage device 52 stores a dictionary of discriminators learned using the training data (a large number of pairs of crowd patches and training labels) generated by the training data generation device 10 shown in FIG1 . A discriminator is an algorithm for recognizing crowd states, and the discriminator dictionary is used to perform crowd state recognition processing based on the algorithm. The discriminator dictionary stored in the crowd state recognition dictionary storage device 52 is obtained, for example, by performing machine learning using a large number of pairs of crowd patches and training labels generated by the training data generation device 10. The machine learning may be well-known machine learning.

The data processing device 4 includes a crowd status recognition device 41 .

The crowd state recognition means 41 extracts the local area image corresponding to the set of rectangular areas stored in the search window storage means 51 from the image acquired by the image acquisition device 3, and normalizes the extracted local area image to match the crowd patch size. The crowd state recognition means 41 then recognizes (determines) the crowd state in the normalized local area image based on the recognition algorithm (or discriminator) by using the dictionary of the discriminator stored in the crowd state recognition dictionary storage means 52.

The training data generation device 10 shown in FIG1 can generate a large amount of training data (pairs of crowd patches and training labels) as desired by the operator. The crowd state recognition device 41 recognizes the crowd state in the local area image by using a dictionary of discriminators obtained as a result of machine learning using this training data. Thus, the crowd state recognition device 30 can recognize various crowd states.

FIG8 is a schematic diagram illustrating how to identify the degree of congestion (the number of people) as the state of a crowd in an image by way of example. For example, it is assumed that the operator of the training data generating device 10 mainly controls the "number of people" in a step-by-step manner and obtains a number of crowd patches and training labels (see the upper part in FIG8 ). It is then assumed that the dictionary of the discriminator obtained by machine learning based on the training data is stored in the crowd state recognition dictionary storage device 52. In the image 61 shown in FIG8 , the rectangular area from which the local area image is extracted is indicated by a dotted line. The recognition result of the crowd state of the local area image extracted from the rectangular area is expressed corresponding to the dotted area. This applies to FIG9 to FIG11 described below. In addition, the actual rectangular area is basically set to cover the entire image, but only some rectangular areas are illustrated by way of example for the purpose of simply illustrating the recognition result. In this example, the crowd state recognition device 41 can recognize the number of people (degree of congestion) in various areas in the image 61 as shown in FIG8 .

FIG9 is a schematic diagram illustrating, by way of example, how to identify the direction of a crowd as a crowd state in an image. For example, it is assumed that the operator of the training data generation device 10 primarily controls "direction of people" and acquires a number of crowd patches and training labels (see the upper portion of FIG9 ). It is then assumed that a dictionary of discriminators acquired through machine learning based on the training data is stored in the crowd state recognition dictionary storage device 52. In this example, the crowd state recognition device 41 can identify the direction of a crowd in various areas of the image 62, as shown in FIG9 .

FIG10 is a schematic diagram illustrating, by way of example, how to identify a non-abnormal crowd (non-significantly crowded crowd) or an abnormal crowd (significantly crowded crowd) as a crowd state in an image. For example, it is assumed that the operator of the training data generating device 10 mainly controls the "number of people" and acquires a number of crowd patches and training labels. Here, it is assumed that a large amount of training data is acquired according to two categories including when the number of people is less than n and when the number of people is n or greater (see the upper part of FIG10 ). It is then assumed that a dictionary of discriminators obtained by machine learning based on the training data is stored in the crowd state recognition dictionary storage device 52. In this example, the crowd state recognition device 41 can recognize whether the crowd state in various areas in the image 63 is a non-abnormal crowd or an abnormal crowd as shown in FIG10 .

FIG11 is a schematic diagram illustrating, by way of example, how to identify a disordered state (people's directions are not uniform) or an ordered state (people's directions are uniform) as the state of a crowd in an image. For example, it is assumed that the operator of the training data generating device 10 obtains a large amount of training data according to two categories, including when "people's directions" are uniform and when they are not uniform (see the upper part of FIG11). It is then assumed that a dictionary of discriminators obtained through machine learning based on the training data is stored in the crowd state recognition dictionary storage device 52. In this example, the crowd state recognition device 41 can identify whether the crowd state in various areas of the image 64 is a disordered state or an ordered state, as shown in FIG11.

Because a large amount of operator-desired training data can be generated, the crowd state recognition device 41 can recognize various states, such as a discrete state in which a crowd disperses and runs, a gathering state in which a crowd gathers at one location, an avoidance state in which a crowd avoids something, a hanging state indicating a special crowd cluster, and a line state, in addition to the situations shown in Figures 8 to 11.

The processing procedure of the training data generating device 10 according to the present invention will be described below. FIG12 is a flowchart illustrating an exemplary processing progress of the training data generating device 10.

The background extraction means 11 selects a background image from the group of background images stored in the background image storage means 21 and extracts an image to be used as a background of a crowd patch (step S1 ).

FIG13 is a flowchart illustrating an exemplary process of step S1. In step S1, the background extraction device 11 first selects a background image from the set of background images stored in the background image storage device 21 (step S101). The selection method is not particularly limited. For example, the background extraction device 11 may select any one background image from the set of background images.

The background extraction means 11 then calculates the aspect ratio of the crowd patches stored in the learned partial image information storage means 22 and temporarily extracts a background of an appropriate position and an appropriate size to satisfy the aspect ratio from the selected background image (step S102 ).

The background extraction means 11 enlarges or reduces (or normalizes) the temporarily extracted background image to match the size of the crowd patch, thereby acquiring an image as the background of the crowd patch (step S103 ). This is the end of step S1 .

After step S1, the background person state determination means 12 determines the state of the person corresponding to the background (step S2).

14 is a flowchart illustrating an exemplary processing progress of step S2. The background person state determination means 12 defines the arrangement of people, the direction of people, the number of people, the shooting angle of people, the lighting of people, the posture of people, the clothing of people, the body shape of people, the hairstyle of people, the size of people when synthesized with the crowd patch, etc., and temporarily determines the state of people corresponding to the background based on the multi-person state control designation stored in the crowd state control designation storage means 23 and the individual person state control designation stored in the person state control designation storage means 24 (step S201).

The background person state determining means 12 then determines whether the person state temporarily determined in step S201 satisfies the condition of the background person state (step S202). The condition has been described above, so its description will be omitted here.

The multi-person state control designation or the individual person state control designation may include a "random" designation, etc., and thus the state temporarily determined in step S201 may not satisfy the condition of the background person state. In this case ("No" in step S202), the background person state determination means 12 repeatedly performs the processing in and after step S201.

When the state temporarily determined in step S201 satisfies the condition of the background person state ("Yes" in step S202), the background person state determining means 12 defines the latest person state temporarily determined in step S201 as the state of the person corresponding to the background (step S203). This is the end of step S2.

After step S2, the foreground person state determination means 13 determines the state of the person corresponding to the foreground (step S3).

FIG15 is a flowchart illustrating an exemplary processing progress of step S3. The foreground person state determining means 13 defines the arrangement of people, the direction of people, the number of people, the shooting angle of people, the lighting of people, the posture of people, the clothing of people, the body shape of people, the hairstyle of people, the size of people when synthesized with the crowd patch, etc., and temporarily determines the state of the person corresponding to the foreground based on the multi-person state control designation stored in the crowd state control designation storage means 23 and the individual person state control designation stored in the person state control designation storage means 24 (step S301).

The foreground person state determining means 13 then determines whether the person state temporarily determined in step S301 satisfies the foreground person state condition (step S302). The condition has been described above, so its description will be omitted here.

The multi-person state control designation or the individual person state control designation may include a "random" designation, etc., and thus the state temporarily determined in step S301 may not satisfy the condition of the foreground person state. In this case ("No" in step S302), the foreground person state determination means 13 repeatedly performs the processing in and after step S301.

When the state temporarily determined in step S301 satisfies the foreground person state condition ("Yes" in step S302), the foreground person state determining means 13 defines the latest person state temporarily determined in step S301 as the state of the person corresponding to the foreground (step S303). This is the end of step S3.

After step S3 , the crowd state image synthesizing means 14 generates crowd patches based on the human states determined in steps S2 and S3 , reads training labels corresponding to the crowd patches, and outputs pairs of the crowd patches and the training labels (step S4 ).

16 is a flowchart illustrating an exemplary processing progress of step S4. The crowd state image synthesizing means 14 selects and reads a person image that satisfies the person state (such as the direction of the person, the number of people, the shooting angle of the person, the lighting for the person, the posture of the person, the clothing of the person, the body shape of the person, and the hairstyle of the person) determined in steps S2 and S3 from the group of person images in the person image storage means 25 (step S401).

The crowd state image synthesizing means 14 then reads each person region image corresponding to each person image selected at step S401 from the person region image storage means 26. The crowd state image synthesizing means 14 crops only a person image for each person image by using the person region image corresponding to the person image (step S402).

The crowd state image synthesizing means 14 determines the arrangement state for each of the images containing only people generated in step S402 based on the “arrangement of people” and the “size of people when synthesized with the crowd patch” determined in steps S2 and S3 (step S403). The crowd state image synthesizing means 14 then synthesizes each of the images containing only people with the background image acquired in step S1 based on the arrangement state to thereby generate a crowd patch (step S404).

The crowd state image synthesizing means 14 then obtains the training labels corresponding to the crowd patches (step S405). That is, the crowd state image synthesizing means 14 reads the contents of the multi-person state control designation of the item with the specified training label from the crowd state control designation storage means 23, and reads the contents of the individual person state control designation of the item with the specified training label from the person state control designation storage means 24. The read contents correspond to the training labels.

The crowd state image synthesizing device 14 outputs a pair of the crowd patch generated in step S404 and the training label obtained in step S405 (step S406 ). This is the end of step S4 .

After step S4, the control device 16 determines whether the number of repetitions of the processes in steps S1 to S4 has reached a predetermined number of times (step S5). When the number of repetitions of the processes in steps S1 to S4 has not reached the predetermined number of times ("No" in step S5), the control device 16 causes the background extraction device 11, the human state determination device 15 (specifically, the background human state determination device 12 and the foreground human state determination device 13), and the crowd state image synthesis device 14 to repeatedly perform the processes in steps S1 to S4.

When the number of repetitions of the processes in steps S1 to S4 reaches a predetermined number ("YES" in step S5), the processes are terminated.

The processing in steps S1 to S4 is performed once to obtain a pair of crowd patches and training labels. Therefore, the data processing device 1 repeatedly performs the processing in steps S1 to S4 a predetermined number of times to obtain a large amount of training data. For example, when the predetermined number of times is defined as 100,000, 100,000 pairs of crowd patches and training labels that match multi-person status control indications and individual person status control indications are obtained.

The order of steps S1 , S2 and S3 may be replaced in the flowchart shown in FIG. 12 .

The processing progress of the crowd state recognition device 30 according to the present invention will be described below. FIG17 is a flowchart illustrating an exemplary processing progress of the crowd state recognition device 30 ...

The image acquisition device 3 acquires an image to be recognized for a crowd state, and inputs the image into the crowd state recognition means 41 (step S21 ).

The crowd state recognition means 41 then determines whether the entire set of rectangular areas stored in the search window storage means 51 has been selected (step S22).

When there is an unselected rectangular area in the group of rectangular areas stored in the search window storage device 51 ("No" in step S22), the crowd state recognition device 41 selects an unselected rectangular area from the group of rectangular areas (step S23).

The crowd state recognition device 41 then extracts a local area image corresponding to the selected rectangular area from the image input in step S21 (step S24). The crowd state recognition device 41 then normalizes the local area image to match the crowd patch size (step S25).

The crowd state recognition means 41 then recognizes the crowd state within the normalized local area image by using the dictionary of discriminators stored in the crowd state recognition dictionary storage means 52 (step S26 ).

After step S26, the crowd state recognition means 41 repeatedly performs the processing in and after step S22. When determining that the entire set of rectangular areas has been selected ("Yes" in step S22), the crowd state recognition means 41 then terminates the processing.

By using the training data generating device according to the present invention, the human state determining means 15 determines the states of the people constituting the crowd based on the multi-person state control designation (state designation for multiple people, such as "arrangement of people", "direction of people" and "number of people") and the individual person state control designation (state designation for individual people, such as "shooting angle of people", "lighting on people", "posture of people", "clothing of people", "body shape of people", "hair style of people" and "size of people when synthesized with crowd patches") defined by the operator. The crowd state image synthesizing means 14 then synthesizes the image of people in the determined state to thereby generate a crowd patch, and reads the training label corresponding to the crowd patch. Then, the process of determining the human state, generating the crowd patch and specifying the training label is repeatedly performed a predetermined number of times so that a large amount of various training data (a plurality of pairs of crowd patches and training labels) of the crowd state desired by the operator can be automatically generated.

In addition, if a large amount of such training data is acquired, the dictionary of the discriminator can be machine-learned based on the training data. The crowd state recognition device 30 can then easily recognize complex crowd states in a still image by using the dictionary.

The crowd state recognition device 41 in the crowd state recognition apparatus 30 recognizes the state of a crowd in a given image using a dictionary learned based on crowd patches expressing a crowd and training labels corresponding to the crowd patches. Therefore, the crowd state recognition device 41 recognizes the state of a crowd not based on a single object such as a person's head, but rather on a larger unit of a group of people whose reference parts are captured. This allows the recognition of crowd states in small areas where heads or individual people cannot be recognized.

In the crowd state recognition device 30 according to the present invention, the crowd state recognition means 41 recognizes the crowd state by using a dictionary (a discriminator's dictionary). Therefore, the accuracy of recognizing the crowd state is independent of the frame rate. Thus, the crowd state recognition device according to the present invention can preferably recognize the crowd state in an image regardless of the frame rate. For example, the crowd state recognition device 30 according to the present invention can preferably recognize the crowd state even in a still image.

The training data generation device 10 according to the above exemplary embodiment determines the human state of "human arrangement" such as overlap between people based on the multi-person state control specification, and generates a crowd patch indicating the human state. When machine learning is performed using such a crowd patch, states including occlusion between people can also be learned. Therefore, even when overlap (occlusion) between people occurs, which is difficult to recognize by head recognition or person recognition, the crowd state recognition device 30 can preferably recognize the crowd state by using the dictionary acquired as a result of learning.

The training data generation device 10 according to the above exemplary embodiment determines a person state, generates a crowd patch that captures the person in that state, and assigns a training label corresponding to the crowd patch based on information indicating the person state of multiple people (multi-person state control designation) and information indicating the person state of each person (individual person state control designation). Thus, the operator defines the multi-person state control designation or the individual person state control designation to easily obtain training data for identifying different types of crowd states. The training data is then machine-learned, thereby easily creating a crowd state recognition device 30 for identifying different types of crowd states.

According to the above exemplary embodiment, if the camera parameters indicating the position, posture, focal length, and lens distortion of the image acquisition device (camera) 3 in a crowd shooting environment can be obtained, a multi-person state control designation or an individual person state control designation limited to that environment can be defined by using the camera parameters. The training data generation device 10 determines the state of a person based on the multi-person state control designation or the individual person state control designation and generates training data, thereby learning a dictionary of discriminators suitable for the crowd shooting environment. As a result, the crowd state recognition device 30 can recognize complex crowd states in still images, etc. with high accuracy.

According to the exemplary embodiment described above, if camera parameters indicating the position, posture, focal length, and lens distortion of the image acquisition device 3 in the recognition environment can be acquired, the human state of each person and the human state of each person can be controlled according to the local area on the image. Then, by synthesizing human images based on the controlled human state, a large number of crowd patches desired by the operator and training labels corresponding to the crowd patches can be automatically generated. Then, based on the crowd patches and training labels, a discriminator dictionary can be learned according to the local area on the image, and by using the discriminator dictionary according to the area on the image, the accuracy of recognizing complex crowd states can be increased.

The specific structure of the training data generating device and the crowd state recognition device according to the present invention will be described below by way of example. Figure 18 is a block diagram illustrating the specific structure of the training data generating device according to the present invention by way of example. Components identical to those shown in Figure 1 are denoted by the same reference numerals as in Figure 1, and their detailed descriptions will be omitted. In the exemplary structure shown in Figure 18, a storage device 2 including a background image storage device 21, a learning local image information storage device 22, a crowd state control indication storage device 23, a human state control indication storage device 24, a human image storage device 25, and a human area image storage device 26 is connected to the computer 100. A computer-readable storage medium 102 for storing the training data generating program 101 therein is also connected to the computer 100.

The computer-readable storage medium 102 is implemented by, for example, a magnetic disk, a semiconductor memory, or the like. For example, when activated, the computer 100 reads the training data generation program 101 from the computer-readable storage medium 102. The computer 100 then operates according to the training data generation program 101 as the background extraction device 11, the human state determination device 15 (more specifically, the background human state determination device 12 and the foreground human state determination device 13), the crowd state image synthesis device 14, and the control device 16 in the data processing apparatus 1 shown in FIG.

FIG19 is a block diagram illustrating, by way of example, the specific structure of a crowd state recognition device according to the present invention. Components identical to those shown in FIG7 are denoted by the same reference numerals as in FIG7 , and their detailed descriptions will be omitted. In the exemplary structure shown in FIG19 , a storage device 5 including a search window storage device 51 and a crowd state recognition dictionary storage device 52 is connected to a computer 150. A computer-readable storage medium 104 storing a crowd state recognition program 103 is also connected to the computer 150.

The computer-readable storage medium 104 is implemented by, for example, a magnetic disk, a semiconductor memory, etc. For example, when activated, the computer 150 reads the crowd state recognition program 103 from the computer-readable storage medium 104. The computer 100 then operates as the crowd state recognition device 41 in the data processing apparatus 4 shown in FIG. 7 according to the crowd state recognition program 103.

In the above exemplary embodiment, the case where the crowd state recognition dictionary storage device 52 (see FIG. 7 ) stores a dictionary acquired through learning using the training data generated by the training data generating device 10 (see FIG. 1 ) has been described. In other words, in the above exemplary embodiment, the case where a dictionary acquired through machine learning using a plurality of pairs of crowd patches and training labels of the crowd patches is stored in the crowd state recognition dictionary storage device 52 has been described, the crowd patches being acquired by synthesizing images of people that match the state of people controlled to a desired state.

The crowd state recognition dictionary storage device 52 can store, as training data, a dictionary acquired through machine learning using data other than the training data generated by the training data generation device 10. Even for training data other than the training data generated by the training data generation device 10, multiple pairs of crowd patches and training labels for the crowd patches are prepared and used as training data. The crowd patches include a person whose reference parts are expressed as large as the reference parts of the person defined for the size of the crowd patches. That is, a dictionary of discriminators acquired through machine learning using multiple pairs of crowd patches and training labels can be stored in the crowd state recognition dictionary storage device 52. In this case, too, the crowd state in an image can be preferably recognized regardless of the frame rate.

The main parts of the present invention will be described below. Figure 20 is a block diagram illustrating the main parts of the training data generation device according to the present invention. The training data generation device according to the present invention includes a background extraction unit 71, a human state determination unit 72, and a crowd state image synthesis unit 73.

The background extraction unit 71 (for example, the background extraction device 11 ) selects a background image from a plurality of pre-prepared background images, extracts a region in the background image, and enlarges or reduces the image corresponding to the extracted region to an image of a predetermined size.

The human state determination unit 72 (for example, the human state determination device 15) determines the human state of the crowd based on the multi-person state control indication as indication information about the human state of people and the individual person state control indication as indication information about the state of individual people among the crowd.

The crowd state image synthesis unit 73 generates a crowd state image (such as a crowd patch) which is an image in which a human image corresponding to the human state determined by the human state determination unit 72 is synthesized with an image of a predetermined size obtained by the background extraction unit 71, specifies a training label of the crowd state image, and outputs a pair of crowd state images and training labels.

For example, the background extraction unit 71, the human state determination unit 72, and the crowd state image synthesis unit 73 repeat these operations sequentially. The operations of the background extraction unit 71, the human state determination unit 72, and the crowd state image synthesis unit 73 may not be performed sequentially. For example, the background extraction unit 71 and the human state determination unit 72 may perform operations in parallel.

With this structure, a large amount of training data for machine learning a dictionary of a discriminator for recognizing crowd states can be easily generated.

21 is a block diagram illustrating the main parts of the crowd state recognition device according to the present invention. The crowd state recognition device according to the present invention includes a rectangular area group storage unit 81, a crowd state recognition dictionary storage unit 82, and a crowd state recognition unit 83.

The rectangular region group storage unit 81 (eg, the search window storage device 51 ) stores a group of rectangular regions indicating portions on an image to be recognized for a crowd state.

The crowd state recognition dictionary storage unit 82 (e.g., the crowd state recognition dictionary storage device 52) stores a dictionary of discriminators obtained by machine learning using multiple pairs of crowd state images (such as crowd patches) and training labels of the crowd state images, where the crowd state image is an image including a person whose reference part is represented as the same size as the reference part of the person defined for a predetermined size of the image representing the crowd state.

The crowd state recognition unit 83 (eg, the crowd state recognition device 41 ) extracts the area indicated by the group of rectangular areas stored in the rectangular area group storage unit 81 from a given image, and recognizes the state of the crowd captured in the extracted image based on a dictionary.

With this structure, the state of a crowd in an image can be preferably recognized regardless of the frame rate.

Part or all of the above exemplary embodiments may be as described in the following supplementary notes, but are not limited to the following.

(Supplementary Note 1)

A training data generating device, comprising:

Background extraction means for selecting a background image from a plurality of pre-prepared background images, extracting a region in the background image, and enlarging or reducing an image corresponding to the extracted region to an image of a predetermined size;

a person state determining means for determining the person state of a group of people based on a multi-person state control designation as designation information on the person state of a plurality of people and an individual person state control designation as designation information on the state of an individual person among the plurality of people; and

Crowd state image synthesis means is used to generate a crowd state image, specify a training label for the crowd state image, and output a pair of the crowd state image and the training label, wherein the crowd state image is an image in which a human image corresponding to the human state determined by the human state determination means is synthesized with an image of a predetermined size acquired by the background extraction means.

(Supplementary Note 2)

The training data generating device according to Supplementary Note 1,

The human state determination device temporarily determines the human state of the crowd based on the multi-person state control indication and the individual person state control indication, and when the temporarily determined human state meets the conditions of the size of the reference part of a person defined for a predetermined size and how the reference part is expressed, the temporarily determined human state is determined as the human state of the crowd, and when the temporarily determined human state does not meet these conditions, the temporary determination of the human state of the crowd is repeated.

(Supplementary Note 3)

The training data generating device according to Supplementary Note 1 or 2, comprising:

Crowd state control indication storage means for storing a multi-person state control indication defined according to a project and storing the existence of a training label for the indication defined for the project; and

A person state control indication storage device for storing individual person state control indications defined according to a project and storing the existence of training labels for the indications defined for the project,

wherein the human state determining means determines the human state of the crowd according to the multi-person state control indication stored in the crowd state control indication storage means and the individual human state control indication stored in the human state control indication storage means, and

The crowd state image synthesis device specifies the training label by reading the multi-person state control designation defined as an item with the specified training label from the crowd state control designation storage device, and reading the individual person state control designation defined as an item with the specified training label from the person state control designation storage device.

(Supplementary Note 4)

The training data generating device according to Supplementary Note 3,

wherein the crowd state control specifies that the storage device stores at least one item as having a specified training label, and

The crowd state image synthesizing means reads a multi-person state control specification defined as an item having a specified training label from the crowd state control specification storage means.

(Supplementary Note 5)

The training data generating device according to Supplementary Note 3 or 4,

wherein the crowd state control indication storage device stores the multi-person state control indication and the existence of the indicated training label according to items such as the arrangement of people, the direction of people, and the number of people, and stores the multi-person state control indication corresponding to each item in any one of a first form indicating a specific state, a second form indicating that an arbitrary state can be defined, and a third form indicating that a state can be defined within a predetermined rule,

The person state control indication storage device stores the existence of individual person state indications and the indicated training labels according to items such as the shooting angle of the person, the lighting of the person, the posture of the person, the clothes of the person, the body shape of the person, the hairstyle of the person, and the size of the person when synthesized with the crowd state image, and stores the individual person state control indication corresponding to each item in any one of the first form, the second form, and the third form, and

The human state determining means determines the human state of the crowd based on the multi-person state control designation stored in the crowd state control designation storage means and the individual human state control designation stored in the human state control designation storage means.

(Supplementary Note 6)

The training data generating device according to any one of Supplementary Notes 1 to 5,

The crowd state image synthesis device selects a human image that matches a determined human state such as the direction of the human, the number of people, the shooting angle of the human, the lighting of the human, the posture of the human, the clothes of the human, the body shape of the human and the hairstyle of the human from a group of pre-prepared human images, crops the human area from the selected human image, thereby generating an image of only the human, and synthesizes the image of only the human with an image of a predetermined size obtained by the background extraction device based on the arrangement of the people determined to be in human state and the size of the people when synthesized with the crowd state image.

(Supplementary Note 7)

The training data generating device according to Supplementary Note 6,

The crowd state image synthesis device sequentially synthesizes the image of only people corresponding to the farthest arrangement position from the camera with the image of a predetermined size acquired by the background extraction device.

(Supplementary Note 8)

The training data generating device according to any one of Supplementary Notes 1 to 7,

Wherein, the human state determination device includes:

Background human state determining means for temporarily determining the human state of a crowd serving as a background in a crowd state image based on a multi-person state control designation and an individual person state control designation, determining the temporarily determined human state as the human state of the crowd serving as the background when the temporarily determined human state satisfies a first condition of a size of a reference part of a person defined for a predetermined size of the crowd state image and how the reference part is expressed, and repeatedly performing the temporary determination of the human state of the crowd serving as the background when the temporarily determined human state does not satisfy the first condition;

A foreground person state determination device is used to temporarily determine the person state of the crowd serving as the foreground in a crowd state image based on a multi-person state control indication and an individual person state control indication, and to determine the temporarily determined person state as the person state of the crowd serving as the foreground when the temporarily determined person state meets a second condition of the size of a reference part of a person defined for a predetermined size of the crowd state image and how the reference part is expressed, and to repeatedly perform the temporary determination of the person state of the crowd serving as the foreground when the temporarily determined person state does not meet the second condition.

(Supplementary Note 9)

The training data generating device according to Supplementary Note 8,

The first condition is that the reference part of the person is not within the crowd state image, or the size of the reference part is much larger or smaller than the size of the reference part of the person defined for the predetermined size, and

The second condition is that the reference part of the person is within the crowd state image, and the size of the reference part is as large as the size of the reference part of the person defined for a predetermined size.

(Supplementary Note 10)

A crowd status recognition device, comprising:

Rectangular region group storage means for storing a group of rectangular regions indicating portions of an image to be recognized for crowd status;

a crowd state recognition dictionary storage device for storing a dictionary of a discriminator obtained by performing machine learning using a plurality of pairs of crowd state images and training labels of the crowd state images, the crowd state images being images expressing a crowd state in a predetermined size and including images of persons whose reference parts are expressed as large as the size of the reference parts of persons defined for the predetermined size; and

Crowd state recognition means is configured to extract, from a given image, an area indicated in the group of rectangular areas stored in the rectangular area group storage means, and recognize the state of a crowd captured in the extracted image based on a dictionary.

(Supplementary Note 11)

According to the crowd state recognition device described in Supplementary Note 10,

wherein the crowd state recognition dictionary storage device stores a dictionary of discriminators obtained by machine learning using a plurality of pairs of crowd state images and training labels of the crowd state images, the crowd state images being obtained by synthesizing images of people whose states match those of people controlled to a desired state, and

The crowd state recognition means recognizes the state of a crowd captured in an image based on a dictionary.

(Supplementary Note 12)

The crowd state recognition device according to Supplementary Note 10 or 11,

wherein the rectangular area group storage means stores rectangular areas defined in a set of sizes based on camera parameters indicating the position, posture, focal length, and lens distortion of an image acquisition device used to acquire an image, and sizes of reference parts of a person defined for predetermined sizes, and

The crowd state recognition device extracts an area indicated in the set of rectangular areas from a given image.

(Supplementary Note 13)

The crowd state recognition device according to any one of Supplementary Notes 10 to 12,

wherein the crowd state recognition dictionary storage means stores a dictionary of discriminators obtained by changing the number of people represented in a crowd state image and by performing machine learning using a plurality of pairs of crowd state images and training labels prepared for the number of people, and

The crowd state recognition means recognizes the number of persons in a crowd captured in an image based on a dictionary.

(Supplementary Note 14)

The crowd state recognition device according to any one of Supplementary Notes 10 to 13,

wherein the crowd state recognition dictionary storage means stores a dictionary of discriminators obtained by changing the directions of people represented in crowd state images and by performing machine learning using a plurality of pairs of crowd state images and training labels prepared for these directions of people, and

The crowd state recognition device recognizes the direction of a crowd captured in an image based on a dictionary.

(Supplementary Note 15)

The crowd state recognition device according to any one of Supplementary Notes 10 to 14,

wherein the crowd state recognition dictionary storage means stores a dictionary of discriminators obtained by machine learning using a plurality of pairs of crowd state images and training labels prepared for non-significantly crowded crowds and significantly crowded crowds, and

The crowd state recognition means recognizes whether a crowd captured in an image is remarkably crowded based on a dictionary.

(Supplementary Note 16)

The crowd state recognition device according to any one of Supplementary Notes 10 to 15,

wherein the crowd state recognition dictionary storage means stores a dictionary of discriminators obtained by machine learning using pairs of crowd state images and training labels prepared for a crowd in which the directions of people are uniform and a crowd in which the directions of people are not uniform, and

The crowd state recognition device recognizes whether the directions of people in a crowd captured in an image are unified based on a dictionary.

(Supplementary Note 17)

A training data generation method, comprising:

A background extraction step of selecting a background image from a plurality of pre-prepared background images, extracting a region in the background image, and enlarging or reducing an image corresponding to the extracted region to an image of a predetermined size;

a person state determining step of determining the person state of the group based on the multi-person state control designation as designation information on the person state of the plurality of persons and the individual person state control designation as designation information on the state of an individual person among the plurality of persons; and

A crowd state image synthesis step generates a crowd state image, specifies a training label for the crowd state image, and outputs a pair of the crowd state image and the training label, wherein the crowd state image is an image in which a human image corresponding to the human state determined in the human state determination step is synthesized with an image of a predetermined size obtained in the background extraction step.

(Supplementary Note 18)

The training data generation method according to Supplementary Note 17 includes:

The human state of a group of people is temporarily determined based on the multi-person state control indication and the individual person state control indication. When the temporarily determined human state meets the conditions of the size of the human reference part defined for the predetermined size and how the reference part is represented, the temporarily determined human state is determined as the human state of the group of people, and when the temporarily determined human state does not meet these conditions, the human state determination step of temporarily determining the human state of the group of people is repeated.

(Supplementary Note 19)

According to the training data generation method described in Supplementary Note 17 or 18,

wherein the crowd state control indication storage device stores a multi-person state control indication defined according to a project and stores the existence of a training label for the indication defined for the project, and

The person state control designation storage device stores individual person state control designations defined according to a project and stores the existence of training labels for the designations defined for the project.

The method includes:

a human state determining step of determining the human state of the crowd based on the multi-person state control designation stored in the crowd state control designation storage device and the individual human state control designation stored in the human state control designation storage device; and

The crowd state image synthesis step specifies the training label by reading the multi-person state control indication defined as an item with the specified training label from the crowd state control indication storage device and reading the individual person state control indication defined as an item with the specified training label from the person state control indication storage device.

(Supplementary Note 20)

According to the training data generation method described in Supplementary Note 19,

wherein the crowd state control instructs the storage device to store at least one item as having a specified training label, and

The method includes a crowd state image synthesis step of reading a multi-person state control specification defined as an item having a specified training label from a crowd state control specification storage device.

(Supplementary Note 21)

According to the training data generation method described in Supplementary Note 19 or 20,

wherein the crowd state control designation storage device stores the multi-person state control designation and the existence of the designated training label according to items such as arrangement of people, direction of people, and number of people, and stores the multi-person state control designation corresponding to each item in any one of a first form indicating a specific state, a second form indicating that an arbitrary state can be defined, and a third form indicating that a state can be defined within a predetermined rule, and

The person state control indication storage device stores the existence of individual person state indications and the indicated training labels according to items such as the shooting angle of the person, the lighting of the person, the posture of the person, the clothes of the person, the body shape of the person, the hairstyle of the person, and the size of the person when synthesized with the crowd state image, and stores the individual person state control indication corresponding to each item in any one of the first form, the second form, and the third form.

The method includes a human state determining step of determining the human state of a crowd based on multi-person state control indications stored in a crowd state control indication storage device and individual human state control indications stored in a human state control indication storage device.

(Supplementary Note 22)

The training data generation method according to any one of Supplementary Notes 17 to 21, comprising:

A crowd state image synthesis step selects a person image that matches a determined person state such as the person's direction, the number of people, the shooting angle of the person, the lighting of the person, the person's posture, the person's clothes, the person's body shape and the person's hairstyle from a group of pre-prepared person images, crops the person's area from the selected person image, thereby generating an image of only the person, and synthesizes the image of only the person with an image of a predetermined size obtained by a background extraction device based on the arrangement of the people determined to be in the human state and the size of the person when synthesized with the crowd state image.

(Supplementary Note 23)

The training data generation method according to Supplementary Note 22 includes:

The crowd state image synthesis step sequentially synthesizes the image of only people corresponding to the farthest arrangement position from the camera with the image of a predetermined size acquired by the background extraction device.

(Supplementary Note 24)

According to the training data generation method described in any one of Supplementary Notes 17 to 23,

The steps of determining the status of a person include:

a background person state determining step of temporarily determining the person state of the crowd serving as the background in the crowd state image based on the multi-person state control designation and the individual person state control designation, determining the temporarily determined person state as the person state of the crowd serving as the background when the temporarily determined person state satisfies a first condition of how the size of a reference part of a person and the reference part are represented defined for a predetermined size of the crowd state image, and repeatedly making the temporary determination of the person state of the crowd serving as the background when the temporarily determined person state does not satisfy the first condition;

A foreground person state determination step is to temporarily determine the person state of the crowd serving as the foreground in the crowd state image based on the multi-person state control indication and the individual person state control indication, and to determine the temporarily determined person state as the person state of the crowd serving as the foreground when the temporarily determined person state meets a second condition of the size of the person's reference part defined for a predetermined size of the crowd state image and how the reference part is represented, and to repeatedly make a temporary determination of the person state of the crowd serving as the foreground when the temporarily determined person state does not meet the second condition.

(Supplementary Note 25)

According to the training data generation method described in Supplementary Note 24,

The first condition is that the reference part of the person is not within the crowd state image or the size of the reference part is much larger or smaller than the size of the reference part of the person defined for the predetermined size, and

The second condition is that the reference part of the person is within the crowd state image and the size of the reference part is as large as the size of the reference part of the person defined for a predetermined size.

(Supplementary Note 26)

A crowd state recognition method,

wherein the rectangular region group storage means stores a group of rectangular regions indicating portions of the image to be recognized for the crowd state, and

The crowd state recognition dictionary storage device stores a dictionary of discriminators obtained by performing machine learning using a plurality of pairs of crowd state images and training labels of the crowd state images, the crowd state images being images expressing a crowd state in a predetermined size and including images of people whose reference parts are expressed as large as the reference parts of people defined for the predetermined size,

The method includes a crowd state recognition step of extracting, from a given image, an area indicated among the group of rectangular areas stored in a rectangular area group storage device, and recognizing a state of a crowd photographed in the extracted image based on a dictionary.

(Supplementary Note 27)

According to the crowd state recognition method described in Supplementary Note 26,

The crowd state recognition dictionary storage device stores a dictionary of discriminators obtained by machine learning using a plurality of pairs of crowd state images and training labels of the crowd state images, wherein the crowd state images are obtained by synthesizing images of people whose states match those of people controlled to a desired state.

The method includes a crowd state recognition step of recognizing the state of a crowd captured in an image based on a dictionary.

(Supplementary Note 28)

According to the crowd state recognition method described in Supplementary Note 26 or 27,

wherein the rectangular area group storage means stores rectangular areas defined by a set of sizes based on camera parameters indicating a position, posture, focal length, and lens distortion of an image acquisition device used to acquire an image, and sizes of reference parts of a person defined for predetermined sizes,

The method includes a crowd state recognition step of extracting, from a given image, an area indicated in the set of rectangular areas.

(Supplementary Note 29)

According to the crowd state recognition method described in any one of Supplementary Notes 26 to 28,

wherein the crowd state recognition dictionary storage means stores a dictionary of discriminators obtained by varying the number of persons expressed in a crowd state image and by performing machine learning using a plurality of pairs of crowd state images and training labels prepared for the number of persons,

The method includes a crowd state recognition step of recognizing the number of people in a crowd captured in an image based on a dictionary.

(Supplementary Note 30)

According to the crowd state recognition method described in any one of Supplementary Notes 26 to 29,

wherein the crowd state recognition dictionary storage means stores a dictionary of discriminators obtained by changing the directions of people expressed in crowd state images and by performing machine learning using a plurality of pairs of crowd state images and training labels prepared for these directions of people,

The method includes a crowd state recognition step of recognizing the direction of a crowd captured in an image based on a dictionary.

(Supplementary Note 31)

According to the crowd state recognition method described in any one of Supplementary Notes 26 to 30,

wherein the crowd state recognition dictionary storage device stores a dictionary of discriminators obtained by machine learning using a plurality of pairs of crowd state images and training labels prepared for non-significantly crowded crowds and significantly crowded crowds,

The method includes a crowd state recognition method for recognizing whether a crowd captured in an image is significantly crowded based on a dictionary.

(Supplementary Note 32)

According to the crowd state recognition method described in any one of Supplementary Notes 26 to 31,

wherein the crowd state recognition dictionary storage means stores a dictionary of discriminators obtained by performing machine learning using a plurality of pairs of crowd state images and training labels prepared for a crowd in which the directions of people are uniform and a crowd in which the directions of people are not uniform,

The method includes a crowd state recognition method for recognizing whether the directions of people in a crowd captured in an image are unified based on a dictionary.

(Supplementary Note 33)

A training data generating program for causing a computer to execute the following processing:

a background extraction process of selecting a background image from a plurality of pre-prepared background images, extracting a region in the background image, and enlarging or reducing the image corresponding to the extracted region to an image of a predetermined size;

a person state determination process of determining the person state of a group of people based on a multi-person state control designation as designation information on the person states of a plurality of people and an individual person state control designation as designation information on the states of individual people among the plurality of people; and

Crowd state image synthesis processing generates a crowd state image, specifies a training label for the crowd state image, and outputs a pair of the crowd state image and the training label. The crowd state image is an image in which a human image corresponding to the human state determined in the human state determination processing is synthesized with an image of a predetermined size obtained in the background extraction processing.

(Supplementary Note 34)

The training data generation program according to Supplementary Note 33, the program being configured to cause a computer to execute:

Human state determination processing, temporarily determining the human state of a group of people based on the multi-person state control indication and the individual person state control indication, determining the temporarily determined human state as the human state of the group when the temporarily determined human state meets the conditions of the size of the human reference part defined for the predetermined size and how the reference part is expressed, and repeating the temporary determination of the human state of the group when the temporarily determined human state does not meet these conditions.

(Supplementary Note 35)

The training data generation program according to Supplementary Note 33 or 34 is configured to cause a computer including a crowd state control instruction storage device and a person state control instruction storage device to execute:

a human state determining step of determining the human state of the crowd based on the multi-person state control designation stored in the crowd state control designation storage device and the individual human state control designation stored in the human state control designation storage device; and

a crowd state image synthesis step of specifying a training label by reading a multi-person state control designation defined as an item having a specified training label from a crowd state control designation storage device and reading an individual person state control designation defined as an item having a specified training label from a person state control designation storage device;

The crowd state control indication storage device is used to store the multi-person state control indication defined according to the project and store the existence of the specified training label defined for the project, and

The person state control indication storage device is used to store individual person state control indications defined according to a project and to store the existence of a specified training label defined for the project.

(Supplementary Note 36)

The training data generation program according to Supplementary Note 35 is configured to cause a computer including a crowd state control device for storing at least one item with a specified training label to execute:

The crowd state image synthesis process reads a multi-person state control specification defined as an item having a specified training label from a crowd state control specification storage device.

(Supplementary Note 37)

The training data generation program according to Supplementary Note 35 or 36, the program being configured to cause a computer including a crowd state control instruction storage device and a person state control instruction storage device to execute:

The human state determination processing determines the human state of the crowd based on the multi-person state control indication stored in the crowd state control indication storage device and the individual person state control indication stored in the human state control indication storage device,

wherein the crowd state control designation storage device is used to store the multi-person state control designation and the existence of the designated training label according to items such as the arrangement of people, the direction of people, and the number of people, and stores the multi-person state control designation corresponding to each item in any one of a first form indicating a specific state, a second form indicating that an arbitrary state can be defined, and a third form indicating that a state can be defined within a predetermined rule, and

The person status control indication storage device is used to store the existence of individual person status control indications and indicated training labels according to items such as the person's shooting angle, the person's lighting, the person's posture, the person's clothes, the person's body shape, the person's hairstyle and the person's size when synthesized with the crowd status image, and stores the individual person status control indication corresponding to each item in any one of the first form, the second form and the third form.

(Supplementary Note 38)

The training data generation program according to any one of Supplementary Notes 33 to 37, the program being configured to cause a computer to execute:

Crowd state image synthesis processing, selecting a human image that matches a determined human state such as the direction of the human, the number of people, the shooting angle of the human, the lighting of the human, the posture of the human, the clothes of the human, the body shape of the human and the hairstyle of the human from a group of pre-prepared human images, cropping the human area from the selected human image, thereby generating an image of only the human and synthesizing the image of only the human with an image of a predetermined size obtained by a background extraction device based on the arrangement of the human when determined to be in a human state and the size of the human when synthesized with the crowd state image.

(Supplementary Note 39)

The training data generation program according to Supplementary Note 38, the program being configured to cause a computer to execute:

A crowd state image synthesis process is used to sequentially synthesize an image of only people corresponding to the farthest arrangement position from the camera with an image of a predetermined size acquired by a background extraction device.

(Supplementary Note 40)

The training data generation program according to any one of Supplementary Notes 33 to 39, the program being configured to cause a computer to execute, in a human state determination process:

Background person state determination processing of temporarily determining a person state of a crowd serving as a background in a crowd state image based on a multi-person state control designation and an individual person state control designation, determining the temporarily determined person state as the person state of the crowd serving as the background when the temporarily determined person state satisfies a first condition of how a reference part of a person is expressed and a size defined for a predetermined size of the crowd state image, and repeatedly performing the temporary determination of the person state of the crowd serving as the background when the temporarily determined person state does not satisfy the first condition;

The foreground person state determination processing temporarily determines the person state of the crowd as the foreground in the crowd state image based on the multi-person state control indication and the individual person state control indication, determines the temporarily determined person state as the person state of the crowd as the foreground when the temporarily determined person state meets the second condition of the size of the person's reference part defined for the predetermined size of the crowd state image and how the reference part is expressed, and repeats the temporary determination of the person state of the crowd as the foreground when the temporarily determined person state does not meet the second condition.

(Supplementary Note 41)

According to the training data generation procedure described in Supplementary Note 40,

The first condition is that the reference part of the person is not within the crowd state image or the size of the reference part is much larger or smaller than the size of the reference part of the person defined for the predetermined size, and

The second condition is that the reference part of the person is within the crowd state image and the size of the reference part is as large as the size of the reference part of the person defined for a predetermined size.

(Supplementary Note 42)

A crowd state recognition program is configured to cause a computer including a rectangular region group storage device and a crowd state recognition dictionary storage device to execute:

a crowd state recognition process of extracting, from a given image, an area indicated in the group of rectangular areas stored in the rectangular area group storage means and recognizing the state of a crowd photographed in the extracted image based on a dictionary,

The rectangular region group storage device is used to store a group of rectangular regions indicating the portion of the image to be identified for the crowd state, and

The crowd state recognition dictionary storage device is used to store a dictionary of a discriminator obtained by machine learning using multiple pairs of crowd state images and training labels of the crowd state images, wherein the crowd state image is an image of a person whose reference part is expressed as the same size as the reference part of a person defined for the predetermined size and represents a crowd state in a predetermined size.

(Supplementary Note 43)

According to the crowd state recognition program described in Supplementary Note 42, the program is configured to cause a computer including a crowd state recognition dictionary storage device to execute:

Crowd state recognition processing, based on the dictionary to identify the state of the crowd captured in the image,

The crowd state recognition dictionary storage device is used to store a dictionary of discriminators obtained by machine learning using multiple pairs of crowd state images and training labels of crowd state images, and the crowd state images are obtained by synthesizing human images that match the human state controlled to the desired state.

(Supplementary Note 44)

The crowd state recognition program according to Supplementary Note 42 or 43 is configured to cause a computer including a rectangular region group storage device to execute:

Crowd state recognition processing, extracting the area indicated in the set of rectangular areas from a given image,

The rectangular area group storage device is used to store rectangular areas defined by a set of sizes based on camera parameters indicating the position, posture, focal length and lens distortion of an image acquisition device used to acquire images, and the sizes of reference parts of a person defined for predetermined sizes.

(Supplementary Note 45)

A crowd state recognition program according to any one of Supplementary Notes 42 to 44, the program being configured to cause a computer including a crowd state recognition dictionary storage device to execute:

Crowd state recognition processing of recognizing the number of people in a crowd captured in an image based on a dictionary,

Among them, the group state recognition dictionary storage device is used to store a dictionary of discriminators obtained by changing the number of people expressed in the crowd state image and by performing machine learning using multiple pairs of crowd state images and training labels prepared for the number of people.

(Supplementary Note 46)

The crowd state recognition program according to any one of Supplementary Notes 42 to 45, the program being configured to cause a computer including a crowd state recognition dictionary storage device to execute:

Crowd state recognition processing, based on the dictionary to identify the direction of the crowd captured in the image,

The crowd state recognition dictionary storage device is used to store a dictionary of discriminators obtained by changing the directions of people expressed in crowd state images and by performing machine learning using multiple pairs of crowd state images and training labels prepared for these directions of people.

(Supplementary Note 47)

The crowd state recognition program according to any one of Supplementary Notes 42 to 46, the program being configured to cause a computer including a crowd state recognition dictionary storage device to execute:

Crowd state recognition processing, based on the dictionary to identify whether the crowd captured in the image is significantly crowded,

The crowd state recognition dictionary storage device stores a dictionary of discriminators obtained by performing machine learning using a plurality of pairs of crowd state images and training labels prepared for non-significantly crowded crowds and significantly crowded crowds.

(Supplementary Note 48)

The crowd state recognition program according to any one of Supplementary Notes 42 to 47, the program being configured to cause a computer including a crowd state recognition dictionary storage device to execute:

Crowd state recognition processing, based on the dictionary to identify whether the orientation of people in the crowd captured in the image is unified,

The crowd state recognition dictionary storage device stores a dictionary of discriminators obtained by machine learning using a plurality of pairs of crowd state images and training labels prepared for a crowd in which the directions of people are unified and a crowd in which the directions of people are not unified.

The present invention has been described with reference to the exemplary embodiments, but the present invention is not limited to the above exemplary embodiments. The structure and details of the present invention can be variously changed within the scope of the present invention that can be understood by those skilled in the art.

This application claims the benefit of priority based on Japanese Patent Application No. 2013-135915, filed on Jun. 28, 2013, the disclosure of which is incorporated herein in its entirety by reference.

Industrial Applicability

The present invention can be suitably applied to a training data generating device for generating training data when learning a dictionary of a discriminator for recognizing a crowd state.

The present invention is suitably applied to a crowd state recognition device for identifying the state of a crowd in an image. Specifically, the present invention can be suitably applied to identifying the state of a crowd in an image with a low frame rate. In addition, the present invention can also be suitably utilized when the frame rate is unstable and the crowd state recognition processing using time information cannot be performed. In addition, the present invention can be suitably used to identify a complex crowd state including overlaps between people based on a still image. In addition, the present invention can be used for suspicious person recognition, left suspicious object recognition, rear-end recognition, abnormal state recognition, abnormal behavior recognition, etc. in the field of surveillance to identify the state of a crowd from an image acquired by a camera. In addition, the present invention can be used to output the recognition result of the state of a crowd in an image together with the position (2D position or 3D position) of the crowd to other systems. In addition, the present invention can be used to obtain the recognition result of the state of a crowd in an image and the position (2D position or 3D position) of the crowd and use the obtained object as a trigger to perform a video search.

Label list

11 Background extraction device

12 Background person status determination device

13. Foreground Person Status Determination Device

14 Crowd state image synthesis device

15. Person status determination device

16 Control Device

21 Background image storage device

22 Learning local image information storage device

23 Crowd status control indication storage device

24-person status control indicating storage device

25-person image storage device

26-person area image storage device

41 Crowd Status Recognition Device

51 Search Window Storage

52 Crowd state recognition dictionary storage device

Claims (7)

1. A training data generation device, comprising: The background extraction unit is used to select a background image from multiple pre-prepared background images, extract a region from the background image, and enlarge or reduce the image corresponding to the extracted region to a predetermined size. A person state determination unit is used to determine the person state of a crowd based on a multi-person state control specification as specifying information about the person state of multiple people and an individual state control specification as specifying information about the state of an individual among the multiple people; and A crowd state image synthesis unit is configured to generate a crowd state image, specify training labels for the crowd state image, and output a pair of crowd state images and training labels. The crowd state image is an image synthesized by combining a person image corresponding to the person state determined by the person state determination unit with an image of a predetermined size obtained by the background extraction unit. The human state determination unit includes: A background person state determination unit is configured to temporarily determine the person state of a crowd serving as the background in a crowd state image based on the multi-person state control specification and the individual person state control specification; when the temporarily determined person state satisfies a first condition regarding the size of a reference part of a person defined for the predetermined size of the crowd state image and how the reference part is represented, the temporarily determined person state is determined as the person state of the crowd serving as the background; and when the temporarily determined person state does not satisfy the first condition, the temporary determination of the person state of the crowd serving as the background is repeated. A foreground person state determination unit is configured to temporarily determine the person state of a crowd as the foreground in a crowd state image based on the multi-person state control specification and the individual person state control specification. When the temporarily determined person state satisfies a second condition—the size of a reference portion defined for a predetermined size of the crowd state image and how the reference portion is expressed—the temporarily determined person state is confirmed as the person state of the crowd as the foreground. When the temporarily determined person state does not satisfy the second condition, the temporary determination of the person state of the crowd as the foreground is repeated. The first condition is that the reference part of the person is not within the crowd state image, or the size of the reference part is much larger or much smaller than the size of the reference part of the person defined for the predetermined size. The second condition is that the reference part of the person is within the crowd state image, and the size of the reference part is the same as the size of the reference part of the person defined for the predetermined size. The training data generation device includes: A crowd state control specification storage unit is provided for storing the crowd state control specification as defined in the project definition, and storing the presence of training labels for the specification defined in the project definition; and The human state control specification storage unit is used to store individual human state control specifications as defined by the project, as well as the presence of training labels for the specifications defined by the project. The person status determination unit determines the person status of the crowd based on the multi-person status control indication stored in the crowd status control indication storage unit and the individual person status control indication stored in the person status control indication storage unit. The crowd state image synthesis unit specifies training labels by reading the multi-person state control specifications defined as items with specified training labels from the crowd state control specification storage unit, and the individual state control specifications defined as items with specified training labels from the individual state control specification storage unit. The crowd state control specifies that the storage unit stores the presence of the multi-person state control specification and the specified training label according to items related to the arrangement, orientation, and number of people, and stores the multi-person state control specification corresponding to each item in any of the following forms: a first form specifying a specific state, a second form specifying that any state can be defined, and a third form specifying that a state can be defined within predetermined rules. The person state control specifies that the storage unit stores the presence of the individual state specification and the specified training label according to items related to the person's shooting angle, lighting, posture, clothing, body shape, hairstyle, and size when composited with a crowd state image. Furthermore, it stores the individual state control specification corresponding to each item in any of the first, second, and third forms. The human status determination unit determines the human status of the crowd based on the multi-person status control indication stored in the crowd status control indication storage unit and the individual status control indication stored in the human status control indication storage unit. 2. The training data generation device according to claim 1, The person state determination unit temporarily determines the person state of the crowd based on the multi-person state control specification and the individual person state control specification. When the temporarily determined person state satisfies the conditions for the size of the reference part of the person defined for the predetermined size and how the reference part is expressed, the temporarily determined person state is determined as the person state of the crowd. When the temporarily determined person state does not satisfy the conditions, the temporary determination of the person state of the crowd is repeated. 3. The training data generation device according to claim 1, The crowd state control specifies that the storage unit stores at least one item with a specified training label, and The crowd state image synthesis unit reads the multi-person state control specification defined as having specified training labels from the crowd state control specification storage unit. 4. The training data generation device according to claim 1, The crowd state image synthesis unit selects a set of pre-prepared human images that match the determined human state, which is determined by factors such as the direction of the human, the number of human, the shooting angle of the human, the lighting of the human, the posture of the human, the clothing of the human, the body shape of the human, and the hairstyle of the human. It then crops the human region from the selected human images to generate an image containing only the human. Based on the arrangement of the human in the determined human state and the human size when synthesized with the crowd state image, it synthesizes the image containing only the human with the image of a predetermined size obtained by the background extraction unit. 5. The training data generation device according to claim 4, The crowd state image synthesis unit sequentially synthesizes images containing only people from the furthest placement position from the camera with images of a predetermined size obtained by the background extraction unit. 6. A method for generating training data, comprising: The background extraction step involves selecting a background image from multiple pre-prepared background images, extracting a region from the background image, and enlarging or shrinking the image corresponding to the extracted region to a predetermined size. The person status determination step involves determining the person status of a group based on a multi-person status control specification, which serves as specification information regarding the person status of multiple people, and an individual status control specification, which serves as specification information regarding the status of individual individuals within the multiple people; and The crowd state image synthesis step includes generating a crowd state image, specifying training labels for the crowd state image, and outputting a pair of crowd state images and training labels. The crowd state image is an image synthesized from a person image corresponding to the person state determined in the person state determination step and an image of a predetermined size obtained in the background extraction step. The steps for determining the person's status include: The background person state determination step involves temporarily determining the person state of the crowd as background in the crowd state image based on the multi-person state control specification and the individual person state control specification. If the temporarily determined person state satisfies a first condition regarding the size of a reference part of a person defined for the predetermined size of the crowd state image and how the reference part is represented, the temporarily determined person state is determined as the person state of the crowd as background. If the temporarily determined person state does not satisfy the first condition, the temporary determination of the person state of the crowd as background is repeated. The foreground person state determination step involves temporarily determining the person states of the crowd as foreground in the crowd state image based on the multi-person state control specification and the individual person state control specification. If the temporarily determined person state satisfies a second condition regarding the size of the reference portion defined for the predetermined size of the crowd state image and how the reference portion is represented, the temporarily determined person state is confirmed as the person state of the crowd as foreground. If the temporarily determined person state does not satisfy the second condition, the temporary determination of the person state of the crowd as foreground is repeated. The first condition is that the reference part of the person is not within the crowd state image, or the size of the reference part is much larger or much smaller than the size of the reference part of the person defined for the predetermined size. The second condition is that the reference part of the person is within the crowd state image, and the size of the reference part is the same as the size of the reference part of the person defined for the predetermined size. in The crowd state control specifies that the storage unit stores the presence of training labels specified in the project definition, in accordance with the crowd state control specification. The human state control specifies that the storage unit stores the individual human state control specified according to the project definition, and stores the existence of the specified training labels for the project definition. The steps for determining the person's status include: The person status of the crowd is determined based on the multi-person status control indication stored in the crowd status control indication storage unit and the individual person status control indication stored in the person status control indication storage unit. The crowd state image synthesis step includes: Training labels are specified by reading the multi-person status control specification defined as an item with a specified training label from the crowd status control specification storage unit, and the individual status control specification defined as an item with a specified training label from the person status control specification storage unit. The crowd state control specifies that the storage unit stores the presence of the multi-person state control specification and the specified training label according to items related to the arrangement, orientation, and number of people, and stores the multi-person state control specification corresponding to each item in any of the following forms: a first form specifying a specific state, a second form specifying that any state can be defined, and a third form specifying that a state can be defined within predetermined rules. The person state control specifies that the storage unit stores the presence of the individual state specification and the specified training label according to items related to the person's shooting angle, lighting, posture, clothing, body shape, hairstyle, and size when composited with a crowd state image. Furthermore, it stores the individual state control specification corresponding to each item in any of the first, second, and third forms. The steps for determining the person's status include: The person status of the crowd is determined based on the multi-person status control specification stored in the crowd status control specification storage unit and the individual person status control specification stored in the person status control specification storage unit. 7. A computer-readable recording medium wherein a training data generation program is recorded, the training data generation program causing a computer to perform the following processes: Background extraction processing involves selecting a background image from multiple pre-prepared background images, extracting a region from the background image, and enlarging or shrinking the image corresponding to the extracted region to a predetermined size. The person status determination process determines the person status of a group based on a multi-person status control specification, which serves as specification information about the person status of multiple people, and an individual status control specification, which serves as specification information about the status of individual individuals within the multiple people; and The crowd state image synthesis process generates a crowd state image, specifies training labels for the crowd state image, and outputs a pair of crowd state images and training labels. The crowd state image is an image synthesized from a person image corresponding to the person state determined in the person state determination process and an image of a predetermined size obtained in the background extraction process. The training data generation procedure causes the computer to execute the human state determination process: Background person state determination processing involves temporarily determining the person states of the crowd as background in a crowd state image based on the multi-person state control specification and the individual person state control specification. When the temporarily determined person state satisfies a first condition regarding the size of a reference part of a person defined for the predetermined size of the crowd state image and how the reference part is represented, the temporarily determined person state is determined as the person state of the crowd as background. If the temporarily determined person state does not satisfy the first condition, the temporary determination of the person state of the crowd as background is repeated. The foreground person state determination process temporarily determines the person state of the crowd as the foreground in the crowd state image based on the multi-person state control specification and the individual person state control specification. If the temporarily determined person state satisfies the second condition regarding the size of the reference portion defined for the predetermined size of the crowd state image and how the reference portion is represented, the temporarily determined person state is confirmed as the person state of the crowd as the foreground. If the temporarily determined person state does not satisfy the second condition, the temporary determination of the person state of the crowd as the foreground is repeated. The first condition is that the reference part of the person is not within the crowd state image, or the size of the reference part is much larger or much smaller than the size of the reference part of the person defined for the predetermined size. The second condition is that the reference part of the person is within the crowd state image, and the size of the reference part is the same as the size of the reference part of the person defined for the predetermined size. The computer mentioned above includes: A crowd state control specification storage unit is provided for storing the crowd state control specification as defined in the project definition, and storing the presence of training labels for the specification defined in the project definition; and The human state control specification storage unit is used to store individual human state control specifications as defined by the project, as well as the presence of training labels for the specifications defined by the project. The training data generation program causes the computer to execute: In the person status determination process, the person status of the crowd is determined based on the multi-person status control specifications stored in the crowd status control specification storage unit and the individual person status control specifications stored in the person status control specification storage unit. In the crowd state image synthesis process, training labels are specified by reading the multi-person state control specifications defined as items with specified training labels from the crowd state control specification storage unit, and the individual state control specifications defined as items with specified training labels from the individual state control specification storage unit. The crowd state control specifies that the storage unit stores the presence of the multi-person state control specification and the specified training label according to items related to the arrangement, orientation, and number of people, and stores the multi-person state control specification corresponding to each item in any of the following forms: a first form specifying a specific state, a second form specifying that any state can be defined, and a third form specifying that a state can be defined within predetermined rules. The person state control specifies that the storage unit stores the presence of the individual state specification and the specified training label according to items related to the person's shooting angle, lighting, posture, clothing, body shape, hairstyle, and size when composited with a crowd state image. Furthermore, it stores the individual state control specification corresponding to each item in any of the first, second, and third forms. The training data generation procedure causes the computer to execute, in the person state determination process, to determine the person state of the crowd based on the multi-person state control specification stored in the crowd state control specification storage unit and the individual person state control specification stored in the person state control specification storage unit.