JP2017162438A - Risk prediction method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a danger prediction method capable of predicting a dangerous area where a travelling vehicle is likely to be exposed to danger.SOLUTION: The danger prediction method executed by a computer of a danger predictor using convolutional neural networks includes: an acquisition step (S1) for causing the neural networks to acquire an input image captured by an on-vehicle camera mounted on a vehicle; and an output step (S2) for causing the convolutional neural networks to estimate a dangerous area in the input image acquired in the acquisition step, where a collision may occur between the vehicle and a moving object appearing on a travel route if the vehicle keeps travelling, and characteristics of the dangerous area and then to output them as a degree of danger predicted for the input image.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a risk prediction method.

  For example, Patent Literature 1 discloses a driving support technology device for supporting safety confirmation of a driver who drives a vehicle. According to this driving support technology, for example, when the signal changes from blue to red at an intersection, there is a possibility of suddenly changing the speed against the driver's expectation, such as a pedestrian who suddenly runs, leading to an accident. Highly dangerous moving objects can be accurately detected at an early timing.

Japanese Patent No. 4967015

Jonathan Long, Evan Shelhamer, Trevor Darrell, “Fully Convolutional Networks for Semantic Segmentation”; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3431-3440

  However, the driving support technology apparatus in Patent Document 1 has a problem that it is merely predicting a dangerous state by observing a speed change of a moving object that can be actually visually recognized. In other words, in the driving support technology device in Patent Document 1 described above, an area where there is a danger of collision with a vehicle when a pedestrian who gets off the bus pops out, such as an area behind a stopped bus (danger There is a problem that the (region) cannot be detected.

  The present disclosure has been made in view of the above-described circumstances, and an object thereof is to provide a risk prediction method capable of predicting a dangerous area in which danger is likely to occur for a traveling vehicle.

  In order to solve the above problem, a risk prediction method according to an embodiment of the present disclosure is a risk prediction method performed by a computer of a risk predictor using a convolutional neural network, and is captured by an in-vehicle camera mounted on a vehicle. An acquisition step of causing the convolutional neural network to acquire the input image, and a travel of the vehicle when the vehicle travels as it is in the dangerous region in the input image acquired in the acquisition step of the convolutional neural network. An output step of estimating a dangerous area where a moving object may appear and collide with the vehicle and a feature of the dangerous area and outputting the predicted risk level to the input image. .

  These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM. The system, method, integrated circuit, computer You may implement | achieve with arbitrary combinations of a program and a recording medium.

  According to the present disclosure, it is possible to realize a danger prediction method capable of predicting a dangerous area in which danger is likely to occur for a traveling vehicle.

FIG. 1 is a block diagram illustrating an example of a configuration of a risk predictor according to the first embodiment. FIG. 2 is a diagram showing an outline of the structure of a convolutional neural network used by the risk predictor shown in FIG. FIG. 3 is a diagram illustrating an example of a prediction result of the risk predictor according to the first embodiment. FIG. 4 is a diagram illustrating another example of the prediction result of the risk predictor according to the first embodiment. FIG. 5 is a flowchart illustrating an example of the prediction process of the risk predictor according to the first embodiment. FIG. 6 is a flowchart showing an outline of learning processing of the convolutional neural network in the first embodiment. FIG. 7A is an explanatory diagram of the learning data prepared in step S1. FIG. 7B is an explanatory diagram of the learning data prepared in step S1. FIG. 7C is an explanatory diagram of the learning data prepared in step S1. FIG. 7D is an explanatory diagram of the learning data prepared in step S1. FIG. 8 is a diagram illustrating an example of a dangerous area output as a result of the learning process and its characteristics. FIG. 9 is a block diagram illustrating an example of a configuration of the learning system in the embodiment. FIG. 10 is a flowchart showing a process for creating annotated data performed by the learning data creation apparatus. FIG. 11 is a flowchart illustrating a learning process performed by the learning device. FIG. 12A is another example of the annotated image prepared in step S1. FIG. 12B is a diagram illustrating another example of the dangerous area output as a result of the learning process and its characteristics. FIG. 13 is a diagram illustrating an example of detailed area information. FIG. 14A is an example of a learning image. FIG. 14B is an example of an annotated image with an annotation indicating a dangerous area. FIG. 15A is an example of a learning image. FIG. 15B is an example of an annotated image with an annotation indicating a dangerous area. FIG. 16A is an example of a learning image. FIG. 16B is an example of an annotated image with an annotation indicating a dangerous area. FIG. 17A is an example of a learning image. FIG. 17B is an example of an annotated image with an annotation indicating a dangerous area. FIG. 18 is a flowchart showing an outline of a two-stage learning process in the second embodiment. FIG. 19 is an explanatory diagram for conceptually explaining the first-stage learning process. FIG. 20 is an explanatory diagram for conceptually explaining the learning process at the second stage. FIG. 21 is another explanatory diagram for conceptually explaining the first-stage learning process.

(Background to obtaining one embodiment of the present invention)
In recent years, the performance of image recognition has improved dramatically by using Deep Learning. Deep learning is known as a machine learning methodology using a multilayer neural network, and a convolutional neural network (CNN) is often used for such a multilayer neural network. Here, the convolutional neural network is composed of a multilayer neural network that repeats convolution and pooling of a local region.

  Here, the convolutional neural network is based on a convolution layer that extracts features by convolution processing using multiple filters, a pooling layer that acquires local data invariance by pooling processing that collects reactions in a certain region, and a Softmax function. And a fully connected layer that performs recognition using probability.

  However, the image recognition process using such a convolutional neural network has a problem that it cannot be executed in real time.

  On the other hand, for example, Non-Patent Document 1 proposes a structure of a convolutional neural network in which all concatenated layers constituting the convolutional neural network are convolutional layers. By using a convolutional neural network having this structure (complete convolutional neural network), it is possible to execute image recognition processing in real time.

  Accordingly, the inventors (e.g.) have come up with the idea of solving the above-described problem using a complete convolution neural network.

  That is, the risk prediction method according to an aspect of the present disclosure is a risk prediction method performed by a computer of a risk predictor using a convolutional neural network, and an input image captured by an in-vehicle camera mounted on a vehicle is An acquisition step that causes the convolutional neural network to acquire, and a moving object that is a dangerous region in the input image acquired in the acquisition step and that the vehicle travels as it is in the convolutional neural network when the vehicle travels as it is. An output step of estimating a dangerous area that may appear and collide with the vehicle and a characteristic of the dangerous area and outputting the estimated risk level for the input image.

  As a result, since the degree of danger for the acquired input image can be estimated, it is possible to realize a danger prediction method capable of predicting a dangerous area in which danger is likely to occur for a running vehicle.

  Here, for example, in the output step, the convolutional neural network is caused to estimate the risk level of the risk area as a characteristic of the risk area, and the likelihood map indicating the estimated risk area and the risk level of the risk area May be output as the degree of risk.

  Further, for example, in the output step, the convolutional neural network is caused to estimate the type of the moving object associated with the dangerous area as the characteristic of the dangerous area, and the estimated dangerous area and the type are used as the risk level. May be output as

  Further, for example, the risk prediction method may further include a learning image including the dangerous area and an annotated image in which an annotation indicating the dangerous area is attached to the learning image before performing the obtaining step. A learning step may be included in which the convolutional neural network learns the weight of the convolutional neural network for estimating the dangerous area and the feature of the dangerous area in the learning image.

  Further, for example, the learning step includes a dangerous area image of a region that is a partial region of each of the learning images and an annotation that indicates the dangerous region, and a safety region that is not annotated. A first weight of the first neural network for causing the first neural network, which is a convolutional neural network having a fully connected layer, to determine that the partial region is a safe region or a dangerous region using an image And the initial value of the weight of the second neural network having a configuration in which all the connection layers of the first neural network are changed to convolutional layers, the first learning step learned in the first learning step. An image for learning that includes the dangerous area, updated to one weight, and the dangerous area in the learning image The second neural network for causing the second neural network to estimate the dangerous area in the learning image and the feature of the dangerous area using the annotated image with the annotation shown. A second learning step of learning the weight of the convolutional neural network having the same configuration as the second neural network by learning the second weight may be included.

  In addition, for example, the risk prediction method further includes acquiring a plurality of time-sequential images taken by an in-vehicle camera mounted on a vehicle, and is included in at least some of the acquired images. A dangerous area, which is a dangerous area where a moving object may appear in the traveling route of the vehicle and collide with the vehicle when the vehicle travels as it is, and the at least part of the image is determined. An assigning step for giving an annotation indicating the determined dangerous area, and in the learning step, the at least a part of the images to which the annotation is given in the giving step, and the plurality of images among the plurality of images An image corresponding to at least a part of the image is acquired as the learning image and the annotated image, and the convolution new It may train the said weights Le networks.

  Here, for example, the dangerous area is an area including a part of the area of the shielding object existing in the learning image, the moving object is hidden, and the moving object is moved from the shielding object to the travel route. This is the area before it appears inside.

  In addition, for example, the danger area may be an area between the two or more moving objects that will cross the travel route of the vehicle when two or more moving objects including a person approach each other.

  Further, for example, the annotation may indicate the danger area and the category of the moving object associated with the danger area.

  Further, for example, the annotation may indicate the danger area and control information including a brake strength or a steering wheel angle of the vehicle at the time of capturing the learning image.

  For example, the annotation may be segment information of the dangerous area.

  Each of the embodiments described below shows a specific example of the present disclosure. Numerical values, shapes, components, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements. In all the embodiments, the contents can be combined.

(Embodiment 1)
Hereinafter, the risk prediction method of the risk predictor 10 according to the first embodiment will be described with reference to the drawings.

[Configuration of danger predictor 10]
FIG. 1 is a block diagram showing an example of the configuration of the danger predictor 10 in the present embodiment. FIG. 2 is a diagram showing an outline of the structure of a convolutional neural network used by the risk meter 10 shown in FIG. FIG. 3 is a diagram illustrating an example of a prediction result of the risk predictor 10 in the present embodiment.

  A risk predictor 10 shown in FIG. 1 is a risk predictor using a convolutional neural network, and is realized by a computer or the like. The convolutional neural network used by the risk predictor 10 is a complete convolutional neural network in which, for example, as shown in FIG.

  Here, the convolutional neural network is often used in the field of image recognition, and extracts a feature amount from an image by performing convolution with a filter on a two-dimensional image. As described above, the convolutional neural network is composed of a multilayer network that repeats convolution and pooling. Then, a filter coefficient (weight) effective for identification constituting the convolution layer in the convolutional neural network is learned using a large amount of learning data such as a large amount of learning images. The coefficient (weight) is obtained by performing learning to acquire invariance with respect to various deformations by repeating convolution by a filter and pooling that collects reactions in a certain region using a large amount of data. It has been found that the discrimination performance of the convolutional neural network depends on the filters constituting the convolutional layer.

  When the input image captured by the in-vehicle camera is input, the risk predictor 10 illustrated in FIG. 1 uses a convolutional neural network as illustrated in FIG. Estimate features and output as predicted risk. The danger predictor 10 predicts a higher degree of danger as it is closer to a partial area of an image immediately before an image that has become dangerous due to the occurrence of a person jumping out in the past, that is, an area that has led to danger in the past.

  More specifically, the risk predictor 10 shown in FIG. 1 acquires an input image taken by an in-vehicle camera mounted on the vehicle. The danger predictor 10 is a dangerous area in the acquired input image, and when the vehicle travels as it is, there is a danger area where a moving object may appear in the traveling route of the vehicle and may collide with the vehicle. The characteristics of the dangerous area are estimated by a convolutional neural network. Then, the danger predictor 10 outputs the degree of danger predicted for the acquired input image. Here, the risk predictor 10 estimates the risk level of the risk area as a characteristic of the risk area, and outputs the estimated risk area and a likelihood map indicating the risk level of the risk area as the risk level.

  In the present embodiment, the risk predictor 10 acquires, as an input image, for example, an image 50 as shown in FIG. 3, that is, an image 50 including a stopped bus 501 and pedestrians 502 and 503. Suppose that In this case, the danger predictor 10 estimates in the image 50 that the area behind the stopped bus 501, the area of the person 502, and the area of the person 503 are dangerous areas. Then, an image 50a in which the estimated risk degree (likelihood) of the dangerous region is superimposed on the image 50 is output as the predicted risk degree. In the image 50a, as the likelihood map, the likelihood 504 in the area behind the stopped bus 501, the likelihood 505 in the area of the person 502, and the likelihood 506 in the area of the person 503 are superimposed. . This likelihood indicates the risk (risk level) that the vehicle may collide with. In the present embodiment, this likelihood indicates a high risk (risk) as an area that is not visible to humans and is an area that leads to danger even for an area that is likely to jump out.

  Note that the risk level output by the risk predictor 10 is not limited to the likelihood map indicating the risk area and the risk level as shown in FIG. Hereinafter, this example will be described with reference to FIG.

  FIG. 4 is a diagram illustrating another example of the prediction result of the risk predictor 10 according to the present embodiment. Elements similar to those in FIG. 3 are denoted by the same reference numerals, and detailed description thereof is omitted.

  The danger predictor 10 estimates that the area 507 behind the stopped bus 501, the area 508 of the person 502, and the area 509 of the person 503 are dangerous areas for the image 50 acquired as the input image. Then, an image 50b in which the estimated dangerous area and its category are superimposed on the image 50 is output. In other words, the danger predictor 10 has an area 507 and its category dangerous area (car), an area 508 and its category dangerous area (person), and an area 509 and its category dangerous area (person). ) Is output. Here, in the image 50b, since the type of the moving object associated with the region 507, which is a dangerous region, is a car, a dangerous region (vehicle) is superimposed as a category of the dangerous region. Similarly, in the image 50b, since the type of the moving object associated with the areas 508 and 509 which are the dangerous areas is a person (person), the dangerous area (person) is superimposed as the category of the dangerous area. .

  Thus, the danger predictor 10 may estimate the type of the moving object associated with the dangerous area as the characteristic of the dangerous area, and output the estimated dangerous area and the type as the degree of danger. .

[Prediction process of danger predictor 10]
Next, the prediction process of the risk predictor 10 according to the present embodiment will be described with reference to the drawings.

  FIG. 5 is a flowchart showing an example of the prediction process of the risk predictor 10 in the present embodiment.

  As shown in FIG. 5, the computer of the risk predictor 10 first causes the convolutional neural network to acquire an input image taken by an in-vehicle camera mounted on the vehicle (S1). In the present embodiment, the computer of the risk predictor 10 causes the convolutional neural network to acquire, for example, the above-described image 50 shown in FIG. 5 as an input image.

  Next, the computer of the risk predictor 10 causes the convolutional neural network to estimate the risk region and its features and output the predicted risk level for the input image (S2). More specifically, the computer of the risk predictor 10 causes the convolutional neural network to show a moving object in the travel route of the vehicle when the vehicle travels as it is in the danger region in the input image acquired in S1. Thus, the danger area that may collide with the vehicle and the characteristics of the danger area are estimated. Then, it is output as a predicted risk level for the input image.

  In the present embodiment, the computer of the risk predictor 10 causes the convolutional neural network to estimate the risk level of the risk area as a characteristic of the risk area, and indicates the estimated risk area and the risk level of the risk area. Output likelihood map as risk. For example, the computer of the risk predictor 10 causes the convolutional neural network to output, for example, the image 50a shown in FIG.

  As described above, the computer of the danger predictor 10 causes the convolutional neural network to estimate the type of the moving object associated with the dangerous area dangerous area as the characteristic of the dangerous area, and the estimated dangerous area. The type of the dangerous area may be output as the degree of danger.

[Effects of danger predictor 10]
As described above, according to the risk predictor 10 according to the first embodiment, it is possible to estimate the risk area in the input image captured by the in-vehicle camera mounted on the vehicle and the characteristics of the risk area. In particular, the risk predictor 10 according to the first embodiment generates a danger for a traveling vehicle even in a region where a pedestrian getting off the bus is not visible, such as a region behind a stopped bus, but is likely to jump out. It can be predicted as a likely danger area.

  Thereby, for example, when a vehicle in automatic driving includes the danger predictor 10, the vehicle predicts a dangerous area where danger is likely to occur using an image of the in-vehicle camera, and avoids the predicted dangerous area. Can be performed more safely.

[Learning process of danger predictor 10]
Hereinafter, a learning process for realizing such a risk predictor 10 will be described. A convolutional neural network that functions as a convolutional neural network used for the risk predictor 10 by performing the learning process will be described as a convolutional neural network 10a.

  FIG. 6 is a flowchart showing an outline of the learning process of the convolutional neural network 10a in the present embodiment. 7A to 7D are explanatory diagrams of learning data prepared in step S1. FIG. 8 is a diagram illustrating an example of a dangerous area output as a result of the learning process and its characteristics.

  First, learning data is prepared (S11). More specifically, learning data including a learning image including a dangerous area and an annotated image in which an annotation indicating the dangerous area is added to the learning image is prepared.

  Here, the learning data will be described with reference to FIGS. 7A to 7D.

  FIG. 7A and FIG. 7B are examples of learning images, and are examples of images that have led to danger among a plurality of images captured by the in-vehicle camera in the past. The image 52 shown in FIG. 7B shows a bus 511 that is stopped, a person 512 that is a pedestrian, and a person 513 that is a pedestrian that appears from the area behind the bus 511. The image 51 shown in FIG. 7A shows a bus 511 that is stopped and a person 512 that is a pedestrian, and the person 513 is present in the area behind the bus 511 but is not visible.

  An image 51a shown in FIG. 7C is an example of an annotated image, and an annotation is given to an area of the image 51 that is a predetermined time before the person appears. More specifically, when the vehicle equipped with the in-vehicle camera that captured the image 51 travels as it is, the image 51a may cause a moving object that is the person 513 to appear in the travel route of the vehicle and collide with the vehicle. Annotation indicating a certain dangerous area is given.

  It should be noted that the annotation indicating the dangerous area is attached to the area 514 behind the bus 511 including the person 513 that is not visible but likely to jump out, as shown in the image 51a in FIG. 7C, that is, the area 514 immediately before the jumping occurs. Not exclusively. For example, as shown in FIG. 7D, in addition to the area 514, an annotation indicating a dangerous area for the area 515 of the visible person 512 is given, and a safety area is shown for other areas (for example, the area 516). An annotation may be given.

  Hereinafter, the description will be returned to FIG.

  Next, the computer performs learning processing using the learning data in order to cause the convolutional neural network 10a to function as the risk predictor 10 (S12). More specifically, using the learning image including the dangerous area and the annotated image in which the learning image is annotated with the annotation indicating the dangerous area, the convolutional neural network 10a is used to transmit the dangerous area in the learning image. And a learning process for learning the weight of the convolutional neural network 10a for estimating the characteristics of the dangerous area.

  For example, the learning process is performed on the convolutional neural network 10a using the image 51 illustrated in FIG. 7A as a learning image including a dangerous area and the image 51b illustrated in FIG. 7D as an image with an annotation in which an annotation indicating the dangerous area is added to the learning image. Suppose that In this case, the convolutional neural network 10a learns the weights for estimating the risk region and the likelihood map indicating the risk level of the risk region as shown in the image 51c of FIG. Here, in the image 51c of FIG. 8, the area behind the bus 511 and the area of the person 512 are dangerous areas, the likelihood 517 of the area behind the bus 511, and the area of the person 512 Likelihood 518 is superimposed.

  In preparing the learning data, the task of adding an annotation indicating the dangerous area to the learning image is done manually, for example, by requesting a crowdsourcing worker, etc. It may be allowed. Hereinafter, a case where the computer of the learning system performs an operation of attaching an annotation indicating a dangerous area to a learning image and performs learning processing on the convolutional neural network 10a using the learning data will be described as an example.

(Example)
[Learning system configuration]
FIG. 9 is a block diagram illustrating an example of a configuration of the learning system in the embodiment. The learning system shown in FIG. 9 includes a learning data creation device 20 and a learning device 30 and performs a learning process for realizing the risk predictor 10.

  The learning data creation device 20 includes a storage unit 201, a storage unit 203, and an annotation assignment unit 202, and creates learning data from video data. The storage unit 201 is configured by an HDD (Hard Disk Drive), a memory, and the like, and stores video data including a plurality of time-series images captured by an in-vehicle camera mounted on the vehicle. This video data is also used as a learning image. The storage unit 203 includes an HDD (Hard Disk Drive), a memory, and the like, and stores annotation-added data in which an annotation indicating a dangerous area is attached to a learning image (video data). The annotation assigning unit 202 attaches at least an annotation indicating a dangerous area to the learning image (video data) acquired from the storage unit 201 and stores the annotation in the storage unit 203.

  The learning device 30 includes an error calculation unit 301 and a weight adjustment unit 302, and performs learning processing on the convolutional neural network 10a using the learning data acquired from the learning data creation device 20. Here, the convolutional neural network 10a has the same configuration as the convolutional neural network used for the risk predictor 10, and functions as a convolutional neural network used for the risk predictor 10 as a result of the learning process. The error calculation unit 301 is a value (correct value) indicating the risk area in the learning image to be estimated by the convolutional neural network 10a and the characteristic of the risk area (correct value), and the output (estimation) that the convolutional neural network 10a actually outputs (estimates). The error is calculated by using an error function with respect to the error from the estimated risk level. The weight adjustment unit 302 adjusts the weight of the convolutional neural network 10a so that the error calculated by the error calculation unit 301 is reduced.

[Learning system operation]
Next, the operation of the learning system configured as described above will be described with reference to FIGS. 10 and 11. FIG. 10 is a flowchart showing a process for creating annotated data performed by the learning data creation apparatus 20. FIG. 11 is a flowchart showing a learning process performed by the learning device 30. 10 corresponds to an example of the detailed process of S11 shown in FIG. 9, and FIG. 11 corresponds to an example of the detailed process of S12 shown in FIG.

  As shown in FIG. 10, the learning data creation device 20 first acquires video data stored in the storage unit 201 (S111). More specifically, the learning data creation device 20 acquires a plurality of time-sequential images taken by an in-vehicle camera mounted on the vehicle as video data.

  Next, the learning data creation device 20 determines a dangerous area included in the video data (S112). More specifically, the learning data creation device 20 is equipped with an in-vehicle camera that is a dangerous region included in at least a part of the acquired plurality of images (video data) and has captured the plurality of images. When a vehicle travels as it is, a dangerous area where a moving object appears in the travel route of the vehicle and may collide with the vehicle is determined.

  Next, the learning data creation apparatus 20 adds an annotation indicating a dangerous area to the video data (S113). More specifically, the learning data creation device 20 assigns an annotation indicating the determined dangerous area to the at least some images. Then, video data (annotated data) obtained by adding an annotation indicating a dangerous area to at least a part of the image is stored in the storage unit 203. In the present embodiment, the learning data creation device 20 gives an annotation indicating a dangerous area to a partial area of an image a predetermined time before a person appears, but is not limited thereto. An annotation indicating a dangerous area may be given to a specific object area of an image a predetermined time before a person appears, such as a car with a ball, a hazard lamp, or a curved mirror showing a person. In any case, when a vehicle equipped with an in-vehicle camera that has captured video data travels as it is, a moving object that is a person 513 appears in the traveling route of the vehicle and indicates a danger area that may collide with the vehicle. This is because an annotation is added.

  Subsequently, as illustrated in FIG. 11, the learning device 30 acquires learning data from the learning data creation device 20 (S121). Specifically, the learning device 30 acquires the image corresponding to the at least a part of the at least a part of the image to which the annotation is added in S113 and the at least a part of the plurality of images (video data) as a learning image. And as an annotated image. For example, as described above, the learning device 30 acquires the learning image including the dangerous region illustrated in FIG. 7A and the annotated image illustrated in FIG. 7D from the learning data creation device 20.

  Next, the learning device 30 causes the convolutional neural network 10a to output a value indicating the estimated risk using the learning image (S122). In the present embodiment, the learning device 30 uses, for example, a risk in the learning image as a value indicating the degree of risk estimated by the convolutional neural network 10a for the learning image including the dangerous region shown in FIG. 7A. A value indicating the region and the characteristics of the dangerous region, that is, the dangerous region and the degree of risk (likelihood map) is output.

  Next, the learning device 30 determines the difference (error) between the value output in S122 and the risk area in the learning image to be estimated by the convolutional neural network 10a and the value (correct value) indicating the characteristic of the risk area. ) Is calculated (S123). Here, the correct answer value is the risk area shown in the image 51c in FIG. 8 calculated from the image 51b shown in FIG. 7D and the annotated image in which the annotation indicating the danger area is added to the learning image, and the risk of the danger area. It is a value indicating a likelihood map indicating the degree.

  Next, when the difference between the value indicating the degree of risk output in S123 and the correct answer value is not minimum (No in S124), the learning device 30 updates the weight of the convolutional neural network 10a so that the difference becomes small. (S125). And the learning apparatus 30 performs the regression process repeated from the process of S122.

  On the other hand, when the difference between the value indicating the degree of risk output in S122 and the correct answer value is minimum, the process is terminated. In other words, the learning device 30 causes the convolutional neural network 10a to learn the weight by adjusting the weight of the convolutional neural network 10a so as to minimize the error calculated in S123. Then, the learning device 30 determines the weight of the convolutional neural network 10a when the error calculated in S123 is minimized as the weight of the convolutional neural network used for the risk predictor 10.

  For convenience of describing FIGS. 10 and 11, the creation processing and learning processing for one image have been described, but the present invention is not limited to this. You may perform a creation process and a learning process for every 10-20 images. Further, the weight of the convolutional neural network 10a with the smallest error is described as being determined as the weight of the convolutional neural network used for the risk predictor 10, but the present invention is not limited to this. If the weight value and the error do not move even when the above regression processing is repeated, the weight 10a of the convolutional neural network when the weight value and the error do not move is determined as the weight of the convolutional neural network used for the risk predictor 10. May be. Further, the minimum error may mean the minimum error in the regression process up to the upper limit number when the upper limit number of regression processes is determined.

(Modification 1)
FIG. 12A is another example of the annotated image prepared in step S1. FIG. 12B is a diagram illustrating another example of the dangerous area output as a result of the learning process and its characteristics. 7A to 8 are denoted by the same reference numerals, and detailed description thereof is omitted.

  In the learning process of the risk predictor 10 described above, the image 51 shown in FIG. 7A and the image 51b shown in FIG. 7D are examples of the learning image and the annotated image, and the correct value is the risk area in the image 51c shown in FIG. Although the case where it is the value which shows the risk degree was demonstrated, it is not restricted to this. The image 51 shown in FIG. 7A and the image 51d shown in FIG. 12A may be a learning image and an annotated image, and the correct value may be a value indicating the dangerous area and its category in the image 51e shown in FIG. 12B.

  In the image 51 shown in FIG. 12A, an area 514d behind the stopped bus 511 is a danger area where an invisible person may jump out and collide with the vehicle, and the danger category 1 is a category of the danger area. It is an annotation to show. Further, the image 51d is further annotated with the region 515 of the visible person 512 and the danger category 2 thereof. The area 515 of the person 512 corresponds to a danger area that may collide with the vehicle, and the danger category 2 corresponds to the category of the danger area. The image 51d is also provided with an annotation indicating that it is a safe area for areas other than these (for example, the area 516d).

  In the image 51e shown in FIG. 12B, the area 519 behind the stopped bus 511 is a dangerous area, the category is a dangerous area (car), and the area 520 of the person 512 is a dangerous area. , It is indicated that the category is a dangerous area (person).

(Modification 2)
The annotation attached to the learning image is not limited to information (such as a frame) indicating the dangerous area described in the embodiment or the category of the moving object associated with the dangerous area described in the first modification. For example, the control information of the vehicle when the learning image such as the steering angle of the vehicle and the brake strength is captured may be detailed area information different from the object area as shown in FIG.

  FIG. 13 is a diagram illustrating an example of detailed area information. The detailed area information shown in FIG. 13 is, for example, segment information 54 for FIG. 7A. An area 541 indicates that the area behind the bus 511 illustrated in FIG. 7A is a dangerous area, and an area 542 indicates that the area of the person 512 illustrated in FIG. 7A is a dangerous area.

  As described above, the annotation attached to the learning image may indicate the dangerous area in the input image and the control information including the brake strength or the steering wheel angle of the vehicle when the learning image is captured. Further, the annotation added to the learning image may be segment information of the dangerous area in the input image.

(Modification 3)
In the first embodiment, the example, and the first and second modifications described above, the danger area where danger is likely to occur for the traveling vehicle is an area including a part of the area of the shielding object existing in the learning image. In the above description, it is assumed that the moving object is hidden and the moving object is an area before the moving object appears in the travel route from the shield. The dangerous area may be an area between two or more moving objects that cross the traveling route of the vehicle when two or more moving objects including a person approach each other.

  Hereinafter, the learning image and the annotated image used for learning processing of such a dangerous area will be exemplified.

  14A, 15A, 16A, and 17A are examples of learning images. FIG. 14B, FIG. 15B, FIG. 16B, and FIG. 17B are examples of annotated images with annotations indicating dangerous areas.

  The learning image 56a shown in FIG. 14A shows a car 561 and a person 562 that are parked and stopped. The annotation-added image 56b shown in FIG. 14B has an annotation indicating that the area 563 between the parked and stopped automobile 561 and the person 562 is a dangerous area with respect to the learning image 56a. Yes.

  The learning image 57a shown in FIG. 15A shows a person 571, an object 572 indicating a bus stop, and an automobile 573 parked and stopped. Annotated image 57b shown in FIG. 15B is annotated with respect to learning image 57a, indicating that area 574 between person 571 and object 572 indicating the bus stop is a dangerous area. ing.

  In addition, the learning image 58a shown in FIG. 16A shows a person 581 as a child and an object 582 used by the person 581 such as a ball for play. In the image 58b with annotation shown in FIG. 16B, an annotation indicating that a region 583 between the person 581 and the object 582 used for play by the person 581 is a dangerous region is attached to the learning image 58a. Has been.

  In addition, the learning image 59a shown in FIG. 17A shows a person 591 who is a child and a person 592 such as a child's parent. In the image 59b with annotation shown in FIG. 17B, an annotation indicating that the area 593 between the person 591 and the person 592 is a dangerous area is added to the learning image 59a.

  As described above, when two or more moving objects including a person approach each other, the vehicle travels across the traveling path. Therefore, the region between the two or more moving objects travels when the vehicle travels as it is. This is considered a dangerous area where a moving object may appear in the route and collide with the vehicle. Therefore, the danger predictor 10 may predict an area between the two or more moving objects as a dangerous area.

(Embodiment 2)
In the first embodiment, the learning process is described using only the convolutional neural network 10a having the same configuration as the convolutional neural network used in the risk predictor 10, but the present invention is not limited to this. First, a first-stage learning process is performed using a convolutional neural network having a fully connected layer, and then a second-stage learning process is performed using a convolutional neural network in which the all-connected layer is changed to a convolutional layer. Good. Hereinafter, this case will be described as a second embodiment. In the following, description will be made centering on differences from the first embodiment.

[Learning process]
The configuration of the risk predictor 10 is the same as that described in the first embodiment, and the description thereof is omitted. The second embodiment is different from the first embodiment in the learning process performed by the computer using the learning data.

  FIG. 18 is a flowchart showing an overview of a two-stage learning process in the present embodiment. FIG. 19 is an explanatory diagram for conceptually explaining the first-stage learning process. FIG. 20 is an explanatory diagram for conceptually explaining the learning process at the second stage. Elements similar to those in FIGS. 3 and 7D and the like are denoted by the same reference numerals, and detailed description thereof is omitted. FIG. 18 corresponds to an example of detailed processing in step S12 illustrated in FIG. Also, the first neural network 10b shown in FIG. 19 is a convolutional neural network having a fully connected layer used in the first stage learning process. The second neural network 10c shown in FIG. 20 is a convolutional neural network used in the learning process at the second stage, and corresponds to a configuration in which all the connection layers of the first neural network 10b are changed to convolutional layers. The second neural network 10 c has the same configuration as the convolutional neural network used for the risk predictor 10.

  First, the computer performs a first learning process in which the first neural network 10b learns the weight (first weight) using the learning data (S221). More specifically, the computer includes a dangerous area image of a part of each of the learning images and an area with an annotation indicating the dangerous area, and a safe area image of an area without the annotation. Is used to learn the first weight of the first neural network for causing the first neural network 10b, which is a convolutional neural network having a fully connected layer, to determine that a partial region is a safe region or a dangerous region. Let

  For example, in the example illustrated in FIG. 19, the computer inputs the dangerous area image 61 that is a partial image of the image 51 b that is an annotated image to the first neural network 10 b and performs the determination so as to perform the determination as the dangerous area. A state in which the first weight of one neural network 10b is learned is shown.

  Next, the computer performs the second learning process on the second neural network 10c using the learning data (S221 to S224). More specifically, the computer first generates a second neural network in which the total connection layer of the first neural network is changed to a convolutional layer (S222). Subsequently, the computer reads the weight (first weight) of the first neural network learned in S221 (S223), and updates the initial value of the weight of the second neural network generated in S222 to the first weight. Subsequently, the computer uses the learning image including the dangerous area and the annotated image in which the learning image is annotated with the annotation indicating the dangerous area, to the second neural network 10c, to detect the danger in the learning image. The weight (second weight) of the second neural network 10c for estimating the region and the feature of the dangerous region is learned (S224).

  For example, in the example illustrated in FIG. 20, the computer inputs an image 50 that is a learning image (input image) to the second neural network 10 c, and the likelihood indicating the risk area and the risk level of the risk area as the risk level. The second weight of the second neural network 10c is updated (learned) so as to estimate the map.

  By performing such second learning processing, the weight of the convolutional neural network used for the risk predictor 10 having the same configuration as that of the second neural network can be learned.

  In the first learning process, the learning area input to the first neural network 10b using the dangerous area image with the annotation indicating the dangerous area and the safe area image without the annotation as the learning image. Although the first weight for determining that the image for use is safe or dangerous is learned, the present invention is not limited to this.

  For example, as shown in FIG. 21, in the first learning process, a partial region of the learning image with the annotation indicating the dangerous region and its category may be used as the learning image. Then, the first neural network 10b is caused to determine whether the input learning image is safe or dangerous, and when it is determined to be dangerous, the first weight for determining the category indicated by the learning image is set. You may learn.

  Here, FIG. 21 is another explanatory diagram for conceptually explaining the first-stage learning process.

  In the example shown in FIG. 21, the computer inputs a dangerous area image 63 obtained by cutting out a region 514d, which is a partial area of the image 51d, which is an annotated image indicating the dangerous area and its category, to the first neural network 10b. Yes. Then, the computer makes a determination that the dangerous area image 63 is a dangerous area, and further, determines the category indicated by the dangerous area image 63 (the dangerous category (vehicle) in the figure) of the first neural network 10b. A state in which one weight is learned is shown. Since there are various variations of the dangerous area other than those described above, in the first learning process, the dangerous category is divided into individual categories (detailed categories) such as each appearance of the object and learned. You may let them.

  Further, the risk level output by the risk predictor 10 is not limited to the likelihood map indicating the risk area and the risk level as shown in FIG. In other words, the second learning process is not limited to the case where the second weight is learned so as to output a likelihood map indicating the dangerous area and the degree of danger as the dangerous area and its feature. The second weight may be learned so as to output the dangerous area and its category.

[Possibility of other embodiments]
As mentioned above, although the risk prediction method of this indication was demonstrated in Embodiment 1-3, it does not specifically limit regarding the main body and apparatus in which each process is implemented. It may be processed by a processor or the like (described below) embedded in a specific device located locally. Further, it may be processed by a cloud server or the like arranged at a location different from the local device.

  Further, the learning image and the input image at the time of danger prediction may be an image (entire image) taken by an in-vehicle camera or a partial image (partial image) of the entire image. As described above, the partial image may be an image of an area estimated to be dangerous. The entire image may be an image when a dangerous situation occurs or before the occurrence.

  In addition, this indication is not limited to the said embodiment. For example, another embodiment realized by arbitrarily combining the components described in this specification and excluding some of the components may be used as an embodiment of the present disclosure. Further, the present disclosure also includes modifications obtained by making various modifications conceivable by those skilled in the art without departing from the gist of the present disclosure, that is, the meanings of the words described in the claims. It is.

  The present disclosure further includes the following cases.

  (1) Specifically, the above apparatus is a computer system including a microprocessor, ROM, RAM, a hard disk unit, a display unit, a keyboard, a mouse, and the like. A computer program is stored in the RAM or hard disk unit. Each device achieves its functions by the microprocessor operating according to the computer program. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions for the computer in order to achieve a predetermined function.

  (2) A part or all of the constituent elements constituting the above-described apparatus may be configured by one system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of components on a single chip, and specifically, a computer system including a microprocessor, ROM, RAM, and the like. . A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to the computer program.

  (3) A part or all of the constituent elements constituting the above-described device may be constituted by an IC card that can be attached to and detached from each device or a single module. The IC card or the module is a computer system including a microprocessor, a ROM, a RAM, and the like. The IC card or the module may include the super multifunctional LSI described above. The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may have tamper resistance.

  (4) Moreover, this indication may be the method shown above. Further, the present invention may be a computer program that realizes these methods by a computer, or may be a digital signal composed of the computer program.

  (5) In addition, the present disclosure provides a computer-readable recording medium such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD ( It may be recorded on a Blu-ray (registered trademark) Disc), a semiconductor memory, or the like. The digital signal may be recorded on these recording media.

  In addition, the present disclosure may transmit the computer program or the digital signal via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like.

  The present disclosure may be a computer system including a microprocessor and a memory, the memory storing the computer program, and the microprocessor operating according to the computer program.

  In addition, the program or the digital signal is recorded on the recording medium and transferred, or the program or the digital signal is transferred via the network or the like and executed by another independent computer system. You may do that.

  The present disclosure particularly relates to a risk prediction method for predicting a danger area where danger is likely to occur in an in-vehicle camera and system mounted on a vehicle for automatic driving, a device or system for driving assistance, and the like. Available to:

DESCRIPTION OF SYMBOLS 10 Risk predictor 10a Convolution neural network 10b 1st neural network 10c 2nd neural network 20 Learning data creation apparatus 30 Learning apparatus 50, 50a, 50b, 51, 51a, 51b, 51c, 51d, 51e, 52 Image 54 Segment information 56b, 57b, 58b, 59b Annotated image 61, 63 Danger area image 101, 102 Convolution layer 201, 203 Storage unit 202 Annotation unit 301 Error calculation unit 302 Weight adjustment unit 501, 511 Bus 502, 503, 512, 513, 562, 571, 581, 591, 592 Person 504, 505, 506, 517, 518 Likelihood 507, 508, 509, 514, 514d, 515, 515d, 516, 516d, 519, 5 0,541,542,563,574,583,593 area 561,573 motor vehicles 572,582 object

Claims (11)

A risk prediction method performed by a computer of a risk predictor using a convolutional neural network,
An acquisition step of causing the convolutional neural network to acquire an input image captured by an in-vehicle camera mounted on the vehicle;
In the convolutional neural network, there is a possibility that a moving object appears in the travel route of the vehicle and collides with the vehicle when the vehicle travels as it is in the dangerous region in the input image acquired in the acquisition step. An output step for estimating a certain dangerous area and a feature of the dangerous area and outputting the estimated risk level for the input image.
Risk prediction method. In the output step, the convolutional neural network is caused to estimate a risk level of the risk area as a characteristic of the risk area, and a likelihood map indicating the estimated risk area and the risk level of the risk area is used as the risk level. Output,
The risk prediction method according to claim 1. In the output step, the convolutional neural network is caused to estimate the type of the moving object associated with the dangerous area as the characteristic of the dangerous area, and output the estimated dangerous area and the type as the degree of danger.
The risk prediction method according to claim 1. The risk prediction method further includes:
Before performing the acquisition step, the learning image including the dangerous area and the annotated image with the annotation indicating the dangerous area are added to the convolutional neural network using the learning image. A learning step for learning the weight of the convolutional neural network for estimating the dangerous area in the image and the feature of the dangerous area;
The risk prediction method according to any one of claims 1 to 3. The learning step includes
Using all of the learning images, a dangerous region image in a region with an annotation indicating the dangerous region, and a safe region image in a region without the annotation, the entire connection layer A first learning step of causing a first neural network that is a convolutional neural network to learn the first weight of the first neural network to determine that the partial region is a safe region or a dangerous region;
Updating an initial value of the weight of the second neural network having a configuration in which all the connection layers of the first neural network are changed to a convolution layer to the first weight learned in the first learning step; Using the learning image and the annotated image in which the learning image is annotated with the annotation indicating the dangerous region, the dangerous region in the learning image and the dangerous image in the second neural network. A second learning step of learning a weight of the convolutional neural network having the same configuration as the second neural network by learning a second weight of the second neural network for estimating the feature of the region; Including,
The risk prediction method according to claim 4. The risk prediction method further includes:
A plurality of time-sequential images taken by an in-vehicle camera mounted on the vehicle are acquired, and the dangerous region is included in at least some of the acquired images, and the vehicle travels as it is In such a case, a dangerous area in which a moving object appears in the traveling route of the vehicle and may collide with the vehicle is determined, and an annotation indicating the determined dangerous area is given to the at least some images. An granting step,
In the learning step, the at least a part of the image to which the annotation is given in the giving step, and an image corresponding to the at least a part of the plurality of images, the learning image and the annotated image And learning the weight of the convolutional neural network,
The risk prediction method according to claim 4 or 5. The dangerous area is an area including a part of the area of the shielding object existing in the learning image, and the moving object is hidden before the moving object appears in the travel route from the shielding object. The area of
The risk prediction method according to any one of claims 4 to 6. The dangerous area is an area between the two or more moving objects that will cross the traveling route of the vehicle when two or more moving objects including a person approach each other.
The risk prediction method according to any one of claims 4 to 6. The annotation indicates the dangerous area and the category of the moving object associated with the dangerous area.
The risk prediction method according to any one of claims 4 to 6. The annotation indicates the dangerous area and control information including a brake strength or a steering wheel angle of the vehicle at the time of capturing the learning image.
The risk prediction method according to any one of claims 4 to 6. The annotation is segment information of the dangerous area.
The risk prediction method according to any one of claims 4 to 6.

Images