WO2022244062A1 - Information processing device, information processing method, and computer program - Google Patents

Abstract

The purpose of the present invention is to provide an information processing device, an information processing method, and a computer program, which enable assorted devices to suitably demonstrate the full functionality thereof.　The information processing device comprises an image acquisition unit, a probability acquisition unit, and a determination unit. The image acquisition unit is configured to be able to acquire an image, the image including a plurality of frames of different timings. The plurality of frames include a first frame and a second frame the timing of which comes after that of the first frame. The probability acquisition unit is configured to be able to acquire a presence probability on the basis of the image, the presence probability being the probability that an object is present within the frame. The determination unit is configured to be able to determine the state of the object on the basis of the presence probability of the first frame and of the second frame.

Description

Information processing device, information processing method, and computer program

The present invention relates to an information processing device, an information processing method, and a computer program.

Information processing devices have various functions, but for example, there are information processing devices that are used to grasp the status of objects (including not only tangible objects but also intangible objects). Examples of such information processing devices include devices that can determine whether or not objects are likely to collide by acquiring distances between objects (an example of targets), human emotions (targets, etc.). For example), it is possible to estimate what kind of situation it is in. For example, Patent Literature 1 proposes an information processing apparatus capable of obtaining biological information from a sensor and estimating human emotions.

JP 2014-144052 A

It is desired that the situation of the target can be properly grasped regardless of the scene in which the information processing device is used.

The present invention has been made in view of such circumstances, and aims to provide an information processing device, an information processing method, and a computer program that can appropriately grasp the situation of a target.

According to the present invention, an image acquisition unit, a probability acquisition unit, and a determination unit are provided, the image acquisition unit is configured to be capable of acquiring an image, the image includes a plurality of frames with different timings, and the The plurality of frames includes a first frame and a second frame whose timing is later than the first frame, and the probability acquisition unit is configured to acquire the existence probability based on the image, The existence probability is a probability that an object exists in the frame, and the determination unit is configured to determine the state of the object based on the existence probabilities of the first and second frames. , an information processing apparatus is provided.

In the present invention, the determination unit is configured to determine the state of the object based on the existence probabilities of the first and second frames, so that the state of the target (object) can be properly grasped.

Various embodiments of the present invention are illustrated below. The embodiments shown below can be combined with each other.
Preferably, the probability acquisition unit is configured to be capable of further acquiring coordinates of the object based on the image, and the determination unit determines in advance the distance between the coordinates of the object in the first and second frames. An information processing apparatus is provided that compares the existence probabilities and determines the situation of the object when it is determined that the distance is equal to or less than a predetermined distance.
Preferably, the probability acquisition unit is configured to be capable of further acquiring type data based on the image, the type data is data specifying the type of the object, and the determination unit includes the first and An information processing apparatus is provided that compares the existence probabilities and determines the situation of the object when it is determined that the type of the object in a second frame is the same.
Preferably, in the type data, a type identification probability is specified for each candidate of the type of the object, and the determination unit selects the candidate with the highest type identification probability among the candidates of the type, An information processing device is provided configured to identify as said type of said object.
Preferably, the information processing device is provided, wherein the determining unit determines the situation of the object by comparing the existence probabilities when determining that the object in the first and second frames is the same. be.
Preferably, the situation corresponds to movement of the object, and the determination unit determines a difference between the existence probability of the second frame and the existence probability of the first frame as a predetermined first threshold. There is provided an information processing device configured to determine that the object is approaching when determining that the object is larger than.
Preferably, an information processing device is provided, wherein the object includes a ship, and the information processing device is configured to determine the situation of relative movement of the ship.
Preferably, a correction probability acquisition unit is further provided, and the correction probability acquisition unit is configured to acquire a correction probability based on the existence probability of the second frame and the existence probability of the first frame, and The correction probability is a probability that the object exists in the frame between the timing of the first frame and the timing of the second frame, and the determination unit determines the An information processing device is provided that is configured to determine said situation of an object.
Preferably, when the existence probability of the second frame is higher than the existence probability of the first frame, the correction probability acquired by the correction probability acquisition unit is higher than the existence probability of the second frame. , an information processing apparatus is provided.
Preferably, when the existence probability of the first frame is higher than the existence probability of the second frame, the corrected probability acquired by the corrected probability acquisition unit is lower than the existence probability of the second frame. , an information processing apparatus is provided.
Preferably, the determination unit determines the situation of the object based on the correction probability when both the existence probabilities of the first and second frames are lower than a predetermined second threshold. An information processing device is provided.
Preferably, the information processing device is provided, wherein the correction probability is higher than the second threshold.
Preferably, the determination unit determines the situation of the object without using the correction probability when at least one of the existence probabilities of the first and second frames is higher than the second threshold. An information processing device configured as follows is provided.

According to another aspect of an embodiment of the present invention, there is provided an information processing method comprising an image acquisition step, a probability acquisition step, and a determination step, wherein the image acquisition step acquires an image, and the image is a plurality of frames having different timings, the plurality of frames having a first frame and a second frame having a later timing than the first frame; and in the probability obtaining step, based on the image and obtaining an existence probability, the existence probability being the probability that an object exists in the frame, and the determining step determining the state of the object based on the existence probability of the first and second frames. A method is provided for determining.

According to another aspect of an embodiment of the present invention, there is provided a computer program for executing an information processing method comprising an image obtaining step, a probability obtaining step, and a determining step, wherein the image obtaining step obtains an image wherein the image includes a plurality of frames with different timings, the plurality of frames having a first frame and a second frame whose timing is later than the first frame, and in the probability obtaining step obtaining an existence probability based on the image, wherein the existence probability is the probability that an object exists in the frame; and in the determining step, based on the existence probability of the first and second frames, A computer program is provided for determining the condition of the object.

According to another aspect of an embodiment of the present invention, it comprises a probability obtaining unit and a modified probability obtaining unit, wherein the probability obtaining unit includes a first situation probability and a timing later than the first situation probability. and a second situation probability, wherein the first and second situation probabilities are probabilities representing the situation of a target, and the modified probability acquisition unit, based on the first and second situation probabilities, A modified probability is obtainable, and the modified probability is a probability representing the situation of the target between the timing of obtaining the first situation probability and the timing of obtaining the second situation probability. , an information processing apparatus is provided.
Preferably, when the second situation probability is higher than the first situation probability, the correction probability acquired by the correction probability acquisition unit is higher than the second situation probability.
Preferably, when the first situation probability is higher than the second situation probability, the correction probability acquired by the correction probability acquisition unit is lower than the second situation probability.

FIG. 1 is a functional block diagram of an information processing device 100 according to an embodiment. FIG. 2A is an explanatory diagram of an image D1 input to the probability acquisition unit 2 and an output D2 output from the probability acquisition unit 2. FIG. FIG. 2B is a schematic diagram for explaining the type data d3. FIG. 3 is a schematic diagram of the image D1. FIG. 4 is a position explanatory diagram of each object, showing the state of each object in the image D1 shown in FIG. 3 as viewed from above. FIG. 5A is a schematic diagram of the first frame of image D1. FIG. 5B is a schematic diagram of the second frame of image D1. FIG. 6 is a control flowchart of the information processing device 100 according to the embodiment. FIG. 7 is a functional block diagram of an information processing apparatus 100 according to Modification 1. As shown in FIG. FIG. 8 is a control flowchart of the information processing apparatus 100 according to Modification 1. As shown in FIG. FIG. 9 is a schematic diagram of an image frame according to Modification 2. FIG. FIG. 10A is a schematic diagram of the first frame of the image D1 according to Modification 3. FIG. 10B is a schematic diagram of the second frame of the image D1 according to Modification 3. FIG. FIG. 11A is an explanatory diagram of an image D1 input to the probability acquisition unit 2 and an output D2 output from the probability acquisition unit 2 according to Modification 3. FIG. 11B is a schematic diagram for explaining the existence probability d1 according to Modification 3. FIG. FIG. 12 is a control flowchart of the information processing apparatus 100 according to Modification 3. As shown in FIG. FIG. 13 is a functional block diagram of an information processing apparatus 100 according to Modification 4. As shown in FIG.

Embodiments of the present invention will be described below with reference to the drawings. Various features shown in the embodiments shown below can be combined with each other. In addition, the invention is established independently for each characteristic item.

1 Description of Overall Configuration The overall configuration of the information processing apparatus 100 according to the embodiment will be described. In the embodiment, the information processing device 100 will be described as being mounted on a ship (the ship obt on which the information processing device 100 shown in FIG. 4 is mounted). The information processing device 100 has a function of quickly determining whether or not a predetermined target on the sea is approaching the onboard vessel obt. Here, in the embodiment, the target is an object (tangible object). Also, it is not limited whether or not the object operates by its own power. Also, the object is not limited to an object floating on the sea, and may be an object floating or flying. Objects are, for example, ships, buoys, yachts, buoys, and the like. Some ships are difficult to turn abruptly and speed up quickly, so it is important to know in advance whether an object is approaching the ship at an early stage to avoid a collision between the ship and the object. It is useful from the viewpoint of

In addition to the information processing device 100, the imaging device 20 and the external device 21 shown in FIG. The imaging device 20 is configured to output to the information processing device 100 data (image D<b>1 ) obtained by imaging the surroundings (front in the embodiment) of the onboard ship obt. The imaging device 20 is attached to the loading ship obt. The external device 21 is composed of, for example, a speaker, a display, or the like, and has a function of informing passengers of the on-board ship obt whether or not an object is approaching by means of voice, text, or the like. Information processing apparatus 100 can determine whether an object is approaching based on a plurality of frames of image D<b>1 and output the determination result to external device 21 . As a result, the information processing apparatus 100 can prompt the passengers of the on-board ship obt to take an avoidance action at an early stage.

The information processing device 100 includes an image acquisition unit 1 , a probability acquisition unit 2 , a determination unit 3 and an output unit 4 . Each component of the information processing apparatus 100 may be realized by software or by hardware. When realized by software, various functions can be realized by the CPU executing a computer program. The program may be stored in a built-in storage unit, or may be stored in a computer-readable non-temporary recording medium. Alternatively, a program stored in an external storage unit may be read out and implemented by so-called cloud computing. When implemented by hardware, it can be implemented by various circuits such as ASIC, FPGA, or DRP. In this embodiment, various information and concepts including this are handled, and these are represented by high and low signal values as binary bit aggregates composed of 0 or 1, and the above software or hardware Communication and computation can be performed according to the aspect of .

2 Detailed Configuration Description of Information Processing Apparatus 100 2-1 Image Acquisition Unit 1
As shown in FIG. 1, the image acquisition unit 1 is configured to acquire an image D1 from an imaging device 20. As shown in FIG. Note that the processing performed by the image acquisition unit 1 is an example of the image acquisition step. Here, in the embodiment, the image D1 has multiple frames. A plurality of frames is data for which time-series information is specified. Specifically, the multiple frames include a first frame D11 and a second frame D12 whose timing is later than the first frame, as shown in FIGS. 5A and 5B. The timing (time) in the first frame is defined as first timing, and the timing (time) in the second frame is defined as second timing. In the example frames shown in FIGS. 5A and 5B, the ship ob1 is approaching the onboard ship obt. Therefore, in the frame shown in FIG. 5B, the ship ob1 is displayed gradually larger and more clearly than in the frame shown in FIG. 5A.

A configuration example of the image D1 will be described with reference to FIG. In the embodiment, the frame of the image D1 shows the ship ob1, the sky ob2, and the sea ob3. In the embodiment, the object refers to an object such as a buoy, a yacht, or the like, other than the ship ob1, which may become an obstacle to the progress of the ship.

The ship ob1 is located in front of the onboard ship obt (imaging device 20). As shown in FIG. 4, defining the Cartesian coordinates centering on the on-board ship obt, the ship ob1 is positioned on the y-axis. In FIG. 4, the y-axis is parallel to the traveling direction of the on-board vessel obt, and the x-axis is orthogonal to the y-axis. Vessel ob1 is an object approaching onboard vessel obt. Specifically, the ship ob1 is moving in the -y-axis direction.

In the embodiment, the determination unit 3, which will be described later, performs situation determination such as whether or not an object is approaching based on the difference in the situation probabilities of the object in the two frames. A situation probability is a probability representing a situation of an object (an object in the embodiment), and a situation probability in the embodiment is an existence probability of the object (an object in the embodiment). Existence probability is the probability that a predetermined object exists in the frame of the image.
The difference in timing between the first frame and the second frame is not particularly limited, but is preferably set according to the device on which the information processing apparatus 100 is installed. For example, when the information processing apparatus 100 is mounted on a ship as in the embodiment, the moving speed of an object that can be an obstacle is lower than that of a vehicle, an airplane, or the like. Therefore, the difference between the first timing and the second timing is, for example, 10 seconds, 20 seconds, 30 seconds, 40 seconds, 50 seconds, 1 minute, 2 minutes, 3 minutes, 4 minutes, 5 minutes, 6 minutes, 7 minutes, 8 minutes, 9 minutes, 10 minutes. Also, the difference between the first timing and the second timing may be represented by a range of two values among the values listed here.

2-2 Probability acquisition unit 2
The probability acquisition unit 2 acquires an output D2 including the existence probability d1 based on the image D1, as shown in FIG. 2A. The probability acquisition unit 2 then outputs the output D2 to the determination unit 3 . Note that the processing performed by the probability acquisition unit 2 is an example of the probability acquisition step. The output D2 has existence probability d1 (existence probability data), coordinates d2 (coordinate data), and type data d3. Note that the existence probability d1 is data (situation probability data) in which the situation probability (existence probability in the embodiment) is specified. Here, a configuration example of these data will be described.

The existence probability d1 is data specifying the value of the existence probability of an object in the frame of the image D1. The existence probability of the existence probability d1 is the probability that a predetermined object exists in the frame of the image D1.

The coordinates d2 are data specifying the coordinates of the object in the frame of the image D1. In an embodiment, the coordinates of coordinate d2 are in the xz coordinate system. The coordinates d2 also include data specifying the width W and height H of the object in the frame of the image D1.

The type data d3 is data for specifying the type of the object in the frame of the image D1. The type data d3 is data in which the probability is specified for each predetermined type for each object in the frame of the image D1. For example, as shown in FIG. 2B, predetermined types in an embodiment include ships, buoys, yachts, and the like. In embodiments, the objects in the frame of image D1 include ships, buoys and yachts. For example, the type data d3 specifies the probabilities of all candidate objects, such as the probability of being a ship, the probability of being a buoy, and the probability of being a yacht, with respect to the object corresponding to the ship ob1. It is

The above output D2 (existence probability d1, coordinates d2, and type data d3) can be calculated based on a learning model that inputs image D1 and outputs output D2. A learning model is a model that is trained using a large amount of teacher data (a set of known input data and correct answer data) to make it possible to predict future outputs. In this embodiment, the input data of the teacher data is image data (frames) obtained by transferring an object such as a ship on the sea, and the correct data is the existence probability of the object shown in this image data, the coordinates of the object, Includes the width and height of the object and the type of object. In this embodiment, the learning model (machine learning model) can employ various object detection algorithms such as YOLO (You Only Look Once) and SSD (Single Shot Detector).

2-3 Determination unit 3
The determination unit 3 has a function of determining the presence or absence of the coordinate d2 (hereinafter also referred to as a first function) and a function of determining whether the distance between coordinates between objects in different frames is equal to or less than a predetermined distance (hereinafter referred to as a first function). , also referred to as a second function), a function of determining whether the types of objects in different frames are the same (hereinafter also referred to as a third function), and a function of comparing the existence probabilities of objects in different frames. (hereinafter also referred to as a fourth function). The determination unit 3 is configured to be able to output a determination result indicating the state of the object.

First, the first function will be explained. The determination unit 3 determines the presence or absence of the coordinate d2 of the second frame (chronologically later frame). If the existence probability of the second frame (chronologically subsequent frame) is 0%, the output D2 does not include the coordinate d2, and if the existence probability is higher than 0%, the output D2 does not include the coordinate d2. d2 is included. That is, in the embodiment, when the coordinate d2 exists in the output D2, it is equivalent to the case where the existence probability is higher than 0%. Note that the first function may be replaced by threshold determination. That is, the determination unit 3 may be replaced with a function of determining whether or not the existence probability is higher than a predetermined threshold value (for example, 5%). In other words, under conditions where the existence probability is considerably low, the information processing apparatus 100 may process the object as not existing in the frame.

Next, the second function will be explained. The image acquisition unit 1 acquires the coordinates of the object in the first frame (the frame chronologically earlier) and the coordinates of the object in the second frame (the frame chronologically later). The coordinates are output to the determination unit 3 as the output D2. The determination unit 3 determines whether the inter-coordinate distance between the coordinates of the object in the first frame and the coordinates of the object in the second frame is equal to or less than a predetermined distance. The information processing apparatus 100 uses this inter-coordinate distance as a criterion for determining whether or not the object in each frame is the same. This is because, if the distance between coordinates is equal to or less than the predetermined distance, it increases the possibility that the object in the first frame and the object in the second frame are the same.

Next, the third function will be explained. The image acquisition unit 1 acquires the object type data d3 of the first frame (chronologically earlier frame) and the object type data d3 of the second frame (chronologically later frame). These data are output to the determination unit 3 as the output D2.
Here, the determination unit 3 identifies the type of object using the type data d3 of each frame. Specifically, the determination unit 3 identifies the candidate with the highest probability among the object type candidate probabilities (type identification probabilities) in the first frame as the object type. For example, in the case of the image D1 shown in FIG. 3, since the ship is clearly displayed, the probability of the ship is calculated high, and the probability of the buoy and the yacht is calculated low. The determination unit 3 compares the magnitude of these probabilities and identifies the ship, which is the highest candidate, as the type of object. The determination unit 3 similarly identifies the type of object for the second frame. That is, the determination unit 3 identifies the candidate with the highest probability among the object type candidate probabilities in the second frame as the object type.
Then, the determination unit 3 determines whether or not the type of the object in the first frame is the same as the type of the object in the second frame. In this way, the information processing apparatus 100 uses the type of object as a criterion for determining whether the objects in each frame are the same. This is because if the types are the same, the possibility that the object in the first frame and the object in the second frame are the same increases.

Next, the fourth function will be explained. The image acquisition unit 1 acquires the object existence probability of the first frame (chronologically earlier frame) and the object existence probability of the second frame (chronologically later frame), These existence probabilities are output to the determination unit 3 as the output D2. The determination unit 3 has a function of comparing the existence probability of the object in the first frame and the existence probability of the object in the second frame. Specifically, the determination unit 3 determines that a value (difference) obtained by subtracting the existence probability of the first frame from the existence probability of the second frame is greater than a predetermined threshold Th1 (an example of the first threshold). is determined, it is determined that the object is approaching. In other words, the determination unit 3 determines that an object is approaching when the existence probability rises above the predetermined threshold value Th1 from the first frame to the second frame. In this way, the information processing apparatus 100 makes a situation judgment using the difference in the existence probability instead of judging the situation based on the existence probability alone. Note that the determination result of the determination unit 3 corresponds to the determination result D3.

Here, the value of the threshold Th1 is not particularly limited, but the value of the threshold Th1 can be set to any numerical value from 10 to 50, for example. The value of the threshold Th1 can be appropriately set according to the information processing apparatus 100 installed therein.

2-4 Output section 4
The output unit 4 is configured to output the determination result D3 to the external device 21 . The external device 21 is not limited to equipment such as a display mounted on the onboard ship obt. It may be output to a control center that supervises the operation status of the on-board ship obt.

3 Description of Operation The operation of the information processing apparatus 100 will be described with reference to FIG. The control flowchart in FIG. 6 can be implemented at the stage when frames with different timings are acquired.

Step S1: Judgment of presence/absence of coordinate d2 (first function)
The determination unit 3 determines whether or not the coordinate d2 exists in the second frame (in the later frame in chronological order). If it exists, the process proceeds to step S2. If it does not exist, the process proceeds to step S5-2.

Step S2: Threshold determination of distance between coordinates (second function)
The determination unit 3 determines whether or not the distance between the coordinates of the object in the first and second frames is equal to or less than a predetermined distance. If the distance is less than or equal to the predetermined distance, there is a possibility that both the objects in the first and second frames are the same, so the process proceeds to step S3. If the distance is not equal to or less than the predetermined distance, the process proceeds to step S5-2.

Step S3: Judgment of Same Type (Third Function)
The determination unit 3 determines whether or not the types of objects in the first and second frames are the same. In making this determination, the determination unit 3 specifies in advance the types of objects in the first and second frames. That is, the determination unit 3 identifies the candidate with the highest probability (type identification probability) among the object type candidates as the object type for each of the first and second frames. If the types of objects are the same, the possibility that both objects are the same increases, so the process proceeds to step S4. If the types of objects are not the same, the process proceeds to step S5-2.

Step S4: Comparison of existence probabilities (fourth function)
The information processing apparatus 100 according to the embodiment assumes that the objects in the first and second frames are the same when steps S1 to S3 have been performed. That is, steps S1 to S3 can be said to be steps for determining whether the objects in the first and second frames are the same. The determination unit 3 acquires the difference in the existence probability of the objects considered to be identical (a value obtained by subtracting the existence probability of the object in the first frame from the existence probability of the object in the second frame). Then, the determination unit 3 determines whether or not this difference is greater than a predetermined threshold value Th1. If it is greater than the threshold Th1, the process proceeds to step S5-1. If it is not greater than the threshold, the process proceeds to step S5-2. Step S4 is an example of a determination step.

Step S5-1: Approaching The determination unit 3 generates a determination result D3 indicating that the object is approaching.

Step S5-2: Not Approaching The determination unit 3 generates a determination result D3 indicating that the object is not approaching.

4 Effect of Embodiment Conventionally, when it is determined that the probability of being a specific object is equal to or greater than a predetermined threshold, a specific control (control related to avoidance of an object) is performed. is known. In such a device, for example, when the value of the probability fluctuates rapidly, there is an increased possibility that specific control cannot be executed appropriately. Also, in the case of a vehicle, such as a large ship, in which it is difficult to quickly increase speed or quickly change the direction of travel, there is an increased possibility that specific control cannot be executed appropriately. In other words, depending on the scene in which the conventional information processing device is used, it may not be possible to properly grasp the situation of the target (object), and as a result, it may not be possible to properly execute specific control. increase.
The information processing apparatus 100 according to this embodiment is configured such that the determination unit 3 determines the state of an object based on the existence probabilities of the first and second frames. That is, unlike the conventional information processing apparatus, the information processing apparatus 100 uses the difference between the existence probabilities to perform the avoidance action, instead of using the probability alone to determine whether the avoidance action should be taken. It is determined whether or not it should be. Therefore, the information processing apparatus 100 can determine whether or not there is a situation in which specific control should be performed more in advance than the information processing apparatus of the related art. control function) can be exhibited appropriately.

5 Modification 5-1 Modification 1: Modified Probability Acquisition Unit 5
In the embodiment, the situation is determined based on the difference between the existence probabilities of the first and second frames, but the present invention is not limited to this. The information processing apparatus 100 may further include a correction probability acquisition unit 5, as shown in FIG.

The correction probability acquisition unit 5 is configured to be able to acquire the correction probability D21 based on the difference between the existence probability of the second frame and the existence probability of the first frame. The correction probability D21 is data in which the value of the correction probability is specified. The correction probability is the probability that an object exists in the frame between the timing of the first frame and the timing of the second frame. In the embodiment, the existence probability difference itself is compared with the threshold value, but in the modified example, the difference is replaced with the probability (correction probability Pc) again.

The correction probability Pc can be obtained based on, for example, a predetermined relational expression. That is, once the existence probabilities of the first and second frames are determined, the correction probability Pc is uniquely determined.
This relational expression is set so that, for example, when the existence probability of the second frame is higher than the existence probability of the first frame, the correction probability is higher than the existence probability of the second frame. That is, when the existence probability increases, it can be said that there is a high possibility that an object is approaching, so the correction probability Pc is set higher than the existence probability of the second frame.
In addition, when the existence probability is lowered, it can be said that the object is not approaching. Therefore, in this relational expression, when the existence probability of the first frame is higher than the existence probability of the second frame, the correction probability Pc is set to a value lower than the existence probability of the second frame. .

Also, the flowchart in the modified example differs from the flowchart in the embodiment in that it includes steps S4 and S5 as shown in FIG. Step S4 is a step in which the correction probability acquisition unit 5 acquires the correction probability D21 based on the difference between the existence probability of the second frame and the existence probability of the first frame. In step S5, the determination unit 3 determines whether or not the correction probability Pc of the correction probability D21 is greater than the threshold Th2. If the correction probability Pc is greater than the threshold Th2, it is determined that the object is approaching (step S6-1).

For example, a case where the existence probability of the object in the first frame is 1% and the existence probability of the object in the second frame is 6%, and a case where the existence probability of the object in the first frame is 20% and The difference itself is the same as when the existence probability of the object is 25%. However, the magnitude of the existence probability differs between the former case and the latter case. Therefore, even if the difference itself is the same between the former case and the latter case, it may be preferable to differentiate the determination contents. For example, in the former case, since the existence probability itself is very low, an error may have increased the existence probability. On the other hand, in the latter case, since the existence probability is relatively high, the increase in the existence probability is not an error but is likely to be significant. Therefore, according to the predetermined relational expression of the modified probability acquisition unit 5 according to the first modification, even if the difference in the existence probability is the same, if the magnitude of the existence probability itself is different, there is a difference in the determination content. It is preferable that the

Moreover, it is preferable to select whether to use the difference or the correction probability for the situation determination.
For example, when both the existence probabilities of the first and second frames are lower than a predetermined threshold value (an example of the second threshold value), the determination unit 3 determines the state of the object based on the correction probability Pc. It may be configured as In this case, the correction probability Pc is higher than a predetermined threshold (an example of the second threshold).
Further, when at least one of the existence probabilities of the first and second frames is higher than the threshold (an example of the second threshold), the determination unit 3 determines the state of the object without using the correction probability Pc. may be configured to In this way, by determining whether or not to use the correction probability according to the magnitude of the existence probability of the first and second frames, it is possible to realize appropriate situation determination according to the scene.

5-2 Modified Example 2: When Multiple Objects Exist In the embodiment, a form in which only one object to be determined (ship ob1) is contained in the frame has been described, but the present invention is not limited to this. . Multiple objects may fit in the frame. For example, as shown in FIG. 9, buoy ob4 and ship ob5 may appear in the frame of the image.

Here, since buoy ob4 exists near the onboard ship, the probability of existence of buoy ob4 is quite high. For an object with a high probability of existence, it is preferable to use another known method to determine whether the object is approaching, instead of using the series of controls (steps S1 to S5-2 in FIG. 6) as described in the embodiment. . In other words, if the existence probability is higher than a predetermined threshold, it can be assumed that the object is essentially near the onboard ship. (Steps S1 to S5-2 in FIG. 6) are preferably not executed. In this case, for example, as a step preceding step S1 in the flowchart shown in FIG. Then, if the existence probability is less than the threshold value, the process proceeds to step S1, and if the existence probability is equal to or greater than the threshold value, approach determination using a known method may be performed. In a known method, for example, approach determination can be performed based on changes in the size of an object or changes in the coordinates of an object.

Also, the ship ob1 and ob5 are contained in the frame, and the difference between these objects is determined by whether the distance between the coordinates in step S2 is equal to or less than the threshold. That is, when the flowchart shown in FIG. 6 is executed for the ship ob1, there are coordinates d2 of two objects (buoy ob4 and ship ob5) in addition to the coordinates d2 of the ship ob1. However, while the difference (inter-coordinate distance) between the coordinates d2 of the ship ob1 in the first frame and the coordinates d2 of the ship ob1 in the second frame is short, the coordinates d2 of the ship ob1 in the first frame and the buoys in the second frame The difference between the coordinates d2 of ob4 and the difference between the coordinates d2 of the ship ob1 in the first frame and the coordinates d2 of the ship ob5 in the second frame are long. Therefore, by performing step S2, it is possible to avoid misjudgment that the ship ob1 in the second frame is the buoy ob4 or the ship ob5. The same is true for the buoy ob4 and the ship ob5. In other words, by performing step S2, it is possible to avoid erroneous determination of the buoy ob4 as the ship ob1 or ob5, or erroneous determination of the ship ob5 as the ship ob1 or the buoy ob4.

5-3 Modified Example 3: Color State Change In the embodiment, the information processing device 100 is mounted on a ship, and it is explained that the situation is determined whether or not the ship is approaching an obstacle. It is not limited. This situation determination may be a determination of a change in color of the object. For example, it may be applied to changes in the color of traffic lights or changes in the color of leaves.

In Modified Example 3, a traffic light will be used as an example. Further, in Modified Example 3, the information processing device 100 is described as being mounted in a vehicle. Furthermore, a situation in which the color of the traffic light changes from red (color indicating stop) to blue (color indicating possible start) while the vehicle is stopped will be described as an example.

In the embodiment, for the output D2, the existence probability and the type of object are separated as separate data. Since the situation determination in Modification 3 is a change in the color of the signal, the probability acquisition unit 2 acquires the existence probability d1 and the coordinates d2 as the output D2, as shown in FIG. 11A. Then, as shown in FIG. 11B, the structure of the existence probability d1 indicates the probability that each color exists.

A configuration example of the image D1 will be described with reference to FIGS. 10A and 10B. In Modified Example 3, the object is a traffic light, and the objects are distinguished according to the lighting color. As shown in FIG. 10A, the first frame D11 of the image D1 shows a traffic light ob61 lit in red, a road ob7, and a sidewalk ob8. As shown in FIG. 10B, the second frame D12 of the image D1 shows a traffic light ob62 that is lit in blue, a road ob7, and a sidewalk ob8.

Next, the operation of the information processing apparatus 100 according to Modification 3 will be described with reference to FIG. Steps S1 and S2 are the same as in FIG. 6, so the description is omitted.

Step S3: Determination of Color Type In step S3, the determination unit 3 determines whether or not the color of the traffic light in the first and second frames has changed based on the existence probability d1. In making this determination, the determination unit 3 specifies in advance the color of the traffic light in the first and second frames based on the existence probability d1. That is, the determination unit 3 identifies the candidate with the highest probability among the candidates for the color of the traffic light as the type of color for each of the first and second frames.
When the determination unit 3 determines that the color of the traffic light has changed, the process proceeds to step S4-1. In other words, when the determination unit 3 determines that the color of the traffic light has changed from red to blue, the process proceeds to step S4-1.
When the determination unit 3 determines that the color of the traffic light has not changed, the process proceeds to step S4-2. In other words, if the determination unit 3 determines that the color of the traffic signal has not changed, or if the color of the traffic signal has changed but does not change from red to blue, the process proceeds to step S4-2.

Step S4-1: Change from Red to Blue The determination unit 3 generates a determination result D3 representing a situation in which the color of the traffic light changes from red to blue. Accordingly, the information processing device 100 can inform the passenger that the vehicle may start. Further, when the vehicle is automatically driven, the information processing device 100 can output information indicating that the vehicle is ready to proceed to the control device of the vehicle.

Step S4-2: No Change from Red to Blue The determination unit 3 generates a determination result D3 representing a situation where the color of the traffic light has not changed from red to green. Thereby, the information processing device 100 can inform the passenger of the situation that the vehicle needs to be maintained in a stopped state. Further, in the case of automatic driving, the information processing device 100 can output to the control device of the vehicle information indicating that the vehicle needs to be maintained in a stopped state.

5-4 Modification 4: Emotion analysis using voice as input The situation determination in the embodiment and Modifications 1 to 3 was the movement of tangible objects and the change in color of tangible objects, but is limited to this. is not. The information processing apparatus 100 can also be applied to, for example, emotion analysis using voice as input. Specifically, the situation determination in Modification 4 corresponds to the determination of the degree of emotion. In Modified Example 4, the degree of emotion is, for example, a minimum of 0% and a maximum of 100%, with a higher number representing a better impression.

It should be noted that emotions are an example of targets. Also, the degree of emotion is an example of the situation probability.

In Modification 4, the above-mentioned situational probability is used instead of the existence probability. The concept of situation probability includes the concept of existence probability in the embodiment and modifications 1-3 thereof. In other words, the existence probability is one aspect of the situational probability. The situation probability is a probability representing the situation of any target, not limited to tangible objects.

As shown in FIG. 13, the information processing apparatus 100 according to Modification 4 includes a voice acquisition section 1t instead of the image acquisition section 1 according to the embodiment. The voice acquisition unit 1t is configured to be able to acquire a voice D1t (for example, words) from a voice input device 20t such as a microphone.

The probability acquisition unit 2 is configured to output the situation probability (degree of emotion) as the output D2 when the voice D1t is input from the voice acquisition unit 1t. The situation probability is data specifying the value of the situation probability of the target at a certain timing. For example, assume that when voice data at the first timing is input to the probability obtaining unit 2, data specifying a situation probability of 10% (an example of the first situation probability) is output from the probability obtaining unit 2. Further, when the speech at the second timing, which is later than the first timing, is input to the probability acquisition unit 2, the probability acquisition unit 2 outputs data specifying a situation probability of 20% (an example of the second situation probability). shall have been

The determination unit 3 has a function of calculating the difference in situation probabilities. Specifically, the determination unit 3 predetermines a value (difference) obtained by subtracting the situation probability (10%) of the situation probability at the first timing from the situation probability (20%) of the situation probability at the second timing. It is configured to determine that the emotion has improved when it is determined to be greater than a set threshold value (eg, 5%). In the example of Modification 4, the difference is 10%, and since the difference is larger than the threshold, the determination unit 3 determines that the emotions have improved.

Modification 4 can also be combined with Modification 1. FIG. That is, the correction probability acquisition unit 5 may be configured to be able to acquire the correction probability based on the difference between the situation probabilities at the first and second timings.
Note that the correction probability is data in which the value of the target correction probability is specified. Further, the correction probability is a probability representing the situation of the target (emotion in the present modified example 4) between the first timing and the second timing.
For example, when the situation probability at the second timing is higher than the situation probability at the first timing, the corrected situation probability acquired by the correction probability acquisition unit 5 is set to be higher than the situation probability at the second timing. be.
Further, for example, when the situation probability at the first timing is higher than the situation probability at the second timing, the corrected situation probability acquired by the correction probability acquisition unit 5 is set to be lower than the situation probability at the second timing. set.

1: image acquisition unit, 1t: voice acquisition unit, 2: probability acquisition unit, 3: determination unit, 4: output unit, 5: correction probability acquisition unit, 20: imaging device, 20t: voice input device, 21: external device , 100: information processing device, D1: image, D11: first frame, D12: second frame, D1t: audio, D2: output, D21: correction probability, D3: determination result, H: vertical width, Pc: correction probability , Th1: threshold, Th2: threshold, W: width, d1: existence probability, d2: coordinates, d3: type data, ob1: ship, ob2: sky, ob3: sea, ob4: buoy, ob5: ship, ob61: red ob62: traffic light lit in blue, ob7: road, ob8: sidewalk, obt: loaded ship

Claims (18)

An image acquisition unit, a probability acquisition unit, and a determination unit,
The image acquisition unit is configured to acquire an image,
The image includes a plurality of frames with different timings,
The plurality of frames has a first frame and a second frame whose timing is later than the first frame,
The probability acquisition unit is configured to acquire an existence probability based on the image,
the existence probability is the probability that an object exists in the frame;
The information processing apparatus, wherein the determination unit is configured to determine the state of the object based on the existence probabilities of the first and second frames. The information processing device according to claim 1,
The probability acquisition unit is configured to be able to further acquire the coordinates of the object based on the image,
The determination unit determines the situation of the object by comparing the existence probabilities when determining that the distance between the coordinates of the object in the first and second frames is equal to or less than a predetermined distance. information processing device. The information processing device according to claim 1 or claim 2,
The probability acquisition unit is configured to be able to further acquire type data based on the image,
The type data is data specifying the type of the object,
The information processing apparatus, wherein the determining unit determines the situation of the object by comparing the existence probabilities when determining that the types of the object are the same in the first and second frames. The information processing device according to claim 3,
a type identification probability is specified for each type candidate of the object in the type data;
The information processing device, wherein the determination unit is configured to identify the candidate with the highest type identification probability among the candidates of the type as the type of the object. The information processing device according to claim 1,
The information processing apparatus, wherein the determining unit determines the situation of the object by comparing the existence probabilities when determining that the object in the first and second frames is the same. The information processing device according to any one of claims 1 to 5,
the situation corresponds to movement of the object;
The determining unit determines that the object is approaching when determining that a difference between the existence probability of the second frame and the existence probability of the first frame is greater than a predetermined first threshold. An information processing device configured to determine. The information processing device according to claim 6,
the object includes a ship;
An information processing device, wherein the information processing device is configured to determine the situation of relative movement of the ship. The information processing device according to claim 1,
further comprising a modified probability acquisition unit,
The correction probability acquisition unit is configured to be able to acquire a correction probability based on the existence probability of the second frame and the existence probability of the first frame,
the correction probability is a probability that the object exists in the frame between the timing of the first frame and the timing of the second frame;
The information processing device, wherein the determination unit is configured to determine the situation of the object based on the correction probability. The information processing device according to claim 8,
information processing, wherein when the existence probability of the second frame is higher than the existence probability of the first frame, the correction probability acquired by the correction probability acquisition unit is higher than the existence probability of the second frame; Device. The information processing device according to claim 8 or claim 9,
information processing, wherein when the existence probability of the first frame is higher than the existence probability of the second frame, the correction probability acquired by the correction probability acquisition unit is lower than the existence probability of the second frame; Device. The information processing device according to any one of claims 8 to 10,
The determination unit is configured to determine the situation of the object based on the correction probability when both the existence probabilities of the first and second frames are lower than a predetermined second threshold. information processing equipment. The information processing device according to claim 11,
The information processing device, wherein the correction probability is higher than the second threshold. The information processing device according to claim 11 or claim 12,
The determination unit is configured to determine the situation of the object without using the correction probability when at least one of the existence probabilities of the first and second frames is higher than the second threshold. Information processing equipment. An information processing method comprising an image acquisition step, a probability acquisition step, and a determination step,
The image acquisition step acquires an image,
The image includes a plurality of frames with different timings,
The plurality of frames has a first frame and a second frame whose timing is later than the first frame,
In the probability acquisition step, an existence probability is acquired based on the image;
the existence probability is the probability that an object exists in the frame;
In the determining step, the method determines the state of the object based on the existence probabilities of the first and second frames. A computer program for executing the information processing method according to claim 14. A probability acquisition unit and a modified probability acquisition unit,
The probability acquisition unit is configured to be able to acquire a first situation probability and a second situation probability whose timing is later than the first situation probability,
The first and second situation probabilities are probabilities representing the situation of the target,
The modified probability acquisition unit is configured to be able to acquire modified probabilities based on the first and second situation probabilities,
The information processing apparatus, wherein the modified probability is a probability representing the situation of the target between the timing of acquiring the first situation probability and the timing of acquiring the second situation probability. The information processing device according to claim 16,
The information processing apparatus, wherein the correction probability acquired by the correction probability acquisition unit is higher than the second situation probability when the second situation probability is higher than the first situation probability. The information processing device according to claim 16 or 17,
The information processing device, wherein the correction probability acquired by the correction probability acquisition unit is lower than the second situation probability when the first situation probability is higher than the second situation probability.

Images