JP2020011675A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of improving the reliability of information regarding a parking frame obtained based on an image taken by a camera. An image processing device includes a recognition unit that recognizes a parking frame based on a captured image obtained by a camera that photographs the surroundings of a vehicle, and management information including position information of the parking frame that has been recognized in the past. It is provided with a management unit that updates based on observation information including the position information of the parking frame obtained from the currently captured image. Based on the management information and the observation information, the management unit classifies the update type for each section line constituting the parking frame to be managed, and updates the management information based on the classification. [Selection diagram] Fig. 3

Description

  The present invention relates to an image processing device and an image processing method, and more particularly, to a technology for recognizing and managing a parking frame by image processing.

  Patent Literature 1 discloses that a white line is detected from an image captured by a camera and a region between the two white lines is detected as a parking space. Patent Document 1 discloses that information on a detected parking space is updated by using a movement amount of a vehicle from a processing cycle immediately before output by a vehicle motion acquisition unit to a current processing cycle.

JP 2017-87758 A

  In the method disclosed in Patent Document 1, information on the traveling direction and speed of a vehicle is acquired via a CAN bus, and the amount of movement of the vehicle is obtained based on the information to estimate the current position of the parking space. With this configuration, when the estimation error of the movement amount of the vehicle is accumulated, the position of the parking space may be shifted from an appropriate position. Further, due to the influence of distortion of the lens or the like, a white line cannot be appropriately detected from a captured image at a position distant from the camera, and an appropriate parking space may not be detected.

  The present invention has been made in view of the above circumstances, and has as its object to provide a technique capable of improving the reliability of information on a parking frame obtained based on a captured image of a camera.

  In order to achieve the above object, an image processing apparatus according to the present invention includes a recognition unit that recognizes a parking frame based on a captured image obtained by a camera that captures a periphery of a vehicle, and position information of the parking frame recognized in the past. A management unit that updates management information based on observation information including position information of the parking frame obtained from a current captured image, and the management unit is based on the management information and the observation information. An update type is classified for each lane line constituting the parking frame to be managed, and the management information is updated based on the classification (first configuration).

  In the image processing device having the first configuration, the management unit performs a process of obtaining estimation information including position information of the parking frame based on the movement information of the vehicle and the management information. , A configuration (second configuration) that is performed based on the estimation information and the observation information.

  In the image processing apparatus having the second configuration, the management unit may classify the lane marking in which the position information is included in both the observation information and the estimation information into a first update type, and position the lane marking only in the observation information. It is preferable that the lane marking including information is classified into a second update type, and the lane marking including positional information only in the estimation information is classified into a third update type (third configuration).

  In the image processing apparatus having the third configuration, when updating the management information, the position information of the lane marking of the first update type is updated to the value of the observation information, and the lane marking of the second update type is updated. Is temporarily registered with the value of the observation information, and the position information of the lane marking of the third update type is a correction value obtained based on the value of the estimation information based on the observation information and the estimation information. It is preferable to adopt a configuration (a fourth configuration) in which the value is updated to the value corrected in (4).

  In the image processing apparatus having the fourth configuration, the management unit may collectively correct the values of the estimation information including the position information of the lane markings of the plurality of third update types with the same correction value. (Fifth configuration) is preferable.

  In the image processing device according to any one of the first to fifth configurations, the camera is a side camera provided on a side of the vehicle, and the recognition unit uses a partial area of the captured image. It is preferable that the parking frame is recognized (sixth configuration).

  In order to achieve the above object, an image processing method according to the present invention is an image processing method executed by an image processing apparatus, which recognizes a parking frame based on a captured image obtained by a camera that captures a periphery of a vehicle. And a management step of updating management information including position information of the parking frame recognized in the past based on observation information including position information of the parking frame obtained from a current captured image. The process includes a process of classifying an update type for each lane line constituting the parking frame to be managed based on the management information and the observation information, and updating the management information based on the classification (the second process). 7 configuration).

  ADVANTAGE OF THE INVENTION According to this invention, the reliability of the information regarding the parking frame obtained based on the image picked up by the camera can be improved.

The figure which shows an example of 1 structure of a parking assistance system Schematic diagram for explaining the search area of the lane marking Table showing examples of update types of lane markings constituting parking frames Flowchart illustrating management processing Schematic diagram for explaining management processing 4 is a flowchart showing a specific example of the update process shown in FIG. Schematic diagram showing a comparison between the current recognition image of the lane markings and the past recognition image Schematic diagram for explaining a method of calculating a correction value

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following description, the direction in which the vehicle travels straight, that is, the direction from the driver's seat to the steering wheel is referred to as “forward”. The direction in which the vehicle travels straight, that is, the direction from the steering wheel to the driver's seat is referred to as “rearward direction”. Further, a direction from the right side to the left side of the driver facing the forward direction, which is a direction perpendicular to the straight traveling direction of the vehicle and the vertical line, is referred to as a “left direction”. Further, a direction that is perpendicular to the straight traveling direction of the vehicle and the vertical line and that is directed from the left side to the right side of the driver who is facing forward is referred to as “right direction”. A vehicle equipped with the parking assistance system including the image processing device of the present invention is referred to as a “own vehicle”.

<1. Parking assistance system>
The image processing device of the present invention can be applied to, for example, a parking support system. FIG. 1 is a diagram illustrating a configuration example of a parking assistance system to which the image processing device 1 of the present invention is applied. As shown in FIG. 1, the parking assistance system includes an image processing device 1, a photographing unit 2, a parking control ECU (Electronic Control Unit) 3, an EPS (Electric Power Steering) -ECU 4, and a vehicle-mounted network 5. Prepare.

  The image processing device 1 is connected to the photographing unit 2 and connected to the parking control ECU 3 and the EPS-ECU 4 via a vehicle-mounted network 5 such as a CAN (Controller Area Network). The image processing apparatus 1 performs recognition of a parking frame and management of the recognized parking frame based on an image captured by the image capturing unit 2. That is, the image processing device 1 functions as a parking frame management device. In addition, the image processing apparatus 1 calculates the target parking position, for example, as a place where the empty parking frame closest to the host vehicle should be parked when the host vehicle stops. Details of the image processing apparatus 1 will be described later.

  Note that a display device such as a liquid crystal display may be connected to the image processing device 1. The image processing device 1 may perform a process of generating a display image in which an index indicating the target parking position is superimposed on the captured image and displaying the display image on the display device.

  The photographing unit 2 includes a camera that photographs the periphery of the vehicle. In this embodiment, the camera has a fisheye lens. The photographing unit 2 includes a side camera 20 provided on a side of the own vehicle. The side camera 20 is included in a camera that captures the surroundings of the host vehicle, and is provided on, for example, a door mirror of the host vehicle. The side camera 20 preferably includes a left side camera and a right side camera. The left side camera has an optical axis extending in the left-right direction of the own vehicle, and captures the left direction of the own vehicle. The right side camera has an optical axis extending in the left-right direction of the own vehicle, and captures the right direction of the own vehicle. It is preferable that the photographing unit 2 includes, in addition to the side camera 20, a front camera that photographs the front of the own vehicle and a back camera that photographs the rear of the own vehicle.

  The parking control ECU 3 calculates a steering amount based on the target parking position received from the image processing device 1 and an output of a clearance sonar sensor (not shown), and transmits information on the steering amount to the EPS-ECU 4.

  The EPS-ECU 4 performs automatic steering at the time of the parking operation of the own vehicle based on the information on the steering amount received from the parking control ECU 3. In the present embodiment, the driver operates the accelerator operation and the brake operation.

<2. Image processing device>
As shown in FIG. 1, the image processing apparatus 1 includes a control unit 11 and a storage unit 12. The control unit 11 controls the entire image processing apparatus 1. The control unit 11 is configured by, for example, a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The storage unit 12 stores computer programs and data necessary for the operation of the control unit 11 in a nonvolatile manner. As the storage unit 12, for example, an electrically erasable programmable read-only memory (EEPROM), a flash memory, or the like can be used.

  The video acquisition unit 111, the search unit 112, the recognition unit 113, and the management unit 114 illustrated in FIG. 1 are functions of the control unit 11 realized by executing a computer program stored in the storage unit 12. In other words, the image processing device 1 includes the video acquisition unit 111, the search unit 112, the recognition unit 113, and the management unit 114.

  At least one of the functional units 111 to 114 included in the control unit 11 may be configured by hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The function units 111 to 114 included in the control unit 11 are conceptual components. The functions performed by one component may be distributed to a plurality of components, or the functions of the plurality of components may be integrated into one component.

  The video acquisition unit 111 acquires an analog or digital captured image from each camera included in the imaging unit 2 including the side camera 20 continuously in a predetermined cycle (for example, 1/30 second cycle). If the acquired temporally continuous captured images (acquired videos) are analog, the video acquisition unit 111 converts the analog captured images into digital captured images (A / D conversion).

  The search unit 112 searches for a lane marking that forms a parking frame based on the captured image of the side camera 20 output from the video acquisition unit 111. When the side camera 20 includes the left side camera and the right side camera, the search unit 112 searches for a lane marking on an image captured by each camera.

  The search unit 112 starts searching for a lane marking, for example, when the vehicle speed of the own vehicle becomes equal to or lower than a predetermined speed. In the present embodiment, the search unit 112 detects a lane marking from the image captured by the side camera 20 at a period of, for example, 100 ms using image processing such as edge extraction. The lane markings are drawn on the pavement of the parking lot with white lines, yellow lines, or the like.

  FIG. 2 is a schematic diagram for explaining a search area for a lane marking. FIG. 2 shows a state in which the host vehicle V has entered the parking lot and a plurality of lane markings SL exist on the left side of the host vehicle V. Each partition line SL extends in the left-right direction. That is, each division line SL extends in a direction parallel to the optical axis of the left side camera 20L. The plurality of division lines SL are arranged at predetermined intervals in the front-rear direction. In FIG. 2, a region R surrounded by a broken line schematically shows a photographing region of the left side camera 20L of the host vehicle V.

  The resolution of the image captured by the left side camera 20L is high in the front direction along the optical axis of the left side camera 20L, and the resolution decreases as the distance from the optical axis in the front-back direction decreases due to the influence of lens distortion and the like. For this reason, as shown by the graph in FIG. 2, when the lane markings SL are detected from the entire region of the image captured by the left side camera 20L, the information of the lane markings SL detected from the front has high reliability, The reliability of the information of the division line SL detected from a position away from the optical axis in the front-rear direction is low. Such a tendency also applies to the right side camera. That is, in the configuration in which the lane markings SL are detected from the entire region of the image captured by the side camera 20, information on the lane markings SL with high reliability and information on the lane markings SL with low reliability may be mixed.

  In consideration of this point, in the present embodiment, the search unit 112 searches for the lane markings SL in the image captured by the side camera 20 in a region where the lane marking information with high reliability can be obtained. It has a configuration. The search unit 112 searches for a lane marking SL using a narrow area centered on the front direction along the optical axis of the side camera 20 in the captured image of the side camera 20 as a search area. The search area has, for example, an area in which about three lane markings SL configuring the parking frame P can enter.

  Returning to FIG. 1, the recognition unit 113 recognizes the parking frame P based on the information about the lane marking SL detected by the search unit 112. In other words, the recognition unit 113 recognizes the parking frame P based on the captured image obtained by the side camera 20. The recognition unit 113 calculates the parallelism between the two lane markings SL, the width between the lane markings SL, and the like, and recognizes the parking frame based on these results. For example, the recognition unit 113 recognizes, as a parking frame P, an area between two lane markings SL arranged in parallel at a predetermined interval. The predetermined interval is, for example, the width of a standard parking area for general public defined by laws and regulations on parking lots.

  In the present embodiment, as described above, the search unit 112 searches for the lane markings SL by narrowing down, in the image captured by the side camera 20, a narrow area having high image reliability. For this reason, the recognition unit 113 recognizes the parking frame P using a partial area including the center of the image captured by the side camera 20. More specifically, the recognition unit 113 recognizes the parking frame P using an image of a narrow area centered on the front direction along the optical axis of the side camera 20 among the images captured by the side camera 20. The narrow region may be, for example, a region having a size that allows about one to two parking frames P. In the present embodiment, the narrow region is a region having a size that allows about two parking frames P. According to this, the recognizing unit 113 can recognize the parking frame P by narrowing down to the lane marking SL obtained from a region having high image reliability in the captured image. For this reason, the reliability of the parking frame P recognized by the recognition unit 113 can be improved.

  The management unit 114 manages information on the parking frame P recognized by the recognition unit 113 as management information 121 in a time-series manner. The management information 121 is stored in the storage unit 12 and is updated as appropriate. Note that the management unit 114 does not need to manage all of the parking frames recognized by the recognition unit 113 in the past. For example, the management unit 114 manages only a plurality of parking frames P that have been recognized most recently in time. The information of P may be appropriately excluded from the management target. As a result, the storage capacity required for the storage unit 12 can be reduced.

  In the present embodiment, the management unit 114 converts the management information 121 including the position information of the parking frame P recognized in the past into the observation information including the position information of the parking frame P obtained from the current captured image of the side camera 20. Update based on Note that the position information of the parking frame P includes the position information of the lane marking SL that forms the parking frame P. The management information 121 has position information on the parking frame P and the lane marking SL, for example, as a combination of a plurality of vertex coordinates. Each parking frame P may be managed as a combination of the vertex coordinates of the lane markings SL constituting the parking frame P.

  Specifically, based on the management information 121 and the observation information, the management unit 114 classifies the update type for each lane line SL configuring the parking frame P to be managed. The management unit 114 updates the management information 121 based on the classification of the update type. According to this, the updating process can be performed on each of the lane markings SL configuring the parking frame P to be managed by an appropriate method according to the classification, and the reliability of the management information 121 can be improved. .

  More specifically, the management unit 114 performs a process of obtaining estimated information including the position information of the parking frame P based on the movement information of the host vehicle V and the management information 121. The estimation information is for estimating to which position the position of each parking frame P included in the management information 121 has moved due to the movement of the host vehicle V. The estimation information includes information on the lane marking SL that forms the parking frame P. The movement information of the own vehicle V is, for example, information such as a steering angle, a vehicle speed, and a shift of the own vehicle V acquired from the on-vehicle network 5. In this case, the management unit 114 estimates the movement amount of the own vehicle V from the movement information of the own vehicle V, and estimates the current position of each parking frame P recognized in the past based on the estimated movement amount and the management information 121. I do. When the management information 121 includes information on a plurality of parking frames P, the management unit 114 estimates the current position of each of the plurality of parking frames P.

  The movement information of the host vehicle V is not limited to the measurement information such as the steering angle and the vehicle speed, but may be information obtained by processing the measurement information such as the estimated moving amount of the host vehicle V. In this case, the estimated moving amount of the vehicle V may be obtained by, for example, the parking control ECU 3. Then, the management unit 114 estimates the current position of the parking frame P based on the estimated movement amount of the vehicle V acquired from the parking control ECU 3 and the information on the parking frame P stored in the past as the management information 121. May do it.

  In the present embodiment, the classification of the update type of the lane marking SL constituting the parking frame P is performed based on the estimation information and the observation information. According to this, the management information 121 can be updated by giving priority to the result of the observation information with high reliability, and the reliability of the management information 121 can be improved.

  FIG. 3 is a table showing an example of the update types of the lane markings SL constituting the parking frame P. In FIG. 3, an observation value of “○” corresponds to a lane marking SL in which lane marking information exists in the observation information. In FIG. 3, the observation value “x” corresponds to a lane marking SL in which lane marking information does not exist in the observation information. In FIG. 3, the estimated value “○” corresponds to the lane marking SL in which the lane marking information exists in the estimation information. In FIG. 3, the estimated value “x” corresponds to the lane marking SL in which the lane marking information does not exist in the estimation information.

  A lane marking SL in which lane marking information exists in the observation information and the estimation information becomes an observation value “○” and an estimation value “○”, and is classified into the first update type. A lane marking SL in which lane marking information exists in the observation information and in which the lane marking information does not exist in the estimation information has an observation value of “○” and an estimated value of “×”, and is classified into the second update type. A lane marking SL in which lane marking information does not exist in the observation information and lane marking information exists in the estimation information becomes an observation value "x" and an estimation value "o", and is classified into the third update type. . Note that a lane marking SL in which lane marking information does not exist in the observation information and the estimation information cannot be processed, and is not given an update type.

  That is, the management unit 114 classifies the lane marking SL in which the position information is included in both the observation information and the estimation information into the first update type, and classifies the lane marking SL in which the position information is included only in the observation information into the second update type. And the lane marking SL in which the position information is included only in the estimation information is classified into the third update type. According to this, the position information of the plurality of lane markings SL acquired in a time series can be appropriately classified, and the management information 121 of the parking frame P can be appropriately updated.

  In the present embodiment, the management unit 114 updates the management information 121 based on these three update types. FIG. 4 is a flowchart illustrating a management process performed by the management unit 114. In the description of the management process shown in FIG. 4, the description will be given with reference to the schematic example shown in FIG.

  FIG. 5 is a schematic diagram for explaining the management process. The vehicle V that has entered the parking lot passes through the lane markings SL1 to SL10 in the order of the first lane marking SL1, the second lane marking SL2, the third lane marking SL3, and so on. As the host vehicle V moves forward, the recognition unit 113 sequentially recognizes the lane markings SL1 to SL10 as lane markings constituting the parking frame P based on the captured images sequentially acquired from the left side camera 20L.

  In the example illustrated in FIG. 5, the first division line SL1 to the ninth division line SL9 are already recognized as the division lines SL configuring the parking frame P by the recognition unit 113, and these division line information is included in the management information 121. Have been. That is, the management unit 114 can obtain estimation information for the first to ninth lane markings SL1 to SL9. Further, the eighth to tenth lane markings SL8 to SL10 are recognized as lane markings constituting the parking frame P from the current captured image of the left side camera 20L.

  The management process (management process) by the management unit 114 is started, for example, when the recognition unit 113 completes the process of recognizing the parking frame P for the current captured image. First, the management unit 114 acquires observation information (step S1). In the example illustrated in FIG. 5, the management unit 114 acquires information on the eighth to tenth division lines SL8 to SL10 in step S1.

  Next, the management unit 114 acquires estimation information (step S2). The management unit 114 calculates an estimated movement amount of the host vehicle V from the time when the immediately preceding captured image is acquired to the time when the current captured image is acquired. Then, the management unit 114 obtains estimation information based on the estimated movement amount and the management information 121 currently stored in the storage unit 12 at the present time. In the example illustrated in FIG. 5, the management unit 114 calculates the estimated positions of the first to ninth division lines SL1 to SL9 in step S2. Step S2 may be performed before step S1, or may be performed in parallel with step S1.

  When acquiring the observation information and the estimation information, the management unit 114 classifies the update type for each lane marking SL (step S3). Specifically, the management unit 114 determines which of the first update type, the second update type, and the third update type each lane line SL belongs to.

  In the example shown in FIG. 5, the eighth lane marking SL8 and the ninth lane marking SL9 correspond to lane markings where lane marking information exists in the observation information and the estimated information. Therefore, the eighth division line SL8 and the ninth division line SL9 are classified into the first update type. The tenth lane marking SL10 corresponds to a lane marking having lane marking information in the observation information and no lane marking information in the estimation information. For this reason, the tenth division line SL10 is classified into the second update type. The first to seventh lane markings SL1 to SL7 correspond to lane markings where lane marking information does not exist in the observation information and lane marking information exists in the estimation information. Therefore, the first to seventh division lines SL1 to SL7 are classified into the third update type.

  The management unit 114 determines whether each of the lane markings SL included in the observation information is the same as any of the lane markings SL recognized in the past, based on the position obtained from the observation information and the position obtained from the estimation information. Is determined by comparing In the present embodiment, even if the position obtained from the observation information and the position obtained from the estimation information do not completely match, if the difference between the positions is within a predetermined threshold value, it is determined that the same division line SL exists. to decide.

  After classifying the update type of each lane marking SL, the management unit 114 updates the management information 121 currently stored in the storage unit 12 based on the classification result (step S4). When the management information 121 is updated, the position information of the lane marking SL of the first update type is updated to the value of the observation information (observation update). The position information of the lane marking SL of the second update type is provisionally registered with the value of the observation information (new registration). Note that, at the stage of temporary registration, the lane marking SL is not treated as a lane marking SL recognized in the past in the classification of the update type, and is not used for calculating a correction value described later. If the temporary registration is performed a plurality of times, the demarcation line SL becomes the main registration. Thereby, the reliability of the management information 121 can be further improved. However, a configuration may be adopted in which the actual registration is performed without the temporary registration in the first observation. In this case, the first update type and the second update type are not divided into two, and may be the same classification. Further, the position information of the division line SL of the third update type is updated to a value obtained by correcting the value of the estimated information with a correction value obtained based on the observation information and the estimated information.

  In the example illustrated in FIG. 5, the management unit 114 updates the position information of the eighth division line SL8 and the ninth division line SL9 to the value of the observation information. The management unit 114 temporarily registers the position information of the tenth lane marking SL10 with the value of the observation information. The management unit 114 updates the position information of the first to seventh seven division lines SL1 to SL7 with a value obtained by correcting each estimated value with a correction value. With these updates, the update of the management information 121 including the information on the parking frame P is completed.

  In the present embodiment, since the observation information is obtained by using the region of the captured image with high reliability, the reliability of the lane line information included in the observation information can be increased. That is, it is possible to improve the reliability of the update information of the lane markings of the first update type and the second update type updated with the observation value. Then, the position information of the lane marking SL of the third update type, which cannot be recognized by the photographed image and needs to be updated with the result estimated from the moving amount of the vehicle V, is corrected by the correction value obtained from the observation information. Is performed to obtain update information. For this reason, it is possible to suppress the deviation of the vehicle from the original position due to accumulation of errors in the estimation of the movement amount of the vehicle during the update process for the lane marking of the third update type. That is, according to the present embodiment, the reliability of the management information 121 of the parking frame P can be improved.

  A specific example of the update process (step S4) shown in FIG. 4 will be described. FIG. 6 is a flowchart illustrating a specific example of the update process illustrated in FIG. After classifying the update type of each lane marking SL, the management unit 114 checks whether or not the first update type exists in the classified update types (step S41). In the example shown in FIG. 5, since the two partition lines SL8 and SL9 are classified into the first update type, the management unit 114 determines that the first update type exists.

  When determining that the first update type exists (Step S41 Yes), the management unit 114 calculates a correction value (Step S42). The correction value is used to correct the estimation information. A method of calculating the correction value will be described with reference to FIGS.

  FIG. 7 is a schematic diagram showing a comparison between the current recognition image I (t) of the lane marking SL and the past recognition image I (t-1). In FIG. 7, the solid line is the current recognition image I (t), and the broken line is the past recognition image I (t-1). The current recognition image I (t) is obtained from an image captured by the left side camera 20L of the own vehicle V (t), and the past recognition image I (t-1) is obtained from the own vehicle V (t- This is obtained from the image captured by the left side camera 20L in 1). The past in FIG. 7 refers to a time one cycle (camera shooting cycle) before the present. As shown in FIG. 7, the position of the lane marking SL recognized by the captured image is also moving due to the movement of the host vehicle V. In addition, the type of the demarcation line SL recognized by the captured image due to the movement of the host vehicle V also changes. That is, in the past recognition image I (t-1), the seventh to ninth three division lines SL7 to SL9 are recognized, but in the current recognition image I (t), the eighth to tenth division lines SL (s) are recognized. Three partition lines SL8 to SL10 are recognized.

  FIG. 8 is a schematic diagram for explaining a method of calculating a correction value. FIG. 8 is a diagram in which only the current recognition image I (t) in FIG. 7 is extracted, and items necessary for explaining the method of calculating the correction value are added. When calculating the correction value, the management unit 114 compares the lane marking information included in the observation information with the lane marking information included in the estimation information.

  This will be specifically described with reference to an example shown in FIG. The management unit 114 determines the current (t) position and the estimated position (position indicated by the one-dot chain line) estimated from the current (t) position and the position one cycle before (t-1) with respect to the eighth division line SL8 and the ninth division line SL9. Are compared to calculate a position difference value Δ. The management unit 14 uses the difference value Δ as a correction value.

  It should be noted that the difference value Δ obtained using the eighth division line SL8 may or may not be the same as the difference value Δ obtained using the ninth division line SL9. When a plurality of difference values Δ having different values are obtained, for example, an average value of the plurality of difference values may be used as the correction value. As another method, of the plurality of difference values Δ, the difference value Δ determined to be highly reliable may be used as the correction value. For example, it is estimated that the lane marking information recognized at the center of the captured image has higher reliability than the lane marking information recognized at a position shifted from the center. For this reason, the lane marking information recognized from a position closer to the center position of the captured image may be regarded as having high reliability, and the difference value Δ obtained from the lane marking information may be used as the correction value. In this example, the number of difference values to be obtained is two, but this number may be changed as appropriate, and may be one, or may be three or more.

  After calculating the correction value, the management unit 114 performs a correction process (step S43). Specifically, the management unit 114 sets the value of the estimated information to the value obtained by correcting the position information of the partition line SL of the third update type with the correction value. In the example illustrated in FIG. 8, it is determined that the estimated movement amount M of the host vehicle V is estimated to be smaller than the actual amount, and correction to increase the estimated movement amount M is performed.

  In the present embodiment, the management unit 114 collectively corrects the values of the estimation information including the position information of the plurality of third update type lane markings SL with the same correction value. In the example illustrated in FIG. 5, the management unit 114 sets the values of the pieces of estimated information to the values obtained by correcting the values of the pieces of estimated information collectively with the same correction value for the first to seventh partition lines SL1 to SL7. According to this, it is possible to efficiently update the partition line SL of the third update type for which the prediction update is performed, by correcting it to a value with high reliability.

  After performing the correction process, the management unit 114 updates each lane marking information to an update value determined according to each update type, and performs update registration of the management information 121 on the parking frame P (step S44). The method of obtaining the update value determined according to each update type has been described above, and will not be described here.

  If it is determined in step S41 that there is no lane marking SL of the first update type, the correction value cannot be calculated based on the observation information. Therefore, when it is determined that the lane marking SL of the first update type does not exist (No in Step S41), the management unit 114 sets the correction value to zero and proceeds to Step S43. In other words, when it is determined that the lane marking SL of the first update type does not exist, the management unit 114 does not substantially perform the correction processing of the estimation information.

  It should be noted that the state in which the lane marking SL of the first update type does not exist is a case where there is no recognition information of the lane marking SL in the observation information, and only the newly recognized lane marking information exists in the observation information. There are cases. The former is, for example, when the host vehicle V passes through the area where the parking frame P exists, or when an object (stationary or moving object) exists between the lane marking SL and the side camera 20 or when the lane marking SL is For example, the case where the division line SL cannot be recognized due to being covered with a shadow or the division line SL is disappearing or the like can be cited. The latter is, for example, when the own vehicle V has just started to approach the area where the parking frame P exists, when an object exists between the lane marking SL and the side camera 20, or when the lane marking SL becomes a shadow. The case where the parting line SL cannot be recognized because the parting line SL is covered or the parting line SL is about to disappear, and the like can be cited.

  Further, even when it is determined that the division line SL of the first update type exists, a state may occur in which the division line SL to be recognized in the current captured image is not recognized due to the presence of an object or a shadow, for example. However, such a lane marking SL can be updated and registered with a value obtained by correcting the estimated value obtained from the estimation of the moving amount of the vehicle with a correction value due to the presence of the lane marking SL of the first update type.

<3. Notes>
Various technical features disclosed in the present specification can be variously modified without departing from the spirit of the technical creation in addition to the above-described embodiment. That is, the above-described embodiment is to be considered in all respects as illustrative and not restrictive. The technical scope of the present invention is defined not by the description of the embodiment but by the claims. It is to be understood that all changes that fall within the meaning and scope equivalent to the appended claims are included. In addition, a plurality of embodiments and modifications shown in this specification may be implemented in combination within a possible range.

  In the above, the case where the recognition unit 113 recognizes the parking frame P using the side camera 20 has been described as an example. However, in the present invention, the recognition unit 113 uses a camera other than the side camera 20 (for example, a back camera). The present invention can also be applied to a configuration in which a parking frame is recognized by using the method.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 20 ... Side camera (an example of a camera)
113: recognition unit 114: management unit 121: management information P: parking frame SL: division line V: own vehicle (vehicle)

Claims (7)

A recognition unit that recognizes a parking frame based on a captured image obtained by a camera that captures the periphery of the vehicle,
A management unit that updates management information including position information of the parking frame recognized in the past based on observation information including position information of the parking frame obtained from a current captured image,
With
An image processing apparatus that, based on the management information and the observation information, classifies an update type for each lane line configuring the parking frame to be managed, and updates the management information based on the classification; . The management unit performs a process of obtaining estimated information including position information of the parking frame based on the movement information of the vehicle and the management information,
The image processing apparatus according to claim 1, wherein the classification of the update type is performed based on the estimation information and the observation information. The management unit,
Classifying the lane markings in which position information is included in both the observation information and the estimation information into a first update type,
Classifying the lane markings whose location information is included only in the observation information into a second update type,
The image processing apparatus according to claim 2, wherein the lane markings in which only the estimation information includes position information are classified into a third update type. When updating the management information,
The position information of the lane marking of the first update type is updated to a value of the observation information,
The position information of the lane marking of the second update type is provisionally registered with the value of the observation information,
The position information of the lane marking of the third update type is updated to a value obtained by correcting the value of the estimation information with a correction value obtained based on the observation information and the estimation information. Image processing device.   The image processing apparatus according to claim 4, wherein the management unit corrects the values of the estimation information including the position information of the lane markings of the plurality of third update types collectively with the same correction value. The camera is a side camera provided on a side of the vehicle,
The image processing device according to any one of claims 1 to 5, wherein the recognition unit recognizes the parking frame using a partial area including a central part of the captured image. An image processing method executed by the image processing device,
A recognition step of recognizing a parking frame based on a captured image obtained by a camera that captures the periphery of the vehicle,
A management step of updating management information including position information of the parking frame recognized in the past based on observation information including position information of the parking frame obtained from a current captured image,
With
The management step includes, based on the management information and the observation information, classifying an update type for each lane line constituting the parking frame to be managed, and updating the management information based on the classification. Image processing method.

Images