KR102478458B1 - Apparatus and method for rear camera image processing - Google Patents

Abstract

A method in which a rear camera image processing apparatus operated by at least one processor processes an original rear image collected by a rear camera. When the rear camera collects the original rear image, the original rear image is transmitted through a first path and a second path. After each processing, the output image generated through the first path is transferred to the display device, and the reduced rearward image generated through the second path is temporarily stored. The validity of the output image generated through the first path is determined, and if the output image is not valid, a temporarily stored reduced rear image is generated as a rear output image and transmitted to the display device.

Description

Apparatus and method for rear camera image processing}

The present invention relates to a rear camera image processing apparatus and method, and in order to support a vehicle S-RVC (Safety Rear View Camera) function, a safety mechanism is applied to a distortion correction function to distort an image collected by a rear view camera. It is a technology for a system and image processing method that safely corrects

A rear view camera (RVC) supports the driver to safely reverse or park by showing the driver the situation around the vehicle at the rear of the vehicle. Due to its importance, components that process the images collected by the rearview camera must have safety mechanisms applied. Safety mechanisms are defined in the ISO 26262 guidelines, which are automotive functional safety standards, and safety mechanisms must operate properly during unit tests and integration tests of vehicle software or hardware tests.

The safety mechanism applied to the rearview camera is also called data validation. In addition, the safety mechanism function should be applied to the components that process the images collected by the rear camera.

Currently, rear cameras have safety mechanisms applied to ISP (Image Signal Processor), Dewarp, S-RVC, and Open Low-Voltage Differential Signaling Display Interface (OpenLDI). However, a safety mechanism is not applied to a main bus through which images or signals are transmitted/received and a DRAM in which images collected by a rear camera are stored.

In order to apply a safety mechanism, it is common to add an end-to-end error correction code function to the main bus. In this case, safety mechanisms must be applied to all other direct memory access (DMA) or subsystems connected to the main bus. Therefore, there is a difficulty in inserting functions for applying a safety mechanism into a chip processing an image collected by a rear camera.

In addition, in the case of DRAM, when reading/writing data, an error correction code bit (ECC-bit) must be placed to determine validity of data. However, since it is difficult to apply a safety mechanism to the main bus, adding this function only to DRAM does not have a significant effect on data validation.

Therefore, if a problem occurs in the main bus or other components, the validity of DRAM data used by the rear camera cannot be guaranteed. As a result, a problem arises in that safety mechanism functions for all signal paths are not satisfied.

Due to this problem, a method of modifying the frame memory so that the rear camera can use it alone is also being studied. However, in order to store the 2048×1440 resolution images collected by the rear camera, the SRAM must have a capacity of about 12 MB, resulting in an increase in the size of the chip.

Therefore, the present invention applies a safety mechanism to the dewarp function, so that even if a problem occurs in the main path for processing the image collected by the rearview camera, the image collected by the rearview camera can be safely provided to the driver by the safety mechanism. , It provides a rear camera image processing device and method.

As a rear camera image processing device, which is one feature of the present invention for achieving the technical problem of the present invention,

An image signal processor generating a rear image by processing the original rear image collected by the rear camera, and a first processor generating a distortion correction image by correcting distortion of the rear image, and generating an output image that can be displayed through a display device. , The resolution of the rear image is reduced to generate a reduced rear image and temporarily stored, and when an operation control signal is received, the distortion of the temporarily stored reduced rear image is corrected, and then the resolution is restored to the resolution of the rear image and output A second processing unit that generates a rear image, and transmits the output image generated by the first processing unit to the display device, determines the validity of the output image based on a setting signal transmitted from the outside during an initial operation, and an image processing unit generating the operation control signal so that the second processing unit operates when the output image is not valid.

The first processing unit receives a non-safe area dewarp for generating a distortion-corrected image by correcting distortion of the rear image, a memory for storing the distortion-corrected image, and a distortion-corrected image stored in the memory, and the display It may include a display controller that converts an output image that can be displayed through a device.

The first processing unit may further include an OpenLDI that receives any one of the output image and output rear image from the image additional processing unit and transmits the received image to the display device.

The display controller may transmit the output image to either the image additional processing unit or the OpenLDI based on gear state information received from the outside.

The second processing unit receives the rearward image, reduces the resolution of the rearward image, and generates the reduced rearward image by using a first resolution adjusting unit, an internal memory for temporarily storing the reduced rearward image, and the operation control signal. If received, a safety area dewarp for generating a reduction correction image by correcting distortion by receiving a reduced rear image temporarily stored in the internal memory, and restoring the resolution of the reduction correction image to the resolution of the rear image to restore the output rear image It may include a second resolution adjusting unit that generates an image.

The image addition processing unit may receive the output image and image information from the first processing unit, and determine validity of the output image by comparing the setting signal and the image information.

The setting signal may include an operation clock, a horizontal sync signal and a vertical sync signal of an image, and the image information may include horizontal sync information and vertical sync information of the output image.

As another feature of the present invention for achieving the technical problem of the present invention, a rear camera image processing apparatus operated by at least one processor processes an original rear image collected by a rear camera,

When the rear camera collects the original rear image, processing the original rear image into a first path and a second path, respectively, the output image generated through the first path is transmitted to a display device, and the second path temporarily storing the scaled-down corrected image generated through; determining the validity of the output image generated through the first path; if the output image is not valid, converting the temporarily stored scaled-down corrected image to a rear output image generating, and transmitting the rear output image to a display device.

Before the step of processing each of the first path and the second path, if a set signal is input from the outside, converting to an operation ready state, wherein the set signal is an operation clock, a horizontal sync signal of an image and a vertical sync. Signals may be included.

The step of processing each of the first path and the second path includes generating a rearward image by image signal processing of the original rearward image through the first path, and a distortion-corrected image by correcting distortion of the rearward image. and generating the distortion correction image in a memory, and generating the stored distortion correction image as the output image displayable on the display device.

The processing of the first path and the second path may include generating the reduced rear image by reducing a resolution of the rear image through the second path.

The generating of the rearward output image may include, when an operation control signal is input, correcting distortion of the reduced rearward image to generate a scaled-down corrected image, and restoring the resolution of the scaled-down corrected image to the resolution of the rearward image. and generating the rear output image.

According to the present invention, even if elements installed in the non-safety area cause malfunction, since the rear camera image can be processed in the safety area, image distortion of the rear camera can be safely corrected and provided to the driver.

In addition, by adjusting the resolution of the image and providing it, the memory for the dewarp function can be implemented in the safe area without affecting the overall chip size.

1A and 1B are exemplary diagrams generally providing images collected from a rear camera.
2 is a structural diagram of a rear camera image processing device according to an embodiment of the present invention.
3 is a flowchart of a rear camera image processing method according to an embodiment of the present invention.
4 is an exemplary diagram of a distortion-corrected rear camera image provided according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated.

Hereinafter, a rear camera image distortion correction apparatus and method according to an embodiment of the present invention will be described with reference to the drawings. Prior to describing an embodiment of the present invention, an example of providing an image collected by a general rear camera will be first described with reference to FIGS. 1A and 1B.

1A and 1B are exemplary diagrams generally providing images collected from a rear camera.

In a general situation, a path for processing an image collected by the rear camera 50 of a vehicle is as shown in FIG. 1A. At this time, the image processing device 10 is connected to the ISP 21 , the dewarp 22 , the S-RVC 23 , the OpenLDI 24 , and the display controller 31 through the bus 25 . Also, the image processing device 10 may be connected to the DRAM 40 through the main bus 32 .

The ISP (21) performs a function of processing and processing the image collected by the rear camera (50). That is, the ISP 21 receives an image obtained from the image sensor (CCD, CIS) of the rear camera 50, and basically processes luminance and color, focus, inversion, mosaic, and CIS (CMOS Image Sensor). ), video format, etc. are additionally performed. The functions of the ISP 21 are already known, and detailed descriptions are omitted in the embodiments of the present invention.

The dewarp 22 corrects distortion of images processed by the ISP 21 . In order for the dewarp 22 to correct distortion in an image, set values of distortion correction parameters are required, and these set values can be calculated by the ISP 21. At this time, the dewarp 22 reads the image stored in the DRAM 40 several times and corrects the distortion. The function of the dewarp 22 to correct distortion is already known, and a detailed description thereof is omitted in the embodiment of the present invention.

The S-RVC 23 basically receives the video signal transmitted from the display controller 31 and performs additional video processing, such as drawing a parking line on the received video signal or adding a warning message to the video signal. . Here, the video signal is a video signal converted by the display controller 31 so that the OpenLDI 24 can receive the distortion-corrected video and display it on the display device 60 .

In addition, when an error occurs in the display controller 31, the S-RVC 23 receives the distortion-corrected image from the dewarp 22, performs additional image processing, and transmits the image to the OpenLDI 24.

OpenLDI (24) is a universal digital video interface standard for high-speed data transmission, which provides high bit rates, low power, and improved noise control performance. According to the control of OpenLDI 24, an additional image processing image or a distortion-corrected image is transmitted to the display device 60. Here, the function of the OpenLDI 24 is already known, and a detailed description thereof is omitted in an embodiment of the present invention.

Here, since a safety mechanism is applied to the ISP 21, the dewarp 22, the S-RVC 23, and the OpenLDI 24, the area including these components is referred to as the safe area 20. And since the safety mechanism is not applied to the display controller 31 and the main bus 32, the area including these two components is referred to as a non-safety area 30.

As shown in FIG. 1A , the image collected by the rear camera 50 in a general situation is stored in the DRAM 40 via the ISP 21 , the dewarp 22 , and the main bus 32 . At this time, if a problem occurs in any one of the display controller 31, the main bus 32, or the DRAM 40 configured in the non-safe area 30, the path through which the image is transmitted as shown in FIG. 1B is changed differently from the transmission path in normal circumstances.

Image data transmitted from the rear camera 50 in the direction of arrows ⑤ and ⑥ is image-processed by the ISP 21 and converted into a YUV format. Image data of the changed YUV format is transferred to the dewarp 22 and distortion is corrected.

At this time, all of the image data processed by the ISP 21 is stored in the DRAM 40 implemented outside the image processing device 10 (⑤). In the dewarp 22, image distortion is corrected using image data stored in the DRAM 40. The distortion-corrected image is subjected to additional image processing in the S-RVC 23, and the image is output from the display device 60 through the OpenLDI 24.

At this time, if a problem occurs in the DRAM 40 or the main bus 32, the image stored in the DRAM 40 cannot be reloaded by the dewarp 22, so the rear camera 50 through the display device 60 will not be able to display the collected images.

Therefore, in the embodiment of the present invention, even if an error occurs in components located in a non-safe area, a technology that can safely display the image collected by the rear camera 50 on a display device and provide it to the driver is proposed. .

2 is a structural diagram of a rear camera image processing device according to an embodiment of the present invention.

As shown in FIG. 2 , the rear camera image processing device 100 processing the image collected by the rear camera 150 has a safety mechanism reflection area (hereinafter referred to as 'safe area' or 'first processing unit' for convenience of explanation). ') 110 and a safety mechanism non-reflecting area (hereinafter, referred to as 'non-safety area' or 'second processing unit' for convenience of description) 120, a component that processes the collected images of the rear camera. are implemented.

The safe area 110 includes an ISP 111, a safe area dewarp 112, a first resolution adjusting unit 113, an internal SRAM 114, a second resolution adjusting unit 115, an image addition processing unit 116, And OpenLDI (118) is located. Meanwhile, in the non-safe area 120, the non-safe area dewarp 119, the main bus 130, and the DRAM 140 are implemented.

The rear camera image processing device 100 also interworks with the rear camera 150 and the display device 160 . The rear camera image processing device 100 can be implemented as a single chip, and in an embodiment of the present invention, the shape of the implemented chip is not limited to one.

The rear image collected by the rear camera 150 is simultaneously processed through two paths in the rear camera image processing device 100 . However, an image processed in one of the two paths is displayed through the display device 160 .

Here, two routes can be defined as a normal route (⑦) and an emergency route (⑧). The general path is a path that processes a rear image through components implemented in the non-safe area and some components implemented in the safety area. The emergency path is a path that processes a rear image only with components implemented in the safety area due to a malfunction of components implemented in the non-safe area.

For convenience of explanation, components when an image is transferred through a normal path are first described, and components when an image is transferred through an emergency path are subsequently described. However, in reality, images are transmitted simultaneously to the normal route and the emergency route.

When the ISP 111 receives the original rear image obtained from the image sensor (CCD, CIS) of the rear camera 150, it generates a rear image by performing general image processing functions such as luminance, color, focus, and inversion of the rear image. do. The functions of the ISP 111 are already known, and detailed descriptions are omitted in the embodiments of the present invention.

The rear image generated by the ISP 111 is transferred to the non-safe area dewarp 119. The non-safe area dewarp 119 and the non-safe area dewarp 119 correct distortion of the rear image processed by the ISP 111 to generate a distortion corrected image. In order for the non-safe area dewarp 119 to correct distortion in an image, set values of distortion correction parameters are required, and these set values can be calculated by the ISP 111. This is a known technique, and detailed descriptions are omitted in the embodiments of the present invention.

The distortion correction image generated by the non-safe area dewarp 119 is stored in the DRAM 140 through the main bus 130 . In the embodiment of the present invention, it is shown that the distortion is corrected once in the non-safe area dewarp 119 and stored in the DRAM 140, but the distortion correction image stored in the DRAM 140 is dewarped in the non-safe area 119. ), the distortion can be corrected by repeating reading.

The display controller 117 reads the distortion correction image stored in the DRAM 140, converts the distortion correction image into a form that the OpenLDI 118 can process, and generates an output image. Then, image information including an output image and vertical/horizontal sync information for the output image is generated. A method for the display controller 117 to process a distortion correction image to generate an output image and a method for obtaining vertical sync and horizontal sync information for the output image are already known techniques, and detailed descriptions are omitted in the embodiments of the present invention. do.

The display controller 117 transfers image information to the image additional processing unit 116 or to the OpenLDI 118 based on vehicle state information transmitted from a vehicle main processor (not shown). Here, the vehicle state information includes whether the vehicle is currently in a driving state and gear state information operated by a driver for parking. is not limited to any one method.

The display controller 117 transmits image information to the OpenLDI 118 when the vehicle state information is a driving state. However, if the vehicle state information is a case where the gear is changed to attempt parking, the display controller 117 transmits the image information to the image addition processing unit 116 to provide additional images such as a parking line or a notification message to the output image. do.

The OpenLDI 118 receives an output image from the image additional processing unit 116 or the display controller 117 and transfers the output image to the display device 160 . OpenLDI 118 is a universal interface standard for high-speed data transmission, which increases the bit rate of output images, lowers power, and provides improved noise control performance.

On the other hand, if the components are described according to the order in which rearward images are delivered to the emergency path, the first resolution adjusting unit 113 receives the rearward image generated by the ISP 111 and adjusts the image resolution to form a reduced rearward image. generate In the embodiment of the present invention, the resolution of the rear image is described as 2048 * 1440 as an example, and the first resolution adjusting unit 113 reduces the resolution of the rear image by 1/4 to generate a resolution of 512 * 360 explain with an example.

At this time, the first resolution controller 113 may receive the distortion-corrected image in which the distortion is corrected in the safe area dewarp 112 and adjust the resolution. In the embodiment of the present invention, for convenience of description, an example of adjusting the resolution without correcting distortion in the rear image generated by the ISP 111 is described, but it is not necessarily limited thereto.

The internal SRAM 114 stores the reduced rear image generated by the first resolution adjusting unit 113 . In addition to the function of storing the reduced rear image, the internal SRAM 114 may determine the validity of the stored reduced rear image using the ECC function as an error correction code. Since the ECC function is already known, a detailed description thereof is omitted in an embodiment of the present invention. In addition, an example of performing the ECC function in the internal SRAM 114 will be described, but is not necessarily limited thereto.

When the safe region dewarp 112 receives an operation control signal from the image addition processing unit 116, it reads the reduced rear image stored in the internal SRAM 114, corrects distortion, and then generates a reduced correction image. As another embodiment, the safe area dewarp 112 may correct distortion in the rear image generated by the ISP 111 and then transfer the corrected distortion to the first resolution adjusting unit 113 .

The second resolution adjusting unit 115 receives the scaled-down corrected image generated by the safe area dewarp 112, restores the scaled-down resolution to the resolution of the rearward image, and generates an output rearward image. A method for restoring the resolution of the reduced image by the second resolution adjusting unit 115 is already known, and a detailed description thereof is omitted in the embodiment of the present invention.

The second resolution adjusting unit 115 may additionally generate image information about the output rear image. The image information may include an output rear image and vertical/horizontal sync information for the output rear image.

When receiving an output image from the display controller 117 or an output rear image from the second resolution adjusting unit 115, the image addition processing unit 116 adds a parking line to each image or adds a warning message to the image. etc., and performs image processing procedures to provide additional information. Since the method of processing the image to provide additional information to the image by the image additional processing unit 116 may be performed in various ways, the embodiment of the present invention is not limited to any one method.

Also, the image addition processing unit 116 checks the validity of the output image based on the image information included in each output image. In order to check validity from the output image, the image addition processing unit 116 receives a setting signal from a control device (not shown) that controls electric devices of the vehicle.

Here, the setting signal includes an operation clock of the image additional processing unit 116, a horizontal sync signal and a vertical sync signal for the video, and when the vehicle is started and the rear camera image processing device 100 is executed, the control device Receives the setting signal. Here, the operating clock is a reference clock used to determine the validity of the sync signal, and means a 24 MHz XIN clock in the embodiment of the present invention. The image addition processing unit 116 determines the validity of the output image based on the setting signal.

That is, when the image additional processing unit 116 receives the output image and the image information, the clock, horizontal sync information, and vertical sync information included in the image information are compared with the previously received setting signal, and the output video is input as much as the set value. determine whether there is or not. And it checks whether there is an error in the components installed in the non-safe area by checking whether only the still image is continuously being input, whether a monochrome image is being input, and whether the setting signal and other image information are being input.

If it is determined that there is an error, the additional image processing unit 116 attempts to convert the image receiving path from a normal path to an emergency path. In addition, an operation control signal is transmitted to the safe area dewarp 112 in order to receive the image whose resolution is adjusted by the second resolution controller 115 . Through this, even if an error occurs in the DRAM 140 implemented in the non-safe area, the main bus 130, etc., the rear camera image stored in the internal SRAM 114 can be transmitted to the display device 160.

A method of processing an image of a rear camera by the rear camera image processing apparatus 100 described above will be described with reference to FIG. 3 .

3 is a flowchart of a rear camera image processing method according to an embodiment of the present invention.

As shown in FIG. 3 , the rear camera image processing device 100 is initially driven (S100). At this time, all components constituting the rear camera image processing device 100 are changed to an operation ready state, and in particular, the image additional processing unit 116 receives a setting signal from the control device. Also, the image additional processing unit 116 is first set to a ready state for receiving an image through a general path.

When the rear camera 150 collects images (S101), the original rear images collected through the normal route and the emergency route are transmitted and the images are processed respectively (S102). At this time, the rear camera image processing apparatus 100 is set so that an output image processed through a general path is provided first.

Accordingly, the image addition processing unit 116 of the rear camera image processing device 100 determines the validity of the output image generated through the general path (S103). The image addition processing unit 116 checks whether the output image is valid based on the image information included in the output image and the setting signal received from the control device during initial driving in step S100 (S104).

If it is confirmed that the output image is valid, the rear camera image processing device 100 displays the output image processed through the general path through the display device 160 (S105).

On the other hand, if it is confirmed that the output image is invalid, the image additional processing unit 116 of the rear camera image processing device 100 generates a control signal to receive the rear output image processed as an emergency path (S106). According to the control signal, the second resolution adjusting unit 115 generates a reduced rear image stored in the internal SRAM 114 as an output rear image, and displays the generated output rear image through the display device 160 (S107).

4 is an exemplary diagram of a distortion-corrected rear camera image provided according to an embodiment of the present invention.

(a) of FIG. 4 is an original rear image collected by the rear camera 150. In addition, FIG. 4(b) is an output image processed in a normal path, and FIG. 4(c) is a rear output image processed in an emergency path.

Compared to the output image processed through the normal path in FIG. 4(b), the rear output image processed through the emergency path in FIG. 4(c) may have picture quality degradation due to resolution reduction/enlargement. However, even if a problem occurs in components implemented in the non-safety area of the rear image processing system 100, the driver can normally check the operation of the rear camera 150.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims are also included in the scope of the present invention. that fall within the scope of the right.

Claims (12)

As a rear camera image processing device,
An image signal processor for generating a rear image by processing the original rear image collected by the rear camera;
A first processing unit that corrects the distortion of the rear image to generate a distortion correction image and generates an output image that can be displayed through a display device;
The resolution of the rear image is reduced to generate a reduced rear image and temporarily stored, and when an operation control signal is received, the distortion of the temporarily stored reduced rear image is corrected and the resolution is restored to the resolution of the rear image to output the rear image. A second processing unit for generating an image, and
The output image generated by the first processing unit is transferred to the display device, the validity of the output image is determined based on a setting signal transmitted from the outside during an initial operation, and if the output image is not valid, the second processing Image supplementary processing unit for generating the operation control signal to operate the supplementary unit
Including, rear camera image processing device. According to claim 1,
The first processing unit,
A non-safe area dewarp for generating a distortion correction image by correcting the distortion of the rear image;
a memory for storing the distortion correction image; and
A display controller that receives the distortion correction image stored in the memory and converts it into an output image that can be displayed through the display device.
Including, rear camera image processing device. According to claim 2,
The first processing unit,
OpenLDI for receiving any one of the output image and the output rear image from the image additional processing unit and transferring the received image to the display device.
Further comprising a rear camera image processing device. According to claim 2,
The display controller,
A rear camera image processing device that transmits the output image to either the image additional processing unit or OpenLDI based on gear state information received from the outside. According to claim 4,
The second processing unit,
A first resolution adjusting unit for receiving the rear image and reducing the resolution of the rear image to generate the reduced rear image;
An internal memory for temporarily storing the reduced rear image,
When the operation control signal is received, a safe area dewarp for receiving a scaled back image temporarily stored in the internal memory and correcting distortion to generate a scaled-down corrected image; and
A second resolution adjusting unit for generating the output rear image by restoring the resolution of the downscaling corrected image to the resolution of the rear image.
Including, rear camera image processing device. According to claim 5,
The image additional processing unit,
The rear camera image processing apparatus for receiving the output image and image information from the first processing unit, and determining validity of the output image by comparing the setting signal and the image information. According to claim 6,
The setting signal includes an operation clock, a horizontal sync signal and a vertical sync signal of an image, and the image information includes horizontal sync information and vertical sync information of the output image. A method in which a rear camera image processing apparatus operated by at least one processor processes an original rear image collected by a rear camera,
When the rear camera collects the original rear image, processing the original rear image into a first path and a second path, respectively;
Transferring the output image generated through the first path to a display device and temporarily storing the scaled-down corrected image generated through the second path;
Determining validity of an output image generated through the first path;
If the output image is not valid, generating the temporarily stored reduction correction image as a backward output image; and
Transmitting the rear output image to a display device
Including, rear camera image processing method. According to claim 8,
Prior to the step of processing each of the first path and the second path,
Converting to an operation ready state when a setting signal is input from the outside
including,
The setting signal includes an operation clock, a horizontal sync signal and a vertical sync signal of an image, rear camera image processing method. According to claim 9,
The step of processing each of the first path and the second path,
Through the first path,
image signal processing of the original rear image to generate a rear image;
Correcting the distortion of the rear image to generate a distortion corrected image, and
Storing the distortion-corrected image in a memory, and generating the stored distortion-corrected image as the output image displayable on the display device.
Including, rear camera image processing method. According to claim 10,
The step of processing each of the first path and the second path,
Through the second path,
Generating the reduced rear image by reducing the resolution of the rear image
Including, rear camera image processing method. According to claim 11,
The step of generating the rear output image,
When an operation control signal is input, generating a reduced correction image by correcting distortion of the reduced rear image, and
Generating the backward output image by restoring the resolution of the downscaling correction image to the resolution of the backward image
Including, rear camera image processing method.

Images