TWI913008B - Method and device of synchronizing images applied to vehicles - Google Patents

Abstract

A method of synchronizing images applied to vehicles is disclosed, which includes: calculating a buffer frame number for each of multiple buffer queues in a memory of a vehicle based on multiple delay times between multiple camera circuits and a processor; controlling the multiple camera circuits through a transmission circuit to shoot around the vehicle in multiple shooting directions to generate multiple image streams, and controlling the transmission circuit to respectively transmit the multiple image streams to the multiple buffer queues in the memory; detecting a queue frame number of the image stream temporarily stored in each of the multiple buffer queues; determining whether to capture a top frame of the image stream temporarily stored in each of the multiple buffer queues based on the buffer frame numbers and the queue frame numbers, thereby combining all top frames into a combined image.

Description

Image synchronization method and device for vehicles

This disclosure relates to image processing techniques, and more particularly to image synchronization methods and devices applied to vehicles.

Most vehicles currently equipped with surveillance cameras. The most common setup involves mounting the cameras at the front or rear of the vehicle, allowing users to view the captured footage on a display inside the vehicle. While multiple traditional surveillance cameras can simultaneously capture road conditions in front and behind, these images are transmitted and stored separately to the vehicle's onboard system. The onboard system typically does not record the time each camera captured the image. Furthermore, when these surveillance cameras transmit images to the onboard system, there may be varying transmission delays between the cameras and the system. These different delays can cause the simultaneously captured images to be out of sync. Therefore, how to solve the image asynchrony caused by different transmission delays is a problem that technical personnel in this field urgently want to solve.

The main purpose of this disclosure is to provide an image synchronization method and device for use in vehicles, which can solve the problem of image asynchrony caused by different transmission delays.

To achieve the above objectives, the present disclosure discloses an image synchronization method for a vehicle, comprising: Step a) Detecting multiple delay times between multiple camera circuits and the processor via a transmission circuit on the vehicle using a processor on the vehicle; Step b) Calculating, using the processor, a buffer frame number for each of multiple buffer sequences in a memory on the vehicle based on the multiple delay times, wherein the multiple buffer sequences correspond to the multiple camera circuits respectively; Step c) The processor, via the transmission circuit, controls the multiple camera circuits to capture images of the area around the vehicle from multiple shooting directions to generate multiple image streams, and controls the transmission circuit to transmit the multiple image streams to the multiple buffer queues in the memory respectively; Step d) The processor, via the transmission circuit, detects the number of frames of one buffer queue of the image stream temporarily stored in each buffer queue; Step e) The processor, based on the number of buffer frames and the number of frames in each buffer queue, determines whether to retrieve the first frame of the temporarily stored image stream in each of the plurality of buffer queues, wherein the first frame is the first image frame of each image stream stored in the corresponding buffer queue; Step f) When it is determined that the first frame of the temporarily stored image stream in each of the plurality of buffer queues should be retrieved, the processor merges all the first frames into a single merged image; and If, in step g), it is determined that the first frame of the image stream temporarily stored in each of the buffer queues should not be retrieved, then steps c) through e) are executed again.

To achieve the above objectives, the image synchronization device for a vehicle disclosed herein includes: a plurality of camera circuits disposed on a vehicle and configured to capture images of the area surrounding the vehicle from multiple shooting directions; a transmission circuit connected to the plurality of camera circuits and configured to receive an image stream; a memory connected to the transmission circuit and configured to store a plurality of instructions and a plurality of buffer queues, wherein the plurality of buffer queues correspond to the plurality of camera circuits; a processor connected to the transmission circuit and the memory and configured to access the plurality of instructions to perform the following steps: Step a) Detect multiple delay times between the plurality of camera circuits and the processor via the transmission circuit; Step b) Calculate the number of buffered frames for each of the plurality of buffered columns on the vehicle based on the multiple delay times; Step c) Control the plurality of camera circuits via the transmission circuit to capture images of the area around the vehicle from multiple shooting directions to generate multiple image streams, and control the transmission circuit to transmit the multiple image streams to the plurality of buffered columns respectively; Step d) Detect the number of frames in a column of the image stream temporarily stored in each buffered column; Step e) Based on the number of buffer frames and the number of frames in each buffer queue, determine whether to retrieve the first frame of the temporarily stored image stream in each of the plurality of buffer queues, wherein the first frame is the first image frame of each image stream stored in the corresponding buffer queue; Step f) When it is determined that the first frame of the temporarily stored image stream in each of the plurality of buffer queues should be retrieved, merge all the first frames into a single merged image; and If, in step g), it is determined that the first frame of the image stream temporarily stored in each of the buffer queues should not be retrieved, then steps c) through e) are executed again.

Compared to previous technologies, this invention first detects the latency of the camera circuits around the shooting vehicle to set the number of buffer frames for each buffer sequence, and compares the number of frames in each buffer sequence with the number of buffer frames to determine when to retrieve the first frame of each buffer sequence for ambient image synchronization. In this way, this invention can solve the problem of image asynchrony caused by different transmission delays in each camera circuit.

Referring to FIG1, FIG1 illustrates a block diagram of an image synchronization device 100 applied to a vehicle according to some embodiments of the present invention. As shown in FIG1, the image synchronization device 100 applied to a vehicle (hereinafter referred to as the image synchronization device 100) includes multiple camera circuits 111-11n, a transmission circuit 120, a memory 130, and a processor 140, where n is a positive integer greater than 1. The transmission circuit 120 is connected to the camera circuits 111-11n, the memory 130, and the processor 140. The memory 130 is connected to the processor 140. In some embodiments, the camera circuits 111-11n are all wirelessly connected to the transmission circuit 120. It is worth noting that the wireless connection method will result in multiple transmission delays between the camera circuits 111~11n and the transmission circuit 120, and these transmission delays may be the same or different (e.g., 200 milliseconds, 150 milliseconds and 100 milliseconds).

In other embodiments, a portion of the camera circuits 111-11n is wirelessly connected to the transmission circuit 120, while another portion of the camera circuits 111-11n is wiredly connected to the transmission circuit 120. It is worth noting that the wireless connection results in several different transmission delays (e.g., 200 milliseconds and 100 milliseconds) between the portion of the camera circuits 111-11n and the transmission circuit 120, while the wired connection eliminates any transmission delay between the other portion of the camera circuits 111-11n and the transmission circuit 120 (i.e., the transmission delay is 0 seconds). It is worth noting that if camera circuits 111-11n are connected to transmission circuit 120 via wired connections to avoid transmission delays, it will incur significant wiring costs. Conversely, if camera circuits 111-11n are connected to transmission circuit 120 via wireless connections, although wiring costs can be greatly reduced, transmission delays will occur.

In this embodiment, camera circuits 111-11n are respectively disposed at the same or different positions on the vehicle and are used to capture images of the surroundings of the vehicle from multiple shooting directions. In some embodiments, camera circuits 111-11n correspond to multiple shooting directions. In other words, each camera circuit can capture images from the vehicle from the corresponding shooting direction to generate its own image stream (i.e., multiple consecutive images). In some embodiments, each camera circuit can be disposed at any position on the vehicle that can capture images of the surroundings of the vehicle. In some embodiments, the shooting direction can be any direction towards the surroundings of the vehicle. In some embodiments, the vehicle can be a vehicle, electric bicycle, motorcycle, or bicycle, etc. In some embodiments, each imaging circuit can be implemented by any image capture circuit (e.g., optical imaging circuit, infrared imaging circuit, or 3D imaging circuit).

The following describes the configuration of each camera circuit using practical examples. Referring also to Figure 2, Figure 2 illustrates the configuration of camera circuits 111-114 in some embodiments disclosed herein. As shown in Figure 2, assuming the image synchronization device 100 has four camera circuits 111-114, camera circuit 111 can be installed at the front edge of the vehicle VH, camera circuit 112 can be installed on one side door pillar of the vehicle VH, camera circuit 113 can be installed on the other side door pillar of the vehicle VH, and camera circuit 114 can be installed at the rear edge of the vehicle VH.

Furthermore, the camera circuits 111-114 each have multiple shooting ranges 210-240, and the shooting directions of the camera circuits 111-114 correspond to the shooting ranges 210-240, respectively. That is, the shooting direction of camera circuit 111 can be directly facing the shooting range 210 of camera circuit 111, the shooting direction of camera circuit 112 can be directly facing the shooting range 220 of camera circuit 112, the shooting direction of camera circuit 113 can be directly facing the shooting range 230 of camera circuit 113, and the shooting direction of camera circuit 114 can be directly facing the shooting range 240 of camera circuit 114.

Returning to Figure 1, in this embodiment, the transmission circuit 120 is used to receive image streams from each of the camera circuits 111-11n. In some embodiments, the transmission circuit 120 can be implemented by any communication circuit used for communication (e.g., a controller area network (CAN) communication circuit, a Wi-Fi communication circuit, or a Bluetooth communication circuit).

In this embodiment, memory 130 stores multiple buffer queues, each corresponding to a camera circuit 111 to 11n. For example, memory 130 may store n buffer queues, where the first buffer queue corresponds to camera circuit 111, the second buffer queue corresponds to camera circuit 112, the third buffer queue corresponds to camera circuit 11n, and so on. In some embodiments, the multiple buffer queues are all first-in-first-out (FIFO) temporary blocks in memory 130. In some embodiments, each buffer queue temporarily stores multiple consecutive frames from the image stream generated by the corresponding camera circuit. In other words, whenever the transmission circuit 120 sequentially receives multiple frames of the image stream from one of the camera circuits, the processor 140 can control the transmission circuit 120 to sequentially transmit these frames to the buffer queue corresponding to that camera circuit. In this way, the corresponding buffer queue can sequentially store these frames in a first-in-first-out manner.

The following describes the buffer array with a practical example. Referring also to Figure 3, Figure 3 illustrates a schematic diagram of buffer arrays 131-13n in some embodiments. As shown in Figure 3, assuming that buffer arrays 131-13n correspond to camera circuits 111-11n respectively, buffer arrays 131-13n can temporarily store multiple frames of multiple image streams img1-imgn generated by camera circuits 111-11n in a first-in-first-out manner.

Returning to Figure 1, in some embodiments, memory 130 stores multiple instructions, upon which processor 140 executes the detailed steps described in subsequent paragraphs. In some embodiments, memory 130 may be implemented using flash memory, read-only memory, a hard disk, or any equivalent storage device, but is not limited thereto. In some embodiments, processor 140 may be implemented using a central processing unit (CPU), a microcontroller unit (MCU), a programmable logic controller (PLC), a system-on-a-chip (SoC), or a field-programmable gate array (FPGA), but is not limited thereto. In some embodiments, memory 130 is also used to store merged images of subsequent segments.

In some embodiments, the image synchronization device 100 may further include a display 150 connected to the processor 140 for displaying merged images of subsequent segments. In some embodiments, the display 150 may be implemented using any display panel or display circuit (e.g., an LCD screen or a touch panel). In some embodiments, the transmission circuit 120, memory 130, processor 140, and display 150 may be located in an onboard system (e.g., a control area network) on a vehicle.

Referring also to Figure 4, which illustrates a flowchart of an image synchronization method for a vehicle in some embodiments, wherein the image synchronization method is applicable to the image synchronization device 100 shown in Figure 1.

As shown in Figure 4, the image synchronization method applied to the vehicle includes steps S410 to S460. First, in step S410, the processor 140 detects multiple delay times between the camera circuits 111-11n and the processor 140 via the transmission circuit 120. In some embodiments, each delay time indicates the time consumed by the corresponding camera circuits 111-11n to transmit the image stream to the processor 140.

The detection of delay time will be further explained below. Referring also to Figure 5, Figure 5 illustrates a flowchart of steps S411 to S413, which are included in step S410 of the image synchronization method for a vehicle disclosed in some embodiments. As shown in Figure 5, step S410 of Figure 4 includes steps S411 to S413.

In step S411, processor 140 controls transmission circuit 120 to transmit timestamps to camera circuits 111-11n. In some embodiments, processor 140 periodically generates a timestamp and simultaneously transmits this timestamp to camera circuits 111-11n, wherein the timestamp indicates the system time of the vehicle system to which the image synchronization device 100 is connected. In other words, processor 140 can periodically use timestamps to confirm the delay time corresponding to each camera circuit 111-11n.

In step S412, processor 140 controls transmission circuit 120 to receive multiple feedback messages from camera circuits 111 to 11n respectively. In some embodiments, when each camera circuit 111 to 11n receives a timestamp, each camera circuit sends the received timestamp back to transmission circuit 120 as a feedback message, so that processor 140 can receive the respective feedback messages from camera circuits 111 to 11n through transmission circuit 120.

In step S413, processor 140 calculates multiple time differences between the transmission time of the transmission timestamp and multiple reception times of receiving multiple feedback messages, as multiple delay times between camera circuits 111-11n and the processor. In some embodiments, processor 140 records the time of external transmission of timestamps as the transmission time. In some embodiments, processor 140 records the time of receiving feedback messages from each camera circuit as each reception time. In other words, processor 140 can calculate the time difference between the transmission time and each reception time as the delay time corresponding to each camera circuit.

Returning to Figure 4, in step S420, the processor 140 calculates the buffer frame count for each of the multiple buffer columns 131~13n in memory 130 based on the multiple delay times. In some embodiments, the buffer frame count for each buffer column indicates the number of frames that each buffer column needs to buffer (i.e., this buffer column needs to buffer this number of frames to achieve image synchronization).

The calculation of the buffer frame number will be explained further below. Referring also to Figure 6, Figure 6 illustrates a flowchart of steps S420, which includes multiple steps S421 to S422, in an image synchronization method for a vehicle disclosed in some embodiments. As shown in Figure 6, step S410 of Figure 4 includes steps S421 to S422.

In step S421, processor 140 selects the largest of the multiple delay times calculated in step S413 (i.e., the maximum delay time), and calculates multiple time differences between the multiple delay times and the maximum of the multiple delay times. In other words, processor 140 can first obtain the maximum delay time and calculate the maximum delay time and the time differences between each delay time. For example, processor 140 calculates a first time difference between a first delay time and a maximum delay time (corresponding to the first camera circuit), calculates a second time difference between a second delay time and a maximum delay time (corresponding to the second camera circuit), calculates an nth time difference between an nth delay time and a maximum delay time (corresponding to the nth camera circuit), and so on.

In step S422, processor 140 calculates multiple product values between multiple time differences and the frame rates of camera circuits 111-11n as multiple buffer frame numbers for buffer rows 131-13n. In some embodiments, the frame rates of camera circuits 111-11n can be exactly the same. In other words, processor 140 can calculate the product value between each time difference and this frame rate separately as the buffer frame number for each buffer row corresponding to each time difference. In detail, an image frame that enters a buffer queue must wait for a certain number of other image frames to enter the same buffer queue before it can be retrieved from that buffer queue. The specific number of waiting frames is called the buffer frame number, and the buffer frame number is inversely proportional to the delay time of the camera circuit.

For example, taking an image synchronization device 100 with three camera circuits 111-113 corresponding to three buffer arrays 131-133 as an example, assuming that the delay times of the three camera circuits 111-113 are 200 milliseconds, 100 milliseconds and 300 milliseconds respectively, and the frame rate of the three camera circuits 111-113 is 30 frames/second, the processor 140 can first obtain 300 milliseconds as the maximum delay time, and calculate the time difference between the maximum delay time and each delay time as 100 milliseconds (300-200), 200 milliseconds (300-100) and 0 milliseconds (300-300) respectively. Next, processor 140 calculates the product of the time difference of 100 milliseconds, 200 milliseconds, and 0 milliseconds with the frame rate of 30 frames per second, which are used as the buffer frame numbers for the three buffer columns 131 to 133 respectively (i.e., 3 frames, 6 frames, and 0 frames).

Returning to Figure 4, in step S430, the processor 140 controls the camera circuits 111-11n via the transmission circuit 120 to capture images of the area around the vehicle from multiple shooting directions to generate multiple image streams, and controls the transmission circuit 120 to transmit the multiple image streams to the buffer arrays 131-13n in the memory 130 respectively. In other words, the processor 140 can store all the image frames of the image streams captured by each of the camera circuits 111-11n to the buffer arrays 131-13n corresponding to the camera circuits 111-11n respectively.

In step S440, processor 140 detects the number of frames in each buffer queue of the image stream temporarily stored in each buffer queue. In some embodiments, processor 140 periodically detects the number of image frames in each buffer queue that are currently temporarily stored in the image stream as the number of frames in each buffer queue.

In step S450, processor 140 determines whether to retrieve the first frame of each temporarily stored image stream in buffer sequences 131-13n based on the number of buffer frames and the number of frames in each buffer sequence. In this embodiment, the first frame is the image frame of each image stream that is first stored in the corresponding buffer sequence. When processor 140 determines that it should retrieve the first frame of each temporarily stored image stream in buffer sequences 131-13n, processor 140 executes step S460. Conversely, when the processor 140 determines that it does not retrieve the first frame of each of the temporarily stored image streams in the buffer arrays 131 to 13n, the processor 140 executes step S430 again.

The extraction judgment will be further explained below. Referring also to Figure 7, Figure 7 illustrates a flowchart of steps S451 to S454, which are included in step S450 of the image synchronization method for a vehicle disclosed in some embodiments. As shown in Figure 7, step S450 of Figure 4 includes steps S451 to S454.

In step S451, processor 140 compares the number of frames in each buffer column with the number of buffered frames in each buffer column plus one. When the number of frames in one buffer column is greater than the number of buffered frames plus one, processor 140 executes step S452. When the number of frames in each buffer column is neither greater than nor equal to the number of buffered frames in each buffer column plus one, processor 140 executes step S453. In addition, when the number of frames in each buffer column is equal to the number of buffer frames in each buffer column plus one, the processor 140 executes step S454.

In step S452, processor 140 deletes the first frame of the temporarily stored image stream in buffer queues whose frame count is greater than the corresponding buffer frame count plus one. In other words, processor 140 will delete the first frame of the temporarily stored image stream in a buffer queue as soon as it detects that the number of temporarily stored image frames in a buffer queue is greater than the buffer frame count of that buffer queue plus one. Then, processor 140 will execute step S451 again.

In step S453, processor 140 determines not to retrieve the first frame of each temporarily stored image stream in buffer queues 131-13n. In other words, processor 140 will not retrieve the temporarily stored first frame from buffer queues 131-13n as long as it detects that the number of frames in all buffer queues 131-13n is not greater than and not equal to the number of frames in each buffer queue plus one. Then, processor 140 can execute step S430 again (i.e., control buffer queues 131-13n to wait to receive new image streams from the corresponding camera circuits). In some embodiments, when processor 140 detects after a specific period of time (e.g., 30 minutes) that the number of frames in one of the buffered columns is consistently less than the buffered frame count plus one, processor 140 can determine that this buffered column has malfunctioned and therefore will no longer compare the number of frames in this buffered column with the buffered frame count plus one (i.e., only compare the number of frames in other buffered columns with the buffered frame count plus one).

In step S454, processor 140 determines that the first frame of the image stream temporarily stored in each of the buffer queues 131 to 13n can be retrieved. In other words, as long as processor 140 detects that the number of frames in all buffer queues 131 to 13n is equal to the number of buffer frames plus one, it will determine that the temporarily stored first frame can be retrieved from the buffer queues 131 to 13n to continue executing step S460.

For example, continuing from the previous example, suppose the number of buffer frames for buffer columns 131 to 133 are 3, 6, and 0 frames respectively, and the number of frames for each of buffer columns 131 to 133 are 5, 6, and 0 frames respectively. The processor 140 can determine that the number of frames for buffer column 131 is greater than the number of buffer frames for buffer column 131 plus one (i.e., 5 frames is greater than 4 frames). Based on this, the processor 140 deletes the first frame of the image stream temporarily stored in the buffer queue 131 (i.e., leaving 4 frames), and compares the number of frames in each of the buffer queues 131 to 133 with the number of buffered frames in each of the buffer queues 131 to 133 plus one.

At this time, the processor 140 can determine that the number of frames in the buffer queue 131 is equal to the number of buffered frames in the buffer queue 131 plus one (i.e., 4 frames equals 4 frames), determine that the number of frames in the buffer queue 132 is less than the number of buffered frames in the buffer queue 131 plus one (i.e., 6 frames is less than 7 frames), and determine that the number of frames in the buffer queue 133 is less than the number of buffered frames in the buffer queue 131 plus one (i.e., 0 frames is less than 1 frame). In other words, the processor 140 can determine that the number of frames in each of the buffer columns 131-133 is neither greater than nor equal to the number of buffered frames in each of the buffer columns 131-133 plus one. Based on this, the processor 140 controls the buffer columns 131-133 to wait for new image streams to be received from the camera circuits 111-113 (i.e., to execute step S430 again).

It is worth noting that the number of frames in buffer array 131 is greater than the number of frames in buffer array 131 because the shooting time of camera circuit 111 is earlier than that of camera circuits 112-113. Since the image stream captured by camera circuit 111 before camera circuits 112-113 start shooting cannot be synchronized with the images captured by camera circuits 112-113, processor 140 needs to delete the image stream captured earlier. The reason why the number of frames in buffer queue 132 is less than the number of buffered frames in buffer queue 132 is that the first frame of buffer queue 132 needs to wait for one more new image frame to enter buffer queue 132 before it can be output. The reason why the number of frames in buffer queue 133 is less than the number of buffered frames in buffer queue 133 is that no image frame has entered buffer queue 133 yet.

Furthermore, when buffer columns 131-133 receive a frame from a new image stream from camera circuits 111-113, the number of frames per buffer column 131-133 is 5, 7, and 1, respectively. Processor 140 compares the number of frames per buffer column with the number of buffered frames per buffer column plus one. Then, processor 140 can determine that the number of frames per buffer column 131 is greater than the number of buffered frames per buffer column 131 plus one (i.e., 5 frames is greater than 4 frames). Based on this, the processor 140 deletes the first frame of the image stream temporarily stored in the buffer queue 131 (i.e., leaving 4 frames), and compares the number of frames in each of the buffer queues 131 to 133 with the number of buffered frames in each of the buffer queues 131 to 133 plus one.

At this time, the processor 140 can determine that the number of frames in the buffer queue 131 is equal to the number of buffered frames in the buffer queue 131 plus one (i.e., 4 frames equals 4 frames), determine that the number of frames in the buffer queue 132 is equal to the number of buffered frames in the buffer queue 131 plus one (i.e., 7 frames equals 7 frames), and determine that the number of frames in the buffer queue 133 is equal to the number of buffered frames in the buffer queue 131 plus one (i.e., 1 frame equals 1 frame). In other words, processor 140 can determine that the number of frames in each of buffer columns 131 to 133 is equal to the number of buffered frames in each of buffer columns 131 to 133 plus one. Based on this, processor 140 determines that the first frame of the temporarily stored image stream in each of buffer columns 131 to 133 can be retrieved for subsequent merging (i.e., step S460).

Returning to Figure 4, in step S460, processor 140 merges all the first frames into a merged image. In some embodiments, processor 140 controls display 150 to display this merged image, or directly stores this merged image in memory 130 for subsequent applications (e.g., image optimization or image recognition). In other words, processor 140 can not only display the merged image on display 150, but also simply store the merged image in memory 130 without displaying it on display 150, or simultaneously store the merged image in memory 130 and display it on display 150.

In some embodiments, multiple regions in the merged image each include multiple first frames (i.e., image merging). In other embodiments, the merged image can be a panoramic image when the shooting ranges of the camera circuits 111-11n overlap. In still other embodiments, the merged image may also include multiple first frames corresponding to multiple shooting ranges with the same time marker (e.g., 3:10 PM) (i.e., timeline merging). When a user selects a shooting range corresponding to one of the time markers (e.g., using a touch-screen display 150), the processor 140 can control the display 150 to display the first frame corresponding to the selected shooting range. It is worth noting that this merged image can be applied to various scenarios (e.g., stereoscopic vision, panoramic stitching, high-speed photography), thus achieving the effect of maintaining image consistency and alignment. In other words, this disclosure utilizes the above method to achieve time synchronization or image stitching of images from different sources used in vehicles.

The generation of merged images is illustrated below with a practical example. Referring also to FIG8, FIG8 illustrates a schematic diagram of a merged image 800 disclosed in some embodiments. As shown in FIG8, assuming that the number of camera circuits and buffer arrays n is equal to 2, the merged image 800 includes a first region 810 and a second region 820. The first region 810 contains the first frame generated by camera circuit 111 (e.g., a front image of a vehicle taken at a specific time), and the second region 820 contains the first frame generated by camera circuit 112 (e.g., a rear image of a vehicle taken at a specific time). It is worth noting that although this example uses a combined image 800 that includes a first area 810 and a second area 820 displayed vertically, other embodiments may also use a combined image 800 that includes a first area 810 and a second area 820 displayed horizontally.

In summary, the image synchronization method and device for vehicles disclosed herein utilize the delay time settings of the camera circuits around the vehicle to determine the number of buffer frames for each buffer sequence. By comparing the number of frames in each buffer sequence with the number of buffered frames, it determines when to retrieve the first frame of each buffer sequence for ambient image synchronization. This solves the problem of image asynchrony caused by the different transmission delays of each camera circuit. Furthermore, the image synchronization method and device for vehicles disclosed herein can address the latency of wireless networks or the instability of image frames arriving at the system when wireless connections are used for camera circuits, and avoid the increased wiring costs associated with wired connections for camera circuits. On the other hand, the image synchronization method and device for vehicles disclosed herein ensures image consistency and alignment when multiple camera circuits are shooting the same scene simultaneously.

The above description is merely a preferred embodiment of this disclosure and does not limit the scope of the patent. Therefore, all equivalent changes made using the content of this disclosure are similarly included within the scope of this disclosure and are hereby stated.

100: Image synchronization device for the vehicle 111~11n: Camera circuit 120: Transmission circuit 130: Memory 140: Processor 150: Display VH: Vehicle 210~240: Shooting range img1~imgn: Video stream 131~13n: Buffer sequence S410~S460, S411~S413, S421~S422, S451~S454: Steps 800: Merged images 810: First area 820: Second area

Figure 1 is a block diagram illustrating an image synchronization device applied to a vehicle in some embodiments of the present invention.

Figure 2 is a schematic diagram illustrating the arrangement of multiple camera circuits in some embodiments.

Figure 3 illustrates a schematic diagram of multiple buffer formations in some embodiments.

Figure 4 illustrates a flowchart of an image synchronization method for a vehicle, as shown in some embodiments.

Figure 5 illustrates a flowchart of several steps included in one step of an image synchronization method for a vehicle, as shown in some embodiments.

Figure 6 illustrates a flowchart of several steps included in another step of the image synchronization method for a vehicle, as disclosed in some embodiments.

Figure 7 illustrates a flowchart of several steps included in another step of the image synchronization method for a vehicle, as disclosed in some embodiments.

Figure 8 is a schematic diagram illustrating the merged images in some embodiments.

S410~S460: Steps

Claims (10)

An image synchronization method for a vehicle includes: Step a) Detecting multiple delay times between multiple camera circuits and the processor via a transmission circuit on the vehicle using a processor on the vehicle; Step b) Calculating, using the processor, a buffer frame number for each of multiple buffer sequences in a memory on the vehicle based on the multiple delay times, wherein the multiple buffer sequences correspond to the multiple camera circuits respectively; Step c) The processor, via the transmission circuit, controls the multiple camera circuits to capture images of the area around the vehicle from multiple shooting directions to generate multiple image streams, and controls the transmission circuit to transmit the multiple image streams to the multiple buffer queues in the memory respectively; Step d) The processor, via the transmission circuit, detects the number of frames of one buffer queue of the image stream temporarily stored in each buffer queue; Step e) The processor, based on the number of buffer frames and the number of frames in each buffer queue, determines whether to retrieve the first frame of the temporarily stored image stream in each of the plurality of buffer queues, wherein the first frame is the first image frame of each image stream stored in the corresponding buffer queue; Step f) When it is determined that the first frame of the temporarily stored image stream in each of the plurality of buffer queues should be retrieved, the processor merges all the first frames into a single merged image; and If, in step g), it is determined that the first frame of the image stream temporarily stored in each of the buffer queues should not be retrieved, then steps c) through e) are executed again. The image synchronization method for a vehicle as described in claim 1, wherein step a) includes: controlling the processor to transmit a timestamp to the plurality of camera circuits; controlling the processor to receive multiple feedback messages from the plurality of camera circuits respectively; and calculating, by the processor, multiple time differences between a transmission time for transmitting the timestamp and multiple reception times for receiving the multiple feedback messages respectively, as the multiple delay times between the plurality of camera circuits and the processor. The image synchronization method for a vehicle as described in claim 1, wherein step b) comprises: calculating, by the processor, multiple time differences between the multiple delay times and the largest of the multiple delay times; and calculating, by the processor, multiple product values between the multiple time differences and a frame rate of the multiple camera circuits as the multiple buffer frames of the multiple buffer sequences. The image synchronization method for a vehicle as described in claim 1, wherein step e) includes: Step e1) comparing the frame count of each buffer queue with the buffer frame count plus one for each buffer queue, using the processor; and Step e2) when the frame count of each buffer queue is equal to the corresponding buffer frame count plus one, determining, using the processor, to retrieve the first frame of the image stream temporarily stored in each of the plurality of buffer queues. The image synchronization method for a vehicle as described in claim 4, wherein step e) further includes: When the number of frames in one of the buffer sequences is greater than the number of buffer frames plus one, the processor deletes the first frame of the image stream temporarily stored in the buffer sequence whose number of frames is greater than the corresponding number of buffer frames plus one, and performs step e1) again; and When the number of frames in each of the buffer queues is neither greater than nor equal to the number of buffered frames in each buffer queue plus one, the processor determines not to retrieve the first frame of the image stream temporarily stored in each of the multiple buffer queues. An image synchronization device for a vehicle includes: a plurality of camera circuits disposed on a vehicle and configured to capture images of the area surrounding the vehicle from multiple shooting directions; a transmission circuit connected to the plurality of camera circuits and configured to receive an image stream from each of the camera circuits; a memory connected to the transmission circuit and configured to store a plurality of instructions and a plurality of buffer queues, wherein the plurality of buffer queues correspond to the plurality of camera circuits; and a processor connected to the transmission circuit and the memory and configured to access the plurality of instructions to perform the following steps: Step a) Detect multiple delay times between the plurality of camera circuits and the processor via the transmission circuit; Step b) Calculate the number of buffered frames for each of the plurality of buffered columns on the vehicle based on the multiple delay times; Step c) Control the plurality of camera circuits via the transmission circuit to capture images of the area around the vehicle from multiple shooting directions to generate multiple image streams, and control the transmission circuit to transmit the multiple image streams to the plurality of buffered columns respectively; Step d) Detect the number of frames in a column of the image stream temporarily stored in each buffered column; Step e) Based on the number of buffer frames and the number of frames in each buffer queue, determine whether to retrieve the first frame of the temporarily stored image stream in each of the plurality of buffer queues, wherein the first frame is the first image frame of each image stream stored in the corresponding buffer queue; Step f) When it is determined that the first frame of the temporarily stored image stream in each of the plurality of buffer queues should be retrieved, merge all the first frames into a single merged image; and If, in step g), it is determined that the first frame of the image stream temporarily stored in each of the buffer queues should not be retrieved, then steps c) through e) are executed again. The image synchronization device for a vehicle as described in claim 6, wherein in step a), the processor is configured to perform the following steps: transmitting a timestamp to the plurality of camera circuits via the transmission circuit; receiving multiple feedback messages from the plurality of camera circuits respectively via the transmission circuit; and calculating multiple time differences between a transmission time for transmitting the timestamp and multiple reception times for receiving the multiple feedback messages, as the multiple delay times between the plurality of camera circuits and the processor. The image synchronization device for a vehicle as described in claim 6, wherein in step b), the processor is configured to perform the following steps: Calculate multiple time differences between the plurality of delay times and the largest of the plurality of delay times; and Calculate multiple product values between the plurality of time differences and a frame rate of the plurality of camera circuits as the plurality of buffered frames in the plurality of buffer sequences. The image synchronization device for a vehicle as described in claim 6, wherein in step e), the processor is configured to perform the following steps: Step e1) comparing the frame count of each buffer sequence with the buffer frame count plus one; and Step e2) determining that the first frame of the image stream temporarily stored in each of the plurality of buffer sequences is retrieved when the frame count of each buffer sequence is equal to the buffer frame count plus one. The image synchronization device for a vehicle as described in claim 9, wherein in step e), the processor is further configured to perform the following steps: When the number of frames in one of the buffer sequences is greater than the number of buffer frames plus one, the processor deletes the first frame of the image stream temporarily stored in the buffer sequence whose number of frames is greater than the corresponding number of buffer frames plus one, and performs step e1) again; and When the number of frames in each of the buffer queues is neither greater than nor equal to the number of buffered frames in each buffer queue plus one, the processor determines not to retrieve the first frame of the image stream temporarily stored in each of the multiple buffer queues.