CN118521797B - Vehicle-mounted reversing image system of heavy truck - Google Patents
Vehicle-mounted reversing image system of heavy truck Download PDFInfo
- Publication number
- CN118521797B CN118521797B CN202410668154.1A CN202410668154A CN118521797B CN 118521797 B CN118521797 B CN 118521797B CN 202410668154 A CN202410668154 A CN 202410668154A CN 118521797 B CN118521797 B CN 118521797B
- Authority
- CN
- China
- Prior art keywords
- vehicle
- image
- enhancement
- computing unit
- edge computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
- G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
- G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Molecular Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Biodiversity & Conservation Biology (AREA)
- Image Processing (AREA)
Abstract
The invention relates to the field of a reversing image system, in particular to a vehicle-mounted reversing image system of a heavy truck, which comprises a vehicle-mounted terminal, an edge computing unit and a cloud service center, wherein the vehicle-mounted terminal is used for collecting image data of the surrounding environment of the vehicle and displaying images processed by the edge computing unit, the edge computing unit is used for receiving the image data collected by the vehicle-mounted terminal and carrying out layered circular filtering enhancement processing on the image data to generate enhanced images and transmitting the enhanced images to the vehicle-mounted terminal for display, and the cloud service center is used for receiving the image data and processing logs uploaded by the edge computing unit and generating an image enhancement strategy based on global data and sending the image enhancement strategy to the edge computing unit, and a characteristic layer of the images under different spatial frequencies is adaptively extracted through a set of circular Gaussian filters with different sizes by utilizing the spatial scale characteristics of the images. The nonlinear mapping mode of each layer is designed pertinently, and differentiated segmentation strategies are adopted in a dark area, a bright area and a transition area.
Description
Technical Field
The invention relates to the field of reversing image systems, in particular to a vehicle-mounted reversing image system of a heavy truck.
Background
With the high-speed development of the transportation industry, heavy trucks have become the middle stream whetspost for cargo transportation. However, due to the large size of the heavy truck body, the rear dead zone is wide, the reversing difficulty is high, accidents are frequent, and hidden hazards are brought to drivers and pedestrians. Therefore, it is needed to develop an advanced vehicle-mounted looking-around system, which provides an omnibearing and high-definition reversing image for a driver with a heavy truck, improves the environment sensing capability in a complex scene, and reduces the occurrence of accidents.
Currently, vehicle-mounted camera technology is widely applied to the fields of passenger cars and commercial vehicles. The main scheme is that a fish-eye camera is arranged at the rear part of a vehicle, images at the rear part of the vehicle are collected through a wide-angle lens, and then the visual effect is improved through algorithms such as image distortion correction, contrast enhancement and the like, so that a driver is assisted to reverse. But are limited by the computational power and the power consumption of a vehicle-standard hardware platform, the algorithms are easy, and the algorithms are difficult to exert ideal effects under complex working conditions faced by heavy trucks due to insufficient illumination, strong backlight, rain and fog and other severe environments and insufficient adaptability. Especially when the vehicle is reversed in the environments of narrow channels, construction sites and the like, the problems of underexposure, overexposure, incomplete vision and the like often occur on the picture, and difficulty is brought to the judgment of a driver.
To overcome the adverse environmental impact, the industry began to introduce multi-camera solutions and intelligent enhancement algorithms. For example, a plurality of cameras at front, back, left and right are adopted to cooperatively collect panoramic images, and technologies such as multi-frame noise reduction, intelligent contrast enhancement and the like are combined to improve the visual quality under complex illumination. In the existing looking-around system, the pixel brightness is adaptively adjusted by using a CLAHE algorithm, and noise is reduced by multi-frame weighted average, so that imaging effects under extreme conditions such as backlight and the like are improved to a certain extent. But is limited by bicycle data and calculation force, the scheme can not cope with the serious change of illumination, partial shielding and the like, and the algorithm parameter adjustment is time-consuming and labor-consuming, so that the method is difficult to adapt to the difference of the working condition and the driving habit of the vehicle.
With the development of the internet of vehicles and the automatic driving technology, academics begin to explore a new paradigm of cloud collaborative awareness. The basic idea is that image data acquired by the vehicle is uploaded to a cloud by utilizing a vehicle-mounted communication module, a perception model is optimized by means of mass data and calculation force of the cloud, and a result is fed back to a vehicle-mounted end to guide a perception decision. The cloud learning edge execution architecture enables the algorithm to move from a bicycle to a motorcade, and greatly expands the optimization space of the perception algorithm based on data evolution. However, the existing research on the automatic driving of the multi-focus passenger car does not fully consider the particularities of the reversing scene of the heavy truck, such as shielding of a cargo box, narrow channels, more severe working conditions and the like, and the scheme of the passenger car is difficult to directly apply. In addition, the time delay and bandwidth limitation of cloud-edge data interaction also make such schemes difficult to guide reversing instructions in real time.
In summary, the existing reversing auxiliary system for the heavy truck still cannot meet the actual use requirements in terms of sensing precision, scene adaptability, cloud enabling and the like. The root of the method is as follows:
1. The existing image enhancement algorithm does not fully consider the characteristics of near-large-far-small and rich details of the reversing image of the heavy truck, adopts single-scale filtering, only focuses on overall brightness adjustment, but neglects the fine enhancement of different levels of characteristics, and is difficult to consider between detail textures and overall contours.
2. The existing self-adaptive enhancement strategy has single parameter, is regular to solidify, lacks a flexible mapping mechanism, is difficult to cope with extreme illumination conditions such as backlight, dark night and the like, so that a key target is easy to submerge in an underexposure or overexposure area, and has potential safety hazards.
3. The existing fusion method adopts simple weighting, cannot fully exert the complementary effect of different layers of characteristics, and breaks through the elbows when balancing the sharpness of details and the overall naturalness.
4. The existing system lacks cloud data aggregation and strategy evolution mechanisms, the perception algorithm is limited to single vehicle learning, the commonality rules of different vehicle types and different road condition data cannot be fully mined, and the generalization of the algorithm and the adaptability of long-tail scenes are to be improved. Meanwhile, cloud capacity fails to meet the first line in real time, vehicle end perception and cloud learning are disjointed, and closed-loop evolution is not formed.
Therefore, a new vehicle-mounted image enhancement scheme facing the reversing of the heavy truck is needed.
Disclosure of Invention
The invention aims to solve the technical problems, and provides a vehicle-mounted reversing image system of a heavy truck, which fully utilizes the circular imaging characteristics of reversing images of the heavy truck in an algorithm level, performs refinement and differentiation enhancement on features of different spatial scales, realizes seamless fusion from local to whole, finds an optimal balance point between sharpness and whole coordination, realizes self-adaptive piecewise nonlinear enhancement on brightness adjustment aiming at different illumination and different road conditions, furthest expands imaging dynamic range, ensures that a key target is still striking under extreme conditions, breaks the condition of single vehicle perception in an architecture level, fully utilizes the data of a motorcade to promote perception capability, forms a continuous evolution way across vehicles and scenes in a cloud, realizes depth coordination of cloud learning and vehicle end execution, and unifies advanced algorithm and vehicle rule level constraint harmony so as to lead perception experience to be integrated with driving habit. Yun Bianduan algorithm is cooperatively innovated, the comprehensive improvement of the reversing image enhancement of the heavy truck in the aspects of precision, real-time performance, adaptability, flexibility and the like is realized, an intelligent reversing auxiliary system which is safe and reliable and continuously evolved is constructed, the cost reduction and the synergy of the boosting heavy truck industry are realized, and the national life is served.
The invention provides a vehicle-mounted reversing image system of a heavy truck, which comprises a vehicle-mounted terminal, an edge computing unit and a cloud service center, wherein,
The vehicle-mounted terminal is used for collecting image data of the surrounding environment of the vehicle and displaying the image processed by the edge computing unit;
The edge computing unit is used for receiving the image data acquired by the vehicle-mounted terminal, carrying out layered circular filtering enhancement processing on the image data, generating an enhanced image and transmitting the enhanced image to the vehicle-mounted terminal for display;
The cloud service center is used for receiving the image data and the processing log uploaded by the edge computing unit, generating an image enhancement strategy based on global data and sending the image enhancement strategy to the edge computing unit.
Specifically, the vehicle-mounted terminal includes:
the cameras are respectively arranged at different positions at the rear part of the vehicle and are used for collecting image data of the surrounding environment of the vehicle at multiple angles;
the display screen is arranged in the vehicle cab and used for displaying the enhanced image transmitted by the edge computing unit;
The camera is connected with the edge computing unit through a CAN bus, and the display screen is connected with the edge computing unit through a USB.
Specifically, the functions of the cloud service center include:
global data management, which is to receive, store and analyze the image data and the processing log uploaded by the edge computing unit;
Self-adaptive strategy learning, training and generating image enhancement strategies applicable to different scenes based on global data;
and (3) managing and monitoring the fleet, monitoring the states of all the accessed vehicles in real time, and providing remote management and upgrading services.
The edge computing unit comprises a hierarchical circular filtering enhancement module, a cloud coordination module and a lightweight reasoning engine;
The layering circle filtering enhancement module is used for receiving the images acquired by the vehicle-mounted terminal, performing self-adaptive layering enhancement and outputting enhanced images;
The cloud coordination module is used for interacting data with the cloud service center and receiving layered enhancement parameters issued by the cloud service center;
the lightweight reasoning engine is used for loading the end-to-end layered enhancement model trained by the cloud service center and processing the images in real time.
The hierarchical circular filtering enhancement module comprises a hierarchical module, an adaptive enhancement module and a hierarchical fusion module;
the layering module is configured to use N circular gaussian filters { H i|Hi∈RK×K, i=1, 2,) with different sizes to perform a layer-by-layer convolution on an input image I e R H×W to obtain an N-layer scale image { I i|Ii e R, i=1, 2,) and N }, where the convolution process is:
Ii＝I*Hi,i=1,2,...,N
wherein, the filter radius increases according to the equal ratio number series:
ri＝r0·qi-1,i=1,2,...,N
wherein r 0 and q are the initial radius and the coefficient of the equal ratio respectively;
The adaptive enhancement module is configured to perform adaptive nonlinear mapping on the layered image { I i }, and output an enhanced image { I' i }:
I′i＝fi(Ii;αi,βi,γi),i=1,2,...,N
wherein f i is a piecewise mapping function:
Where I i represents some intensity or value of the input, and the parameter α i,βi,γi,δi is determined by the enhancement policy received by the cloud coordination module;
The hierarchical fusion module is configured to perform weighted summation on the enhanced image I 'i to obtain a final enhanced result I':
wherein w i is the weight, and N is the total number of images;
Where a i is the i-th layer score issued by the cloud service center, and norm represents the normalization function.
Specifically, the functions of the cloud coordination module include:
The environment sensing is carried out, image statistical characteristics are extracted and uploaded to the cloud service center, and environment categories returned by the cloud service center are received;
The data is returned, and the layering characteristics and the enhancement result are uploaded to the cloud service center for enhancing policy learning;
And strategy updating, namely receiving the latest hierarchical enhancement parameters which are issued by the cloud service center and comprise a filtering scale, a mapping coefficient, a fusion weight and the like.
Specifically, the lightweight inference engine is obtained based on cloud knowledge distillation and is used for executing real-time layered enhancement at a vehicle end, and the lightweight inference engine has the structure that:
Wherein, Is an end-to-end mapping network trained in the cloud, e represents an environmental parameter, and Θ is a model weight;
The lightweight inference engine performs on-end acceleration optimization through the TVM, converts the model into a highly optimized hardware-related program by using a heterogeneous computational graph, and performs low-latency inference by using the vehicle-mounted GPU and NVDLA.
Specifically, the edge computing unit adopts an Injeida Jetson AGX Xavier embedded system module, and comprises an 8-core ARM CPU, a 512-core Volta GPU, a 64-tensor core and 2 NVDLA accelerators, and is provided with 16GB LPDDR4 memory and 32GB eMMC memory.
Specifically, the camera in the vehicle-mounted terminal is a fisheye camera, a Sony IMX586 sensor is adopted, the resolution is 8000 multiplied by 6000, the pixel size is 0.8 mu m, the angle of view is 200 degrees, the display screen is a 12.3-inch LCD liquid crystal screen, the resolution is 1920 multiplied by 720, the brightness is 1000nit, and the touch control function is supported.
The invention has the following beneficial effects:
1. Multi-scale hierarchical feature extraction and adaptive enhancement are efficiently coordinated. The traditional image enhancement method generally adopts a fixed scale and unified strategy, and is difficult to consider the integral structure and local detail of the reversing image of the heavy truck. The invention skillfully utilizes the spatial scale characteristics of the image, and adaptively extracts the characteristic layers of the image under different spatial frequencies through a group of circular Gaussian filters with different sizes. The nonlinear mapping mode of each layer is designed pertinently, differential segmentation strategies are adopted in a dark area, a bright area and a transition area, and dynamic ranges are stretched and compressed in a vector manner. The two links complement each other, firstly, destructive decoupling is carried out on numerous original image data, the original image data is integrated into zero, then each component is accurately regulated and controlled, each component is as long as possible, and finally, fusion reconstruction is carried out, and the original image data is integrated into zero. Therefore, the proper balance among the whole, the part, the brightness and the primary and secondary is realized, the image is rich in level and not disordered, and the detail is clear and not abrupt.
2. Dynamic gaming integrating weight self-adaptive adjustment and layering characteristics. When multi-scale layered features are fused, the traditional method often adopts fixed weight addition or simply gives a higher weight to a high-frequency layer, and the potential of the features of each layer cannot be fully exerted. The invention develops a new way to dynamically adjust the weights of all layers according to the scene semantic information issued by the cloud in real time. When the reversing scene is mainly close-range and detail is the king, the weight of the middle-high frequency layer is increased, and when the scene is wide and the overall coordination is more important, the weight of the low-frequency base layer is increased. Dynamic games of high and low frequency layers are kept at all times, so that detail loss caused by excessive smoothing is avoided, and noise interference caused by excessive sharpening is also avoided. Just like an automatic gain control system, environmental feedback is monitored constantly, gains of all frequency bands are dynamically adjusted, the receiving and releasing degrees are guaranteed, the best balance of overall coordination and detail enrichment is guaranteed, and image quality varies from scene to scene and from time to time.
3. Closed loop iteration of cloud data aggregation and end-side policy updating. The problems of limited training data and insufficient scene coverage of bicycle intelligence all the time exist, and cloud learning forms a massive scene data set by returning and converging data, so that a solid foundation is provided for improving algorithm performance. Meanwhile, the cloud end utilizes strong calculation force to continuously quench essence strategies, and issues edge end guiding parameter updating. So, the continuous data supply and strategy evolution make the system break through the perception of the ceiling and obtain qualitative leaps in scene adaptability and robustness. The edge end is not passively accepted, but actively adapts to hardware constraint, vehicle rule level calculation potential is brought into full play, and the cloud is optimized in real-time performance and deployment efficiency. Cloud teaching and learning are performed, cloud understanding is performed, and a two-way rush architecture not only extends cloud bonus to the end, but also achieves phase-to-phase reflection among the end clouds, and brings comprehensive benefits of far-beyond single-vehicle unidirectional intelligence.
4. And 3, high-efficiency light-weight end-side reasoning and real-time high-definition visual experience is realized. The final objective of image enhancement is to provide a driver with a lossless visual perception, which places severe demands on computational latency. The cloud knowledge pouring system skillfully utilizes knowledge distillation to enable cloud knowledge pouring to be taught, and a small and precise network is designed for edges. And the cloud performance is approximated to the maximum limit under the limited computing power through soft and hard collaborative optimization, communication optimization, encoding and decoding acceleration and the like. Therefore, a high-efficiency and convenient bridge is built between the complex algorithm and the vehicle-mounted hardware, so that the cloud advanced result is embedded in the terminal, and the terminal-side limited resources are used and used as far as possible, so that the contradiction between high-delay, high-energy consumption and high-performance is avoided, and the high-definition, smooth and real-time seamless experience is brought.
In conclusion, the invention systematically solves the defects of the traditional scheme in the aspects of scene adaptability, overall detail balance, end cloud cooperativity and the like by skillfully combining layered feature extraction with self-adaptive enhancement, dynamic weight adjustment with multi-scale fusion, end cloud cooperativity with closed-loop evolution, light-weight reasoning with real-time high-definition display and mutual promotion, and brings all-round breakthrough progress for the reversing image enhancement of the heavy truck. It is believed that intelligent, high-definition and robust visual assistance will be a new standard for the heavy-duty industry, and driving safety is guaranteed. The innovative concept of the invention is necessary to provide a precious design paradigm for the vehicle-mounted vision field and inject forward power for further development of end cloud fusion.
Drawings
FIG. 1 is a frame diagram of a truck-mounted reverse imaging system for a heavy truck in accordance with the present invention;
Detailed Description
In order to further describe the technical means and effects adopted by the invention to achieve the preset aim, the following detailed description refers to the specific implementation, structure, characteristics and effects of a vehicle-mounted reversing image system of a heavy truck according to the invention with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" means that the embodiments are not necessarily the same. Furthermore, the particular features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The following specifically describes a specific scheme of the vehicle-mounted reversing image system of the heavy truck provided by the invention with reference to the accompanying drawings.
Example 1
Referring to fig. 1, the invention discloses a vehicle-mounted reversing image system of a heavy truck, which comprises a vehicle-mounted terminal 1, an edge computing unit 2 and a cloud service center 3, wherein the three work cooperatively to realize self-adaptive enhancement of an environment image in a reversing process of the vehicle.
The vehicle-mounted terminal 1 is mounted on a vehicle and mainly comprises a camera and a display screen. The camera adopts a fisheye camera, is provided with a Sony IMX586 sensor, has ultra-high resolution of 8000 multiplied by 6000 and wide view angle of 200 degrees, and can collect the environment image behind the vehicle without dead angles. The cameras are respectively arranged at different positions at the rear part of the vehicle and connected with the edge computing unit 2 through the CAN bus, and the collected image data are transmitted in real time. The display screen is a 12.3-inch LCD liquid crystal screen, has the resolution ratio of 1920 multiplied by 720 and the brightness of up to 1000nit, supports touch operation, is arranged in a cab, and is used for displaying clear reversing images processed by the edge computing unit 2 in real time and assisting a driver in judging reversing environments.
The edge computing unit 2 is the core of the system and is responsible for carrying out self-adaptive enhancement processing on the images acquired by the vehicle-mounted terminal 1. The system adopts an Inlet-Weida Jetson AGX Xavier embedded system module, has strong heterogeneous computing capacity, comprises an 8-core ARM CPU, a 512-core Volta GPU, a 64-tensor core and 2 NVDLA accelerators, is provided with 16GB LPDDR4 memory and 32GB eMMC memory, and can efficiently execute a complex image enhancement algorithm. The software of the edge computing unit 2 mainly comprises a hierarchical circular filtering enhancement module, a cloud coordination module and a lightweight reasoning engine.
The hierarchical circular filtering enhancement module comprises a hierarchical module, a self-adaptive enhancement module and a hierarchical fusion module;
the layering module is configured to use N circular gaussian filters { H i|Hi∈RK×K, i=1, 2,) with different sizes to perform a layer-by-layer convolution on an input image I e R H×W to obtain an N-layer scale image { I i|Ii e R, i=1, 2,) and N }, where the convolution process is:
Ii＝I*Hi,i=1,2,...,N
wherein, the filter radius increases according to the equal ratio number series:
ri＝r0·qi-1,i=1,2,...,N
wherein r 0 and q are the initial radius and the coefficient of the equal ratio respectively;
The adaptive enhancement module is configured to perform adaptive nonlinear mapping on the layered image { I i }, and output an enhanced image { I' i }:
I′i＝fi(Ii;αi,βi,γi),i=1,2,...,N
wherein f i is a piecewise mapping function:
Where I i represents some intensity or value of the input, and the parameter α i,βi,γi,δi is determined by the enhancement policy received by the cloud coordination module;
The hierarchical fusion module is configured to perform weighted summation on the enhanced image I 'i to obtain a final enhanced result I':
wherein w i is the weight, and N is the total number of images;
where a i is the i-th layer score issued by the cloud service center 3, and norm represents the normalization function.
The hierarchical circular filtering enhancement module is one of the core components of the edge calculation unit 2, and is configured to receive the original image data from the vehicle-mounted terminal 1, and perform adaptive and hierarchical image enhancement. The method comprises the steps of firstly, carrying out layer-by-layer convolution on an original image by using a group of circular Gaussian filters with different sizes, and decomposing the image into N scale layers { I i|Ii∈RH×W, i=1, 2. The radius r i of the Gaussian filters is increased according to an equal ratio array, wherein the initial radius r 0 and the public ratio q are decided in real time by the cloud coordination module according to the current scene characteristics. The smaller radius filter extracts high-frequency components such as detail textures of the image, and the larger radius filter extracts low-frequency components such as background and contour. The layering strategy can fully utilize the circular imaging characteristics of the fisheye camera, and highlight detail information of different layers while maintaining the overall structure.
After the layered enhancement module obtains the layered image { I i }, respectively performing adaptive nonlinear enhancement on each layer, wherein the mapping function is f i(Ii;αi,βi,γi, and outputting a corresponding enhancement layer I' i. The mapping function adopts piecewise nonlinear design, the input brightness is divided into three sections of low illumination, medium illumination and high illumination, different strategies are respectively adopted, namely the low illumination section adopts gamma power function stretching, the detail high illumination section of the dark part is promoted to adopt gamma root function compression, the bright part is prevented from overexposure, the medium illumination section adopts linear transformation, the brightness range is expanded to delta i, and the contrast is increased. The segmentation points alpha i、βi and the mapping coefficients gamma i、δi are issued by the cloud coordination module in real time according to the environmental conditions. The self-adaptive segmentation mapping can intelligently adjust the gain of each brightness interval, and protect key information of different layers while improving the overall visual effect. Taking daytime backlight environment as an example, the layering module can properly reduce the brightness of the low-frequency background layer to avoid overexposure, and simultaneously stretches the details of the high-frequency foreground layer to avoid underexposure. And in the night low-light environment, all frequency bands can be uniformly stretched, and the whole is lightened. The parameter γ is typically between 1.5 and 3.0, the greater the value, the greater the degree of nonlinearity of the mapping curve. Delta is typically 90% to 98%, i.e., the target luminance range is 90% to 98% of the original dynamic range.
Next, the hierarchical fusion module combines each enhancement layer { I 'i } with a corresponding weight { w i } using a weighted fusion strategy to generate the final enhancement result I'. Wherein w i adopts a normalized Softmax form, and is generated in real time by a cloud coordination module according to the current scene characteristics, so that the contribution degree of the key layer is automatically highlighted. When the reversing scene is mainly based on the scene-corresponding details, the fusion module can properly improve the weight of the middle-high frequency layer, otherwise, when the scene depth is larger and the overall sense is more important, the fusion module can properly improve the weight of the low frequency layer. Dynamic balance is kept at all times, so that details and the whole are harmonious and unified. The scale factor a of Softmax is typically 5 to 10, the larger the weight distribution is, the more concentrated the key layer is.
The cloud coordination module is a bridge for data interaction between the vehicle-end system and the cloud. On one hand, the system is responsible for transmitting information such as image data, layering characteristics, enhancement results and the like acquired by the vehicle end back to the cloud end for global data analysis and strategy learning, and on the other hand, the system is responsible for receiving the latest environmental parameters and layering fusion weights issued by the cloud end and guiding the vehicle end algorithm to adjust in real time, so that the continuous evolution of the vehicle end system is realized. In addition, the cloud coordination module also bears the responsibility of environmental perception, extracts the statistical characteristics (such as average brightness, contrast and the like) of the vehicle-end image, and combines the information of vehicle positioning, time and the like to upload to the cloud environment classifier. After deducing the scene category (such as day, night, rain and fog, etc.) of the vehicle, the classifier sends the scene category to the vehicle end to trigger corresponding enhancement strategy switching, so that the system can adapt to different environmental conditions.
The lightweight reasoning engine is obtained based on cloud knowledge distillation and is used for executing real-time layered enhancement at a vehicle end, and the lightweight reasoning engine is structurally characterized in that:
Wherein, Is an end-to-end mapping network trained in the cloud, e represents an environmental parameter, and Θ is a model weight;
The whole layering enhancement flow is performed on the GPU and NVDLA of the edge computing unit 2 in a pipeline mode, and efficient parallelism is achieved. To further enhance computational performance, lightweight inference engines employ highly optimized end-to-end hierarchical enhancement networks The segmentation parameters alpha i、βi、γi、δi and the fusion weight w i of each layer can be directly predicted by forward propagation once. The network adopts a structure of a backbone network and a multi-head regression layer, and uses TVM to carry out depth heterogeneous optimization, so that the model is converted into an efficient and hardware-friendly tensor program. The backbone network can select MobileNet, shuffleNet and other lightweight feature extractors to realize millisecond-level real-time reasoning on the edge equipment. Model parameters Θ of the reasoning engine are obtained by the cloud service center 3 based on massive data and knowledge distillation pre-training, and can be updated regularly according to factors such as vehicle types and environments, so that iteration of a cloud algorithm is followed.
The lightweight inference engine then assumes the heavy duty of the real-time computation at the initiator. The method adopts an end-to-end architecture of a backbone network and a multi-head regression layer, loads parameters obtained by cloud knowledge distillation, and can predict all layered enhancement parameters at one time, including dividing points and slopes of a piecewise function, layered filtering radius, fusion weight and the like. The end-to-end reasoning has lower latency and higher parallelism than traditional fractional computation. In addition, the reasoning engine is specially optimized by utilizing a TVM tool, a customized calculation map is generated aiming at heterogeneous calculation resources (such as GPU, NVDLA and the like) of the Xavier board, and bottom optimization of operator scheduling, memory layout, data prefetching and the like is automatically completed, so that the algorithm can run smoothly on the vehicle-mounted embedded platform.
The cloud service center 3 is the brain of the system and provides data support and decision guidance for the vehicle-end system. The system comprises three functional modules, namely global data management, self-adaptive strategy learning and fleet management and control. The global data management is responsible for receiving, cleaning, storing and managing all the environmental data uploaded by the vehicles, such as original images, layering characteristics, enhancement results, driver feedback and the like, labeling and classifying the environmental data, constructing a massive reversing data set of the heavy truck, and providing materials for strategy learning.
The self-adaptive strategy learning is the core of the cloud service center 3, and utilizes a machine learning algorithm to automatically summarize rules from global data and generate image enhancement optimal strategies aiming at different scenes. The method specifically adopts an active learning paradigm, preferentially samples a small amount of samples from a mass data set and marks the samples, and is used for initializing a self-adaptive layered enhancement model, and then fine adjustment and strategy updating are continuously carried out on the model by utilizing newly-added data, so that the model can continuously evolve, and the image enhancement skills applicable to all-weather and all-road conditions are gradually mastered. In addition, the cloud service center 3 also provides a fleet management and control function, including real-time monitoring of the positions, speeds, electric quantity, abnormal states and the like of all vehicles, and issuing of software update, remote diagnosis and the like, so that unified scheduling is facilitated.
The invention has the working flow that when a driver hangs the vehicle into a reverse gear, a plurality of fisheye cameras in the vehicle-mounted terminal 1 start to work, the environment images at the rear and at the two sides of the vehicle are collected in real time at the speed of 30 frames per second, and the environment images are transmitted to the edge computing unit 2 through a CAN bus. The edge computing unit 2 firstly carries out Gaussian filtering on the original image by using a layered circle filtering module, then uses piecewise nonlinear mapping to respectively strengthen each layer, and finally obtains the clearest and most stable reversing environment image by weighting fusion. Meanwhile, the cloud coordination module of the edge computing unit 2 can transmit the data such as the environmental characteristics and the layering result back to the cloud service center 3 for large data analysis, and can also receive the enhancement strategy update from the cloud in real time, for example, the inflection point position, the slope and the like of the mapping curve are adjusted according to different weather conditions, or the fusion proportion of different layers is adjusted according to driving habits. The cloud service center 3 continuously learns and optimizes the layered enhancement strategy by using the returned data to form a closed loop. Finally, the edge computing unit 2 transmits the real-time enhanced high-definition reversing image to the central control large screen of the vehicle-mounted terminal 1 through the USB for display, so that a driver is assisted to accurately judge the environment behind the vehicle, and safe driving is realized.
The layered circular filtering enhancement algorithm fully considers the circular characteristic of the fisheye camera imaging, adopts a circular filter instead of a rectangular filter to carry out multi-scale decomposition on the image, adopts a self-adaptive enhancement strategy on different scale layers, is more suitable for a physical model of fisheye imaging, and can improve the overall visual effect while reducing the image distortion.
The self-adaptive piecewise mapping function can automatically adjust piecewise points and slopes according to ambient brightness, and adopts different nonlinear stretching or compression strategies for dark areas, bright areas and normal areas, so that the overall brightness and contrast of the image are improved, overexposure and underexposure are avoided, and the reversing image is still clear and distinguishable under complex illumination conditions.
The weighted fusion link removes image noise, realizes seamless integration of different layers of features, retains the whole outline, and highlights local details, so that the visual quality of both near and distant views is improved. The fusion weight of softmax normalization ensures smooth transition of layering characteristics and improves the attractiveness of images.
In addition, yun Bianduan co-architecture allows the system to have a continuous evolution capability. The back transmission of massive vehicle-end data provides a solid foundation for strategy learning, so that the system can automatically mine the intrinsic law of image enhancement in a complex scene from the data. And after cloud closed-loop optimization, the enhancement strategy is issued to all vehicles, so that the overall intelligent level of the motorcade is synchronously improved. The active learning paradigm further reduces the dependence of model training on manpower, and accelerates the strategy iteration speed.
And the end-to-end reasoning and TVM software and hardware collaborative optimization solve the problem of high efficiency of deployment on the algorithm end. The traditional step processing mode has large calculation redundancy, and the general processor has difficulty in playing the optimal performance of the algorithm. The integrated reasoning model provided by the system can generate all layering enhancement parameters end to end, so that the expenditure of intermediate data exchange is avoided. The special Xavier computing platform is perfectly matched with TVM depth optimization software, so that the potential of vehicle-mounted heterogeneous computing resources is furthest mined, the reasoning performance similar to a server is realized on a vehicle-mounted chip, millisecond-level ultralow delay is realized, and the harsh requirement of reversing real-time video processing is met.
It should be noted that the above-mentioned embodiments are only preferred embodiments of the present invention, and are not intended to limit the present invention, and any modifications, equivalent substitutions, improvements, etc. within the principle of the present invention should be included in the protection scope of the present invention.
Claims (7)
1. A vehicle-mounted reversing image system of a heavy truck is characterized by comprising a vehicle-mounted terminal, an edge computing unit and a cloud service center, wherein,
The vehicle-mounted terminal is used for collecting image data of the surrounding environment of the vehicle and displaying the image processed by the edge computing unit;
The edge computing unit is used for receiving the image data acquired by the vehicle-mounted terminal, carrying out layered circular filtering enhancement processing on the image data, generating an enhanced image and transmitting the enhanced image to the vehicle-mounted terminal for display;
The cloud service center is used for receiving the image data and the processing logs uploaded by the edge computing unit, generating an image enhancement strategy based on global data and sending the image enhancement strategy to the edge computing unit;
The edge computing unit comprises a hierarchical circular filtering enhancement module, a cloud coordination module and a lightweight reasoning engine;
The layering circle filtering enhancement module is used for receiving the images acquired by the vehicle-mounted terminal, performing self-adaptive layering enhancement and outputting enhanced images;
The cloud coordination module is used for interacting data with the cloud service center and receiving layered enhancement parameters issued by the cloud service center;
The lightweight reasoning engine is used for loading an end-to-end layered enhancement model trained by the cloud service center and processing images in real time;
The hierarchical circular filtering enhancement module comprises a hierarchical module, a self-adaptive enhancement module and a hierarchical fusion module;
the layering module is configured to use N circular gaussian filters { H i|Hi∈RK×K, i=1, 2,) with different sizes to perform a layer-by-layer convolution on an input image I e R H×W to obtain an N-layer scale image { I i|Ii e R, i=1, 2,) and N }, where the convolution process is:
Ii＝I*Hi,i=1,2,...,N
wherein, the filter radius increases according to the equal ratio number series:
ri＝r0·qi-1,i=1,2,...,N
wherein r 0 and q are the initial radius and the coefficient of the equal ratio respectively;
The adaptive enhancement module is configured to perform adaptive nonlinear mapping on the layered image { I i }, and output an enhanced image { I' i }:
I′i＝fi(Ii;αi,βi,γi),i=1,2,...,N
wherein f i is a piecewise mapping function:
Where I i represents some intensity or value of the input, and the parameter α i,βi,γi,δi is determined by the enhancement policy received by the cloud coordination module;
The hierarchical fusion module is configured to perform weighted summation on the enhanced image I 'i to obtain a final enhanced result I':
wherein w i is the weight, and N is the total number of images;
Where a i is the i-th layer score issued by the cloud service center, and norm represents the normalization function.
2. The on-board reverse imaging system of a heavy truck of claim 1, wherein the on-board terminal comprises:
the cameras are respectively arranged at different positions at the rear part of the vehicle and are used for collecting image data of the surrounding environment of the vehicle at multiple angles;
the display screen is arranged in the vehicle cab and used for displaying the enhanced image transmitted by the edge computing unit;
The camera is connected with the edge computing unit through a CAN bus, and the display screen is connected with the edge computing unit through a USB.
3. The on-board reverse imaging system of a heavy truck of claim 1, wherein the functions of the cloud service center comprise:
global data management, which is to receive, store and analyze the image data and the processing log uploaded by the edge computing unit;
Self-adaptive strategy learning, training and generating image enhancement strategies applicable to different scenes based on global data;
and (3) managing and monitoring the fleet, monitoring the states of all the accessed vehicles in real time, and providing remote management and upgrading services.
4. The on-board reverse imaging system of a heavy truck of claim 1, wherein the functions of the cloud coordination module comprise:
The environment sensing is carried out, image statistical characteristics are extracted and uploaded to the cloud service center, and environment categories returned by the cloud service center are received;
The data is returned, and the layering characteristics and the enhancement result are uploaded to the cloud service center for enhancing policy learning;
and strategy updating, namely receiving the latest hierarchical enhancement parameters of the cloud service center, wherein the latest hierarchical enhancement parameters comprise a filtering scale, a mapping coefficient and a fusion weight.
5. The on-board reverse imaging system of heavy truck of claim 1, wherein the lightweight inference engine is obtained based on cloud knowledge distillation for performing real-time hierarchical enhancement at the vehicle end, and has the structure that:
Wherein, Is an end-to-end mapping network trained in the cloud, e represents an environmental parameter, and Θ is a model weight;
The lightweight inference engine performs on-end acceleration optimization through the TVM, converts the model into a highly optimized hardware-related program by using a heterogeneous computational graph, and performs low-latency inference by using the vehicle-mounted GPU and NVDLA.
6. The on-board reverse imaging system of heavy truck of any of claims 1-5, wherein the edge computing unit employs an inflight Jetson AGX Xavier embedded system module comprising an 8-core ARM CPU, a 512-core Volta GPU, a 64-tensor core and 2 NVDLA accelerators, equipped with 16GB LPDDR4 memory and 32GB eMMC storage.
7. The vehicle-mounted reversing image system of the heavy truck according to claim 2, wherein the camera in the vehicle-mounted terminal is a fisheye camera, a Sony IMX586 sensor is adopted, the resolution is 8000 multiplied by 6000, the pixel size is 0.8 mu m, the angle of view is 200 degrees, the display screen is a 12.3-inch LCD liquid crystal screen, the resolution is 1920 multiplied by 720, the brightness is 1000nit, and the touch control function is supported.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410668154.1A CN118521797B (en) | 2024-05-28 | 2024-05-28 | Vehicle-mounted reversing image system of heavy truck |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410668154.1A CN118521797B (en) | 2024-05-28 | 2024-05-28 | Vehicle-mounted reversing image system of heavy truck |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118521797A CN118521797A (en) | 2024-08-20 |
| CN118521797B true CN118521797B (en) | 2024-12-27 |
Family
ID=92277945
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410668154.1A Active CN118521797B (en) | 2024-05-28 | 2024-05-28 | Vehicle-mounted reversing image system of heavy truck |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118521797B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119788941B (en) * | 2025-03-10 | 2025-05-27 | 深圳市聚芯影像有限公司 | Image processing system based on Internet of things |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112822343A (en) * | 2021-01-05 | 2021-05-18 | 中国电子科技集团公司信息科学研究院 | Night video oriented sharpening method and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100564592B1 (en) * | 2003-12-11 | 2006-03-28 | 삼성전자주식회사 | Video Data Noise Reduction Method |
| US7742652B2 (en) * | 2006-12-21 | 2010-06-22 | Sharp Laboratories Of America, Inc. | Methods and systems for image noise processing |
| US10373019B2 (en) * | 2016-01-13 | 2019-08-06 | Ford Global Technologies, Llc | Low- and high-fidelity classifiers applied to road-scene images |
| US11880770B2 (en) * | 2018-08-31 | 2024-01-23 | Intel Corporation | 3D object recognition using 3D convolutional neural network with depth based multi-scale filters |
| CN111489319A (en) * | 2020-04-17 | 2020-08-04 | 电子科技大学 | Infrared image enhancement method based on multi-scale bilateral filtering and visual saliency |
| CN117557765B (en) * | 2023-11-15 | 2024-04-09 | 兰州交通大学 | Small-target water-float garbage detection method based on APM-YOLOv7 |
-
2024
- 2024-05-28 CN CN202410668154.1A patent/CN118521797B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112822343A (en) * | 2021-01-05 | 2021-05-18 | 中国电子科技集团公司信息科学研究院 | Night video oriented sharpening method and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118521797A (en) | 2024-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11704781B2 (en) | Enhanced high-dynamic-range imaging and tone mapping | |
| Li et al. | A deep learning based image enhancement approach for autonomous driving at night | |
| CN118710571A (en) | Image enhancement method and image enhancement device | |
| CN113076815B (en) | Automatic driving direction prediction method based on lightweight neural network | |
| Ding | LENet: Lightweight and efficient LiDAR semantic segmentation using multi-scale convolution attention | |
| CN118521797B (en) | Vehicle-mounted reversing image system of heavy truck | |
| Duan et al. | M-YOLOv8s: An improved small target detection algorithm for UAV aerial photography | |
| Mandal et al. | Real-time fast low-light vision enhancement for driver during driving at night | |
| CN117726940A (en) | A method and system for fine-grained detection of remote sensing images based on region proposal enhancement | |
| CN118397599A (en) | Training method and device for driving decision reasoning model | |
| CN115984133A (en) | Image enhancement method, vehicle capture method, equipment and medium | |
| Wang et al. | Lightweight CNN-based low-light-image enhancement system on FPGA platform | |
| DE102025121875A1 (en) | Generative AI models for image rendering and inverse rendering | |
| Xing et al. | Traffic sign recognition from digital images by using deep learning | |
| Liu et al. | Importance biased traffic scene segmentation in diverse weather conditions | |
| CN115661556B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN113705427A (en) | Fatigue driving monitoring and early warning method and system based on vehicle gauge chip SoC | |
| CN117237603A (en) | An improved YOLOv8s traffic target detection method based on FPGA acceleration | |
| CN120929247A (en) | Electronic outside rear-view mirror control method, device, vehicle and storage medium | |
| CN116543278A (en) | Image detection method and system under rainy day condition | |
| CN120598834A (en) | Night image enhancement method based on multi-directional information extraction and color loss optimization | |
| CN116977826B (en) | Reconfigurable neural network target detection method under edge computing architecture | |
| CN119888656A (en) | Self-supervision blind image decomposition method oriented to automatic driving scene | |
| CN117237612A (en) | A target detection method in complex road scenes based on YOLOX model | |
| CN119851116B (en) | Road network information acquisition method integrating remote sensing image and track data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |