CN121029004A

CN121029004A - Blind spot display methods and related equipment

Info

Publication number: CN121029004A
Application number: CN202511448734.0A
Authority: CN
Inventors: 周磊
Original assignee: Zhejiang Geely Holding Group Co Ltd; Zhejiang Zeekr Intelligent Technology Co Ltd
Current assignee: Zhejiang Geely Holding Group Co Ltd; Zhejiang Zeekr Intelligent Technology Co Ltd
Priority date: 2025-10-11
Filing date: 2025-10-11
Publication date: 2025-11-28

Abstract

This application relates to the field of display technology, specifically to a blind spot display method and related equipment. The blind spot display method includes: acquiring external environmental information corresponding to the target user's gaze direction, visual information collected by a head-mounted display device worn by the target user in the gaze direction, and first pose information of the head-mounted display device, where the first pose information is the pose information of the head-mounted display device in the vehicle coordinate system; determining the gaze position of the target user's gaze point in the vehicle coordinate system based on the gaze direction and the first pose information; if the gaze position is determined to be located on a vehicle visual obstacle based on the gaze position, performing image processing on the external environmental image and the visual image based on the target user's gaze direction and the first pose information to determine a target image; and displaying the target image on the head-mounted display device. This application can improve driving safety by displaying external environmental information that the user is gazing at but is obscured by a vehicle visual obstacle through a head-mounted display device.

Description

Blind area display method and related equipment

Technical Field

The application relates to the technical field of display, in particular to a blind area display method and related equipment.

Background

The body of a vehicle includes components such as an a-pillar, a B-pillar, a C-pillar, a roof and a chassis, for example, the a-pillar, the B-pillar and the C-pillar are load-bearing pillars connecting a vehicle windshield and a cockpit door. Although the arrangement of the A column, the B column and the C column ensures the structural safety of the vehicle body, certain shielding is caused to the sight line of the driving, so that a blind area exists when a driver looks at the condition outside the vehicle when the driver is positioned in the vehicle.

In order to solve the problem of the driving blind area, related technologies propose to display the information of the surrounding environment outside the vehicle, which is blocked by the sight line obstacle of the vehicle, through the intelligent glasses, and particularly determine and display the information of the surrounding environment outside the vehicle, which is blocked by the sight line obstacle of the vehicle, according to the pose information of the intelligent glasses relative to the sight line obstacle of the vehicle. However, this approach may cause the smart glasses to display information of the surrounding environment outside the vehicle, which is blocked by the vehicle sight line obstacle, regardless of whether the driver looks at the vehicle sight line obstacle, resulting in poor visual naturalness of the user and poor user experience.

Disclosure of Invention

In view of the above, the embodiment of the application provides a blind area display method and related equipment, which aim to solve the problem that in the prior art, the visual naturalness of information of the surrounding environment outside a vehicle, which is shielded by a sight obstacle of the vehicle, is poor by displaying the information through a head-mounted display device.

In a first aspect, an embodiment of the present application provides a blind area display method, which includes acquiring external environment information corresponding to a sight line direction of a target user, visual information acquired by a head-mounted display device worn by the target user in the sight line direction, and first pose information of the head-mounted display device, wherein the external environment information includes an external environment image which is located outside a vehicle sight line obstacle in the sight line direction, the visual information includes a visual image, the first pose information is pose information of the head-mounted display device in a vehicle coordinate system, a gaze position of a gaze point of the target user in the vehicle coordinate system is determined based on the sight line direction and the first pose information, and in a case that the gaze point of the target user is located on the vehicle sight line obstacle, the external environment image and the visual image are processed based on the sight line direction of the target user and the first pose information, and the target image is determined, and the target image is displayed on the head-mounted display device.

According to the application, the fixation position of the fixation point of the target user under the vehicle coordinate system is determined, so that the image fusion of the external environment image and the visual image acquired by the head-mounted display device is conveniently triggered when the fixation point of the target user is determined to be positioned on the vehicle sight obstacle, the visual consistency and the visual naturalness of the target user when the head-mounted display device is worn can be improved, and the situation in the blind area in the sight direction can be intuitively acquired by the target user through the head-mounted display device when the target user drives the vehicle, so that the driving safety can be improved.

In some embodiments, the vehicle exterior environment information further comprises first point cloud data, the visual information further comprises second point cloud data, and the blind zone display method further comprises, before acquiring the first pose information of the head-mounted display device, acquiring second pose information of the head-mounted display device, wherein the second pose information is pose information in a coordinate system of the head-mounted display device, determining a pose transformation matrix based on the first point cloud data and the second point cloud data, and converting the second pose information into the first pose information in the coordinate system of the vehicle based on the pose transformation matrix.

In some embodiments, the visual information further comprises second point cloud data, the image processing is performed on the vehicle exterior environment image and the visual image based on the sight direction of the target user and the first pose information, the target image is determined, the method comprises the steps of determining first acquisition parameters based on the visual image, performing left-right image pixel matching processing on the visual image, determining a parallax image, determining a first depth image based on the second point cloud data, the first acquisition parameters and the parallax image, constructing a three-dimensional reconstruction model based on the first depth image and the visual image, the three-dimensional reconstruction model represents a space structure of an area, which is blocked by the vehicle sight obstacle, in the visual image, performing image synthesis processing on the vehicle exterior environment image and the visual image, determining a synthesized image, and rendering the target image based on the sight direction of the target user, the second pose information and the three-dimensional reconstruction model.

In some embodiments, the determining a first depth map based on the second point cloud data, the first acquisition parameters and the disparity map includes determining a second depth map based on the second point cloud data, interpolating the second depth map to determine a third depth map, determining a fourth depth map based on the disparity map and the first acquisition parameters, and fusing the third depth map and the fourth depth map to determine the first depth map.

In some embodiments, the image synthesis processing is performed on the vehicle exterior environment image and the visual image to determine a synthesized image, and the method comprises the steps of determining the shielding degree of the vehicle sight obstacle relative to the target user based on the first depth map, determining the view angle of the target user based on the first pose information and the sight direction of the target user, determining fusion transparency based on the shielding degree and the view angle, and performing image fusion on the vehicle exterior environment image and the visual image based on the fusion transparency to determine the synthesized image.

In some embodiments, the image fusion of the vehicle exterior environment image and the visual image based on the fusion transparency comprises determining a first image fusion area based on the position of the gaze point of the target user under the vehicle coordinate system and the position of the vehicle vision obstacle under the vehicle coordinate system, adjusting the first image fusion area based on the gaze direction of the target user, the second pose information and the shielding degree, determining a second image fusion area, and performing image fusion of the vehicle exterior environment image and the visual image based on the fusion transparency and the second image fusion area.

In some embodiments, the blind area display method further comprises performing image restoration processing on the visual image based on the first depth map, the parallax map and the vehicle exterior environment image to determine a visual restoration image, and performing image synthesis processing on the vehicle exterior environment image and the visual image to determine a synthesized image, including performing image synthesis processing on the vehicle exterior environment image and the visual restoration image to determine the synthesized image.

In a second aspect, the embodiment of the application further provides a blind area display device, which comprises an acquisition module, a determination module and a processing module, wherein the acquisition module is used for acquiring vehicle external environment information corresponding to the sight line direction of a target user, visual information acquired by a head-mounted display device worn by the target user in the sight line direction and first pose information of the head-mounted display device, the vehicle external environment information comprises a vehicle external environment image which is positioned outside a vehicle sight line obstacle in the sight line direction, the visual information comprises a visual image, the first pose information is pose information of the head-mounted display device in a vehicle coordinate system, the determination module is used for determining the gazing position of a gazing point of the target user in the vehicle coordinate system based on the sight line direction and the first pose information, the processing module is used for processing the vehicle external environment image and the visual image based on the sight line direction of the target user and the first pose information in the condition that the gazing point of the target user is positioned on the vehicle sight line obstacle, and the determination module is used for displaying the target image on the head-mounted display device.

In a third aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a processor and a memory, where the memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory, so that the electronic device executes the blind area display method according to the first aspect.

In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, in which a computer program is stored, the computer program implementing the blind area display method according to the first aspect when executed by a processor.

Drawings

Fig. 1 is a flowchart illustrating steps of a blind area displaying method according to an embodiment of the present application.

Fig. 2 is an interaction schematic diagram of a head-mounted display device and a vehicle-mounted sensor according to an embodiment of the present application.

Fig. 3 is a flowchart illustrating a sub-step of a blind area displaying method according to an embodiment of the present application.

Fig. 4 is a flowchart illustrating another sub-step of the blind area displaying method according to an embodiment of the present application.

Fig. 5 is a schematic functional block diagram of a blind area display device according to an embodiment of the application.

Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the application.

Detailed Description

In order that the above-recited objects, features and advantages of the present application will be more clearly understood, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description. The embodiments of the present application and the features in the embodiments may be combined with each other without collision.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application, and the described embodiments are merely some, rather than all, of the embodiments of the present application.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the description of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

It is further intended that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The term "at least one" in the present application means one or more, and "a plurality" means two or more. "and/or" describes an association relationship of associated objects, meaning that there may be three relationships, e.g., A and/or B may mean that A alone exists, while A and B together exist, and B alone exists, where A, B may be singular or plural. The terms "first," "second," "third," "fourth" and the like in the description and in the claims and drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

In embodiments of the application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

The embodiment of the application provides a blind area display method which can be applied to one or more electronic devices. The electronic device may be a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and its hardware includes, but is not limited to, a Processor, a micro-program controller (Microprogrammed Control Unit, MCU), an Application SPECIFIC INTEGRATED Circuit (ASIC), a Programmable gate array (Field-Programmable GATE ARRAY, FPGA), a digital Processor (DIGITAL SIGNAL Processor, DSP), an embedded device, and the like. For example, the electronic device may be integrated in a vehicle, that is, an in-vehicle electronic device, or the electronic device may be a head-mounted display device communicatively connected to the vehicle, which is not limited by the embodiment of the present application.

The head-mounted display device may include an augmented reality device (e.g., AR glasses) and a mixed reality device (e.g., MR glasses). Wherein, AR equipment superimposes digital content on the basis of real world, fuses virtual information with real environment, provides extra information or interactive function. The MR device not only superimposes digital content into a real environment, but also enables real-time interaction and fusion of virtual objects and real objects. In some embodiments, the head mounted display device is exemplified as AR glasses.

The electronic device may be communicatively coupled with an in-vehicle sensor in the vehicle, which may include, but is not limited to, an image sensor, a lidar sensor, etc., which may be used to collect in-vehicle exterior environment information of the vehicle and transmit in-vehicle exterior environment information to the electronic device.

Wherein, the vehicle exterior environment information can reflect the running environment of the vehicle. The vehicle exterior environment information may include road images and laser point cloud data acquired by the vehicle-mounted sensor, but is not limited thereto.

When the target user wearing the head-mounted display device is located in the vehicle, the blind area exists when the target user looks at the condition outside the vehicle due to the arrangement of the vehicle body of the vehicle. In order to facilitate the target user to view the situation in the blind area in time and avoid the problem that the head-mounted display device is triggered to display the information of the surrounding environment outside the vehicle, which is shielded by the vehicle vision obstacle, no matter whether the vision of the driver looks at the vehicle vision obstacle or not, so that the visual naturalness of the user is poor, the application determines the looking position of the point of regard of the target user under the vehicle coordinate system by acquiring the information of the environment outside the vehicle in the blind area and the visual information acquired by the head-mounted display device in the visual direction, and triggering the image fusion of the external environment image and the visual image acquired by the head-mounted display device when the fixation point of the target user is determined to be positioned on the vehicle sight line obstacle, so that the visual consistency and the visual naturalness of the target user when the head-mounted display device is worn can be improved, the situation in the blind area in the sight line direction can be intuitively acquired by the target user through the head-mounted display device when the vehicle is driven, and the driving safety can be improved.

Referring to fig. 1, fig. 1 is a flowchart illustrating steps of an embodiment of a blind zone display method according to the present application. The order of the steps in the flow chart may be changed and some steps may be omitted according to different needs. The blind area display method may include the following steps.

Step 101, obtaining the vehicle exterior environment information corresponding to the sight line direction of the target user, the visual information acquired by the head-mounted display device worn by the target user in the sight line direction and the first pose information of the head-mounted display device.

In some embodiments, the target user is located within the vehicle, which may be a driver driving the vehicle. The first pose information is pose information of the head-mounted display device in a vehicle coordinate system, and the vehicle coordinate system is constructed based on a vehicle body of the vehicle, for example, the first pose information may be constructed by taking a geometric center of the vehicle as a coordinate origin, a traveling direction of the vehicle as an X-axis, a left-side direction of the vehicle as a Y-axis, and a top-end direction of the vehicle as a Z-axis. Taking the example of the head-mounted display device as AR glasses, the AR glasses may be connected to the vehicle by a wired or wireless manner, so that the vehicle data and the glasses data may be mutually transmitted. The AR glasses may include various sensors, for example, cameras, lidar, gyroscopes, accelerometers, etc., and the AR glasses may enable determination of their pose and acquisition of visual information through data acquired by the sensors.

In some embodiments, the in-vehicle exterior environment information may include an in-vehicle exterior environment image in a line-of-sight direction and outside of the vehicle line-of-sight obstacle. For example, an outside environment image of the vehicle is acquired by a camera provided on the vehicle.

In some embodiments, the vehicle exterior environment information may further include first point cloud data, where the first point cloud data is a first three-dimensional space data set generated by an environment acquisition device disposed on the vehicle when scanning an environment in which the vehicle is located. For example, a lidar sensor disposed on a vehicle generates a first three-dimensional spatial data set while scanning an ambient environment in which the vehicle is located. The first three-dimensional spatial data set characterizes information such as geometry, spatial location, and surface characteristics of objects in the surrounding environment in which the vehicle is located.

In some embodiments, the visual information may include visual images captured by the head mounted display device. For example, visual images are acquired by a camera provided on a head-mounted display device.

In some embodiments, the visual information may further include second point cloud data, which is a second three-dimensional spatial dataset generated by a sensor on the head-mounted display device while scanning the surrounding environment in which the target user is located. For example, a lidar sensor provided at the head-mounted display device generates a second three-dimensional spatial data set while scanning the surrounding environment in which the target user is located. The second three-dimensional spatial data set characterizes information such as geometry, spatial location, and surface characteristics of objects in the surrounding environment in which the target user is located.

The vehicle outside environment image outside the vehicle vision obstacle may refer to a blind area existing due to shielding of the vehicle vision obstacle, and the blind area corresponds to the vehicle outside environment image. The vehicle visual line obstacle may be set in advance, for example, the vehicle visual line obstacle may be set based on a blind area observation requirement when the driver drives the vehicle, and the vehicle visual line obstacle may include an obstacle that can block the driver's visual line, such as an a pillar, a B pillar, a C pillar, a head portion region (engine compartment portion region), a tail portion region (trunk portion region), or the like of the vehicle. The blind area is an area that is located outside the vehicle in the line of sight direction and is shielded by an obstacle. For example, the blind area is an area outside the vehicle when the driver looks into the a pillar on the left side of the vehicle when driving the vehicle. The following is illustrative of a vehicle line of sight obstruction as the a-pillar. In other embodiments, when the target user looks at the body parts such as the B-pillar, the C-pillar, the head, etc., a blind area is also present, and the processing may be performed with reference to the case when the target user looks at the a-pillar.

Referring to fig. 2, an in-vehicle sensor in a vehicle may include an environment acquisition device for acquiring an in-vehicle environment image and a radar device for acquiring first point cloud data, for example, the environment acquisition device may include one or more of a panoramic camera, a fisheye camera, a side camera, a pan around camera, and the like. The radar apparatus may include one or more of a lidar, a millimeter wave radar, and the like.

The data collected by the head mounted display device may include a visual image, second point cloud data, and first pose information. The visual image is an image acquired by the head-mounted display device in the sight line direction of the target user, the second point cloud data is a second three-dimensional space data set generated by the head-mounted display device under the condition of scanning the surrounding environment of the target user, and the first pose information is used for representing the position, the motion state and the space pose of the head-mounted display device under the vehicle coordinate system. Further, since the head-mounted display device is worn on the head of the target user, the first pose information moves along with the movement of the head of the target user, and can also be used for reflecting the position, the movement state and the spatial pose of the head of the target user under the vehicle coordinate system.

In the following, the description will be made taking the head-mounted display device as AR glasses and the blind area display method performed on the AR glasses as an example, it will be understood that in other embodiments, the blind area display method may be performed by a vehicle, and after the vehicle obtains the target image, the target image is sent to the AR glasses to display, or the blind area display method is performed by the AR glasses and the vehicle together.

The AR glasses may include a vision acquisition device, a display module, and a processing module, both communicatively coupled to the processing module. The vision acquisition device may include a camera unit (e.g., a binocular camera) for acquiring a vision image and a radar unit (e.g., a lidar) for acquiring second point cloud data.

The display module may include a first inertial measurement unit, which may include an accelerometer, a gyroscope, a magnetometer, and the like, an eye tracking module, and an optical display module. The accelerometer is used for measuring the acceleration of an object in three axial directions, the gyroscope is used for measuring the angular velocity of the object around the three axial directions (namely, 6-axis gyroscope data), and the magnetometer can provide the information of the earth magnetic field to assist in determining the direction. By fusion processing of the sensor data, pose information of the AR glasses can be determined, and then movement of the head of a user can be tracked. The eyeball tracking module is used for collecting eyeball motion data of a target user so as to determine the information such as the sight line direction, the fixation point and the like. The optical display module may be used to display the target image.

The processing module may include a data fusion module, a gesture parsing module, a device processor, and a rendering module. The data fusion module is used for integrating data acquired by the vehicle-mounted sensor and data acquired by the sensor on the head-mounted display device, so that fusion of multi-source data is realized. In the process of fusion of the multi-source data, the data collected by the vehicle-mounted sensor and the data collected by the sensor on the head-mounted display device can be integrated under the same coordinate system, for example, the data are integrated under the vehicle coordinate system. The gesture analysis module is used for processing the data acquired by the first inertial measurement unit to obtain first pose information, and processing the data acquired by the eyeball tracking module to determine the sight line direction of the target user, the gaze point (the intersection point of the sight line of the target user and the inner surface of the cabin) and the like. The rendering module is used for displaying the image to the target user. For example, the rendering module processes the obstacle data sent by the data fusion module and the 3D environment data sent by the device processor to obtain the AR enhanced picture. And then, the AR enhancement picture is sent to an optical display module for image display.

The vehicle-mounted sensor CAN send the acquired data to the data fusion module of the AR glasses through the CAN bus.

For example, the millimeter wave radar sends the acquired obstacle coordinates to the data fusion module. And the laser radar sends the acquired point cloud data to a data fusion module. The fisheye camera sends the acquired RGB image to the data fusion module. The vehicle-mounted sensor can also send running state data such as the speed, the steering angle and the like of the vehicle to the data fusion module.

In some embodiments, the step of obtaining a gaze direction of a target user wearing a head mounted display device located within the vehicle comprises:

(1) And acquiring an iris image and a pupil image acquired by the eyeball tracking module, and performing image processing on the iris image and the pupil image by utilizing a pupil edge detection algorithm to obtain a first position of the pupil and a second position of the iris. And then, calculating the first position and the second position to obtain the corresponding relation between the iris and the pupil. And determining coordinates of the cornea reflection points based on the corresponding relationship.

In some embodiments, the radius of the cornea may be determined using the following formula:

,

Wherein d is the distance between the iris and the pupil, and can be calculated by the first position and the second position. For the radius of the pupil,Is the radius of the cornea. In determiningThen, corneal reflection point coordinates are determined based on the radius of the pupil, the radius of the cornea, and pupil center coordinates.

(2) And extracting characteristic points in the iris image and the pupil image, wherein the characteristic points comprise pupil centers, iris textures, pupil edge points and the like. And obtaining a distance value between the pupil center and the cornea reflection point based on the characteristic points by using the deep learning model. And obtaining a sight line direction vector based on the distance value, the pupil center coordinate and the cornea reflection point coordinate. Wherein the line of sight direction is an indication direction of the line of sight direction vector. The gaze direction vector v may be expressed as v= (p 1-p 2)/d 1, p1 being pupil center coordinates, p2 being corneal reflection point coordinates, d1 being the distance from the pupil center to the corneal reflection point.

In some embodiments, a GazeNet model may be used to extract feature points in the iris image and the pupil image, for example, a deep neural network in the GazeNet model may be used to perform feature extraction on the iris image and the pupil image to obtain feature vectors. And then, processing the characteristic vector by using a regression or classification method to obtain the characteristic points. In other embodiments, the Pupil Labs algorithm may also be used to extract feature points in iris and Pupil images.

When the target user looks at the a-pillar, the a-pillar camera is required to capture an image of the outside environment of the vehicle in the blind area in order to display the image in the blind area to the target user. In order to ensure the image quality of the vehicle exterior environment image, the A-pillar camera can be calibrated before the A-pillar camera collects the image. The step of calibrating the a-pillar camera may include:

(1) And acquiring a plurality of groups of checkerboard images acquired by the A-pillar camera, and respectively extracting pixel coordinates of the checkerboard corners in each group of checkerboard images by using tools such as OpenCV, wherein the pixel coordinates are positions of each checkerboard corner in an image coordinate system. World coordinates of the checkerboard corner points of each group of checkerboards are obtained, and the world coordinates are positions of each checkerboard corner point in a world coordinate system. And calling calibrateCamera functions to process the pixel coordinates and the world coordinates to obtain an internal reference matrix and a distortion coefficient of the A-column camera.

(2) And acquiring the spatial position of the A-pillar camera under the vehicle coordinate system through laser ranging or structured light scanning. And performing spatial registration on the space by using the CAD model of the vehicle to obtain an external parameter matrix. The vehicle coordinate system is a coordinate system constructed based on the body of the vehicle.

(3) And correcting the image acquired by the A-column camera based on the internal reference matrix, the external reference matrix and the distortion coefficient.

In some embodiments, in order to enable the target image displayed in the head-mounted display device to accurately reflect the real situation in the blind area when the target user looks at the a pillar, the acquired vehicle exterior environment information and visual information need to be subjected to time synchronization processing.

In some embodiments, the head mounted display device may obtain second pose information based on sensors of a gyroscope, an accelerometer, or the like deployed by itself, the second pose information being pose information in a head mounted display device coordinate system. The head-mounted display device can further convert the second pose information into the first pose information under the vehicle coordinate system, and data integration is achieved by unifying the first pose information to the vehicle coordinate system.

Taking the head-mounted display device as an example of AR glasses, the conversion of the second pose information into the first pose information in the vehicle coordinate system may be achieved by spatially positioning the AR glasses in the vehicle coordinate system. For example, converting the second pose information to the first pose information in the vehicle coordinate system may be accomplished by determining a pose transformation matrix based on the first point cloud data and the second point cloud data, and converting the second pose information to the first pose information in the vehicle coordinate system based on the pose transformation matrix.

For example, the second pose information can be specifically converted into the first pose information under the vehicle coordinate system by the following manner, so that the spatial registration of the pose of the head-mounted display device and the vehicle coordinate system is realized, and the spatial consistency of the subsequent image fusion is ensured:

(1) And constructing a dense point cloud map of the environment outside the vehicle based on the first point cloud data sensed by the vehicle laser radar/millimeter wave radar.

(2) Determining a first pose transformation matrix of the AR glasses under the vehicle coordinate system based on the second point cloud data of the AR glasses and a dense point cloud map of the vehicle external environment, for example, extracting characteristic points from the second point cloud data, matching the extracted characteristic points from the dense point cloud map, establishing characteristic point pairs, adopting ICP (Iterative Closest Point) or NDT (Normal Distributions Transform) algorithm and the like, minimizing Euclidean distance between the characteristic point pairs, estimating the pose transformation matrix of the AR glasses relative to the vehicle, combining the data of a first inertial measurement unit of the AR glasses, and processing the estimated pose transformation matrix through Extended Kalman Filtering (EKF) or graph optimization (such as factor graph SLAM) to obtain the first pose transformation matrix of the AR glasses under the vehicle coordinate system.

In some embodiments, the visual image may be a depth image, or feature points may be extracted from the visual image, and matched with feature points extracted from a dense point cloud map, to create feature point pairs.

(3) Based on the second point cloud data of the AR glasses and the dense point cloud map, the first pose transformation matrix is optimized to obtain a second pose transformation matrix of the AR glasses under the vehicle coordinate system, for example, the second point cloud data of the AR glasses can be transformed under the vehicle coordinate system through the first pose transformation matrix, a corresponding area is searched in the dense point cloud map, and local ICP/NDT is adopted for further fine registration to obtain an optimized first pose transformation matrix (namely, the second pose transformation matrix).

(4) The second pose information is converted into first pose information in a vehicle coordinate system based on the second pose transformation matrix.

In some embodiments, a second inertial measurement unit may also be provided within the vehicle for acquiring body attitude. The vehicle body posture comprises a pitch angle, a roll angle, a yaw angle and the like of the vehicle.

Based on the method, the second pose information can be converted into the first pose information under the vehicle coordinate system based on the vehicle body pose and the second pose transformation matrix, so that the influence of vehicle body motion on the first point cloud data acquisition can be overcome, and the accuracy of the first pose information is further improved.

After the first pose information is obtained, the target image displayed by the head-mounted display device can be rendered based on the first pose information, and the target image with better quality is obtained.

Step 102, determining the fixation position of the fixation point of the target user under the vehicle coordinate system based on the sight line direction and the first pose information.

In some embodiments, the first pose information is used to characterize a position and pose of the head mounted display device in a vehicle coordinate system, and after determining the first pose information, a gaze location of a gaze point of the target user in the vehicle coordinate system may be determined based on the gaze direction and the position of the head mounted display device in the vehicle coordinate system.

For example, the gaze location P _gaze of the target user's gaze point in the vehicle coordinate system may be determined by P _gaze=P_g +d2 Xv, where P _g is the position of the head mounted display device in the vehicle coordinate system, P _g may be determined based on the first pose information, d2 is the gaze distance, and v is the gaze direction vector. The gaze distance d2 may be determined based on the first depth map. For example, a depth value of a corresponding pixel in the first depth map may be determined as the gaze distance by determining the corresponding pixel in the gaze direction.

In some embodiments, the first depth map may refer to a depth map constructed based on data acquired by the head mounted display device. For example, the head-mounted display device comprises a binocular vision acquisition device (e.g., a binocular camera), and the first depth map can be constructed by acquiring first acquisition parameters based on a vision image, wherein the first acquisition parameters are parameters for the binocular vision acquisition device to acquire the vision image, performing left and right image pixel matching processing on the vision image to determine a parallax map, and determining the first depth map based on second point cloud data, the first acquisition parameters and the parallax map.

In some embodiments, determining the first depth map based on the second point cloud data, the first acquisition parameters, and the disparity map may include determining the second depth map based on the second point cloud data, e.g., the second point cloud data acquired by the head-mounted display device may be projected onto a pixel plane to construct a sparse depth map, i.e., the second depth map is a sparse depth map, interpolating the second depth map to obtain a third depth map, e.g., the sparse depth map may be interpolated by a bilateral Filtering (Bilateral Filtering) algorithm or a Guided Filtering (Guided Filtering) algorithm to determine a dense depth map (i.e., the third depth map), determining the fourth depth map based on the disparity map and the first acquisition parameters, e.g., the disparity map may be converted to the fourth depth map based on a disparity versus depth relationship in combination with the first acquisition parameters, and in particular the fourth depth mapCan be represented by the following expression:

,

Wherein f is the focal length of the camera in the first acquisition parameter, b is the binocular baseline length in the first acquisition parameter, Is a disparity map;

After the third depth map and the fourth depth map are obtained, the third depth map and the fourth depth map can be fused to determine the first depth map.

According to the application, through fusion point cloud depth and binocular depth, the depth map of depth information under the scene where the current head-mounted display equipment is positioned can be accurately determined, and the subsequent accurate determination of the shielding degree of the vehicle sight obstacle relative to the target user is facilitated.

Specifically, for each pixelAccording to the point cloud density and binocular parallax consistency of the second point cloud data, confidence weight of the third depth map is respectively determinedConfidence weight for fourth depth mapSatisfies the following conditions. Then carrying out fusion treatment according to the following formula:

,

Wherein, the As a result of the first depth map being a first depth map,For the third depth-map of the image,Is the fourth depth map. When the reliability of the point cloud data is higher,The value of (c) may be set higher and when the point cloud data is missing or noisy,The value of (2) may be set higher. In the fusion processing process, guided filtering or bilateral filtering can be adopted to ensure that the depth edge is consistent with the image structure, so as to avoid blurring and jumping.

It can be understood that the visual images acquired by the binocular vision acquisition device include left and right images, and performing left and right image pixel matching processing on the visual images may refer to calculating the parallax of each pixel by using a Semi-global block matching algorithm (Semi-Global Block Matching, SGBM) on the left and right images, so that a parallax map may be constructed based on the parallax. For example, the disparity value of each pixel is coded in gray or color to form a disparity map.

Step 103, in a case where it is determined that the gaze point of the target user is located on the vehicle vision obstacle based on the gaze position, image processing is performed on the vehicle exterior environment image and the visual image based on the gaze direction of the target user and the first pose information, and the target image is determined.

In some embodiments, after determining the gaze location of the target user's gaze point in the vehicle coordinate system, it may be further determined whether the target user's gaze point is located on a vehicle line of sight obstacle based on the gaze location. For example, it may be determined whether the gaze point of the target user is located on the vehicle line of sight obstacle by comparing the gaze position of the target user in the vehicle coordinate system with the position area of the vehicle line of sight obstacle in the vehicle coordinate system. If the gaze point of the target user is located on the vehicle vision obstacle, image processing is continuously performed on the vehicle exterior environment image and the visual image based on the gaze direction and the first pose information of the target user, so that image fusion is ensured only when the driver gazes the vehicle vision obstacle which can generate blind areas, and space consistency and visual naturalness are improved. If the gaze point of the target user is not located on the vehicle gaze obstacle, the process may end, i.e., image processing of the vehicle exterior environment image and the visual image based on the gaze direction and the first pose information of the target user is not performed.

In some embodiments, if the gaze point of the target user is located on a vehicle gaze obstacle, taking the vehicle gaze obstacle as an a-pillar example, the vehicle exterior environment image is an image in a blind zone when the target user looks at the a-pillar, and the visual image is an image acquired by the head mounted display device when the target user looks at the a-pillar. And carrying out image processing on the external environment image and the visual image of the vehicle based on the sight direction and the first pose information of the target user, determining the target image, and subsequently displaying the target image through the head-mounted display device, so that the target user can conveniently watch the target image through the head-mounted display device, and the situation in the blind area is known.

The specific steps of determining the target image on the basis of the gaze direction of the target user and the first pose information for performing image processing on the vehicle exterior environment image and the visual image are described in detail below, and are not repeated here.

Step 104, displaying the target image on the head-mounted display device.

After the target image is obtained, the target image can be displayed on the head-mounted display device, so that a target user can know the situation in the blind area, and the driving safety is improved.

In some embodiments, if the blind area display method is performed by the vehicle-mounted electronic device integrated in the vehicle, after the target image is obtained, the target image may be sent to the head-mounted display device by the vehicle-mounted electronic device for display.

Compared with the prior art, the embodiment of the application has the following advantages:

On the one hand, by determining the fixation position of the fixation point of the target user under the vehicle coordinate system, the image fusion of the external environment image and the visual image acquired by the head-mounted display device is conveniently triggered when the fixation point of the target user is determined to be positioned on the vehicle sight obstacle, so that the visual consistency and visual naturalness of the target user when wearing the head-mounted display device can be improved, the situation in a blind area in the sight direction can be intuitively acquired by the target user through the head-mounted display device when driving the vehicle, and the driving safety can be improved.

On the other hand, the first pose information of the head-mounted display device under the vehicle coordinate system and the sight direction of the target user are used for fusing the external environment image and the visual image, so that the spatial registration of the pose of the head-mounted display device and the vehicle coordinate system is realized, and the spatial consistency of the image fusion is ensured.

Referring to fig. 3, fig. 3 is a schematic flow chart of a blind area displaying method according to an embodiment of the application. This embodiment is a detailed description of step 103. The specific steps can include:

Step 1031, based on the visual image, obtains a first acquisition parameter.

In some embodiments, the binocular vision acquisition apparatus may be a binocular camera and the first acquisition parameters may include an extrinsic matrix, an intrinsic matrix, a focal length, a binocular baseline length, and the like of the binocular camera.

Step 1032, performing left-right pixel matching processing on the visual image to determine a disparity map.

In some embodiments, a left-right pixel matching process may be performed on the visual image to determine a disparity map. For example, based on the focal length and the binocular base line length, a parallax value of each pixel in the left image and the right image is calculated, and a parallax map is determined based on the parallax value of each pixel in the left image and the right image.

Step 1033, determining a first depth map based on the second point cloud data, the first acquisition parameters, and the disparity map.

The determination of the first depth map may be described with reference to the above embodiments, and will not be described herein.

Step 1034, constructing a three-dimensional reconstruction model based on the first depth map and the visual image, the three-dimensional reconstruction model characterizing a spatial structure of an area within the visual image that is occluded by the vehicle line-of-sight obstacle.

In some embodiments, constructing the three-dimensional reconstruction model based on the first depth map and the visual image may specifically include:

(1) For each pixel in the visual image, back projecting the corresponding depth in the first depth map to a three-dimensional space to generate three-dimensional point cloud data;

In some embodiments, for occluded or deep absent areas, a deep learning algorithm may be employed for repair, which may include DEEPFILL V algorithm, PATCHMATCH algorithm. For example, for each pixel of the occluded region, a block of known regions that is most similar to the neighborhood may be searched based on the PATCHMATCH algorithm, filling in the pixel values.

(2) And processing the three-dimensional point cloud data by using a preset surface reconstruction algorithm (such as Poisson Surface Reconstruction algorithm) to determine a three-dimensional reconstruction model.

And 1035, performing image synthesis processing on the external environment image and the visual image of the vehicle to determine a synthesized image.

In some embodiments, the image synthesis processing of the vehicle exterior environment image and the visual image may specifically include the steps of:

(1) Based on the first depth map, a degree of occlusion of the vehicle vision obstruction relative to the target user is determined.

In some embodiments, the occlusion degree may be determined by the first depth map or the second point cloud data. For example, taking a vehicle sight line obstacle as an A column as an example, the nearest distance between the sight line direction and the surface of the A column can be calculated based on the first depth map or the second point cloud data, or the pixel duty ratio of the A column in the current field of view can be counted, and then the shielding degree of the A column relative to the target user can be determined based on the nearest distance or the pixel duty ratio. For example, the occlusion levels corresponding to different nearest distances or pixel ratios may be predefined, and then scaled based on the nearest distances or pixel ratios.

(2) And determining the visual angle of the target user based on the second pose information and the sight line direction of the target user.

In some embodiments, the cone vertex and direction may be determined based on the second pose information of the head-mounted display device, and the cone range may be calculated in combination with the field of view (FOV) of the head-mounted display device, and the cone range may be corrected by eye tracking to obtain the final cone range (i.e., the view angle of the target user).

(3) Determining fusion transparency based on the shielding degree and the viewing angle;

In some embodiments, the blended transparency may refer to transparency when the virtual and real contents are superimposed, and the blended transparency may be dynamically adjusted based on the occlusion degree and the viewing angle, for example, the blended transparency α: α=f (θ, S _degree) is set according to the user viewing angle θ and the occlusion degree S _degree.

For another example, when the overlap ratio of the user viewing angle and the virtual object is high (i.e. the user is opposite to the virtual object), the fusion transparency can be properly improved, the virtual-real fusion feeling can be enhanced, and when the user viewing angle deviates from the virtual object, the fusion transparency can be properly reduced, and the visual interference can be reduced.

For example, the higher the shielding degree is, the fusion transparency can be properly improved, the virtual-real fusion feeling is enhanced, and the lower the shielding degree is, the fusion transparency can be properly reduced, and the visual interference is reduced.

In some embodiments, the fusion transparency may also be adjusted to account for ambient light, for example, when the light is strong, the fusion transparency may be suitably increased to avoid excessive protrusion. When the illumination is weak, the fusion transparency can be properly reduced, so that the virtual content is more clearly visible.

(4) And carrying out image fusion on the vehicle exterior environment image and the visual image based on the fusion transparency, and determining a composite image.

In some embodiments, after determining the fusion transparency, the image within the fusion area and the visual image may be image fused to determine the composite image. The composite image I _blend can be expressed as:

I_blend=α×I_warp+(1-α) ×I_G,

Wherein, alpha is fusion transparency, I _G is visual image, and I _warp is vehicle exterior environment image.

In some embodiments, in order to eliminate spatial misalignment between the vehicle exterior environment image and the visual image, and avoid the problems of misalignment of the overlapping area, hard segmentation boundary and the like of the composite image, the image synthesis processing for the vehicle exterior environment image and the visual image may further specifically include the following steps:

(1) An in-vehicle exterior environment image is image aligned based on the visual image, and an in-vehicle exterior environment image is determined.

Specifically, taking an external environment image of a vehicle as an example, a second acquisition parameter of an environment acquisition device of a vehicle can be acquired, the second acquisition parameter can comprise an internal parameter matrix, external parameter data and a distortion coefficient of an A-pillar camera, the external environment image is subjected to image correction based on the second acquisition parameter to obtain a corrected external environment image of the vehicle, for example, the external environment image of the vehicle is subjected to image correction by utilizing the internal parameter matrix and the distortion coefficient to obtain a corrected external environment image of the vehicle, and the corrected external environment image of the vehicle is subjected to image alignment based on a visual image to determine an aligned parking space environment image.

In some embodiments, the first set of feature points of the corrected vehicle exterior environment image and the second set of feature points of the visual image may be extracted first. The method comprises the steps of obtaining a first characteristic point set, a second characteristic point set, a single adaptation matrix and a perspective transformation, wherein the first characteristic point set can be structural point characteristics of an obstacle, the second characteristic point set can be structural point characteristics of an A column, characteristic matching is conducted on the first characteristic point set and the second characteristic point set to obtain characteristic matching pairs, the single adaptation matrix is calculated based on the characteristic matching pairs, the single adaptation matrix characterizes the corresponding relation between pixel points of an external environment image of a vehicle and pixel points of a visual image, and finally perspective transformation is conducted on the external environment image of the vehicle by the aid of the single adaptation matrix to determine an aligned parking space environment image.

(2) Based on the first depth map, a degree of occlusion of the vehicle vision obstruction relative to the target user is determined.

(3) And determining the visual angle of the target user based on the second pose information and the sight line direction of the target user.

(4) Based on the occlusion degree and the viewing angle, the fusion transparency is determined.

(5) And (5) performing image fusion on the external environment image and the visual image of the aligned vehicle based on the fusion transparency, and determining a composite image.

In some embodiments, after determining the fusion transparency, the aligned image and the visual image within the fusion area may be image fused to determine the composite image. The composite image I _blend can be expressed as:

I_blend=α×I_warp'+(1-α) ×I_G,

Where α is the fusion transparency, I _G is the visual image, and I _warp' is the alignment vehicle exterior environment image. Taking a vehicle sight line obstacle as an A column as an example, I _warp' is an aligned A column external environment image.

In some embodiments, aligning the vehicle exterior environment image and the visual image based on the fusion transparency may specifically include determining a first image fusion area based on a gaze location of a gaze point of a target user in a vehicle coordinate system and a location of a vehicle vision obstruction in the vehicle coordinate system, adjusting the first image fusion area based on a gaze direction of the target user, first pose information, and a degree of occlusion, determining a second image fusion area, and aligning the vehicle exterior environment image and the visual image based on the fusion transparency and the second image fusion area.

Taking a vehicle line of sight obstacle as an example for the a-pillar, the first fusion zone may be determined by:

Calculating the gazing position of the gazing point of the target user under a vehicle coordinate system in real time according to the sight direction of the target user and the first pose information;

Judging whether the current gaze point passes through the A column shielding area or not by utilizing the vehicle CAD model and the A column space position;

And if the first fusion area passes through the A column shielding area, projecting a space area corresponding to the A column on a display plane of the head-mounted display device, and determining the first fusion area.

In some embodiments, the shape and size of the first fused region may change in real-time with the gaze direction of the target user, the first pose information, and the occlusion degree. For example, the shape and size of the first fusion area may be adjusted by constructing corresponding change rules in advance, the second image fusion area may be determined, and the corresponding change rules may be set based on the line of sight direction of the target user, the first pose information, and the shielding degree.

Step 1036, performing image rendering on the synthetic image based on the sight direction of the target user, the first pose information and the three-dimensional reconstruction model, and determining the target image.

According to the method, based on the sight direction of the target user, the first pose information and the three-dimensional reconstruction model, the image rendering is carried out on the synthesized image, the target image is determined, the real-time and accurate rendering of the synthesized image of the spatial structure of the area which contains the shielding of the vehicle sight obstacle according to the sight and the head movement of the target user can be realized, and the augmented reality experience is improved.

In some embodiments, taking a vehicle sight line obstacle as an a-pillar as an example, performing image rendering on a synthetic image based on a sight line direction of a target user, first pose information and a three-dimensional reconstruction model, and determining a target image may specifically include:

(1) Determining a gaze point (an intersection point of the gaze direction and the A column shielding region) of a target user according to the first pose information and the gaze direction, and determining a projection region of the A column shielding region under the current view angle based on the gaze point, wherein the projection region is used as a virtual-real superposition region;

in some embodiments, the virtual-real superposition area may be used for superposition of the three-dimensional reconstruction model.

(2) Establishing a mapping relation between the three-dimensional reconstruction model and the synthetic image;

In some embodiments, the synthetic image is used as a texture, mapped onto a visible surface of the three-dimensional reconstruction model to obtain texture coordinates, and a mapping relationship between the three-dimensional reconstruction model and the synthetic image is established based on the texture coordinates.

(3) And performing image rendering on the synthetic image based on the three-dimensional reconstruction model, the virtual-real superposition area and the mapping relation between the three-dimensional reconstruction model and the synthetic image, and determining a target image.

In some embodiments, image rendering of the composite image may be performed by a GPU of the head mounted display device, and may employ inter-frame prediction techniques to reduce display latency of the image.

Compared with the prior art, the embodiment has at least the following advantages:

According to the embodiment of the application, the image correction is carried out on the vehicle exterior environment image based on the second acquisition parameters, so that the distortion corrected vehicle exterior environment image can be obtained, and the subsequent distortion occurrence during the display of the target image in the head-mounted display device is avoided. By performing image alignment on the vehicle exterior environment image and the visual image, the image overlapping or division and the like can be prevented when the image is displayed in the head-mounted display device. And (3) carrying out image fusion based on the determined fusion transparency, realizing real-time adjustment of fusion effect according to the visual angle, shielding degree and the like of a driver, and enhancing virtual-real fusion feeling. And finally, rendering the synthesized image based on the sight direction of the target user, the first pose information and the three-dimensional reconstruction model to obtain a target image which can be directly displayed on the head-mounted display device, and ensuring the spatial consistency of virtual-real fusion.

Referring to fig. 4, fig. 4 is a schematic flow chart of a blind area displaying method according to an embodiment of the application. This embodiment is a detailed description of step 103.

Step 1031, based on the visual image, obtains a first acquisition parameter.

Step 1037, performing image restoration processing on the visual image based on the first depth map, the parallax map and the vehicle exterior environment image, and determining a visual restoration image.

In some embodiments, due to occlusion of the vehicle line of sight obstacle, a region of the visual image that is occluded by the vehicle line of sight obstacle may be present, and the content within the occluded region may be restored based on the first depth map, the disparity map, and the vehicle exterior environment image, and a restored visual image (i.e., a visual restoration image) may be determined.

In some embodiments, spatial uniformity and visual integrity of blind areas may be improved by image restoration of visual images followed by composition.

Step 1038, performing image synthesis processing on the vehicle exterior environment image and the visual repair image, and determining a synthesized image.

In some embodiments, the image synthesis processing of the vehicle exterior environment image and the visual repair image may refer to step 1035, which is not described herein.

As shown in fig. 5. The embodiment of the present application further provides a blind area display device 50, where the blind area display device 50 may include:

the obtaining module 501 is configured to obtain vehicle external environment information corresponding to a sight line direction of a target user, visual information collected in the sight line direction by a head-mounted display device worn by the target user, and first pose information of the head-mounted display device.

The vehicle exterior environment information comprises an image of the vehicle exterior environment in the sight line direction and outside the vehicle sight line obstacle, the visual information comprises a visual image, and the first pose information is pose information of the head-mounted display device in a vehicle coordinate system.

A determining module 502 is configured to determine a gaze location of the gaze point of the target user in the vehicle coordinate system based on the gaze direction and the first pose information.

A processing module 503, configured to, in a case where it is determined that the gaze point of the target user is located on the vehicle vision obstacle based on the gaze location, perform image processing on the vehicle exterior environment image and the visual image based on the gaze direction of the target user and the first pose information, and determine a target image.

A display module 504 for displaying the target image on the head mounted display device.

The modules may be programmable software instructions stored in a memory and executable by a processor call. It will be appreciated that in other embodiments, the modules may be program instructions or firmware (firmware) that are resident in the processor.

Fig. 6 is a schematic diagram of an embodiment of an electronic device according to the present application. The electronic device 100 comprises a memory 20, a processor 30 and a computer program 40 stored in the memory 20 and executable on the processor 30. The steps in the above-described blind zone display method embodiment are implemented by the processor 30 when executing the computer program 40.

By way of example, the computer program 40 may likewise be partitioned into one or more modules/units that are stored in the memory 20 and executed by the processor 30. The one or more modules/units may be a series of computer program instruction segments capable of performing particular functions for describing the execution of the computer program 40 in the electronic device 100.

It will be appreciated by those skilled in the art that the schematic diagram is merely an example of the electronic device 100 and is not meant to be limiting of the electronic device 100, and may include more or fewer components than shown, or may combine certain components, or different components, e.g., the electronic device 100 may also include a display device, an input-output device, a network access device, a bus, etc.

In some embodiments, the electronic device 100 may be an in-vehicle electronic device integrated in a vehicle, or a functional component integrated in a head mounted display device.

The Processor 30 may be a central processing unit (Central Processing Unit, CPU), a graphics Processor (Graphics Processing Unit, GPU), but may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application SPECIFIC INTEGRATED Circuits (ASICs), off-the-shelf Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general purpose processor may be a microprocessor, a single-chip microcomputer or the processor 30 may be any conventional processor or the like.

The memory 20 may be used to store computer programs 40 and/or modules/units, and the processor 30 implements various functions of the electronic device 100 by running or executing the computer programs and/or modules/units stored in the memory 20, as well as invoking data stored in the memory 20. The memory 20 may mainly include a storage program area that may store an operating system, application programs required for at least one function (such as a sound playing function, an image playing function, etc.), etc., and a storage data area that may store data created according to the use of the electronic device 100 (such as audio data), etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SMART MEDIA CARD, SMC), secure Digital (SD) card, flash memory card (FLASH CARD), at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device.

The modules/units integrated with the electronic device 100 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as a stand alone product. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, where the computer program, when executed by a processor, may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

The embodiment of the application also provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the blind area display method is realized when the computer program is executed by a processor.

The embodiment of the present application also provides a vehicle, which may include the blind area display device 50 provided in the above embodiment, or the electronic apparatus 100 provided in the above embodiment.

In the several embodiments provided in the present application, it should be understood that the disclosed electronic device and method may be implemented in other manners. For example, the above-described embodiments of the electronic device are merely illustrative, and for example, the division of the units is merely a logical function division, and there may be other manners of division when actually implemented.

In addition, each functional unit in the embodiments of the present application may be integrated in the same processing unit, or each unit may exist alone physically, or two or more units may be integrated in the same unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.

It will be evident to those skilled in the art that the application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the embodiments are to be considered in all respects as illustrative and not restrictive.

Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present application and not for limiting the same, and that it should be understood by those skilled in the art that the technical solution of the present application may be modified or substituted without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A blind area display method, characterized in that the blind area display method comprises:

Acquiring vehicle external environment information corresponding to a sight line direction of a target user, visual information acquired in the sight line direction by a head-mounted display device worn by the target user and first pose information of the head-mounted display device, wherein the vehicle external environment information comprises a vehicle external environment image which is positioned outside a vehicle sight line obstacle in the sight line direction, the visual information comprises a visual image, and the first pose information is pose information of the head-mounted display device in a vehicle coordinate system;

Determining a gaze location of a gaze point of the target user in the vehicle coordinate system based on the gaze direction and the first pose information;

in the case that the gaze point of the target user is determined to be located on the vehicle vision obstacle based on the gaze position, performing image processing on the vehicle exterior environment image and the visual image based on the gaze direction of the target user and the first pose information, and determining a target image;

the target image is displayed on the head-mounted display device.

2. The blind zone display method according to claim 1, wherein the vehicle exterior environment information further includes first point cloud data, the visual information further includes second point cloud data, and the blind zone display method further includes, before acquiring the first pose information of the head-mounted display device:

Acquiring second pose information of the head-mounted display device, wherein the second pose information is pose information under a coordinate system of the head-mounted display device;

determining a pose transformation matrix based on the first point cloud data and the second point cloud data;

the second pose information is converted into the first pose information in the vehicle coordinate system based on the pose transformation matrix.

3. The blind area display method according to claim 2, wherein the visual information further includes second point cloud data, the determining a target image by performing image processing on the vehicle exterior environment image and the visual image based on the line-of-sight direction of the target user and the first pose information includes:

determining a first acquisition parameter based on the visual image;

performing left and right image pixel matching processing on the visual image to determine a parallax image;

Determining a first depth map based on the second point cloud data, the first acquisition parameters and the disparity map;

Constructing a three-dimensional reconstruction model based on the first depth map and the visual image, wherein the three-dimensional reconstruction model characterizes a spatial structure of a region, which is shielded by the vehicle sight barrier, in the visual image;

performing image synthesis processing on the vehicle exterior environment image and the visual image to determine a synthesized image;

And performing image rendering on the synthesized image based on the sight direction of the target user, the second pose information and the three-dimensional reconstruction model, and determining the target image.

4. The blind zone display method according to claim 3, wherein said determining a first depth map based on said second point cloud data, said first acquisition parameters, and said disparity map comprises:

Determining a second depth map based on the second point cloud data;

Performing interpolation processing on the second depth map to determine a third depth map;

determining a fourth depth map based on the disparity map and the first acquisition parameters;

And carrying out fusion processing on the third depth map and the fourth depth map, and determining the first depth map.

5. The blind area display method according to claim 3, wherein said performing image synthesis processing on said vehicle exterior environment image and said visual image to determine a synthesized image includes:

determining a degree of occlusion of the vehicle line of sight barrier relative to the target user based on the first depth map;

Determining a viewing angle of the target user based on the first pose information and a viewing direction of the target user;

determining fusion transparency based on the occlusion degree and the viewing angle;

And carrying out image fusion on the vehicle exterior environment image and the visual image based on the fusion transparency, and determining the synthetic image.

6. The blind area display method according to claim 5, wherein said image fusion of said vehicle exterior environment image and said visual image based on fusion transparency includes:

Determining a first image fusion area based on a position of a gaze point of the target user in the vehicle coordinate system and a position of the vehicle line of sight obstacle in the vehicle coordinate system;

Adjusting the first image fusion area based on the sight direction of the target user, the second pose information and the shielding degree to determine a second image fusion area;

And carrying out image fusion on the vehicle exterior environment image and the visual image based on the fusion transparency and the second image fusion area.

7. The blind zone display method according to claim 3, characterized in that the blind zone display method further comprises:

performing image restoration processing on the visual image based on the first depth map, the parallax map and the vehicle exterior environment image, and determining a visual restoration image;

the image synthesis processing is performed on the vehicle exterior environment image and the visual image, and the synthetic image is determined, including:

And performing image synthesis processing on the vehicle exterior environment image and the visual restoration image, and determining the synthesized image.

8. A blind area display device, characterized in that the blind area display device includes:

The system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring vehicle external environment information corresponding to the sight direction of a target user, visual information acquired by a head-mounted display device worn by the target user in the sight direction and first pose information of the head-mounted display device, the vehicle external environment information comprises a vehicle external environment image which is positioned outside a vehicle sight barrier in the sight direction, the visual information comprises a visual image, and the first pose information is pose information of the head-mounted display device in a vehicle coordinate system;

a determining module, configured to determine a gaze location of a gaze point of the target user in the vehicle coordinate system based on the gaze direction and the first pose information;

A processing module, configured to, in a case where it is determined that the gaze point of the target user is located on the vehicle vision obstacle based on the gaze location, perform image processing on the vehicle exterior environment image and the visual image based on the gaze direction of the target user and the first pose information, and determine a target image;

And the display module is used for displaying the target image on the head-mounted display device.

9. An electronic device comprising a processor and a memory, wherein the memory is configured to store instructions, and wherein the processor is configured to invoke the instructions in the memory, such that the electronic device performs the blind zone display method of any of claims 1 to 7.

10. A computer-readable storage medium, in which a computer program is stored, which when executed by a processor implements the blind zone display method according to any one of claims 1 to 7.