[go: up one dir, main page]

CN115883816A - Display method, device, head-mounted display device, and storage medium - Google Patents

Display method, device, head-mounted display device, and storage medium Download PDF

Info

Publication number
CN115883816A
CN115883816A CN202211494355.1A CN202211494355A CN115883816A CN 115883816 A CN115883816 A CN 115883816A CN 202211494355 A CN202211494355 A CN 202211494355A CN 115883816 A CN115883816 A CN 115883816A
Authority
CN
China
Prior art keywords
image
real scene
scene image
user
brightness
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211494355.1A
Other languages
Chinese (zh)
Inventor
杨青河
王玉影
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Goertek Techology Co Ltd
Original Assignee
Goertek Techology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Goertek Techology Co Ltd filed Critical Goertek Techology Co Ltd
Priority to CN202211494355.1A priority Critical patent/CN115883816A/en
Publication of CN115883816A publication Critical patent/CN115883816A/en
Pending legal-status Critical Current

Links

Images

Landscapes

  • Controls And Circuits For Display Device (AREA)

Abstract

本发明公开了一种显示方法、装置、头戴显示设备及存储介质,涉及头戴显示设备领域,方法应用于安装有采集真实场景图像的第一相机的头戴显示设备。首先,根据用户在真实场景图像中的感兴趣区域的图像内容确定采集真实场景图像的第一相机的曝光,然后,将第一相机基于目标曝光采集的目标真实场景图像进行显示。从而通过基于感兴趣区域的图像内容确定的曝光采集符合用户亮度需求的真实场景图像,解决人眼视觉区域因曝光不合适导致头戴显示设备显示的画面亮度偏暗和过亮,从而不满足用户人眼对于画面亮度的观看需要的技术问题,通过显示亮度合适的真实场景画面来提升头戴显示设备的VST视频透视显示效果,提升头戴显示设备用户的使用体验。

Figure 202211494355

The invention discloses a display method, a device, a head-mounted display device, and a storage medium, and relates to the field of head-mounted display devices. The method is applied to a head-mounted display device equipped with a first camera for collecting real scene images. First, the exposure of the first camera that captures the real scene image is determined according to the image content of the user's interest region in the real scene image, and then the target real scene image collected by the first camera based on the target exposure is displayed. Therefore, through the exposure determined based on the image content of the region of interest, the real scene image that meets the user's brightness requirements is collected, and the brightness of the screen displayed by the head-mounted display device is too dark and too bright due to improper exposure of the human eye's visual area, which does not satisfy the user. The technical problem that the human eye needs to watch the brightness of the picture, by displaying the real scene picture with appropriate brightness to improve the VST video perspective display effect of the head-mounted display device, and improve the user experience of the head-mounted display device.

Figure 202211494355

Description

显示方法、装置、头戴显示设备及存储介质Display method, device, head-mounted display device, and storage medium

技术领域technical field

本发明涉及头戴显示技术领域,尤其涉及一种显示方法、显示装置、头戴显示设备及计算机可读存储介质。The present invention relates to the field of head-mounted display technology, and in particular to a display method, a display device, a head-mounted display device, and a computer-readable storage medium.

背景技术Background technique

头戴显示设备作为目前主流的交互工具,其VST(Vi deo See-Through,视频透视)的工作原理如下,用户戴上头戴显示设备后,头戴显示设备上的相机采集到真实场景图像,然后将相机采集得到的真实场景图像通过头戴显示设备的显示屏幕上展示以供用户观看。The head-mounted display device is currently the mainstream interactive tool. The working principle of its VST (Vi deo See-Through) is as follows. After the user wears the head-mounted display device, the camera on the head-mounted display device collects real scene images. Then, the real scene image collected by the camera is displayed on the display screen of the head-mounted display device for the user to watch.

但是目前头戴显示设备的视频透视存在以下问题:用户人眼观看到的画面偏暗和过亮,进而影响头戴显示设备用户的适用体验。However, the current video see-through of the head-mounted display device has the following problems: the picture viewed by the user's human eyes is too dark and too bright, which in turn affects the application experience of the head-mounted display device user.

发明内容Contents of the invention

本发明的主要目的在于提供一种显示方法、显示装置、头戴显示设备及计算机可读存储介质,旨在解决现有技术中头戴显示设备显示的画面亮度偏暗和过亮,不满足用户人眼对于画面亮度的观看需要的技术问题。The main purpose of the present invention is to provide a display method, a display device, a head-mounted display device, and a computer-readable storage medium, aiming at solving the problem that the screen brightness displayed by the head-mounted display device in the prior art is too dark and too bright, which does not satisfy users. The technical problem that the human eye needs to watch the brightness of the picture.

为实现上述目的,本发明提供一种显示方法,所述显示方法应用于头戴显示设备,在所述头戴显示设备上安装有采集真实场景图像的第一相机,所述显示方法包括:In order to achieve the above object, the present invention provides a display method, the display method is applied to a head-mounted display device, a first camera for collecting real scene images is installed on the head-mounted display device, and the display method includes:

确定用户在真实场景图像中的感兴趣区域;Determine the user's region of interest in the real scene image;

基于所述感兴趣区域的图像内容确定所述第一相机的目标曝光;determining a target exposure for the first camera based on image content of the region of interest;

获取所述第一相机基于所述目标曝光采集的目标真实场景图像,将所述目标真实场景图像进行显示。Acquiring the target real scene image collected by the first camera based on the target exposure, and displaying the target real scene image.

可选地,所述确定用户在真实场景图像中的感兴趣区域的步骤,包括:Optionally, the step of determining the region of interest of the user in the real scene image includes:

获取用户的眼睛图像,并根据所述眼睛图像确定用户在真实场景图像中的感兴趣区域。An eye image of the user is acquired, and an area of interest of the user in the real scene image is determined according to the eye image.

可选地,所述根据所述眼睛图像确定用户在真实场景图像中的感兴趣区域的步骤,包括:Optionally, the step of determining the user's region of interest in the real scene image according to the eye image includes:

根据所述眼睛图像计算出用户的注视点坐标,基于所述注视点坐标确定用户在真实场景图像中的感兴趣区域。The gaze point coordinates of the user are calculated according to the eye image, and the region of interest of the user in the real scene image is determined based on the gaze point coordinates.

可选地,所述基于所述注视点坐标确定用户在真实场景图像中的感兴趣区域的步骤,包括:Optionally, the step of determining the user's region of interest in the real scene image based on the gaze point coordinates includes:

确定人眼注视内容清晰时对应的预设角度范围;Determine the corresponding preset angle range when the content of the human eye is clear;

根据所述注视点坐标和预设角度范围在真实场景图像中的感兴趣区域。The region of interest in the real scene image according to the gaze point coordinates and the preset angle range.

可选地,所述将所述目标真实场景图像进行显示的步骤之后,还包括:Optionally, after the step of displaying the target real scene image, it also includes:

基于所述目标真实场景图像的真实图像亮度调整虚拟场景图像的虚拟图像亮度,得到目标虚拟场景图像;adjusting the virtual image brightness of the virtual scene image based on the real image brightness of the target real scene image to obtain the target virtual scene image;

将所述目标虚拟场景图像和所述目标真实场景图像进行融合显示。Fusion displaying the target virtual scene image and the target real scene image.

可选地,所述显示方法,还包括:Optionally, the display method further includes:

在采集到的真实场景图像上确定所述感兴趣区域对应的外围区域,并确定所述外围区域对应的外围亮度;Determining a peripheral area corresponding to the region of interest on the collected real scene image, and determining peripheral brightness corresponding to the peripheral area;

将所述外围区域对应的真实场景图像调整为所述外围亮度得到外围场景图像;adjusting the real scene image corresponding to the peripheral area to the peripheral brightness to obtain a peripheral scene image;

将所述目标真实场景图像以及所述外围场景图像进行显示。and displaying the target real scene image and the peripheral scene image.

可选地,所述确定所述外围区域对应的外围亮度的步骤,包括:Optionally, the step of determining the peripheral brightness corresponding to the peripheral area includes:

确定所述目标真实场景图像的真实图像亮度,并基于所述真实图像亮度确定所述外围区域对应的外围亮度;determining the real image brightness of the target real scene image, and determining the peripheral brightness corresponding to the peripheral region based on the real image brightness;

其中,所述真实图像亮度大于所述外围亮度。Wherein, the real image brightness is greater than the peripheral brightness.

此外,为实现上述目的,本发明还提供一种显示装置,所述显示装置,包括:In addition, in order to achieve the above object, the present invention also provides a display device, the display device comprising:

感兴趣区域确定模块,用于确定用户在真实场景图像中的感兴趣区域;A region of interest determination module is used to determine the region of interest of the user in the real scene image;

目标曝光确定模块,用于基于所述感兴趣区域的图像内容确定第一相机的目标曝光;A target exposure determination module, configured to determine the target exposure of the first camera based on the image content of the region of interest;

显示模块,用于获取所述第一相机基于所述目标曝光采集的目标真实场景图像,将所述目标真实场景图像进行显示。The display module is configured to acquire the target real scene image collected by the first camera based on the target exposure, and display the target real scene image.

此外,为实现上述目的,本发明还提供一种头戴显示设备,所述头戴显示设备包括:采集真实场景图像的第一相机、采集用户的眼睛图像的第二相机、存储器、处理器、及存储在所述存储器上并可在所述处理器上运行的计算机程序,所述计算机程序被所述处理器执行时实现如上任一项所述的显示方法的步骤。In addition, in order to achieve the above object, the present invention also provides a head-mounted display device, the head-mounted display device includes: a first camera for collecting real scene images, a second camera for collecting user's eye images, a memory, a processor, and a computer program stored on the memory and operable on the processor, when the computer program is executed by the processor, the steps of the display method described in any one of the above items are realized.

此外,为实现上述目的,本发明还提供一种计算机可读存储介质,所述计算机可读存储介质上存储有计算机程序,所述计算机程序被处理器执行时实现如上任一项所述的显示方法的步骤。In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the display as described in any one of the above items is realized. method steps.

本发明实施例提出的一种显示方法、显示装置、头戴显示设备及计算机可读存储介质,所述显示方法应用于头戴显示设备,在所述头戴显示设备上安装有采集真实场景图像的第一相机,所述显示方法包括:确定用户在真实场景图像中的感兴趣区域;基于所述感兴趣区域的图像内容确定所述第一相机的目标曝光;获取所述第一相机基于所述目标曝光采集的目标真实场景图像,将虚所述目标真实场景图像进行显示。A display method, a display device, a head-mounted display device, and a computer-readable storage medium proposed in an embodiment of the present invention, the display method is applied to a head-mounted display device, and a device for collecting real scene images is installed on the head-mounted display device The first camera of the first camera, the display method includes: determining the region of interest of the user in the real scene image; determining the target exposure of the first camera based on the image content of the region of interest; obtaining the target exposure of the first camera based on the The real scene image of the target is exposed and collected, and the real scene image of the target is displayed.

首先,根据用户在真实场景图像中的感兴趣区域的图像内容确定采集真实场景图像的第一相机的曝光,然后,将第一相机基于目标曝光采集的目标真实场景图像进行显示。First, the exposure of the first camera that captures the real scene image is determined according to the image content of the user's interest region in the real scene image, and then the target real scene image collected by the first camera based on the target exposure is displayed.

从而通过基于感兴趣区域的图像内容确定的曝光采集符合用户亮度需求的真实场景图像,解决人眼视觉区域因曝光不合适导致头戴显示设备显示的画面亮度偏暗和过亮,从而不满足用户人眼对于画面亮度的观看需要的技术问题,通过显示亮度合适的真实场景画面来提升头戴显示设备的VST视频透视显示效果,提升头戴显示设备用户的使用体验。Therefore, through the exposure determined based on the image content of the region of interest, the real scene image that meets the user's brightness requirements is collected, and the brightness of the screen displayed by the head-mounted display device is too dark and too bright due to improper exposure of the human eye's visual area, which does not satisfy the user. The technical problem that the human eye needs to watch the brightness of the picture, by displaying the real scene picture with appropriate brightness to improve the VST video perspective display effect of the head-mounted display device, and improve the user experience of the head-mounted display device.

附图说明Description of drawings

图1是本发明实施例方案涉及的硬件运行环境的终端结构示意图;Fig. 1 is a schematic diagram of the terminal structure of the hardware operating environment involved in the solution of the embodiment of the present invention;

图2为本发明显示方法一实施例的流程示意图;Fig. 2 is a schematic flow chart of an embodiment of the display method of the present invention;

图3为本发明显示方法一实施例的传输区域示意图;3 is a schematic diagram of a transmission area of an embodiment of the display method of the present invention;

图4为本发明显示装置一实施例的装置示意图。FIG. 4 is a device schematic diagram of an embodiment of the display device of the present invention.

本发明目的的实现、功能特点及优点将结合实施例,参照附图做进一步说明。The realization of the purpose of the present invention, functional characteristics and advantages will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

具体实施方式Detailed ways

应当理解,此处所描述的具体实施例仅仅用以解释本发明,并不用于限定本发明。It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

参照图1,图1为本发明实施例方案涉及的硬件运行环境的运行设备的结构示意图。Referring to FIG. 1 , FIG. 1 is a schematic structural diagram of an operating device of a hardware operating environment involved in an embodiment of the present invention.

如图1所示,该运行设备可以包括:处理器1001,例如中央处理器(CentralProcess i ng Un it,CPU),通信总线1002、用户接口1003,网络接口1004,存储器1005。其中,通信总线1002用于实现这些组件之间的连接通信。用户接口1003可以包括显示屏(Disp l ay)、输入单元比如键盘(Keyboard),可选用户接口1003还可以包括标准的有线接口、无线接口。网络接口1004可选的可以包括标准的有线接口、无线接口(如无线保真(WI re less-FI de l ity,WI-FI)接口)。存储器1005可以是高速的随机存取存储器(RandomAccess Memory,RAM)存储器,也可以是稳定的非易失性存储器(Non-Vo l at i l eMemory,NVM),例如磁盘存储器。存储器1005可选的还可以是独立于前述处理器1001的存储装置。As shown in FIG. 1 , the running device may include: a processor 1001 , such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a wireless fidelity (WI less-FI de lity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

本领域技术人员可以理解,图1中示出的结构并不构成对运行设备的限定,可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件布置。Those skilled in the art can understand that the structure shown in FIG. 1 does not constitute a limitation on the operating equipment, and may include more or less components than shown in the figure, or combine some components, or arrange different components.

如图1所示,作为一种存储介质的存储器1005中可以包括操作系统、数据存储模块、网络通信模块、用户接口模块以及计算机程序。As shown in FIG. 1 , memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, and computer programs.

在图1所示的运行设备中,网络接口1004主要用于与其他设备进行数据通信;用户接口1003主要用于与用户进行数据交互;本发明运行设备中的处理器1001、存储器1005可以设置在运行设备中,所述运行设备通过处理器1001调用存储器1005中存储的计算机程序,并执行以下操作:In the running device shown in Figure 1, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the running device of the present invention can be set in In the running device, the running device invokes the computer program stored in the memory 1005 through the processor 1001, and performs the following operations:

确定用户在真实场景图像中的感兴趣区域;Determine the user's region of interest in the real scene image;

基于所述感兴趣区域的图像内容确定所述第一相机的目标曝光;determining a target exposure for the first camera based on image content of the region of interest;

获取所述第一相机基于所述目标曝光采集的目标真实场景图像,将所述目标真实场景图像进行显示。Acquiring the target real scene image collected by the first camera based on the target exposure, and displaying the target real scene image.

进一步地,处理器1001可以调用存储器1005中存储的计算机程序,还执行以下操作:Further, the processor 1001 can call the computer program stored in the memory 1005, and also perform the following operations:

所述确定用户在真实场景图像中的感兴趣区域的步骤,包括:The step of determining the region of interest of the user in the real scene image includes:

获取用户的眼睛图像,并根据所述眼睛图像确定用户在真实场景图像中的感兴趣区域。An eye image of the user is acquired, and an area of interest of the user in the real scene image is determined according to the eye image.

进一步地,处理器1001可以调用存储器1005中存储的计算机程序,还执行以下操作:Further, the processor 1001 can call the computer program stored in the memory 1005, and also perform the following operations:

所述根据所述眼睛图像确定用户在真实场景图像中的感兴趣区域的步骤,包括:The step of determining the region of interest of the user in the real scene image according to the eye image includes:

根据所述眼睛图像计算出用户的注视点坐标,基于所述注视点坐标确定用户在真实场景图像中的感兴趣区域。The gaze point coordinates of the user are calculated according to the eye image, and the region of interest of the user in the real scene image is determined based on the gaze point coordinates.

进一步地,处理器1001可以调用存储器1005中存储的计算机程序,还执行以下操作:Further, the processor 1001 can call the computer program stored in the memory 1005, and also perform the following operations:

所述基于所述注视点坐标确定用户在真实场景图像中的感兴趣区域的步骤,包括:The step of determining the region of interest of the user in the real scene image based on the gaze point coordinates includes:

确定人眼注视内容清晰时对应的预设角度范围;Determine the corresponding preset angle range when the content of the human eye is clear;

根据所述注视点坐标和预设角度范围在真实场景图像中的感兴趣区域。The region of interest in the real scene image according to the gaze point coordinates and the preset angle range.

进一步地,处理器1001可以调用存储器1005中存储的计算机程序,还执行以下操作:Further, the processor 1001 can call the computer program stored in the memory 1005, and also perform the following operations:

所述将所述目标真实场景图像进行显示的步骤之后,还包括:After the step of displaying the target real scene image, it also includes:

基于所述目标真实场景图像的真实图像亮度调整虚拟场景图像的虚拟图像亮度,得到目标虚拟场景图像;adjusting the virtual image brightness of the virtual scene image based on the real image brightness of the target real scene image to obtain the target virtual scene image;

将所述目标虚拟场景图像和所述目标真实场景图像进行融合显示。Fusion displaying the target virtual scene image and the target real scene image.

进一步地,处理器1001可以调用存储器1005中存储的计算机程序,还执行以下操作:Further, the processor 1001 can call the computer program stored in the memory 1005, and also perform the following operations:

所述显示方法,还包括:The display method also includes:

在采集到的真实场景图像上确定所述感兴趣区域对应的外围区域,并确定所述外围区域对应的外围亮度;Determining a peripheral area corresponding to the region of interest on the collected real scene image, and determining peripheral brightness corresponding to the peripheral area;

将所述外围区域对应的真实场景图像调整为所述外围亮度得到外围场景图像;adjusting the real scene image corresponding to the peripheral area to the peripheral brightness to obtain a peripheral scene image;

将所述目标真实场景图像以及所述外围场景图像进行显示。and displaying the target real scene image and the peripheral scene image.

进一步地,处理器1001可以调用存储器1005中存储的计算机程序,还执行以下操作:Further, the processor 1001 can call the computer program stored in the memory 1005, and also perform the following operations:

所述确定所述外围区域对应的外围亮度的步骤,包括:The step of determining the peripheral brightness corresponding to the peripheral area includes:

确定所述目标真实场景图像的真实图像亮度,并基于所述真实图像亮度确定所述外围区域对应的外围亮度;determining the real image brightness of the target real scene image, and determining the peripheral brightness corresponding to the peripheral region based on the real image brightness;

其中,所述真实图像亮度大于所述外围亮度。Wherein, the real image brightness is greater than the peripheral brightness.

参照图2,本发明提供一种显示方法,所述显示方法应用于头戴显示设备,在所述头戴显示设备上安装有采集真实场景图像的第一相机,所述显示方法包括:Referring to FIG. 2 , the present invention provides a display method, the display method is applied to a head-mounted display device, and a first camera for collecting real scene images is installed on the head-mounted display device, and the display method includes:

步骤S10,确定用户在真实场景图像中的感兴趣区域。Step S10, determining the region of interest of the user in the real scene image.

头戴显示设备如AR眼镜采集真实场景图像的第一相机为相机(可以为单色相机或者RGB相机),头戴显示设备如AR眼镜采集用户的眼睛图像的第二相机为Eyetracki ng(眼部追踪)相机。其中,真实场景图像是指头戴显示设备如AR眼镜上的相机采集到的真实世界的场景图像,感兴趣区域是指通过Eyetracki ng相机采集用户注视点具体坐标位置而确定的区域。The first camera for the head-mounted display device such as AR glasses to collect real scene images is a camera (may be a monochrome camera or an RGB camera), and the second camera for the head-mounted display device such as AR glasses to collect user's eye images is Eyetracking (eye tracking). tracking) camera. Among them, the real scene image refers to the scene image of the real world collected by the camera on the head-mounted display device such as AR glasses, and the region of interest refers to the region determined by the specific coordinate position of the user's gaze point collected by the Eyetracking camera.

进一步的,在采集到的真实场景图像上确定感兴趣区域的步骤,还包括:获取用户的语音信息,解析所述语音信息得到用户确定的主要物体,基于所述主要物体在所述真实场景图像上匹配与所述主要物体相似度大于预设阈值的目标主体,将所述目标主体所处区域作为所述感兴趣区域。从而除了通过用户注视点确定感兴趣区域之外,还可以让用户主动在真实场景图像上确定感兴趣区域,进而提高显示方法的适用性,给用户提供更多的实现融合显示的可选操作。Further, the step of determining the region of interest on the collected real scene image further includes: acquiring the voice information of the user, analyzing the voice information to obtain the main object determined by the user, and based on the presence of the main object in the real scene image Match the target subject whose similarity with the main object is greater than a preset threshold, and use the area where the target subject is located as the region of interest. Therefore, in addition to determining the region of interest through the user's gaze point, the user can also actively determine the region of interest on the real scene image, thereby improving the applicability of the display method and providing the user with more optional operations for realizing fusion display.

进一步地,在采集到的真实场景图像上确定感兴趣区域的步骤,包括:识别用户的第一手势动作,并根据所述第一手势动作确定用户在真实场景图像上的感兴趣区域。除了通过用户的语音信息确定真实场景图像上的感兴趣区域之外,还可以通过用户的第一手势动作确定的感兴趣区域。用户可以在经过预设划分方式划分之后的真实场景图像上通过第一手势动作选取其中的一个或多个子区域作为感兴趣区域,也可以在未经过划分的真实场景图像上通过第一手势动作圈选出一个区域作为感兴趣区域。在本实施例中,对第一手势动作的具体手势和动作不做限定。从而,充分利用头戴显示设备的显示能力,给用户提供除语音选取之外通过选中或者圈选方式确定感兴趣区域。Further, the step of determining the region of interest on the collected real scene image includes: identifying the user's first gesture, and determining the user's region of interest on the real scene image according to the first gesture. In addition to determining the region of interest on the real scene image through the voice information of the user, the region of interest determined through the user's first gesture action can also be used. The user can select one or more sub-regions as regions of interest through the first gesture action on the real scene image divided by the preset division method, or use the first gesture action circle on the real scene image that has not been divided. Select a region as the region of interest. In this embodiment, the specific gesture and action of the first gesture action are not limited. Therefore, by making full use of the display capability of the head-mounted display device, it is provided for the user to determine the region of interest through selection or circle selection in addition to voice selection.

步骤S20,基于所述感兴趣区域的图像内容确定所述第一相机的目标曝光。Step S20, determining the target exposure of the first camera based on the image content of the region of interest.

在确定了感兴趣区域之后,就需要进一步根据感兴趣区域的图像内容确定第一相机的目标曝光,第一相机再基于目标曝光采集得到目标真实场景图像,进一步将目标真实场景图像进行显示。After the region of interest is determined, it is necessary to further determine the target exposure of the first camera based on the image content of the region of interest, and then the first camera acquires the real scene image of the target based on the target exposure, and further displays the real scene image of the target.

步骤S30,获取所述第一相机基于所述目标曝光采集的目标真实场景图像,将所述目标真实场景图像进行显示。Step S30, acquiring the target real scene image collected by the first camera based on the target exposure, and displaying the target real scene image.

视频透视(VST)是指头戴显示设备如AR眼镜通过安装在眼镜上的微型摄像头采集真实场景的图像,AR眼镜再通过场景理解和分析将所要添加的信息和图像信号叠加在摄像机的视频信号上,同时将AR眼镜生成的虚拟场景与真实场景进行融合,最后通过AR眼镜显示屏幕呈现给用户。Video see-through (VST) refers to head-mounted display devices such as AR glasses that collect images of real scenes through a miniature camera installed on the glasses. AR glasses then superimpose the information and image signals to be added on the video signals of the camera through scene understanding and analysis. At the same time, the virtual scene generated by the AR glasses is fused with the real scene, and finally presented to the user through the display screen of the AR glasses.

可选的,所述将所述目标真实场景图像进行显示的步骤之后,还包括:基于所述目标真实场景图像的真实图像亮度调整虚拟场景图像的虚拟图像亮度,得到目标虚拟场景图像;将所述目标虚拟场景图像和所述目标真实场景图像进行融合显示。Optionally, after the step of displaying the target real scene image, further comprising: adjusting the virtual image brightness of the virtual scene image based on the real image brightness of the target real scene image to obtain the target virtual scene image; The target virtual scene image and the target real scene image are fused and displayed.

在将目标真实场景图像进行显示的过程中,可能存在目标真实场景图像的亮度与原始的虚拟场景图像的亮度不一致,从而导致用户的眼部不适以及沉浸体验上的割裂感。所以,在基于感兴趣区域的图像内容确定第一相机的目标曝光、改变第一相机采集到的真实场景图像的亮度之后,还需要对原始的虚拟场景图像的亮度进行同步调整。因此,基于目标真实场景图像的真实图像亮度调整虚拟场景图像的虚拟图像亮度,得到目标虚拟场景图像。优选地,将虚拟图像亮度调整为与真实图像亮度一样。进一步地,由于人眼在视物的过程中,并不会注意到视野中的全部细节,只有中间视觉焦点附近是清晰的,任何超出了人眼注视区中心预设角度范围如5度以上的区域都会逐渐降低清晰度。所以,将感兴趣区域内的虚拟图像亮度设置为与真实图像亮度一样,将非感兴趣区域内的虚拟图像亮度设置为比真实图像亮度低。最终,在调整了虚拟场景图像的虚拟图像亮度得到目标之后,将调整得到的目标虚拟场景图像和采集得到的目标真实场景图像进行融合显示,除了避免单一调整真实场景图像亮度引起的亮度不一致的问题之外,将真实场景图像和虚拟场景图像调整为一致或者虚拟图像亮度设置为比真实图像亮度低,也可以使用户观看头戴显示设备显示的画面时更加符合肉眼生物属性带来的观看习惯。In the process of displaying the target real scene image, the brightness of the target real scene image may not be consistent with the brightness of the original virtual scene image, which may cause discomfort to the user's eyes and a sense of fragmentation in the immersive experience. Therefore, after determining the target exposure of the first camera based on the image content of the region of interest and changing the brightness of the real scene image captured by the first camera, it is also necessary to synchronously adjust the brightness of the original virtual scene image. Therefore, the virtual image brightness of the virtual scene image is adjusted based on the real image brightness of the target real scene image to obtain the target virtual scene image. Preferably, the brightness of the virtual image is adjusted to be the same as the brightness of the real image. Furthermore, since the human eye does not notice all the details in the field of vision during the process of viewing objects, only the vicinity of the intermediate vision focus is clear, and any angle beyond the preset angle range of the center of the human eye's gaze area, such as 5 degrees or more Areas are progressively less sharp. Therefore, the brightness of the virtual image in the region of interest is set to be the same as the brightness of the real image, and the brightness of the virtual image in the non-interest region is set to be lower than the brightness of the real image. Finally, after adjusting the virtual image brightness of the virtual scene image to obtain the target, the adjusted target virtual scene image and the collected target real scene image are fused and displayed, in addition to avoiding the problem of brightness inconsistency caused by single adjustment of the real scene image brightness In addition, adjusting the real scene image and the virtual scene image to be consistent or setting the brightness of the virtual image to be lower than the real image brightness can also make the user watch the picture displayed by the head-mounted display device more in line with the viewing habits brought about by the biological attributes of the naked eye.

在本实施例中,所述显示方法应用于头戴显示设备,在所述头戴显示设备上安装有采集真实场景图像的第一相机,所述显示方法包括:确定用户在真实场景图像中的感兴趣区域;基于所述感兴趣区域的图像内容确定所述第一相机的目标曝光;获取所述第一相机基于所述目标曝光采集的目标真实场景图像,将所述目标真实场景图像进行显示。In this embodiment, the display method is applied to a head-mounted display device, and a first camera for collecting real scene images is installed on the head-mounted display device, and the display method includes: determining the user's position in the real scene image A region of interest; determining the target exposure of the first camera based on the image content of the region of interest; acquiring a target real scene image collected by the first camera based on the target exposure, and displaying the target real scene image .

首先,根据用户在真实场景图像中的感兴趣区域的图像内容确定采集真实场景图像的第一相机的曝光,然后,将第一相机基于目标曝光采集的目标真实场景图像进行显示。First, the exposure of the first camera that captures the real scene image is determined according to the image content of the user's interest region in the real scene image, and then the target real scene image collected by the first camera based on the target exposure is displayed.

从而通过基于感兴趣区域的图像内容确定的曝光采集符合用户亮度需求的真实场景图像,解决人眼视觉区域因曝光不合适导致头戴显示设备显示的画面亮度偏暗和过亮,从而不满足用户人眼对于画面亮度的观看需要的技术问题,通过显示亮度合适的真实场景画面来提升头戴显示设备的VST视频透视显示效果,提升头戴显示设备用户的使用体验。Therefore, through the exposure determined based on the image content of the region of interest, the real scene image that meets the user's brightness requirements is collected, and the brightness of the screen displayed by the head-mounted display device is too dark and too bright due to improper exposure of the human eye's visual area, which does not satisfy the user. The technical problem that the human eye needs to watch the brightness of the picture, by displaying the real scene picture with appropriate brightness to improve the VST video perspective display effect of the head-mounted display device, and improve the user experience of the head-mounted display device.

进一步的,在本发明显示方法的另一实施例中,所述确定用户在真实场景图像中的感兴趣区域的步骤,包括:获取用户的眼睛图像,并根据所述眼睛图像确定用户在真实场景图像中的感兴趣区域。Further, in another embodiment of the display method of the present invention, the step of determining the user's region of interest in the real scene image includes: acquiring the user's eye image, and determining the user's area of interest in the real scene image according to the eye image region of interest in the image.

注视点追踪又称眼球追踪,是利用传感器如红外摄像机捕获、提取眼球特征信息,测量眼睛的运动情况从而估计视线方向或眼睛注视点位置。头戴显示设备如AR眼镜采集用户的眼睛图像的第二相机为Eyetracki ng(眼部追踪)相机,通过头戴显示设备的第二相机采集获取得到用户的眼睛图像,从而可以根据眼睛图像确定用户在真实场景图像上的感兴趣区域。Gaze tracking, also known as eye tracking, uses sensors such as infrared cameras to capture and extract eye feature information, and measure eye movement to estimate the direction of sight or the position of eye gaze. A head-mounted display device such as AR glasses collects the second camera of the user's eye image as an Eyetracking (eye tracking) camera, and obtains the user's eye image through the second camera of the head-mounted display device, so that the user's eye image can be determined according to the eye image. Regions of interest on real scene images.

可选地,所述根据所述眼睛图像确定用户在真实场景图像中的感兴趣区域的步骤,包括:根据所述眼睛图像计算出用户的注视点坐标,基于所述注视点坐标确定用户在真实场景图像中的感兴趣区域。Optionally, the step of determining the region of interest of the user in the real scene image according to the eye image includes: calculating the gaze point coordinates of the user according to the eye image, and determining the gaze point coordinates of the user in the real scene image based on the gaze point coordinates. A region of interest in a scene image.

在通过头戴显示设备的第二相机采集获取得到用户的眼睛图像之后,根据眼睛图像计算出用户的注视点坐标,从而基于注视点坐标确定的感兴趣区域。在本实施例中,采用瞳孔角膜反射法确定注视点坐标:在头戴显示设备的注视点追踪系统中红外光源和eye-tracki ng摄像机的位置都是不变条件下,并且在眼球模型的结构基础上,利用闪烁点与光源位置计算得到角膜曲率中心。利用图像处理技术计算获得瞳孔中心,通过角膜曲率中心与瞳孔中心的连线求得眼球光轴,并利用光轴和视轴之间的夹角计算得到了真实的视线方向即视轴以及注视点坐标。After the user's eye image is captured by the second camera of the head-mounted display device, the coordinates of the user's gaze point are calculated according to the eye image, so as to determine the region of interest based on the gaze point coordinates. In this embodiment, the pupil-corneal reflection method is used to determine the coordinates of the fixation point: in the fixation point tracking system of the head-mounted display device, the positions of the infrared light source and the eye-tracking camera are all constant, and the structure of the eyeball model On the basis, the center of corneal curvature is calculated by using the flicker point and the position of the light source. The center of the pupil is calculated by image processing technology, the optical axis of the eyeball is obtained through the connection between the center of corneal curvature and the center of the pupil, and the real line of sight direction, that is, the visual axis and fixation point, is calculated by using the angle between the optical axis and the visual axis coordinate.

可选的,所述基于所述注视点坐标确定用户在真实场景图像中的感兴趣区域的步骤,包括:确定人眼注视内容清晰时对应的预设角度范围;根据所述注视点坐标和预设角度范围在真实场景图像中的感兴趣区域。Optionally, the step of determining the region of interest of the user in the real scene image based on the gaze point coordinates includes: determining the corresponding preset angle range when the gaze content of the human eye is clear; according to the gaze point coordinates and the preset Set the angle range to the region of interest in the real scene image.

人眼在视物的过程中,并不会注意到视野中的全部细节,只有中间视觉焦点附近是清晰的,任何超出了人眼注视区中心预设角度范围如5度以上的区域都会逐渐降低清晰度,超出了人眼注视区中心预设角度范围如5度以上的区域作为忽略区域。这是由于负责观察色彩和细节的视网膜上的视锥细胞的浓度不同,拥有高密度视锥细胞的区域叫中央凹与人眼视野中的注视点相对应。由于人类视网膜的结构,人眼中央凹的分辨率最高,外围视场的视觉质量相对更低,中央凹位置称为人眼注视点区域。在VST技术中,相机对焦区域要与人眼注视点区域实时重合才能确保感知视图与视觉范围保持一致。所以,在本实施例中,基于注视点坐标和预设角度范围在真实场景图像中确定得到感兴趣区域。其中,所确定的感兴趣区域可以是圆形区域,也可以矩形区域,具体的区域形状或者大小至少包括以注视点坐标为中心的预设角度范围内的区域。In the process of seeing objects, the human eye does not notice all the details in the field of vision, only the vicinity of the intermediate vision focus is clear, and any area beyond the preset angle range of the center of the human eye's gaze area, such as 5 degrees or more, will gradually decrease Clarity, beyond the preset angle range of the center of the human eye's gaze area, such as the area above 5 degrees, is regarded as an ignored area. This is due to differences in the concentration of cones on the retina, which are responsible for seeing color and detail. The area with a high density of cones called the fovea corresponds to the point of fixation in the human eye's field of view. Due to the structure of the human retina, the resolution of the fovea of the human eye is the highest, and the visual quality of the peripheral field of view is relatively low. The position of the fovea is called the focal point area of the human eye. In VST technology, the focus area of the camera must coincide with the focus area of the human eye in real time to ensure that the perceived view is consistent with the visual range. Therefore, in this embodiment, the region of interest is determined in the real scene image based on the gaze point coordinates and the preset angle range. Wherein, the determined region of interest may be a circular region or a rectangular region, and the specific shape or size of the region at least includes a region within a preset angle range centered on the gaze point coordinates.

在本实施例中,首先通过eye-tracki ng摄像头和红外补光灯I R-Led来采集用户眼睛图像,利用注视点追踪算法经过包括图像的灰度化、二值化和图像边缘检测等的图像预处理,然后分别进行瞳孔中心定位和角膜反射光斑中心定位,计算出用户的注视点坐标,以此得出用户的注视方向和具体的注视点坐标值。进而可以根据注视点坐标和预设角度范围在真实场景图像中的感兴趣区域。从而除了通过用户的语音信息确定真实场景图像上的感兴趣区域、以及通过选中或者圈选的方式确定感兴趣区域之外,还可以仅需要根据用户的注意力关注的注视点即可确定用户在真实场景图像上的感兴趣区域,进一步做到无感智能的融合显示,提升头戴显示设备的增强现实现实效果和体验。In this embodiment, first, the user's eye image is collected through the eye-tracking camera and the infrared supplementary light I R-Led, and the gaze point tracking algorithm is used to go through the steps including image grayscale, binarization, and image edge detection. Image preprocessing, and then perform pupil center positioning and corneal reflection spot center positioning respectively, and calculate the user's gaze point coordinates, so as to obtain the user's gaze direction and specific gaze point coordinate values. Furthermore, the region of interest in the real scene image can be determined according to the gaze point coordinates and the preset angle range. Therefore, in addition to determining the region of interest on the real scene image through the user's voice information, and determining the region of interest by selecting or circling the region, it can also be determined only according to the gaze point of the user's attention. The area of interest on the real scene image is further achieved with non-sensing and intelligent fusion display, which improves the augmented reality effect and experience of the head-mounted display device.

进一步的,在本发明显示方法的另一实施例中,所述显示方法,还包括:Further, in another embodiment of the display method of the present invention, the display method further includes:

在采集到的真实场景图像上确定所述感兴趣区域对应的外围区域,并确定所述外围区域对应的外围亮度;Determining a peripheral area corresponding to the region of interest on the collected real scene image, and determining peripheral brightness corresponding to the peripheral area;

将所述外围区域对应的真实场景图像调整为所述外围亮度得到外围场景图像;adjusting the real scene image corresponding to the peripheral area to the peripheral brightness to obtain a peripheral scene image;

将所述目标真实场景图像以及所述外围场景图像进行显示。and displaying the target real scene image and the peripheral scene image.

在确定了感兴趣区域之后,在采集到的真实场景图像上确定感兴趣区域对应的外围区域,其中,所述外围区域可以是所述感兴趣区域的外围环形区域,或者上下左右四边等宽的矩形区域,外围区域的确定方式以及外围区域的区域大小、区域形状等在本实施例中不做限定。考虑到由于负责观察色彩和细节的视网膜上的视锥细胞的浓度不同导致的人眼在视物的过程中,并不会注意到视野中的全部细节,只有中间视觉焦点附近是清晰的,任何超出了人眼注视区中心预设角度范围如5度以上的区域都会逐渐降低清晰度,所以在本实施例中不仅仅只是考虑清晰度最高、用户眼睛最能关注到的感兴趣区域,还考虑到所述感兴趣区域对应的外围区域,将外围区域作为逐渐降低亮度的对应区域,将外围区域对应的真实场景图像调整为外围亮度得到外围场景图像。最终,将目标真实场景图像以及外围场景图像进行显示,尽可能模拟并还原用户的真实观看体验。从而减轻用户在佩戴使用头戴显示设备时虚拟与现实之间的割裂感,为用户尽可能模拟人眼的观看体验、尽可能提供接近现实的观看感受。After the region of interest is determined, the peripheral region corresponding to the region of interest is determined on the collected real scene image, wherein the peripheral region can be the peripheral ring region of the region of interest, or an area of equal width from top to bottom, left, right, and four sides The rectangular area, the manner of determining the peripheral area, and the size and shape of the peripheral area are not limited in this embodiment. Considering that the concentration of cone cells on the retina responsible for observing color and details is different, the human eye will not notice all the details in the visual field during the visual process, and only the vicinity of the intermediate vision focus is clear. Areas that exceed the preset angle range of the center of the human gaze area, such as 5 degrees or more, will gradually reduce the definition. Therefore, in this embodiment, not only the area of interest with the highest definition and the user's eyes can pay attention to is considered, but also To the peripheral area corresponding to the region of interest, the peripheral area is used as the corresponding area whose brightness gradually decreases, and the real scene image corresponding to the peripheral area is adjusted to the peripheral brightness to obtain the peripheral scene image. Finally, the target real scene image and peripheral scene images are displayed to simulate and restore the user's real viewing experience as much as possible. In this way, the sense of separation between virtual and reality can be alleviated when the user wears the head-mounted display device, and the viewing experience of the human eye can be simulated as much as possible for the user, and the viewing experience as close to reality as possible can be provided.

可选的,所述确定所述外围区域对应的外围亮度的步骤,包括:Optionally, the step of determining the peripheral brightness corresponding to the peripheral area includes:

确定所述目标真实场景图像的真实图像亮度,并基于所述真实图像亮度确定所述外围区域对应的外围亮度;determining the real image brightness of the target real scene image, and determining the peripheral brightness corresponding to the peripheral region based on the real image brightness;

其中,所述真实图像亮度大于所述外围亮度。Wherein, the real image brightness is greater than the peripheral brightness.

在确定了外围区域之后,基于目标真实场景图像的真实图像亮度确定外围区域对应的外围亮度。进一步的,真实图像亮度确定所述外围区域对应的外围亮度的步骤,包括:确定所述外围区域所处的划分层级,基于所述划分层级确定目标真实场景图像的真实图像亮度对应的所述外围场景图像的外围亮度。预先对感兴趣区域的外围区域进行划分,得到不同的距离层级并且不同距离层级预设有根据所述真实图像亮度向外逐渐降低的亮度,外围区域根据其所处的划分层级直接确定其对应的外围亮度。或者,预先对感兴趣区域的外围区域进行划分,对不同距离的外围区域设置不同的小于1且逐渐降低的亮度权重,基于外围区域所处的距离和该距离对应的亮度权重确定所述外围区域对应的外围亮度。After the peripheral area is determined, the peripheral brightness corresponding to the peripheral area is determined based on the real image brightness of the target real scene image. Further, the step of determining the peripheral brightness corresponding to the peripheral area by the real image brightness includes: determining the division level where the peripheral area is located, and determining the peripheral area corresponding to the real image brightness of the target real scene image based on the division level. Peripheral brightness of the scene image. The peripheral area of the region of interest is divided in advance to obtain different distance levels, and different distance levels are preset with brightness that gradually decreases outwards according to the real image brightness, and the peripheral area directly determines its corresponding Peripheral brightness. Alternatively, divide the peripheral area of the region of interest in advance, set different brightness weights less than 1 and gradually decreasing for peripheral areas of different distances, and determine the peripheral area based on the distance of the peripheral area and the brightness weight corresponding to the distance Corresponding peripheral brightness.

其中,同样考虑到由于负责观察色彩和细节的视网膜上的视锥细胞的浓度不同导致的人眼在视物的过程中,并不会注意到视野中的全部细节,只有中间视觉焦点附近是清晰的,任何超出了人眼注视区中心预设角度范围如5度以上的区域都会逐渐降低清晰度,所以目标真实场景图像的真实图像亮度大于外围区域的外围亮度,从而在头戴显示设备显示的画面中凸显感兴趣区域。Among them, it is also considered that due to the different concentration of cone cells on the retina responsible for observing colors and details, the human eye will not notice all the details in the field of vision during the process of seeing objects, and only the vicinity of the intermediate vision focus is clear. Yes, any area that exceeds the preset angle range of the center of the human gaze area, such as 5 degrees or more, will gradually reduce the definition, so the real image brightness of the target real scene image is greater than the peripheral brightness of the peripheral area, so that the head-mounted display device displays The area of interest is highlighted on the screen.

进一步的,在本发明显示方法的另一实施例中,所述显示方法,还包括:Further, in another embodiment of the display method of the present invention, the display method further includes:

识别所述感兴趣区域中的主要物体;identifying principal objects in the region of interest;

基于所述主要物体的图像内容确定所述第一相机的目标曝光。A target exposure for the first camera is determined based on image content of the primary object.

除了基于感兴趣区域的图像内容确定第一相机的目标曝光之外,还可以只基于感兴趣区域中的主要物体精准确定第一相机的目标曝光。进一步的,所述识别所述感兴趣区域中的主要物体的步骤包括:识别所述感兴趣区域中的各个待选物体,在所述待选物体中选择物体面积最大的待选物体作为所述主体物体,或者在所述待选物体中选择处于最上图层的待选物体作为所述主体物体,或者确定各个所述待选物体所处为前景或背景,将前景中的一个或者多个待选物体作为所述主体物体。同样的,在多个待选物体中确定得到所述主要物体也可以是通过语音、手势等操作实现,具体操作步骤与上述在采集到的真实场景图像上确定感兴趣区域的方式类似,在此不做赘述。In addition to determining the target exposure of the first camera based on the image content of the region of interest, the target exposure of the first camera can also be accurately determined only based on the main object in the region of interest. Further, the step of identifying the main object in the region of interest includes: identifying each candidate object in the region of interest, and selecting the candidate object with the largest object area among the candidate objects as the The main object, or select the candidate object in the uppermost layer among the candidate objects as the main object, or determine that each of the candidate objects is in the foreground or background, and place one or more candidate objects in the foreground Select an object as the main object. Similarly, determining the main object among multiple candidate objects can also be achieved through operations such as voice and gestures. The specific operation steps are similar to the above-mentioned method of determining the region of interest on the collected real scene image. Here I won't go into details.

在所述感兴趣区域中识别到主要物体,并根据主要物体的图像内容确定第一相机的目标曝光,从而可以降低头戴显示设备进行显示时的数据量以及计算量,提升头戴显示设备的处理效率,并进一步提高曝光的精准度,最终提升头戴显示设备的显示效果以及用户的增强现实的体验。The main object is identified in the region of interest, and the target exposure of the first camera is determined according to the image content of the main object, thereby reducing the amount of data and calculation when the head-mounted display device is displayed, and improving the performance of the head-mounted display device. processing efficiency, and further improve the accuracy of exposure, and ultimately improve the display effect of the head-mounted display device and the user's augmented reality experience.

参照图3,在本发明显示方法的另一实施例中,首先用户戴上头戴显示设备如VR/AR眼镜,在采集真实场景图像的相机、eye-tracki ng摄像头和I R-LED采集图像功能正常时,调用成熟的注视点跟踪算法计算出用户的注视点方向。具体的,利用eye-tracki ng摄像头和I R-LED采集清晰的眼睛图像,之后经过图像预处理包括图像的灰度化、图像滤波、二值化和图像边缘检测等,然后分别进行瞳孔中心定位和角膜反射光斑中心定位,计算出用户的注视点。然后基于注视点位置信息的图像内容确定采集真实场景图像的第一相机的目标曝光,最终将虚拟场景图像和第一相机基于目标曝光采集的目标真实场景图像进行融合,显示到VR/AR眼镜显示屏上。从而通过基于感兴趣区域的图像内容确定的曝光采集符合用户亮度需求的真实场景图像,解决人眼视觉区域因曝光不合适导致头戴显示设备显示的画面亮度偏暗和过亮,从而不满足用户人眼对于画面亮度的观看需要的技术问题,通过显示亮度合适的真实场景画面来提升头戴显示设备的VST视频透视显示效果,进而提升头戴显示设备的VST视频透视显示效果,提升头戴显示设备用户的使用体验。Referring to Fig. 3, in another embodiment of the display method of the present invention, firstly, the user puts on a head-mounted display device such as VR/AR glasses, and collects images with a camera, an eye-tracking camera, and an IR-LED for collecting real scene images When the function is normal, the mature gaze point tracking algorithm is called to calculate the direction of the user's gaze point. Specifically, use the eye-tracking camera and I R-LED to collect clear eye images, and then undergo image preprocessing including image grayscale, image filtering, binarization, and image edge detection, etc., and then perform pupil center positioning respectively and the center of the corneal reflection spot to calculate the user's gaze point. Then determine the target exposure of the first camera that collects the real scene image based on the image content of the gaze point position information, and finally fuse the virtual scene image with the target real scene image collected by the first camera based on the target exposure, and display it on the VR/AR glasses display screen. Therefore, through the exposure determined based on the image content of the region of interest, the real scene image that meets the user's brightness requirements is collected, and the brightness of the screen displayed by the head-mounted display device is too dark and too bright due to improper exposure of the human eye's visual area, which does not satisfy the user. The technical problem that the human eye needs to watch the screen brightness is to improve the VST video perspective display effect of the head-mounted display device by displaying a real scene image with appropriate brightness, thereby improving the VST video perspective display effect of the head-mounted display device and improving the head-mounted display Device user experience.

此外,参照图4,本发明实施例还提供一种显示装置,所述显示装置,包括:In addition, referring to FIG. 4 , an embodiment of the present invention also provides a display device, the display device including:

感兴趣区域确定模块M1,用于确定用户在真实场景图像中的感兴趣区域;The region of interest determination module M1 is used to determine the region of interest of the user in the real scene image;

目标曝光确定模块M2,用于基于所述感兴趣区域的图像内容确定第一相机的目标曝光;A target exposure determination module M2, configured to determine the target exposure of the first camera based on the image content of the region of interest;

显示模块M3,用于获取所述第一相机基于所述目标曝光采集的目标真实场景图像,将所述目标真实场景图像进行显示。The display module M3 is configured to obtain the target real scene image collected by the first camera based on the target exposure, and display the target real scene image.

可选的,感兴趣区域确定模块M1,还用于获取用户的眼睛图像,并根据所述眼睛图像确定用户在真实场景图像中的感兴趣区域。Optionally, the ROI determining module M1 is further configured to acquire an eye image of the user, and determine the ROI of the user in the real scene image according to the eye image.

可选的,感兴趣区域确定模块M1,还用于根据所述眼睛图像计算出用户的注视点坐标,基于所述注视点坐标确定用户在真实场景图像中的感兴趣区域。Optionally, the ROI determination module M1 is further configured to calculate the gaze point coordinates of the user based on the eye image, and determine the ROI of the user in the real scene image based on the gaze point coordinates.

可选的,感兴趣区域确定模块M1,还用于确定人眼注视内容清晰时对应的预设角度范围;Optionally, the region-of-interest determination module M1 is also used to determine the corresponding preset angle range when the gaze content of the human eye is clear;

根据所述注视点坐标和预设角度范围在真实场景图像中的感兴趣区域。The region of interest in the real scene image according to the gaze point coordinates and the preset angle range.

可选的,所述显示装置还包括:进阶融合模块,用于基于所述目标真实场景图像的真实图像亮度调整虚拟场景图像的虚拟图像亮度,得到目标虚拟场景图像;Optionally, the display device further includes: an advanced fusion module, configured to adjust the virtual image brightness of the virtual scene image based on the real image brightness of the target real scene image, to obtain the target virtual scene image;

将所述目标虚拟场景图像和所述目标真实场景图像进行融合显示。Fusion displaying the target virtual scene image and the target real scene image.

可选的,所述显示装置还包括:外围区域融合模块,用于在采集到的真实场景图像上确定所述感兴趣区域对应的外围区域,并确定所述外围区域对应的外围亮度;Optionally, the display device further includes: a peripheral region fusion module, configured to determine a peripheral region corresponding to the region of interest on the collected real scene image, and determine peripheral brightness corresponding to the peripheral region;

将所述外围区域对应的真实场景图像调整为所述外围亮度得到外围场景图像;adjusting the real scene image corresponding to the peripheral area to the peripheral brightness to obtain a peripheral scene image;

将所述目标真实场景图像以及所述外围场景图像进行显示。and displaying the target real scene image and the peripheral scene image.

可选的,外围区域融合模块,还用于确定所述目标真实场景图像的真实图像亮度,并基于所述真实图像亮度确定所述外围区域对应的外围亮度;Optionally, the peripheral region fusion module is further configured to determine the real image brightness of the target real scene image, and determine the peripheral brightness corresponding to the peripheral region based on the real image brightness;

其中,所述真实图像亮度大于所述外围亮度。Wherein, the real image brightness is greater than the peripheral brightness.

本发明提供的显示装置,采用上述实施例中的显示方法,解决现有技术中头戴显示设备显示的画面亮度偏暗和过亮,不满足用户人眼对于画面亮度的观看需要的技术问题。与现有技术相比,本发明实施例提供的显示装置的有益效果与上述实施例提供的显示方法的有益效果相同,且该显示装置中的其他技术特征与上述实施例方法公开的特征相同,在此不做赘述。The display device provided by the present invention adopts the display method in the above-mentioned embodiments to solve the technical problem that the brightness of the picture displayed by the head-mounted display device in the prior art is too dark or too bright, which does not meet the viewing needs of the user's human eyes for the brightness of the picture. Compared with the prior art, the beneficial effect of the display device provided by the embodiment of the present invention is the same as that of the display method provided by the above embodiment, and other technical features of the display device are the same as those disclosed by the method of the above embodiment, I won't go into details here.

此外,本发明实施例还提供一种头戴显示设备,所述头戴显示设备包括:采集真实场景图像的第一相机、采集用户的眼睛图像的第二相机、存储器、处理器、及存储在所述存储器上并可在所述处理器上运行的计算机程序,所述计算机程序被所述处理器执行时实现如上任一项所述的显示方法的步骤。In addition, an embodiment of the present invention also provides a head-mounted display device, the head-mounted display device includes: a first camera for collecting real scene images, a second camera for collecting user's eye images, a memory, a processor, and a A computer program on the memory and operable on the processor, when the computer program is executed by the processor, the steps of the display method described in any one of the above items are realized.

此外,本发明实施例还提供一种计算机可读存储介质,所述计算机可读存储介质上存储有计算机程序,所述计算机程序被处理器执行时实现如上任一项所述的显示方法的步骤。In addition, an embodiment of the present invention also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the display method described in any one of the above items are implemented. .

需要说明的是,在本文中,术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者系统不仅包括那些要素,而且还包括没有明确列出的其他要素,或者是还包括为这种过程、方法、物品或者系统所固有的要素。在没有更多限制的情况下,由语句“包括一个……”限定的要素,并不排除在包括该要素的过程、方法、物品或者系统中还存在另外的相同要素。It should be noted that, as used herein, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or system comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or system. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article or system comprising that element.

上述本发明实施例序号仅仅为了描述,不代表实施例的优劣。The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件,但很多情况下前者是更佳的实施方式。基于这样的理解,本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在如上所述的一个存储介质(如ROM/RAM、磁碟、光盘)中,包括若干指令用以使得一台终端设备(可以是手机,计算机,服务器,空调器,或者网络设备等)执行本发明各个实施例所述的方法。Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present invention can be embodied in the form of a software product in essence or the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM) , magnetic disk, optical disk), including several instructions to make a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present invention.

以上仅为本发明的优选实施例,并非因此限制本发明的专利范围,凡是利用本发明说明书及附图内容所作的等效结构或等效流程变换,或直接或间接运用在其他相关的技术领域,均同理包括在本发明的专利保护范围内。The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process conversion made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

Claims (10)

1. A display method is applied to a head-mounted display device, wherein a first camera for collecting real scene images is installed on the head-mounted display device, and the display method comprises the following steps:
determining a region of interest of a user in a real scene image;
determining a target exposure of the first camera based on image content of the region of interest;
and acquiring a target real scene image acquired by the first camera based on the target exposure, and displaying the target real scene image.
2. The display method of claim 1, wherein the step of determining the region of interest of the user in the image of the real scene comprises:
the method comprises the steps of acquiring an eye image of a user, and determining an interested area of the user in a real scene image according to the eye image.
3. The display method according to claim 2, wherein the step of determining the region of interest of the user in the image of the real scene from the eye image comprises:
and calculating the fixation point coordinate of the user according to the eye image, and determining the region of interest of the user in the real scene image based on the fixation point coordinate.
4. The display method according to claim 3, wherein the step of determining the region of interest of the user in the image of the real scene based on the gaze point coordinates comprises:
determining a corresponding preset angle range when the gaze content of the human eyes is clear;
and according to the gazing point coordinate and the region of interest of the preset angle range in the real scene image.
5. The display method according to claim 1, wherein the step of displaying the target real scene image is followed by further comprising:
adjusting the virtual image brightness of the virtual scene image based on the real image brightness of the target real scene image to obtain a target virtual scene image;
and fusing and displaying the target virtual scene image and the target real scene image.
6. The display method according to claim 1, further comprising:
determining a peripheral area corresponding to the region of interest on the acquired real scene image, and determining peripheral brightness corresponding to the peripheral area;
adjusting the real scene image corresponding to the peripheral area to the peripheral brightness to obtain a peripheral scene image;
and displaying the target real scene image and the peripheral scene image.
7. The display method according to claim 6, wherein the step of determining the peripheral brightness corresponding to the peripheral area comprises:
determining real image brightness of the target real scene image, and determining peripheral brightness corresponding to the peripheral area based on the real image brightness;
wherein the real image brightness is greater than the peripheral brightness.
8. A display device, comprising:
the interesting area determining module is used for determining the interesting area of the user in the real scene image;
a target exposure determination module for determining a target exposure of the first camera based on image content of the region of interest;
and the display module is used for acquiring a target real scene image acquired by the first camera based on the target exposure and displaying the target real scene image.
9. A head-mounted display device, the head-mounted display device comprising: a first camera capturing images of a real scene, a second camera capturing images of a user's eyes, a memory, a processor, and a computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, implements the steps of the display method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, carries out the steps of the display method according to any one of claims 1 to 7.
CN202211494355.1A 2022-11-25 2022-11-25 Display method, device, head-mounted display device, and storage medium Pending CN115883816A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211494355.1A CN115883816A (en) 2022-11-25 2022-11-25 Display method, device, head-mounted display device, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211494355.1A CN115883816A (en) 2022-11-25 2022-11-25 Display method, device, head-mounted display device, and storage medium

Publications (1)

Publication Number Publication Date
CN115883816A true CN115883816A (en) 2023-03-31

Family

ID=85764140

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211494355.1A Pending CN115883816A (en) 2022-11-25 2022-11-25 Display method, device, head-mounted display device, and storage medium

Country Status (1)

Country Link
CN (1) CN115883816A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118921558A (en) * 2023-05-08 2024-11-08 万有引力(宁波)电子科技有限公司 Image processing method, processing device and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130050432A1 (en) * 2011-08-30 2013-02-28 Kathryn Stone Perez Enhancing an object of interest in a see-through, mixed reality display device
CN112153360A (en) * 2020-08-24 2020-12-29 青岛小鸟看看科技有限公司 Method, device and equipment for determining exposure parameters of head-mounted equipment
CN112584127A (en) * 2019-09-27 2021-03-30 苹果公司 Gaze-based exposure
CN113141502A (en) * 2021-03-18 2021-07-20 青岛小鸟看看科技有限公司 Camera shooting control method and device of head-mounted display equipment and head-mounted display equipment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130050432A1 (en) * 2011-08-30 2013-02-28 Kathryn Stone Perez Enhancing an object of interest in a see-through, mixed reality display device
CN112584127A (en) * 2019-09-27 2021-03-30 苹果公司 Gaze-based exposure
CN112153360A (en) * 2020-08-24 2020-12-29 青岛小鸟看看科技有限公司 Method, device and equipment for determining exposure parameters of head-mounted equipment
CN113141502A (en) * 2021-03-18 2021-07-20 青岛小鸟看看科技有限公司 Camera shooting control method and device of head-mounted display equipment and head-mounted display equipment

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118921558A (en) * 2023-05-08 2024-11-08 万有引力(宁波)电子科技有限公司 Image processing method, processing device and storage medium
WO2024230504A1 (en) * 2023-05-08 2024-11-14 万有引力(宁波)电子科技有限公司 Image processing method and processing apparatus, and storage medium

Similar Documents

Publication Publication Date Title
US9720238B2 (en) Method and apparatus for a dynamic “region of interest” in a display system
CN109633907B (en) Method for automatically adjusting brightness of monocular AR (augmented reality) glasses and storage medium
US10182720B2 (en) System and method for interacting with and analyzing media on a display using eye gaze tracking
US9911214B2 (en) Display control method and display control apparatus
CN101866215B (en) Human-computer interaction device and method adopting eye tracking in video monitoring
RU2672502C1 (en) Device and method for forming cornea image
US20190004600A1 (en) Method and electronic device for image display
EP3382510A1 (en) Visibility improvement method based on eye tracking, machine-readable storage medium and electronic device
CN107193383A (en) A kind of two grades of Eye-controlling focus methods constrained based on facial orientation
CN106484116A (en) Media file processing method and device
US12299192B2 (en) Visual assistance
CN111886564A (en) Information processing apparatus, information processing method, and program
CN111710050A (en) Image processing method and device for virtual reality equipment
CN111556305A (en) Image processing method, VR device, terminal, display system and computer-readable storage medium
CN113552947B (en) Virtual scene display method, device and computer readable storage medium
WO2020215960A1 (en) Method and device for determining area of gaze, and wearable device
JPWO2019021601A1 (en) Information processing apparatus, information processing method, and program
Lander et al. hEYEbrid: A hybrid approach for mobile calibration-free gaze estimation
CN114895790A (en) Man-machine interaction method and device, electronic equipment and storage medium
CN113325947A (en) Display method, display device, terminal equipment and storage medium
KR20250113454A (en) Method, device and eye movement tracking device for determining gaze information
CN115883816A (en) Display method, device, head-mounted display device, and storage medium
CN114816065B (en) Screen backlight adjusting method, virtual reality device and readable storage medium
CN113960788A (en) Image display method, image display device, AR glasses, and storage medium
US10083675B2 (en) Display control method and display control apparatus

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination