CN120471816A - Image display method and electronic device - Google Patents

Abstract

The present application provides an image display method and electronic device. The method is performed by the electronic device and includes: acquiring a first image, where the first image is any image to be enhanced and displayed in the electronic device; processing the first image using a preset model to generate a pre-enhanced image corresponding to the first image; calculating the information difference between the pre-enhanced image and the first image, and creating and displaying an enhanced image corresponding to the first image based on the information difference and the first image, wherein the display dynamic range of the enhanced image is greater than or equal to the display dynamic range of the first image. As a result, the process no longer requires the acquisition of scene brightness information or gain maps, and can achieve dynamic range enhancement and display of any image, displaying the corresponding enhanced image, thus having strong applicability.

Description

Image display method and electronic device

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image display method and an electronic device.

Background

With the continuous improvement of various performances of electronic devices, a display screen of the electronic device can support displaying a high dynamic range (HIGH DYNAMIC RANGE, HDR) image, and compared with a traditional standard dynamic range (STANDARD DYNAMIC RANGE, SDR) image, the HDR image can acquire more picture details and display a wider brightness range.

In the related art, if an HDR image is to be displayed, it is generally required to simultaneously acquire scene brightness information or a gain map during the process of acquiring a base image (such as an SDR image), and then tone-map the base image based on the scene brightness information or the gain map to display the corresponding HDR image. However, since the related art needs to collect scene brightness information or gain map, some electronic devices with weak hardware capability may not collect the information, and then the corresponding HDR image cannot be displayed.

Disclosure of Invention

The application provides an image display method and electronic equipment, which can realize the dynamic range enhancement display of any image, display corresponding enhancement images and have stronger applicability.

In a first aspect, the present application provides an image display method, which is performed by an electronic device, comprising:

The method comprises the steps of obtaining a first image, wherein the first image is any image to be enhanced and displayed in electronic equipment, processing the first image through a preset model to generate a pre-enhanced image corresponding to the first image, calculating information difference between the pre-enhanced image and the first image, and creating and displaying an enhanced image corresponding to the first image according to the information difference and the first image, wherein the display dynamic range of the enhanced image is larger than or equal to that of the first image.

The first image (i.e., the base image) may be any image stored in the electronic device, and the electronic device may perform enhanced display on any first image. For example, the first image may be an SDR image, which may display a higher dynamic range when the first image is displayed enhanced, such as from standard dynamic range SDR display to high dynamic range HDR. In the implementation manner of the application, the electronic device can generate the pre-enhanced image by processing the first image, and further, the information difference between the pre-enhanced image and the first image is used as gain information (or gain map) to enhance and display the first image, so that the process does not need to collect scene brightness information or gain map any more, and the dynamic range enhancement display of any image can be realized without depending on the hardware acquisition capability of the electronic device, so that the corresponding enhanced image is displayed, and the method has stronger applicability.

With reference to the first aspect, in some implementations of the first aspect, creating and displaying the enhanced image corresponding to the first image according to the information difference and the first image includes creating the enhanced image according to the information difference, the display brightness of the electronic device, and the first image, and displaying the enhanced image.

In the display process of the enhanced image, besides the information difference, the electronic device can also consider the display brightness of the electronic device, so that the enhanced image is more suitable for the display effect of the electronic device. Thus, the electronic device can create an enhanced image based on the information difference, the current display brightness, and the first image, and display the final enhanced image.

With reference to the first aspect, in some implementations of the first aspect, creating the enhanced image according to the information difference, the display brightness of the electronic device, and the first image includes determining to create the enhanced image according to the information difference, an expandable dynamic range, and the first image, where the expandable dynamic range is determined by the display brightness of the electronic device and metadata corresponding to the information difference.

In some implementations, in order to keep the data uniformity of the first image during the calculation, the electronic device may map the first image to the linear space for normalization processing, and then perform enhanced display on the first image.

For example, assuming that the bit width of the first image (BaseImage or BI) is 8 bits and the transfer function is Gamma 2.2, the electronic device may convert the first image into LINEAR space by bi_line=clip ((BI/255) × 2.2,0,1) and normalize to the [0,1] interval, where ×2.2 represents Gamma 2.2 of the pixel value in the image.

With reference to the first aspect, in some implementations of the first aspect, creating the enhanced image according to the information difference, the expandable dynamic range, and the first image includes creating the enhanced image according to a relational expression including BaseImage ×gap×h/Hmax, baseImage indicating the first image, gap indicating the information difference, H indicating the current expandable dynamic range of the electronic device, hmax indicating a maximum value of the expandable dynamic range of the first image (which may be included in metadata corresponding to the information difference), h= Lheadroom/Lcurrent, lheadroom indicating an expanded luminance that may be provided by the electronic device, lcurrent indicating a current display luminance of the electronic device, lheadroom being greater than or equal to Lcurrent, and Lheadroom being less than or equal to a maximum display luminance that may be provided in a current state of the electronic device.

That is, assuming that the relationship between the display brightness of the electronic device and the current expandable dynamic range H of the electronic device may be expressed as h= Lheadroom/Lcurrent, the electronic device may create the enhanced image according to a relationship expression including BaseImage Gap H/Hmax. In some implementations, after the electronic device creates the enhanced image (ENHANCEDIMAGE or EI), it may also perform normalization processing, for example, the bit width of the enhanced image is 8 bits, the transfer function is gamma2.2, and normalization processing may be performed by the manner of ei_normalized= (clip ((EI-EImin)/(EImax-EImin), 0, 1) ×1/2.2)) ×255, EImax represents the maximum value of EI, and EImin represents the minimum value of EI.

With reference to the first aspect, in some implementations of the first aspect, the preset model may be an artificial intelligence model, for example, a neural network model, to implement dynamic range enhancement on a low dynamic range image, and generate a relatively high dynamic range image.

With reference to the first aspect, in some implementations of the first aspect, the preset model includes a first model and a second model, the first model is used for performing color mapping on pixels in the first image to generate a second image with a different display dynamic range from the first image, and the second model is used for performing detail enhancement on the second image to generate a pre-enhanced image.

Illustratively, the first model may be a color mapping model, and the pixel-by-pixel one-to-one color mapping relationship may be implemented by a convolutional neural network model. The second model can be a detail enhancement model, the output result of the color mapping model can be further enhanced, the problems of artifacts, faults and the like are removed, and an image with higher quality is reconstructed. Then, after the electronic device inputs the first image into the preset model, the pre-enhanced image may be output through processing of the color mapping model and the detail enhancement model, respectively.

With reference to the first aspect, in some implementations of the first aspect, the first model is an image pyramid structure and the second model is a self-encoder structure.

After the electronic device inputs the first image into the first model, the image pyramid structure can be combined with color information of more scales to serve as color prior information, fine adjustment is carried out on the output of the color mapping model according to a primary function, and the first image is converted into an initial pre-enhanced image. Then the original pre-enhanced image can be reconstructed through the self-encoder (namely the encoding-decoding structure), and the obtained color prior information can be combined in the reconstruction process, so that details in the generated pre-enhanced image are more perfect, and the problems of more artifacts, faults and the like are removed.

With reference to the first aspect, in some implementations of the first aspect, calculating the information difference between the pre-enhanced image and the first image includes calculating a ratio between a value of a first channel of each pixel point in the pre-enhanced image and a value of a first channel of a corresponding pixel point in the first image, and determining the ratio of each pixel point as the information difference.

In some implementations, the electronic device may perform pixel-by-pixel comparison of the pre-enhanced image and the first image, taking the information differences between the pixels as the information differences of the whole.

In other implementations, the electronic device may compare the value of the first channel of each pixel in the pre-enhanced image with the value of the first channel of the corresponding pixel in the first image, and calculate the ratio corresponding to each pixel as the overall information difference.

In the case that the first channel is a Y channel, the electronic device may define a ratio between a value of the pixel Y channel in the pre-enhanced image and a value of the corresponding pixel Y channel in the first image as the information difference. Illustratively, the information difference Gap = Y-HDR/Y-SDR, Y-HDR representing the value of the pixel Y channel in the pre-enhanced image, Y-SDR representing the value of the pixel Y channel in the first image.

In the case where the first channel includes an R channel, a G channel, and a B channel, the electronic device may define the ratio between the value of the pixel point R, G, B channel in the pre-enhanced image and the value of the corresponding pixel point R, G, B channel in the first image as the above information difference, respectively. Illustratively, the information difference Gap (R) =r-HDR/R-SDR, gap (G) =g-HDR/G-SDR, gap (B) =b-HDR/B-SDR, R-HDR represents the value of the pixel R channel in the pre-emphasis image, R-SDR represents the value of the pixel R channel in the first image, G-HDR represents the value of the pixel G channel in the pre-emphasis image, G-SDR represents the value of the pixel G channel in the first image, B-HDR represents the value of the pixel B channel in the pre-emphasis image, and B-SDR represents the value of the pixel B channel in the first image.

With reference to the first aspect, in some implementations of the first aspect, after calculating the information difference between the pre-enhanced image and the first image, the method further includes storing the first image and the information difference package in a same image file.

That is, after the electronic device calculates the information difference, the information difference may be encoded and saved, and the saved information difference and the first image are packaged into the same image file for subsequent reading and displaying.

With reference to the first aspect, in some implementations of the first aspect, creating and displaying the enhanced image corresponding to the first image according to the information difference and the first image includes decoding an image file to obtain the first image and the information difference, and creating and displaying the enhanced image based on the information difference and the first image.

Under the condition that the electronic equipment encapsulates and stores the first image and the information difference in the same image file, the electronic equipment can decode the image file to obtain the first image and the information difference in the process of displaying the enhanced image, and then the enhanced image is created and displayed through the information difference and the first image in the process of displaying.

With reference to the first aspect, in some implementations of the first aspect, in a case where the first image and the information difference are stored in a same image file in a package, the method further includes storing metadata corresponding to the information difference, where the metadata includes at least one of a maximum value of the information difference, a minimum value of the information difference, and a maximum value (Hmax) of a scalable dynamic range of the first image.

In other words, the electronic device can save metadata corresponding to the information difference besides the information difference corresponding to the first image, so as to provide more data references when the first image is enhanced and displayed later, and display an enhanced image with a better effect.

In a second aspect, the present application provides an apparatus, which is included in an electronic device, the apparatus having a function of implementing the above first aspect and the behavior of the electronic device in the possible implementation manners of the above first aspect. The functions may be realized by hardware, or may be realized by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the functions described above. Such as a receiving module or unit, a processing module or unit, etc.

In a third aspect, the present application provides an electronic device comprising one or more processors, and a memory;

The memory is coupled to the one or more processors, and the memory is configured to store computer program code, where the computer program code includes computer instructions that the one or more processors call to cause the electronic device to perform any one of the methods of the first aspect.

In a fourth aspect, the present application provides a chip system for application to an electronic device, the chip system comprising one or more processors for invoking computer instructions to cause the electronic device to perform the method of the first aspect and any possible implementation thereof.

Optionally, the chip system further comprises a memory, and the memory is connected with the processor through a circuit or a wire.

Further optionally, the chip system further comprises a communication interface.

In a fifth aspect, the present application provides a computer readable storage medium, including instructions, which when executed on an electronic device, cause the electronic device to perform any one of the methods according to the first aspect.

In a sixth aspect, the present application provides a computer program product comprising computer program code which, when run on an electronic device, causes the electronic device to perform any one of the methods of the first aspect.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

FIG. 2 is a block diagram of a software architecture of an example electronic device according to an embodiment of the present application;

FIG. 3 is a schematic diagram illustrating an example of a high dynamic display of an HDR picture provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of another embodiment of the present application for starting HDR picture high dynamic presentation;

Fig. 5 is a schematic diagram of still another example of a high dynamic display of an HDR picture provided by an embodiment of the present application;

FIG. 6 is a flowchart illustrating an example of an image display method according to an embodiment of the present application;

FIG. 7 is a schematic diagram illustrating a process of processing a preset model according to an embodiment of the present application;

FIG. 8 is a schematic diagram of an example of an image pyramid structure according to an embodiment of the present application;

FIG. 9 is a schematic diagram of an example self-encoder structure provided by an embodiment of the present application;

FIG. 10 is a schematic diagram showing an example of determining an information difference based on a first image and a pre-enhanced image according to an embodiment of the present application;

Fig. 11 is a flowchart of another exemplary image display method according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. In the description of the embodiment of the present application, unless otherwise indicated, "/" means or, for example, a/B may represent a or B, and "and/or" herein is merely an association relationship describing an association object, which means that three relationships may exist, for example, a and/or B, and that three cases, i.e., a alone, a and B together, and B alone, exist. In addition, in the description of the embodiments of the present application, "plurality" means two or more than two.

The terms "first," "second," "third," and the like, are used below for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first", "a second", or a third "may explicitly or implicitly include one or more such feature.

In order to facilitate understanding of the embodiments of the present application, related concepts related to the embodiments of the present application will be briefly described.

1. Dynamic range (DYNAMIC RANGE, DR)

Dynamic range is used to represent the ratio of the maximum to the minimum of the signal. For example, the DR value may be represented by a base 10 logarithm. When the DR value is reflected in the image, it can be understood that there is a highlight region and a dark region in the image, and the ratio between the luminance value of the highlight region and the luminance value of the dark region is the same.

2. High dynamic range HDR

A high dynamic range means that the ratio between the highest value and the lowest value of the signal (i.e., the dynamic range) is greater than or equal to a preset first threshold. Where HDR is reflected in an image, it is understood that there are areas of high light and areas of low light in the image, the ratio between the luminance values of the high light areas and the luminance values of the low light areas exceeding a first threshold, it being understood that the corresponding image may also be referred to as an HDR image.

3. Standard dynamic range SDR

The standard dynamic range refers to a ratio (i.e., dynamic range) between a maximum value and a minimum value of a signal that is less than a preset first threshold and greater than or equal to a preset second threshold (the second threshold is less than the first threshold). Reflecting SDR in an image may be understood as the presence of a highlight region and a dim region in the image, where the ratio between the luminance value of the highlight region and the luminance value of the dim region is less than a preset first threshold and greater than or equal to a preset second threshold, and it is understood that the corresponding image may also be referred to as an SDR image.

4. Red Green Blue (RGB)

RGB represents the three basic colors used for color reproduction in red, green, and blue, i.e., displays, panels, projectors, printers, or other devices. Wherein the color space of RGB is based on independent color coordinates, so that colors can be corresponding to the same color coordinate system in different device use transmissions without being influenced by different color coordinates of the devices.

5. HDR transfer function (transfer functions)

In HDR display of an image, adjustment from real ambient brightness to display brightness of an electronic device is required, so that the displayed image is better, and since human vision is nonlinear to sense brightness and sensitive to darkness more than bright, the adjustment process of real ambient brightness to display brightness of an electronic device is nonlinear, and the process converts a linear signal into a nonlinear signal, that is, based on an HDR transfer function (also referred to as a transfer function). Currently, the more common transfer functions include, but are not limited to, perceptual quantization curves (perceptual quantizer, PQ) and hybrid log-gamma (HLG).

6. Metadata is information that is appended to an HDR image or video that can help a display device display the picture content in a better way, and the content of metadata can include information of maximum brightness, minimum brightness, average brightness, color gamut, etc. of the image. The HDR metadata may include static metadata and dynamic metadata, where the static metadata is a fixed brightness and contrast range formulated for a series of images, and may be applied to a video stream formed by the series of images, and the dynamic metadata is dynamically generated according to each frame of image, and may dynamically adjust brightness and contrast for each frame of image.

With the continuous improvement of various performances of electronic devices, the display screen of the electronic device can support to display an HDR image, and as can be seen from the description above, the HDR image has a higher display dynamic range compared with a traditional SDR image, so that the HDR image can acquire more picture details and display a wider brightness range.

In the related art, if an HDR image is to be displayed, it is generally required to simultaneously acquire scene brightness information or a gain map during the process of acquiring a base image (such as an SDR image), and then tone-map the base image based on the scene brightness information or the gain map to display the corresponding HDR image. For example, the electronic device may have an HDR photographing function, and when the HDR photographing function is turned on, the electronic device may collect the base image and the scene brightness information at the same time, and store the base image and the scene brightness information in one image file, and if the user clicks the display image, the electronic device may tone-map the base image according to the scene brightness information, and display the corresponding HDR image.

It can be seen that, in the related art, the basic image is enhanced and displayed by collecting the scene brightness information in the photographed scene, so that the HDR image is displayed, but for some electronic devices with weak hardware capability (such as weak image collection capability), the scene brightness information may not be collected, so that the corresponding HDR image cannot be displayed. Further, for images acquired in electronic devices that are not photographed scenes, HDR display cannot be performed due to the absence of corresponding scene brightness information or gain maps.

In view of this, an embodiment of the present application provides an image display method, in which an electronic device generates a pre-enhancement image by preprocessing a base image, generates a gain image based on the pre-enhancement image and the base image, and further performs enhancement display on the base image based on the generated gain image, where the process does not need to perform acquisition of scene brightness information or gain image, and does not need to rely on hardware acquisition capability of the electronic device, so that dynamic range enhancement display on any image can be implemented, and a corresponding enhancement image (such as an HDR image) is displayed, which has relatively strong applicability. It can be appreciated that the image display method provided by the embodiment of the application can be applied to electronic devices with image processing capability, such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (augmented reality, AR)/Virtual Reality (VR) devices, notebook computers, personal computers (personal computer, PCs), netbooks, personal digital assistants (personal DIGITAL ASSISTANT, PDAs) and the like, and the embodiment of the application does not limit the specific types of the electronic devices.

Fig. 1 is a schematic structural diagram of an electronic device 100 according to an embodiment of the present application. Taking the example of the electronic device 100 being a mobile phone, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identity module (subscriber identity module, SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (IMAGE SIGNAL processor, ISP), a controller, a memory, a video codec, a digital signal processor (DIGITAL SIGNAL processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a neural hub and a command center of the electronic device 100, among others. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it may be called directly from memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, and the like. The display 194 includes a display panel. The display panel may employ a Liquid Crystal Display (LCD) CRYSTAL DISPLAY, an organic light-emitting diode (OLED), an active-matrix organic LIGHT EMITTING diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microLed, micro-oLed, a quantum dot LIGHT EMITTING diode (QLED), or the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, N being a positive integer greater than 1.

The electronic device 100 may implement photographing functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent recognition of the electronic device 100, for example, image recognition, face recognition, voice recognition, text understanding, etc., can be realized through the NPU.

The internal memory 121 may be used to store computer-executable program code that includes instructions. The processor 110 executes various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data created during use of the electronic device 100 (e.g., audio data, phonebook, etc.), and so on. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like.

It should be understood that the illustrated structure of the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the application, electronic device 100 may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

In examples where the electronic device 100 is a cell phone, the software system of the electronic device 100 may employ a layered architecture, an event driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. In the embodiment of the application, taking an Android system with a layered architecture as an example, a software structure of the electronic device 100 is illustrated.

Illustratively, FIG. 2 is a software architecture block diagram of an electronic device 100 in accordance with an embodiment of the present application. The layered architecture divides the software into several layers, each with distinct roles and branches. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer, an application framework layer, an Zhuoyun rows (Android runtime) and system libraries, and a kernel layer, respectively. The application layer may include a series of application packages.

As shown in fig. 2, the application package may include applications for cameras, gallery, calendar, phone calls, maps, navigation, WLAN, bluetooth, music, video, short messages, etc. It can be appreciated that images taken by a user using a camera, images downloaded over a WLAN network, images captured from video, or the like, may be stored in a gallery application, and subsequent users may respond to a user's viewing operation by the electronic device when viewing the images via the gallery application, displaying the corresponding images.

In some implementations, when a user views an image in a gallery application, the gallery application may provide an HDR display portal corresponding to the image, and if the user selects to display in HDR, the gallery application may perform dynamic range enhancement processing on the image and invoke a display driver to display the HDR image corresponding to the image.

The application framework layer provides an application programming interface (application programming interface, API) and programming framework for the application of the application layer. The application framework layer includes a number of predefined functions.

As shown in FIG. 2, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

The window manager is used for managing window programs. The window manager can acquire the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like. The content provider is used to store and retrieve data and make such data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebooks, etc. The view system includes visual controls, such as controls to display text, controls to display pictures, and the like. The telephony manager is used to provide the communication functions of the electronic device 100. Such as the management of call status (including on, hung-up, etc.). The resource manager provides various resources for the application program, such as localization strings, icons, pictures, layout files, video files, and the like. The notification manager allows the application to display notification information in a status bar, can be used to communicate notification type messages, can automatically disappear after a short dwell, and does not require user interaction. Such as notification manager is used to inform that the download is complete, message alerts, etc.

Android runtime include core libraries and virtual machines. Android runtime is responsible for scheduling and management of the android system.

The core library comprises two parts, wherein one part is a function required to be called by java language, and the other part is an android core library.

The application layer and the application framework layer run in a virtual machine. The virtual machine executes java files of the application program layer and the application program framework layer as binary files. The virtual machine is used for executing the functions of object life cycle management, stack management, thread management, security and exception management, garbage collection and the like.

The system library may include a plurality of functional modules. Such as surface manager (surface manager), media library (media library), three-dimensional graphics processing library (e.g., openGL ES), 2D graphics engine (e.g., SGL), etc. The surface manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications. Media libraries support a variety of commonly used audio, video format playback and recording, still image files, and the like. The media library may support a variety of audio and video encoding formats, such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like. The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is a layer between hardware and software. The inner core layer at least comprises a display driver, a camera driver, an audio driver and a sensor driver.

For easy understanding, the following embodiments of the present application will take an electronic device having a structure shown in fig. 1 and fig. 2 as an example, and specifically describe an image display method provided by the embodiments of the present application with reference to the accompanying drawings and application scenarios.

In some scenarios, taking an electronic device as a mobile phone as an example, as shown in fig. 3, a user may click on a gallery application icon on a desktop interface, triggering a gallery application to run. On the running interface of the gallery application, a list of captured or downloaded images and videos is presented, while on the running interface, a setting control 31 is presented. In the case that the user clicks the setting control 31, the electronic device may display a setting interface corresponding to the library application, on which the switch control 32 for the HDR picture high dynamic display is presented, and if the user triggers to open the switch for the HDR picture high dynamic display, when viewing a certain HDR image (for example, an image captured with the HDR photographing function), the electronic device may display the image in HDR. But for non-HDR images, the electronic device is normally not able to perform HDR display. For this case, the embodiment of the application provides an HDR display portal in the gallery application, so that the electronic device can display any image in HDR.

For example, as shown in fig. 4, when a user clicks any image on the running interface of the gallery application, the electronic device may display the image in a large scale, and multiple operation controls, such as sharing, collecting, editing, deleting, and the like, are presented below the display interface. If the user clicks on the edit control 41, the electronic device can jump to the editing interface of the image on which the HDR display control 42 is presented. If the user clicks on the HDR display control 42, the electronic device may perform dynamic range enhancement processing on the image to obtain and display a corresponding HDR image, and the processing procedure of the image by the electronic device is described in detail in the following embodiments.

In other scenarios, taking the electronic device as a PC, as shown in fig. 5, a storage space (such as a hard disk) of the electronic device may store a plurality of images and videos, and in a case that a user clicks on an image, the electronic device may open the image using an installed picture viewer. If the user wants to display the image in HDR, the electronic device can click on the HDR display control 51 on the image display interface, and perform dynamic range enhancement processing on the image, so as to obtain and display a corresponding HDR image, and the processing procedure of the electronic device on the image is described in the following embodiments.

It should be noted that, the image display method provided by the embodiment of the present application is not only applicable to the above two scenes, but also applicable to any scene that needs to perform HDR display on an image in an electronic device capable of supporting HDR display.

Fig. 6 is a flowchart of an example of an image display method according to an embodiment of the present application, where the method may be executed by an electronic device, and may specifically include:

s101, acquiring a first image.

Wherein the first image (i.e. the base image) may be any one of the images stored in the electronic device. For example, in the case where the electronic device is a mobile phone, the first image may be any one image in a gallery application, and in the case where the electronic device is a PC, the first image may be any one image stored in a hard disk of the PC.

In some implementations, the first image may be an SDR image. In some implementations, the electronic device may perform the following processing on the first image upon receiving a user trigger HDR display operation.

S102, processing the first image through a preset model to generate a pre-enhanced image corresponding to the first image.

The purpose of the preset model to process the first image to generate the pre-enhanced image is that the subsequent electronic device may generate an information difference (or a gain map) based on the pre-enhanced image and the first image, that is, the information difference is obtained by the electronic device through the preset model and calculation, and does not need to be acquired.

In some implementations, where the first image is an 8-bit wide SDR image, the pre-emphasis image may be a 10-bit wide HDR image. It will be appreciated that an 8-bit wide image may provide 256 samples, a 10-bit wide image may provide 1024 samples, and that the color accuracy and the display quality of the 10-bit wide image are better than those of the 8-bit wide image.

In some implementations, the pre-set model may be an artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) model, such as a neural network model, that may enable dynamic range enhancement of low dynamic range images, generating relatively high dynamic range images.

In some implementations, the preset model may include a first model and a second model, the first model may be used to color map pixels in the first image, generate an image with a different display dynamic range than the first image, and the second model may be used to further enhance a result of the color mapping of the first model.

Optionally, the first model may be a color mapping model, the color mapping model may be a convolutional neural network model, that is, the electronic device may implement a pixel-by-pixel one-to-one color mapping relationship through the convolutional neural network model, and the second model may be a detail enhancement model, which may further enhance an output result of the color mapping model, remove problems such as artifacts and faults, and reconstruct an image with higher quality. Then, as shown in fig. 7, after the electronic device inputs the first image into the preset model, the pre-enhanced image may be output through processing of the color mapping model and the detail enhancement model, respectively.

Illustratively, as shown in fig. 8, the color mapping model may be an image pyramid structure, which is one of multi-scale expressions of images, and an effective structure for interpreting images at multiple resolutions. An image pyramid of an image is a series of image resolution sets that decrease progressively (bottom-up) in the shape of a pyramid and that are derived from the same original image, the higher the level of the pyramid, the smaller the image and the lower the resolution. Therefore, after the electronic device inputs the first image into the color mapping model, the image pyramid structure can be combined with color information of more scales to serve as color prior information, fine adjustment is performed on the output of the color mapping model according to a primary function, and the first image is converted into an initial pre-enhanced image. For example, an 8-bit wide SDR image is converted to an initial 10-bit wide HDR image.

Then, as shown in fig. 9, the detail enhancement model may be a self-encoder (autoencoder, AE) structure, which is a type of artificial neural network (ARTIFICIAL NEURAL NETWORKS, ans) used in semi-supervised learning and non-supervised learning, whose function is to characterize learning (representation learning) of the input information by taking the input information as a learning target. The self-encoder comprises an encoder (encoder) and a decoder (decoder), and because the input data and the output data structure are basically consistent, the network structure of the self-encoder can show large-small-large symmetry, and the network from high dimension to low dimension is the encoder, and the network from low dimension to high dimension is the decoder. Therefore, after the electronic device outputs the initial pre-enhancement image through the color mapping model, the self-encoder (namely the encoding-decoding structure) can reconstruct the initial pre-enhancement image, and the obtained color prior information can be combined in the reconstruction process, so that details in the obtained pre-enhancement image are more perfect, and more artifacts, faults and other problems are removed.

S103, calculating an information difference between the pre-enhanced image and the first image, and packaging the first image and the information difference into the same image file.

After the electronic device generates the pre-enhanced image described above, an information difference (Gap) between the pre-enhanced image and the first image (i.e., the base image) may be calculated, which may characterize the pixel difference between the first image and the pre-enhanced image.

In some implementations, the electronic device may perform pixel-by-pixel comparison of the pre-enhanced image and the first image, taking the pixel information differences between the pixels as the overall information differences.

In other implementations, the electronic device may calculate the information difference between the pre-emphasis image and the first image by assuming that the first image is an 8-bit wide SDR image, the encoding format is a YUV format, the pre-emphasis image is a 10-bit wide HDR image, the transfer function is PQ or HLG, and the encoding format is a YUV format, and then the electronic device may define a ratio between a value of a pixel Y channel in the pre-emphasis image and a value of a corresponding pixel Y channel in the first image as the information difference. Alternatively, in the case where the value of the Y channel in the first image and the value of the Y channel in the pre-emphasis image can be converted onto a linear domain, then the information difference gap=y-HDR/Y-SDR, Y-HDR representing the value of the pixel Y channel in the pre-emphasis image and Y-SDR representing the value of the pixel Y channel in the first image. It will be appreciated that for each pixel in the first image, the information difference between it and the pixel at the corresponding position in the pre-emphasis image may be calculated.

Assuming that the first image is an SDR image with 8bit width, the encoding format is an RGB format, the pre-enhanced image is an HDR image with 10bit width, the transfer function is PQ or HLG, and the encoding format is an RGB format, the electronic device may define the ratio between the value of the pixel point R, G, B channel in the pre-enhanced image and the value of the corresponding pixel point R, G, B channel in the first image as the above information difference, respectively. Alternatively, in case the values of the channels in the first image and the values of the channels in the pre-emphasis image can be converted onto a linear domain, the information difference Gap (R) =r-HDR/R-SDR, gap (G) =g-HDR/G-SDR, gap (B) =b-HDR/B-SDR, R-HDR represents the value of the pixel point R channel in the pre-emphasis image, R-SDR represents the value of the pixel point R channel in the first image, G-HDR represents the value of the pixel point G channel in the pre-emphasis image, G-SDR represents the value of the pixel point G channel in the first image, B-HDR represents the value of the pixel point B channel in the pre-emphasis image, and B-SDR represents the value of the pixel point B channel in the first image.

In some implementations, after the electronic device calculates the information difference, the information difference may be saved in code. Alternatively, the electronic device may store the information differences in the form of an image (i.e., an image is formed by the information differences corresponding to each pixel point), where the information differences are stored in the form of an image, the information differences may be represented as a gain map. For example, the electronic device calculates the information difference based on the first image and the pre-enhanced image, and saves the information difference in the form of an image, see fig. 10. Meanwhile, the electronic equipment can package the stored information difference and the first image into the same image file for subsequent reading and displaying.

In some implementations, when the electronic device encodes and saves the information difference, metadata corresponding to the information difference may also be saved. Alternatively, the content contained in the metadata may include, but is not limited to, information difference maximum GapMax, information difference minimum GapMin, and maximum Hmax of the first image's scalable dynamic range.

In some implementations, after the information difference between the pre-enhanced image and the first image is calculated, the electronic device may further normalize (or perform other linear or nonlinear conversion on) the information difference, and then encode and store the normalized information difference, so that the data of the information difference has a certain uniformity. Alternatively, the electronic device may normalize the information difference in such a way that G_normalized= (Gap-GapMin)/(GapMax-GapMin).

S104, creating and displaying an enhanced image corresponding to the first image based on the image file.

After the first image and the information difference are encapsulated, the electronic device may start to create and display an enhanced image corresponding to the first image, for example, display an HDR image corresponding to the SDR image.

In some implementations, in the process of displaying the enhanced image corresponding to the first image, the electronic device may first decode the image file to obtain the first image and the information difference, and then create and display the enhanced image based on the information difference, the display brightness of the electronic device, and the first image in the process of displaying.

The electronic device may create the enhanced image by assuming a bit width of the first image (BaseImage or BI) of 8 bits and a transfer function of Gamma 2.2, by first converting the first image into LINEAR space for normalization, for example by converting the first image into LINEAR space by means of bi_line=clip ((BI/255) × 2.2,0,1), and normalizing to the [0,1] interval, where 2.2 represents the Gamma 2.2 of the pixel values in the image. It is then assumed that the relationship between the display luminance of the electronic device and the current expandable dynamic range H of the electronic device can be expressed as h= Lheadroom/Lcurrent, where Lcurrent is the current display luminance of the electronic device, lheadroom is the expandable luminance that can be provided by the electronic device, where Lheadroom is greater than or equal to Lcurrent and Lheadroom is less than or equal to the maximum display luminance that can be provided by the electronic device in the current state. Then, the enhanced image (ENHANCEDIMAGE or EI) may be obtained by means of ei=bi_line_gap_h/Hmax, where Hmax represents the maximum value of the expandable dynamic range of the first image included in the metadata corresponding to the information difference. Alternatively, the enhanced image may be obtained by BaseImage x Gap H/Hmax without considering normalization of the first image. Still alternatively, where normalization is required to account for information differences, the enhanced image may be obtained by bi_line_g_normalized H/Hmax or BaseImage g_normalized H/Hmax. Thus, the electronic device may display the final enhanced image on the display screen after creating the final enhanced image, for example, the enhanced image may be displayed with a selected bit width of 8 bits or 10 bits.

Optionally, after obtaining the Enhanced Image (EI), the electronic device may also perform normalization processing, for example, the bit width of the enhanced image is 8 bits, and the transfer function is gamma2.2, where the normalization processing may be performed by means of eijnormalized= (clip ((EI-EImin)/(EImax-EImin), 0, 1) ×1/2.2))×255, where EImax represents the maximum value of EI, and EImin represents the minimum value of EI.

In some implementations, the electronic device may also create the enhanced image based only on the information difference and the first image, if the display brightness effect of the electronic device is not considered. For example, the electronic device may create the enhanced image based on a product relationship between the information difference and the first image.

For the implementation process, in some implementation manners, under the condition that the processing performance of the preset model is good enough, for example, the precision of the generated pre-enhanced image is high enough, the electronic device can also directly take the pre-enhanced image generated by the preset model as the final enhanced image, and calculation of information difference is not performed any more, so that the image processing efficiency is improved.

According to the image display method, after the electronic equipment acquires any image, the image can be preprocessed to generate the pre-enhancement image, then the information difference is calculated based on the pre-enhancement image and the basic image (for example, the information difference can be a gain image), and further the enhancement image is created and displayed (for example, an image with a higher dynamic range is displayed) based on the information difference and the basic image.

Based on the procedure of the foregoing embodiment, the image display method provided in the embodiment of the present application will be further described by taking an application scenario as an example, and illustratively, taking the scenario shown in fig. 4 as an example, after the user clicks the HDR display control 42, the electronic device may begin to execute the following image processing procedure, as shown in fig. 11, where the procedure may include:

S201, the gallery application receives a first operation on a first image.

Illustratively, the first operation may be a click operation on the HDR display control 42 in fig. 4, i.e., triggering HDR display of the first image (the image displayed by the current gallery application).

S202, the gallery application calls a preset model to process the first image in response to the first operation, and a pre-enhanced image corresponding to the first image is generated.

That is, after receiving the first operation, the gallery application may respond to the first operation and call the preset model to process the first image. The gallery application may invoke the preset model, for example, via an interface provided by the preset model to generate the pre-enhanced image corresponding to the first image. The process of processing the first image by the gallery application may be referred to the description of S102, which is not repeated herein.

S203, the gallery application calculates an information difference between the pre-enhanced image and the first image, and encapsulates the first image and the information difference into the same image file.

The process of calculating the information difference with respect to the gallery application may be referred to the description of S103 above, and will not be repeated here.

S204, the gallery application decodes the image file to obtain a first image and an information difference.

S205, the gallery application creates an HDR image from the information difference, the display brightness of the electronic device, and the first image.

The implementation process of steps S204 to S205 may be referred to the description of S104.

In some implementations, because the current user triggers the gallery application to display the first image in HDR in real time, in an actual scenario, the gallery application may not package and save the first image and the information difference first, but directly create the HDR image through the information difference, the display brightness of the electronic device, and the first image, so as to improve the processing efficiency.

In other implementations, the gallery application may save the HDR image after creating the HDR image, and if the user triggers HDR display of the first image again next, the gallery application may directly obtain the saved HDR image for display.

S206, the gallery application calls a display driver to display the HDR image.

I.e. the gallery application, after creating the resulting HDR image, may invoke a display driver through an interface provided by the system to display the HDR image.

According to the image display method, when a user triggers HDR display on any image through the gallery application, the gallery application can preprocess the image to generate a pre-enhanced image, then calculate information difference based on the pre-enhanced image and the basic image, and further create and display an enhanced image (for example, display from an SDR image to an HDR image) based on the information difference and the basic image, the process does not need to collect scene brightness information or gain images, and is not limited to shooting the image by a camera, so that dynamic range enhancement display of any image can be realized, corresponding enhanced images are displayed, and the method has strong applicability.

Examples of the image display method provided by the embodiment of the present application are described in detail above. It will be appreciated that the electronic device, in order to achieve the above-described functions, includes corresponding hardware and/or software modules that perform the respective functions. Those of skill in the art will readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may implement the described functionality using different approaches for each particular application in conjunction with the embodiments, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The embodiment of the application can divide the functional modules of the electronic device according to the method example, for example, each function can be divided into each functional module, for example, a detection unit, a processing unit, a display unit, and the like, and two or more functions can be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules. It should be noted that, in the embodiment of the present application, the division of the modules is schematic, which is merely a logic function division, and other division manners may be implemented in actual implementation.

It should be noted that, all relevant contents of each step related to the above method embodiment may be cited to the functional description of the corresponding functional module, which is not described herein.

The electronic device provided in this embodiment is configured to execute the image display method, so that the same effects as those of the implementation method can be achieved.

In case an integrated unit is employed, the electronic device may further comprise a processing module, a storage module and a communication module. The processing module can be used for controlling and managing the actions of the electronic equipment. The memory module may be used to support the electronic device to execute stored program code, data, etc. And the communication module can be used for supporting the communication between the electronic device and other devices.

Wherein the processing module may be a processor or a controller. Which may implement or perform the various exemplary logic blocks, modules and circuits described in connection with this disclosure. A processor may also be a combination that performs computing functions, e.g., including one or more microprocessors, digital Signal Processing (DSP) and a combination of microprocessors, and the like. The memory module may be a memory. The communication module can be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip and other equipment which interact with other electronic equipment.

In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device according to this embodiment may be a device having the structure shown in fig. 1.

The embodiment of the application also provides a computer readable storage medium, in which a computer program is stored, which when executed by a processor, causes the processor to execute the image display method of any of the above embodiments. The storage medium may include a U disk, a removable hard disk, a Read Only Memory (ROM), a random access memory (random access memory, RAM), a magnetic disk or an optical disk, etc. various media that can store program codes.

The embodiment of the application also provides a computer program product, which when run on a computer, causes the computer to execute the above related steps to implement the image display method in the above embodiment.

In addition, the embodiment of the application also provides a device which can be a chip, a component or a module, and the device can comprise a processor and a memory which are connected, wherein the memory is used for storing computer execution instructions, and when the device runs, the processor can execute the computer execution instructions stored in the memory so that the chip can execute the image display method in the method embodiments.

The electronic device, the computer readable storage medium, the computer program product or the chip provided in this embodiment are used to execute the corresponding method provided above, so that the beneficial effects thereof can be referred to the beneficial effects in the corresponding method provided above, and will not be described herein.

It will be appreciated by those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional modules is illustrated, and in practical application, the above-described functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to perform all or part of the functions described above.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.

Claims (15)

1. A method of displaying an image, the method performed by an electronic device, the method comprising:

Acquiring a first image, wherein the first image is any image to be enhanced and displayed in the electronic equipment;

processing the first image through a preset model to generate a pre-enhanced image corresponding to the first image;

And calculating an information difference between the pre-enhanced image and the first image, and creating and displaying an enhanced image corresponding to the first image according to the information difference and the first image, wherein the display dynamic range of the enhanced image is larger than or equal to that of the first image.

2. The method of claim 1, wherein creating and displaying an enhanced image corresponding to the first image from the information difference and the first image comprises:

and creating the enhanced image according to the information difference, the display brightness of the electronic equipment and the first image, and displaying the enhanced image.

3. The method of claim 2, wherein the creating the enhanced image from the information difference, the display brightness of the electronic device, and the first image comprises:

and creating the enhanced image according to the information difference, an expandable dynamic range and the first image, wherein the expandable dynamic range is determined by the display brightness of the electronic device and metadata corresponding to the information difference.

4. A method according to claim 3, wherein said creating said enhanced image from said information difference, said expandable dynamic range and said first image comprises:

Creating the enhanced image according to a relational expression comprising BaseImage x Gap H/Hmax, wherein BaseImage represents the first image, gap represents the information difference, H represents the current expandable dynamic range of the electronic device, hmax represents the maximum value of the expandable dynamic range of the first image contained in the metadata, h= Lheadroom/Lcurrent, lheadroom represents the expandable brightness available to the electronic device, lcurrent represents the current display brightness of the electronic device, lheadroom is greater than or equal to Lcurrent, and Lheadroom is less than or equal to the maximum display brightness available in the current state of the electronic device.

5. The method of any one of claims 1 to 4, wherein the pre-set model comprises a first model for color mapping pixels in the first image to generate a second image having a different display dynamic range than the first image, and a second model for detail enhancement of the second image to generate the pre-enhanced image.

6. The method of claim 5, wherein the first model is an image pyramid structure and the second model is a self-encoder structure.

7. The method according to any one of claims 1 to 6, wherein said calculating an information difference between the pre-enhanced image and the first image comprises:

And calculating the ratio between the value of the first channel of each pixel point in the pre-enhanced image and the value of the first channel of the corresponding pixel point in the first image, and determining the ratio of each pixel point as the information difference.

8. The method of claim 7, wherein the first channel is a Y channel.

9. The method of claim 7, wherein the first channel comprises an R channel, a G channel, and a B channel.

10. The method according to any one of claims 1 to 9, wherein after said calculating the information difference between the pre-enhanced image and the first image, the method further comprises:

and packaging and storing the first image and the information difference into the same image file.

11. The method of claim 10, wherein creating and displaying an enhanced image corresponding to the first image from the information difference and the first image comprises:

decoding the image file to obtain the first image and the information difference;

the enhanced image is created and displayed based on the information difference and the first image.

12. The method according to claim 10 or 11, wherein in case the first image and the information difference package are saved in the same image file, the method further comprises:

and storing metadata corresponding to the information difference, wherein the metadata comprises at least one of an information difference maximum value, an information difference minimum value and a maximum value of the expandable dynamic range of the first image.

13. An electronic device, the electronic device comprising:

one or more processors, and memory;

The memory is coupled with the one or more processors, the memory for storing computer program code comprising computer instructions that are invoked by the one or more processors to cause the electronic device to perform the method of any one of claims 1-12.

14. A chip system for application to an electronic device, the chip system comprising one or more processors to invoke computer instructions to cause the electronic device to perform the method of any of claims 1 to 12.

15. A computer readable storage medium comprising instructions that, when run on an electronic device, cause the electronic device to perform the method of any one of claims 1 to 12.