WO2023005751A1 - Rendering method and electronic device - Google Patents

Abstract

The present application relates to the technical field of computer graphics. Provided are a rendering method and an electronic device. A rendering duration for a single frame can be effectively shortened by means of parallel rendering, and a screen refresh rate can be guaranteed, thereby improving the use experience of a user. The method comprises: an electronic device acquiring rendering objects to be displayed, wherein the rendering objects comprise a control and a layout; acquiring a control tree which comprises the rendering objects to be displayed, and starting to traverse the control tree from a root node of the control tree, which involves: setting different control tree levels as different rendering levels, and setting rendering objects, which intersect each other, in the same control tree level as different rendering levels; and then, performing parallel rendering on rendering objects in the same rendering level, and performing serial rendering on rendering objects in different rendering levels.

Description

Rendering method and electronic device

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office on July 30, 2021, with application number 202110875756.0, and the title of the invention is "Rendering Method and Electronic Equipment", the entire contents of which are incorporated by reference in this application.

technical field

The embodiments of the present application relate to the technical field of computer graphics, and in particular, to a rendering method and an electronic device.

Background technique

With the development of terminal technology, electronic devices (such as mobile phones, large-screen devices, etc.) can provide users with better display experience. For example, some electronic devices are equipped with displays with a refresh rate of 144 hertz (Hz) per second. Compared with displays with a refresh rate of 60 Hz, the higher refresh rate can make interface operations smoother.

Generally, in order to ensure the refresh rate, it is necessary to reduce the rendering time of a single frame of the display screen, otherwise it is difficult to ensure the full frame display effect of a high refresh rate display, and it is prone to frame loss and stuttering, which affects the user experience. For a display with a refresh rate of 144Hz, the rendering time of a single frame needs to be less than 6ms, and there may be a demand for a higher refresh rate in the future, which requires a shorter rendering time of a single frame. , it is already difficult to complete a single frame rendering within 6ms.

Contents of the invention

The rendering method and electronic device provided by the embodiments of the present application can effectively reduce the rendering time of a single frame through parallel rendering, ensure a screen refresh rate, and improve user experience.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

In the first aspect, the embodiment of the present application provides a rendering method applied to an electronic device, and the method may include: the electronic device, acquiring a rendering object to be displayed, where the rendering object includes a control and a layout, and acquiring a control including the rendering object to be displayed Tree. Traverse the control tree from the root node of the control tree: set different control tree levels to different rendering levels, set the intersecting rendering objects of the same control tree level to different rendering levels; render rendering objects of the same rendering level in parallel, and serialize Render objects of different rendering levels.

In some embodiments, the control tree is used to represent controls and layouts organized in a tree structure. The control tree level is used to represent the tree node relationship between controls and layouts in the control tree determined after traversing the control tree. Each node in the control tree corresponds to a rendering object (that is, a control or a layout), and each control tree level contains one or more rendering objects (that is, including one or more controls, one or more layouts). Then, after obtaining the control tree level, it is necessary to determine whether the rendering objects (that is, controls and layouts) at the same control tree level in the control tree intersect, so as to divide the rendering level according to the intersection.

In some embodiments, serial rendering includes invoking threads to sequentially render rendering objects according to the order of rendering levels; parallel rendering includes invoking multiple rendering threads to simultaneously execute rendering tasks of the same rendering level.

In this way, through the method provided by the embodiment of the present application, the rendering time can be effectively reduced, and the frame dropping phenomenon of the displayed image can be avoided. That is to say, the possibility of frame loss when an electronic device displays an image can be reduced by means of parallel rendering, and the smoothness of the image displayed on the display screen can be ensured, thereby improving the user's visual experience.

In a possible implementation, the rendering objects of the same rendering level are rendered in parallel; including: parallel rendering of the constituent element data in the same instruction group in the rendering objects of the same rendering level, and serial rendering of different instructions in the rendering objects of the same rendering level The constituent element data in the group.

In a possible implementation manner, the constituent element data includes one or more of the following contents: constituent element data of the foreground layer, constituent element data of the content layer, and constituent element data of the background layer.

Among them, the background layer constituent element data corresponds to the background of the control or layout after rendering, the content layer constituent elements correspond to the display content of the control or layout after rendering, and the foreground layer constituent element data corresponds to the foreground of the control or layout after rendering. For example, if the control is a text control, the element data of the content layer of the control includes corresponding text data.

In some embodiments, after the electronic device completes the division of the rendering levels, it can render the data of different rendering levels according to the order of the rendering levels, but for the data of the same rendering level, it is necessary to further determine the composition of the element data in the same rendering level. Rendering order. Optionally, perform instruction grouping on the constituent element data in the same rendering level, and perform control and layout rendering according to the instruction grouping results.

In this way, further classifying the data of the rendering object according to the data of the constituent elements can further optimize the parallel rendering tasks executed in the same rendering level. Wherein, the data of the rendering object includes the following situations: if the rendering object is a control, the data of the rendering object is control data; if the rendering object is a layout, then the data of the rendering object is layout data.

In a possible implementation, the data types of the constituent element data in the same instruction group are the same, and the data types of the constituent element data in different instruction groups are different; or, the constituent element data in the same instruction group are in the corresponding rendering object The execution order of the corresponding instructions in is the same, and the execution order of the corresponding instructions in the corresponding rendering objects of the constituent element data in different instruction groups is different.

In some embodiments, the background layer constituent elements in the same rendering level are divided into one instruction group data, the content layer constituent elements are divided into one instruction group data, and the foreground layer constituent elements are divided into one instruction group data. For example, divide the background data in the same rendering level into an instruction group, divide the text data into an instruction group, and so on. Optionally, record the instruction group group result mark into the corresponding control data or layout data, then in the rendering process, the element data instruction group can be formed according to the background layer and the element data can be formed by the content layer in the same rendering level according to the mark The command group and the foreground layer are rendered in the order of the element data command group.

In this way, according to the category of the constituent element data, the division of the rendering object data is quickly completed, and the data of the command group that can be rendered in parallel is determined to improve the rendering efficiency.

In a possible implementation, setting the intersecting rendering objects at the same control tree level to different rendering levels includes: obtaining the smallest rectangular area corresponding to the rendering objects at the same control tree level; setting the smallest rectangle area at the same control tree level Render objects whose regions intersect are set to different rendering levels.

In some embodiments, the instruction execution order includes, for example, the original rendering order of the component data of the control data and the layout data. Specifically, the original rendering order of the component data of the control data or layout data is generally to first render the background layer to form the element data, then render the content layer to form the element data, and finally render the foreground layer to form the element data. If the control data or layout data does not include a certain layer or layers of constituent element data, it is allowed to skip this layer of data and directly render the next layer of constituent element data. Then, according to the instruction execution sequence, the data with the same instruction execution sequence in the same rendering level is divided into the same instruction group, and the data with different instruction execution sequences is divided into different instruction groups. Optionally, mark and record the grouping result of the instruction group into the corresponding control data or layout data, and then perform rendering according to the marked grouping result of the instruction group and the execution order of the instruction group during the rendering process.

In this way, the problem that the rendering thread pauses and waits when the data of the rendering object does not contain a certain type of constituent element data is avoided, and unnecessary rendering waiting time is reduced.

In a possible implementation manner, the smallest rectangular area corresponding to the rendering object is a rectangular area covering the rendering object and having the smallest area.

In some embodiments, during the process of drawing the control and the layout, the control and the layout have a corresponding drawing area, and if the drawing area partially or completely intersects, it means that the corresponding control and the layout intersect. Optionally, the corresponding drawing area can be obtained according to the control data and layout data, and the smallest rectangular area can be used to represent the drawing area of the control or layout. The smallest rectangular area includes, for example, the smallest area that can cover the corresponding control or layout.

In this way, the electronic device determines the intersection of the rendering objects through the intersection of the smallest rectangular area.

In a possible implementation manner, before rendering the rendering objects of the same rendering level in parallel, the method further includes: obtaining resource conditions. Based on resources, determines the maximum number of rendering threads that are allowed to be called in parallel.

In some embodiments, the electronic device allocates rendering threads to perform data rendering according to the resources available for operation in the CPU and GPU. For example, the UI thread generates a rendering instruction based on the result of the rendering hierarchy division and the instruction group grouping result to trigger the rendering thread to perform data rendering, and the rendering thread performs data rendering after receiving the rendering instruction.

In a possible implementation, rendering the rendering objects of the same rendering level in parallel includes: if the first number of rendering threads required to render the rendering objects of the same rendering level in parallel is less than or equal to the maximum number, calling the first number of Rendering threads render rendering objects of the same rendering level in parallel. If the first number is greater than the maximum number, call the maximum number of rendering threads to render the rendering objects of the same rendering level in parallel.

In some embodiments, after the electronic device completes the instruction group grouping of the data to be rendered, it determines whether the number of rendering threads that can be called in parallel meets the rendering requirements of the instruction group according to the executable resources of the CPU and GPU. If the number satisfies the rendering requirements of the instruction group, a corresponding number of rendering threads will be called to render the data in the instruction group in parallel. If the number of threads that can be called in parallel does not meet the rendering requirements of the instruction group, call the maximum number of rendering threads to render part of the data in the instruction group in parallel, and render all the data in the instruction group in batches.

In this way, before executing the rendering task, the electronic device first determines the maximum number of rendering threads that are allowed to be called according to the resource situation, so as to ensure the smooth execution of parallel rendering tasks and avoid rendering exceptions.

In a second aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory; the memory is coupled to the processor, and the memory is used to store computer program codes, and the computer program codes include computer instructions. When the processor reads from the memory The computer instructions enable the electronic device to perform the following operations to obtain a rendering object to be displayed, where the rendering object includes a control and a layout, and obtain a control tree including the rendering object to be displayed. Traverse the control tree from the root node of the control tree: set different control tree levels to different rendering levels, set the intersecting rendering objects of the same control tree level to different rendering levels; render rendering objects of the same rendering level in parallel, and serialize Render objects of different rendering levels.

In a possible implementation, the rendering objects of the same rendering level are rendered in parallel; including: parallel rendering of the constituent element data in the same instruction group in the rendering objects of the same rendering level, and serial rendering of different instructions in the rendering objects of the same rendering level The constituent element data in the group.

In a possible implementation manner, the constituent element data includes one or more of the following contents: constituent element data of the foreground layer, constituent element data of the content layer, and constituent element data of the background layer.

In a possible implementation, the data types of the constituent element data in the same instruction group are the same, and the data types of the constituent element data in different instruction groups are different; or, the constituent element data in the same instruction group are in the corresponding rendering object The execution order of the corresponding instructions in is the same, and the execution order of the corresponding instructions in the corresponding rendering objects of the constituent element data in different instruction groups is different.

In a possible implementation, setting the intersecting rendering objects at the same control tree level to different rendering levels includes: obtaining the smallest rectangular area corresponding to the rendering objects at the same control tree level; setting the smallest rectangle area at the same control tree level Render objects whose regions intersect are set to different rendering levels.

In a possible implementation manner, the smallest rectangular area corresponding to the rendering object is a rectangular area covering the rendering object and having the smallest area.

In a possible implementation manner, when the processor reads the computer instruction from the memory, it further causes the electronic device to perform the following operation: obtain resource information. Based on resources, determines the maximum number of rendering threads that are allowed to be called in parallel.

In a possible implementation, rendering the rendering objects of the same rendering level in parallel includes: if the first number of rendering threads required to render the rendering objects of the same rendering level in parallel is less than or equal to the maximum number, calling the first number of Rendering threads render rendering objects of the same rendering level in parallel. If the first number is greater than the maximum number, call the maximum number of rendering threads to render the rendering objects of the same rendering level in parallel.

In addition, for the technical effect of the electronic device described in the second aspect, reference may be made to the technical effect of the rendering method described in the first aspect, which will not be repeated here.

In a third aspect, an embodiment of the present application provides an electronic device, where the electronic device has a function of implementing the rendering method described in the foregoing first aspect and any possible implementation manner thereof. This function may be implemented by hardware, or may be implemented by executing corresponding software on the hardware. The hardware or software includes one or more modules corresponding to the above functions.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are run on the electronic device, the electronic device executes any of the steps in the first aspect and any possible implementation manners. One of the described rendering methods.

In the fifth aspect, the embodiment of the present application provides a computer program product, which, when the computer program product is run on the electronic device, enables the electronic device to execute any one of the first aspect and any one of the possible implementations. rendering method.

In a sixth aspect, an embodiment of the present application provides a circuit system, where the circuit system includes a processing circuit configured to execute the rendering method described in the first aspect and any possible implementation manner thereof.

In the seventh aspect, the embodiment of the present application provides a chip system, including at least one processor and at least one interface circuit. When executing the instruction, at least one processor executes the rendering method described in the first aspect and any possible implementation manner thereof.

Description of drawings

Figure 1 is a schematic diagram of the interface provided by the embodiment of the present application;

FIG. 2 is a schematic diagram of the form of an electronic device provided by an embodiment of the present application;

FIG. 3 is a first schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of a software structure block diagram of an electronic device provided in an embodiment of the present application;

FIG. 5 is a flowchart of a rendering method provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of a control tree structure provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a rendering layered scene provided by an embodiment of the present application;

FIG. 8 is a first schematic diagram of an instruction grouping scenario provided by an embodiment of the present application;

FIG. 9 is a second schematic diagram of an instruction group grouping scenario provided by the embodiment of the present application;

FIG. 10 is a schematic diagram of an image frame refresh cycle provided by an embodiment of the present application;

FIG. 11 is a second structural schematic diagram of the electronic device provided by the embodiment of the present application;

FIG. 12 is a third structural schematic diagram of the electronic device provided by the embodiment of the present application.

Detailed ways

The rendering method and the electronic device provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

The terms "including" and "having" mentioned in the description of the embodiments of the present application and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes other unlisted steps or units, or optionally also includes Other steps or elements inherent to the process, method, product or apparatus are included.

It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more. The "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, and B exists alone These three situations.

First, for ease of understanding, related terms and concepts involved in the embodiments of the present application are firstly introduced below.

(1) Controls and control data

A control (view), also referred to as a component (widget) in some embodiments, is a user interface element that can serve as a human-computer interaction interface and receive user operations on electronic devices. For example, the electronic device receives a control command input by the user by detecting the user's operation on the control. Wherein, the control may include a text control (text view), a button (button), a picture control (image view), a progress bar (progress bar), etc.

Exemplarily, the main interface 101 shown in (a) in FIG. 1 has an application program icon displayed on the main interface 101. The application program icon can be used as a control to start the application program. The application program icon here is equivalent to a launcher The control of the application, the launcher is the launcher, that is, the system application that displays the desktop of the user's mobile phone, including displaying and managing application icons and various desktop controls on the desktop. For example, after the mobile phone detects that the user clicks on the control 1011 (here, the application program is taken as an example of the “settings” application), the interface 102 shown in (b) in FIG. 1 is displayed. As shown in the interface 102, the controls may include a picture control 1021, a text control 1022, a button 1023, and the like.

Control data, generated by the application that includes the control. For example, the above-mentioned application icon control of the launcher application is a control used by the launcher to display and facilitate the user to open other installed applications. Therefore, the control data is obtained and generated by the launcher according to the other installed applications, and the data of the control can also be changed in the launcher, for example, the user can adjust the position and size of the control. The electronic device draws the control according to the control data, so as to display the control visible to the user.

(2) Layout and layout data

The layout (layout) is used to indicate the position and size of a certain display area, and the position display effect of the controls in the display area. Types of layout include linear layout, relative layout, absolute layout, etc. Among them, the linear layout is used to indicate that the controls are displayed in a linear arrangement in a horizontal or vertical direction. Relative layout is used to indicate where a control is displayed relative to other controls. Absolute layout is used to indicate the display position of the control in the display interface. For example, the layout of the control A includes that the center pixel of the control A coincides with the center point of the display screen, and the size is 200 pixels (pixels), then the layout of the control A is an absolute layout. The layout of control B includes that the center pixel of control B is located N pixels to the left of the center pixel of control A, and the size of control B is the same as that of control A, so the layout of control B is a relative layout.

Exemplarily, as shown in the interface 102 in (b) of FIG. 1 , the type of layout corresponding to the display position of each setting option such as Bluetooth, mobile network, desktop and wallpaper includes a linear layout.

The layout data includes information about the position and size of a certain display area, the controls contained in the display area, and the type of layout corresponding to the controls.

In some embodiments, the process of displaying images by the electronic device at least includes a user interface (user interface, UI) thread and a rendering thread. Among them, the UI thread is used to control the display, update and control interaction of the UI interface, and can generate graphic data and rendering instructions. The rendering thread completes rendering of control data, layout data, etc. according to graphics data and rendering instructions, and generates image frames.

It should be noted that, during the graphics drawing process of the electronic device, one or more UI threads and rendering threads are called. The UI thread in the embodiment of the present application can also be described as a UI process, UI task, etc. in different operating systems, and the rendering thread in the embodiment of the present application can also be described as a rendering process, rendering task, etc. in different operating systems. It will not be explained below.

In some embodiments, the display frame rate indicates the smoothness of the image displayed by the electronic device. The higher the display frame rate, the smoother the displayed image, and the more realistic display effect can be achieved. At present, users have higher and higher requirements on the display effect of electronic equipment. In order to ensure the display effect, it is necessary to increase the display frame rate, which requires an increase in the image generation rate. Among them, the rendering time of control data, layout data, etc. has a great influence on the image generation rate.

Exemplarily, if the electronic device needs to achieve a high display frame rate of 144HZ, it needs to ensure that the rendering time of a single frame image is less than 6ms. However, in complex scenarios, such as when there are many controls included in a single frame image, it is difficult to ensure that the rendering time of a single frame image is less than 6ms. Electronic devices display image freeze (i.e. frame loss).

Based on this, the embodiment of the present application provides a rendering method, which can effectively reduce the rendering time by layering and parallel rendering the control data and layout data to be rendered, meet the requirements of high display frame rate for image generation rate, and avoid the occurrence of electronic device display images Caton phenomenon.

The rendering method provided in the embodiment of the present application may be applied to electronic devices. For example, as shown in FIG. 2 , the electronic device can specifically be a large-screen display device 21, a mobile phone 22, a notebook computer 23, a tablet computer 24, a vehicle-mounted device 25, a wearable device (such as a smart watch 26), a super mobile personal computer ( Ultra-mobile personal computer, UMPC), netbook, personal digital assistant (personal digital assistant, PDA), artificial intelligence (artificial intelligence) equipment and other terminal equipment with display functions, the embodiment of this application does not make any restrictions on the specific type of electronic equipment .

Exemplarily, FIG. 3 shows a schematic structural diagram of the electronic device 100 .

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (subscriber identification module, SIM) card interface 195 wait.

It can be understood that, the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can be respectively coupled to a touch sensor, a charger, a flashlight, a camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor through an I2C interface, so that the processor 110 communicates with the touch sensor through the I2C bus interface to realize the touch function of the electronic device 100 .

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device 100 .

The USB interface 130 is an interface conforming to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100 , and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present application is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. While the charging management module 140 is charging the battery 142 , it can also provide power for electronic devices through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 .

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

The wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the electronic device 100. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou satellite navigation system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

Wherein, the display screen 194 in the embodiment of the present application may be a touch screen. That is, a touch sensor is integrated in the display screen 194 . The touch sensor may also be referred to as a "touch panel". That is to say, the display screen 194 may include a display panel and a touch panel, and a touch screen composed of a touch sensor and the display screen 194 is also called a “touch screen”. The touch sensor is used to detect a touch operation on or near it. After the touch operation detected by the touch sensor, it can be passed to the upper layer by the driver of the kernel layer (such as the TP driver) to determine the type of the touch event. Visual output related to the touch operation can be provided through the display screen 194 . In some other embodiments, the touch sensor can also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .

Exemplarily, after the electronic device 100 uses the touch sensor to detect the user's touch operation on the display screen 194, it may trigger the position of the controls displayed on the display screen 194 to change, so when the next frame of image is displayed, it needs to re-align the image. layer data for rendering. For example, after the electronic device 100 detects that the user has long pressed and dragged the application icon, the position of the application icon (that is, the control) may change in each frame of the image during the dragging process, and the control data and layout Rendering of data.

The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 121 may be used to store computer-executable program codes including instructions. The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 . The electronic device 100 may implement audio functions through the audio module 170 and the application processor. Such as music playback, recording, etc. Wherein, the audio module may include a loudspeaker, a receiver, a microphone, and an earphone jack.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .

The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.

The SIM card interface 195 is used for connecting a SIM card. The SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100 .

FIG. 4 is a block diagram of the software structure of the electronic device 100 according to the embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, which are respectively the application program layer, the application program framework layer, the system library of Android runtime (Android runtime), and the kernel layer from top to bottom.

The application layer can consist of a series of application packages.

As shown in Figure 4, the application package may include application programs such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 4, the application framework layer can include window managers, content providers, view systems, resource managers, notification managers, etc.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

The activity manager is used to manage the Activity, and is responsible for the startup, switching, scheduling of each component in the system and the management and scheduling of the application program.

Content providers are used to store and retrieve data and make it accessible to applications. Said data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction.

Android Runtime includes core library and virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

The core library includes two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

A system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing, etc.

The two-dimensional graphics engine is a graphics engine for 2D graphics. Common graphics engines include Vulkan, Canvas, etc., for example.

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

Among them, when the user performs an input operation on the electronic device (such as triggering the electronic device to start an application program), the kernel layer can generate a corresponding input event (such as an application program start event) according to the input operation, and report the event to the application framework layer. event. The view system of the application framework layer determines the corresponding control data and layout data of the application, and determines the process logic of the rendering engine rendering, and sends them to the two-dimensional graphics engine of the system library for processing, and then sends the processed data to the kernel The display driver of the layer is used to display the corresponding application interface on the display screen.

FIG. 5 is a schematic diagram of a rendering method provided by an embodiment of the present application. The method may be applied to electronic devices. As shown in FIG. 5 , the method may include S501-S503.

S501. Obtain control data and layout data in the control tree to be drawn, and determine the control tree levels of the controls and layout in the control tree.

In some embodiments, after the electronic device detects an event that triggers rendering, it acquires the rendering object to be displayed and the control tree of the rendering object to be displayed (ie, the control tree to be drawn), and then determines the control of the rendering object in the control tree tree hierarchy. Among them, the rendering objects include controls and layouts. The data of the rendering object includes the following situations: if the rendering object is a control, the data of the rendering object is control data; if the rendering object is a layout, then the data of the rendering object is layout data.

In some embodiments, the event that triggers rendering is, for example, a user's touch operation on the display screen detected by the electronic device, such as clicking an interactive control, or the user's sliding on the display screen (such as sliding up and down, left and right sliding, etc.); or it may be the user’s voice input detected by the electronic device; or it may be an event that automatically switches the screen of the foreground application of the electronic device, etc., which are not limited here. The foreground application is an application corresponding to the interface currently displayed on the display screen of the electronic device.

It should be noted that the screen content of the foreground application can include not only the screen that the user can see, but also the content without user interface, the content of transparent layer, or the content that is blocked by other application interfaces and is invisible to the user. Obtain all control data and layout data to be drawn corresponding to the screen content.

In some embodiments, a control tree (view tree) is used to represent controls and layouts organized in a tree structure. The confirmation of the control tree level is to traverse the control tree to determine the tree node relationship between the controls and the layout in the control tree, and fill in the determined controls and the control tree level of the layout into the corresponding control data and layout data, so as to facilitate subsequent In step S502, the rendering levels are divided according to the control tree level.

It should be noted that "," is used to indicate the object association relationship of "and/or". For example, control data and layout data are used to indicate that control data exists alone, layout data exists alone, and there are three types of control data and layout data at the same time. situation, which will not be described further below.

Exemplarily, it is assumed that controls and layouts in the control tree include absolute layout 0, absolute layout 1, absolute layout 2, absolute layout 3, relative layout 1, button (button) 0, button 1, button 2, button 3, text ( text)0, text1, text2, and text3. As shown in (a) of FIG. 6 , it shows a schematic diagram of the positional relationship between controls and layouts in the control tree. Taking an electronic device configured with an Android system as an example, the electronic device starts an Activity corresponding to a window (window) after detecting a user's touch operation on a display screen, and creates a corresponding control tree. After that, the control data and layout data in the control tree are obtained, and according to the node depth of the control tree where the control tree node is located, it is divided into the level corresponding to the depth of the control tree, and the corresponding control tree can be filled in the corresponding control data and layout data according to the level The level marks are used to divide the rendering levels in the subsequent step S502.

Specifically, in the control tree creation process, the root view (root view) will be determined first, and then the setContentView() method will be called to complete the creation of other nodes above the root view. Then, the node relationship in the control tree can be determined according to the Z-axis value, and the Z-axis value is related to the order of creation of controls and layouts. As shown in (a) in Figure 6, the root view in the current control tree is absolute layout 0, then as shown in (b) in Figure 6, determine that the level of absolute layout 0 in the control tree is control tree level 0, and the control tree The level 0 mark is filled in the layout data corresponding to absolute layout 0.

Afterwards, using absolute layout 0 as the parent node (parentCount), according to the positional relationship shown in (a) in Figure 6, determine that the child nodes (childrenCount) of absolute layout 0 include absolute layout 1, absolute layout 2, and relative layout 1. Therefore, as shown in (b) of FIG. 6 , determine the levels of absolute layout 1, absolute layout 2, and relative layout 1 in the control tree as control tree level 1, and record the mark of control tree level 1 into the corresponding layout data.

Afterwards, take Absolute Layout 1, Absolute Layout 2, and Relative Layout 1 respectively as parent nodes, and determine the child nodes corresponding to each parent node according to the positional relationship shown in Figure 6 (a), for example, the child nodes of Absolute Layout 1 include Button 0 and button 1, the child nodes of absolute layout 2 include text 0 and text 1, and the child nodes of relative layout 1 include button 2, button 3 and absolute layout 3, then determine each child node as shown in (b) in Figure 6 The level in the control tree is control tree level 2, and the mark of control tree level 2 is recorded in the corresponding control data and layout data.

Finally, according to the positional relationship shown in (a) in Figure 6, use absolute layout 3 as the parent node, and determine that its child nodes include text 2 and text 3, and determine that absolute layout 3 is in the control as shown in (b) in Figure 6 The level in the tree is level 3 of the control tree, and the mark of level 3 of the control tree is recorded in the corresponding control data.

In this way, the confirmation of the control tree level is completed according to the above steps, and the data structure of the control tree as shown in (b) in FIG. 6 is obtained. It can be understood that other traversal algorithms can also be used to complete the confirmation of the control tree level of the control tree, which will not be repeated here.

It should be noted that one or more steps in S501 may be completed by the electronic device calling a UI thread. Wherein, the embodiment of the present application does not limit the creation order of nodes in the control tree. For example, the creation order of nodes can be customized by the user. After obtaining the control tree, it is only necessary to determine the control tree level according to the node relationship in the control tree.

In addition, after traversing the control tree to complete the confirmation of the control tree level, if the control data and layout data included in the subsequent control tree have not changed, you only need to maintain the control tree, instead of traversing the control tree before rendering, only inherit previous data. Or, if the control data and layout data included in the control tree are changed, it is only necessary to reconfirm the control tree level and its sub-control tree levels of the changed part.

It can be understood that the above control tree level 0, control tree level 1, control tree level 2 and control tree level 3 are used to represent the order of the control tree levels, which is only an exemplary illustration. For example, the control tree level can also be expressed as a control Tree level 1-control tree level 4, control tree level A-control tree level D, etc.

S502. Divide the rendering levels according to the intersection condition.

In some embodiments, each node in the control tree corresponds to a rendering object (that is, a control or a layout), and each control tree level contains one or more rendering objects (that is, including one or more controls, one or more layout). Then, after obtaining the control tree level, it is necessary to determine whether the rendering objects (that is, controls and layouts) at the same control tree level in the control tree intersect, so as to divide the rendering level according to the intersection.

For example, according to the control tree level, the controls and layouts are initially divided into the corresponding rendering levels (that is, different control tree levels are set to different rendering levels), and then on this basis, the intersecting controls and layouts in the same control tree level Divide into different rendering levels (that is, set intersecting rendering objects at the same control tree level to different rendering levels), and divide disjoint controls and layouts into the same rendering level to complete the division of rendering levels. Optionally, fill in the determined rendering level tags of controls and layouts into corresponding control data and layout data, so as to facilitate subsequent rendering of controls and layouts according to the rendering level in step S503.

In some embodiments, during the process of drawing the control and the layout, the control and the layout have a corresponding drawing area, and if the drawing area partially or completely intersects, it means that the corresponding control and the layout intersect. Optionally, the corresponding drawing area can be obtained according to the control data and layout data, and the smallest rectangular area can be used to represent the drawing area of the control or layout. The smallest rectangular area includes, for example, the smallest area that can cover the corresponding control or layout.

For example, as shown in (a) in Figure 6, the area in the rectangular frame indicated by reference numeral 61 can completely cover the drawing area of text 2 and has the smallest area, then it is the smallest rectangular area corresponding to text 2, and reference numeral 62 indicates The area within the rectangular frame of can completely cover the drawing area of text 3 and has the smallest area, so it is the smallest rectangular area corresponding to text 3. Further, as shown in (b) in Figure 6, both text 2 and text 3 have been set to control tree level 3 (that is, the same control tree level), then it is necessary to determine the two rendering objects (that is, text 2 and text 3 ) to determine the rendering level. As shown in (a) of FIG. 6 , by judging whether the smallest rectangular area intersects, it is further judged whether the two rendering objects intersect.

It should be noted that, in the schematic diagram of the positional relationship between each control and layout in the control tree shown in (a) in Figure 6, the minimum rectangular area corresponding to the control and layout has been shown, that is, the area outside each rectangle shown in the figure The area within the border, which will not be described below.

For the control tree scene shown in Figure 6, how to divide the rendering levels according to the intersection situation is described below.

Exemplarily, as shown in (b) in FIG. 6 , the control tree level information of the control tree is obtained, and the level of the absolute layout 0 is the control tree level 0. As shown in FIG. 7 , the rendering level of the absolute layout 0 is divided into Render level 0.

After that, determine the next level according to the order of the control tree level, that is, the rendering objects corresponding to control tree level 1 include absolute layout 1, absolute layout 2 and relative layout 1, as shown in (a) in Figure 6, relative layout 1 corresponds to If the minimum rectangular area does not intersect with the minimum rectangular area corresponding to absolute layout 1 and the minimum rectangular area corresponding to absolute layout 2, then it is determined that relative layout 1 does not intersect with both absolute layout 1 and absolute layout 2, then it corresponds to the control tree level, such as As shown in FIG. 7 , the rendering level of relative layout 1 is divided into rendering level 1. As shown in (a) of Figure 6, if the smallest rectangular area corresponding to absolute layout 2 intersects the smallest rectangular area corresponding to absolute layout 1, it is determined that absolute layout 2 intersects with absolute layout 1, and the intersecting area of absolute layout 2 is within absolute layout 1 above. Then corresponding to the control tree level, as shown in Figure 7, the rendering level of absolute layout 1 is divided into rendering level 1; the rendering level of absolute layout 2 is divided into rendering level 2 by adding 1 to the corresponding control tree level.

After that, as shown in (b) in Figure 6, confirm the next level, that is, the rendering objects corresponding to control tree level 2 include button 0, button 1, text 0, text 1, button 2, button 3 and absolute layout 3. Among them, as shown in (a) in Figure 6, according to the corresponding minimum rectangular area, it is determined that button 0 and button 1 intersect, and the intersecting area button 1 is above button 0; text 0 and text 1 intersect, and the intersecting area text 1 Above Text0; Button2, Button3, and AbsoluteLayout3 are all disjoint from other render objects. Then corresponding to the control tree level, as shown in Figure 7, the rendering levels of button 0, button 2, button 3 and absolute layout 3 are divided into rendering level 2, and the rendering level of button 1 is divided into rendering level 3. And, since the rendering level of absolute layout 2 is rendering level 2, and text 0 and text 1 are child node controls of absolute layout 2, then add 1 to the rendering level of text 0 on the basis of rendering level 2, and divide it into rendering level 3 ; Add 1 to the rendering level of text 1 on the basis of rendering level 3, and divide it into rendering level 4.

Finally, as shown in (b) of FIG. 6 , confirm the next level, that is, the rendering objects corresponding to level 3 of the control tree include text 2 and text 3 . Wherein, as shown in (a) of FIG. 6 , according to the corresponding minimum rectangular area, it is determined that the text 2 and the text 3 intersect, and the text 3 is above the text 2 in the intersecting area. Then, corresponding to the control tree level, as shown in FIG. 7 , the rendering level of text 2 is divided into rendering level 3 , and the rendering level of text 3 is divided into rendering level 4 .

In this way, the hierarchy of the control tree is traversed to complete the division of the rendering hierarchy of the rendering object.

It can be understood that the above rendering level 0, rendering level 1, rendering level 2, rendering level 3 and rendering level 4 are used to indicate the rendering order, which is only an exemplary illustration. For example, the rendering level can also be expressed as rendering level 1- Rendering level 5, rendering level A-rendering level E, etc.

It should be noted that the division of rendering levels can be confirmed by the UI thread after confirming the control tree level, and then in step S503, the rendering thread directly obtains the rendering level for rendering; or after the rendering thread obtains the control tree level, it divides the rendering according to the control tree level Level; or divide the rendering level by other threads, and the rendering thread performs data rendering after receiving the confirmed rendering level division result and data.

S503. Call the rendering thread to render the control and layout.

In some embodiments, the electronic device invokes the rendering thread to render rendering objects (eg, including controls and layouts) in order of rendering levels. For example, render objects of different rendering levels serially according to the order of rendering levels, and render objects of the same rendering level in parallel. Among them, serial rendering includes calling threads to render rendering objects sequentially according to the order of rendering levels; parallel rendering includes calling multiple rendering threads to perform rendering tasks at the same time.

Exemplarily, as shown in FIG. 7 , the current rendering level includes 5 levels, and the rendering thread is called to perform serial rendering in the order of rendering from rendering level 0 to rendering level 4, that is, serially rendering from a low rendering level to a high rendering level. For example, call the same rendering thread to sequentially render absolute layout 0, absolute layout 1, and button 0. Moreover, the data in the same rendering level is rendered in parallel, for example, after rendering absolute layout 0, call two rendering threads to render absolute layout 1 and relative layout 1 in parallel.

In some scenarios, the control data or layout data includes one or several items of background layer constituent element data, content layer constituent element data, and foreground layer constituent element data. Among them, the background layer constituent element data corresponds to the background of the control or layout after rendering, the content layer constituent elements correspond to the display content of the control or layout after rendering, and the foreground layer constituent element data corresponds to the foreground of the control or layout after rendering. For example, if the control is a text control, the element data of the content layer of the control includes corresponding text data.

Therefore, after the electronic device completes the division of the rendering levels, the data of different rendering levels can be rendered according to the order of the rendering levels, but for the data of the same rendering level, the rendering order of the constituent element data in the same rendering level needs to be further determined. Optionally, perform instruction grouping on the constituent element data in the same rendering level, and perform control and layout rendering according to the instruction grouping results.

For example, as shown in FIG. 7 , button 0 and button 2 are both divided into rendering level 2, assuming that button 0 includes background 1 and content 1, and button 2 only includes content 2. Then, although button 0 and button 1 are at the same rendering level, it is necessary to divide the rendering order of background 1, content 1 and content 2.

Two instruction grouping methods constituting element data are described below.

Method 1: Group the control data and layout data in the same rendering level according to the data categories that make up the element data, divide the data of the same data category into the same instruction group, and divide the data of different data categories into different instruction groups.

In some embodiments, the background layer constituent elements in the same rendering level are divided into one instruction group data, the content layer constituent elements are divided into one instruction group data, and the foreground layer constituent elements are divided into one instruction group data. For example, divide the background data in the same rendering level into an instruction group, divide the text data into an instruction group, and so on. Optionally, record the instruction group group result mark into the corresponding control data or layout data, then in the rendering process, the element data instruction group can be formed according to the background layer and the element data can be formed by the content layer in the same rendering level according to the mark The command group and the foreground layer are rendered in the order of the element data command group.

Exemplarily, as shown in FIG. 8 , assuming that the method described in step S502 is used to determine the rendering levels of the control data and layout data included in the control tree, including rendering level 1 and rendering level 2, the data corresponding to rendering level 1 includes Text 1, Text 3, Text 5, Text 7, and Text 9, and the data corresponding to rendering level 2 include Text 2, Text 4, Text 6, Text 8, and Text 10. Divide the data in the same rendering level into command groups, assuming that each text data includes background layer constituent element data and content layer constituent element data (i.e. text content), as shown in Figure 8, the background layer can be divided into each rendering level The constituent element data is divided into an instruction group (such as instruction group 1 or instruction group 3), and the content layer constituent element data is divided into an instruction group (such as instruction group 2 or instruction group 4). After the instruction group is divided, the data of rendering level 1 includes the data of instruction group 1 and the data of instruction group 2, and the data of rendering level 2 includes the data of instruction group 3 and the data of instruction group 4.

Method 2: According to the execution order of the instructions constituting the element data, group the control data and layout data in the same rendering level into instruction groups. Data with different instruction execution sequences are divided into different instruction groups.

In some embodiments, the instruction execution order includes, for example, the original rendering order of the component data of the control data and the layout data. Specifically, the original rendering order of the component data of the control data or layout data is generally to first render the background layer to form the element data, then render the content layer to form the element data, and finally render the foreground layer to form the element data. If the control data or layout data does not include a certain layer or layers of constituent element data, it is allowed to skip this layer of data and directly render the next layer of constituent element data. Then, according to the instruction execution sequence, the data with the same instruction execution sequence in the same rendering level is divided into the same instruction group, and the data with different instruction execution sequences is divided into different instruction groups. Optionally, mark and record the grouping result of the instruction group into the corresponding control data or layout data, and then perform rendering according to the marked grouping result of the instruction group and the execution order of the instruction group during the rendering process.

Exemplarily, as shown in FIG. 9 , assuming that the method described in step S502 is used to determine the rendering levels of the control data and layout data included in the control tree, including rendering level 1 and rendering level 2, the data corresponding to rendering level 1 includes Text 1, Text 3, Text 5, Text 7, and Text 9, and the data corresponding to rendering level 2 include Text 2, Text 4, Text 6, Text 8, and Text 10. Divide the data in the same rendering level into command groups, as shown in Figure 9, the text 1, text 5, text 7 and text 9 in the rendering level 1 all include the element data of the background layer and the element data of the content layer. Text 3 indicated by mark 91 includes only content layer constituent element data, and does not include background layer constituent element data. The order of instruction execution is to first execute the rendering of the background layer to form the element data, and then execute the rendering of the content layer to form the element data, then the text 3 can directly render the content layer to form the element data during the rendering process because the text 3 does not include the background layer to form the element data, and the rendering layer Other data in 1 need to render the element data of the background layer first, and then render the element data of the content layer. Therefore, if the instruction group is divided according to the element category, it will cause the rendering thread that is responsible for rendering the element data of the background layer of Text 3 to pause when rendering the element data composed of the background layers of Text 1, Text 5, Text 7, and Text 9 during the rendering process. Work, wait for the next instruction group to render, and increase unnecessary rendering waiting time. Therefore, the electronic device can divide instruction groups according to the execution order of instructions in the control. As shown in Figure 9, after the instruction groups are divided, the data of rendering level 1 includes the data of instruction group 1 and the data of instruction group 2, and the data of instruction group 1 includes The background layer constituent element data of text 1, text 5, text 7 and text 9, and the content layer constituent data of text 3; the data of command group 2 includes the content layer constituent elements of text 1, text 5, text 7 and text 9 data.

It should be noted that the command group group can be confirmed by the UI thread after dividing the rendering level, and then the rendering thread directly obtains the rendering level and the command group group for rendering during the rendering process; or, after the rendering thread obtains the control tree level, according to the control tree The level confirms the rendering level and instruction group grouping; or, the electronic device calls other threads to complete the rendering level layering and instruction group grouping according to the control tree level information determined by the UI thread, and then calls the rendering thread to perform rendering according to the rendering level and instruction group grouping.

In some embodiments, based on the above step S503, on the basis of serially rendering rendering objects of different rendering levels, the data of different instruction groups in the same rendering level are rendered serially according to the instruction group order, and the data in the same instruction group are rendered in parallel rendering. Wherein, the rendering order of the command group data corresponds to the rendering order of the element data of the background layer, the element data of the content layer, and the element data of the foreground layer. That is, the component element data in the same instruction group in the rendering objects of the same rendering level are rendered in parallel, and the component element data in different instruction groups in the rendering objects of the same rendering level are rendered serially.

Exemplarily, as shown in FIG. 8 , the control data and layout data in rendering level 1 should be rendered first, and then the control data and layout data in rendering level 2 should be rendered. Among them, the instruction group 1 data and instruction group 2 data belonging to the rendering level 1 are serially rendered, the background layer corresponding to instruction group 1 is rendered first to form element data, and then the content layer corresponding to instruction group 2 is rendered to form element data, and the same Parallel rendering within instruction groups. For example, after the electronic device calls thread 1, thread 2, thread 3, thread 4, and thread 5 in parallel rendering instruction group 1, the background layer constituent element data (that is, background data) is completed, and then calls thread 1, thread 2, thread 3, thread 4 and thread 5 render in parallel the content layer in instruction group 2 to form element data (ie text data).

In some embodiments, the electronic device allocates rendering threads to perform data rendering according to the resources available for operation in the CPU and GPU. For example, the UI thread generates a rendering instruction based on the result of the rendering hierarchy division and the instruction group grouping result to trigger the rendering thread to perform data rendering, and the rendering thread performs data rendering after receiving the rendering instruction.

Exemplarily, after the electronic device completes the instruction group grouping of the data to be rendered, it determines whether the number of rendering threads that can be called in parallel meets the rendering requirements of the instruction group according to the executable resources of the CPU and GPU. If the number of threads that can be called in parallel meets the If the rendering requirement of the instruction group is met, a corresponding number of rendering threads are called to render the data in the instruction group in parallel. If the number of threads that can be called in parallel does not meet the rendering requirements of the instruction group, call the maximum number of rendering threads to render part of the data in the instruction group in parallel, and render all the data in the instruction group in batches.

For example, in the scene shown in Figure 8, according to the grouping results of the final instruction group, the electronic device determines that instruction group 1-instruction group 4 need to call 5 rendering threads for parallel rendering, and the electronic device determines the resources that can be run in the current CPU and GPU If the requirements are met (for example, the maximum number of rendering threads that can be invoked in parallel is greater than or equal to 5), the running resource allows 5 rendering threads to be invoked for parallel rendering, and then threads 1-5 are invoked for parallel rendering of data in the instruction group.

For another example, in the scene shown in Figure 8, the electronic device determines that there are 5 constituent element data (such as background 1-background 5 included in instruction group 1) that need to be rendered in parallel according to the grouping of the final instruction group, assuming that the electronic device determines the most Three rendering threads can be called for parallel rendering (that is, the maximum number of parallel calling rendering threads is 3), then the constituent element data in the command group that needs to be rendered in parallel are grouped again and then rendered serially. For example, among the 5 background layer constituent element data to be rendered included in instruction group 1 shown in FIG. 8, three background layer constituent element data to be rendered are divided into one instruction group and the other two background constituent elements The element data is divided into an instruction group (for example, background 1-background 3 are divided into one instruction group, and background 4 and background 5 are divided into one instruction group). Afterwards, use the 3 rendering threads that can be called in parallel to first render the element data of the 3 background layers to be rendered in parallel, and then call any 2 of the rendering threads to render the remaining 2 background layers to be rendered. Render background 1-background 3, then render background 4 and background 5 in parallel). That is to say, when there are insufficient running resources in the CPU and GPU, the constituent element data divided into parallel rendering is divided into partial serial rendering constituent element data again, so as to realize batch rendering of data to avoid rendering failure.

In some embodiments, the rendering of the rendering object can be performed by the display device, that is, after the electronic device completes the division of rendering levels and the grouping of instruction groups, it sends the result to the display device (that is, other electronic devices with display functions) for rendering and show. In this way, while reducing the rendering time, the display device can also reduce the power consumption of dividing rendering levels and dividing instruction groups.

In this way, through the method provided by the embodiment of the present application, the rendering time can be effectively reduced, and the frame dropping phenomenon of the displayed image can be avoided. That is to say, the possibility of frame loss when an electronic device displays an image can be reduced by means of parallel rendering, and the smoothness of the image displayed on the display screen can be ensured, thereby improving the user's visual experience.

In addition, in some scenarios, if the graphics data (such as control data and layout data) included in adjacent image frames does not change, the graphics data does not need to be redrawn. For example, application icons are displayed on the main interface of the electronic device, and the user presses and drags one of the application icons to a blank display area for display. During the above application icon dragging process, the position and display content of other application icons will not change, so the electronic device can no longer redraw the constant controls and layouts in adjacent image frames, so as to save the power of the electronic device. consumption. That is to say, the rendering method provided in step S501-step S503 above is applied to the graphic data that needs to be drawn, and for the graphic data that does not need to be re-rendered, the rendering result of the previous frame image can be obtained directly.

Exemplarily, as shown in FIG. 10 , the image display process includes an input event phase, a UI data update phase, a rendering phase, a synthesis phase, and a display phase. Wherein, as shown in FIG. 10 , a vertical synchronization signal (vsync) is used to trigger different stages. For example, the vertical synchronization signal 1 may be used to trigger drawing of one or more layers, and trigger rendering of the one or more layers. In the embodiment of the present application, in response to the vertical synchronization signal 1 , the electronic device may draw one or more layers for each application through each of the multiple drawing threads. That is, in response to the vertical synchronization signal 1 , the electronic device can simultaneously execute drawing tasks for one or more applications, so as to draw one or more layers corresponding to each application. Wherein, the rendering method corresponding to each layer may be the rendering method described in the foregoing embodiments. The vertical sync signal 2 can be used to trigger image frame compositing after rendering. The vertical sync signal 3 can be used to trigger display of the synthesized image frame. The signal period of each vertical synchronization signal is determined according to the screen refresh rate of the display screen of the electronic device. For example, the signal period of the vertical synchronization signal is the reciprocal of the screen refresh rate of the display screen (such as LCD or OLED) of the electronic device. Wherein, the screen refresh rate of the electronic device may be the same as the frame rate of the electronic device. A high frame rate of an electronic device is a high screen refresh rate.

As shown in (a) in FIG. 10 , in response to the vertical synchronization signal 1, the electronic device processes the layer data to obtain the corresponding rendering object in the control tree and the control tree level corresponding to the rendering object, and then determines the rendering level of the rendering object. Finally, the rendering thread is called to render the rendering object. In the rendering phase, rendering a (that is, rendering thread a) and rendering b (that is, rendering thread b) are rendered in parallel, and then rendering c (that is, rendering thread c) is executed. In this way, the rendering task is completed within the signal period of the vertical synchronization signal to ensure the execution of subsequent steps. As shown in (b) in Figure 10, rendering a, rendering b, and rendering c all need to be serially rendered in sequence, resulting in failure to complete the rendering task of rendering c within the signal period of the vertical synchronization signal, and the electronic device detection After the vertical synchronous signal 2, there is no rendered image frame that can be synthesized, and then the image frame cannot be refreshed and displayed in response to the vertical synchronous signal 3, and the display screen of the LCD cannot be updated, and the phenomenon of frame loss will occur. In this way, the coherence and fluency of images displayed on the display screen will be affected, thereby affecting the user's visual experience.

It should be noted that in different systems or architectures, the name of the vertical synchronization signal may be different. For example, in some systems or architectures, the name of the signal used to trigger the processing of one or more layers of data (ie, the vertical synchronization signal 1 ) may not be the vertical synchronization signal. However, no matter what the name of the signal is, as long as it is a synchronization signal with similar functions and conforms to the technical idea of the method provided by the embodiment of this application, it should be covered within the scope of protection of this application.

It can be seen from (a) in FIG. 10 that, through the method provided in the embodiment of the present application, frame loss of the displayed image can be avoided. That is to say, the method of the embodiment of the present application can reduce the possibility of frame loss when the electronic device displays an image, and can ensure the smoothness of the image displayed on the display screen, thereby improving the user's visual experience.

The rendering method provided by the embodiment of the present application has been described in detail above with reference to FIGS. 5-10 . The electronic device provided by the embodiment of the present application will be described in detail below with reference to FIG. 11 and FIG. 12 .

In a possible design, FIG. 11 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As shown in FIG. 11 , an electronic device 1100 includes: an acquiring unit 1101 , a dividing unit 1102 and a rendering unit 1103 . The electronic device 1100 may be used to implement the functions of the electronic device involved in the foregoing method embodiments. Wherein, the electronic device 1100 may be the electronic device itself, or may be a functional unit or chip in the electronic device, or a device matched with the electronic device.

Optionally, the acquiring unit 1101 is configured to support the electronic device 1100 to execute step S501 in FIG. 5 .

Optionally, the dividing unit 1102 is configured to support the electronic device 1100 to execute step S502 in FIG. 5 .

Optionally, the rendering unit 1103 is configured to support the electronic device 1100 to execute step S503 in FIG. 5 .

Optionally, the electronic device 1100 shown in FIG. 11 may further include a sending unit (not shown in FIG. 11 ), which is configured to send signals to other communication devices.

Optionally, the electronic device 1100 shown in FIG. 11 may further include a storage unit (not shown in FIG. 11 ), which stores programs or instructions. When the acquiring unit 1101 , dividing unit 1102 and rendering unit 1103 execute the program or instruction, the electronic device 1100 shown in FIG. 11 can execute the rendering method shown in FIG. 5 .

For technical effects of the electronic device 1100 shown in FIG. 11 , reference may be made to the technical effects described in the foregoing method embodiments, and details are not repeated here.

Wherein, the acquisition unit and the sending unit may be collectively referred to as a transceiver unit, which may be implemented by a transceiver or a transceiver-related circuit component, and may be a transceiver or a transceiver module. The operations and/or functions of each unit in the electronic device 1100 are to implement the corresponding process of the rendering method described in the above method embodiment, and for the sake of brevity, details are not described here.

In the case of using integrated units, FIG. 12 shows another possible composition diagram of the electronic device involved in the above embodiment. As shown in FIG. 12 , the electronic device 1200 may include: a processing module 1201 .

The processing module 1201 is used to execute the steps performed by the acquiring unit 1101, the dividing unit 1102 and the rendering unit 1103 shown in FIG. For the sake of brevity, the corresponding process is not repeated here.

The electronic device 1200 may also include a storage module for storing program codes and data of the electronic device. The storage module may be a memory.

Wherein, the processing module 1201 may be a processor or a controller. It can realize or execute various exemplary logical blocks, modules and circuits described in conjunction with the disclosed content of the embodiments of the present application. The processor can also be a combination of computing functions, for example, a combination of one or more microprocessors, a combination of DSP and a microprocessor, and so on.

The embodiment of the present application also provides a chip system, including: a processor, the processor is coupled with a memory, and the memory is used to store programs or instructions, and when the programs or instructions are executed by the processor, the The system on chip implements the method in any one of the foregoing method embodiments.

Optionally, there may be one or more processors in the chip system. The processor can be realized by hardware or by software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like. When implemented by software, the processor may be a general-purpose processor implemented by reading software codes stored in a memory.

Optionally, there may be one or more memories in the chip system. The memory may be integrated with the processor, or may be configured separately from the processor, which is not limited in this embodiment of the present application. Exemplarily, the memory can be a non-transitory processor, such as a read-only memory ROM, which can be integrated with the processor on the same chip, or can be respectively arranged on different chips. The embodiment of the present application is specific to the type of memory, and The arrangement manner of the memory and the processor is not specifically limited.

Exemplarily, the chip system may be a field programmable gate array (field programmable gate array, FPGA), an application specific integrated circuit (ASIC), or a system on chip (SoC), It can also be a central processing unit (central processor unit, CPU), it can also be a network processor (network processor, NP), it can also be a digital signal processing circuit (digital signal processor, DSP), it can also be a microcontroller (micro controller unit, MCU), and can also be a programmable logic device (programmable logic device, PLD) or other integrated chips.

It should be understood that each step in the foregoing method embodiments may be implemented by an integrated logic circuit of hardware in a processor or instructions in the form of software. The method steps disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

Embodiments of the present application also provide a storage medium for storing instructions used by the above-mentioned communication device.

The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores computer instructions, and when the computer instructions are run on the server, the server executes the above-mentioned related method steps to realize the rendering in the above-mentioned embodiments method.

An embodiment of the present application further provides a computer program product, which, when running on a computer, causes the computer to execute the above-mentioned related steps, so as to implement the rendering method in the above-mentioned embodiment.

In addition, the embodiment of the present application also provides a device, which may specifically be a component or a module, and the device may include one or more connected processors and memories; wherein, the memories are used to store computer programs, and one or more computers Programs include instructions. When the instruction is executed by one or more processors, the device is executed to execute the rendering method in the foregoing method embodiments.

Wherein, the device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding method provided above, therefore, the beneficial effects it can achieve can refer to the above-mentioned The beneficial effects of the corresponding method will not be repeated here.

The steps of the methods or algorithms described in connection with the disclosure of the embodiments of the present application may be implemented in the form of hardware, or may be implemented in the form of a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (random access memory, RAM), flash memory, read only memory (read only memory, ROM), erasable programmable read-only memory ( erasable programmable ROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), registers, hard disk, removable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may be located in an application specific integrated circuit (ASIC).

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated according to needs It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working process of the above-described system, device, and unit, reference may be made to the corresponding process in the foregoing method embodiments, and details are not repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of modules or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The above is only a specific implementation of the application, but the protection scope of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application should be covered within the protection scope of the application . Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (11)

A rendering method, characterized in that it is applied to an electronic device, the method comprising: said electronic device, Obtain the rendering object to be displayed, the rendering object includes controls and layouts, Acquiring a control tree including the rendering object to be displayed, Start traversing the control tree from the root node of the control tree: Set different control tree levels to different rendering levels, Set the intersecting rendering objects of the same control tree level to different rendering levels; Render objects of the same rendering level in parallel, Serially render render objects of different render levels. The method according to claim 1, wherein the parallel rendering of rendering objects of the same rendering level includes: The component element data in the same instruction group in the rendering objects of the same rendering level are rendered in parallel, and the component element data in different instruction groups in the rendering objects of the same rendering level are serially rendered. The method according to claim 2, wherein the constituent element data includes one or more of the following contents: constituent element data of the foreground layer, constituent element data of the content layer, and constituent element data of the background layer. The method according to claim 2 or 3, characterized in that, the data types of the constituent element data in the same instruction group are the same, and the data types of the constituent element data in different instruction groups are different; or, the constituent elements in the same instruction group The execution order of the instructions corresponding to the data in the corresponding rendering objects is the same, and the execution orders of the corresponding instructions of the constituent element data in different instruction groups are different in the corresponding rendering objects. The method according to any one of claims 1-4, wherein setting the intersecting rendering objects at the same control tree level to different rendering levels includes: Get the minimum rectangular area corresponding to the rendering object at the same control tree level; Set the rendering objects intersected by the smallest rectangular area in the same control tree level to different rendering levels. The method according to claim 5, wherein the smallest rectangular area corresponding to the rendering object is a rectangular area covering the rendering object and having the smallest area. The method according to any one of claims 1-6, wherein before rendering the rendering objects of the same rendering level in parallel, the method further comprises: access to resources; According to the resource situation, determine the maximum number of rendering threads that are allowed to be called in parallel. The method according to claim 7, wherein the parallel rendering of rendering objects of the same rendering level comprises: If the first number of rendering threads required to render the rendering objects of the same rendering level in parallel is less than or equal to the maximum number, calling the first number of rendering threads to render the rendering objects of the same rendering level in parallel; If the first number is greater than the maximum number, calling the maximum number of rendering threads to render the rendering objects of the same rendering level in parallel. An electronic device, characterized in that it includes: a processor and a memory, the memory is coupled to the processor, the memory is used to store computer program codes, the computer program codes include computer instructions, when the processor The computer instruction is read from the memory, so that the electronic device executes the rendering method according to any one of claims 1-8. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a computer program, and when the computer program is run on an electronic device, the electronic device executes any one of claims 1-8. method described in the item. A computer program product, characterized in that, when the computer program product is run on a computer, the computer is made to execute the method according to any one of claims 1-8.

Images