CN116166257A - Interface generation method and electronic equipment - Google Patents

Abstract

The application discloses an interface generation method and an electronic device. The interface generation method provided by this application can determine the content that needs to be rendered by animation and perform animation rendering on the content in the process of generating the interface based on the rendering tree, and generate an interface with animation effect through one continuous rendering The content prepared in advance before the rendering tree generates the interface and rendered by animation reduces the number of renderings required to generate an interface with animation effects, simplifies the process of generating an interface with animation effects, and reduces the need for electronic devices to generate animation effects. s expenses.

Description

Interface generation method and electronic device

technical field

The present application relates to the field of electronic technology, in particular to an interface generation method and electronic equipment.

Background technique

With the development of electronic technology, more and more electronic devices are involved in users' daily life. When a user uses an electronic device such as a smart phone, the animation effect of the UI interface of the application program directly affects the user's experience.

For example, when the user switches applications, or switches out of the multitasking window through the navigation bar or full-screen gestures, the electronic device will present a blurred background; another example, after the user presses and holds the icons of other applications on the desktop application , the interface other than the icon will become blurred; this blurring effect can make the user perceive a coherent switching animation, thereby improving the user experience.

However, it is obvious that the realization of various animation effects including blurring involves multiple renderings of the view, the realization is more complicated, the computing resources and memory resources are consumed, and the time is longer, which is easy to cause the interface to be blurred. Caton, and will increase the power consumption and heat generation of electronic equipment.

Contents of the invention

Embodiments of the present application provide a method for generating an interface and an electronic device. The interface generation method determined in the embodiment of this application does not need to pre-generate the content after animation when implementing animation effects including blurring on controls or arbitrary areas, but by modifying the rendering nodes and rendering properties of the rendering tree, and based on In the process of generating the interface by the rendering tree, the animation rendering of the control and any area is completed based on the modified rendering nodes and rendering properties, and then an interface with animation effects is generated. The interface generation method provided by the embodiment of the present application reduces the number of times of rendering through a continuous GPU call, and does not need to prepare the content of the animation effect in the interface before the rendering phase, thereby reducing power consumption and heat generation of electronic devices.

In the first aspect, the interface generation method provided by the embodiment of the present application, the method includes: the first process determines the target rendering tree, and the target rendering tree is used to draw the target interface; in the first process, the target interface is generated based on the target rendering tree In the process, the first process determines the first content based on at least one rendering node in the target rendering tree, and performs animation rendering based on the first animation parameters on the first content to obtain the second content, and the at least one rendering The animation parameter of the node is the first animation parameter; or, during the process of the first process generating the target interface based on the target rendering tree, the first process determines the first content based on the first area on the first surface, and performing animation rendering on the first content based on the first animation parameter to obtain the second content, the first process generating the target interface on the first surface based on the target rendering tree, the first animation parameter being the first The process is determined; the first process generates the target interface, the target interface includes the second content, and the animation rendering is performed in an off-screen buffer.

In the above embodiment, in the process of generating the interface based on the rendering tree, the electronic device may draw the content that requires animation rendering, perform animation rendering on the content to obtain the rendered content, and then obtain the target interface. First of all, it does not require the process to prepare the rendered content before the rendering stage, reducing the overhead of multiple renderings, thereby reducing the overhead of electronic equipment.

With reference to some embodiments of the first aspect, in some embodiments, before the first process determines the target rendering tree, the method further includes: the first process adds the at least one rendering node in the target rendering tree, the at least A rendering node is configured with the first animation parameter; or, the first process configures the first animation parameter for the at least one rendering node.

In the above embodiment, the first process can modify the parameters in the target rendering tree. For example, the rendering node corresponding to the control that needs animation rendering can be configured with animation parameters, so that in the process of generating the interface based on the rendering tree, the first process Animation rendering can be performed on controls that require animation rendering; or, for another example, the process can insert one or more rendering nodes configured with animation parameters in the rendering tree, and the one or more rendering nodes themselves and their own child nodes are The content of the animation rendering, and then the animation rendering.

With reference to some embodiments of the first aspect, in some embodiments, when the animation parameter of the at least one rendering node indicates that the foreground is blurred, the first content includes the view corresponding to the at least one rendering node; When the animation parameter of the node indicates that the background is blurred, the first content includes the content of the region corresponding to the at least one rendering node on the first surface.

In the above embodiment, when the animation parameter indicates that the foreground is blurred, the content rendered by animation is the view corresponding to the rendering node configured with animation parameters; when the animation parameter indicates that the background is blurred, the content rendered by animation is For the content of the area corresponding to the first surface of the rendering node configured with animation parameters, only the interface generation method provided by this application can arbitrarily select the content to be animated based on the value of the animation parameters, and then realize various animation effects.

With reference to some embodiments of the first aspect, in some embodiments, before the first process determines the target rendering tree, the method further includes: the first process generates the target rendering tree based on the second rendering tree and the third rendering tree, Wherein, the second rendering tree is a rendering tree generated by the second process for drawing a frame interface of the second process; the third rendering tree is a rendering tree generated by the third process for drawing the third process The rendering tree of the frame interface.

In the foregoing embodiments, the target rendering tree may be synthesized by rendering trees of multiple application programs, and then a target interface including multiple application program interfaces is generated through one GPU call, that is, one rendering.

With reference to some embodiments of the first aspect, in some embodiments, the at least one rendering node includes rendering nodes in the second rendering tree and rendering nodes in the third rendering tree, and the first content includes the second process part or all of the content of the one-frame interface of the third process and part or all of the content of the one-frame interface of the third process.

In the above-mentioned embodiments, the target rendering includes rendering nodes of different application programs, and then part or all of the contents of multiple application program interfaces can be selected at one time for animation rendering, and then the target interface can be obtained.

With reference to some embodiments of the first aspect, in some embodiments, the first rendering node in the second rendering tree is configured with the first animation parameter by the second process; the second rendering node in the third rendering tree The first animation parameter is configured by the third process; the at least one rendering node includes the first rendering node and the second rendering node.

In the above embodiment, the application can decide which controls need animation effects, and then configure animation parameters for the rendering nodes corresponding to these controls. The first process will recognize these animation parameters during the process of generating the interface based on the target rendering tree. Determine the content rendered by the animation, complete the animation rendering, and then generate the interface.

With reference to some embodiments of the first aspect, in some embodiments, when the animation parameter indicates background blur or foreground blur, the animation parameter further includes at least one of blur degree, blur precision, and blur radius.

In the above-mentioned embodiment, in the case that the animation parameter indicates that the background is blurred or the foreground is blurred, the blur degree, blur precision, and blur radius can be further configured. These parameters can be configured by the application program or by the first process. Realize different types of blur with different effects, thereby improving the user experience.

In a second aspect, an embodiment of the present application provides an electronic device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program codes, The computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute: the first process determines the target rendering tree, and the target rendering tree is used to draw the target interface; in the first process based on When the target rendering tree generates the target interface, the first process determines first content based on at least one rendering node in the target rendering tree, and performs animation rendering based on first animation parameters on the first content to obtain the The second content is that the animation parameter of the at least one rendering node is the first animation parameter; or, when the first process generates the target interface based on the target rendering tree, the first process based on the first surface on the first surface An area determines first content, and performs animation rendering on the first content based on first animation parameters to obtain the second content, the first process generates the target interface on the first surface based on the target rendering tree, the The first animation parameter is determined by the first process; the first process generates the target interface, the target interface includes the second content, and the animation rendering is performed in an off-screen buffer.

In the above embodiment, in the process of generating the interface based on the rendering tree, the electronic device may draw the content that requires animation rendering, perform animation rendering on the content to obtain the rendered content, and then obtain the target interface. First of all, it does not require the process to prepare the rendered content before the rendering stage, reducing the overhead of multiple renderings, thereby reducing the overhead of electronic equipment.

With reference to some embodiments of the second aspect, in some embodiments, the one or more processors are further configured to call the computer instruction to make the electronic device execute: the first process adds the at least A rendering node, the at least one rendering node is configured with the first animation parameter; or, the first process configures the at least one rendering node with the first animation parameter.

In the above embodiment, the first process can modify the parameters in the target rendering tree. For example, the rendering node corresponding to the control that needs animation rendering can be configured with animation parameters, so that in the process of generating the interface based on the rendering tree, the first process Animation rendering can be performed on controls that require animation rendering; or, for another example, the process can insert one or more rendering nodes configured with animation parameters in the rendering tree, and the one or more rendering nodes themselves and their own child nodes are The content of the animation rendering, and then the animation rendering.

With reference to some embodiments of the second aspect, in some embodiments, when the animation parameter of the at least one rendering node indicates that the foreground is blurred, the first content includes the view corresponding to the at least one rendering node; When the animation parameter of the node indicates that the background is blurred, the first content includes the content of the region corresponding to the at least one rendering node on the first surface.

In the above embodiment, when the animation parameter indicates that the foreground is blurred, the content rendered by animation is the view corresponding to the rendering node configured with animation parameters; when the animation parameter indicates that the background is blurred, the content rendered by animation is For the content of the area corresponding to the first surface of the rendering node configured with animation parameters, only the interface generation method provided by this application can arbitrarily select the content to be animated based on the value of the animation parameters, and then realize various animation effects.

With reference to some embodiments of the second aspect, in some embodiments, the one or more processors are further configured to invoke the computer instructions to cause the electronic device to execute: the first process is based on the second rendering tree and the third rendering tree to generate the target rendering tree, wherein the second rendering tree is a rendering tree generated by the second process for drawing a frame interface of the second process; the third rendering tree is a rendering tree generated by the third process for Draw a rendering tree of a frame interface of the third process.

In the foregoing embodiments, the target rendering tree may be synthesized by rendering trees of multiple application programs, and then a target interface including multiple application program interfaces is generated through one GPU call, that is, one rendering.

With reference to some embodiments of the second aspect, in some embodiments, the at least one rendering node includes rendering nodes in the second rendering tree and rendering nodes in the third rendering tree, and the first content includes the second process part or all of the content of the one-frame interface of the third process and part or all of the content of the one-frame interface of the third process.

In the above-mentioned embodiments, the target rendering includes rendering nodes of different application programs, and then part or all of the contents of multiple application program interfaces can be selected at one time for animation rendering, and then the target interface can be obtained.

With reference to some embodiments of the second aspect, in some embodiments, the first rendering node in the second rendering tree is configured with the first animation parameter by the second process; the second rendering node in the third rendering tree The first animation parameter is configured by the third process; the at least one rendering node includes the first rendering node and the second rendering node.

In the above embodiment, the application can decide which controls need animation effects, and then configure animation parameters for the rendering nodes corresponding to these controls. The first process will recognize these animation parameters during the process of generating the interface based on the target rendering tree. Determine the content rendered by the animation, complete the animation rendering, and then generate the interface.

With reference to some embodiments of the second aspect, in some embodiments, when the animation parameter indicates background blur or foreground blur, the animation parameter further includes at least one of blur degree, blur precision, and blur radius.

In the above-mentioned embodiment, in the case that the animation parameter indicates that the background is blurred or the foreground is blurred, the blur degree, blur precision, and blur radius can be further configured. These parameters can be configured by the application program or by the first process. Realize different types of blur with different effects, thereby improving the user experience.

In the third aspect, the embodiment of the present application provides a chip system, the chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to invoke computer instructions so that the electronic device executes the first Aspect and the method described in any possible implementation manner of the first aspect.

In the fourth aspect, the embodiment of the present application provides a computer program product containing instructions, when the above computer program product is run on the electronic device, the above electronic device is made to execute any possible implementation of the first aspect and the first aspect described method.

In the fifth aspect, the embodiment of the present application provides a computer-readable storage medium, including instructions, which, when the above-mentioned instructions are run on the electronic device, cause the above-mentioned electronic device to execute any possible implementation of the first aspect and the first aspect. described method.

It can be understood that the above-mentioned electronic device provided in the second aspect, the chip system provided in the third aspect, the computer program product provided in the fourth aspect, and the computer storage medium provided in the fifth aspect are all used to execute the method provided in the embodiment of the present application . Therefore, the beneficial effects that it can achieve can refer to the beneficial effects in the corresponding method, and will not be repeated here.

Description of drawings

1A-1D are exemplary diagrams of animation effects of a single application involved in the present application.

FIG. 2 is an exemplary schematic diagram of interface generation of a single application program provided by the embodiment of the present application.

FIG. 3 is an exemplary schematic diagram of an animation rendering implementation transition interface provided by an embodiment of the present application.

FIG. 4A and FIG. 4B are exemplary schematic diagrams of interfaces for cross-window/cross-application animation effects provided by the embodiment of the present application.

FIG. 5A and FIG. 5B are another exemplary schematic diagrams of an interface for cross-window/cross-application animation effects provided by the embodiment of the present application.

FIG. 6A and FIG. 6B are another exemplary schematic diagrams of an interface for cross-window/cross-application animation effects provided by the embodiment of the present application.

FIG. 7A is an exemplary schematic diagram of the implementation of the cross-window/cross-application animation effect provided by the embodiment of the present application.

FIG. 7B is an exemplary schematic diagram of bitmap changes during the method shown in FIG. 7A .

FIG. 7C is another exemplary schematic diagram of realizing the cross-window/cross-application animation effect provided by the embodiment of the present application.

FIG. 7D is an exemplary schematic diagram of bitmap changes during the method shown in FIG. 7C .

FIG. 8 is an exemplary schematic diagram of an interface generation method provided by an embodiment of the present application.

FIG. 9 is an exemplary schematic diagram of rendering tree generation provided by the embodiment of the present application.

FIG. 10 is an exemplary schematic diagram of a shared memory transfer rendering tree provided by an embodiment of the present application.

FIG. 11A and FIG. 11B are exemplary schematic diagrams of a storage structure of a rendering tree in a shared memory provided by an embodiment of the present application.

FIG. 12 is an exemplary schematic diagram of the architecture of UniRender provided by the embodiment of the present application.

FIG. 13 is an exemplary schematic diagram of an application program writing a rendering tree into a shared memory provided by an embodiment of the present application.

FIG. 14 is another exemplary schematic diagram of the application program writing the rendering tree into the shared memory provided by the embodiment of the present application.

FIG. 15 is an exemplary schematic diagram of synthesizing a rendering tree into a target rendering tree according to an embodiment of the present application.

FIG. 16 is another exemplary schematic diagram of synthesizing a rendering tree into a target rendering tree according to an embodiment of the present application.

Fig. 17 is another exemplary schematic diagram of the process of constructing, rendering and generating an interface of an electronic device provided by the embodiment of the present application.

FIG. 18A is an exemplary schematic diagram of the implementation of foreground blur provided by the embodiment of the present application.

FIG. 18B is an exemplary schematic diagram of the implementation of background blur provided by the embodiment of the present application.

FIG. 19 is an exemplary schematic diagram of cross-application/cross-window animation rendering provided by the embodiment of the present application.

FIG. 20 is another exemplary schematic diagram of cross-application/cross-window animation rendering provided by the embodiment of the present application.

FIG. 21 is an exemplary schematic diagram of pixel processing by blur rendering provided by the embodiment of the present application.

FIG. 22A and FIG. 22B are exemplary schematic diagrams of the blur animation provided by the embodiment of the present application.

FIG. 23A and FIG. 23B are schematic diagrams of an example of the implementation of cross-application/cross-window interface animation effects provided by the embodiment of the present application.

FIG. 24 is an exemplary schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.

FIG. 25 is an exemplary schematic diagram of a software structure of an electronic device provided by an embodiment of the present application.

Detailed ways

The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and appended claims of this application, the singular expressions "a", "an", "the", "above", "the" and "this" are intended to also include Plural expressions, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in this application refers to and includes any and all possible combinations of one or more of the listed items.

Hereinafter, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as implying or implying relative importance or implicitly specifying the quantity of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present application, unless otherwise specified, the "multiple" The meaning is two or more.

The term "user interface (UI)" in the following embodiments of this application is a medium interface for interaction and information exchange between an application program or an operating system and a user, and it realizes the communication between the internal form of information and the form acceptable to the user. conversion. The user interface is the source code written in specific computer languages such as java and extensible markup language (XML). The source code of the interface is parsed and rendered on the electronic device, and finally presented as content that can be recognized by the user. A commonly used representation form of a user interface is a graphical user interface (graphic user interface, GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It may be text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, and other visible interface elements displayed on the display screen of the electronic device.

For ease of understanding, the following first introduces related terms and concepts involved in the embodiments of the present application. The terms used in the embodiments of the present invention are only used to explain specific examples of the present invention, and are not intended to limit the present invention.

In order to improve the user's experience, when the user is using the electronic device, in response to the user's interactive operation, the electronic device may display a transition interface when switching pages and applications to improve the fluency of the UI, thereby improving the user's experience. Among them, the transition interface needs to go through animations such as blurring (Gaussian blur, radial blur, lens blur, etc.), sharpening, distortion, stylization (such as relief effect, oil painting effect, 3D effect, etc.), lens flare, and noise addition. Combination of effects is realized, wherein animation effects require electronic equipment for animation rendering.

(1) First, the interface and implementation of animation effects in several situations involved in the embodiments of the present application will be exemplarily introduced below.

(1.1) Animated interface for a single application

1A-1D are exemplary diagrams of animation effects of a single application involved in the present application.

During the process of using the application program, the user can interact with the electronic device by clicking, long pressing, etc., so that the application program changes the displayed content or switches pages.

For example, the user can click the control 1A01 to switch the played song, such as switching from song 1 to song 2, while using the music application. As shown in FIG. 1A , before the user clicks on the control 1A01 , the interface of the music application displayed on the electronic device includes multiple controls, a song name "Song 1" and a photo of the singer. As shown in FIG. 1D , after the user clicks on the control 1A01, the electronic device plays the next song, and the interface of the music application displayed on the electronic device includes multiple controls, the name of the song "Song 2" and the photo of the singer.

The interface of the application program displayed on the electronic device directly changes from the content shown in FIG. 1A to the content shown in FIG. 1D . However, this way of directly switching the interface is very blunt, and the user experience is poor.

In order to improve the user's experience, when the interface of the application program changes, the transition interface can be used to achieve interface switching, which can be more in line with human visual characteristics. For example, the transition interface may be as shown in FIG. 1B and FIG. 1C. Wherein, the transition interface is an interface after animation rendering.

For the first animation switching method, as shown in Figure 1A, Figure 1B, and Figure 1D, in response to the user clicking on the control 1A01, the electronic device blurs the screen except for the control 1A01, as shown in Figure 1B; then, as shown in Figure 1D As shown, the interface of the music application displayed by the electronic device includes a plurality of controls, the name of the song "song 2" and the photo of the singer.

Or, for the second animation switching mode, as shown in FIGS. 1A, 1B, and 1C, in response to the user clicking on the control 1A01, the electronic device enlarges the control 1A01 and changes the color of the control 1A01, as shown in FIG. 1C; then, As shown in FIG. 1D , the interface of the music application displayed by the electronic device includes a plurality of controls, a song name "song 2", a photo of the singer, and the like.

Obviously, for the first animation switching method, since the content on the interface that needs to be changed is switched by blurring the animation effect, the user will not feel the tearing of the interface switching, which improves the continuity of the interface switching , thereby improving the user experience. Similarly, for the second animation switching method, in response to the user clicking on the control 1A01, the electronic device transforms the size, color, shape and other attributes of the control 1A01 to remind the user that the electronic device has received the user's click operation, thereby improving the user experience. user experience.

(1.2) Realization of the animation effect of a single application

When hardware acceleration is turned on, the realization of the animation effect of a single application is shown in Figure 2 and Figure 3 .

FIG. 2 is an exemplary schematic diagram of interface generation of a single application program provided by the embodiment of the present application.

As shown in Figure 2, the interface generation of a single application can include three steps:

① Invalidate the view hierarchy, and the main thread (UI thread, UI Thread) traverses the views of the application, determines and saves the drawing operation of each view, and relates the view and the drawing operation involved in the view The drawing operation (drawoperationstruct, DrawOP) recorded in the rendering instruction list (displaylist) of the rendering node (RenderNode) of the rendering tree.

Wherein, the rendering node also includes rendering attributes, and the rendering instruction list of the rendering node also includes the entry of the sub-rendering node drawing instruction list. The rendering attribute is used to determine the position and transparency of the drawing content on the surface, and the drawing instruction list is used to determine the drawing content, such as rectangles, lines, circles, etc.

Wherein, a view is a basic element constituting an application program interface, and a control on the interface may correspond to one or more views.

Among them, the drawing operation is a data structure, which is used to draw graphics, such as drawing lines, drawing widening, drawing rectangles, and drawing text. The drawing operation will be converted to the API call of the image processing library in the rendering node, such as the interface call of OpenGL. For example, DrawLineOp is a data structure, which contains the drawn data such as the length and width of the line.

Wherein, the drawing instruction list may be a buffer, which records all drawing operations included in one frame interface of the application program or identifiers of all drawing operations. When an application has multiple windows or is displayed on different display areas (displays), multiple rendering trees need to be independently generated, wherein multiple drawing command lists corresponding to different windows and display areas will be independently generated.

In this embodiment of the present application, the display area may be a screen, or may be a virtual screen (Virtual Display) or the like. The virtual screen may be an area used by the electronic device to carry content displayed on the screen when recording the screen.

Wherein, the rendering tree is generated by the UI thread, and is used to generate a data structure of the application program interface. The rendering tree may include multiple rendering nodes, and each rendering node includes a rendering attribute and a drawing instruction list. The rendering tree records all the information for generating a frame of the application interface.

② The UI thread passes/synchronizes the rendering tree to the rendering thread (Render Thread). Wherein, the rendering tree is located in the stack (stack) of the process corresponding to the application program, and the physical addresses may not be continuously distributed.

③ The Render thread first obtains a hardware canvas (HardwareCanvas), and performs the drawing operation in the rendering tree on the hardware canvas. Wherein, the hardware canvas is located in a surface (surface) held by the application program, and the surface carries bitmap (bitmap) or data for storing image information in other formats. Among them, the surface is provided to the application by the Surface Composer (SurfaceFlinger).

It can be considered that ① is the construction phase, which is mainly responsible for determining the size, position, transparency and other attributes of each view in the application, recording the drawing operation and encapsulating it into the drawing instruction list.

For example, the drawLine in the view in the view can be encapsulated into a DrawLineOp in the construction, which contains the drawn data such as the length and width of the line, and can also contain the interface call corresponding to the drawLine of the underlying graphics processing library.

Similarly, it can be considered that ③ is the rendering stage, which is mainly responsible for performing the drawing operation in the rendering tree on the surface to generate a bitmap. In this process, the Render thread calls the underlying graphics processing library, such as OpenGL, and then calls the GPU to complete the rendering. Generate a bitmap.

Among them, the UI thread is the main thread of the application program, and the Render thread is the thread responsible for rendering. Among them, a bitmap, as a class, can carry image information, for example, it can contain information such as the position of a pixel and the value of a pixel.

Wherein, the bitmap generated in the rendering stage is the bitmap used to present the transition interface, as shown in FIG. 1B .

Among them, SurfaceFlinger is a system-level service for managing and consuming the currently available surface, which carries the bitmap generated by the application Render thread. SurfaceFlinger can also be used to combine multiple surfaces as layers.

Among them, the construction phase needs to occupy the computing resources of the CPU, and the rendering phase needs to occupy the resources of the GPU.

It is worth noting that if hardware acceleration is not enabled, that is, in the case of software drawing, all operations in the construction phase and rendering phase are completed through the UI thread, and there is no need to generate a rendering tree. After traversing the views of the application and drawing the views After the operation, apply for an anonymous shared memory from SurfaceFlinger, and directly render on the memory by calling the interface provided by the underlying graphics library, such as the skia library, to generate a bitmap or other format data for storing image information.

For the transition interface shown in Figure 1C, in response to the user clicking on control 1A01, the application program can modify the rendering properties and drawing operations of the rendering node corresponding to control 1A01 in the rendering tree, and then make the Render thread execute the drawing command list. During the drawing operation, the control 1A01 which is enlarged and whose color changes can be drawn on the surface.

It is worth noting that if you only change the color, size, and transparency of a control in the interface of the application, you can generate a transitional interface by modifying the rendering properties of the rendering node corresponding to the control. However, obviously, the application program cannot change the color, size, and transparency of any delimited area.

For the transition interface shown in FIG. 1B , the transition interface needs to be realized through animation rendering, and the specific process is shown in FIG. 3 .

FIG. 3 is an exemplary schematic diagram of an animation rendering implementation transition interface provided by an embodiment of the present application.

As shown in Figure 3:

① First, the UI thread of the application draws the background bitmap. Wherein, the background bitmap does not include the control 1A01.

②Secondly, the UI thread of the application performs animation rendering on the background bitmap, and performs pixel-by-pixel processing on the background bitmap to achieve animation effects such as blurring. Among them, the UI thread can call the CPU for animation rendering.

③ Again, when the UI thread of the application generates the rendering tree, it adds drawing operations or rendering nodes. The drawing operation, such as drawBitmap, can import the background bitmap after animation rendering as the background into the bitmap generated in the rendering stage; or, the rendering node corresponds to the background bitmap after animation rendering, and in the rendering stage, this rendering node will make The Render thread imports the animated background bitmap as the background to the surface.

④ Finally, when the Render thread of the application generates a bitmap based on the rendering tree, it generates a transition interface as shown in Figure 1B.

Obviously, to realize the animation rendering in the application, the UI thread needs to prepare the bitmap after the animation rendering before the rendering phase starts, so that the UI thread is occupied by the time-consuming operation of animation rendering, which may easily cause the application to freeze. In addition, the UI thread calls the CPU to perform animation rendering, but the efficiency of the CPU to perform image processing operations is low, which further aggravates the freeze of the application.

(1.3) Cross-window/cross-application animation interface

FIG. 4A and FIG. 4B are exemplary schematic diagrams of interfaces for cross-window/cross-application animation effects provided by the embodiment of the present application.

As shown in FIG. 4A , the interface displayed by the electronic device includes an interface of a desktop (launcher) application and an interface of a wallpaper application. The interface of the desktop application includes several application icon controls, such as the icon control 4A01 of the reading application.

In response to the user's interaction with the icon control 4A01 , such as long pressing, the electronic device displays the interface of a blurred wallpaper application and a partially blurred interface of a desktop application, as shown in FIG. 4B . Wherein, in the interface of the desktop application, the icon control 4A01 of the reading application and the interface of the control bar 4A02 are not blurred.

FIG. 5A and FIG. 5B are another exemplary schematic diagrams of an interface for cross-window/cross-application animation effects provided by the embodiment of the present application.

As shown in FIG. 5A , the electronic device displays an interface of a gallery application program. In response to user interaction, such as a full-screen gesture (such as sliding up from the bottom to the center of the screen) or clicking the multitasking control on the bottom navigation bar, the electronic device displays a multitasking interface as shown in FIG. 5B .

As shown in FIG. 5B , the multitasking interface includes a gallery interface and a blurry wallpaper interface.

FIG. 6A and FIG. 6B are another exemplary schematic diagrams of an interface for cross-window/cross-application animation effects provided by the embodiment of the present application.

In a split-screen scenario, as shown in FIG. 6A , the upper half of the screen of the electronic device displays an interface of a desktop application, and the lower half displays an interface of a music application. Wherein, the interface of the music application program includes a control 6A01.

After the user clicks the control 6A01, the interface presented by the electronic device is as shown in FIG. 6B or FIG. 6A.

As shown in FIG. 6B , after the user clicks the control 6A01 , the electronic device displays a blurred interface of a desktop application program and a partially blurred interface of a music application program. Wherein, the interface of the control 6A01 in the interface of the music application program is not blurred.

Apparently, the electronic device displays the interface of the application program to the user through animation effects such as blurring and other means, which helps to improve the user experience.

It is worth noting that the interface of the animation effect of multiple windows (windows) of a single application is similar to the interface of cross-applications, and will not be repeated here.

(1.4) Implementation of cross-window/cross-application animation effects

Different from the realization of the animation effect of a single application, the realization of the animation effect of multiple applications involves the process of layer composition of different applications.

Combining with the content shown in FIG. 7A, the realization of the animation effect of multiple application programs is exemplarily introduced below:

FIG. 7A is an exemplary schematic diagram of the implementation of the cross-window/cross-application animation effect provided by the embodiment of the present application.

As shown in FIG. 7A, the realization of animation effects of multiple application programs may include:

① The UI thread of application 1 copies the interface of application 2, for example, application 1 obtains the interface of application 2 through the screenshot interface, which is recorded as bitmap 1. Wherein, the screenshot interface will trigger the application program 2 to rebuild, render and generate the bitmap 1 . Wherein, the bitmap 1 may be in the heap memory of the application program 1. Alternatively, the screenshot interface may obtain the interface of the last frame of the application program 2. ②The UI thread of application 1 is performing animation rendering on bitmap 1 to generate bitmap 2. Wherein, the animation rendering can be completed by calling computing resources of the CPU. ③ The UI thread software of the application program 1 draws the interface of the application program 1, which only includes the part that needs to be animated and rendered, which is recorded as bitmap 3. ④ The UI thread performs animation rendering on bitmap 3 to generate bitmap 4. ⑤The UI thread of application 1 superimposes the content of bitmap 2 and bitmap 4 to generate bitmap 5, and inserts bitmap 5 into the interface view structure of the application program as an animation layer. ⑥The Render thread of application 1 imports bitmap 5 as the background during the process of generating bitmap based on the rendering tree, and generates bitmap 6. ⑦The UI thread of application 2 generates the rendering tree. ⑧ The Render thread of the application generates bitmap 7 based on the rendering tree.

SurfaceFlinger/hardware composition strategy module (Hardware Composer, HWC) obtains the bitmap 6 generated by application 1 and the bitmap 7 generated by application 2 through the surface, and performs layer composition to obtain the bitmap for display (also called layer for display).

Obviously, the realization of the animation effect shown in Figure 7A requires the UI thread to prepare the bitmap after the animation rendering before the rendering stage starts, so that the UI thread is occupied by the time-consuming operation of animation rendering, which may easily cause the application to freeze. In addition, the UI thread calls the CPU to perform animation rendering, but the efficiency of the CPU to perform image processing operations is low, which further aggravates the freeze of the application.

In addition, there may be a delay in obtaining the interface of other applications through the screenshot interface. In the case that the interface of application 2 keeps changing (for example, the contents of bitmap 1 and bitmap 7 are different), application 1 cannot get the interface of other applications. to the interface of the latest application 2. In addition, the screenshot interface will call the GPU, which will increase the number of calls to the GPU and increase the calculation overhead.

Next, taking the interface shown in FIG. 4B as an example, the contents of bitmap 1 to bitmap 6 in FIG. 7A are exemplarily introduced in conjunction with FIG. 7B .

FIG. 7B is an exemplary schematic diagram of bitmap changes during the method shown in FIG. 7A .

In the interface shown in FIG. 4B , application 1 in FIG. 7A is a desktop application, and application 2 is a wallpaper application.

As shown in Figure 7B, bitmap 1 is the interface of the wallpaper application obtained by the desktop application through the screenshot interface; bitmap 2 is the generated bitmap after blurring bitmap 1; bitmap 3 does not include the control 4A01 The interface of the desktop application program; bitmap 4 is the bitmap generated after blurring bitmap 3; bitmap 5 is the bitmap generated by superimposing the pixels of bitmap 4 and bitmap 5; bitmap 6 is the desktop application interface; bitmap 7 is the interface of the program for wallpaper application.

Among them, blur rendering is a kind of animation rendering.

It is worth noting that the white in bitmap 1 to bitmap 7 in Figure 7B (such as the content outside the icon on the desktop application) is the pixel that the application does not draw, which can be empty, or can be a default value, such as black pixels(255, 255, 255). The pixel may participate in the calculation of animation rendering, or may not participate in the calculation of animation rendering.

FIG. 7C is another exemplary schematic diagram of realizing the cross-window/cross-application animation effect provided by the embodiment of the present application.

Different from the content shown in FIG. 7A , the realization of the animation effect shown in FIG. 7C mainly relies on SurfaceFlinger to perform off-screen rendering of the content requiring animation rendering.

As shown in FIG. 7C, the realization of animation effects of multiple applications includes:

①Application 1 and application 2 are independently constructed and rendered to generate bitmap 1 of application 1 and bitmap 2 of application 2. Among them, generating bitmap 1 and bitmap 2 needs to call CPU and GPU respectively.

② The Render thread of application 1 passes bitmap 1 to SurfaceFlinger, and SurfaceFlinger copies the content that needs animation rendering in bitmap 2 to off-screen buffer 1. The content in off-screen buffer 1 is called bitmap 3. Wherein, the off-screen buffer can be applied for and constructed in advance, or can be applied for and constructed after responding to the animation rendering request.

③ The Render thread of application 2 transfers bitmap 2 to SurfaceFlinger, and copies the content in bitmap 2 that needs animation rendering to off-screen buffer 2. The content in off-screen buffer 2 is called bitmap 4.

④SurfaceFlinger blurs the content in off-screen buffer 1 and off-screen buffer 2 respectively, and generates bitmap 5 and bitmap 6 respectively. Among them, generating bitmap 5 and bitmap 6 needs to call the GPU respectively.

⑤SurfaceFlinger performs layer synthesis on bitmap 1, bitmap 2, bitmap 5, and bitmap 6. Among them, the layer synthesis calls the GPU once. Wherein, the content of layer synthesis is bitmap 7, and bitmap 7 is copied to the on-screen buffer (on-screenbuffer).

Among them, the off-screen buffer is an off-screen buffer developed by SurfaceFlinger, which is used to store the intermediate state of the layer/bitmap. The on-screen buffer is a buffer in the GPU that can be obtained by the display subsystem, carries a bitmap, and can be directly sent to the display.

Obviously, SurfaceFlinger completes cross-window/cross-application animation effects through off-screen rendering in the process of compositing bitmaps of multiple applications as layers. However, SurfaceFlinger performs animation rendering through off-screen rendering. Since the graphics processing library called by animation rendering depends on the context, context switching caused by animation rendering on different off-screen buffers leads to large computing and memory overheads for electronic devices.

Secondly, when SurfaceFlinger implements cross-window/cross-application animation effects, it needs to generate multiple off-screen buffers to save different intermediate results, and it needs to synthesize multiple layers carrying intermediate results through layer composition, which requires computational overhead and memory. The overhead is high. Furthermore, the intermediate results in different off-screen buffers have a large amount of repeated content, which wastes computing resources and memory resources of the electronic device.

Taking the interface shown in FIG. 4B as an example, the contents included in bitmap 1 to bitmap 7 in FIG. 7C are exemplarily introduced in conjunction with FIG. 7D .

FIG. 7D is an exemplary schematic diagram of bitmap changes during the method shown in FIG. 7C .

As shown in Figure 7D, bitmap 1 is a bitmap generated by desktop application construction and rendering; bitmap 2 is a bitmap generated by wallpaper application construction and rendering; bitmap 3 is the content that needs to be blurred in bitmap 1; Fig. 4 is the content that needs to be blurred in bitmap 2; Bitmap 5 is the bitmap after bitmap 3 is blurred; Bitmap 6 is the bitmap after bitmap 4 is blurred; Bitmap 7 is bitmap 1, Bitmap 2, Bitmap 5, and Bitmap 6 are synthesized and used to send the bitmap for display.

Obviously, as shown in FIG. 7A above, in order to achieve cross-window/cross-application animation rendering, the animation rendering needs to be completed through software drawing. However, since the hardware structure of the CPU is not suitable for graphics operations compared to the GPU, the energy efficiency ratio of software rendering is low, and the rendering speed of animation is slow, which may easily cause freezes.

Obviously, as shown in Figure 7C above, in order for SurfaceFlinger to achieve cross-window/cross-application animation rendering, it is necessary to open up multiple off-screen buffers and complete animation rendering through off-screen rendering. In this process, there are a lot of repeated operations, and the animation rendering speed is slow, which may easily lead to freeze.

(2) Next, the interface generation method provided by the embodiment of the present application is introduced below.

In the interface generation method provided by the embodiment of the present application, firstly, the rendering tree of one or more applications is obtained through the UniRender process, and one or more rendering trees are reassembled into a target rendering tree; secondly, the UniRender process is based on the target rendering tree Renders to a bitmap that carries information about one or more application images. The UniRender process can perform animation rendering during the process of rendering and generating bitmaps.

Obviously, first of all, the interface generation method provided by the embodiment of the present application merges the target rendering tree of one or more application programs into a target rendering tree, and can generate bitmaps carrying image information of one or more application programs through one rendering. image, avoiding the need to obtain bitmaps of other applications through screenshot calls, reducing the number of calls to the GPU, thereby reducing power consumption.

Secondly, the interface generation method provided by the embodiment of the present application can reduce the number of off-screen buffers that need to be opened by performing animation rendering during the process of rendering and generating bitmaps, thereby reducing computing costs and memory costs.

Thirdly, since the bitmap produced by rendering is a bitmap that carries the image information of one or more applications, SurfaceFlinger/HWC may not be required for layer synthesis, which can reduce at least one GPU call, further saving calculation and memory overhead.

The interface generation method provided by the embodiment of the present application is exemplarily introduced below.

FIG. 8 is an exemplary schematic diagram of an interface generation method provided by an embodiment of the present application.

As shown in Figure 8, the interface generation method provided by the embodiment of the present application includes:

S801: Build and generate a rendering tree after receiving a vertical signal.

When the application needs to update the interface, it can request a vertical synchronization signal (Vsync-APP) from the UniRender process. After the application program receives the vertical synchronization signal, it executes the measure() method, the layout() method, and the draw() method in the UI thread. Among them, when the UI thread executes the draw() method, it traverses the views of the application, determines the drawing instructions required for rendering each view, and continuously records/records them in the drawing instruction list of the rendering node corresponding to the view.

Among them, the interface that the application needs to display is composed of multiple view nests, and the drawing instruction list corresponding to the root view (DecorView) includes the entry of the drawing instruction list of the subview of the root view, that is, the nesting between the drawing instruction lists The relationship is the same as the nesting relationship of view, so the nesting relationship of rendering nodes is the same as that of view. Among them, the definition of related concepts of rendering tree and rendering node can refer to the text description corresponding to Figure 3 above.

FIG. 9 is an exemplary schematic diagram of constructing a rendering tree provided by an embodiment of the present application.

After the application UI thread performs measure(), layout(), and draw(), it can obtain the parent-child structure of multiple views of the interface to be updated, and determine the view to be displayed on each view during the process of traversing the views. The content and the interface calls required to generate the content, such as drawCircle, drawLine, etc.

The application encapsulates drawing interface calls such as drawCircle and drawLine into corresponding DrawOps such as DrawCircleOp and DrawLineOp, and saves them in the drawing instruction list.

As shown in FIG. 9 , the view structure of the interface of the application program is: a root view, the subviews of the root view are View 1 and View 2 , and the subview of View 2 is View 3 . Wherein, the view 2 includes a rectangle and a circle, and the drawing operations in the drawing instruction list in the rendering node 2 corresponding to the view 2 include: an operation of drawing a rectangle, an operation of drawing a circle, and the like.

When executing the draw() method, the UI thread of the application can start from the root view, traverse all views according to the parent-child relationship between Views, determine the drawing operation in each view, and encapsulate the drawing operation as DrawOp. After the UI thread of the application program generates the drawing instruction list, it will further encapsulate the drawing instruction list into a rendering tree.

The rendering node in the rendering tree includes a drawing instruction list and rendering attributes, among which, the rendering attribute is used to determine the position, size, transparency and other attributes of the view to be rendered by the rendering node in the surface, and the drawing instruction list is used to determine the rendering node to be rendered. The content of the rendered view, such as lines, rectangles, circles, etc.

The surface is applied for by the application, and the application determines the size of the surface by itself. In the case that SurfaceFlinger exists, the surface can be applied to SurfaceFlinger by the application, and in the absence of SurfaceFlinger, it can be applied to the UniRender process. Among them, SurfaceFlinger may not allocate a surface to the application.

Optionally, in some embodiments of the present application, the UniRender process may adjust the frequency of the vertical synchronization signal (Vsync-APP) on each display area after determining the screen refresh rate of each display area, so that the display area 1 displays The application generates the render tree at the screen refresh rate of Display Zone 1.

In this embodiment of the present application, the display area may be a screen, or may be a virtual screen (Virtual Display) or the like. The virtual screen may be an area used by the electronic device to carry content displayed on the screen when recording the screen.

It is worth noting that the UI thread of an application can generate multiple rendering trees, such as in multi-display scenarios such as multi-screen, virtual screen, and multi-window scenarios.

S802: passing the rendering tree across processes

After the UI thread of the application generates the rendering tree, it will pass the rendering tree to the UniRender process through IPC communication. The corresponding UniRender process needs to receive the rendering tree delivered by different applications, and determine the corresponding relationship between the rendering tree and the application.

Among them, multiple applications in the foreground pass the rendering tree to the UniRender process. If the application meets any one of the following three conditions, the application is a foreground application, the application has a visible activity (Activity), the application has a foreground service, and other foreground applications are associated with the application.

Since the memory of different processes is not shared, data interaction between processes needs to be completed through InterProcess Communication (IPC). Among them, the application program can pass the rendering tree to the UniRender process to realize IPC communication through methods such as Binder, AIDL, shared memory, Socket, etc., which is not limited here.

The following uses shared memory as an example of IPC communication to exemplarily introduce the way to transfer the rendering tree across processes.

(a) The application writes the render tree to shared memory

FIG. 10 is an exemplary schematic diagram of a shared memory transfer rendering tree provided by an embodiment of the present application.

As shown in Figure 10, first, the application will apply for shared memory from the UniRender process each time it starts. After the UniRender process receives the shared memory request initiated by the application program, it applies for a piece of shared memory and a handle corresponding to the shared memory through the anonymous shared memory (Anonymous Shared Memory, Ashmem) subsystem.

Secondly, after the UniRender process successfully applies for shared memory from the Ashmem subsystem, it will receive a handle returned by the Ashmem subsystem for reading and writing physical memory. The UniRender process returns the handle to the application, so that the application can use the handle to write the rendering tree into the physical memory. The UniRender process can directly read the physical memory from its own process space, and then directly read the rendering tree of the application.

Wherein, the shared memory may be a virtual file established in the internal memory (RAM) through a temporary file system (tmpfs), which are respectively mapped to user spaces of different processes.

Among them, the application program applies for shared memory to the UniRender process, the UniRender process applies for the shared memory to the Ashmem subsystem, and the UniRender process returns the handle corresponding to the shared memory obtained by the application to the application program. Cross-process communication can be realized through Binder communication.

It can be understood that the rendering tree stored on the stack of the application process can be delivered to the UniRender process through other IPC methods such as shared memory, which is not limited here.

Optionally, in some embodiments of the present application, when the application has multiple layers, that is, when the application generates multiple rendering trees, the application can apply to the UniRender process for two pieces of shared memory, which are used for Store different layers, that is, store data of different rendering trees.

(b) The storage data structure of the rendering tree in shared memory

In order to further improve the efficiency of IPC communication, in the embodiment of the present application, the rendering tree is stored in the shared memory in the form of a memory tree. The following example introduces the data structure format of the rendering tree stored in the shared memory:

Among them, the memory tree can be composed of multiple segments of data, and the data of different segments store layer information, rendering data, etc. respectively. The structure of the memory tree is exemplarily introduced below by taking the contents of FIG. 11A and FIG. 11B as examples. Wherein, the offsets of different segments of data in the shared memory may be fixed, or not fixed, and adaptive.

FIG. 11A and FIG. 11B are exemplary schematic diagrams of a storage structure of a rendering tree in a shared memory provided by an embodiment of the present application.

As shown in FIG. 11A , the storage structure of the rendering tree includes three pieces of data, namely the HEAD field, the MAPPING field, and the NODES field.

Among them, the HEAD field includes layerkey and rootid; the MAPPING field includes nodeid and the address corresponding to the nodeid; the NODES field includes currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist.

Among them, layerkey is the ID of the rendering tree as a whole as a layer; rootid is the ID of the root node of the rendering tree; nodeid is the ID of the rendering node of the rendering tree except the root node, and a nodeid corresponds to an address, the address is The starting address of the rendering properties (renderproperties/properties) and the drawing instruction list (displaylist) in the rendering tree node; stagingproperties is the rendering properties written by the application, stagingdisplaylist is the drawing instruction list written by the application, and currentproperties is the UniRender process The read rendering attributes and currentdisplaylist are the list of drawing instructions read by the UniRender process.

It is worth noting that if "stagingproperties and stagingdisplaylist" are regarded as the first set of data, and "currentproperties and currentdisplaylist" are the second set of data, then the data written in the application program is the first set of data, and the next time the application program writes data is The second set of data implements a double-buffering mechanism; similarly, the data read by the UniRender process is the first set of data, and the data read next time is the second set of data.

Optionally, in some embodiments of the present application, the storage structure of the rendering tree in the shared memory may be as shown in FIG. 11B , where the NODES field may only include a set of data, that is, only include rendering attributes and a drawing instruction list.

The layerkey is used for: before the UniRender process reads the layer data in the shared memory through the handle, it can obtain the application that needs to be displayed and the layer ID of the application that needs to participate in layer composition from the WMS. The UniRender process checks the layer ID with the layer ID included in the layerkey from the shared memory.

rootid is used: rootid is used as the entry of the rendering tree, which stores the entry of other rendering nodes. After obtaining the rootid, the UniRender process can read the data of the rendering tree and restore the nested structure of the rendering tree.

currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist are used for: After the application finishes writing the drawing instruction list and rendering properties, exchange the values of currentproperties and stagingproperties, exchange the values of currentdisplaylist and stagingdisplaylist, and the UniRender process reads the rendering properties and values from currentproperties and current displaylist Draw command list.

For example, application 1 is an application displayed in the foreground, and application 1 can have multiple layers, that is, application 1 will generate multiple rendering trees after receiving the vertical synchronization signal (Vsync-APP), such as rendering tree 1 and render tree 2. The UniRender process determines from the WMS that the rendering tree corresponding to the layer participating in layer composition is rendering tree 1.

Since the offset of layerkey and rootid is fixed, the UniRender process can determine the address of the root node. The UniRender process finds the position of the root node at the NODES side in the MAPPING segment, and reads the rendering command for rendering the node. If there is a child node, there will be a corresponding DrawRenderNode command, which stores the id of the child node, and finds the child node at the node through the hash value. The location of the MAPPING segment. For example, the parent-child relationship of the render nodes of the render tree will be saved in the DrawOP operation. For example, the drawing instruction list of the rendering node 2 includes several DrawOP operations and the operation of "Draw RenderNode3" (drawing the rendering node 3), and then the UniRender process can determine that the rendering node 3 is a child node of the rendering node 2.

It is understandable that the rendering tree in the shared memory still has the same nesting relationship as the view of the application, so the UniRender process can read data from the root node, and then read all the data of the rendering tree.

Among them, by dividing currentproperties, stagingproperties, stagingdisplaylist, and currentdisplaylist, the security of the display data read by the UniRender process and the data written by the application is guaranteed, and half of the data written by the application is prevented from being read and rendered by the UniRender process as the latest data. Generate the application's interface. Among them, for the security guarantee of simultaneous reading and writing of the rendering tree, please refer to the textual description of the reading and writing of the rendering tree in the shared memory in (2.3) below, which will not be repeated here.

Optionally, in some embodiments of the present application, the sizes of the three pieces of data may be fixed. That is, when the UniRender process applies for shared memory from the Ashmem subsystem, the obtained shared memory is a memory with a size of (a+b+c). Among them, from the start address (physical address) to the start address + a is the position to fill the HEAD field; from the start address + a + 1 to the start address + a + b is the position to fill the MAPPING field; from the start address +a+b+1 to start address +a+b+c are the positions to fill the NODES field.

It is understandable that when the size of the three pieces of data is fixed, the UniRender process can determine the start of each piece of data according to the fixed offset, and then find the Mapping section, in which the rendering nodes of the rendering tree are stored in The offset in the NODES field, and in turn the data for each render node can be found.

Optionally, in some embodiments of the present application, when the size of the three pieces of data is fixed, and the rendering node written by the application exceeds the size of b, a second piece of shared memory is applied to the UniRender process. Wherein, the format of the second piece of shared memory may be the same as that of the first piece of shared memory, and the NODES field of the second piece of shared memory continues to store the rendering instruction list and the attributes of the rendering node. Wherein, the HEAD field and/or the MAPPING field of the second shared memory may be empty or may not exist. That is, in some embodiments of the present application, only the NODES field is included in the second shared memory.

(c) UniRender reads the rendering tree in shared memory

(c.1) UniRender architecture

FIG. 12 is an exemplary schematic diagram of the architecture of UniRender provided by the embodiment of the present application.

As shown in Figure 12, the UniRender provided by this application may include four parts, namely NodeManager, LayerManager, DisplayManager, and UniRenderCore.

Among them, NodeManager is the node management module in the UniRender process, which is responsible for receiving the rendering tree sent by the application program. Wherein, the composition of the target rendering tree can refer to the text description in step S903.

Among them, LayerManager is the layer management module in the UniRender process, which is responsible for synchronizing layer information from the Window Manager Service (WMS), such as layer creation, destruction, property change, etc. Among them, a bitmap is equivalent to a layer.

Among them, DisplayerManager is a display device management module in the UniRender process, responsible for synchronizing display device information from the Display Manager Service (DMS), such as the size of the screen.

Among them, UniRenderCore is the rendering management module in the UniRender process, which is responsible for: establishing a corresponding rendering node for each layer; receiving the rendering tree corresponding to different applications maintained in the NodeManager, and removing the layer information instruction of the application from the LayerManager and insert the rendering node; for the active display device maintained in the DisplayManager, merge the rendering trees corresponding to all the visible layers; traverse the merged rendering tree for each display area, and UniRender assigns it to the buffer Generate bitmaps etc.

(c.2) UniRender reads the rendering tree

The UniRender process first determines the applications displayed on each display area from the DMS and WMS, and these applications are the applications that participate in layer composition. Among them, the UniRender process can further determine the application program that performs layer composition in the UniRender process in combination with the white list. Among them, the UniRender process can determine the layer ID of each application through WMS.

Among them, the DisplayerManager in the UniRender process is responsible for communicating with the DMS, and the LayerManager in the UniRender process is responsible for communicating with the WMS.

Because the UniRender process saves the handle of the shared memory corresponding to the application, after the UniRender process determines the application participating in layer composition, it can determine the shared memory corresponding to the application through the handle. The UniRender process reads the rendering tree from the shared memory through this handle.

Among them, the NodeManager in the UniRender process is responsible for managing the handle of the shared memory and reading the rendering tree from the shared memory.

The process of the UniRender process reading the rendering tree from the shared memory includes:

First, the UniRender process reads the layer key from the starting address of the shared memory, and verifies the layer ID. Among them, the UniRender process compares the layer ID determined from the layer key with the layer ID determined by the WMS. When the verification is consistent, the UniRender process starts to read the rendering tree from the root node root id.

Secondly, after the UniRender process finds the address of the root node, it determines the starting address of the root node in the NODES segment in the address field in the MAPPING segment, and starts to read the rendering command list and rendering attributes of the rendering node. Wherein, if the root node has a child node, the drawing list instruction of the root node stores the entry of the child node, such as a DrawRenderNode instruction. Since the DrawRenderNode instruction includes the ID of the child node, the UniRender process finds the corresponding node id in the MAPPING section through hash operation, and then can determine the drawing command list of the child node and the position of the rendering attribute in the NODES section, and read the child node A list of drawing instructions and rendering properties.

(d) Reading and writing to the rendering tree in shared memory

The rendering tree located in the shared memory can be read and written by two or more processes. In order to reduce and avoid read-write conflicts and cause data errors in the rendering tree, you can configure synchronization locks between processes to ensure the read-write security of the rendering tree .

The process of reading and writing the rendering tree is exemplarily introduced below in conjunction with the contents shown in FIG. 13 and FIG. 14 .

FIG. 13 is an exemplary schematic diagram of an application program writing a rendering tree into a shared memory provided by an embodiment of the present application.

Each application saves at least one lock variable A to prevent the application and the UniRender process from reading and writing the shared memory at the same time. Among them, the UniRender process obtains the status (holding or releasing) of the locked variables of different applications through IPC communication.

As shown in Figure 13, ① First, when the application needs to update the interface, it requests and receives the vertical synchronization signal (Vsync-APP), and then obtains the lock variable A of the shared memory corresponding to the application, and starts rendering from the root node. The node's attributes and drawing instruction list are computed and updated.

②Secondly, the application program writes the updated rendering node properties and drawing command list into the staging properties and staging displaylist data segments of the NODES segment in the shared memory, and the changed rendering node id is added to the properties_dirty queue and displaylist_dirty queue , the queue is saved in the shared memory management class singleton on the application side.

It is understandable that the changed rendering nodes are marked in the properties_dirty queue and the displaylist_dirty queue, which can achieve differential update of the rendering tree.

Optionally, in some embodiments of the present application, the properties_dirty queue and the displaylist_dirty queue may not be saved, so as to realize full rendering tree update.

③ Again, the application copies the stagingproperties segment corresponding to the rendering node in the properties_dirty queue to the current properties segment. The application program exchanges the draw_pointer and record_pointer corresponding to the rendering node in the displaylist_dirty queue, that is, copies the staging displaylist section corresponding to the rendering node in the displaylist_dirty queue to the current displaylist; or the application program copies the staging displaylist section to the current displaylist.

It is understandable that, compared to the previous vertical synchronization signal (Vsync-APP), in response to this vertical synchronization signal (Vsync-APP), the application only changes the data of the rendering node corresponding to displaylist_dirty, and the application queues displaylist_dirty If the draw_pointer and record_pointer corresponding to the rendering node are exchanged, the current displaylist can be updated by difference.

It is understandable that the application program copies the staging displaylist segment to the currentdisplaylist to achieve a full update, that is, directly writes all the data of the rendering tree generated by the application program in response to the vertical synchronization signal (Vsync-APP) to the shared memory to realize Simpler.

Optionally, in some embodiments of the present application, without saving the properties_dirty queue and the displaylist_dirty queue, the staging properties section of all rendering nodes of the application is copied to the current properties section, and the staging displaylist section is copied to the current displaylist section.

④Through the IPC communication, the application transmits the information that the lock variable A has been released to the UniRender process.

⑤ Again, the UniRender process holds the lock variable A.

⑥Finally, corresponding to ③, the UniRender process reads the current displaylist and current properties from the shared memory, or reads the stagingdisplaylist segment corresponding to the rendering node in the displaylist_dirty queue and copies it to the current displaylist.

After the UniRender process reads the data, it can release the lock variable A and notify the application that the lock variable A has been released. Then, when the next vertical synchronization signal (Vsync-APP) arrives, the lock variable A is held, and then the rendering tree is written into the shared memory. At this point, the roles of the staging data segment and the current data segment are switched, and the UniRender process finally reads the staging displaylist and staging properties segments to implement a "double buffering" mechanism to ensure the robustness of interface generation.

Optionally, in some embodiments of the present application, if there is a lock variable between the application program and the UniRender process, the NODES field may only include staging displaylist and staging properties, or only include current displaylist and current properties. Among them, the application program and the UniRender process realize read and write security through the lock variable, so that the UniRender process can read the correct rendering tree.

Optionally, in some embodiments of the present application, each application program may hold more lock variables, for example, each application program holds lock variable A and lock variable B. Furthermore, after the application program releases the lock A, it does not need to wait for the UniRender process to release the lock A. After holding the lock variable B, after receiving the next vertical synchronization signal (Vsync-APP), it directly writes the rendering tree data into the shared memory .

Among them, the holding, release, and inter-process synchronization of the lock variable B are the same as the holding, release, and inter-process synchronization of the lock variable A. For details, please refer to the text description shown in FIG. 14 , and will not repeat them here.

FIG. 14 is another exemplary schematic diagram of the application program writing the rendering tree into the shared memory provided by the embodiment of the present application.

As shown in Figure 14, when the application saves a lock variable A, after receiving the vertical synchronization signal 1 (Vsync-APP), the application holds the lock variable A, and in the holding phase, the generated rendering tree written to shared memory. After the application writes the render tree to shared memory, release the lock variable A. After the UniRender process determines that the lock variable A is released by the application program, it holds the lock variable A and starts to read the rendering tree generated by the application program in response to the vertical synchronization signal 1 from the shared memory.

During the time when the UniRender process holds the lock variable A, the application program receives the vertical synchronization signal 2 (Vsync-APP) after the vertical synchronization signal 1. Since the lock variable A is held by UniRender, the application program cannot write the rendering tree in time. Into the shared memory, it needs to wait for a while until the application determines that the UniRender process releases lock A.

However, it is obvious that in Figure 13, the UniRender process reads the current displaylist field and the current properties field, while the application program writes the staging displaylist field and the stagingproperties field. You can realize different fields in the shared memory by increasing the number of lock variables concurrent read and write.

As shown in Figure 14, when the application saves two lock variables B, after receiving the vertical synchronization signal 1 (Vsync-APP), the application holds the lock variable A, and the generated rendering The tree is written to shared memory. After the application writes the render tree to shared memory, release the lock variable A. After the UniRender process determines that the lock variable A is released by the application program, it holds the lock variable and starts to read the rendering tree generated by the application program in response to the vertical synchronization signal 1 from the shared memory.

During the time when the UniRender process holds the lock variable A, the application program receives the vertical synchronization signal 2 (Vsync-APP) after the vertical synchronization signal 1, and holds the lock variable B at this time, and the application program can write the rendering tree in time In shared memory, there is no need to wait for a period of time until the application determines that the UniRender process releases lock A.

Correspondingly, after the UniRender process determines that the lock variable B is released by the application program, it holds the lock variable B and starts to read the rendering tree generated by the application program in response to the vertical synchronization signal 2 from the shared memory.

It should be noted that the number of lock variables held by the application program may be related to the content included in the NODES field in the shared memory, or may be related to the value of Vsync-offset.

For example, consider the current displaylist and current properties as the first group, and the stagingdisplaylist and staging properties as the second group, then you can configure two synchronization lock variables in the application and UniRender processes. Similarly, for another example, if the NODES field includes three sets of data, then three synchronization lock variables can be configured in the application program and the UniRender process.

Among them, a lock variable corresponds to a set of data. For example, lock variable A corresponds to current displaylist and current properties, and the change of lock variable A from the holding state to the release state indicates that the application successfully updates the current displaylist and current properties in the shared memory. The data of the current displaylist and current properties can be read by the UniRender process; or, the change of the lock variable A from the holding state to the release state indicates that the UniRender process and the reading of the data of the current displaylist and current properties are completed from the shared memory , the data of current displaylist and current properties can be updated by the application.

Wherein, the number of lock variables may be related to the value of Vsync-offset.

That is, it may be related to the difference Vsync-offset between the vertical synchronization signal (Vsync-APP) and the vertical synchronization signal (Vsync-UR). If the Vsync-offset is large, the lock variable may not be configured. In the case of not configuring the lock variable, the UniRender process reads the rendering tree from the shared memory after receiving the vertical synchronization signal (Vsync-UR). Due to the large Vsync-offset, when the UniRender process reads the rendering tree, the application has completely written the rendering tree into the shared memory.

S803: Generate a target rendering tree based on the obtained rendering tree.

Among them, the UniRender process can start to read the rendering tree from the shared memory after receiving the vertical synchronization signal (Vsync-UR), and start to synthesize the rendering tree after obtaining one or more rendering trees. Or, the UniRender process can start to read the rendering tree from the shared memory while holding the lock variable, and start compositing the rendering tree after receiving the vertical synchronization signal (Vsync-UR).

When the number of rendering trees received by the UniRender process is one, the rendering tree is the target rendering tree; when the number of rendering trees received by the UniRender process is more than one, multiple rendering trees are synthesized to generate a target render tree.

FIG. 15 is an exemplary schematic diagram of synthesizing a rendering tree into a target rendering tree according to an embodiment of the present application.

As shown in Figure 15, the UniRender process can create a new root rendering node, such as the new root rendering node in Figure 15, and use the processed rendering trees of multiple applications as child nodes of the root rendering node. That is, the root rendering node can store the parent-child relationship of the processed rendering trees of different applications, so that the UniRender process can traverse the target rendering tree from the root rendering node to find the processed rendering trees of different applications.

FIG. 16 is another exemplary schematic diagram of synthesizing a rendering tree into a target rendering tree according to an embodiment of the present application.

The UniRender process can first determine the Z order of the layers corresponding to different applications according to the window management service, that is, determine the upper and lower masking relationships between the layers of the applications. Furthermore, in the process of generating the target rendering tree, the rendering nodes corresponding to the completely occluded views are deleted, thereby reducing the amount of calculation in the process of generating the bitmap in step S905 and increasing the generation speed of the bitmap.

For example, as shown in FIG. 16 , the layer corresponding to the rendering tree generated by application 1 is layer 1 , and the layer corresponding to the rendering tree generated by application 2 is layer 2 . Among them, the Z order of layer 1 is higher than that of layer 2. Among them, the rendering tree corresponding to layer 1 is root rendering node 1, rendering node 1, rendering node 2, rendering node 3, rendering node 1 and rendering node 2 are the child rendering nodes of root rendering node 1, and rendering node 3 is the rendering node 2's child rendering nodes; the rendering tree corresponding to layer 2 is root rendering node 2, rendering node 4, rendering node 5, rendering node 6, rendering node 4 and rendering node 5 are the child rendering nodes of the new root rendering node, and rendering nodes 6 is a child render node of render node 5.

The UniRender process can traverse the rendering sub-nodes of the rendering tree corresponding to layer 1 and the sub-nodes of the rendering tree corresponding to layer 2, determine the position of the view corresponding to each rendering node in the (surface allocated by the UniRender process) surface, and Combining the layer Z order of layer 1 and the Z order of layer 2, determine the fully shaded view and the rendering node of the completely shaded view.

For example, UniRender can determine that the view corresponding to rendering node 6 in the corresponding rendering tree of layer 2 is completely obscured, and then delete rendering node 6.

After the UniRender process deletes the rendering node corresponding to the completely occluded view, it can merge all rendering trees that trigger off-screen rendering instructions into the target rendering tree as shown in Figure 16.

Optionally, in some embodiments of the present application, UniRender can also traverse the rendering nodes, and optimize the parameters of the target rendering tree from the granularity of the DrawOP drawing operation. For example, delete the DrawOP drawing operation that does not affect the interface, which may mean that the graphics drawn by the DrawOP drawing operation are not displayed on the interface; for another example, for different applications, modify the position of the DrawOP operation in the drawing node so that the same type The DrawOP drawing operations can be performed together, for example, modifying the DrawOP drawing operation of the rendering node 2 of the application program 1 into the drawing instruction list of the rendering node 1 of the application program 2, which is not limited here.

Optionally, in some embodiments of the present application, after step S804 and before step S805, after the UniRender process generates the target rendering tree, it decapsulates the RenderNode drawing instruction list to obtain a series of DrawOPs, and then uses the target rendering tree as The DrawOP operation is performed as a whole to perform batch (Batch) and merge (Merge), and in step S805 , a target rendering tree that can generate a bitmap with less computation is obtained.

It can be understood that by merging multiple rendering trees and optimizing the merged rendering trees, transitional rendering can be avoided or reduced, thereby increasing the speed of bitmap generation in step S905, saving power consumption of electronic devices, and improving user experience .

It is worth noting that the UniRender process can also optimize other parameters of the target rendering tree during the process of generating the target rendering tree, which is not limited here.

S804: Generate a bitmap and complete animation rendering

After the UniRender process generates the target rendering tree, it can execute the drawing operation in the drawing instruction list in the rendering tree, and complete the process of generating bitmaps in the surface allocated by the UniRender process for multiple applications. Animated rendering of bitmaps. Wherein, the bitmap bears image information of one or more application programs.

Wherein, the surface may be the same size as the display area (display). That is, the surface can be the same size as the screen of the electronic device. For example, in the scenes shown in Figures 6A and 6B, the size of the surface is consistent with the size of the screen. Wherein, the surface can be located on the screen buffer.

FIG. 17 is an exemplary schematic diagram of animation rendering in the process of generating a bitmap provided by the embodiment of the present application.

As shown in Figure 17, after the UniRender process obtains the rendering tree generated by the UI thread of one or more applications through IPC communication, it synthesizes one or more rendering trees into a target rendering tree, and then calls the GPU to perform target rendering through the underlying graphics library The drawing operation of the drawing instruction list in the tree generates a bitmap, and animation rendering is completed during the process of generating the bitmap.

Next, from two aspects, whether the animation is cross-window/cross-application or not, the method of animation rendering by the UniRender process in the process of generating bitmaps is exemplarily introduced.

(a) Animations within a single application

In order to facilitate the development of the application program developers, the interface generation method provided by the embodiment of the present application can enable the developers to configure different animation parameters for any view attribute of the application program, such as foreground blur, background blur and other animation parameters. Correspondingly, during the process of generating the rendering tree, the UI thread of the application will add animation parameters to the rendering properties of the rendering node corresponding to the view.

Wherein, the UniRender process can identify the animation parameters in the rendering properties of the rendering node in the rendering tree corresponding to the animation parameters of the view, and execute the corresponding animation rendering operation. Alternatively, the UniRender process can add animation parameters to the rendering properties of any rendering node in the target rendering tree.

That is, the animation parameter is a parameter that the application developer can configure on the view, or a parameter that the UniRender process adds to the rendering node, and is used to add animation effects to the selected content on the interface.

The interface generation method provided by the embodiment of the present application also provides developers with a BlurView, and the range covered by the BlurView is the range where the animation takes effect. Among them, BlurView can inherit view, and besides all configurable properties of view, it can also have other properties.

The UniRender process can identify the corresponding rendering node of BlurView in the rendering tree, and execute the corresponding animation rendering operation. The animation rendering operation includes: the UniRender process will copy the drawn content on the surface covered by BlurView to the off-screen buffer for animation rendering, and then copy it back to the surface, or directly copy it to the on-screen buffer.

①The UniRender process can allocate a surface for the bitmap of the application, and the UniRender process continuously draws the view of the application on the surface based on the target rendering tree to generate the bitmap of the application.

②When the UniRender process draws to a rendering node with animation parameters, if the animation parameters are parameters such as scaling, simple color change, and transparency change, the parameters of the drawing operation in the drawing command list in the rendering node will be modified accordingly , and then perform that drawing operation for animation rendering.

For example, if the animation parameter of the render attribute of the render node is magnified by 2 times, and the drawing command list of the render node includes DrawCircleOp((x0, y0), 3), then the UniRender process will drawCircleOp((x0) when performing the drawing operation , y0), 3) after converting to DrawCircleOp((x0, y0), 6), execute DrawCircleOp((x0, y0), 6).

Among them, DrawCircleOp((x0, y0), 3) is: draw a circle with a radius of 3 at the center of (x0, y0); DrawCircleOp((x0, y0), 6) is: at (x0, y0) is Draw a circle with a radius of 6 at the center.

The animation parameters are filter effects, such as blur (Gaussian blur, radial blur, lens blur, etc.), sharpening, distortion, stylization (such as embossing effect, oil painting effect, 3D effect, etc.), lens flare, adding noise, etc. When the color and other parameters are selected, the UniRender process will start off-screen rendering, copy the content at the position of the rendering node to the off-screen buffer, perform animation rendering, and then copy it back to the surface.

③ Continue to draw to generate a bitmap.

The following takes the foreground blur and background blur as the filter effect as an example to introduce the animation rendering process of the UniRender process.

FIG. 18A is an exemplary schematic diagram of the implementation of foreground blur provided by the embodiment of the present application.

As shown in FIG. 18A, when the UniRender process traverses to a rendering node whose rendering attribute includes animation parameters, such as RenderNode3, during the process of generating a bitmap based on the rendering tree, it first determines the type of the animation parameter, for example, it is determined as blurred foreground. Among them, the view corresponding to RenderNode3 is view3.

After the UniRender process determines that the animation parameter is blurred foreground, it executes the drawing operation in the drawing instruction list of RenderNode3 in the off-screen buffer 3, and then draws view3.

The UniRender process performs animation rendering on the content in the off-screen buffer, that is, blur rendering, and the generated content is called view31. Then, the UniRender process copies the contents of the off-screen buffer to the surface, and continues to traverse other nodes of the rendering tree.

Wherein, the size of the off-screen buffer can be consistent with the size of view3, and the UniRender process can determine the size of view3 from the attribute of RenderNode3.

FIG. 18B is an exemplary schematic diagram of the implementation of background blur provided by the embodiment of the present application.

As shown in FIG. 18B , in the process of generating bitmaps based on the rendering tree, when the UniRender process traverses to a rendering node whose rendering attributes include animation parameters, such as RenderNode3, it first determines the type of animation parameters, for example, background blur. Among them, the view corresponding to RenderNode3 is view3.

After the UniRender process determines that the animation parameter is background blur, it copies the content at the position of view3 on the surface to the off-screen buffer, and performs animation rendering on the content in the off-screen buffer, that is, performs blur rendering.

Then, the UniRender process copies the content in the off-screen buffer to the surface, executes the drawing command list of RenderNode3, and continues to draw view3 on the surface.

The UniRender process continues to traverse other nodes of the render tree.

Among them, before drawing view3, the content copied by the UniRender process to the position of view3 on the surface in the off-screen buffer is the background of view3.

Optionally, in some embodiments of the present application, when the reference coordinate system according to which the rendering tree generated by the application program is inconsistent with the reference coordinate system of the surface allocated by the UniRender process, after the UniRender process receives the rendering tree of the application program , will also add coordinate transformation instructions in the rendering attributes of the rendering tree. For example, the UniRender process will convert DrawCircleOp((x0, y0), 3) to DrawCircleOp((x1, y1), 3).

(b) Cross-app/cross-window animation

The implementation of the cross-application/cross-window animation is the same as that of the single-application animation, which can be referred to the text description above, and will not be repeated here.

First, the UniRender process allocates a surface for multiple applications, and completes the animation rendering of the bitmap during the process of generating bitmaps of multiple applications in the surface. Wherein, the surface can be located in the screen buffer.

Second: The UniRender process can insert a surface anywhere on the surface, and/or insert a BlurView between any rendering nodes in the target rendering tree; the UniRender process can also recognize the BlurView of the application. Wherein, the range covered by the BlurView is the content for animation rendering.

FIG. 19 is an exemplary schematic diagram of cross-application/cross-window animation rendering provided by the embodiment of the present application.

As shown in Figure 19, cross-application/cross-window animation rendering may include the following three steps:

① First, the UniRender process generates a bitmap in the surface after traversing the drawing instruction list of the execution target rendering tree, and the bitmap carries the image information of one or more applications.

②Secondly, the UniRender process adds BlurView to the surface, and copies the content at the position of BlurView to the off-screen buffer.

③ Finally, the UniRender process animates and renders the content of the off-screen buffer, and copies the content in the off-screen buffer back to the surface.

Among them, the superposition of the content in the off-screen buffer and the original content on the surface can be considered as a kind of layer synthesis, that is, the content in the off-screen buffer can cover the content on the surface, or the content in the off-screen buffer can be on the surface The content is superimposed in various ways.

FIG. 20 is another exemplary schematic diagram of cross-application/cross-window animation rendering provided by the embodiment of the present application.

As shown in Figure 20, cross-application/cross-window animation rendering may include the following three steps:

① First, the UniRender process can add a rendering node corresponding to BlurView at any level of the rendering tree during the process of traversing the drawing instruction list of the execution target rendering tree (as shown in Figure 20). Alternatively, the UniRender process may identify the rendering node corresponding to the BlurView of the application program during the process of traversing the drawing instruction list of the execution target rendering tree.

When BlurView is not the root node of the target rendering tree, the UniRender process will first perform drawing operations on the surface to generate a bitmap.

②Secondly, the UniRender process draws the child nodes of BlurView in the off-screen buffer. After the UniRender process draws all the sub-nodes of BlurView, the content in the off-screen buffer is animated and rendered, and copied back to the surface after the animation is rendered.

③ Finally, the UniRender process continues to traverse the target rendering tree to complete the drawing of the bitmap.

It is worth noting that the position and size of the BlurView specified by the application can be generated based on the coordinate system of the surface of the UniRender process, or based on other coordinate systems, which are not limited here.

Optionally, in some embodiments of the present application, BlurView may also be configured with a sigma attribute, which represents different meanings in different animation renderings. For example, when animation rendering is blurred rendering, sigma represents the degree of blurring, and the value range is an integer. When sigma=0, BlurView does not take effect, that is, no blurring is performed. Wherein, the definition of blur degree can refer to the contents in (c) blur animation rendering, which will not be repeated here.

(c) Fuzzy animation rendering

In the embodiment of the present application, only the content that needs to be rendered by animation has been copied to the off-screen buffer, or directly drawn in the off-screen buffer, and animation rendering can be performed on the content in the off-screen buffer.

The following takes animation as blur as an example to introduce the process of blur rendering.

The embodiment of the present application provides blurred rendering with various degrees of blurring, wherein the lower the degree of blurring is, the closer the value of the pixel after blurring is to the value of the pixel before blurring.

The embodiments of the present application provide multiple precision blur renderings, which are respectively referred to as high-precision blur rendering, medium-precision blur rendering, and low-precision blur rendering for convenience of description. The embodiments of the present application also provide blur rendering with different blur radii under blur rendering with different precisions.

Among them, the blur degree, blur precision, and blur radius can be configured by the application program or the UniRender process. Among them, the application can configure the blur degree, blur precision, and blur radius by configuring BlurView properties, and the BlurView properties will be encapsulated into the rendering properties of the rendering node corresponding to the BlurView, and then obtained by the UniRender process. When the application program does not configure animation parameters such as blur degree, blur precision, and blur radius of BlurView, the UniRender process can adaptively configure animation parameters such as blur degree, blur precision, and blur radius of BlurView.

The embodiment of the present application provides that the UniRender process can select animation rendering with different precisions according to the current GPU load and GPU capability. For example, when the GPU load of the electronic device is high, animation rendering with low precision is selected; when the GPU load of the electronic device is low, animation rendering with high precision is selected.

FIG. 21 is an exemplary schematic diagram of pixel processing by blur rendering provided by the embodiment of the present application.

Blur is an animation that resets the value of each pixel in the image, where the value of each pixel after reset is related to the value of the surrounding pixels before the pixel is reset.

For example, as shown in Figure 21, the bitmap before blurring is bitmapA, where the resolution of bitmapA is 8*8, that is, each row has 8 pixels, and each column has 8 pixels, that is, there are 64 pixels in bitmapA. are A ₁₁ , A ₁₂ to A ₈₈ , respectively.

The electronic device generates bitmapB after blurring bitmapA. Wherein, bitmapB may have the same resolution as bitmapA, or may have the same resolution as bitmapA. If the resolutions of generated bitmapB and bitmapA are different, downsampling and interpolation can be used to make the resolution of bitmapB the same as that of bitmapA.

After the electronic device generates bitmapB, the value of any pixel in bitmapB is related to multiple pixel values in bitmapA.

For example, for pixel B _ij in bitmapB, its generation process is as follows (1):

In formula (1), k(r(i,j,k,l)) is the coefficient related to the distance r(i,j,k,l), where r(i,j,k,l) means the distance with ( The distance between i, j) and (k, l), A _kl is the pixel in row k and column l, and B _ij is the pixel in row i and column j.

Among them, k(r(i,j,k,l)) can be related to the configured blur radius, for example, when r(i,j,k,l)>blur radius, k(r(i,j,k ,l))=0. That is, if the distance between A _kl and B _ij is greater than the blur radius, the value of the pixel of B _ij will not be affected by the value of the pixel of A _kl .

For example, if the blur radius is 2, then B ₄₅ =avg(A45+0.5*(A35+A44+A46+A55)+0.3*(A34+A36+A54+A56)), where avg() is the average.

Among them, k(0) is related to the degree of blur. For example, when k(0)=1, B ₄₅ =avg(A45+0.5*(A35+A44+A46+A55)+0.3*(A34+A36+A54+A56)), k(0)=2 In the case of B ₄₅ =avg(2*A45+0.5*(A35+A44+A46+A55)+0.3*(A34+A36+A54+A56)). Then, the smaller the value of k(0), the larger the degree of blur.

When the relationship between the value of k(r(i,j,k,l)) and the distance is a Gaussian function, Gaussian blur can be realized.

Considering that the pixels close to the edge of the blurred bitmapA, such as A ₁₈ , need the content outside the bitmapA during blur rendering, so when copying the blurred content from the screen buffer to the off-screen buffer, the copied content can be enlarged , a better blur has been implemented.

FIG. 22A and FIG. 22B are exemplary schematic diagrams of the blur animation provided by the embodiment of the present application.

As shown in FIG. 22A , after determining BlurView, the electronic device determines the size of BlurView1 according to the blur radius and BlurView, and then copies the content within the defined range of BlurView1 to the off-screen buffer. After blurring the content in the BlurView in the off-screen buffer, copy the content within the range defined by the blurred BlurView to the on-screen buffer.

As shown in Figure 22B, after the electronic device obtains the content that requires blur rendering and copies the rendered content to the off-screen buffer, it first down-samples the blurred content, and performs blur rendering on the down-sampled bitmap ; Then after getting the blurred rendered bitmap, perform interpolation to restore the resolution of the bitmap, and then copy it back to the screen buffer.

For high-precision rendering, one or more downsampling can be performed in the downsampling stage (when the downsampling ratio is fixed), and the calculation is performed pixel by pixel during blur rendering. Finally, the bitmap copied back to the screen buffer is obtained by interpolation.

For medium-precision rendering, one downsampling can be performed in the downsampling stage (when the downsampling ratio is fixed), and when blurring any pixel during blur rendering, only calculate the pixels in the same row and column as the pixel. Influence. For example, for B ₄₅ , only the effects of A ₄₄ , A ₄₅ , A ₄₆ , A ₃₅ , and A ₅₅ are considered. Finally, the bitmap copied back to the screen buffer is obtained by interpolation.

For low-precision rendering, downsampling may not be performed, and when blurring any pixel during blur rendering, only the influence of pixels in the same row and column as the pixel is calculated. Finally, the bitmap copied back to the screen buffer is obtained by interpolation.

It is worth noting that other parameters such as the number of times of downsampling, whether to perform downsampling, the blur radius during blur rendering, the number of pixels to be considered during blur rendering, and other parameters can also be combined arbitrarily to achieve rendering with different precision, which is not limited here. .

(3) Again, taking FIG. 23A and FIG. 23B as examples, the process of generating the interface shown in FIG. 4B by the electronic device through animation rendering is introduced.

FIG. 23A and FIG. 23B are schematic diagrams of an example of the implementation of cross-application/cross-window interface animation effects provided by the embodiment of the present application.

① After the UniRender process generates the target rendering tree, it can traverse the drawing instruction list in the target rendering tree, and draw the generated bitmap 1 on the surface.

Wherein, the object rendering tree is shown in FIG. 23B , the root node of the object rendering is node 1, and the child nodes of node 1 are node 2, node 3, node 4, and node 5 respectively. Among them, the view corresponding to node 2 is the interface of the wallpaper application program; node 3 is used to determine the layout of the view corresponding to the child nodes of node 3; node 4 is the rendering node corresponding to BlurView; node 5 is used to determine the layout of the view corresponding to the child nodes of node 3; The layout of the view corresponding to the child nodes of the number node. The child nodes of node 3 are nodes 6 and 7, and the content corresponding to node 6 is the icon and text of the "Gallery" on the desktop application; the content corresponding to node 7 is "dial, information" on the desktop application , Contact” icon and text; the content corresponding to node 8 is icon control 4A01; the content corresponding to node 9 is control bar 4A02.

Among them, the order of the UniRender process traversing the target rendering tree is: node 1, node 2, ..., node 9.

Wherein, the bitmap 1 may only include the content that needs to be rendered by animation in the interface of two or more application programs, or the complete interface of two or more application programs.

In FIG. 23A, bitmap 1 is the content that needs to be rendered by animation in the interface of the two application programs.

When the UniRender process traverses to the No. 4 node, after determining that the No. 4 node is the rendering node corresponding to BlurView, the UniRender process determines the content to be rendered by animation according to the size and range of the BlurView.

In Figure 23A, BlurView is sized and bounded to cover the entire surface. That is, the content rendered by animation is bitmap 1.

②The UniRender process creates an off-screen buffer, and copies the content covered by BlurView to the off-screen buffer; or directly draws the content drawn after BlurView in the off-screen buffer.

In FIG. 23A, the UniRender process copies bitmap 1 to the off-screen buffer, which is referred to as bitmap 2 for short.

③The UniRender process performs animation rendering on the content in the off-screen buffer. Wherein, animation rendering includes fuzzy rendering.

In FIG. 23A , the UniRender process performs blur rendering on bitmap 2 to obtain bitmap 3 .

④The UniRender process copies the content in the off-screen buffer back to the surface, and continues to traverse the target rendering tree. After traversing the target rendering tree, the final bitmap 4 for display is obtained.

In Figure 23A, the UniRender process copies bitmap 3 back to the surface, and continues to draw, completes rendering, and generates bitmap 4. Among them, the bitmap 4 will be passed to the display subsystem by the UniRender process for display.

It can be understood that the interface generation method provided by the implementation of this application can generate bitmaps carrying image information of multiple applications through one continuous rendering (one GPU call), avoiding additional calls to the GPU during the layer composition process.

It can be understood that, the interface generation method provided by the embodiment of the present application can reduce the size of the area for repeated drawing by combining multiple rendering trees of the application program into a target rendering tree.

It can be understood that the animation parameters of the view and the BlurView provided in the embodiment of the present application can implement animation rendering of any control and any area, reducing the workload of application developers.

It is worth noting that the rendering tree of the application received by the UniRender process is not limited to the data structure coupled by RenderNode in a tree structure, but can also be other data structures including the drawing instruction list and rendering attributes, which are not limited here .

(4) Finally, the hardware architecture and software architecture of the electronic device provided by the embodiment of the present application are introduced.

FIG. 24 is an exemplary schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.

Hereinafter, an electronic device is taken as an example to describe the embodiment in detail. It should be understood that an electronic device may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.

The electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194 and user An identification module (subscriber identification module, SIM) card interface 195 and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.

It can be understood that, the structure shown in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the illustrations, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. . Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Wherein, the controller may be the nerve center and command center of the electronic equipment. The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuitsound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver (universal asynchronous receiver) /transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or A universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flashlight, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the electronic device.

The I2S interface can be used for audio communication. The PCM interface can also be used for audio communication, sampling, quantizing and encoding the analog signal. The UART interface is a universal serial data bus used for asynchronous communication.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . The MIPI interface includes a camera serial interface (camera serial interface, CSI), a display serial interface (displayserial interface, DSI), and the like. In some embodiments, the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device. The processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device. The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193 , the display screen 194 , the wireless communication module 160 , the audio module 170 , the sensor module 180 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc. The SIM interface can be used to communicate with the SIM card interface 195 to realize the function of transmitting data to the SIM card or reading data in the SIM card. The USB interface 130 is an interface conforming to the USB standard specification, specifically, it may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The charging management module 140 is configured to receive a charging input from a charger. The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 .

The wireless communication function of the electronic device can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor. Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied to electronic devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves and radiate them through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite system ( Global navigation satellite system (GNSS), frequency modulation (frequency modulation, FM), near field communication (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (globalnavigation satellite system, GLONASS), a Beidou satellite navigation system (beidou navigationsatellite system, BDS), a quasi-zenith satellite system (quasi-zenith) satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device realizes the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light emitting diode). , AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

The electronic device can realize the shooting function through ISP, camera 193 , video codec, GPU, display screen 194 and application processor, and then obtain real-time video data.

The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also optimize the algorithm for image noise and brightness. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 193 .

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a complementary metal-oxide-semiconductor (complementary metal-oxide-semiconductor, CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

Video codecs are used to compress or decompress digital video. An electronic device may support one or more video codecs. In this way, the electronic device can play or record videos in various encoding formats, such as: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of electronic devices can be realized through NPU, such as: image recognition, face recognition, voice recognition, text understanding, etc.

The internal memory 121 may include one or more random access memories (random access memory, RAM) and one or more non-volatile memories (non-volatile memory, NVM).

Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous dynamic random access memory, SDRAM), double data rate synchronous Dynamic random access memory (double data rate synchronous dynamic random access memory, DDR SDRAM, such as the fifth generation DDR SDRAM is generally called DDR5 SDRAM), etc.;

Non-volatile memory may include magnetic disk storage devices, flash memory (flash memory).

In the embodiment of the present application, the non-real-time video may be located in a non-volatile memory.

According to the operating principle, flash memory can include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. According to the potential order of storage cells, it can include single-level storage cells (single-level cell, SLC), multi-level storage cells (multi-level cell, MLC), triple-level cell (TLC), quad-level cell (QLC), etc., can include universal flash storage (English: universal flash storage, UFS), An embedded multimedia memory card (embedded multi media Card, eMMC), etc.

The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of an operating system or other running programs, and can also be used to store data of users and application programs.

The non-volatile memory can also store executable programs and data of users and application programs, etc., and can be loaded into the random access memory in advance for the processor 110 to directly read and write.

The external memory interface 120 can be used to connect an external non-volatile memory, so as to expand the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and video are stored in an external non-volatile memory.

The electronic device can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. Speaker 170A, also referred to as a "horn", is used to convert audio electrical signals into sound signals. Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

The earphone interface 170D is used for connecting wired earphones. The earphone interface 170D may be the USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, pressure sensor 180A may be disposed on display screen 194 . The gyro sensor 180B can be used to determine the motion posture of the electronic device. The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall sensor. The acceleration sensor 180E can detect the acceleration of the electronic device in various directions (generally three axes). The distance sensor 180F is used to measure the distance. Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The ambient light sensor 180L is used for sensing ambient light brightness. The fingerprint sensor 180H is used to collect fingerprints. Electronic devices can use the collected fingerprint features to unlock fingerprints, access application locks, take pictures with fingerprints, answer incoming calls with fingerprints, etc. The temperature sensor 180J is used to detect temperature.

Touch sensor 180K, also known as "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194 . In some other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device, which is different from the position of the display screen 194 .

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device can receive key input and generate key signal input related to user settings and function control of the electronic device. The motor 191 can generate a vibrating reminder. The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like. The SIM card interface 195 is used for connecting a SIM card. The electronic device interacts with the network through the SIM card to realize functions such as calling and data communication.

FIG. 25 is an exemplary schematic diagram of a software structure of an electronic device provided by an embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the system is divided into four layers, which are application program layer, application program framework layer, system library, and kernel layer from top to bottom.

The application layer can consist of a series of application packages. As shown in FIG. 20, the application package may include application programs (also called applications) such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 25, the application framework layer can include window management service, display management service, content provider, view system, phone manager, resource manager, notification manager, local profile management assistant (LocalProfile Assistant, LPA), etc. .

The window management service is responsible for the startup, addition, and deletion of windows. It can determine the applications displayed on the windows and the creation, destruction, and property changes of the layers of the applications. It can determine whether there is a status bar, lock the screen, and capture the screen.

The display management service can obtain the number and size of display areas, and is responsible for starting, adding, and deleting display areas.

Content providers are used to store and retrieve data and make it accessible to applications. Said data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.

The phone manager is used to provide communication functions of electronic devices. For example, the management of call status (including connected, hung up, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog interface. For example, prompting text information in the status bar, issuing a prompt sound, vibrating the electronic device, and flashing the indicator light, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

The view system also includes UniRender, which can receive one or more application render trees. UniRender can synchronize layer information through the window management service, such as layer creation, destruction, property change, etc. UniRender can synchronize the information of the display area, such as the size of the screen, from the display management service.

Optionally, in some embodiments of the present application, the view system further includes SurfaceFlinger. On an electronic device configured with a whitelist, when the application does not belong to the whitelist, after the UI thread of the application generates the rendering tree, the Render thread of the application generates the bitmap, and then passes it to SurfaceFlinger for layering Synthesis.

Optionally, in some embodiments of the present application, when there are applications in the whitelist and applications not in the whitelist displayed on the display area, the bitmap of the application in the whitelist is generated by UniRender Responsible, after UniRender generates the bitmap, it passes the bitmap to SurfaceFlinger, and SurfaceFlinger then combines the bitmap with other applications not in the whitelist to generate a bitmap for display.

The runtime includes the core library and virtual machine. The runtime is responsible for the scheduling and management of the operating system.

The core library consists of two parts: one part is the function function that the java language needs to call, and the other part is the core library.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

A system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), graphics processing library, wherein, the graphics processing library includes: 3D graphics processing library (eg: OpenGLES), 2D graphics engine (eg: SGL) and so on.

The surface manager is used to manage the display subsystem, and provides fusion of two-dimensional (2-Dimensional, 2D) and three-dimensional (3-Dimensional, 3D) layers for multiple applications.

The media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, layer composition, and layer processing. 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, and a virtual card driver.

In the embodiment of the present application, one or more application programs in the application program layer deliver the rendering trees produced by their respective UI threads to the UniRender process of the view system. The UniRender process obtains window control information and display area information from the window management service and display management service, and then synthesizes the rendering tree of the application program on a display area into a target rendering tree. After the UniRender process generates the target rendering tree, it calls the layer processing library, executes the DrawOP operation in the drawing instruction list in the target rendering tree, and then generates a bitmap. UniRender passes the generated bitmap to the display driver for display.

As used in the above embodiments, depending on the context, the term "when" may be interpreted to mean "if" or "after" or "in response to determining..." or "in response to detecting...". Similarly, depending on the context, the phrase "in determining" or "if detected (a stated condition or event)" may be interpreted to mean "if determining..." or "in response to determining..." or "on detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)".

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, e.g. (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state hard disk), and the like.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments are realized. The processes can be completed by computer programs to instruct related hardware. The programs can be stored in computer-readable storage media. When the programs are executed , may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: ROM or random access memory RAM, magnetic disk or optical disk, and other various media that can store program codes.

Claims (11)

1. An interface generation method applied to electronic equipment is characterized by comprising the following steps:

the first process determines a target rendering tree, wherein the target rendering tree is used for drawing a target interface;

in the process that the first process generates the target interface based on the target rendering tree, the first process determines first content based on at least one rendering node in the target rendering tree, and performs animation rendering based on first animation parameters on the first content to obtain second content, wherein the animation parameters of the at least one rendering node are the first animation parameters;

or in the process that the first process generates the target interface based on the target rendering tree, the first process determines first content based on a first area on a first surface, and performs animation rendering based on first animation parameters on the first content to obtain the second content, wherein the first process generates the target interface on the first surface based on the target rendering tree, and the first animation parameters are determined by the first process;

The first process generates the target interface, the target interface including the second content, the animated rendering being rendering in an off-screen buffer.

2. The method of claim 1, wherein before the first process determines the target rendering tree, the method further comprises:

the first process adding the at least one rendering node in the target rendering tree, the at least one rendering node configured with the first animation parameters;

alternatively, the first process configures the first animation parameters for the at least one rendering node.

3. A method according to claim 1 or 2, characterized in that,

in the case that the animation parameters of the at least one rendering node indicate a foreground blur, the first content includes a view corresponding to the at least one rendering node;

in the event that the animation parameters of the at least one rendering node indicate background blur, the first content includes content of a corresponding region of the at least one rendering node on the first surface.

4. A method according to any of claims 1-3, wherein before the first process determines the target rendering tree, the method further comprises:

The first process generates the target rendering tree based on a second rendering tree and a third rendering tree, wherein the second rendering tree is a rendering tree generated by a second process and used for drawing a frame interface of the second process; the third rendering tree is a rendering tree generated by a third process for drawing a frame interface of the third process.

5. The method of claim 4, wherein the step of determining the position of the first electrode is performed,

the at least one rendering node includes a rendering node in the second rendering tree and a rendering node in the third rendering tree, and the first content includes a portion or all of a frame interface of the second process and a portion or all of a frame interface of the third process.

6. The method according to claim 4 or 5, wherein,

a first rendering node in the second rendering tree is configured with the first animation parameters by the second process;

a second rendering node in the third rendering tree is configured with the first animation parameters by the third process;

the at least one rendering node includes the first rendering node and the second rendering node.

7. The method according to any one of claims 1-6, wherein in case the animation parameters indicate background blur or foreground blur, the animation parameters further comprise at least one of blur degree, blur precision, blur radius.

8. An electronic device, the electronic device comprising: one or more processors and memory;

the memory is coupled to the one or more processors, the memory for storing computer-readable instructions that are invoked by the one or more processors to cause the electronic device to perform the method of any one of claims 1-7.

9. A chip system for application to an electronic device, the chip system comprising one or more processors to invoke computer instructions to cause the electronic device to perform the method of any of claims 1-7.

10. A computer readable storage medium comprising instructions which, when run on an electronic device, cause the electronic device to perform the method of any of claims 1-7.

11. A computer program product comprising computer readable instructions which, when executed by one or more processors, implement the method of any of claims 1-7.

Images