CN120011096A - Multimedia resource processing method, device, electronic device and storage medium - Google Patents
Multimedia resource processing method, device, electronic device and storage medium Download PDFInfo
- Publication number
- CN120011096A CN120011096A CN202311529637.5A CN202311529637A CN120011096A CN 120011096 A CN120011096 A CN 120011096A CN 202311529637 A CN202311529637 A CN 202311529637A CN 120011096 A CN120011096 A CN 120011096A
- Authority
- CN
- China
- Prior art keywords
- processing
- multimedia
- target
- strategy
- resource
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/541—Interprogram communication via adapters, e.g. between incompatible applications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
本公开是关于一种多媒体资源处理方法、装置、电子设备及存储介质。该方法包括：检测多媒体资源的处理指令；响应于所述处理指令，调用为WebAssembly应用程序提供的多媒体接口进行多媒体资源处理，所述多媒体接口由WebAssembly的兼容层提供底层实现，所述WebAssembly的兼容层动态适配当前的电子设备。该方法为WebAssembly应用程序提供了多媒体接口，且由于该多媒体接口由WebAssembly的兼容层提供底层实现，而WebAssembly的兼容层动态适配当前的电子设备，因此，该多媒体接口适用于多种类型的电子设备，可以快速、方便地接入多种类型的电子设备，在多种类型的电子设备上高效运行，以进行多媒体资源处理。
The present disclosure relates to a multimedia resource processing method, device, electronic device and storage medium. The method includes: detecting a processing instruction of a multimedia resource; in response to the processing instruction, calling a multimedia interface provided for a WebAssembly application to process the multimedia resource, wherein the multimedia interface is provided with an underlying implementation by a WebAssembly compatibility layer, and the WebAssembly compatibility layer dynamically adapts to the current electronic device. The method provides a multimedia interface for a WebAssembly application, and since the multimedia interface is provided with an underlying implementation by a WebAssembly compatibility layer, and the WebAssembly compatibility layer dynamically adapts to the current electronic device, the multimedia interface is suitable for a variety of electronic devices, can be quickly and conveniently connected to a variety of electronic devices, and can run efficiently on a variety of electronic devices to process multimedia resources.
Description
Technical Field
The disclosure relates to the field of computer technology, and in particular, to a method and device for processing multimedia resources, an electronic device and a storage medium.
Background
WebAssembly (WASN) is a new code that runs in modern web browsers and provides new performance characteristics and effects, has a compact binary format, can run at near the performance of native languages, and provides an efficient compilation target for lower source languages such as C, C ++.
Early WASM was mainly applied to web applications, but as they continue to develop, more and more projects use them as the final format of the smart contract, load runs using corresponding virtual machines (VM Virtual Machine), such as wavm, wagon, and specific parsing and running of bytecodes are performed in the core virtual machine module. The virtual machine (VM Virtual Machine) refers to a complete computer system which is simulated by software, has complete hardware system functions and operates in an isolated environment, such as a Vmware virtual machine virtualized by an operating system, a JVM virtual machine of a software operating environment, and the like.
Disclosure of Invention
In order to overcome the problems in the related art, the present disclosure provides a multimedia resource processing method, apparatus, electronic device, and storage medium.
According to a first aspect of embodiments of the present disclosure, there is provided a multimedia resource processing method, the method including:
Detecting a processing instruction of a multimedia resource;
And in response to the processing instruction, calling a multimedia interface provided for WebAssembly application programs to process multimedia resources, wherein the multimedia interface is realized by a bottom layer provided by a compatible layer of WebAssembly, and the compatible layer of WebAssembly is dynamically adapted to the current electronic equipment.
In some embodiments, the calling the multimedia interface provided for WebAssembly application to perform multimedia resource processing in response to the processing instruction includes:
Responding to the processing instruction, and acquiring resource capability information of the electronic equipment;
Determining a target processing strategy based on the resource capacity information, wherein the target processing strategy is used for processing multimedia resources;
And calling the multimedia interface to process the multimedia resource based on the target processing strategy.
In some embodiments, the method further comprises:
Configuring a plurality of processing strategies corresponding to the WebAssembly application programs;
The determining a target processing strategy based on the resource capability information comprises the following steps:
the target processing policy is selected from the plurality of processing policies based on the resource capability information.
In some embodiments, the selecting a target processing policy from the plurality of processing policies based on the resource capability information comprises:
determining a target weight corresponding to each processing strategy based on the resource capacity information, wherein the target weight is used for representing the data processing capacity of the processing strategy under the current resource capacity information, and the data processing capacities of the same processing strategy under different resource capacity information are different;
And selecting the target processing strategy from the plurality of processing strategies based on the target weights corresponding to the plurality of processing strategies.
In some embodiments, the method further comprises:
Configuring initial weights corresponding to each processing strategy;
the determining the target weight corresponding to each processing strategy based on the resource capability information comprises the following steps:
And adjusting the initial weight corresponding to each processing strategy based on the resource capacity information to obtain the target weight corresponding to each processing strategy.
In some embodiments, in a case that the processing policy is used for processing a multimedia resource, the selecting, based on the target weights corresponding to the multiple processing policies, the target processing policy from the multiple processing policies includes:
acquiring the corresponding processing time length of each processing strategy, wherein the processing time length is the time length of acquiring the decoded multimedia resource based on the corresponding processing strategy;
And selecting the target processing strategy from the plurality of processing strategies based on the processing time lengths and the target weights corresponding to the plurality of processing strategies.
In some embodiments, the plurality of processing strategies includes a first strategy for acquiring decoded multimedia resources from a server and a plurality of second strategies for acquiring and decoding undecoded multimedia resources from the server;
the selecting the target processing strategy from the plurality of processing strategies based on the processing time length and the target weight corresponding to the plurality of processing strategies comprises the following steps:
when the processing time length corresponding to the first strategy is smaller than or equal to the processing time length corresponding to the plurality of second strategies, determining the first strategy as the target processing strategy;
And when the processing time length corresponding to the first strategy is longer than the processing time length corresponding to the plurality of second strategies, determining the target processing strategy based on the target weights corresponding to the plurality of second strategies, wherein the target weights corresponding to the target processing strategies are greater than the target weights corresponding to other second strategies.
In some embodiments, the calling the multimedia interface to perform multimedia resource processing based on the target processing policy includes:
Calling the multimedia interface to acquire the decoded multimedia resource from the server based on the first policy, or
And calling the multimedia interface to acquire an undecoded multimedia resource from the server based on the second strategy, and decoding the multimedia resource.
In some embodiments, the processing instructions for detecting a multimedia asset include:
receiving the processing instruction sent by intelligent equipment, wherein the intelligent equipment is connected with the electronic equipment;
after the multimedia interface is called to process the multimedia resource based on the target processing strategy, the method further comprises the following steps:
And sending the decoded multimedia resources to intelligent equipment so that the intelligent equipment displays the multimedia resources.
In some embodiments, the obtaining the resource capability information of the electronic device includes:
And acquiring the resource capacity information through the WebAssembly compatible layer.
In some embodiments, before the acquiring, by the compatible layer WebAssembly, the resource capability information of the electronic device, the method further includes:
the open source code of the WebAssembly application is converted to machine code or intermediate code by WebAssembly compilation tools.
According to a second aspect of embodiments of the present disclosure, there is provided a multimedia resource processing apparatus, the apparatus comprising:
an instruction detection module configured to detect a processing instruction of a multimedia resource;
And the processing module is configured to respond to the processing instruction, call a multimedia interface provided for WebAssembly application programs to perform multimedia resource processing, the multimedia interface is realized by a bottom layer provided by a compatible layer of WebAssembly, and the compatible layer of WebAssembly is dynamically adapted to the current electronic equipment.
In some embodiments, the processing module is configured to:
Responding to the processing instruction, and acquiring resource capability information of the electronic equipment;
Determining a target processing strategy based on the resource capacity information, wherein the target processing strategy is used for processing multimedia resources;
And calling the multimedia interface to process the multimedia resource based on the target processing strategy.
In some embodiments, the apparatus further comprises:
The strategy configuration module is configured to configure various processing strategies corresponding to the WebAssembly application programs;
the processing module is configured to select the target processing policy from the plurality of processing policies based on the resource capability information.
In some embodiments, the processing module is configured to:
determining a target weight corresponding to each processing strategy based on the resource capacity information, wherein the target weight is used for representing the data processing capacity of the processing strategy under the current resource capacity information, and the data processing capacities of the same processing strategy under different resource capacity information are different;
And selecting the target processing strategy from the plurality of processing strategies based on the target weights corresponding to the plurality of processing strategies.
In some embodiments, the apparatus further comprises:
the initial weight configuration module is configured to configure initial weights corresponding to each processing strategy;
the processing module is configured to adjust the initial weight corresponding to each processing strategy based on the resource capacity information to obtain the target weight corresponding to each processing strategy.
In some embodiments, the processing module is configured to:
acquiring the corresponding processing time length of each processing strategy, wherein the processing time length is the time length of acquiring the decoded multimedia resource based on the corresponding processing strategy;
And selecting the target processing strategy from the plurality of processing strategies based on the processing time lengths and the target weights corresponding to the plurality of processing strategies.
In some embodiments, the plurality of processing strategies includes a first strategy for acquiring decoded multimedia resources from a server and a plurality of second strategies for acquiring and decoding undecoded multimedia resources from the server;
the processing module is configured to:
when the processing time length corresponding to the first strategy is smaller than or equal to the processing time length corresponding to the plurality of second strategies, determining the first strategy as the target processing strategy;
And when the processing time length corresponding to the first strategy is longer than the processing time length corresponding to the plurality of second strategies, determining the target processing strategy based on the target weights corresponding to the plurality of second strategies, wherein the target weights corresponding to the target processing strategies are greater than the target weights corresponding to other second strategies.
In some embodiments, the processing module is configured to:
Calling the multimedia interface to acquire the decoded multimedia resource from the server based on the first policy, or
And calling the multimedia interface to acquire an undecoded multimedia resource from the server based on the second strategy, and decoding the multimedia resource.
In some embodiments, the instruction detection module is configured to receive a multimedia resource acquisition instruction sent by an intelligent device, where the intelligent device is connected with the electronic device;
the apparatus further comprises:
and the resource sending module is configured to send the decoded multimedia resource to the intelligent equipment so as to enable the intelligent equipment to display the multimedia resource.
In some embodiments, the processing module is configured to obtain the resource capability information through a compatible layer of the WebAssembly.
In some embodiments, the apparatus further comprises:
A transcoding module configured to convert the open source code of the WebAssembly application into machine code or intermediate code via WebAssembly compilation tools.
According to a third aspect of embodiments of the present disclosure, there is provided an electronic device, comprising:
A processor;
a memory for storing processor-executable instructions;
Wherein the processor is configured to perform the method as described in the first aspect of the disclosed embodiments.
According to a fourth aspect of embodiments of the present disclosure, there is provided a non-transitory computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the method as described in the first aspect of embodiments of the present disclosure.
The method has the following beneficial effects:
The method provided by the embodiment of the disclosure provides a multimedia interface for WebAssembly application programs, and because the multimedia interface is realized by the bottom layer provided by the compatible layer of WebAssembly and the compatible layer of WebAssembly is dynamically adapted to the current electronic equipment, the multimedia interface is suitable for various types of electronic equipment, can be quickly and conveniently accessed into various types of electronic equipment, and can be efficiently operated on various types of electronic equipment to process multimedia resources.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a flow chart illustrating a method of multimedia asset processing according to an exemplary embodiment;
FIG. 2 is a flow chart illustrating a method of multimedia asset processing according to an exemplary embodiment;
FIG. 3 is a flowchart illustrating a method of multimedia asset processing according to an exemplary embodiment;
FIG. 4 is a block diagram of a multimedia asset processing device, according to an exemplary embodiment;
Fig. 5 is a block diagram of an electronic device, according to an example embodiment.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the invention. Rather, they are merely examples of apparatus and methods consistent with aspects of the invention as detailed in the accompanying claims.
The WebAssembly application in the embodiments of the present disclosure may be a graphics rendering application, a multimedia application (audio playing application, video playing application), or the like.
For WebAssembly application programs, in the related art, corresponding webapis are required to be called through glue codes written in JavaScript language to realize corresponding functions, so that the capability of WebAssembly application programs is limited, additional data transmission is added, overhead is increased, and the performance of electronic equipment is poor.
The method provided by the embodiment of the disclosure is executed by electronic equipment, and the electronic equipment can be equipment provided with WebAssembly application programs, such as mobile phones, tablet computers, notebook computers, desktop computers, vehicle-mounted terminals, wearable equipment, intelligent household equipment and the like.
Fig. 1 is a flowchart illustrating a multimedia asset processing method, performed by an electronic device, see fig. 1, according to an exemplary embodiment, the method comprising the steps of:
step S101, detecting a processing instruction of the multimedia resource.
The processing instruction is used for indicating to process the multimedia resource, for example, the processing instruction is an audio playing instruction, a video playing instruction, a graphic drawing instruction and the like.
In step S102, in response to the processing instruction, the multimedia interface provided for WebAssembly application is called to perform multimedia resource processing, the multimedia interface is implemented by the bottom layer provided by the compatible layer of WebAssembly, and the compatible layer of WebAssembly dynamically adapts to the current electronic device.
When the electronic device detects a processing instruction in the running process, the electronic device indicates that multimedia resource processing is needed, and in the case that the application program is WebAssembly application program (also called WASM application program), the multimedia resource processing is performed by calling a multimedia interface provided for the WebAssembly application program. The multimedia interface is implemented by providing a bottom layer by the compatible layer WebAssembly, and the compatible layer WebAssembly is dynamically adapted to the current electronic device, that is, the compatible layer WebAssembly provides a bottom layer for the multimedia interface, so that the multimedia interface can be suitable for various types of electronic devices.
The bottom implementation of the multimedia interface supports a native multimedia interface, for example, the native multimedia interface is an interface supporting framebuffer, or the native rendering interface may also be an interface supporting DRM (DIRECT RENDERING MANAGER ). Optionally, the number of the multimedia interfaces may be one or more, and in this embodiment of the disclosure, the multimedia interfaces are collectively referred to as interfaces supporting multimedia resource processing, and do not refer to any particular interface in particular.
The multimedia resources comprise audio resources, video resources, dynamic effect resources, graphic resources and the like, and the multimedia resources are processed by acquiring decoded audio resources, video resources, dynamic effect resources, graphic drawing and the like.
The method provided by the embodiment of the disclosure provides a multimedia interface for WebAssembly application programs, and because the multimedia interface is realized by the bottom layer provided by the compatible layer of WebAssembly and the compatible layer of WebAssembly is dynamically adapted to the current electronic equipment, the multimedia interface is suitable for various types of electronic equipment, can be quickly and conveniently accessed into various types of electronic equipment, and can be efficiently operated on various types of electronic equipment to process multimedia resources.
In addition, compared with the scheme of setting glue codes in the related art, the embodiment of the disclosure provides a WebAssembly compatible layer, and no glue codes are required to be set.
In one related technology, for WebAssembly applications, decoded multimedia resources such as audio and video or special effects are obtained from a Web server and then played in an electronic device. The method is easy to be influenced by network bandwidth, when the network bandwidth is smaller, the playing is caused to have a cartoon and bad listening visual experience, but if the un-decoded multimedia resources are obtained from the Web server and then are decoded and played in the electronic equipment, the performance of the electronic equipment is difficult to be ensured because the encoding and decoding capabilities of different terminals on the multimedia resources are uneven. Aiming at the problem of poor performance, referring to the implementation manner shown in fig. 2 below, the multimedia resource processing method provided by the embodiment of the disclosure can improve the performance of the electronic device and enhance the use experience of the user.
Fig. 2 is a flowchart illustrating a method of multimedia asset processing, performed by an electronic device, see fig. 2, according to an exemplary embodiment, the method comprising the steps of:
Step S201, detecting a processing instruction of the multimedia resource.
Step S202, in response to the processing instruction, resource capability information of the electronic equipment is acquired.
The resource capability information at least comprises hardware capability, network bandwidth, multimedia resource information to be processed and the like of the electronic equipment. The resource capability information is used to characterize the capability of the electronic device to currently perform data processing. For example, when the resource capability information includes a hardware capability of the electronic device, the better the hardware capability is, the stronger the capability of the electronic device for currently performing data processing is, the worse the hardware capability is, the worse the capability of the electronic device for currently performing data processing is, and for example, when the resource capability information includes a network bandwidth, the larger the network bandwidth is, the stronger the capability of the electronic device for currently performing data processing is, the smaller the network bandwidth is, and the worse the capability of the electronic device for currently performing data processing is. Wherein the data processing includes multimedia asset processing.
In some embodiments, the resource capability information is obtained through a compatible layer of WebAssembly.
Step S203, determining a target processing policy based on the resource capability information.
The target processing policy is used for processing the multimedia resource, and the target processing policy may be hardware or software such as encapsulated hardware, a custom architecture, an open source architecture, and the like. Because the processing strategy depends on the resource capability information of the electronic equipment, when the resource capability information is different, the data processing capability of the processing strategy is also different, so that the target processing strategy with the best data processing capability under the current condition is determined for enabling the electronic equipment to have better performance under the current resource capability information.
Step S204, based on the target processing strategy, calling the multimedia interface to process the multimedia resource.
Calling the multimedia interface to process the multimedia resources based on the target processing strategy, wherein the method comprises the steps of calling the multimedia interface to obtain decoded audio resources, video resources and dynamic resources based on the target processing strategy, or calling the multimedia interface to draw graphics based on the target processing strategy.
According to the method provided by the embodiment of the disclosure, in the running process of the WebAssembly application program, the resource capacity information of the electronic equipment is acquired, the target processing strategy with the highest data processing capacity is determined based on the resource capacity information, and the multimedia interface is called for multimedia resource processing based on the target processing strategy, so that the electronic equipment is ensured to have higher performance when the WebAssembly application program processes the multimedia resource, and the use experience of a user is improved.
Fig. 3 is a flowchart illustrating a method of multimedia asset processing, performed by an electronic device, see fig. 3, according to an exemplary embodiment, the method comprising the steps of:
In step S301, a plurality of processing policies corresponding to the application program are configured WebAssembly.
In the embodiment of the disclosure, in the compiling stage, a plurality of processing strategies corresponding to WebAssembly application programs are configured, so that when the subsequent WebAssembly application programs run, data processing is performed based on the configured processing strategies. The processing policy is used for processing the multimedia resource, and the processing policy may refer to hardware or software such as encapsulated hardware, custom architecture, open source architecture, and the like. For example, the open source architecture may be ffmpeg, gtreamer, pluseaudio, etc., and for example, the open source architecture may be LVGL, WESTON, EGL, openGL, etc.
The data processing capabilities of different processing strategies differ for a plurality of processing strategies. For example, the processing time length when the decoded multimedia resources are acquired based on different processing strategies is different.
In some embodiments, a configuration file is provided, and during the compilation stage, a plurality of processing strategies are configured based on the configuration file. The configuration file is used for configuring the processing strategy.
In step S302, the open source code of the WebAssembly application is converted into machine code or intermediate code by WebAssembly compiling means.
The WebAssembly compiling tool is used for converting open source codes, and the open source codes can be application program codes written in any language. In the compiling stage, when the open source code is converted into the machine code through the WebAssembly compiling tool, the electronic equipment can directly run the application program based on the converted machine code without performing other conversion, and the running efficiency can be improved.
In addition, in the compiling stage, the open source codes are converted into machine codes or intermediate codes through WebAssembly compiling tools, so that the isolation of the open source codes is realized, and the copyright protection is realized.
Step S303, detecting a processing instruction of the multimedia resource.
Step S304, responding to the processing instruction, and acquiring the resource capability information through a WebAssembly compatible layer.
In the embodiment of the present disclosure, at the compatible layer WebAssembly, the resource capability information of the electronic device is dynamically identified, that is, the current resource capability information is identified in real time. The compatible layer WebAssembly in the embodiments of the present disclosure can dynamically identify which information is not limited.
Step S305, based on the resource capability information, a target weight corresponding to each processing policy is determined.
The target weight is used for representing the data processing capability of the processing strategy under the current resource capability information, and the data processing capability of the same processing strategy under different resource capability information is different, namely the corresponding target weight of the same processing strategy under different resource capability information is different.
In some embodiments, initial weights corresponding to each processing strategy are configured, and based on the resource capacity information, the initial weights corresponding to each processing strategy are adjusted to obtain target weights corresponding to each processing strategy. Wherein the initial weight is a pre-configured weight. The initial weights are also used for representing the data processing capacity of the processing strategies, but the initial weights are irrelevant to the resource capacity information, so that under the current resource capacity information, the initial weights corresponding to each processing strategy are required to be adjusted to obtain target weights which are more in line with the current resource capacity information, and the target weights can more accurately represent the data processing capacity of the processing strategies under the current resource capacity information compared with the initial weights.
Optionally, in the compiling stage, under the same data processing scene, determining time consumption when each processing strategy performs data processing, and then configuring initial weight corresponding to each processing strategy based on the time consumption corresponding to each processing strategy.
In some embodiments, a correspondence between a target weight corresponding to each processing policy and the resource capability information is preset, and based on the correspondence, the target weight corresponding to each processing policy under the current resource capability information is determined.
In some embodiments, a processing duration corresponding to each processing policy is obtained, and a target processing policy is selected from the plurality of processing policies based on the processing durations and the target weights corresponding to the plurality of processing policies. The processing time length is the time length of acquiring the decoded multimedia resource based on the corresponding processing strategy, and the longer the processing time length is, the worse the data processing capability of the processing strategy is, the shorter the processing capability is, and the better the data processing capability of the processing strategy is, so that the target processing strategy is selected based on the processing time length and the target weight.
Optionally, when selecting the target processing policy, the processing duration is preferentially considered, or the target weight is preferentially considered, or the processing duration and the target weight are comprehensively considered.
In some embodiments, the plurality of processing strategies includes a first strategy for obtaining decoded multimedia resources from the server and a plurality of second strategies for obtaining and decoding undecoded multimedia resources from the server. Under the condition of priority of performance, the first strategy for acquiring the decoded multimedia resources is considered preferentially, and if the data processing capacity of the first strategy is poor, the second strategy for acquiring the un-decoded multimedia resources is considered. The method comprises the steps of determining a first strategy as a target processing strategy when processing time length corresponding to the first strategy is smaller than or equal to processing time length corresponding to a plurality of second strategies, and determining the target processing strategy based on target weights corresponding to the plurality of second strategies when the processing time length corresponding to the first strategy is longer than the processing time length corresponding to the plurality of second strategies, wherein the target weight corresponding to the target processing strategy is greater than the target weights corresponding to other second strategies.
Optionally, the data processing capability of the first policy is determined based on the current network bandwidth, and the larger the current network bandwidth is, the better the data processing capability of the first policy is, the smaller the current network bandwidth is, and the worse the data processing capability of the first policy is.
Step S306, selecting a target processing strategy from the plurality of processing strategies based on the target weights corresponding to the plurality of processing strategies.
In some embodiments, the processing policy with the largest corresponding target weight is used as the target processing policy, that is, the processing policy with the highest data processing capability is selected as the target processing policy, so that better performance is ensured when the data processing is performed subsequently.
Step S307, based on the target processing strategy, the multimedia interface is called to process the multimedia resource.
The multimedia resources include audio resources, video resources, dynamic resources, graphics resources, and the like.
In some embodiments, the multimedia interface is invoked to obtain decoded multimedia resources from the server based on the first policy when the target processing policy is the first policy, or the multimedia interface is invoked to obtain undecoded multimedia resources from the server and decode multimedia resources based on the second policy when the target processing policy is any second policy. The decoded multimedia resources refer to decoded audio resources, video resources and dynamic resources.
It should be noted that, after the electronic device obtains the decoded multimedia resource, the multimedia resource may also be displayed.
In some embodiments, when the electronic device uses the local codec capability, the electronic device performs the above steps S301-S307, resulting in a decoded multimedia resource. When the electronic device needs to use the codec capability provided by the remote network, it needs to acquire the decoded multimedia resource from another device connected to the electronic device, and the other device may perform the above steps S301 to S307, and then send the acquired decoded multimedia resource to the electronic device.
Alternatively, for a smart device to connect to an electronic device, the smart device may need to use the codec capabilities provided by the remote network. And after the decoded multimedia resources are obtained, the electronic equipment sends the decoded multimedia resources to the intelligent equipment so as to enable the intelligent equipment to display the multimedia resources.
In some embodiments, when the multimedia resource is a graphics resource, the underlying hardware that has the GPU (Graphics Processing Unit, graphics processor) acceleration selected during the rendering process may have a higher performance and better user experience when the graphics rendering application is running. The method comprises the steps of calling a multimedia interface to process multimedia resources based on a target processing strategy, wherein the method comprises the step of calling the multimedia interface to process the multimedia resources based on bottom hardware accelerated by a CPU.
According to the method provided by the embodiment of the disclosure, a plurality of corresponding processing strategies are configured for the WebAssembly-based application program, and a WebAssembly compatible layer is arranged, in the running process of the application program, the resource capability information of the electronic device is obtained through the WebAssembly compatible layer, the target processing strategy with the highest data processing capability is selected from the plurality of processing strategies based on the resource capability information, and the data processing is performed based on the target processing strategy, so that the electronic device has higher performance when the WebAssembly-based application program processes multimedia resources or performs graphic drawing, and the use experience of a user is improved. In addition, compared with the scheme of setting glue codes in the related art, the embodiment of the disclosure is provided with the WebAssembly compatible layer, and the glue codes are not required to be set.
In addition, a compatible layer of WebAssembly is provided in the embodiment of the present disclosure, and a unified preset interface is provided for the application program based on WebAssembly, so that efficient running on various types of devices can be realized, the application program is isolated under the condition that the unified preset interface is kept unchanged, and consistent use experience is provided for users. In addition, in the compiling stage, the open source codes are converted into machine codes or intermediate codes through WebAssembly compiling tools, so that the isolation of the open source codes is realized, and the copyright protection is realized.
Fig. 4 is a block diagram of a multimedia resource processing apparatus configured to an electronic device, see fig. 4, according to an exemplary embodiment, the apparatus comprising:
An instruction detection module 401 configured to detect processing instructions of the multimedia resource;
The processing module 402 is configured to respond to the processing instruction, call the multimedia interface provided for WebAssembly application programs to perform multimedia resource processing, the multimedia interface is implemented by the bottom layer provided by the compatible layer of WebAssembly, and the compatible layer of WebAssembly is dynamically adapted to the current electronic device.
In some embodiments, the processing module 402 is configured to:
responding to the processing instruction, and acquiring resource capability information of the electronic equipment;
determining a target processing strategy based on the resource capacity information, wherein the target processing strategy is used for processing the multimedia resource;
And calling the multimedia interface to process the multimedia resource based on the target processing strategy.
In some embodiments, the apparatus further comprises:
The strategy configuration module is configured to configure WebAssembly a plurality of processing strategies corresponding to the application program;
A processing module 402 configured to select a target processing policy from a plurality of processing policies based on the resource capability information.
In some embodiments, the processing module 402 is configured to:
Determining a target weight corresponding to each processing strategy based on the resource capability information, wherein the target weight is used for representing the data processing capability of the processing strategy under the current resource capability information, and the data processing capability of the same processing strategy under different resource capability information is different;
and selecting a target processing strategy from the plurality of processing strategies based on the target weights corresponding to the plurality of processing strategies.
In some embodiments, the apparatus further comprises:
The initial weight configuration module is configured to configure initial weights corresponding to each processing strategy;
The processing module 402 is configured to adjust an initial weight corresponding to each processing policy based on the resource capability information, so as to obtain a target weight corresponding to each processing policy.
In some embodiments, the processing module 402 is configured to:
acquiring a processing time length corresponding to each processing strategy, wherein the processing time length is the time length of acquiring the decoded multimedia resource based on the corresponding processing strategy;
and selecting a target processing strategy from the plurality of processing strategies based on the processing time lengths and the target weights corresponding to the plurality of processing strategies.
In some embodiments, the plurality of processing strategies includes a first strategy for obtaining decoded multimedia resources from the server and a plurality of second strategies for obtaining and decoding undecoded multimedia resources from the server;
A processing module 402 configured to:
when the processing time length corresponding to the first strategy is smaller than or equal to the processing time length corresponding to the plurality of second strategies, determining the first strategy as a target processing strategy;
When the processing time length corresponding to the first strategy is longer than the processing time length corresponding to the plurality of second strategies, determining a target processing strategy based on the target weights corresponding to the plurality of second strategies, wherein the target weights corresponding to the target processing strategies are greater than the target weights corresponding to the other second strategies.
In some embodiments, the processing module 402 is configured to:
calling the multimedia interface to obtain the decoded multimedia resource from the server based on the first strategy, or
And calling the multimedia interface to acquire the undecoded multimedia resource from the server based on the second strategy, and decoding the multimedia resource.
In some embodiments, the instruction detection module 401 is configured to receive a multimedia resource acquisition instruction sent by an intelligent device, where the intelligent device is connected to the electronic device;
The apparatus further comprises:
And the resource sending module is configured to send the decoded multimedia resource to the intelligent device so as to enable the intelligent device to display the multimedia resource.
In some embodiments, the processing module 402 is configured to obtain the resource capability information through a compatible layer of WebAssembly.
In some embodiments, the apparatus further comprises:
A transcoding module configured to convert the open source code of WebAssembly application into machine code or intermediate code by WebAssembly compilation tools.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
The embodiment of the disclosure also provides electronic equipment, which comprises a processor and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the multimedia resource processing method in the embodiment.
Fig. 5 is a block diagram of an electronic device 500, shown in accordance with an exemplary embodiment.
Referring to FIG. 5, an electronic device 500 may include one or more of a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input/output (I/O) interface 512, a sensor component 514, and a communication component 516.
The processing component 502 generally controls overall operation of the electronic device 500, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 502 can include one or more modules that facilitate interactions between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.
The memory 504 is configured to store various types of data to support operations at the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 504 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
The power supply component 506 provides power to the various components of the electronic device 500. The power components 506 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 500.
The multimedia component 508 includes a screen between the electronic device 500 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front-facing camera and/or a rear-facing camera. When the electronic device 500 is in an operational mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
The audio component 510 is configured to output and/or input audio signals. For example, the audio component 510 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 500 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 further comprises a speaker for outputting audio signals.
The I/O interface 512 provides an interface between the processing component 502 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, a home button, a volume button, an activate button, and a lock button.
The sensor assembly 514 includes one or more sensors for providing status assessment of various aspects of the electronic device 500. For example, the sensor assembly 514 may detect an on/off state of the electronic device 500, a relative positioning of components such as a display and keypad of the electronic device 500, a change in position of the electronic device 500 or a component of the electronic device 500, the presence or absence of a user's contact with the electronic device 500, an orientation or acceleration/deceleration of the electronic device 500, and a change in temperature of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 516 is configured to facilitate communication between the electronic device 500 and other devices, either wired or wireless. The electronic device 500 may access a wireless network based on a communication standard, such as WiFi,2G, or 3G, or a combination thereof. In one exemplary embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device 500 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.
In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, such as memory 504, including instructions executable by processor 620 of electronic device 500 to perform the above-described method. For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
The disclosed embodiments also provide a non-transitory computer readable storage medium, which when executed by a processor of an electronic device, enables the electronic device to perform the multimedia resource processing method in the above embodiments.
Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the invention is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (14)
1. A method for processing a multimedia resource, the method comprising:
Detecting a processing instruction of a multimedia resource;
And in response to the processing instruction, calling a multimedia interface provided for WebAssembly application programs to process multimedia resources, wherein the multimedia interface is realized by a bottom layer provided by a compatible layer of WebAssembly, and the compatible layer of WebAssembly is dynamically adapted to the current electronic equipment.
2. The method of claim 1, wherein said invoking the multimedia interface provided for WebAssembly applications for multimedia asset processing in response to the processing instruction comprises:
Responding to the processing instruction, and acquiring resource capability information of the electronic equipment;
Determining a target processing strategy based on the resource capacity information, wherein the target processing strategy is used for processing multimedia resources;
And calling the multimedia interface to process the multimedia resource based on the target processing strategy.
3. The method according to claim 2, wherein the method further comprises:
Configuring a plurality of processing strategies corresponding to the WebAssembly application programs;
The determining a target processing strategy based on the resource capability information comprises the following steps:
the target processing policy is selected from the plurality of processing policies based on the resource capability information.
4. The method of claim 3, wherein the selecting the target processing policy from the plurality of processing policies based on the resource capability information comprises:
determining a target weight corresponding to each processing strategy based on the resource capacity information, wherein the target weight is used for representing the data processing capacity of the processing strategy under the current resource capacity information, and the data processing capacities of the same processing strategy under different resource capacity information are different;
And selecting the target processing strategy from the plurality of processing strategies based on the target weights corresponding to the plurality of processing strategies.
5. The method according to claim 4, wherein the method further comprises:
Configuring initial weights corresponding to each processing strategy;
the determining the target weight corresponding to each processing strategy based on the resource capability information comprises the following steps:
And adjusting the initial weight corresponding to each processing strategy based on the resource capacity information to obtain the target weight corresponding to each processing strategy.
6. The method of claim 4, wherein the selecting the target processing policy from the plurality of processing policies based on the target weights corresponding to the plurality of processing policies comprises:
acquiring the corresponding processing time length of each processing strategy, wherein the processing time length is the time length of acquiring the decoded multimedia resource based on the corresponding processing strategy;
And selecting the target processing strategy from the plurality of processing strategies based on the processing time lengths and the target weights corresponding to the plurality of processing strategies.
7. The method of claim 6, wherein the plurality of processing strategies includes a first strategy for acquiring decoded multimedia resources from a server and a plurality of second strategies for acquiring and decoding undecoded multimedia resources from a server;
the selecting the target processing strategy from the plurality of processing strategies based on the processing time length and the target weight corresponding to the plurality of processing strategies comprises the following steps:
when the processing time length corresponding to the first strategy is smaller than or equal to the processing time length corresponding to the plurality of second strategies, determining the first strategy as the target processing strategy;
And when the processing time length corresponding to the first strategy is longer than the processing time length corresponding to the plurality of second strategies, determining the target processing strategy based on the target weights corresponding to the plurality of second strategies, wherein the target weights corresponding to the target processing strategies are greater than the target weights corresponding to other second strategies.
8. The method of claim 7, wherein invoking the multimedia interface for multimedia asset processing based on the target processing policy comprises:
Calling the multimedia interface to acquire the decoded multimedia resource from the server based on the first policy, or
And calling the multimedia interface to acquire an undecoded multimedia resource from the server based on the second strategy, and decoding the multimedia resource.
9. The method of claim 8, wherein detecting processing instructions for a multimedia asset comprises:
receiving the processing instruction sent by intelligent equipment, wherein the intelligent equipment is connected with the electronic equipment;
after the multimedia interface is called to process the multimedia resource based on the target processing strategy, the method further comprises the following steps:
And sending the decoded multimedia resources to intelligent equipment so that the intelligent equipment displays the multimedia resources.
10. The method of claim 2, wherein the obtaining the resource capability information of the electronic device comprises:
And acquiring the resource capacity information through the WebAssembly compatible layer.
11. The method according to any one of claims 1 to 10, further comprising:
the open source code of the WebAssembly application is converted to machine code or intermediate code by WebAssembly compilation tools.
12. A multimedia resource processing apparatus, the apparatus comprising:
an instruction detection module configured to detect a processing instruction of a multimedia resource;
And the processing module is configured to respond to the processing instruction, call a multimedia interface provided for WebAssembly application programs to perform multimedia resource processing, the multimedia interface is realized by a bottom layer provided by a compatible layer of WebAssembly, and the compatible layer of WebAssembly is dynamically adapted to the current electronic equipment.
13. An electronic device, comprising:
A processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to perform the method of any of claims 1-11.
14. A non-transitory computer readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method of any one of claims 1-11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311529637.5A CN120011096A (en) | 2023-11-16 | 2023-11-16 | Multimedia resource processing method, device, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311529637.5A CN120011096A (en) | 2023-11-16 | 2023-11-16 | Multimedia resource processing method, device, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN120011096A true CN120011096A (en) | 2025-05-16 |
Family
ID=95659522
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311529637.5A Pending CN120011096A (en) | 2023-11-16 | 2023-11-16 | Multimedia resource processing method, device, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN120011096A (en) |
-
2023
- 2023-11-16 CN CN202311529637.5A patent/CN120011096A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12417065B2 (en) | Multi-screen collaboration method and system, and electronic device | |
| US20170235435A1 (en) | Electronic device and method of application data display therefor | |
| US9843667B2 (en) | Electronic device and call service providing method thereof | |
| US10181203B2 (en) | Method for processing image data and apparatus for the same | |
| CN111599358A (en) | Voice interaction method and electronic equipment | |
| EP4354270A1 (en) | Service recommendation method and electronic device | |
| CN106033397B (en) | Memory buffer area adjusting method, device and terminal | |
| CN114647390B (en) | Enhanced screen sharing method and system and electronic equipment | |
| CN109710259B (en) | Data processing method, device and storage medium | |
| WO2022078295A1 (en) | Device recommendation method and electronic device | |
| CN112527222A (en) | Information processing method and electronic equipment | |
| CN110413383B (en) | Event processing method, device, terminal and storage medium | |
| US11210449B2 (en) | Page display method and device and storage medium | |
| CN110991368A (en) | Camera scene recognition method and related device | |
| CN109522131A (en) | Resource allocation method, device and equipment | |
| CN110865864A (en) | Interface display method, device, device and storage medium for quick application | |
| US10075798B2 (en) | Method for providing audio and electronic device adapted to the same | |
| CN114007117B (en) | Control display method and device | |
| US20170013118A1 (en) | Electronic device and notification method thereof | |
| CN107679205B (en) | Multimedia data playing method and device, computer equipment and storage medium | |
| CN120011096A (en) | Multimedia resource processing method, device, electronic device and storage medium | |
| CN114116590B (en) | Data acquisition method, device, vehicle, storage medium and electronic equipment | |
| US20250284547A1 (en) | Quick application running method and electronic device | |
| CN110853643A (en) | Method, device, device and storage medium for speech recognition in fast application | |
| CN115314588B (en) | Background synchronization method, device, terminal, equipment, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |