HK40075015B - Cloud audio and video processing method, system, electronic equipment and storage medium - Google Patents

Description

Cloud audio and video processing methods, systems, electronic devices and storage media

Technical Field

This application relates to cloud technology, and more particularly to a cloud audio and video processing method, system, electronic device, and storage medium.

Background Technology

Currently, cloud service platform solutions used in cloud gaming or cloud live streaming audio-visual interactive systems typically integrate modules such as acquisition, encoding, transmission, and uplink event processing services. However, due to variations in host operating systems and hardware configurations, as well as differences in the underlying interfaces for acquisition, encoding, transmission, and events across different hardware configurations and host operating systems, current cloud service platforms require different module configurations and code implementations for different hardware configurations and host operating systems. Therefore, current cloud service platforms exhibit high module coupling and code redundancy, which hinders stable operation, upgrades, and maintenance, thereby reducing the compatibility, stability, and maintainability of cloud audio-visual interactive systems.

Summary of the Invention

This application provides a cloud audio and video processing method, system, electronic device, and storage medium, which can improve the compatibility, stability, and maintainability of the cloud audio and video interaction system.

The technical solution of this application embodiment is implemented as follows:

This application provides a cloud audio and video processing method, including:

The acquisition module is called through the standard audio and video acquisition interface to acquire audio and video data generated by the cloud audio and video process.

The encoding module is called through the encoding transmission standard interface to encode the audio and video data in a preset standard encoding format to obtain an encoded audio and video stream, and then the encoded audio and video stream is transmitted to the interactive device.

The system receives operation events initiated by the interactive device based on the encoded audio and video stream through a standard event input interface, and maps these operation events to target driver events corresponding to the current operating system. The target driver events are used to drive the cloud audio and video process to update the corresponding audio and video data.

The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively.

This application provides a cloud audio and video processing method, including:

The system receives encoded audio and video streams and decodes and plays them to obtain decoded video images and decoded audio. The encoded audio and video streams are generated by the first cloud host through the audio and video acquisition standard interface, which calls the acquisition module to acquire cloud audio and video data; and through the encoding transmission standard interface, which calls the encoding module to encode the audio and video data in a preset standard encoding format.

The system acquires an operation event initiated in response to the decoded video or audio, and sends the operation event to the event standard input interface of the first cloud host. This allows the first cloud host to map the operation event to a target driver event corresponding to the current operating system via the event standard input interface. The target driver event is used to drive the cloud audio/video process to update the corresponding audio/video data.

The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively.

This application provides a cloud audio and video processing method, including:

The encoded audio and video stream is obtained through a preset downlink standard interface; the encoded audio and video stream is generated by the first cloud host by calling the acquisition module through the audio and video acquisition standard interface to acquire cloud audio and video data; and the audio and video data is encoded by the encoding module through the encoding transmission standard interface to perform audio and video encoding in a preset standard encoding format; the preset downlink standard interface is connected to the encoding transmission standard interface on the first cloud host and the interactive device respectively.

The encoded audio and video streams are transmitted to the interactive device using a preset standard transmission format;

The system receives operation events initiated by the interactive device based on the encoded audio and video stream through a preset uplink standard interface; the preset uplink standard interface is connected to the event standard input interface on the first cloud host and the interactive device, respectively.

The operation event is transmitted to the event standard input interface using the preset standard transmission format.

This application provides a first cloud host, including:

The acquisition module is used to acquire audio and video data generated by the cloud audio and video process through the standard audio and video acquisition interface;

The encoding module is used to encode the audio and video data in a preset standard encoding format through an encoding transmission standard interface to obtain an encoded audio and video stream, and then transmit the encoded audio and video stream to the interactive device.

The event handling module is used to receive operation events initiated by the interactive device based on the encoded audio and video stream through the event standard input interface, and to map the operation events to target driving events corresponding to the current operating system; the target driving events are used to drive the cloud audio and video process to update the audio and video data accordingly; wherein...

The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively.

In the aforementioned first cloud host, the first cloud host further includes an initialization module. The initialization module is used to call the acquisition module through the audio and video acquisition standard interface before acquiring the audio and video data generated by the cloud audio and video process. Before acquiring the audio and video data, the initialization module calls the local initialization interfaces corresponding to the acquisition module, the encoding module, and the operating system through the initialization standard interface to initialize the acquisition module, the encoding module, and the operating system. The initialization standard interface is a standardized interface implemented in the cloud service standardized subclass.

In the aforementioned first cloud host, the acquisition module is further configured to periodically call the local acquisition driver interface through the audio and video acquisition standard interface to acquire the current video frame and current audio generated by the cloud audio and video process according to the preset sampling frame rate, so as to obtain the audio and video data; the preset sampling frame rate is the standardized sampling frame rate uniformly set in the preset cloud service standardization base class.

In the aforementioned first cloud host, the encoding module is further configured to call the local encoding driver interface through the encoding transmission standard interface, and encode the audio and video data according to the preset encoding frame rate and the preset standard encoding format to obtain an encoded audio and video stream; the preset encoding frame rate is the standardized encoding frame rate uniformly set in the preset cloud service standardization base class;

In the aforementioned first cloud host, the event processing module is further configured to call a preset instruction conversion service through the event standard input interface to determine the target driving event corresponding to the operation event from a preset correspondence between at least one preset operation event and at least one standard driving event; the at least one standard driving event is a standard driving event in the current operating system; and convert the operation event into the target driving event.

This application provides an interactive device, including:

A standardized client component is provided, which is connected to the encoding and transmission standard interface and the event standard input interface on the first cloud host; wherein...

The client-side standardized component is used to receive the encoded audio and video stream sent by the encoded transmission standard interface; and to decode and play the encoded audio and video stream to obtain decoded video and decoded audio; the encoded audio and video stream is obtained by the first cloud host calling the acquisition module through the audio and video acquisition standard interface to acquire cloud audio and video data generated by the cloud audio and video process; and by calling the encoding module through the encoded transmission standard interface to encode the audio and video data in a preset standard encoding format.

The client-side standardization component is further configured to acquire at least one type of raw operation instruction initiated for the decoded video or decoded audio, and to convert the raw operation instruction of at least one type of raw operation instruction according to a preset standard event format to obtain the operation event. The operation event is then sent to the event standard input interface, so that the first cloud host maps the operation event to a target driving event corresponding to the current operating system through the event standard input interface. The target driving event is used to drive the cloud audio/video process to update the audio/video data accordingly.

The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively.

This application provides a second cloud host, including:

The system includes a preset downlink standard interface and a preset uplink standard interface. The preset downlink standard interface is connected to the encoding transmission standard interface on the first cloud host and the interactive device, respectively. The preset uplink standard interface is connected to the interactive device and the event standard input interface on the first cloud host, respectively.

The preset downlink standard interface is used to receive the encoded audio and video stream sent by the first cloud host through the encoding transmission standard interface; the encoded audio and video stream is obtained by the first cloud host calling the acquisition module through the audio and video acquisition standard interface to acquire cloud audio and video data generated by the cloud audio and video process; and by calling the encoding module through the encoding transmission standard interface to encode the audio and video data in a preset standard encoding format; the encoded audio and video stream is transmitted to the interactive device using the preset standard transmission format;

The preset uplink standard interface is used to receive operation events initiated by the interactive device based on the encoded audio and video stream; and to transmit the operation events to the event standard input interface on the first cloud host using the preset standard transmission format.

In the aforementioned second cloud host, the interactive device is equipped with a client standardization component. The client standardization component is connected to the preset downlink standard interface and the preset uplink standard interface respectively. The preset downlink standard interface is also used to transmit the encoded audio and video stream to the client standardization component on the interactive device using the preset standard transmission format, so that the client standardization component can decode and play the encoded audio and video stream in a preset decoding format to obtain decoded video and decoded audio.

The preset uplink standard interface is also used to receive the operation event issued by the client standardization component on the interactive device, wherein the operation event is obtained by the client standardization component by acquiring at least one original operation instruction of at least one format type initiated for the decoded video screen or decoded audio, and converting the original operation instruction of at least one format type according to the preset standard event format.

In the aforementioned second cloud host, a standardized transmission service is implemented between the preset uplink standard interface and the preset downlink standard interface, and the standardized transmission service includes at least one of the following:

Uplink and downlink network transmission bandwidth assessment service, congestion control service, and connection maintenance and abnormal reconnection service.

In the aforementioned second cloud host, the event standard input interface is connected to the preset uplink standard interface via a first socket; the encoded transmission standard interface is connected to the preset downlink standard interface via a second socket.

In the aforementioned second cloud host, the preset standard transmission format includes at least one of the following:

The system includes a preset application layer standard protocol format, a preset transport layer standard protocol format, and a preset network layer standard protocol format. The preset application layer standard protocol format includes either a web instant messaging protocol or a real-time streaming protocol. The preset transport layer standard protocol format includes either a real-time transmission protocol or a real-time transmission control protocol. The preset network layer standard protocol format includes either a user data packet protocol or a flow control transmission protocol.

This application provides a cloud audio and video processing system, including: a first cloud host, a second cloud host, and an interactive device; the first cloud host runs a cloud audio and video process; the second cloud host is equipped with a preset uplink standard interface and a preset downlink standard interface; the interactive device is equipped with a client standardization component; the client standardization component is connected to an event standard input interface on the first cloud host through the preset uplink standard interface, and the client standardization component is connected to an encoding transmission standard interface on the first cloud host through the preset downlink standard interface; wherein...

The first cloud host is used to call the acquisition module through the audio and video acquisition standard interface to acquire the audio and video data generated by the cloud audio and video process; and to call the encoding module through the encoding transmission standard interface to encode the audio and video data in a preset standard encoding format to obtain an encoded audio and video stream, and to transmit the encoded audio and video stream to the preset downlink standard interface.

The second cloud host is used to transmit the encoded audio and video streams to the client standardization component via the preset downlink standard interface using a preset standard transmission format;

The interactive device is used to decode and play the encoded audio and video stream through the client standardization component to obtain decoded video and decoded audio; and to obtain operation events initiated for the decoded video or decoded audio and send the operation events to the preset uplink standard interface.

The second cloud host is also used to transmit the operation event to the event standard input interface through the preset uplink standard interface using the preset standard transmission format;

The first cloud host is further configured to map the operation event to a target driving event corresponding to the current operating system through the event standard input interface; the target driving event is used to drive the cloud audio and video process to update the audio and video data accordingly;

The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, respectively based on the configuration information of the acquisition module, the encoding module, and the operating system.

This application provides an electronic device, including:

Memory, used to store executable instructions;

When the processor executes the executable instructions stored in the memory, it implements any of the cloud audio and video processing methods provided in the embodiments of this application.

This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement any of the cloud audio and video processing methods described above.

The embodiments of this application have the following beneficial effects:

By implementing standardized interfaces in cloud service standardized subclasses derived from a predefined cloud service standardized base class, based on the configuration information of the acquisition module, encoding module, and operating system on the cloud service platform host, the system can capture audio and video data by calling the acquisition module configured on the cloud service platform host through the audio and video acquisition standard interface; perform standardized audio and video encoding and output by calling the encoding module configured on the cloud service platform host through the encoding and transmission standard interface; and map operation events to target driver events corresponding to the operating system of the cloud service platform host through the event standard input interface. This achieves encapsulation and isolation of different module driver configurations and different cloud host operating systems. Standardized acquisition, encoding, transmission, and event services can be provided externally through a unified cloud service standardized subclass on cloud hosts with different configurations, reducing module coupling and code redundancy, improving the compatibility and stability of the cloud audio and video interaction system. Furthermore, when upgrading or migrating the acquisition, encoding, transmission, and event services to new hardware or host operating system environments, only the functional interfaces of the cloud service standardized subclass need to be implemented, thereby improving the maintainability of the cloud audio and video interaction system.

Attached Figure Description

Figure 1 is a schematic diagram of the cloud gaming interaction process provided in an embodiment of this application;

Figure 2 is a schematic diagram of the architecture of the current cloud gaming server platform provided in the embodiments of this application;

Figure 3 is a schematic diagram of an optional cloud audio and video processing system architecture provided in an embodiment of this application;

Figure 4 is a schematic diagram of an optional structure of the first cloud host provided in an embodiment of this application;

Figure 5 is a schematic diagram of an optional structure of the interactive device provided in an embodiment of this application;

Figure 6 is a schematic diagram of an optional structure of the second cloud host provided in an embodiment of this application;

Figure 7 is an optional flowchart of the cloud audio and video processing method provided in an embodiment of this application;

Figure 8 is a schematic diagram showing the relationship between the preset cloud service standardization base class, the cloud service standardization subclass, and the standard virtual function interface contained in the preset cloud service standardization base class provided in the embodiments of this application.

Figure 9 is an optional flowchart of the cloud audio and video processing method provided in an embodiment of this application;

Figure 10 is an optional flowchart of the cloud audio and video processing method provided in an embodiment of this application;

Figure 11 is a schematic diagram of the client standardization component provided in this application converting at least one format type of original operation instruction;

Figure 12 is an optional flowchart of the cloud audio and video processing method provided in an embodiment of this application;

Figure 13 is an optional flowchart of the cloud audio and video processing method provided in an embodiment of this application;

Figure 14 is a schematic diagram of an optional architecture of the cloud gaming system provided in an embodiment of this application.

Detailed Implementation

To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

If the application documents contain similar descriptions such as "first/second", the following explanation shall be added: In the following description, the terms "first/second/third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first/second/third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

1) Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology applied to the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

2) Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

Based on logical function, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be deployed directly on top of IaaS. PaaS is a platform for running software, such as databases and web virtual containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally speaking, SaaS and PaaS are upper layers compared to IaaS.

3) Cloud gaming: Also known as gaming on demand, it is an online gaming technology based on cloud computing. Cloud gaming technology enables thin clients with relatively limited graphics processing and data processing capabilities to run high-quality games. In cloud gaming, the game does not reside on the player's terminal but runs on a cloud server. The cloud server renders the game scene as a video and audio stream, which is then transmitted to the player's terminal via the network. The player's terminal does not need powerful graphics processing and data processing capabilities; it only needs basic streaming media playback capabilities and the ability to receive player input commands and send them to the cloud server.

4) Unreal Engine: A complete development framework for PC, Xbox 360, iOS, and PlayStation 3, providing a wealth of core technologies, content creation tools, and supporting infrastructure. The design philosophy behind all aspects of Unreal Engine is to make content creation and programming more convenient. Its goal is to give artists and game designers as much control as possible in developing resources within a visual environment, minimizing programmer assistance, while providing programmers with a highly modular, upgradeable, and scalable architecture to develop, test, and release various types of games.

5) X86: The X86 architecture is a set of computer language instructions executed by a microprocessor. It is a standard abbreviation for a series of general-purpose Intel computers and also identifies a set of general-purpose computer instructions. It is the complex instruction set of a Complex Instruction Set Computer (CISC).

6) ARM (Advanced RISC Machine): The ARM processor is the first low-power, low-cost Reduced Instruction Set Computing (RISC) microprocessor designed by Acorn Ltd. in the UK.

7) Houdini: Houdini is an ARM binary translator developed by Intel. Its principle is to translate ARM binary code into x86 instruction set for execution on x86 CPU.

8) Frames Per Second (FPS): FPS is a definition in the field of graphics, referring to the number of frames transmitted per second. Simply put, it refers to the number of frames in an animation or video. FPS measures the amount of information used to store and display dynamic video. The more frames per second, the smoother the displayed motion. It can also be understood as the "refresh rate" (measured in Hz).

9) A virtual container (also known as an emulator) is a complete computer system simulated by a computer program, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in an emulator. Emulators can include Android emulators, Java emulators, Linux emulators, Windows emulators, and so on.

10) A virtual function is a member function modified by `virtual`. It is declared as `virtual` in the base class and redefined in one or more derived classes. When a pointer to the base class operates on its polymorphic class object, it will call the appropriate function depending on the different class objects. Virtual functions can be used to implement polymorphism, which separates the interface from the implementation, achieving the same functionality but using different methods and strategies due to individual differences.

11) Real-Time Streaming Protocol (RTSP) is an application layer protocol in the TCP/IP protocol suite.

12) The Web Real-Time Communication (WebRTC) protocol is an API protocol that enables web browsers to conduct real-time voice or video conversations.

13) Real-time Transport Protocol (RTP) is a network transport protocol; Real-time Transport Control Protocol (RTCP) is its sister protocol.

14) User Datagram Protocol (UDP) provides an application with a method to send encapsulated IP packets without establishing a connection.

15) Stream Control Transmission Protocol (SCTP) is a protocol that allows multiple data streams to be transmitted simultaneously between two ends of a network connection.

16) QEMU is a set of emulated processors whose source code is distributed under the General Public License and is widely used on the GNU/Linux platform.

Currently, cloud gaming is a gaming method based on cloud computing. In the cloud gaming operating mode, all games run on the server side, and the game screen generated by the server is encoded, compressed, and transmitted to the cloud gaming client on the terminal via the network. The gaming device on the cloud gaming client does not need to be equipped with any high-end processors or graphics cards, nor does it need to actively adapt to different software and hardware platforms or have powerful terminal rendering performance; only basic audio and video decompression capabilities are required. The current interaction method of cloud gaming can be shown in Figure 1. The main process of cloud gaming runs on cloud server 10, which is usually an edge computing node with a graphics processing unit (GPU). Cloud server 10 converts the game images generated by the GPU into video streams in formats such as VP8, VP9, H.264, H.265, or AV1, and audio streams in formats such as silk, opus, and aac, and transmits the audio and video streams to various types of terminals 11 via the network, such as TVs, mobile phones, PCs, tablets, etc. The terminal can decode the received audio and video streams and render them to obtain the terminal game screen, which is then displayed to the user on the terminal's display device. Furthermore, the terminal can receive operation commands initiated by the user at the target coordinate position on the terminal game screen through operation devices such as keyboard, mouse, gamepad, and touch screen, and transmit these operation commands and coordinate positions as interactive operation information to the cloud server 10. The cloud server 10 then maps the received interactive operation information into the default game mouse buttons in the cloud game program, and then sends it to the cloud server 10 through the keyboard and mouse driver to complete the entire game service experience.

Currently, cloud gaming systems primarily provide basic service capabilities for cloud gaming servers through the PaaS layer. Based on Figure 1, the basic architecture of the current cloud gaming PaaS layer can be shown in Figure 2. The PaaS layer on the edge computing node 10 of the cloud gaming system can include an audio/video acquisition and encoding module 15, a Real-time Communications (RTC) channel 12, and an event input service 13. The audio/video acquisition and encoding module 15 includes four sub-modules: audio acquisition, audio encoding, video acquisition, and video encoding. Correspondingly, the terminal is configured with an RTC channel 22, an audio/video decoding and rendering module 25, and an event input service 23 to interact with the edge computing node for cloud gaming. The audio and video acquisition and encoding module 15, RTC channel 12, and event input service 13 in the edge computing node 10 are all integrated in the same PaaS process service. The audio and video acquisition and encoding module 15 acquires and encodes the game rendering screen of the cloud gaming process 14 to obtain audio and video streams. The audio and video streams are transmitted to the terminal 20 through the RTC channel 12, and the event input service 13 receives terminal operation instructions, such as operation instructions issued by the terminal through the mouse, keyboard, touch, etc., as terminal uplink events. The received terminal uplink events are simulated as event drivers or event instructions of the host OS system currently running on the edge computing node 10 and sent to the cloud gaming process 14 in the host OS. Because the hardware configurations of computing nodes and the host software systems are typically different combinations of solutions—for example, for PC cloud gaming servers, the host is usually a Windows system, and the hardware configuration environment is mainly a CPU (INTEL/AMD x86) + GPU (NVIDA/AMD) solution; for mobile cloud gaming servers, the host is usually an Android virtual container, and the hardware configuration environment is mainly a CPU (INTEL/AMD x86 Houdini) or ARM + GPU (NVIDA/AMD/INTEL) + Docker virtual container solution—the current PaaS layer structure has modules such as audio and video acquisition and encoding module 15, RTC channel 12, and event input service 13, which are designed for different operating systems and hardware configurations. The coupling between these modules is very high. Furthermore, due to the different hardware or different host OS operating environments and driver-based acquisition, encoding, transmission, and event underlying APIs, the current cloud gaming PaaS layer has the following defects:

1. PaaS platform upgrade costs are high: When the hardware configuration of the server and the host OS environment change, the previous PaaS layer code can hardly be reused, and most of the code related to the hardware and host OS environment needs to be rewritten.

2. Poor stability of PaaS platform: The entire platform architecture code is strongly correlated with hardware configuration and host OS. As long as the hardware configuration and host OS change, the entire upstream and downstream end-to-end link module adaptation and rewritten code must be retested and adapted.

3. High PaaS platform operation and maintenance costs: Different hardware configurations and host OS environment code result in different deployments of platform code and architecture for different hardware and host OS configurations, leading to an exponential increase in operation and maintenance costs.

It is evident that the current implementation of the PaaS service platform in cloud gaming leads to code redundancy and chaos in PaaS services under different environments. If the hardware configuration is upgraded or adjusted, or the host OS changes, the entire PaaS service code must be adapted to the new hardware drivers and host OS-related API interfaces. This makes it difficult for the PaaS layer to be upgraded and migrated to new hardware and OS environments, resulting in high upgrade costs for the entire cloud gaming backend service and poor operation, maintenance, and platform stability.

This application provides a cloud audio and video processing method, system, electronic device, and storage medium, which can improve the compatibility, stability, and maintainability of the cloud audio and video interaction system. The exemplary application of the electronic device provided in this application is described below.

Referring to Figure 3, which is an optional architecture diagram of the cloud audio and video processing system 100 provided in an embodiment of this application, the cloud audio and video processing system 100 includes: a first cloud host 200, a second cloud host 500, and interactive devices 400 (interactive devices 400-1 and 400-2 are shown as examples). The interactive device 400 is connected to the second cloud host 500 via a network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. In some embodiments, the first cloud host 200 can run on an edge computing node of a cloud network. The first cloud host 200 and the second cloud host 500 are interconnected, and the interactive device 400 is connected to the first cloud host 200 through the second cloud host 500. The second cloud host 500 is equipped with a preset uplink standard interface 560 and a preset downlink standard interface 570; the interactive device 400 is equipped with a client standardization component 460 (client standardization component 460-1 and client standardization component 460-2 are shown as examples); the first cloud host 200 runs a cloud audio and video process 200-2; the cloud audio and video process 200-2 can be the main process of a cloud audio and video application, such as the main process of a cloud game. The first cloud host 200 is configured with a capture encoding event framework 200-1, which includes a capture module 260, an encoding module 270, and an event processing module 280, as well as an audio and video capture standard interface 261, an encoding transmission standard interface 271, and an event standard input interface 281. Among them, the audio and video acquisition standard interface 261, the encoding and transmission standard interface 271, and the event standard input interface 281 are standardized interfaces implemented in the cloud service standard subclasses derived from the preset cloud service standard base class, respectively based on the configuration information of the acquisition module 260, the encoding module 270, and the operating system.

As shown in Figure 3, the client standardization component 460-1 is connected to the event standard input interface 281 on the first cloud host 200 through the preset uplink standard interface 560 on the second cloud host 500, and the client standardization component 460-1 is connected to the encoding transmission standard interface 271 on the first cloud host 200 through the preset downlink standard interface 570 on the second cloud host 500; wherein...

The first cloud host 200 can be the cloud service platform host in the cloud audio and video interaction system 100. It is used to call the acquisition module 260 through the audio and video acquisition standard interface 261 to acquire the audio and video data generated by the cloud audio and video process 200-2; and to call the encoding module 270 through the encoding transmission standard interface 271 to encode the audio and video data in a preset standard encoding format to obtain an encoded audio and video stream, and transmit the encoded audio and video stream to the preset downlink standard interface 570.

The second cloud host 500 is used to transmit encoded audio and video streams to the client standardized components through the preset downlink standard interface 570 using the preset standard transmission format.

Interactive device 400, belonging to the end user of cloud audio and video processing system 100, such as a cloud game player, is used to decode and play the encoded audio and video stream through client standardized component 460 to obtain decoded video and audio, which are displayed on graphical interface 470 (graphical interfaces 470-1 and 470-2 are shown for example); it obtains operation events initiated for the decoded video or decoded audio and sends the operation events to the preset uplink standard interface 560;

The second cloud host 500 is also used to transmit operation events to the event standard input interface through the preset uplink standard interface 560 using the preset standard transmission format.

The first cloud host 200 is also used to map operation events to target-driven events corresponding to the current operating system through the event standard input interface 281; the target-driven events are used to drive the cloud audio and video process 200-2 to update the corresponding audio and video data.

In some embodiments, the first cloud host 200 and the second cloud host 500 can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The interactive device 400 can be various types of user terminals such as smartphones, tablets, laptops, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), but is not limited to these. The interactive device 400 and the second cloud host 500, as well as the first cloud host 200 and the second cloud host 500, can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

In the cloud audio and video processing method or apparatus disclosed in this application, multiple cloud hosts or servers can be combined to form a blockchain, and the server is a node on the blockchain.

Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying platform, a platform product service layer, and an application service layer.

The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operational monitoring. The user management module is responsible for managing the identity information of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the correspondence between user real identities and blockchain addresses (access management). Furthermore, under authorization, it monitors and audits transactions of certain real identities and provides risk control rule configuration (risk control audit). The basic services module is deployed on all blockchain node devices to verify the validity of business requests. After consensus is reached on valid requests, they are recorded in storage. For a new business request, the basic services first perform interface adaptation parsing and authentication (interface adaptation), and then encrypt the business information through a consensus algorithm (consensus management). After encryption, the data is transmitted completely and consistently to the shared ledger (network communication) and recorded and stored. The smart contract module is responsible for contract registration, issuance, triggering, and execution. Developers can define contract logic using a programming language and publish it to the blockchain (contract registration). According to the contract terms, the key or other events are invoked to trigger execution and complete the contract logic. It also provides functions for contract upgrades and cancellations. The operation monitoring module is mainly responsible for deployment, configuration modification, contract settings, cloud adaptation, and real-time status visualization output during product release, such as alarms, monitoring network conditions, and monitoring the health status of node devices.

The platform product service layer provides basic capabilities and implementation frameworks for typical applications. Developers can leverage these basic capabilities, add business characteristics, and complete the blockchain implementation of business logic. The application service layer provides application services based on blockchain solutions for use by business participants. In some embodiments, when implementing the cloud audio and video processing system disclosed in this application in the form of a blockchain network, the platform product service layer can be mounted and run based on the first cloud host 200 and the second cloud host 500.

Referring to Figure 4, which is a schematic diagram of the structure of a first cloud host 200 provided in an embodiment of this application, the first cloud host 200 shown in Figure 4 includes: at least one first processor 210, a first memory 250, at least one first network interface 220, and a first user interface 230. The various components in the first cloud host 200 are coupled together through a first bus system 240. It is understood that the first bus system 240 is used to realize the connection and communication between these components. In addition to a data bus, the first bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as the first bus system 240 in Figure 4.

The first processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

The first user interface 230 includes one or more first output devices 231 that enable the presentation of media content, including one or more speakers and/or one or more visual displays. The first user interface 230 also includes one or more first input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

The first memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The first memory 250 may optionally include one or more storage devices physically located remote from the first processor 210.

The first memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The first memory 250 described in this application embodiment is intended to include any suitable type of memory.

In some embodiments, the first memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

The first operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

The first network communication module 252 is used to reach other computing devices via one or more (wired or wireless) first network interfaces 220, exemplary network interfaces 220 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

The first presentation module 253 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more first output devices 231 (e.g., a display screen, a speaker, etc.) associated with the first user interface 230.

The first input processing module 254 is configured to detect and translate one or more user inputs or interactions from one or more first input devices 232.

In some embodiments, the apparatus provided in this application can be implemented in software. FIG4 shows a first cloud audio and video processing apparatus 255 stored in a first memory 250, which can be software in the form of programs and plug-ins, including the following software modules: acquisition module 260, encoding module 270, and event processing module 280. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented.

The functions of each module will be explained below.

In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the cloud audio and video processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

Referring to Figure 5, which is a schematic diagram of the structure of the interactive device 400 provided in an embodiment of this application, Figure 5 shows a second cloud audio and video processing device 455 stored in a second memory 450. This device can be software in the form of programs and plugins, and includes the following software modules: a client standardization component 460. These modules are logically integrated and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below. The functions of at least one second processor 410, the second memory 450, at least one second network interface 420, and the second user interface 430 in Figure 5 are consistent with the functions of the same modules in Figure 4, and will not be repeated here.

Referring to Figure 6, which is a schematic diagram of the structure of the second cloud host 500 provided in an embodiment of this application, Figure 6 shows the third cloud audio and video processing device 555 stored in the third memory 550. This device can be software in the form of programs and plug-ins, including the following software modules: a preset downlink standard interface 570 and a preset uplink standard interface 560. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below. The functions of at least one third processor 510, the third memory 550, at least one third network interface 520, and the third user interface 530 in Figure 6 are consistent with the functions of the same modules in Figure 4, and will not be repeated here.

The cloud audio and video processing method provided in this application will be described by referring to the exemplary applications and implementations of the first cloud host, interactive device and second cloud host provided in the embodiments of this application.

Referring to Figure 7, which is an optional flowchart of the cloud audio and video processing method provided in this application embodiment applied to a first cloud host and an interactive device, the steps shown in Figure 7 will be described in conjunction with the steps shown in Figure 7.

S101, the first cloud host calls the acquisition module through the audio and video acquisition standard interface to acquire audio and video data generated by the cloud audio and video process.

The cloud audio and video processing method provided in this application is mainly applied to cloud audio and video interaction scenarios, such as cloud gaming, cloud live streaming, and virtual engine application scenarios.

In this embodiment, the cloud audio/video process can be the main process of cloud audio/video applications such as cloud gaming and cloud live streaming. These applications are pre-installed and run on a cloud server hosting the audio/video application. In some embodiments, the cloud server can be a game application cloud server, which may run one or more virtual devices. For example, the virtual device can be a virtual machine, and the cloud audio/video process can run within it. The virtual machine's memory may contain game resources, such as image resources, character resources, and audio resources. During operation, the cloud gaming process can call game resources to render relevant game data and generate corresponding game audio, based on the current program's driving logic. The first cloud host can use the standard audio/video capture interface to call its own capture module to obtain the game screen and game audio from the cloud audio/video process as audio/video data.

In this embodiment, the driver interfaces and software development kit (SDK) interfaces provided by the hardware configuration and operating system of the acquisition modules on different cloud hosts are different. The standard audio and video acquisition interface in this embodiment can be implemented by specifically calling the driver interface and SDK interface of the acquisition module configured on the first cloud host within the standard cloud service subclass, in the standard audio and video acquisition virtual function interface. The standard cloud service subclass inherits from a preset standard cloud service base class, which predefines the standard audio and video acquisition virtual function interface. The first cloud host can implement its own corresponding standard cloud service subclass according to its hardware and operating system configuration to be applied on different cloud hosts. Here, the first cloud host implements the standard audio and video acquisition virtual function interface in the standard cloud service subclass according to the driver interface and SDK interface corresponding to its own acquisition module. This allows for standardized acquisition of audio and video data externally through a unified standard audio and video acquisition virtual function interface name, while internally achieving compatibility shielding for acquisition modules with different hardware and software configurations.

In this embodiment, the first cloud host can access the GPU graphics interface on the audio/video application cloud server through the acquisition module to obtain the current video frame from the GPU graphics interface. The first cloud host can also access the audio interface, such as a sound card, on the audio/video application cloud server through the acquisition module to obtain the current audio from the audio interface. The first cloud host uses the current video frame and the current audio as audio/video data. In some embodiments, the GPU graphics interface may include DirectX 9/10/11/12 and OpenGL interfaces, and the audio interface may be the Windows Audio Session API. The specific selection depends on the actual situation and is not limited in this embodiment.

In some embodiments, the preset cloud service standardization base class can predefine a preset sampling frame rate. The first cloud host can periodically call the local acquisition driver interface of the acquisition module through the audio and video acquisition standard interface to acquire the current video frame and current audio generated by the cloud audio and video process, and obtain audio and video data according to the preset sampling frame rate; the preset sampling frame rate is the standardized sampling frame rate uniformly set in the preset cloud service standardization base class.

In some embodiments, the cloud audio and video process can be deployed to run on the same host machine or on other cloud hosts. The specific choice depends on the actual situation, and this application embodiment does not limit it.

S102. The first cloud host calls the encoding module through the encoding transmission standard interface to encode the audio and video data in a preset standard encoding format, obtain the encoded audio and video stream, and transmit the encoded audio and video stream to the interactive device.

In this embodiment, when the first cloud host collects audio and video data, it can invoke its configured encoding module through an encoding output standard interface. In some embodiments, the encoding module may include a video encoder and an audio encoder. The first cloud host can redirect the current video frame in the audio and video data to the video encoder, which encodes the current video frame in a preset standard video format; and it can take over the current audio in the audio and video data through the audio encoder, which encodes the current audio in a preset standard audio format; wherein, the preset standard encoding format includes at least one of a preset video standard format and a preset standard audio format.

In some embodiments, the preset video standard format in the preset standard encoding format may include H.264 and H.265 formats, and the preset standard audio format in the preset standard encoding format may include AAC and Opus. The specific selection is made according to the actual situation. This application embodiment does not limit the specific encoding format type and number of preset standard encoding formats.

In this embodiment, first cloud hosts with different hardware and software configurations encode audio and video data using a preset standard encoding format to obtain a unified encoded audio and video stream in the preset standard encoding format. The first cloud host then transmits the encoded audio and video stream to the interactive device.

In some embodiments, the acquisition module may be the graphics card on the first cloud host, and the interaction device may be a cloud gaming terminal device, such as a computer, mobile phone, game console, tablet computer, etc.

In this embodiment, the driver interfaces and SDK interfaces provided by the hardware configuration and operating system differ for the encoding modules on different cloud hosts. The standard encoding transmission interface in this embodiment can be implemented by calling the driver interface and SDK interface of the encoding module configured on the first cloud host within the internal implementation of the standard encoding output virtual function interface, based on the corresponding cloud service standardization subclass on the first cloud host. Here, the first cloud host implements the standard encoding output virtual function interface in its corresponding cloud service standardization subclass according to the driver interface and SDK interface of its own encoding module. This internally achieves compatibility shielding for encoding modules with different hardware and software configurations, and externally achieves standardized encoding output of audio and video data through the unified name of the standard encoding output virtual function interface.

In some embodiments, the preset cloud service standardization base class may predefine a preset encoding frame rate. The first cloud host can call the local encoding driver interface of the encoding module through the encoding transmission standard interface to encode the audio and video data according to the preset encoding frame rate and the preset standard encoding format to obtain the encoded audio and video stream; the preset encoding frame rate is the standardized encoding frame rate uniformly set in the preset cloud service standardization base class.

S201. The interactive device receives the encoded audio and video streams, decodes and plays the encoded audio and video streams to obtain the decoded video images and decoded audio.

In this embodiment of the application, when the interactive device receives the encoded audio and video stream transmitted by the first cloud host, it can decode and render the encoded audio and video stream to obtain the decoded video image and decoded audio of the cloud audio and video, and present them to the operator of the interactive device, such as a cloud gaming user.

S202. The interactive device acquires operation events initiated for the decoded video or decoded audio and sends the operation events to the event standard input interface of the first cloud host.

In this embodiment, the interactive device acquires operation events initiated by the operator on the decoded video or decoded audio, such as events generated by clicking, pressing, or swiping virtual characters, game scenes, or game options. The interactive device then sends these operation events to the standard event input interface of the first cloud host.

S103, the first cloud host receives operation events initiated by the interactive device based on the encoded audio and video stream through the event standard input interface, and maps the operation events to the target driver events corresponding to the current operating system; the target driver events are used to drive the cloud audio and video process to update the corresponding audio and video data; among which...

The audio and video acquisition standard interface, encoding and transmission standard interface, and event standard input interface are standardized interfaces implemented in the cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, encoding module, and operating system, respectively.

In this embodiment of the application, the first cloud host can receive operation events initiated by the interactive device based on the encoded audio and video stream, and map the operation events to target-driven events corresponding to its own operating system through the event standard input interface.

In this embodiment, the program-driven logic in the cloud audio and video process is typically implemented based on a preset default device type's corresponding operation method. For example, cloud game developers who create cloud games using mobile phones as the default device type usually use mobile phone-type operation methods such as screen clicks or swipes to implement the program execution logic corresponding to the driving events, thereby controlling the operation of the cloud game process. Since cloud gaming and other cloud audio and video applications target various types of interactive devices, it is necessary to convert the operation events initiated by interactive devices of different device types into standard driving events corresponding to the default device type of the cloud audio and video process, so as to achieve driving control of cloud audio and video on different interactive devices.

In some embodiments, the first cloud host can invoke a preset instruction conversion service through the event standard input interface to determine the target driving event corresponding to the operation event from a preset correspondence between at least one preset operation event and at least one standard driving event; and then convert the operation event into the target driving event. The at least one standard driving event is a standard driving event in the current operating system. The first cloud host can continue to drive the cloud audio/video process through the target driving event, so that the cloud audio/video process updates the current video frame and current audio according to the instructions of the target driving event. The first cloud host can continuously collect and encode the updated audio and video data to obtain an updated encoded audio/video stream, and then transmit the updated encoded audio/video stream to the interactive device to achieve continuous interaction with the cloud audio/video process.

In this embodiment, the standard event input interface is obtained by specifically implementing the standard event input virtual function interface in the standardized subclass of cloud services, based on the different hardware on different cloud service host machines and the driver and SDK interfaces provided by the operating system. Internally, the standard event input interface implements interface calls to the hardware drivers and operating system on the current cloud service host machine, achieving compatibility shielding for operating system event drivers with different hardware and software configurations; externally, it provides a unified standard event input virtual function interface name, achieving standardized output of operating system driver events.

In some embodiments, when the first cloud host determines the target driving event, it can input the target driving event into the running cloud audio and video process, so as to trigger the corresponding target program driving logic in the cloud audio and video process through the target driving event, and output the corresponding updated video screen and updated audio as updated audio and video data by executing the target program driving logic, thereby realizing the response to the operation event.

In some embodiments, the relationship between the preset cloud service standardization base class, the cloud service standardization subclasses, and the standard virtual function interfaces contained in the preset cloud service standardization base class can be as shown in Figure 8. The preset cloud service standardization base class may contain standard audio/video acquisition virtual function interfaces, standard encoding output virtual function interfaces, standard event handling virtual function interfaces, and other custom standard virtual function interfaces. On first cloud hosts with different hardware and software configurations, unified standard audio/video acquisition virtual function interfaces, standard encoding output virtual function interfaces, standard event handling virtual function interfaces, and other custom standard virtual function interfaces can be obtained by inheriting from the preset cloud service standardization base class. Furthermore, different cloud service standardization subclasses can be implemented on different first cloud hosts based on their different hardware and software configurations. As shown in Figure 8, the same virtual function interface name can have different implementation methods in different cloud service standardization subclasses, allowing for the invocation of acquisition modules, encoding modules, operating systems, and other modules with different hardware and software configurations on different first cloud hosts through a unified virtual function interface.

It is understood that in this embodiment, by implementing standardized interfaces based on the configuration information of the acquisition module, encoding module, and operating system on the first cloud host in the cloud service standardized subclass derived from the preset cloud service standardized base class, it is possible to call the acquisition module configured on the first cloud host to acquire audio and video data through the audio and video acquisition standard interface; call the encoding module configured on the first cloud host to perform standardized audio and video encoding and output through the encoding and transmission standard interface; and map operation events to the target driver events corresponding to the operating system of the first cloud host through the event standard input interface. This achieves encapsulation and isolation of different module driver configurations and different cloud host operating systems. Standardized acquisition, encoding, transmission, and event services can be provided externally through a unified cloud service standardized subclass on cloud hosts with different configurations. This reduces module coupling and code redundancy, improves the compatibility and stability of the cloud audio and video interaction system, and when upgrading or migrating the acquisition, encoding, transmission, and event services to new hardware or host OS environments, only the functional interfaces of the cloud service standardized subclass need to be implemented, thereby improving the maintainability of the cloud audio and video interaction system.

In some embodiments, referring to FIG9, FIG9 is an optional flowchart of the cloud audio and video processing method provided in the embodiments of this application. The first cloud host also includes initializing a standard interface. Before S101, S001 can also be executed, which will be described in conjunction with each step.

S001. The first cloud host initializes the acquisition module, encoding module and operating system by calling the corresponding local initialization interfaces of the acquisition module, encoding module and operating system through the initialization standard interface. The initialization standard interface is a standardized interface implemented in the cloud service standardized subclass.

In this embodiment of the application, the preset cloud service standard base class may also include an initialization standard interface. For cloud service standard subclasses corresponding to different first cloud hosts, the inherited initialization standard interface can be specifically implemented based on the driver configuration information corresponding to the acquisition module and the encoding module, such as the acquisition and encoding drivers and SDK initialization interfaces provided by different hardware/host OS, so as to realize the initialization of the acquisition module and the encoding module through the initialization standard interface.

In some embodiments, a client standardization component is deployed on the interactive device. The client standardization component can be connected to the encoding transmission standard interface on the first cloud host, and the client standardization component can be connected to the event standard input interface on the first cloud host. Referring to Figure 10, Figure 10 is an optional flowchart of the cloud audio and video processing method provided in the embodiments of this application. S201 can be implemented by executing the process of S2011-S2012, and S202 can be implemented by executing the process of S2021-S2022. The steps shown in Figure 10 will be described in conjunction with the steps shown in Figure 10.

S2011. The interactive device receives the encoded audio and video streams output by the first cloud host through the encoded transmission standard interface via the client-side standardized components.

In this embodiment, the client standard component is a functional component provided by the cloud audio and video processing system of this embodiment to different types of interactive devices such as televisions, mobile phones, and computers, containing a unified standardized client service. The interactive device can receive encoded audio and video streams in a preset standard encoding format output by the first cloud host through the encoding transmission standard interface via the client standard component.

In some embodiments, the client standard component may be in the form of an SDK. In some embodiments, the client standard component may also be integrated on the first cloud host or other network nodes in the cloud audio and video interaction system, depending on the actual situation, and this application embodiment does not limit the choice.

S2012. The interactive device uses standardized client components to decode and play the encoded audio and video streams in a preset decoding format to obtain decoded video images and decoded audio.

In this embodiment, the client-side standardization component may include an audio/video processing standard service. This service is used to decode and play encoded audio/video streams using a preset decoding format. The interactive device can use the audio/video processing standard service in the client-side standardization component to decode and play the encoded audio/video streams using the preset decoding format, obtaining decoded video footage and decoded audio.

S2021. The interactive device obtains at least one type of original operation command for decoding video or audio through the client-side standardized component, and performs format conversion on the original operation command for at least one type of original operation command according to the preset standard event format to obtain the operation event.

In this embodiment, the client-side standardization component may include an uplink event processing standard service. This uplink event processing standard service is used to acquire at least one type of raw operation command from the interactive device, and to convert the raw operation command of at least one type of raw operation command according to a preset standard event format to obtain an operation event.

In this embodiment, the hardware configuration and software driver configuration information of the user interface components on different interactive devices are different. For example, different models and brands of game controllers will issue different formats of raw operation commands when performing the same button operation, such as pressing the up button. For instance, the API interfaces supported by the game controller may include DirectInput, XInput, RawInput, etc., and the format of raw operation commands output by different types of API interfaces is also different. The uplink event processing standard service can convert raw operation commands of at least one format type into a unified standard format operation event according to a preset standard event format, thereby unifying the raw operation commands issued by interactive devices of the same type but different hardware and software configurations, and sending the operation events to the first cloud host.

Here, the user interface component can be an integrated hardware or software component within the interactive device, such as a mobile phone touchscreen, or an extended device connected to the interactive device via a hardware or software extension interface, such as a mouse, keyboard, or game controller connected to a PC terminal, or a microphone, camera, or other component that can issue raw operation commands through sound or gestures. The specific selection depends on the actual situation, and this application embodiment does not limit it.

In some embodiments, as shown in Figure 11, different user interface components such as mouse 1, mouse 2, mouse 3 and touch screen 1, touch screen 2, and touch screen 3 issue different types of raw operation commands when performing click operations, such as mouse 1 click command, mouse 2 click command, etc. The interactive device can convert the different types of mouse click commands into standard mouse click events as operation events through the client standardization component; and convert the different types of touch screen click commands into standard touch screen click events as operation events.

S2022, The event standard input interface through which the interactive device sends operation events to the first cloud host.

In this embodiment, the client-side standardization component transmits the converted operation events in a preset standard event format to the event standard input interface, so as to map the operation events into target-driven events and continue subsequent cloud audio and video processing through the event standard input interface.

It is understood that, in this embodiment of the application, the client-side standardized component can be used to compatiblely shield operation commands initiated by interactive devices with different hardware and software configurations, and convert at least one type of original operation command into an operation event of a preset standard event format and transmit it to the first cloud host for processing. Furthermore, the client-side standardized component can shield the differences in hardware and software media playback devices on different interactive devices, providing a unified audio and video decoding and playback service. This is beneficial for troubleshooting, upgrading, maintenance, and migration of the cloud audio and video interactive system, reduces the risk of instability caused by code redundancy, and further improves the compatibility, stability, and maintainability of the cloud audio and video interactive system.

Referring to Figure 12, which is an optional flowchart of the cloud audio and video processing method provided in this application embodiment applied to a second cloud host, the steps shown in Figure 12 will be described in conjunction with the steps shown in Figure 12.

S301, the second cloud host obtains the encoded audio and video stream through the preset downlink standard interface; the encoded audio and video stream is obtained by the first cloud host calling the acquisition module through the audio and video acquisition standard interface to acquire cloud audio and video data generated by the cloud audio and video process; and by calling the encoding module through the encoding transmission standard interface to encode the audio and video data in the preset standard encoding format; the preset downlink standard interface is connected to the encoding transmission standard interface on the first cloud host and the interactive device respectively.

In this embodiment, a standard data transmission service can be pre-integrated between the preset uplink standard interface and the preset downlink standard interface. This standard data transmission service uses a preset standard transmission format to encapsulate and transmit data transmitted between the preset uplink standard interface and the preset downlink standard interface. In some embodiments, the preset uplink standard interface and the preset downlink standard interface can be deployed on a second cloud host, or on a first cloud host or other cloud hosts, depending on the actual situation. This embodiment does not impose any limitations on this.

In this embodiment, a preset downlink standard interface is connected to both the encoding transmission standard interface on the first cloud host and the interactive device, for transmitting data streams from the first cloud host to the interactive device. The second cloud host can obtain encoded audio and video streams from the encoding transmission standard interface of the first cloud host through the preset downlink standard interface. The encoded audio and video streams are obtained by the first cloud host calling the acquisition module through the audio and video acquisition standard interface to acquire cloud audio and video data generated by the cloud audio and video process; and then calling the encoding module through the encoding transmission standard interface to encode the audio and video data using a preset standard encoding format.

S302, the second cloud host uses a preset standard transmission format to transmit the encoded audio and video streams to the interactive device.

In this embodiment of the application, the second cloud host can utilize standard data transmission services and use a preset standard transmission format to encapsulate and transmit the encoded audio and video streams in the corresponding transmission format, thereby transmitting the encoded audio and video streams to the interactive device.

In some embodiments, the preset standard transmission format includes at least one of the following: a preset application layer standard protocol format, a preset transport layer standard protocol format, and a preset network layer standard protocol format; wherein, the preset application layer standard protocol format includes any one of WebRTC or RTSP; the preset transport layer standard protocol format includes any one of RTP or RTCP; and the preset network layer standard protocol format includes any one of UDP or SCTP.

S303, the second cloud host receives operation events initiated by the interactive device based on encoded audio and video streams through a preset uplink standard interface; the preset uplink standard interface is connected to the event standard input interface on the first cloud host and the interactive device respectively;

In this embodiment, a preset uplink standard interface is connected to both the event standard input interface on the first cloud host and the interactive device, for transmitting data streams from the interactive device to the first cloud host. The second cloud host can receive operation events initiated by the interactive device based on encoded audio and video streams through the preset downlink standard interface.

S304. The second cloud host uses a preset standard transmission format to transmit operation events to the event standard input interface.

In this embodiment, the second cloud host can encapsulate and transmit the content data of the operation event in a corresponding transmission format according to a preset standard transmission format through a standard data transmission service, and then output the operation event to the event standard input interface of the first cloud host through a preset uplink standard interface.

It is understood that, in the embodiments of this application, by standardizing the transmission of operation events and encoded audio and video streams transmitted in the cloud audio and video interaction system, the architecture of the cloud audio and video interaction system is further decoupled, thereby further improving the compatibility, stability and maintainability of the cloud audio and video interaction system.

In some embodiments, the interactive device includes: a client standardization component, which is connected to a preset downlink standard interface and a preset uplink standard interface respectively. Referring to Figure 13, Figure 13 is an optional flowchart of a cloud audio and video processing method provided in an embodiment of this application. Based on Figure 12, S302 can be implemented by executing the process of S3021, and S303 can be implemented by executing the process of S3031. The steps will be described in conjunction with each step.

S3021 The second cloud host transmits the encoded audio and video streams to the client standardization component on the interactive device through a preset downlink standard interface and a preset standard transmission format, so that the client standardization component can decode and play the encoded audio and video streams in a preset decoding format to obtain decoded video images and decoded audio.

In this embodiment, the preset downlink standard interface may include a preset downlink standard ingress point and a preset downlink standard egress point. The preset downlink standard ingress point is connected to the encoding transmission standard interface in the first cloud host, and the preset downlink standard egress point is connected to the client standardization component.

In this embodiment, the encoding module on the first cloud host can transmit encoded audio and video streams in a preset standard encoding format to the preset downlink standard entry point on the second cloud host through the encoding transmission standard interface.

In this embodiment of the application, when the second cloud host receives the encoded audio and video stream through the preset downlink standard inlet, it can use the preset standard transmission format, encapsulate and transmit the encoded audio and video stream in the corresponding transmission format through the encoding transmission standard interface, and output it to the client standardization component on the interactive device through the preset downlink standard outlet.

S3031. The second cloud host receives operation events sent by the client standardized component on the interactive device through a preset uplink standard interface. The operation event is obtained by the client standardized component by acquiring at least one type of original operation instruction for decoding video or decoding audio, and converting the original operation instruction of at least one type of original operation instruction according to the preset standard event format.

In this embodiment, the preset uplink standard interface may include a preset uplink standard ingress point and a preset uplink standard egress point. The preset uplink standard ingress point can be connected to a client-side standardized component, and the preset uplink standard egress point can be connected to an event standard input interface in the first cloud host.

In this embodiment, the client-side standardized component can transmit operation events to a preset uplink standard input on the second cloud host via a network connection. When the second cloud host receives the operation event through the preset uplink standard input, it can use a preset standard transmission format to encapsulate and transmit the operation event in the corresponding transmission format through an encoded transmission standard interface, and output it to the event standard input interface in the first cloud host through the preset uplink standard output.

In some embodiments, the standardized transport service may include at least one of the following: uplink and downlink network transmission bandwidth assessment service, congestion control service, connection maintenance and abnormal reconnection service.

In some embodiments, the event standard input interface is connected to the preset uplink standard interface through a first socket; the encoded transmission standard interface is connected to the preset downlink standard interface through a second socket, thereby making full use of edge computing nodes and CDN proximity access services for flexible deployment according to network conditions, and solving the last-mile network problem of proximity access.

In some embodiments, the first socket and the second socket can be the same or corresponding sockets. The specific choice depends on the actual situation, and this application does not limit the choice.

It is understood that, in this embodiment of the application, by decoupling the acquisition, encoding, transmission, and events in the cloud audio and video interaction system, the encapsulation and isolation of different module driver configurations and different cloud host operating systems are achieved. This allows standardized acquisition, encoding, transmission, and event services to be provided externally through a unified cloud service standardized subclass on cloud hosts with different configurations, reducing code redundancy and improving the compatibility and stability of the cloud audio and video interaction system. Furthermore, when upgrading or migrating the acquisition, encoding, transmission, and event services to new hardware or host OS environments, only the functional interfaces of the cloud service standardized subclass need to be implemented, thereby improving the maintainability of the cloud audio and video interaction system.

The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

This application provides an implementation architecture for a cloud gaming system, as shown in Figure 14. The cloud gaming system includes a cloud gaming terminal 600, a platform access layer 700, and a cloud service platform host 800. The cloud gaming terminal 600 is equivalent to an interactive device 400, which can be any cloud gaming terminal device such as a mobile phone, computer, or television. The cloud gaming terminal 600 is configured with a cloud gaming software toolkit (cloud gaming SDK) 610 and a media player 620, whereby the cloud gaming SDK 610 is equivalent to a client-side standardized component 460. The platform access layer 700 is equivalent to a second host 500, and the cloud service platform host 800 is equivalent to a first cloud host 200. The cloud gaming host 800 consists of a hardware layer 810, a virtual machine layer 820, an operating system layer 830, and a standardized framework for acquisition and encoding events 840. The hardware layer 810 includes hardware components on the host, such as a CPU, GPU, I/O memory, and other hardware. In some embodiments, the CPU model can be x86 or ARM, and the GPU model can be NVIDIA, AMD, or Intel. The virtual machine layer 820 is a virtual machine environment running on cloud service platform host machines with different hardware configurations. In some embodiments, the virtual machine layer 820 can be a KVM, Xen, VMware, or Hyper-V version virtual machine environment. An operating system layer 830 is mounted on the virtual machine layer 820. The operating system layer 830 can be a Linux operating system or a Windows operating system, in which an Android virtual container based on QEMU or Houdini, or a Windows system virtual container, can run. In this embodiment, the acquisition encoding event standardization framework 840 defines a base class for standard acquisition encoding and event input. Cloud service platform host machines with different hardware and software configurations can inherit from the base class and implement different virtual function interfaces in different standardized subclasses (such as subclass 1 corresponding to Intel processor + Android container, subclass 2 corresponding to ARM processor + NVIDIA graphics card + Android container, etc.). This ensures internal compatibility by shielding different acquisition module drivers, encoding module drivers, and operation events at the host's underlying level, and provides a unified standard acquisition encoding stream output and event input parsing service to the platform access layer 700 and cloud gaming terminal 600. In some embodiments, the standardized subclass can implement the following standard interfaces to provide a unified standard service externally:

1. Initialize the standard interface to shield the acquisition and encoding drivers and SDK initialization interfaces provided by different hardware/host OS.

2. Audio and video capture standard interface, used to capture screen content periodically based on the FPS timer set according to the base class, ensuring compatibility with shielding.

3. Encoded transmission standard interface, used to shield compatibility with standard audio and video streams that are encoded and output at timed intervals according to the base class and then output downlink to the access service.

4. The standard event input interface is used to input relevant events into the game process according to the requirements of the drivers and SDK interfaces provided by different hardware and OS host machines. For example, it maps the received mouse operation commands to the corresponding game mouse buttons, and then sends them to the actual game server through the keyboard and mouse driver to complete the entire game service experience.

As shown in Figure 14, the cloud service platform host 800 and platform access layer 700 in this embodiment have completed the construction of operating system, database, middleware, runtime library, etc., and realized the hardware and software function configuration of cloud game server-side related hardware devices, storage, transmission, network bandwidth, security, game updates, etc. The cloud game terminal 600 only needs to integrate or install the cloud game SDK 610, and can interact with the cloud game server-side for audio and video data and operation commands by calling the API in the cloud game SDK 610, thereby running cloud game-related functions on the cloud game terminal 600.

In this embodiment, the platform access layer 700 and the cloud service platform host 800 are independent in software architecture. The platform access layer 700 and the cloud service platform host 800 are separate, and the standardized event acquisition and encoding framework 840 on both the platform access layer 700 and the cloud service platform host 800 uses socket communication. Therefore, the platform access layer 700 can be deployed on the same machine as the cloud service platform host 800, or it can be deployed on different hosts. Specifically, it can be flexibly deployed based on the network conditions of the game users, and this embodiment does not impose any limitations. The platform access layer 700 externally connects to the cloud gaming SDK 610 and internally connects to the standardized event acquisition and encoding framework 840 for standardizing the input of operation events and the output of encoded audio and video streams. The platform access layer 700 provides unified event and stream transmission services for both the cloud gaming SDK 610 and the cloud service platform host 800. In some embodiments, the application layer transport and control protocols in the streaming service can be unified as WebRTC/RTSP, the uplink and downlink transport application layer protocols can be unified as RTP/RTCP, and the network transport layer protocols can be unified as UDP/SCTP. The platform access layer can fully utilize edge computing nodes and content delivery networks (CDNs) for proximity access services, solving the last-mile network problem for cloud gaming users, and can provide unified standard services such as uplink and downlink network transmission bandwidth assessment, congestion control, connection maintenance, and abnormal reconnection for cloud gaming users.

In this embodiment, the cloud gaming system provides a unified cloud gaming SDK 610 to game terminals and applications of different software and hardware types, such as mobile games, PC games, and cloud-rendered apps. In some embodiments, the types of game terminals that the cloud gaming SDK 610 can be compatible with may include Android/iOS/PC/Web/H5/TV/PSP/Xbox/Switch, etc. In some embodiments, the cloud gaming SDK 610 may be a software development kit that has integrated standard WebRTC access, audio and video decoding, and uplink event processing methods for mouse, keyboard, etc. The Cloud Gaming SDK 610 can receive control commands in various formats input by users through different models and types of control devices, such as different models of keyboards and game controllers. It converts these different control commands into keyboard, game controller, and touchscreen events in a preset standard event format. Furthermore, the Cloud Gaming SDK 610 implements standard audio and video format input/output interfaces. Through these interfaces, it receives encoded audio and video streams in standard audio and video formats from the platform access layer and sends uplink audio and video data (such as voice input commands and video input commands) in standard audio and video formats to the platform access layer. In some embodiments, the preset uplink and downlink standard audio and video formats in the Cloud Gaming SDK 610 can be H.264/H.265/AAC/Opus, etc. The preset standard event format can be customized, and the specific selection depends on the actual situation; this application embodiment does not limit this.

It is understood that the cloud gaming service platform provided in this application embodiment, when applied to cloud gaming scenarios, can achieve rapid iteration on different hardware configurations (such as CPU: X86/ARM + GPU: NVDIA/ARM/INTEL, etc.) and different host OS (such as Windows/Android virtual containers, etc.) for cloud mobile games and cloud gaming. When the cloud gaming service platform needs to be upgraded or migrated to new hardware configurations, drivers, and SDKs, it can quickly access, debug, test, and launch applications by implementing the four basic virtual function interfaces of the subclasses, thereby improving the compatibility of cloud gaming and significantly improving stability and operational efficiency.

The following continues to describe the exemplary structure of the first cloud audio and video processing device 255 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG4, the software module stored in the first cloud audio and video processing device 255 in the memory 250 may include:

The acquisition module 260 is used to acquire audio and video data generated by the cloud audio and video process through the audio and video acquisition standard interface;

The encoding module 270 is used to encode the audio and video data in a preset standard encoding format through the encoding transmission standard interface to obtain an encoded audio and video stream, and transmit the encoded audio and video stream to the interactive device 400.

Event processing module 280 is used to receive operation events initiated by the interactive device based on the encoded audio and video stream through the event standard input interface, and to map the operation events to target driving events corresponding to the current operating system; the target driving events are used to drive the cloud audio and video process to update the audio and video data accordingly; wherein...

The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively.

In some embodiments, the first cloud audio and video processing device 255 further includes an initialization module. The initialization module is used to call the acquisition module through the audio and video acquisition standard interface before acquiring the audio and video data generated by the cloud audio and video process. Before acquiring the audio and video data, the initialization module calls the local initialization interfaces corresponding to the acquisition module, the encoding module and the operating system through the initialization standard interface to initialize the acquisition module, the encoding module and the operating system. The initialization standard interface is a standardized interface implemented in the cloud service standardized subclass.

In some embodiments, the acquisition module 260 is further configured to periodically call the local acquisition driver interface through the audio and video acquisition standard interface to acquire the current video frame and current audio generated by the cloud audio and video process according to a preset sampling frame rate, so as to obtain the audio and video data; the preset sampling frame rate is a standardized sampling frame rate uniformly set in the preset cloud service standard base class.

In some embodiments, the encoding module 270 is further configured to call the local encoding driver interface through the encoding transmission standard interface to encode the audio and video data according to the preset encoding frame rate and the preset standard encoding format to obtain an encoded audio and video stream; the preset encoding frame rate is the standardized encoding frame rate uniformly set in the preset cloud service standard base class.

The event processing module 280 is further configured to call a preset instruction conversion service through the event standard input interface to determine the target driving event corresponding to the operation event from a preset correspondence between at least one preset operation event and at least one standard driving event; the at least one standard driving event is a standard driving event in the current operating system; and convert the operation event into the target driving event.

The following continues to describe the exemplary structure of the second cloud audio and video processing device 455 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG5, the software module stored in the second cloud audio and video processing device 455 in the second memory 450 may include:

A client-side standardization component 460 is provided, which is connected to the encoding transmission standard interface and the event standard input interface on the first cloud host 200; wherein...

The client-side standardized component 460 is used to receive the encoded audio and video stream sent by the encoded transmission standard interface; and to decode and play the encoded audio and video stream to obtain decoded video and decoded audio; the encoded audio and video stream is obtained by the first cloud host 200 calling the acquisition module 260 through the audio and video acquisition standard interface to acquire cloud audio and video data generated by the cloud audio and video process; and by calling the encoding module 270 through the encoded transmission standard interface to encode the audio and video data in a preset standard encoding format.

The client standardization component 460 is further configured to acquire at least one type of original operation instruction initiated for the decoded video or decoded audio, and to convert the original operation instruction of at least one type of original operation instruction according to a preset standard event format to obtain the operation event. The operation event is then sent to the event standard input interface, so that the first cloud host 200 maps the operation event to a target driving event corresponding to the current operating system through the event standard input interface. The target driving event is used to drive the cloud audio/video process to update the audio/video data accordingly.

The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively.

The following continues to describe the exemplary structure of the third cloud audio and video processing device 555 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG6, the software module stored in the third cloud audio and video processing device 555 in the third memory 550 may include:

A preset downlink standard interface 570 and a preset uplink standard interface 560 are provided; the preset downlink standard interface 570 is connected to the encoding transmission standard interface on the first cloud host 200 and the interactive device, respectively; the preset uplink standard interface 560 is connected to the interactive device and the event standard input interface on the first cloud host 200, respectively.

The preset downlink standard interface 570 is used to acquire encoded audio and video streams; the encoded audio and video streams are generated by the first cloud host 200 through the audio and video acquisition standard interface calling the acquisition module 260 to acquire cloud audio and video data; and the audio and video data are encoded in a preset standard encoding format by the encoding module 270 through the encoding transmission standard interface; the encoded audio and video streams are transmitted to the interactive device 400 using the preset standard transmission format;

The preset uplink standard interface 560 is used to receive operation events initiated by the interactive device 400 based on the encoded audio and video stream; and to transmit the operation events to the event standard input interface on the first cloud host 200 using the preset standard transmission format.

In some embodiments, the interactive device 400 includes a client standardization component 460, which is connected to the preset downlink standard interface 570 and the preset uplink standard interface 560 respectively. The preset downlink standard interface 570 is further configured to transmit the encoded audio and video stream to the client standardization component 460 on the interactive device 400 using the preset standard transmission format, so that the client standardization component 460 decodes and plays the encoded audio and video stream using a preset decoding format to obtain decoded video images and decoded audio.

The preset uplink standard interface 560 is also used to receive the operation event sent by the client standardization component 460 on the interactive device 400, wherein the operation event is obtained by the client standardization component 460 by acquiring at least one original operation instruction of at least one format type initiated for the decoded video screen or decoded audio, and converting the original operation instruction of at least one format type according to the preset standard event format.

In some embodiments, a standardized transmission service is implemented between the preset uplink standard interface 560 and the preset downlink standard interface 570, and the standardized transmission service includes at least one of the following:

Uplink and downlink network transmission bandwidth assessment service, congestion control service, and connection maintenance and abnormal reconnection service.

In some embodiments, the event standard input interface is connected to the preset uplink standard interface 560 via a first socket; the encoded transmission standard interface is connected to the preset downlink standard interface 570 via a second socket.

In some embodiments, the preset standard transmission format includes at least one of the following:

The system includes a preset application layer standard protocol format, a preset transport layer standard protocol format, and a preset network layer standard protocol format. The preset application layer standard protocol format includes either a web instant messaging protocol or a real-time streaming protocol. The preset transport layer standard protocol format includes either a real-time transmission protocol or a real-time transmission control protocol. The preset network layer standard protocol format includes either a user data packet protocol or a flow control transmission protocol.

It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the cloud audio and video processing method described above in this application.

This application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, they will cause the processor to perform the method provided in this application, such as the method shown in Figures 7, 9, 10, 12, and 13.

In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

In summary, this application embodiment implements standardized interfaces in cloud service standardized subclasses derived from a preset cloud service standardized base class, based on the configuration information of the acquisition module, encoding module, and operating system on the cloud service platform host. This enables the acquisition of audio and video data via the audio/video acquisition standard interface, the standardized encoding and output of audio and video via the encoding and transmission standard interface, and the mapping of operation events to target driver events corresponding to the operating system of the cloud service platform host through the event standard input interface. This achieves encapsulation and isolation of different module driver configurations and different cloud host operating systems. Standardized acquisition, encoding, transmission, and event services can be provided externally through a unified cloud service standardized subclass on cloud hosts with different configurations, reducing module coupling and code redundancy, improving the compatibility and stability of the cloud audio/video interaction system. Furthermore, when upgrading or migrating acquisition, encoding, transmission, and event services to new hardware or host OS environments, only the functional interfaces of the cloud service standardized subclass need to be implemented, thereby improving the maintainability of the cloud audio/video interaction system.

The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims (15)

1. A cloud audio and video processing method, characterized in that it includes: The acquisition module is called through the standard audio and video acquisition interface to acquire audio and video data generated by the cloud audio and video process. The encoding module is called through the encoding transmission standard interface to encode the audio and video data in a preset standard encoding format to obtain an encoded audio and video stream, and then the encoded audio and video stream is transmitted to the interactive device. The system receives operation events initiated by the interactive device based on the encoded audio and video stream through a standard event input interface, and maps these operation events to target driver events corresponding to the current operating system. The target driver events are used to drive the cloud audio and video process to update the corresponding audio and video data. The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively. 2. The method according to claim 1, characterized in that, before calling the acquisition module through the audio and video acquisition standard interface to acquire the audio and video data generated by the cloud audio and video process, the method further includes: The acquisition module, the encoding module, and the operating system are initialized by calling their respective local initialization interfaces through the initialization standard interface. The initialization standard interface is a standardized interface implemented in the cloud service standardized subclass. 3. The method according to claim 1 or 2, characterized in that, the step of calling the acquisition module through the audio and video acquisition standard interface to acquire the audio and video data generated by the cloud audio and video process includes: According to the preset sampling frame rate, the local acquisition driver interface of the acquisition module is called periodically through the audio and video acquisition standard interface to acquire the current video frame and current audio generated by the cloud audio and video process, so as to obtain the audio and video data; the preset sampling frame rate is the standardized sampling frame rate uniformly set in the preset cloud service standard base class. 4. The method according to claim 1 or 2, characterized in that, the step of calling the encoding module through the encoding transmission standard interface to perform audio and video encoding of the audio and video data in a preset standard encoding format to obtain an encoded audio and video stream includes: The local encoding driver interface of the encoding module is called through the encoding transmission standard interface to encode the audio and video data according to the preset encoding frame rate and the preset standard encoding format to obtain an encoded audio and video stream; the preset encoding frame rate is the standardized encoding frame rate uniformly set in the preset cloud service standard base class. 5. The method according to claim 1 or 2, characterized in that, mapping the operation event to a target driver event corresponding to the current operating system includes: By invoking a preset instruction conversion service through the event standard input interface, the target driving event corresponding to the operation event is determined from a preset correspondence between at least one preset operation event and at least one standard driving event; the at least one standard driving event is a standard driving event in the current operating system. The operation event is converted into the target-driven event. 6. A cloud audio and video processing method, characterized in that it includes: The system receives encoded audio and video streams and decodes and plays them to obtain decoded video images and decoded audio. The encoded audio and video streams are generated by the first cloud host through the audio and video acquisition standard interface, which calls the acquisition module to acquire cloud audio and video data; and through the encoding transmission standard interface, which calls the encoding module to encode the audio and video data in a preset standard encoding format. The system acquires an operation event initiated in response to the decoded video or audio, and sends the operation event to the event standard input interface of the first cloud host. This allows the first cloud host to map the operation event to a target driver event corresponding to the current operating system via the event standard input interface. The target driver event is used to drive the cloud audio/video process to update the corresponding audio/video data. The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively. 7. The method according to claim 6, characterized in that, receiving the encoded audio and video stream and decoding and playing the encoded audio and video stream to obtain decoded video images and decoded audio includes: The client-side standardized component receives the encoded audio and video streams output by the first cloud host through the encoded transmission standard interface; The client-side standardized component decodes and plays the encoded audio and video streams using a preset decoding format to obtain the decoded video and the decoded audio. The step of acquiring the operation event initiated for the decoded video frame or decoded audio includes: The client-side standardization component obtains at least one type of original operation command for the decoded video or decoded audio, and performs format conversion on the original operation command according to a preset standard event format to obtain the operation event. 8. A cloud audio and video processing method, characterized in that it includes: The encoded audio and video stream is obtained through a preset downlink standard interface; the encoded audio and video stream is generated by the first cloud host by calling the acquisition module through the audio and video acquisition standard interface to acquire cloud audio and video data; and the audio and video data is encoded by the encoding module through the encoding transmission standard interface to perform audio and video encoding in a preset standard encoding format; the preset downlink standard interface is connected to the encoding transmission standard interface on the first cloud host and the interactive device respectively. The encoded audio and video streams are transmitted to the interactive device using a preset standard transmission format. The system receives operation events initiated by the interactive device based on the encoded audio and video stream through a preset uplink standard interface; the preset uplink standard interface is connected to the event standard input interface on the first cloud host and the interactive device, respectively. The operation event is transmitted to the event standard input interface using the preset standard transmission format. 9. The method according to claim 8, characterized in that the interactive device is configured with a client standardization component, the client standardization component is connected to the preset downlink standard interface and the preset uplink standard interface respectively, and the step of transmitting the encoded audio and video stream to the interactive device using the preset standard transmission format includes: Through the preset downlink standard interface and using the preset standard transmission format, the encoded audio and video stream is transmitted to the client standardization component on the interactive device, so that the client standardization component can decode and play the encoded audio and video stream in a preset decoding format to obtain decoded video and decoded audio. The step of receiving operation events initiated by the interactive device based on the encoded audio and video stream through a preset uplink standard interface includes: The system receives operation events from the client standardization component on the interactive device via a preset uplink standard interface. The operation events are obtained by the client standardization component by acquiring at least one type of original operation instruction for the decoded video or decoded audio and converting the original operation instruction of at least one type of original operation instruction according to a preset standard event format. 10. A cloud audio and video processing system, comprising: a first cloud host, a second cloud host, and an interactive device; the first cloud host running a cloud audio and video process; the second cloud host having a preset uplink standard interface and a preset downlink standard interface deployed thereon; the interactive device having a client standardization component deployed thereon; the client standardization component being connected to an event standard input interface on the first cloud host via the preset uplink standard interface, and the client standardization component being connected to an encoding transmission standard interface on the first cloud host via the preset downlink standard interface; wherein... The first cloud host is used to call the acquisition module through the audio and video acquisition standard interface to acquire the audio and video data generated by the cloud audio and video process; and to call the encoding module through the encoding transmission standard interface to encode the audio and video data in a preset standard encoding format to obtain an encoded audio and video stream, and to transmit the encoded audio and video stream to the preset downlink standard interface. The second cloud host is used to transmit the encoded audio and video streams to the client standardization component via the preset downlink standard interface using a preset standard transmission format; The interactive device is used to decode and play the encoded audio and video stream through the client standardization component to obtain decoded video and decoded audio; and to obtain operation events initiated for the decoded video or decoded audio and send the operation events to the preset uplink standard interface. The second cloud host is also used to transmit the operation event to the event standard input interface through the preset uplink standard interface using the preset standard transmission format; The first cloud host is further configured to map the operation event to a target driving event corresponding to the current operating system through the event standard input interface; the target driving event is used to drive the cloud audio and video process to update the audio and video data accordingly; The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, respectively based on the configuration information of the acquisition module, the encoding module, and the operating system. 11. A first cloud host, characterized in that it comprises: The acquisition module is used to acquire audio and video data generated by the cloud audio and video process through the standard audio and video acquisition interface; The encoding module is used to encode the audio and video data in a preset standard encoding format through an encoding transmission standard interface to obtain an encoded audio and video stream, and then transmit the encoded audio and video stream to the interactive device. The event handling module is used to receive operation events initiated by the interactive device based on the encoded audio and video stream through the event standard input interface, and to map the operation events to target driving events corresponding to the current operating system; the target driving events are used to drive the cloud audio and video process to update the audio and video data accordingly; wherein... The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively. 12. An interactive device, characterized in that it comprises: a client-side standardized component, wherein the client-side standardized component is connected to an encoding transmission standard interface and an event standard input interface on a first cloud host; wherein, The client-side standardized component is used to receive the encoded audio and video stream sent by the encoded transmission standard interface; and to decode and play the encoded audio and video stream to obtain decoded video and decoded audio; the encoded audio and video stream is obtained by the first cloud host calling the acquisition module through the audio and video acquisition standard interface to acquire cloud audio and video data generated by the cloud audio and video process; and by calling the encoding module through the encoded transmission standard interface to encode the audio and video data in a preset standard encoding format. The client-side standardization component is further configured to acquire at least one type of raw operation instruction initiated for the decoded video or decoded audio, and to convert the raw operation instruction of at least one type of raw operation instruction according to a preset standard event format to obtain an operation event. The operation event is then sent to the event standard input interface, so that the first cloud host maps the operation event to a target driver event corresponding to the current operating system through the event standard input interface. The target driver event is used to drive the cloud audio/video process to update the audio/video data accordingly. The audio/video acquisition standard interface, the encoding and transmission standard interface, and the event standard input interface are standardized interfaces implemented in cloud service standard subclasses derived from the preset cloud service standard base class, based on the configuration information of the acquisition module, the encoding module, and the operating system, respectively. 13. A second cloud host, characterized in that it comprises: a preset downlink standard interface and a preset uplink standard interface; the preset downlink standard interface is respectively connected to an encoding transmission standard interface on a first cloud host and an interactive device, and the preset uplink standard interface is respectively connected to the interactive device and an event standard input interface on the first cloud host; wherein... The preset downlink standard interface is used to receive the encoded audio and video stream sent by the first cloud host through the encoding transmission standard interface; the encoded audio and video stream is obtained by the first cloud host calling the acquisition module through the audio and video acquisition standard interface to acquire cloud audio and video data generated by the cloud audio and video process; and by calling the encoding module through the encoding transmission standard interface to encode the audio and video data in a preset standard encoding format; the encoded audio and video stream is transmitted to the interactive device using the preset standard transmission format; The preset uplink standard interface is used to receive operation events initiated by the interactive device based on the encoded audio and video stream; and to transmit the operation events to the event standard input interface on the first cloud host using the preset standard transmission format. 14. An electronic device, characterized in that it comprises: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method as claimed in any one of claims 1 to 5, or claim 6 or 7, or claim 8 or 9. 15. A computer-readable storage medium, characterized in that it stores executable instructions for implementing, when executed by a processor, the method as described in any one of claims 1 to 5, or claim 6 or 7, or claim 8 or 9.