TWI911362B - Method for reducing network bandwidth required for image streaming by using artificial intelligence processing module - Google Patents

Abstract

A method for reducing network bandwidth required for image streaming by using an Artificial Intelligence (AI) processing module. The resolution of the images to be transmitted are firstly reduced on the server side, and then the resolution-reduced low-resolution images are transmitted to the client device through the network, thereby reducing the network bandwidth required to transmit the image streams. A pre-trained AI processing module is used on the client device to restore the received low-resolution images to high-resolution images, allowing users to enjoy high-quality image streaming and low-bandwidth consumption of network at the same time.

Description

A method for reducing network bandwidth required for video streaming using an artificial intelligence processing module

This invention relates to a method for reducing the network bandwidth required for video streaming using an artificial intelligence (AI) processing module. Specifically, it describes a method that first reduces the resolution of the image to be transmitted on the server side before transmitting it over the network to the client, and then uses a pre-trained AI processing module on the client side to restore the received image to high resolution, thereby reducing the network bandwidth required for video streaming.

In recent years, online games have become increasingly popular worldwide. With the development of cloud-based computing systems and technologies, a cloud technology has been developed that provides online gaming services by having servers stream game content over the network to players.

Traditionally, this type of cloud-based online game service is provided by having the server perform almost all the computation. In other words, when providing an online game service, a specific application runs on the server to generate a virtual 3D environment containing many 3D (three-dimensional) objects, including 3D objects that the player can control or move. Then, based on the player's control, the server renders these 3D objects and the virtual 3D environment into a 2D (two-dimensional) game screen for display on the player's device. Next, the server encodes and compresses the rendered image into a 2D video stream and transmits it over the network to the player's device. The player device only needs to decode and "play" the received 2D video stream, without performing 3D rendering calculations. However, this type of cloud-based online gaming service still has several issues to consider, such as the high server load when providing 3D rendering to a large number of players simultaneously, the reduction in image quality caused by encoding compression and streaming processes, and the large amount of communication bandwidth consumed by transmitting 2D video streams over the network.

A common approach to addressing image quality degradation is to increase the resolution of the original image generated by the game application on the server side and increase the bitrate when transmitting the image; that is, to reduce the compression ratio when the server encodes the original image into a 2D video stream. However, it's obvious that this will significantly increase server load and bandwidth consumption due to the high image resolution and high bitrate. For example, assuming both the frame rate and compression ratio are constant, increasing the resolution of the original image generated by the game application on the server from 720p to 1080p will increase the server's computational load and the required network bitrate by 2.25 times. Conversely, attempting to reduce server load or network bandwidth consumption will sacrifice the game's visual quality. Therefore, achieving both perfect visual quality and cost-effective bandwidth consumption becomes an impossible dilemma.

Another way to solve this problem is to reduce the resolution of the original image generated by the server-side game application, or to encode the original image into a 2D video stream at a higher compression rate, or both. This reduces the bandwidth consumption of transmitting the 2D video stream over the network, although the image quality of the game will be sacrificed. Simultaneously, an image enhancement technique is used on the client device. Once the 2D video stream is received, the client device decodes the stream and uses the image enhancement technique to improve the visual effect of the image. Histogram equalization (HE) is one of the most commonly used methods to improve image contrast due to its simplicity and efficiency. However, HE (High-Contrast Enhancement) can lead to excessive contrast enhancement and feature loss, resulting in unnatural appearances and loss of image detail. Furthermore, not only HE but all other known image enhancement techniques in this field face the same dilemma: they all attempt to use the same algorithm to process images with completely different content, which is impractical. Take cloud-based online game services as an example: the content of the original image generated by the server changes significantly due to variations in the game scene. For instance, a city game scene might result in the original image containing many images with simple and clear outlines and colors that, while different, are roughly of the same color family. A dark cave scene will result in a game's original image filled with monotonous, low-toned, and low-saturation colors, but with irregular and unremarkable landscape outlines. Conversely, a lush garden scene will result in an original image containing many vibrant and colorful objects with detailed and complex outlines. Undoubtedly, no single learning enhancement technique can provide equally good image enhancement for such diverse scenes with completely different content.

Furthermore, another drawback of these learned image enhancement techniques is that while their mathematical formulas can improve image quality such as contrast, sharpness, and saturation, these formulas and their parameters are completely unrelated to the original image generated by the server. Therefore, the enhancement process of these techniques never makes the enhanced image visually closer to its corresponding original image. Consequently, client-side gamers cannot fully enjoy the visual effects of the original image generated by the server-side game application.

Therefore, the main objective of this invention is to provide a method for reducing the network bandwidth required for image streaming using an artificial intelligence (AI) processing module. By reducing the resolution of the image to be transmitted on the server side before transmitting the low-resolution image to the client over the network, the network bandwidth required for image streaming is reduced. Then, on the client side, a pre-trained AI processing module restores the received low-resolution image to a high-resolution image, simultaneously enjoying the dual advantages of high image quality and low network bandwidth consumption.

To achieve the above objectives, an embodiment of the method for reducing the network bandwidth required for video streaming using an artificial intelligence processing module includes: Step (A): Executing a first application in a server to generate a plurality of source images corresponding to a plurality of original image images; the plurality of source images having a first resolution; the plurality of source images being encoded and compressed by an encoder within the server to generate a plurality of corresponding encoded images; Step (B): [The text abruptly ends here, likely due to an incomplete translation or missing information.] A second application runs within a client device on the server; the second application is associated with and works in conjunction with the first application; Step (C): The client device connects to the server via a network and receives the encoded images generated by the server via the network as a video stream; Step (D): The client device decodes the encoded images into a corresponding plurality of decoded images and uses an artificial intelligence (AI)... Intelligent (AI) processing module is used to improve the resolution of the decoded images to generate a plurality of high-resolution images; the plurality of high-resolution images have a second resolution; wherein the second resolution is higher than the first resolution, and the resolution of the plurality of original images is equal to the second resolution; Step (E): The client device sequentially outputs the plurality of high-resolution images to a screen as a plurality of output images to be played; wherein the AI processing module uses The decoded images are processed using at least one mathematical formula and a plurality of weighting parameters, derived from analyzing the differences between the decoded images and their corresponding original images. Consequently, the resolution of the resulting high-resolution images is equal to that of the corresponding original images and higher than the resolution of the plurality of source images. The at least one mathematical formula and the plurality of weighting parameters of the AI processing module are pre-defined by a training procedure executed by an artificial neural network module within a training server.

Preferably, the plurality of encoded images described in step (A) are generated by the following steps: executing the first application on the server to generate a plurality of original images having the second resolution; using a resolution reduction procedure to reduce the resolution of the plurality of original images to the first resolution to obtain a corresponding plurality of source images; and using the encoder to encode the plurality of source images to obtain a corresponding plurality of encoded images.

Preferably, the server includes an AI encoding module; the plurality of encoded images in step (A) are generated by the following steps: executing the first application on the server to generate a plurality of original images, the plurality of original images having the second resolution; using the AI encoding module to reduce the resolution of the plurality of original images to obtain a corresponding plurality of source images, and encoding the plurality of source images to obtain a corresponding plurality of encoded images; wherein the AI encoding module includes at least one preset AI encoding formula; the at least one AI encoding formula includes a preset plurality of encoding weighting parameters.

Preferably, the at least one mathematical operation of the AI processing module includes a first preset AI operation and a second preset AI operation; the first preset AI operation includes a plurality of first weighting parameters; the second preset AI operation includes a plurality of second weighting parameters; wherein the first preset AI operation, in combination with the plurality of the first weighting parameters, can be used to improve the resolution of the image, thereby increasing the resolution of the image processed by the first preset AI operation in combination with the plurality of the first weighting parameters from the first resolution to the second resolution; wherein the second preset AI operation, in combination with the plurality of the second weighting parameters, can be used to enhance the quality of the image, thereby increasing the quality of the image processed by the second preset AI operation in combination with the plurality of the second weighting parameters to be higher than the quality of the decoded image and closer to the quality of the original image.

Preferably, after the client device decodes the received plurality of encoded images into a corresponding plurality of decoded images, the client device processes the plurality of decoded images using one of the following methods:

Method 1: The client device first uses the first preset AI algorithm and multiple first weighting parameters to process multiple decoded images to generate multiple resolution-upgraded images with a corresponding second resolution; then, the client device uses the second preset AI algorithm and multiple second weighting parameters to process multiple resolution-upgraded images to generate multiple high-resolution images with high image quality and the second resolution;

Method 2: The client device first uses the second preset AI algorithm and multiple second weighting parameters to process multiple decoded images to generate multiple quality-enhanced images with corresponding high image quality; then, the client device uses the first preset AI algorithm and multiple first weighting parameters to process multiple quality-enhanced images to generate multiple high-resolution images with the second resolution and high image quality.

1. 501, 701: Server

2, 21, 22, 23, 502, 702: Client devices

3: Base station

30: Router

4: Internet

100, 200: Application (App)

101, 201: Memory

102: Encoding

103: Streaming

104: Network Equipment

105: Artificial Neural Network Module

106: Neural Networks

107: Decoding Module

108: Comparison and Training Modules

202: Network Module

203: Decoding Module

204: AI Enhancement Module

205: Output Module

301-308, 400-466, 711-723, 7161-7229: Steps

A preferred embodiment of the present invention will be illustrated with reference to the following figures, wherein:

Figure 1 schematically illustrates the system of this invention that utilizes an artificial intelligence processing module to reduce the network bandwidth required for video streaming;

Figure 2 is a schematic diagram of an embodiment of the system architecture of this invention, which utilizes an artificial intelligence processing module to reduce the network bandwidth required for video streaming;

Figure 3 is a schematic diagram of a first embodiment of the method for processing image streams using an artificial intelligence processing module according to the present invention;

Figure 4 is a schematic diagram of a first embodiment of the training program of the artificial neural network module 105 described in this invention;

Figure 5 is a schematic diagram of a second embodiment of the training program of the artificial neural network module 105 described in this invention;

Figure 6 is a schematic diagram of a third embodiment of the training program of the artificial neural network module 105 described in this invention;

Figure 7 is a schematic diagram of an embodiment of the training procedure for the detector shown in Figure 6;

Figure 8 illustrates an embodiment of the training process of the neural network of this invention, wherein the original image is YUV420 and the output image is RGB or YUV420;

Figure 9 is a schematic diagram of an embodiment of the program for processing images decoded in YUV420 format according to the present invention;

Figure 10 is a schematic diagram of another embodiment of the procedure for processing images decoded in YUV420 format according to the present invention;

Figure 11A is a schematic diagram of a second embodiment of the method of the present invention for reducing the network bandwidth required for video streaming using an artificial intelligence processing module;

Figure 11B is a schematic diagram of a third embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module;

Figure 12A is a schematic diagram of the fourth embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module according to the present invention;

Figure 12B is a schematic diagram of the fifth embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module according to the present invention;

Figure 13 is a schematic diagram of an embodiment of the training method for the first preset AI algorithm and the first weighting parameters of the AI processing module described in this invention;

Figure 14A is a schematic diagram of the sixth embodiment of the method of the present invention for reducing the network bandwidth required for video streaming using an artificial intelligence processing module;

Figure 14B is a schematic diagram of the seventh embodiment of the method of the present invention for reducing the network bandwidth required for video streaming using an artificial intelligence processing module;

Figure 15 is a schematic diagram of an embodiment of the training method for the first preset AI algorithm, the second preset AI algorithm, the first weighting parameter, and the second weighting parameter of the AI processing module described in this invention;

Figure 16 is a schematic diagram of the eighth embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module according to the present invention;

Figure 17 is a schematic diagram illustrating an embodiment of the training method for the AI encoding algorithm, the AI decoding algorithm, the first preset AI algorithm, the second preset AI algorithm, the first weighting parameter, and the second weighting parameter of the artificial neural network module described in this invention.

This invention relates to a method for reducing the network bandwidth required for video streaming using an artificial intelligence (AI) processing module. By reducing the resolution of the image to be transmitted on the server side before transmitting the low-resolution image to the client over the network, the network bandwidth required for video streaming is reduced. Then, on the client side, a pre-trained AI processing module restores the received low-resolution image to a high-resolution image, simultaneously enjoying the dual advantages of high image quality and low network bandwidth consumption.

One application of this invention is cloud-based online games, where players use client devices connected to a server via a network to play games provided by the server. The server responds to commands input by the player and generates corresponding video images. For example, a player can execute a movement command on their client device. This movement command is transmitted to the server via the network, and the server calculates an image based on the movement command, sends the image back, and plays it on the client device. In many games, the server generates 2D images containing several 3D rendered objects within the visible field of view.

Please refer to Figure 1, which schematically illustrates the system of the present invention that utilizes an artificial intelligence processing module to reduce the network bandwidth required for video streaming. A server 1 is used to provide services by an application running on the server 1; the service may be, but is not limited to, a cloud-based online game service. A plurality of client devices 21, 22, 23 may connect to (log in to) the server 1 via a network 4 to use the services provided by the application running on the server 1. In this embodiment, network 4 is the Internet, and client devices 21, 22, and 23 can be any type of network-connected electronic device, such as (but not limited to) a smartphone 21, a digital tablet, a laptop 22, a desktop computer 23, a video game console, or even a smart TV. Some client devices 21 and 22 are wirelessly connected to network 4 via a wireless communication base station 3 or a wireless router 30, while others are wired connected to network 4 via a network router or network sharer. The application running on server 1 generates a virtual 3D environment containing a plurality of 3D objects; some of these 3D objects can be moved or destroyed according to user actions, while others cannot. In a preferred embodiment, the application operates independently for each client device; that is, each application provides the service to only one client device, but multiple applications can run simultaneously on server 1 to provide services to multiple client devices. Client devices 21, 22, and 23 are connected to server 1 via network 4 to receive images generated by the application, which contain at least a portion of 3D objects. The architecture and functionality of the system of this invention will be detailed in Figure 2 and its related description.

Figure 2 is a schematic diagram of an embodiment of the system architecture of the present invention. The application (App) 100, stored in memory 101 and executed on server 1, is an application (typically a 3D game) that can generate a 3D rendered image result composed of a series of original image files. Encoding 102 and streaming 103 are encoding and streaming modules, respectively, which can accept the original image files generated by the application 100 and encode and stream them as a 2D image stream. The 2D image stream is then transmitted via network 4 through network 4 via network device 104 of the server to a remote client device 2. Each client device 2 has an application 200 pre-installed. This application 200 is stored in the memory 201 of the client device 2 and can associate and cooperate with the application 100 on the server 1. The application 200 of the client device 2 can establish a connection with the application 100 of the server 1 and receive the encoded 2D image stream from the server 1 through the network module 202. The encoded 2D image stream is then decoded by the decoding module 203 to generate a decoded image. Due to these encoding, streaming, and decoding processes, the quality of the decoded image is obviously much worse than the original image. The AI processing module 204, built into the client device 2, can enhance the quality of the decoded images to produce corresponding enhanced images. Specifically, the AI processing module 204 processes the decoded images using at least one mathematical expression derived from analyzing the differences between the decoded images and their corresponding original images; thereby, the resulting enhanced images will visually be closer to the original images than the decoded images. The enhanced images are then output (played) on the screen (display panel) of the client device 2 via the output module 205. In this invention, the mathematical formulas used by the AI processing module 204 of the client device 2 are defined by a training program executed by the Artificial Neural Network (ANN) module 105 located on the server 1. The ANN module 105 is located within the server 1 and includes: an artificial neural network 106, a decoding module 107, and a comparison and training module 108. An embodiment of the training program for the ANN module 105 described in this invention will be detailed later.

Figure 3 is a schematic diagram of a first embodiment of the method for processing image streams using an artificial intelligence processing module according to the present invention. By utilizing the system and architecture of the present invention as shown in Figures 2 and 3, the method generally includes the following steps:

Step 301: A first application is executed on a server. This first application generates multiple original image files according to at least one instruction (Step 302). The original image files are then down-processed by a resolution reduction module within the server (Step 3021), and then encoded and compressed by an encoder within the server (Step 303) to generate multiple encoded images. These encoded images are then transmitted to a client device via a network as a 2D video stream (Step 304). Because the images are down-processed before being transmitted to the client device, the network bandwidth required for transmitting the video stream is also reduced.

A second application runs on a client device located remotely from the server (step 305). This second application is connected to and works in conjunction with the first application, allowing the client device to be operated by a user to generate and send commands to the server to enjoy services provided by the server's first application. The client device transmits the command to the server via the network and then receives, via the network, the encoded image generated by the server corresponding to the command. The client device then decodes (step 306) these encoded images into multiple decoded images and uses an AI processing module (step 307) to enhance the quality of these decoded images to produce multiple enhanced images. The AI processing module processes the decoded images using at least one pre-derived mathematical formula derived from analyzing the differences between the decoded images and their corresponding original images. This results in an enhanced image that visually resembles the original image more closely than the decoded image. The client device then outputs the enhanced image (step 308) to the screen (display panel) as the output image to be played.

In this invention, the at least one mathematical expression used by the AI processing module within the client device includes a plurality of weighted parameters. These weighted parameters are related to the differences between the decoded image and the corresponding original image, and are defined by a training program executed by an artificial neural network module within the server. In one embodiment of this invention, the weighted parameters are pre-stored in the client device. In another embodiment, the weighted parameters are downloaded from the server to the client device only when the second application is executed.

In one embodiment of the present invention, the image content contained in the original image generated by the server varies drastically depending on the game scene. For example, a city game scene might result in an original image containing many simple and clear outlines and colors that, while different, are generally of the same color family. Another game scene, a dark cave, would result in an original image filled with monotonous, low-toned, and low-saturation colors, but with irregular but unremarkable landscape outlines. Yet another scene, a lush garden, would result in an original image containing many vibrant and brightly colored objects with detailed and complex outlines. This invention employs several different weighting parameters to adapt to different game scenarios. Therefore, the quality of the output image enhanced by the same AU enhancement module can be maintained at a high and stable level, even if the content of the original image changes drastically.

Preferably, the original images generated by the first application can be divided into multiple sets of scene-modes, each scene containing multiple sets of the original images. The weighting parameters are also divided into multiple sets, each set containing multiple weighting parameters corresponding to one of the scenes. The decoded images corresponding to the original images of different scenes are used by the same AI processing module to perform image enhancement processing using the set of weighting parameters corresponding to the scene from the different sets of weighting parameters. In one embodiment of the invention, all the different sets of weighting parameters are pre-stored in the client device. Whenever the scene changes, the set of weighting parameters corresponding to the new scene is applied to the AI processing module to generate the enhanced image. In another embodiment, all the different sets of weighting parameters are stored on the server. Whenever the scene changes, the corresponding set of weighting parameters for the new scene is transmitted from the server to the client device and then used in the AI processing module to generate the enhanced image.

Figure 4 is a schematic diagram of a first embodiment of the training program of the artificial neural network module 105 described in this invention. In this invention, the mathematical expressions used by the AI processing module 204 of the client device 2 are trained and defined by a training program executed by the artificial neural network module 105 in the server 1. The training program includes the following steps:

Step 400: Execute the first application in a training mode to generate a plurality of training source images (Step 401), and perform resolution reduction processing on these source images (Step 4011);

Step 402: Encode the reduced-resolution original training images into multiple encoded training images using the encoder;

Step 403: Decode the training encoded images into a plurality of training decoded images using a training decoder on the server;

Step 404: The artificial neural network module receives the training decoded images and processes them one by one using at least one training mathematical operation to generate a plurality of training output images (Step 405); the at least one training mathematical operation includes a plurality of training weighting parameters; and

Step 406: The comparison and training module compares the differences between the training output image and the corresponding training original image one by one, and adjusts the training weighting parameters of the at least one training mathematical formula accordingly. The training weighting parameters are adjusted to minimize the differences between the training output image and the corresponding training original image. Each time the training weighting parameters are adjusted, the adjusted training weighting parameters are fed back to the at least one training mathematical formula for processing the next training decoded image in step 404. After comparing a predetermined number of training output images with their corresponding original training images, and adjusting the training weighting parameters a predetermined number of times, the training weighting parameters obtained after training (step 407) are finally extracted and applied to the AI processing module of the client device as weighting parameters for its mathematical calculations.

In a first embodiment of the present invention, the training decoded image is input into the artificial neural network module to generate a corresponding training output image. Then, the training output image and the corresponding original training image are compared to calculate the difference. Subsequently, mathematical optimization algorithms such as Adam algorithm, Stochastic gradient descent (SGD), or Root Mean Square Propagation (RMSProp) are used to learn the weighting parameters (commonly referred to as weight w and bias b) of the artificial neural network, minimizing the difference so that the training output image can more closely approximate its corresponding original training image. Different methods can be used to calculate the difference (or approximation) to suit different needs; for example: mean square error (MSE), L1 regularization (using absolute value error), peak signal-to-noise ratio (PSNR), structure similarity (SSIM), generative adversarial network loss (GAN loss) and/or other methods, etc. In the first embodiment, the following methods are used to calculate the difference: (1) a weighted average of MSE, L1, and GAN loss; (2) MSE; (3) GAN loss while simultaneously training the discriminator; (4) a weighted average of MSE and the edge of MSE. Further details of the training program will be described later.

Figure 5 is a schematic diagram of a second embodiment of the training procedure for the artificial neural network module 105 described in this invention. In this invention, the training procedure of the second embodiment includes the following steps:

Step 410: Execute the first application in a training mode to generate a plurality of training source images (Step 411), wherein the color format of these training source images is RGB (primary colors of light); and perform resolution reduction processing on these training source images (Step 4111);

Step 412: Encode the reduced-resolution original training images into multiple encoded training images using the encoder;

Step 413: Decode the training encoded images into multiple training decoded images using the training decoder on the server;

Step 414: In this second embodiment, when the color format of the training decoded image and the training output image is the same (both are RGB in this second embodiment), a residual network module (also known as a convolutional neural network; CNN) can be used in the artificial neural network module; the output of the residual network module used to process the corresponding training decoded image is summed up with the corresponding training decoded image (step 415); then, the sum of the output of the residual network module and the corresponding training decoded image is output as the training output image (step 416); and

Step 417: The comparison and training module compares the training output image with the corresponding training original image one by one (calculating the difference value), and adjusts the training weighting parameters of the at least one training mathematical formula accordingly. The training weighting parameters are adjusted to minimize the difference between the training output image and the corresponding training original image. Each time the training weighting parameters are adjusted, the adjusted training weighting parameters are fed back to the artificial neural network for processing the next training decoding image in step 414. After comparing a predetermined number of training output images with their corresponding original training images, and adjusting the training weighting parameters a predetermined number of times, the training weighting parameters obtained after training (step 418) are finally extracted and applied to the AI processing module of the client device as weighting parameters for its mathematical calculations.

Figure 6 is a schematic diagram of a third embodiment of the training procedure of the artificial neural network module 105 described in this invention. In the third embodiment, the comparison and training module uses a discriminator to compare the difference between the training output image and the corresponding original training image and adjust the training weighting parameters accordingly. The training procedure of the third embodiment includes the following steps:

Step 420: Execute the first application in a training mode to generate a plurality of training source images (Step 421), wherein the training source images include n channels, where n is a positive integer greater than 2; and perform resolution reduction processing on the training source images (Step 4211);

Step 422: Encode the reduced-resolution original training images into multiple encoded training images using the encoder;

Step 423: Decode the training encoded images into a plurality of training decoded images using a training decoder on the server; wherein each training decoded image includes m channels, where m is a positive integer greater than 2; and

Step 424: The artificial neural network module receives the training decoded images (m channels) and processes them one by one using at least one training mathematical operation to generate a plurality of training output images (n channels) (Step 425); the at least one training mathematical operation includes a plurality of training weighting parameters; the training output images (n channels) and their corresponding training decoded images (m channels) are combined (Step 426) to generate a plurality of training combined images (with m+n channels); then, these training combined images are fed back to a discriminator (Step 427) for analysis of the quality of the training output images, thereby training the artificial neural network.

Figure 7 is a schematic diagram of an embodiment of the training procedure for the detector shown in Figure 6. The training procedure for this detector includes the following steps:

Step 430: Execute the first application in a training mode to generate a plurality of training source images (Step 431), wherein the training source images include n channels, where n is a positive integer greater than 2; and perform resolution reduction processing on the training source images (Step 4311);

Step 432: Encode the reduced-resolution original training images into multiple encoded training images using the encoder;

Step 433: Decode the training encoded images into a plurality of training decoded images using a training decoder on the server; wherein each training decoded image includes m channels, where m is a positive integer greater than 2; and

Step 434: The artificial neural network module receives the training decoded images and processes each training decoded image (m channels) sequentially using at least one training mathematical operation to generate a plurality of training output images (Step 435); the at least one training mathematical operation contains a plurality of training weighting parameters; the training output images include n channels;

Step 436: The training output image of the n channels and its corresponding training decoded image of the m channels are combined to generate a plurality of false samples with m+n channels; and the original training image of the n channels and its corresponding training decoded image of the m channels are combined to generate a plurality of true samples with m+n channels (Step 437); and

Step 438: The simulated fake sample of the m+n channel and the simulated real sample of the m+n channel are fed back to the discriminator of the comparison and training module to train the discriminator to detect and distinguish between the simulated fake sample and the simulated real sample.

After the artificial neural network 105 (as shown in Figure 2) is properly trained on server 1, the obtained weighting parameters (weight w, bias b) are applied to the AI processing module 204 within the client device. The AI processing module 204 and its associated weighting parameters (weight w, bias b) are stored in the client device 2. Subsequently, whenever the client device receives and decodes the encoded images contained in the 2D image stream from the server, each encoded image is processed by the AI processing module to generate an enhanced image. The client device then plays these enhanced images as output images on its screen. The neural network can learn and enhance the color, brightness, and detail of the images. Because some details in the original image may be damaged or lost during encoding and streaming, a properly trained neural network can repair these damaged or lost details. In an embodiment of the invention, the neural network of the AI processing module requires the following information to operate:

Related functions and parameters:

X: Input image;

Conv2d(X,a,b,c,d,w,b): Executes on X; the number of output channels is a; the kernel size is b; the stride is c; the padding size is 2d convolution with bias d; the weighting parameters for this training are kernel w and bias b.

Conv2dTranspose(X,a,b,c, w , b )): Executes on X; the number of output channels is a; the kernel size is b; the stride is c; the cropping size is 2d transpose convolution with bias of d; the weighting parameters for this training are kernel w and bias b.

σ(X): A nonlinear activation function operating on X;

uint8(x): Used to control and limit the value of floating-point x to between 0 and 255 (inclusive). u uses the unconditional discard method to convert to unsigned int8.

R(X,w): Residual blocks that operate on X, including many conv2d and batchnorm blocks, each with its own weighted parameters for training (more information can be found at: https://stats.stackexchange.com/questions/246928/what-exactly-is-a-residual-learning-block-in-the-context-of-deep-residual-networ ).

Since input and output images may have different color formats, such as RGB, YUV420, YUV444, etc., the following discussion will focus more on input and output images with different color formats.

The first scenario: The original image is RGB, and the output image is also RGB.

This is the simplest case, as both input and output images are RGB. To improve processing speed, a relatively large kernel size (e.g., 8x8, stride=4 in convolution and transpose convolution structures) is used to accelerate computation as quickly as possible to handle the high resolution of Full HD video. In this case, a residual network is used to make convergence easier and more stable.

Related functions and parameters:

X: Input image, which is in RGB format, and all colors are in unsigned int8 format;

;

Y=uint8((Conv2dTranspose(σ(Conv2d(X2,a,b,c,d, w _1, b _1)), w _2, b _2)+X2)*128+128);

w_1 is a matrix of size b*b*3*a; b_1 is a vector of size a;

w_2 is a matrix of size b*b*3*a; b_2 is a vector of size 3;

The parameters used include:

The resolution of X is 1280x720;

a=128,b=10,c=5,d=0,σ=leaky relu with alpha=0.2;

a=128,b=9,c=5,d=4,σ=leaky relu with alpha=0.2;

a=128,b=8,c=4,d=0,σ=leaky relu with alpha=0.2;

If the client device has a fast processing speed, the following mathematical formula can be used:

Y=uint8((Conv2dTranspose(R(σ(Conv2d(X2,a,b,c,d, w _1, b _1)), w _R), w _2, b _2)+X2)*128+128);

w_1 is a matrix of size b*b*3*a; b_1 is a vector of size a;

w_2 is a matrix of size b*b*3*a; b_2 is a vector of size 3;

Where R represents residual blocks, which have n layers;

It contains many neural network layers, each with its own weighted parameters for training, collectively referred to as w_R ;

The parameters used include:

a=128,b=8,c=4,d=0, σ =leaky relu with alpha=0.2; n=2;

a=128,b=8,c=4,d=0,σ=leaky relu with alpha=0.2; n=6.

The second scenario: The original image is YUV420, and the output image is RGB or YUV444;

If the input image is YUV420 and the output image is RGB or YUV444, the residual network cannot be directly applied due to the difference in resolution and format between the input and output images. The method of this invention first decodes the YUV420 input image, and then uses another neural network (called network A, where N=3) to process the decoded image and obtain an RGB or YUV444 image (called X2). This X2 image is then fed into the neural network (residual network) described in the first case for training. Furthermore, the same training method is applied to network A to compare the differences between the X2 image and the original image, thereby training network A.

X_y is the Y variable of the input image in YUV420 format, with the format unsigned int8.

X_uv is the UV value of an input image in YUV420 format, with the format unsigned int8.

;

;

X2=Conv2d(X2_y,3,e,1, w _ y , b _ y )+Conv2dTranspose(X2_uv,3,f,2, w _uv, b _uv);

w_y is a matrix of size e*e*1*3; b_y is a vector of size 3;

w_uv is a matrix of size f*f*3*2; b_uv is a vector of size 3;

The above is the first implementation of network A (neural network number A);

Finally, the mathematical formula used to output the image is the same as the formula used in the first case mentioned above when both the input and output images are in RGB format:

Y=uint8((Conv2dTranspose(σ(Conv2d(X2,a,b,c,d, w _1, b _1)), w _2, b _2)+X2)*128+128);

w_1 is a matrix of size b*b*3*a; b_1 is a vector of size a;

w_2 is a matrix of size b*b*3*a; b_2 is a vector of size 3;

The parameters used are the same as those used previously when both the input and output images are in RGB format:

The resolution of X is 1280x720;

a=128,b=8,c=4,d=0,e=1,f=2,σ=leaky relu with alpha=0.2;

a=128,b=8,c=4,d=0,e=1,f=2,σ=leaky relu with alpha=0.2.

Please refer to Figure 8, which illustrates an embodiment of the training process of the neural network of this invention, wherein the original image is YUV420 and the output image is RGB or YUV420. The training process of this neural network includes the following steps:

Step 440: Execute the first application in a training mode to generate a plurality of training source images, wherein the training source images are in RGB or YUV444 format; and perform resolution reduction processing on the training source images (Step 4401);

Step 441: Encode the reduced-resolution original training images into multiple encoded training images using the encoder;

Step 442: The training encoded images are decoded into a plurality of training decoded images using a training decoder on the server; wherein the training decoded images are in YUV420 format;

Step 443: The artificial neural network module includes a first neural network and a second neural network; the first neural network (also referred to as the A network) receives the training decoded images and processes each training decoded image (YUV420) using at least one training mathematical operation to generate a plurality of first output images X2 (also referred to as X2; as in step 444) having the same encoding format as the original training image; the at least one training mathematical operation contains a plurality of training weighting parameters;

Step 445: The second neural network is a Convolutional Neural Network (CNN); the second neural network (also called a CNN network) receives the first output image X2 and processes the first output image X2 one by one using at least one training mathematical operation to generate a plurality of second output images; the at least one training mathematical operation contains a plurality of training weighting parameters; then, the first output image X2 and the second output image are added together (step 446) to generate the training output image (step 447);

The comparison and training module includes a first comparator and a second comparator. In step 448, the first comparator compares the difference between the first output image X2 and its corresponding training original image for training the first neural network. In step 449, the second comparator compares the difference between the training output image and its corresponding training original image for training the second neural network.

Figure 9 is a schematic diagram of an embodiment of the procedure for processing images decoded in YUV420 format according to the present invention. The procedure for processing images decoded in YUV420 format according to the present invention includes:

Step 451: The steps for the first neural network to accept and process training decoded images in YUV420 color format include:

Step 452: Extract the Y-part data from the trained decoded image. Process the Y-part data of the trained decoded image using the neural network of standard size (original size) to generate N-channel Y-part output data (e.g., stride=1 in convolution; as in step 454);

Step 453: Extract the UV-part data from the trained decoded image. Process the UV-part data of the trained decoded image using a neural network with double magnification to generate N-channel UV-part output data (e.g., stride=2 in the transposed convolution; as in step 455);

Step 456: Combine the Y-part output data with the UV-part output data to generate the training output image (Step 457).

The third scenario: The original image is YUV420, and the output image is YUV444, which is processed in a faster way.

If the input image is YUV420 and the output image is YUV444, there is another, faster, way to implement the first neural network (A-network) in addition to the methods described above. The decoded YUV420 image is first converted to YUV444 format (also known as X2) using the first neural network (A-network); then, X2 is fed into the aforementioned neural network (residual network) for training. Furthermore, the same training method is implemented on the A-network to compare the differences between X2 and the original image, thereby training the A-network.

X_y is the Y variable of the input image in YUV420 format, with the format unsigned int8.

X_uv is the UV value of an input image in YUV420 format, with the format unsigned int8.

;

;

X3_uv=Conv2dTranspose(X2_uv,2,2,2, w _ uv , b _ uv );

w_uv is a matrix of size 2*2*2*2; b_uv is a vector of size 2;

X2 = concat(X2_y, X3_uv);

The above is another implementation of network A (neural network number A), where the "concat" function connects the input according to the direction of the channel;

Finally, the mathematical formula used to output the image is the same as the formula used in the first case mentioned above when both the input and output images are in RGB format:

Y=uint8((Conv2dTranspose(σ(Conv2d(X2,a,b,c,d, w _1, b _1)), w _2, b _2)+X2)*128+128);

w_1 is a matrix of size b*b*3*a; b_1 is a vector of size a;

w_2 is a matrix of size b*b*3*a; b_2 is a vector of size 3;

The parameters used are the same as those used previously when both the input and output images are in RGB format:

The resolution of X is 1280x720;

a=128,b=10,c=5,d=0,σ=leaky relu with alpha=0.2;

a=128,b=9,c=5,d=4,σ=leaky relu with alpha=0.2;

a=128,b=8,c=4,d=0,σ=leaky relu with alpha=0.2.

Figure 10 is a schematic diagram of another embodiment of the procedure for processing images decoded in YUV420 format according to the present invention. As shown in Figure 10, the procedure for processing images decoded in YUV420 format according to the present invention includes:

Step 461: The first neural network receives and processes a training decoded image in YUV420 color format using the following steps, wherein the training decoded image includes N channels, and N is a positive integer greater than 2;

Step 462: Extract the Y-part data from the trained decoded image to generate the Y-part output data;

Step 463: Extract the UV-part data from the trained decoded image and process it using a first neural network with double amplification to generate UV-part output data with N-1 channels (e.g., stride=2 in the transposed convolution; as in step 464);

Step 465: Process the Y-part and UV-part data using the concat(concatenates) function to generate the training output image (Step 466).

The fourth scenario: The original image is YUV420, and the output image is also YUV420.

If both the input and output images are YUV420, the processing will be similar to the aforementioned RGB-to-RGB approach. However, due to the different input and output formats, different convolution methods will be applied to different channels. For example, when processing the Y-part of an image using a neural network with a kernel size of 8x8 and a stride of 4, the neural network can be modified to have a kernel size of 4x4 and a stride of 2 to process the UV-part of the image.

X_y is the Y variable of the input image in YUV420 format, with the format unsigned int8.

X_uv is the UV value of an input image in YUV420 format, with the format unsigned int8.

;

;

X3=σ(Conv2d(X2_y,a,b,c, w _ y , b _ y )+Conv2d(X2_uv,a,b/2,c/2, w _ uv , b _ uv ));

w_y is a matrix of size b*b*1*a ; b_y is a vector of size a;

w_uv is a matrix of size (b/2)*(b / 2)*2*a; b_uv is a vector of size a;

X4_y=Conv2dTranspose(X3,1,b,c, w _ 1 , b _ 1 )+X2_y;

X4_uv=Conv2dTranspose(X3,2,b/2,c/2, w _ 2 ,b_2)+X2_uv;

w_1 is a matrix of size b*b*1*a; b_1 is a vector of size 1;

w_2 is a matrix with size (b/2)*(b/2)*2*a; b_2 is a vector with size 2;

The above is another implementation of network A (neural network number A), where the "concat" function connects the input according to the direction of the channel;

Final output:

Y_y = uint8(X4_y * 128 + 128);

Y_uv = uint8(X4_uv * 128 + 128);

Parameters used:

a=128,b=8,c=4,d=0,e=2,f=2,σ=leaky relu with alpha=0.2.

The parameters used in this invention are described in detail below:

Training parameters:

The initial values of this weighting parameter are based on a Gaussian distribution: mean = 0, stddev = 0.02.

The Adam algorithm was used during training, with a learning rate of 1e-4 and beta1 of 0.9.

Mini batch size = 1;

The primary error function is:

100*(L2+L2e)+λ *L1+γ * D+α *Lg;

The standard values for the parameters used are:

λ=0, γ=0, α=0;

λ=0, γ=0, α=100;

λ=0, γ=1, α=0;

λ=10, γ=0, α=0;

λ=10, γ=0, α=100;

λ=10, γ=1, α=0;

in:

L2 = mean (( T - Y ) ^{^2} ); where mean refers to the average value and T is the training target.

L1 = mean (| T - Y |); where mean refers to the average value and T is the training target.

D is the Generative Adversarial Network (GAN) loss, which uses general GAN training methods to train the discriminator to distinguish between (X,Y) and (X,T).

The mathematical expression for Lg is:

For WxH images,

Y_dx(i,j)=Y(i+1,j)-Y(i,j)0<=i<W-1,0<=j<H

T_dx(i,j)=T(i+1,j)-T(i,j)0<=i<W-1,0<=j<H

Y_dy(i,j)=Y(i,j+1)-Y(i,j)0<=i<W,0<=j<H-1

T_dy(i,j)=T(i,j+1)-T(i,j)0<=i<W,0<=j<H-1

L _g = mean (( T _dx - Y _dx ) ² )+ mean (( T _dy - Y _dy ) ² )

In RGB mode, the training target T is the original image of the RGB game video;

In YUV444 mode, the training target T is the original image of the RGB game video;

In RGB->RGB and YUV420->YUV420 modes, L2e=0;

In YUV420->RGB and YUV420->YUV444 modes,

L _{2 e} = mean (( T - X ₂ ) ² ).

As can be seen from the above description, the method of this invention has the following advantages:

It can continuously train the neural network based on various images with different content, so as to apply different enhancement effects to different image content; for example, for images with cartoon style, realistic style, or different scenes, different weighting parameters w and b can be pre-stored in the client device or dynamically downloaded to the client device;

Regarding the method for determining the mode of the original image, the server-side neural network can automatically determine the mode of the original image and transmit this information to the client device. Because the content of the original image is consistent, this determination process can be performed periodically by the server, for example, once per second. However, in another embodiment, the process of determining the image mode can also be performed periodically by the client device, for example, once every few seconds, depending on the computing power of the client device.

The training is conducted using real video footage, allowing for the actual measurement of the enhancement improvement. For example, when using the method of this invention to enhance a 1280x720 resolution video with a bitrate of 3000, the PSNR value of a similar scene can increase by approximately 1.5 to 2.2 dB. This proves that the method of this invention can indeed effectively improve the quality of the output image and make it visually closer to the quality of the original image. Furthermore, this invention differs from well-known image enhancement techniques, which can only increase the contrast, smoothness, and color filtering of the output image, but cannot make the output image visually closer to the original image like this invention.

By employing a simplified model of the neural network algorithm and utilizing large cores and large stride values, the resolution of the neural network is rapidly reduced, while the processing speed of the model can be significantly improved; even client devices with limited computing power can achieve the goal of 60fps and HD resolution output video; and

By incorporating color format conversion (YUV420 and RGB) into the neural network, and leveraging the lower resolution of the UV channel compared to the Y channel by setting the UV channel's step value to half that of the Y channel, the neural network's computational speed can be improved.

Figure 11A is a schematic diagram of a second embodiment of the method of the present invention for reducing the network bandwidth required for video streaming using an artificial intelligence processing module, which includes the following steps:

Step 711: A first application is executed on a server 701. This first application generates a plurality of high-resolution original images according to at least one instruction. The resolution of these original images may be 4K or higher (hereinafter also referred to as a second resolution). The at least one instruction is generated by a client device 702 and transmitted to the server 701 via a network.

Step 712: On server 701, a known sampling method is used to reduce the resolution of the original image to obtain a source image with a low resolution (e.g., 1080i, 720p, or lower, hereinafter referred to as the first resolution). The first resolution is lower than the second resolution.

Step 713: On server 701, an encoder is used to encode and compress the source images to generate a corresponding plurality of encoded images.

Step 714: Server 701, based on instructions from client device 702, transmits the encoded images to client device 702 via the network as a 2D video stream (Step 304). Since the images are downscaled before being transmitted to the client device, the network bandwidth required for transmitting the video stream is also reduced.

Step 715: Client device 702 receives these encoded images and decodes them into a corresponding plurality of decoded images.

In this invention, client device 702 includes an AI processing module comprising at least one preset mathematical expression. The at least one mathematical expression includes a plurality of weighted parameters. These weighted parameters are predefined using a training mode of an artificial neural network module of a training server. A second application, associated with and cooperating with the first application, is executed on client device 702 to allow a user to operate client device 702 to generate the instruction. Client device 702 transmits the instruction to server 701 via a network and receives from the server the encoded image generated according to the instruction.

In this embodiment, the at least one mathematical operation includes a first preset AI operation and a second preset AI operation. The first preset AI operation includes a plurality of first weighting parameters. The second preset AI operation includes a plurality of second weighting parameters. The first preset AI operation, combined with the plurality of the first weighting parameters, can be used to improve the resolution of the image, thereby increasing the resolution of the image processed by the first preset AI operation with the plurality of the first weighting parameters from the first resolution to the second resolution. The second preset AI operation, combined with the plurality of the second weighting parameters, can be used to enhance the image quality, thereby improving the quality of the image processed by the second preset AI operation with the plurality of the second weighting parameters to a higher quality than the decoded image and closer to the quality of the original image.

Step 716: After the client device 702 decodes the received plurality of encoded images into a corresponding plurality of decoded images, the client device first uses the first preset AI formula and a plurality of the first weighting parameters to process the plurality of decoded images to generate a corresponding plurality of resolution-upgraded images with a second resolution. Next, in step 717, the client device 702 uses the second preset AI formula and a plurality of the second weighting parameters to process the plurality of resolution-upgraded images to generate a plurality of high-resolution images with high image quality and the second resolution. Then, as in step 718, the client device 702 outputs these high-resolution images as output images to the screen (display).

The first weighting parameter of the first preset AI algorithm is predefined by analyzing the difference between the low-resolution source image and the corresponding original image, so that the upgraded image visually resembles the original image rather than the source image. Similarly, the second weighting parameter of the second preset AI algorithm is predefined by analyzing the difference between the decoded image and the corresponding original image, so that the high-resolution image visually resembles the original image rather than the decoded image.

Figure 11B is a schematic diagram of a third embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module. Since most of the steps shown in Figure 11B are the same as those shown in Figure 11A, identical or similar steps will be given the same number and their details will not be elaborated. In the embodiment shown in Figure 11B, the first application running on server 701 generates a plurality of source images with a first resolution (step 719). In other words, server 701 directly generates low-resolution source images, so there is no need to execute a separate resolution reduction procedure. These source images are then processed according to steps 713-718 as described in Figure 11A. Since server 701 directly generates low-resolution source images, it consumes fewer computing resources than generating high-resolution original images. Therefore, in addition to the network bandwidth saving advantage of the embodiment described in Figure 11A, the embodiment shown in Figure 11B also has the advantage of saving server computing resources.

Figure 12A is a schematic diagram of a fourth embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module. Since most of the steps shown in Figure 12A are the same as those shown in Figures 11A and 11B, identical or similar steps will be given the same number and their details will not be elaborated. In the embodiment shown in Figure 12A, a first application running on server 701 generates a plurality of original images with a second resolution (step 711). These original images are then processed to reduce their resolution, becoming a plurality of corresponding source images with a first resolution (step 712). Afterwards, these source images are encoded (step 713) into encoded images and transmitted to client device 702 (step 714). Client device 702 decodes the received encoded image into a decoded image (step 715). Then, in step 717 of the embodiment shown in FIG12A, client device 702 first processes the plurality of decoded images using the second preset AI formula and a plurality of the second weighting parameters to generate a plurality of quality-enhanced images with high image quality but still at the first resolution. Next, client device 702 processes the plurality of quality-enhanced images using the first preset AI formula and a plurality of the first weighting parameters to generate a plurality of high-resolution images with the second resolution and high image quality. Then, as in step 718, client device 702 outputs these high-resolution images as output images to the screen.

Figure 12B is a schematic diagram of a fifth embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module. Since most of the steps shown in Figure 12B are the same as those shown in Figures 12A and 11B, identical or similar steps will be given the same number and their details will not be elaborated. In the embodiment shown in Figure 12B, the first application running on server 701 generates a plurality of source images with a first resolution (step 719). In other words, server 701 directly generates low-resolution source images, so there is no need to execute a separate resolution reduction procedure. These source images are then processed according to steps 713-718 as described in Figure 12A.

Figure 13 is a schematic diagram of an embodiment of the training method for the first preset AI operation and the first weighted parameters of the AI processing module described in this invention. In this invention, the first preset AI operation and the complex first weighted parameters in the AI processing module within the client device 702 are predefined by executing a training program for an artificial neural network on the training server. After training is completed, the first preset AI operation and the complex first weighted parameters are applied to the AI processing module of the client device 702 to perform the AI resolution enhancement step as shown in step 716 of Figures 11A, 11B, 12A, and 12B. The steps for training the first preset AI operation and the complex first weighted parameters on the training server include:

Step 7161: Enable a training mode on the training server to generate a plurality of training source images (Step 7162); the plurality of training source images have the second resolution (high resolution).

Step 7163: Perform a resolution reduction procedure to reduce the resolution of a plurality of the training source images from the second resolution to the first resolution, thereby generating a plurality of low-resolution training images with the first resolution (Step 7164).

Step 7165: The artificial neural network module receives and uses a first training operation to process a plurality of the training low-resolution images one by one to generate a plurality of corresponding training output images with the second resolution (Step 7166); the first training operation has a plurality of first training weighting parameters.

Step 7167: Use a comparison module to compare the differences between a plurality of the training output images and the corresponding plurality of the training source images one by one, and adjust the first training weighting parameters of the first training algorithm accordingly. These first training weighting parameters are adjusted to minimize the differences between the training output images and the corresponding training source images. Each time the first training weighting parameters are adjusted, the adjusted first training weighting parameters are fed back to the first training algorithm for processing the next training low-resolution image. After comparing a predetermined number of the training output images with their corresponding original training images, and adjusting a predetermined number of the plurality of the first training weighting parameters, the resulting first training weighting parameters are applied in the AI processing module of the client device 702 as a plurality of weighting parameters for at least one mathematical expression to perform the AI resolution enhancement step described in step 716 as shown in Figures 11A, 11B, 12A, and 12B.

In this embodiment, the training method for the second preset AI formula and the second weighting parameters of the AI processing module of the client device 702 is the same as the training method for the aforementioned artificial neural network module 105 shown in Figures 4, 5, or 6. After training is completed, the obtained second training weighting parameters are applied within the AI processing module of the client device 702 as a plurality of weighting parameters for at least one of the mathematical formulas to perform the AI-enhanced image quality steps described in step 717 as shown in Figures 11A, 11B, 12A, and 12B.

Figure 14A is a schematic diagram of a sixth embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module. Since most of the steps shown in Figure 14A are the same as those shown in Figure 11A, identical or similar steps will be given the same number and their details will not be elaborated. In the embodiment shown in Figure 14A, a first application running on server 701 generates a plurality of original images with a second resolution (step 711). These original images are then processed to reduce their resolution, becoming a plurality of corresponding source images with a first resolution (step 712). Afterwards, these source images are encoded (step 713) into encoded images and transmitted to client device 702 (step 714). The client device 702 decodes the received encoded image into a decoded image (step 715). In this embodiment, the first preset AI formula, the second preset AI formula, the plurality of first weighting parameters, and the plurality of second weighting parameters are all included in the same AI processing module of the client device 702, so as to directly process the plurality of decoded images into a plurality of high-resolution images with high image quality and the second resolution. Therefore, in step 720, the AI processing module of the client device 702 accepts and uses the first preset AI formula, the second preset AI formula, the plurality of first weighting parameters, and the plurality of second weighting parameters to process the decoded images to generate a plurality of high-resolution images with corresponding second resolution. Then, as in step 718, client device 702 outputs these high-resolution images as output images to the screen.

Figure 14B is a schematic diagram of a seventh embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module. Since most of the steps shown in Figure 14B are the same as those shown in Figures 14A and 11B, identical or similar steps will be given the same number and their details will not be elaborated. In the embodiment shown in Figure 14B, the first application running on server 701 generates a plurality of source images with a first resolution (step 719). In other words, server 701 directly generates low-resolution source images, so there is no need to execute a separate resolution reduction procedure. These source images are then processed according to steps 713, 714, 715, 720, and 718 as described in Figure 14A.

Figure 15 is an embodiment of the training method for the first preset AI algorithm, the second preset AI algorithm, the first weighting parameter, and the second weighting parameter of the AI processing module described in this invention. In this invention, the first preset AI algorithm, the second preset AI algorithm, the first weighting parameter, and the second weighting parameter in the AI processing module within the client device 702 are predefined by executing a training program for an artificial neural network on the training server. After training is completed, the first preset AI algorithm, the second preset AI algorithm, the first weighting parameter, and the second weighting parameter are applied in the AI processing module of the client device 702 to perform the AI resolution enhancement step as shown in step 720 of Figures 14A and 14B. The steps for training the first preset AI algorithm, the second preset AI algorithm, the first weighted parameter, and the second weighted parameter on the training server include:

Step 7201: Enable a training mode on the training server to generate a plurality of training source images (Step 7202); the plurality of training source images have the second resolution (high resolution).

Step 7203: Perform a resolution reduction procedure to reduce the resolution of a plurality of the training source images from the second resolution to the first resolution, thereby generating a plurality of low-resolution training images with the first resolution (Step 7204).

Step 7205: Execute an encoding procedure using an encoder within the training server to encode a plurality of the training low-resolution images into a corresponding plurality of trained encoded images.

Step 7206: Execute a decoding procedure using a decoder within the training server to decode the plurality of trained encoded images into a corresponding plurality of trained decoded images; the plurality of trained decoded images have the first resolution.

Step 7207: The artificial neural network module receives and uses a first training operation and a second training operation to process a plurality of the trained and decoded images one by one to generate a plurality of corresponding trained output images with the second resolution (Step 7208). The first training operation has a plurality of first training weighting parameters. The second training operation has a plurality of second training weighting parameters.

Step 7209: Use a comparison module to compare the differences between a plurality of the training output images and the corresponding plurality of the training source images one by one, and adjust the first training weighting parameters of the first training equation and the second training weighting parameters of the second training equation accordingly. The first training weighting parameters and the second training weighting parameters are adjusted to minimize the differences between the training output images and the corresponding training source images. Each time the first training weighting parameters and the second training weighting parameters are adjusted, the adjusted first training weighting parameters and the second training weighting parameters are fed back to the first training algorithm and the second training algorithm to process the next low-resolution training image. After comparing a predetermined number of training output images with their corresponding original training images, and adjusting a predetermined number of the first and second training weighting parameters, the final first and second training weighting parameters are applied within the AI processing module of the client device as weighting parameters for at least one mathematical operation, specifically the first and second training operations, to execute step 720 of Figures 14A and 14B, which involves AI resolution enhancement and image quality improvement.

In a preferred embodiment of the present invention, the AI processing module of the client device 702 includes only a single set of AI formulas and a plurality of weighting parameters, which are trained via steps 7201 to 7209 as described in Figure 15. Therefore, it can also provide the combined function of "AI-enhanced resolution + improved image quality" as described in step 720 of Figures 14A and 14B.

Figure 16 is a schematic diagram of an eighth embodiment of the method of reducing the network bandwidth required for video streaming using an artificial intelligence processing module. Since most of the steps shown in Figure 16 are the same as those shown in Figures 14A and 14B, identical or similar steps will be given the same designation and their details will not be elaborated. In the embodiment shown in Figure 16, server 701 further includes an AI encoding module. A first application running on server 701 generates a plurality of original image images with a second resolution according to instructions (step 721). Then, in step 722, server 701 uses the AI encoding module to reduce the resolution of the plurality of original image images to obtain a corresponding plurality of source images, and to encode the plurality of source images to obtain a corresponding plurality of encoded images. The AI encoding module includes at least one preset AI encoding formula; the at least one AI encoding formula includes a plurality of preset encoding weighting parameters. Then, the encoded image is transmitted to the client device 750 via video streaming (step 714). In this embodiment, the AI processing module of the client device 702 further includes an AI decoding formula for decoding the received encoded image into a corresponding decoded image. In other words, the AI decoding formula, the first preset AI formula, the second preset AI formula, the plurality of first weighting parameters, and the plurality of second weighting parameters are all contained within the same AI processing module of the client device 702, so as to directly process the received plurality of encoded images into a plurality of high-resolution images with high image quality and the second resolution after decoding. Therefore, in step 723, the AI processing module of the client device 702 receives and uses the AI decoding formula, the first preset AI formula, the second preset AI formula, the plurality of first weighting parameters, and the plurality of second weighting parameters to process the encoded images to directly generate a plurality of high-resolution images with corresponding second resolution. Then, as in step 718, client device 702 outputs these high-resolution images as output images to the screen.

Figure 17 is an embodiment of the training method for the AI encoding operation, AI decoding operation, first preset AI operation, second preset AI operation, first weighting parameter, and second weighting parameter of the artificial neural network module described in this invention. In this invention, the AI encoding operation and its weighting parameter in server 701, and the AI decoding operation and its weighting parameter, first preset AI operation, second preset AI operation, first weighting parameter, and second weighting parameter in the AI processing module in client device 702 are all predefined by executing the artificial neural network training program on the training server. After training is complete, the AI encoding formula and its weighting parameters are applied to the AI encoding module of server 701 to execute step 722 (AI encoding step) shown in Figure 16; simultaneously, the AI decoding formula and its weighting parameters, the first preset AI formula, the second preset AI formula, the first weighting parameter, and the second weighting parameter are applied to the AI processing module of client device 702 to execute step 723 (combined processing of AI decoding + resolution enhancement + image quality enhancement) as shown in Figure 16. The steps for training the AI encoding formula and its weighting parameters, the AI decoding formula and its weighting parameters, the first preset AI formula, the second preset AI formula, the first weighting parameter, and the second weighting parameter in the training server include:

Step 7221: Enable a training mode on the training server and execute the first application within the training mode to generate a plurality of training original image images (Step 7222); the plurality of training original image images have the second resolution (high resolution).

Step 7223: Perform a resolution reduction procedure to reduce the resolution of a plurality of the training source images from the second resolution to the first resolution, thereby generating a plurality of low-resolution training images with the first resolution (Step 7224).

Step 7225: A first artificial neural network module is used to receive and process a plurality of the training low-resolution images one by one using a training coding algorithm to generate a plurality of corresponding training coded images with the first resolution (Step 7226); the training coding algorithm has a plurality of training coding weighting parameters.

Step 7227: A second artificial neural network module is used to receive and process a plurality of the trained encoded images one by one using a training decoding algorithm to generate a plurality of corresponding trained output images with the second resolution (Step 7228); the training decoding algorithm has a plurality of training decoding weighting parameters.

Step 7229: Use a comparison module to compare the differences between a plurality of the training output images and the corresponding plurality of the training source images one by one, and adjust the training encoding weighting parameters of the training encoding operation and the training decoding weighting parameters of the training decoding operation accordingly. The training encoding weighting parameters and the training decoding weighting parameters are adjusted to minimize the differences between the training output images and the corresponding training source images. Each time the training encoding weighting parameters and the training decoding weighting parameters are adjusted, the adjusted training encoding weighting parameters and the training decoding weighting parameters are fed back to the training encoding operation and the training decoding operation respectively to process the next training low-resolution image. In step 7220, after comparing a predetermined number of training output images with the corresponding training original images, and adjusting a predetermined number of training encoding weighting parameters and training decoding weighting parameters, the final training encoding weighting parameters are applied to the AI encoding formula of the AI encoding module of the server; and the final training decoding weighting parameters are applied to at least one mathematical formula of the AI processing module of the client device. Therefore, the server's AI encoding module can combine the processes of reducing the resolution of the original image and encoding it, as shown in step 722 of Figure 16; and the client device's AI processing module can combine the processes of decoding, resolution enhancement, and image quality enhancement on the received encoded image, as shown in step 723 of Figure 16.

In a preferred embodiment, the AI processing module of client device 702 contains only a single set of AI formulas and a plurality of weighting parameters, which are trained via steps 7221 to 7229 as described in Figure 17. Therefore, it can also provide the combined function of "AI decoding + AI resolution enhancement + AI image quality enhancement" as described in step 723 of Figure 16.

In one embodiment of the present invention, any of the following existing artificial neural network techniques can be used as the first artificial neural network module to perform AI encoding steps on the server: Autoencoder (AE), Denoising Autoencoder (DAE), Variational Autoencoder (VAE), and Vector-Quantized Variational Autoencoder (VQ-VAE). The second artificial neural network module used in the client device to perform AI decoding, AI resolution enhancement, and AI image quality enhancement can be selected from the following existing artificial neural network techniques: SRCNN, EDSR, RCAN, EnhanceNet, SRGAN, and ESRGAN.

In a preferred embodiment, the plurality of original images may be three-dimensional (3D) images; each 3D image contains a left-eye view and a right-eye view, respectively, combined side-by-side in a single image frame. Therefore, the corresponding output image generated by the client device will also be a 3D image.

In a preferred embodiment, the system according to the invention, which utilizes an artificial intelligence processing module to reduce the network bandwidth required for video streaming, can also be applied to a remote control system for robots. The server of the invention can be a robot, comprising a motion module, a camera module, a communication module, and a control module. The client device of the invention can be a robot control device including a controller module and a display. The robot is remotely connected to the control device via the Internet or other wireless communication technologies. The controller module can be operated by a user to send control commands to the robot, thereby remotely controlling and manipulating the robot's movement and actions. The robot's camera module includes a binocular image capture module to acquire 3D images (left-eye and right-eye views combined side-by-side in a single image frame). Based on control commands received from the control device, the robot can move and perform other actions, and can also capture 3D images of its surrounding environment, then send these 3D images back to the control device and display them on a monitor. Using the method of this invention, the client device (control device) can be equipped with a pre-trained AI processing module; thereby, the robot's binocular image acquisition module only needs to capture a small amount of low-resolution image data and quickly transmit it to the client device with less network bandwidth. The client device can then use the AI processing module to restore the 3D images to high resolution and high image quality. Furthermore, because the robot captures and processes low-resolution images, its required computing resources are relatively lower and it is more power-efficient, which can extend the robot's remote operation time.

The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of its implementation. The scope of protection of the present invention shall be determined by the contents of the patent application. Any modifications and alterations to the present invention made by those skilled in the art may not depart from the spirit and scope of the invention and shall still be covered by the contents of the patent application.

701: Server

702: Client Device

711-718: Steps

Claims (3)

A method for reducing network bandwidth required for video streaming using an artificial intelligence processing module includes: step (A): executing a first application in a server to generate a plurality of source images corresponding to a plurality of original image images; the plurality of source images having a first resolution; the plurality of source images being encoded and compressed by an encoder within the server to generate a plurality of corresponding encoded images; step (B): executing... A second application is executed; the second application is associated with and cooperates with the first application; Step (C): The client device connects to the server via a network and then receives the encoded images generated by the server via the network in the form of a video stream; Step (D): The client device decodes the encoded images into a corresponding plurality of decoded images and uses an artificial intelligence (AI) device built into the client device. Intelligent (AI) processing module to improve the resolution of the decoded images to generate a plurality of high-resolution images; the plurality of high-resolution images have a second resolution; wherein the second resolution is higher than the first resolution, and the resolution of the plurality of original images is equal to the second resolution; Step (E): The client device sequentially outputs the plurality of high-resolution images to a screen as a plurality of output images to be played; wherein the AI processing module obtains at least one mathematical formula in advance by analyzing the differences between the decoded images and the corresponding original images, and A plurality of weighted parameters are used to process the decoded images; thereby, the resolution of the obtained high-resolution images will be equal to the resolution of the corresponding original images and higher than the resolution of the plurality of source images; the at least one mathematical operation and the plurality of weighted parameters of the AI processing module are defined in advance by a training program executed by an artificial neural network module in a training server; characterized in that: the at least one mathematical operation of the AI processing module set in the client device includes a first preset AI operation and a second preset AI operation that are trained and have different functions; the first preset AI operation includes a plurality of weighted parameters. The first preset AI algorithm includes several first weighting parameters; the second preset AI algorithm includes a plurality of second weighting parameters. The first preset AI algorithm, combined with the plurality of the first weighting parameters, is used to improve the resolution of the image, thereby increasing the resolution of the image processed by the first preset AI algorithm with the plurality of the first weighting parameters from the first resolution to the second resolution. The second preset AI algorithm, combined with the plurality of the second weighting parameters, is used to enhance the image quality, thereby improving the quality of the image processed by the second preset AI algorithm with the plurality of the second weighting parameters to a higher quality than the decoded image. Furthermore, the quality is closer to that of the original image; wherein, after the client device decodes the received plurality of encoded images into a corresponding plurality of decoded images, the client device processes the plurality of decoded images using one of the following methods: Method 1: The client device first uses the first preset AI formula and a plurality of the first weighting parameters to process the plurality of decoded images to generate a corresponding plurality of resolution-upgraded images with a second resolution; then, the client device uses the second preset AI formula and a plurality of the second weighting parameters to process the plurality of resolution-upgraded images to generate images with high image quality. Method 1: The client device first uses the second preset AI algorithm and a plurality of the second weighting parameters to process a plurality of the decoded images to generate a plurality of quality-enhanced images with the first resolution and high image quality; then, the client device uses the first preset AI algorithm and a plurality of the first weighting parameters to process a plurality of the quality-enhanced images to generate a plurality of high-resolution images with the second resolution and high image quality; wherein, the training program of the artificial neural network module executed in the training server includes a dedicated training... The following steps are performed to train the first preset AI algorithm and a plurality of the first weighting parameters: A training mode is activated in the training server to generate a plurality of training original images; the plurality of training original images have the second resolution; a resolution reduction procedure is executed to reduce the resolution of the plurality of training original images from the second resolution to the first resolution, thereby generating a plurality of training low-resolution images with the first resolution; without encoding the plurality of training low-resolution images with the first resolution, the artificial neural network module receives and uses a first training algorithm to process the plurality of training low-resolution images one by one. The image generates a plurality of training output images with the second resolution; the first training algorithm has a plurality of first training weighting parameters; and without decoding the plurality of training output images, a comparison module is used to compare the differences between the plurality of training output images and the corresponding plurality of training source images one by one, and adjust the first training weighting parameters of the first training algorithm accordingly; the first training weighting parameters are adjusted to minimize the differences between the training output images and the corresponding training source images; each time the first training weighting parameters are adjusted, the adjusted first training weighting parameters are used to generate a plurality of training output images with the second resolution. Each training weighting parameter is fed back to the first training algorithm to process the next low-resolution training image. After comparing a predetermined number of training output images with their corresponding original training images, and adjusting a predetermined number of the first training weighting parameters, the final first training weighting parameters of the first training algorithm are applied to the AI processing module of the client device as multiple first weighting parameters of the first preset AI algorithm. The training procedures for training the second preset AI algorithm are different from those for training the first preset AI algorithm. As described in claim 1, wherein the plurality of encoded images in step (A) are generated by the following steps: executing the first application on the server to generate a plurality of original images having the second resolution; using a resolution reduction procedure to reduce the resolution of the plurality of original images to the first resolution to obtain a corresponding plurality of source images; and using the encoder to encode the plurality of source images to obtain a corresponding plurality of encoded images. The method described in claim 2, wherein the plurality of original images are three-dimensional (3D) images, each of the 3D images comprising a left-eye view and a right-eye view arranged side-by-side in a single image frame.