HK40070373B - Video processing method and related apparatus - Google Patents

Description

A video processing method and related apparatus

Technical Field

This application relates to the field of data processing, and in particular to a video processing method and related apparatus.

Background Technology

For game applications, in addition to providing game services to users through the game itself, they can also provide immersive experiences by setting up offline game environments.

Currently, offline experiences mainly use physical game props (such as clothing, grips, and accessories) to provide users with a game role-playing experience. However, frequently changing into and out of props is not only inconvenient but also makes it difficult to create a sense of immersion for users.

Therefore, it is clear that providing a high-quality offline immersive gaming experience is a pressing technical problem that needs to be solved.

Summary of the Invention

To address the aforementioned technical issues, this application provides a video processing method and related apparatus that generates target videos to create the illusion of physical objects acting as target game objects within a target game scene. This not only provides a better immersive experience but also adds entertainment value.

The embodiments of this application disclose the following technical solutions:

On one hand, embodiments of this application provide a video processing method, the method comprising:

Acquire video data collected for a display area, the display area being used to display game environment images of the target game, and the display area containing entity objects;

Determine the shape parameters of the entity object in the video data;

Based on the shape parameters, the target game object is determined from among the multiple game objects in the target game;

The virtual props corresponding to the target game object are rendered onto the entity object in the video data to obtain the target video.

On the other hand, embodiments of this application provide a video processing apparatus, the apparatus including an acquisition unit, a determination unit, and a rendering unit:

The acquisition unit is used to acquire video data collected from the display area, which is used to display the game environment image of the target game and has entity objects in the display area;

The determining unit is used to determine the shape parameters of the entity object in the video data;

The determining unit is further configured to determine the target game object among multiple game objects in the target game based on the shape parameters;

The rendering unit is used to render the virtual props corresponding to the target game object onto the entity object in the video data to obtain the target video.

On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

The memory is used to store program code and transmit the program code to the processor;

The processor is configured to execute the methods described above according to instructions in the program code.

On the other hand, embodiments of this application provide a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program, the computer program being used to execute the methods described above.

On the other hand, embodiments of this application provide a computer program product including instructions that, when run on a computer, cause the computer to perform the methods described above.

As can be seen from the above technical solution, the set display area can display the game environment image of the target game. When a physical object enters the display area, video data can be collected for the display area. Since the game environment image displayed in the display area serves as the background of the physical object, the physical object in the video data appears to be in the scene of the target game, creating a sense of presence. By identifying the shape parameters of the physical object in the video data, the target game object can be determined from multiple game objects in the target game based on the shape parameters. The virtual props corresponding to the target game object are then rendered onto the physical object in the video data to obtain the target video. Since the target game object is determined based on the shape parameters and has a correlation with the physical object in appearance, it gives the impression that the physical object is playing the role of the target game object in the target game scene, which not only brings a better immersive experience but also adds fun.

Attached Figure Description

To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

Figure 1 is a schematic diagram of a video processing scenario provided in an embodiment of this application;

Figure 2 is a flowchart of a video processing method provided in an embodiment of this application;

Figure 3 is a schematic diagram of key joint points of the human body provided in an embodiment of this application;

Figure 4 is a schematic diagram of special effects rendering for different human bodies provided in an embodiment of this application;

Figure 5a is a schematic diagram of a video processing flow provided in an embodiment of this application;

Figure 5b is a schematic diagram of an offline experience scenario provided in an embodiment of this application;

Figure 6a shows one of the display contents of an operation console provided in an embodiment of this application;

Figure 6b shows a second display content of an operating console provided in an embodiment of this application;

Figure 6c shows a third display content of an operation console provided in an embodiment of this application;

Figure 7 is a game scene diagram of a three-sided screen display provided in an embodiment of this application;

Figure 8 is a rendering effect of a virtual prop provided in an embodiment of this application;

Figure 9a is a schematic diagram of a single-person experience provided in an embodiment of this application;

Figure 9b is a schematic diagram of a multi-user experience provided in an embodiment of this application;

Figure 10 is a structural diagram of a video processing apparatus provided in an embodiment of this application;

Figure 11 is a structural diagram of a terminal device provided in an embodiment of this application;

Figure 12 is a structural diagram of a server provided in an embodiment of this application.

Detailed Implementation

The embodiments of this application will now be described with reference to the accompanying drawings.

Building high-quality offline immersive gaming experiences is an important way to enhance user engagement and expand the game's audience. However, the relevant technologies mainly rely on the production of physical game props, and the complex operation makes it difficult to achieve a sufficient sense of immersion.

Therefore, this application provides a video processing method that generates a target video to give the impression that an entity object is playing the role of a target game object in a target game scene, which not only brings a better immersive experience but also adds fun.

The video processing method provided in this application can be implemented using computer equipment, which can be a terminal device or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, and vehicle terminals. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited herein. This application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

The embodiments of this application can be applied to artificial intelligence (AI). AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating/interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning/deep learning, autonomous driving, and intelligent transportation.

This application relates to Computer Vision (CV) technology. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and further processing the images to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, 3D object reconstruction, 3D technology, virtual reality, and augmented reality. For example, this application can use computer vision technology to recognize the actions of entities in video data.

It is understood that, in the specific embodiments of this application, data such as shape parameters that may be related to user information, when applied to specific products or technologies, require separate permission or consent from the user, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Figure 1 is a schematic diagram of a video processing scenario provided in an embodiment of this application. In Figure 1, the terminal device 100 is used as an example of the aforementioned computer device for illustration. The camera 200 is used to collect video data from the display area 300, and the terminal device 100 acquires the collected video data through the camera 200. The game environment image of the target game is displayed in the display area 300.

During the process of camera 200 acquiring video data, user a, as an entity, is located in display area 300. Since the game environment image displayed in the display area serves as the background for user a, user a appears to be in the scene of the target game in the video data, creating a sense of presence.

Terminal device 100 can determine the physical parameters of user a based on video data, such as a height of 1.8 meters. Based on the height parameter, it can determine the target game object that matches user a among multiple game objects in the target game, such as a tall male fighter. The virtual prop corresponding to the male fighter, such as boxing gloves, is rendered onto user a in the video data to obtain the target video, as shown in the display interface 20 of terminal device 100.

Since the male fighter determined based on height parameters is similar in height to user A, user A, who is rendered with boxing gloves in the target video, will look more like the male fighter in the target game. This gives the impression that user A is playing the male fighter in the target game scene, which not only brings a better immersive experience, but also adds fun and attracts users to try to play which game character they can play.

Figure 2 is a flowchart of a video processing method provided in an embodiment of this application. In this embodiment, a terminal device is used as the aforementioned computer device for illustration. The method includes:

S201: Acquire video data collected for the display area.

The display area is used to display the game environment image of the target game. When collecting video data, there are physical objects in the display area. In the video acquisition direction, the physical objects are located between the acquisition device and the display area, so that the physical objects in the acquired video data are in the background of the game environment image, which can bring a sense of presence to the target game scene.

The game environment image can be a static image or a dynamic image; this application does not limit this.

An entity object can be any actual object with a physical form, such as a human body, an animal, or various objects, and this application does not limit it.

To enhance the sense of presence in the display area, in one possible implementation, the display area is a three-sided screen, with the three display screens arranged perpendicularly to each other.

The three displays of the three-sided screen can be vertically arranged to form a rectangular corner space. The three-dimensional display area can bring a three-dimensional display effect and improve the sense of realism. When a physical object is in the display area of the three-sided screen, there are realistic game environment images on the left, right, top or bottom sides, which enhances the sense of being in the target game scene.

This application does not limit the number and type of entity objects in the video data. There can be multiple entity objects or one entity object. Entity objects can all be of the same type, such as all being human bodies, or they can include different types, such as human bodies and animals.

S202: Determine the shape parameters of the entity object in the video data.

Shape parameters are used to identify the shape characteristics of an entity object. These characteristics can be represented by its size or appearance. Thus, the terminal device can effectively describe the physical features of the entity object through these shape characteristics, facilitating the subsequent identification of target game objects that match the entity object in shape.

When the entity object is a human body, S202 includes: determining at least one of the facial features, height parameters, and body shape parameters of the human body in the video data as the shape parameter.

It's important to note that while the defined appearance parameters include at least one of the following: facial features, height, and body shape, these parameters are only used for temporary game object matching. After matching is complete, these parameters will not be saved or used for other purposes. Furthermore, the defined appearance parameters are relatively coarse-grained; they only need to reflect the physical characteristics and will not involve user privacy information.

When the entity is an animal, the shape parameters can also identify the animal's physical characteristics, species characteristics, etc.

When the entity object is a physical object, the shape parameters can also identify the object's structural features and physical form.

S203: Based on the shape parameters, determine the target game object among the multiple game objects of the target game.

Multiple game objects in the target game have corresponding virtual appearances, which can be represented, for example, by virtual models. Based on the virtual appearances, the visual characteristics of the game objects can be determined, while shape parameters can represent the physical characteristics of the objects. Therefore, the terminal device can determine the target game object from multiple game objects based on the shape parameters.

The identified target game object is correlated in appearance with the entity object in the video data. This correlation can show a close match or a significant difference in appearance. A close match makes the entity object in the target video appear to be playing the target game object, while a significant difference creates a sense of contrast when the entity object in the target video is rendered as a virtual prop, adding fun to the offline experience.

This application does not limit the type of game object; it can be game characters of various forms and types. The specific appearance and type of the game object can be related to the target game.

In one possible implementation, S203 includes: determining a target game object that matches the entity object among a plurality of game objects in the target game based on the shape parameters.

In this method, the matched target game object is similar in appearance to the physical object. For example, a tall physical object can be matched with a tall game character, a small physical object can be matched with a small game character, and a dog can be matched with a virtual dog avatar in the target game.

Because the target game object is similar in appearance to the physical object, the physical object rendered with virtual props in the target video will be more similar to the target game object, giving the impression that the physical object is playing the role of the target game object in the target game scene.

S204: Render the virtual props corresponding to the target game object onto the entity object in the video data to obtain the target video.

Virtual props corresponding to the target game object can be used to identify the appearance of the target game object and serve to distinguish it from other game objects to a certain extent. These virtual props can be items used for decoration or worn on the target game object in the target game, such as at least one of virtual accessories, clothing, skins, hair, or holding props. The virtual props may also include additional virtual props related to the target game object, such as virtual pets or halo effect materials; this application does not limit this.

Since the matched target game object and the physical object are similar in appearance, the terminal device renders the virtual props corresponding to the target game object onto the physical object in the video data. This makes the physical object appear to be playing the target game object in the target video, giving the impression that the physical object is playing the target game object in the target game scene. Playing the target video can bring a better immersive experience and can also attract more participants to try and determine which target game object they can match, further enhancing the fun of the offline experience.

As mentioned earlier, the target game object determined by the shape parameters may differ in appearance from the entity object in the video data (for example, there may be some differences even when the shapes match, and the differences are greater when the shapes are significantly different). Therefore, in order to better render virtual props and improve realism, in one possible implementation, the method further includes:

S41: Determine the difference in appearance between the entity object and the target game object in the video data based on the game image parameters and the appearance parameters of the target game object.

S42: Adjust the game appearance parameters of the target game object according to the appearance differences to obtain adjusted appearance parameters that are compatible with the appearance parameters.

The corresponding S204 includes: rendering the virtual props corresponding to the target game object onto the entity object in the video data according to the adjusted image parameters, thereby obtaining the target video.

This shape difference identifies the difference between the target game object's shape and the physical object in the video data. After adjusting the game image parameters of the target game object based on the determined shape difference, a bar-shaped image parameter is obtained. This adjusted image parameter can identify the mapping relationship between virtual props and physical objects in the video data. This allows the virtual props of the target game object to be rendered on the physical objects in the video data when they are rendered on the physical objects. The virtual props, which originally had different shapes, can be rendered on the physical objects in the video data in a way that is consistent with the proportions of the physical objects. As a result, the picture is more beautiful and has a better sense of realism.

Therefore, the set display area can display the game environment image of the target game. When a physical object enters the display area, video data can be captured for that area. Since the game environment image displayed in the display area serves as the background for the physical object, the physical object in the video data appears to be in the target game scene, creating a sense of immersion. By identifying the shape parameters of the physical object in the video data, the target game object can be determined from multiple game objects in the target game based on these parameters. The virtual props corresponding to the target game object are then rendered onto the physical object in the video data, resulting in the target video. Because the target game object is determined based on shape parameters and has a correlation with the physical object in appearance, it gives the impression that the physical object is playing the role of the target game object in the target game scene, providing not only a better immersive experience but also added entertainment value.

In one possible implementation, S203 includes:

S2031: Obtain the game character parameters corresponding to the multiple game objects respectively.

S2032: Based on the similarity comparison between the appearance parameters and the game character parameters, determine the target game object from game objects whose similarity meets the threshold.

Game character parameters identify the visual characteristics of the corresponding game object, while appearance parameters reflect the physical appearance of the object. Therefore, a similarity comparison can be performed between appearance parameters and game character parameters to determine the target game object from game objects whose similarity meets a threshold. This threshold can be set to a relatively high value.

There can be at least one game object whose similarity meets the threshold. If there are multiple similar game objects, the one with the highest similarity can be randomly selected as the target game object.

To further improve the similarity between entity objects and target game objects, when the entity object is a human body, a gender parameter can be introduced as a matching criterion. The gender parameter can identify male or female.

In one possible implementation, the method further includes: obtaining the sex parameter of the human body.

Accordingly, S203 includes:

S2033: Determine the candidate game object that matches the gender parameter from the plurality of game objects.

S2034: Based on the shape parameters, determine the target game object that matches the entity object from the list of undetermined game objects.

This scenario primarily involves target games that differentiate game objects by gender. Based on the gender parameter of an entity object, a candidate game object matching that gender can be identified from multiple game objects. Then, the target game object is determined from the candidate game objects using the entity object's shape parameter. This ensures that the identified target game object matches the gender of the entity object, thus improving the user experience.

In some applications, the number of entities included in video data is not limited; one or more entities can be included in the display area. Different processing methods can be used for different numbers of entities.

In one possible implementation, the method further includes:

S11: Determine the number of entities in the video data. If the number of entities is a specified number, execute S12. If the number of entities is not the specified number, execute S14.

This application does not limit the specific numerical value of the specified quantity; for example, it can be 1 or an integer greater than 1. The number of objects can be the number of entity objects of a certain type (e.g., including several human bodies) or the total number of entity objects of all types (e.g., how many human bodies and animals are included in total).

S12: Identify the actions performed by the entity object in the video data.

S13: In response to the action being a specified action, perform the operation of rendering the virtual props corresponding to the target game object onto the entity object in the video data to obtain the target video.

S14: Execute the operation of S204 to render the virtual props corresponding to the target game object onto the entity object in the video data to obtain the target video.

When the number of entity objects reaches a specified quantity, the decision to retrieve the target video is made based on whether the actions performed by the entity objects meet the requirements. This mechanism can bring additional interactive elements to the offline experience by guiding entity objects to perform specified actions, thereby enhancing their sense of participation.

This application does not limit the specified action; it can be related to the type of entity object, the specified quantity, or the target game. For example, when the entity object is a human body, if the specified quantity is 1, the specified action can be a single action such as opening arms or making a heart shape with hands. If the specified quantity is an integer greater than 1, the specified action can be a combined action or multiple single actions. A combined action can require a specified number of entity objects to cooperate in performing a complete action, such as multiple people holding hands or multiple people making a heart shape with hands.

For the human body as a physical object, the actions performed by the physical object in video data can be identified by recognizing the positions of the key joints of the human body.

In one possible implementation, S12 includes:

S121: Determine the positions of the specified number of key joints of the human body in the video data.

S122: Identify the actions of the specified number of human bodies in the video data based on the positions of the joint key points.

The key points of the human joints are used to identify the joints of the human body, such as the shoulder, neck, elbow, ankle, and pelvis, which can produce displacement and rotation. The key point locations are used to identify the positions of the corresponding key points. Based on the key point locations, the actions and postures performed by the human body can be identified.

By identifying the video frames included in the video data, the positions of the key joints of the human body in each video frame can be determined. Then, by using the position of the key joints and the temporal relationship between the video frames, it can be determined what kind of action a specified number of human bodies are performing in the video data.

The identification can be performed by selecting the position of the corresponding joint key point based on the specified action to be identified, or by identifying the position of the default joint key point. This application does not limit this.

In this embodiment, besides determining the corresponding target game object based on the shape characteristics of the physical object, richer interaction methods can also be provided. For example, virtual props can be rendered on the physical object based on the game object selected by the user. That is to say, in addition to providing AR rendering of game objects that match the shape of the physical object, AR rendering based on a specified object can also be performed based on the user's selection.

Therefore, in one possible implementation, the method further includes:

S21: Obtain a game object selection instruction for a specified object, wherein the specified object is one of the plurality of game objects.

S22: Render the virtual props corresponding to the specified object onto the entity object in the video data to obtain the specified video.

This method allows users to choose any game object they want to role-play using a physical object, increasing the diversity and selectivity of virtual prop rendering. For example, if a user, as a physical object, wants to role-play a game object that doesn't match their physical characteristics, they can generate a game object selection command by selecting that object. Then, the terminal device renders the video based on the virtual prop corresponding to the specified object on the physical object in the video data, resulting in the specified video. This satisfies users' needs to role-play game objects of different genders and appearances.

The following example illustrates the specific scenario where the specified quantity is 1 and the specified action is "open your arms".

The terminal device can recognize the specified action of "opening both hands" using a recognition model. This model can identify the positions of multiple key joints on the human body. For example, in the scenario shown in Figure 3, for the right hand, by recognizing the positions of key joints 6 (right shoulder), 8 (right elbow), and 10 (right wrist), it can be determined whether the person has made the action of opening their right hand. Similarly, by recognizing the positions of key joints 5 (left shoulder), 7 (left elbow), and 9 (left wrist), it can be determined whether the person has made the action of opening their left hand. Thus, once the conditions are met, the device can recognize that the person has made the action of "opening both hands".

For example, by calculating the angles of the wrist and elbow, and the angles of the shoulder and elbow, the action is defined as "open arms" when the angle between the left and right shoulders and elbows is less than 180 degrees and the angle between the wrist and elbow is greater than 25 degrees. The code can be shown below:

if(left_shoulder_angleA>25&&left_shoulder_angle A<180

&&left_shoulder_angleB>25&&left_shoulder_angleB<180

&&right_shoulder_angleA>25&&right_shoulder_angleA<180

&&right_shoulder_angleB>25&&right_shoulder_angleB<180){

armOpen = 1;

}else{

armOpen = 0;

}

A and B refer to left and right, respectively.

After determining that the human body has made the "open arms" action, the position and size of virtual props and virtual effects can be calculated by identifying the positions of key joints 6 (right shoulder), 12 (right hip), 16 (right ankle), and 5 (left shoulder), so as to match different styles of props and effects to human bodies of different positions and heights.

The terminal device of this application can have different props and special effects matched to people of different positions and heights.

Therefore, in one possible implementation, the method further includes: determining the video position of the entity object in the video data.

Accordingly, S204 includes: determining the display form corresponding to the virtual prop based on the video position; rendering the virtual prop onto the entity object in the video data based on the display form to obtain the target video.

Within the display area, an entity object can be positioned anywhere; this position is recorded as the video position in the video data. Consequently, the size of the same entity object will differ depending on its position within the captured video data.

Therefore, in order to better render virtual props onto the physical objects in the video data, at least two different display formats can be set for virtual props corresponding to different game objects, based on the distance and offset relationship between the physical object's video position and the video capture device. Before rendering, the display format corresponding to the virtual prop is selected based on the video position of the physical object in the video data, and the virtual prop is rendered onto the physical object in the video data according to the display format to obtain the target video, thereby improving the realism of the virtual prop rendering.

As shown in Figure 4, the left and right images correspond to different human figures (entity objects). For example, the human figure in the left image is taller than the human figure in the right image, and is positioned further back. The special effects in the left image are larger than those in the right image. The special effects include a flying dragon flashing towards the human figure's chest, and virtual props corresponding to the target game object are rendered on the human figure.

The code can be as follows:

let left＝(left_hip.y-left_shoulder.y)/2+(left_shoulder.y);

let humanMidx=1920-(1920-right_shoulder.x)+(1920-right_shoulder.x)-(1920-left_shoulder.x))/2;

let humanMidy;

if(left_shouldlr.y>right_shoulder.y){

humanMidy=left_shoulder.y+(left_shoulder.y-right_shoulder.y)/2;

}else{

humanMidy=right_shoulder.y+(right_shoulder.y-left_shoulder.y)/2;

}

let topHeight=1080-humanMidy;

Where x and y represent the horizontal and vertical coordinates of the joint key points, respectively.

The code above calculates the midpoint between the left and right shoulders and hips of the human body as the midpoint for rendering the virtual prop, and calculates and sets the proportion of the virtual prop based on the distance from the person's shoulders to their feet.

The identified target video can be an augmented reality (AR) video without music. AR technology is a technology that cleverly integrates virtual information with the real world. It simulates and applies computer-generated text, images, 3D models, music, videos, and other virtual information to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

The target video can be generated by calling an interface used for video creation. This interface records the dynamic content of the canvas in the video data. The canvas depicts the real content captured by the camera and virtual props (or virtual effects, etc.). The content related to the human body and the three screens is the real content captured by the camera, while the virtual props are multiple transparent AR videos drawn at specified positions on the canvas. The final result is a WEBM format AR video (i.e., the target video), which demonstrates the interaction between the human body and the virtual props. Specifically, the recorded information can be stored in a `data` array. After recording, the `data` is converted into a file to obtain the target video.

Since the target video may be silent in some scenarios, audio-video fusion is necessary to incorporate background music. The ffmpeg tool (an open-source, free software capable of recording, converting, and streaming various audio and video formats) is used for this audio-video fusion. This tool enables the fusion of silent WEBM + MP3 (background audio) in this solution. The acquired silent video is then fused with the server's audio using ffmpeg commands to obtain a target video with background sound. This target video can be an MP4 file.

After generating the target video, the system can also provide users with a way to access it. This allows users to save the target video on their devices and watch or share it repeatedly when needed.

In one possible implementation, the method further includes:

S31: Save the target video to the storage location.

S32: Generate a corresponding address code based on the storage location, and display the address code.

S33: If a video download request sent by a terminal device is obtained based on the address code, the target video is returned to the terminal device from the storage location.

After the target video is generated, it can be saved. The target video can be stored locally on the terminal device or on a cloud server.

After storing the target video, an address code can be generated based on the storage location. This address code can be in digital form such as a QR code or barcode.

Users can generate and send a video download request by scanning an address code using their terminal device, such as a mobile phone. The terminal device, which is the aforementioned computer device, can then receive the video download request and return the target video from its storage location to the user's terminal device.

For example, after a user scans a QR code (which serves as an address code) via a mini-program, the terminal device obtains the relevant parameters and uses them to make a network request. It then checks if the storage address of the target video (which can be a link) is valid. If the link is valid, it accesses the target video and asks the user if they want to save it. If the link is invalid, it continues the request until the target video, originally in WEBM format, is successfully converted to MP4 format in the cloud, at which point it can be accessed.

Continuing with Figures 5a and 5b, the video processing method provided in this application is further described. As shown in Figure 5b, the offline scenario includes a terminal device 100, a camera 200, a three-sided screen 300, and an operating table 400. The specified quantity is 1, and the specified action is opening both hands.

In other words, it includes two main units and three screens. The two main units are the control panel and the terminal device. The control panel also serves as a server. The control panel is used to select the interactive experience mode and confirm or delete the final generated video. The terminal device is used to record immersive interactive AR videos of the user and hero effects.

The three screens are connected to the control terminal and the display terminal, as well as a large external screen that is synchronized with the display terminal in real time. In addition, in order to better restore the game scene, three screens are also configured in the offline environment to play the game scene in a loop. Users can immerse themselves in the thrill of becoming a hero character on the three screens.

The user can be treated as a physical object, and the camera 200 can capture video data when the user is within the display area 300 of the three-sided screen. Based on the recognition and matching of the video data, the terminal device 100 renders the virtual props of the target game object onto the user in the video data.

Before implementing the embodiments of this application, the display interface 10 of the operating console 400 may be as shown in FIG6a, including steps one, two, and three for participating in the offline experience, as well as single and multi-person options.

Step one instructs the user to enter the video capture area (i.e., the display area); step two instructs the rendering of virtual props; and step three instructs the user to scan a QR code to obtain the target video. Users can also choose to participate individually or in groups, depending on the number of participants.

As shown in Figure 5a, in the case of a single person participating, the start can be initiated by triggering the "Single Person" control in Figure 6a or by triggering the "Multi-Person" control in Figure 6a.

After clicking "Start", the control panel 400 displays the content shown in Figure 6b. The user is instructed to enter the display area 300 by saying "Please go to the photo area for interaction". The camera 200 collects the corresponding video data. The content displayed in the display area 300 can be as shown in Figure 7.

The control panel 400 can also display the content shown in Figure 6c, and instruct the user to perform a specified action through the image and text "Please raise your arms and open them to summon the dragon".

Terminal device 100 determines whether the user has made an action of opening their arms based on video data. If the recognition is successful, it determines whether to click on the special effect, that is, whether to render the virtual prop.

When determining the click effect, the terminal device 100 renders the corresponding virtual props onto the user in the video data based on the determined target game object, and generates the target video. The target video can be shown in Figure 8, which shows the situation where two users are simultaneously in the display area 300, and is displayed through the display interface 20 of the terminal device 100.

In Figure 8, the virtual items of the target game object matched with the user on the left are clothing and headwear, while the virtual item of the target game object matched with the user on the right is a machine gun hanging on the shoulder. Accordingly, in this case, the user can trigger the "multiplayer" control in Figure 6a.

After the terminal device 100 processes the target video, it stores it in the storage location and generates an address code based on the storage location. This address code can be displayed on the terminal device 100 in the scenario shown in Figure 5b, or it can be displayed on the operation console 400.

Users scan the address code via the mini-program. If authorized, the terminal device 100 will return the target video to the user and save it in the user's phone's album. If not authorized, the target video cannot be obtained.

The single-player experience flow is shown in Figure 9a. The user clicks the single-player shooting button on the control panel. The user then stands in the optimal shooting area of the display area with arms outstretched. Based on the transmitted video data, the current environment captured by the camera is displayed on the large screen of the terminal device. After recognizing the outstretched arms gesture, the terminal device adapts the virtual prop effects of the game hero according to the user's position and height, rendering them together with the content captured by the camera. The user can then interact with the hero effects. After the interaction, a target video for the entire AR interactive experience is generated. After the target video is captured, audio can be integrated. The target video is then displayed on the control panel. After the user clicks to save the video, the terminal device saves the target video and generates a corresponding address code. This address code is transmitted to the display end for display, and the user can scan the code using a mini-program to save it.

The multi-player experience flow can be seen in Figure 9b, where users click the multi-player shooting button on the control panel. Because in some situations, when multiple people have their arms outstretched, the recognition using tendorflow.js can cause mutual interference, affecting the recognition of actions, it is optional to actively trigger the camera and hero effects on the control panel. The interaction after activation is the same as the single-player experience, and will not be elaborated further.

The embodiments of this application can be implemented on the web platform, and create an immersive game character experience through the combination of technology and offline environment.

Based on Figures 1-9b above, Figure 10 is a structural diagram of a video processing apparatus provided in an embodiment of this application. The video processing apparatus 1000 includes an acquisition unit 1001, a determination unit 1002, and a rendering unit 1003.

The acquisition unit 1001 is used to acquire video data collected from a display area, the display area being used to display game environment images of the target game, and having entity objects in the display area;

The determining unit 1002 is used to determine the shape parameters of the entity object in the video data;

The determining unit 1002 is further configured to determine the target game object among multiple game objects of the target game based on the shape parameters;

The rendering unit 1003 is used to render the virtual props corresponding to the target game object onto the entity object in the video data to obtain the target video.

In one possible implementation, the determining unit is further configured to determine, based on the shape parameters, a target game object that matches the entity object among a plurality of game objects in the target game.

In one possible implementation, the determining unit is further configured to:

Based on the game image parameters and appearance parameters of the target game object, determine the appearance differences between the entity object in the video data and the target game object;

Adjust the game appearance parameters of the target game object according to the appearance differences to obtain adjusted appearance parameters that are compatible with the appearance parameters;

The rendering unit is also used to render the virtual props corresponding to the target game object onto the entity object in the video data according to the adjusted image parameters, so as to obtain the target video.

In one possible implementation, if the entity object is a human body, the determining unit is further configured to determine at least one of the facial features, height parameters, and body shape parameters of the human body in the video data as the shape parameter.

In one possible implementation, the determining unit is further configured to:

Obtain the game character parameters corresponding to the multiple game objects;

The target game object is determined from game objects whose similarity meets the threshold by comparing the appearance parameters with the game character parameters.

In one possible implementation, the acquisition unit is further configured to acquire the gender parameter of the human body;

The determining unit is further configured to:

From the plurality of game objects, determine the candidate game object that matches the gender parameter;

Based on the shape parameters, a target game object matching the entity object is determined from the undetermined game objects.

In one possible implementation, the device further includes an identification unit:

The determining unit is also used to determine the number of objects of the entity object in the video data;

The recognition unit is used to recognize the actions performed by the entity objects in the video data if the number of objects is a specified number.

The identification unit is also configured to trigger the rendering unit in response to the action being a specified action;

The determining unit is further configured to trigger the rendering unit if the number of objects is not the specified number.

In one possible implementation, the entity object is a human body, and the recognition unit is further configured to:

Determine the locations of the specified number of key joints of the human body in the video data;

Based on the location of the key joint points, identify the actions performed by the specified number of human bodies in the video data.

In one possible implementation, the acquisition unit is further configured to:

Obtain a game object selection instruction for a specified object, wherein the specified object is one of the plurality of game objects;

The virtual props corresponding to the specified object are rendered onto the entity object in the video data to obtain the specified video.

In one possible implementation, the determining unit is further configured to determine the video position of the entity object in the video data;

The rendering unit is also used for:

The display form of the virtual prop is determined based on the video location;

Based on the display format, the virtual props are rendered onto the entity objects in the video data to obtain the target video.

In one possible implementation, the device further includes a storage unit, the storage unit being used for:

Save the target video at the storage location;

Generate a corresponding address code based on the storage location, and display the address code;

If a video download request sent by a terminal device is obtained based on the address code, the target video is returned to the terminal device from the storage location.

In one possible implementation, the display area is a three-sided screen, with the three display screens of the three-sided screen arranged perpendicularly to each other.

Therefore, the set display area can display the game environment image of the target game. When a physical object enters the display area, video data can be captured for that area. Since the game environment image displayed in the display area serves as the background for the physical object, the physical object in the video data appears to be in the target game scene, creating a sense of immersion. By identifying the shape parameters of the physical object in the video data, a target game object matching the physical object can be determined from multiple game objects in the target game based on these parameters. The virtual props corresponding to the target game object are then rendered onto the physical object in the video data, resulting in the target video. Because the target game object is similar in appearance to the physical object, the physical object rendered with virtual props in the target video will resemble the target game object even more closely, creating the illusion that the physical object is playing the role of the target game object in the target game scene. This not only provides a better immersive experience but also adds a sense of fun.

This application also provides a computer device, which is the computer device described above, and may include a terminal device or a server. The aforementioned video processing apparatus may be configured in this computer device. The computer device will now be described in conjunction with the accompanying drawings.

If the computer device is a terminal device, please refer to Figure 11. This application embodiment provides a terminal device, taking a mobile phone as an example:

Figure 11 shows a block diagram of a portion of the structure of a mobile phone related to the terminal device provided in the embodiments of this application. Referring to Figure 11, the mobile phone includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a Wi-Fi module 1470, a processor 1480, and a power supply 1490. Those skilled in the art will understand that the mobile phone structure shown in Figure 11 does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

The following section, with reference to Figure 11, provides a detailed introduction to the various components of the mobile phone:

The RF circuit 1410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1480; in addition, it transmits uplink data to the base station.

The memory 1420 can be used to store software programs and modules, and the processor 1480 executes various functions and data processing of the mobile phone by running the software programs and modules stored in the memory 1420.

The input unit 1430 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432. The touch panel 1431, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1431), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 1431 may include two parts: a touch detection device and a touch controller.

The display unit 1440 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1440 may include a display panel 1441.

The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.

Audio circuit 1460, speaker 1461, and microphone 1462 provide an audio interface between the user and the mobile phone. Audio circuit 1460 converts received audio data into electrical signals and transmits them to speaker 1461, where speaker 1461 converts them into sound signals for output. On the other hand, microphone 1462 converts collected sound signals into electrical signals, which are received by audio circuit 1460, converted into audio data, and then processed by processor 1480 before being transmitted via RF circuit 1410 to, for example, another mobile phone, or the audio data can be output to memory 1420 for further processing.

WiFi is a short-range wireless transmission technology. A mobile phone using the WiFi module 1470 can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 11 shows the WiFi module 1470, it is understood that it is not an essential component of a mobile phone and can be omitted as needed without altering the essence of the invention.

The processor 1480 is the control center of the mobile phone. It connects to various parts of the mobile phone through various interfaces and lines. It performs various functions of the mobile phone and processes data by running or executing software programs and/or modules stored in the memory 1420 and calling data stored in the memory 1420.

In this embodiment, the processor 1480 included in the terminal device also has the following functions:

Acquire video data collected for a display area, the display area being used to display game environment images of the target game, and the display area containing entity objects;

Determine the shape parameters of the entity object in the video data;

Based on the shape parameters, a target game object matching the entity object is determined from among multiple game objects in the target game;

The virtual props corresponding to the target game object are rendered onto the entity object in the video data to obtain the target video.

If the computer device is a server, this application embodiment also provides a server, as shown in Figure 12. Figure 12 is a structural diagram of the server 1500 provided in this application embodiment. The server 1500 can vary considerably due to different configurations or performance, and may include one or more central processing units (CPUs) 1522 (e.g., one or more processors) and memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1522 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the server 1500.

Server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input/output interfaces 1558, and/or one or more operating systems 1541, such as Windows Server ^™ , Mac OS X ^™ , Unix ^™ , Linux ^™ , FreeBSD ^™ , etc.

The steps performed by the server in the above embodiments can be based on the server structure shown in Figure 12.

In addition, this application embodiment also provides a storage medium for storing a computer program for executing the method provided in the above embodiment.

This application also provides a computer program product including instructions that, when run on a computer, cause the computer to perform the methods provided in the above embodiments.

Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store program code.

It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Moreover, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims (13)

1. A video processing method, characterized in that the method comprises: Acquire video data collected from a display area, which is used to display game environment images of the target game and contains entity objects; the entity objects are actual objects with physical forms. Determine the shape parameters of the entity object in the video data; Obtain the game character parameters corresponding to multiple game objects in the target game; Based on the similarity comparison between the appearance parameters and the game character parameters, a target game object matching the entity object is determined from game objects whose similarity meets the threshold, so that the target game object and the entity object are related in appearance; the multiple game objects are game characters of different forms or different types in the target game; the target game object is the game character played by the entity object; Based on the game image parameters and appearance parameters of the target game object, determine the appearance differences between the entity object in the video data and the target game object; Adjust the game appearance parameters of the target game object according to the appearance differences to obtain adjusted appearance parameters that are compatible with the appearance parameters; Based on the adjusted appearance parameters, the virtual props corresponding to the target game object are rendered onto the entity object in the video data to obtain the target video. The adjusted appearance parameters are used to identify the mapping relationship between the virtual props and the entity object in the video data, so that when the virtual props of the target game object are rendered onto the entity object in the video data, the virtual props that originally had different appearances are rendered onto the entity object in the video data. The virtual props corresponding to the target game object are used to identify the appearance features of the target game object and distinguish it from other game objects. 2. The method according to claim 1, wherein if the entity object is a human body, determining the shape parameters of the entity object in the video data includes: At least one of the facial features, height parameters, and body shape parameters of the human body in the video data is determined as the shape parameter. 3. The method according to claim 1, wherein if the entity object is a human body, the method further includes: Obtain the gender parameters of the human body; The step of determining the target game object that matches the entity object among multiple game objects in the target game based on the shape parameters includes: From the plurality of game objects, determine the candidate game object that matches the gender parameter; Based on the shape parameters, a target game object matching the entity object is determined from the undetermined game objects. 4. The method according to claim 1, characterized in that the method further comprises: Determine the number of entities in the video data; If the number of objects is a specified number, identify the actions performed by the entity objects in the video data; In response to the action being a specified action, the operation of rendering the virtual props corresponding to the target game object onto the entity object in the video data is performed to obtain the target video; If the number of objects is not the specified number, the operation of rendering the virtual props corresponding to the target game object onto the entity object in the video data is performed to obtain the target video. 5. The method according to claim 4, wherein the entity object is a human body, and the step of recognizing the action performed by the entity object in the video data includes: Determine the locations of the specified number of key joints of the human body in the video data; Based on the location of the key joint points, identify the actions performed by the specified number of human bodies in the video data. 6. The method according to claim 1, characterized in that the method further comprises: Obtain a game object selection instruction for a specified object, wherein the specified object is one of the plurality of game objects; The virtual props corresponding to the specified object are rendered onto the entity object in the video data to obtain the specified video. 7. The method according to claim 1, characterized in that the method further comprises: Determine the video position of the entity object in the video data; The step of rendering the virtual items corresponding to the target game object onto the entity object in the video data to obtain the target video includes: The display form of the virtual prop is determined based on the video location; Based on the display format, the virtual props are rendered onto the entity objects in the video data to obtain the target video. 8. The method according to any one of claims 1-7, characterized in that the method further comprises: Save the target video at the storage location; Generate a corresponding address code based on the storage location, and display the address code; If a video download request sent by a terminal device is obtained based on the address code, the target video is returned to the terminal device from the storage location. 9. The method according to any one of claims 1-7, wherein the display area is a three-sided screen, and the three display screens of the three-sided screen are arranged perpendicularly to each other. 10. A video processing apparatus, characterized in that the apparatus comprises an acquisition unit, a determination unit, and a rendering unit: The acquisition unit is used to acquire video data collected from a display area, which is used to display game environment images of the target game and has entity objects; the entity objects are actual objects with physical entities. The determining unit is used to determine the shape parameters of the entity object in the video data; The determining unit is further configured to acquire game character parameters corresponding to multiple game objects in the target game; perform a similarity comparison between the appearance parameters and the game character parameters, and determine a target game object matching the entity object from game objects whose similarity meets a threshold, so that the target game object and the entity object are related in appearance; the multiple game objects are game characters of different forms or different types in the target game; the target game object is the game character played by the entity object; The determining unit is further configured to determine the appearance difference between the entity object in the video data and the target game object based on the game image parameters and the appearance parameters of the target game object; adjust the game image parameters of the target game object according to the appearance difference to obtain adjusted image parameters that are adapted to the appearance parameters; The rendering unit is used to render the virtual props corresponding to the target game object onto the entity object in the video data according to the adjusted image parameters, thereby obtaining the target video. The adjusted image parameters are used to identify the mapping relationship between the virtual props and the entity object in the video data, so that when the virtual props of the target game object are rendered onto the entity object in the video data, the virtual props that originally have different shapes are rendered onto the entity object in the video data. The virtual props corresponding to the target game object are used to identify the shape characteristics of the target game object and distinguish it from other game objects. 11. A computer device, characterized in that the computer device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method described in any one of claims 1-9 according to the instructions in the program code. 12. A computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program, the computer program being used to perform the method according to any one of claims 1-9. 13. A computer program product comprising instructions, characterized in that, when run on a computer, it causes the computer to perform the method according to any one of claims 1-9.