CN116711303A - Three-dimensional video call method and electronic device - Google Patents

Abstract

The application provides a three-dimensional video call method and electronic equipment, which can realize three-dimensional video call and reduce time delay for acquiring three-dimensional video. The electronic device includes: the face image acquisition module is used for dividing the face depth image into a plurality of subunits comprising a first subunit and a second subunit; dividing the face two-dimensional image into a plurality of subunits including a third subunit and a fourth subunit; transmitting the first subunit and the third subunit to the video coding module; the second subunit and the fourth subunit are then sent to the video encoding module. The video coding module is used for obtaining a first coding unit according to the first subunit and the third subunit and sending the first coding unit to the network transmission module; and then, obtaining a second coding unit according to the second subunit and the fourth subunit and sending the second coding unit to the network transmission module. The network transmission module is used for sending the first coding unit to the second electronic equipment; after the first coding unit is transmitted, the second coding unit is transmitted to the second electronic device.

Description

Three-dimensional video call method and electronic device technical field

The present application relates to the communication field, and in particular to a three-dimensional video call method and electronic equipment.

Background technique

With the development of video codec technology, video call technology has become one of the more popular social methods. Existing video call solutions have been able to implement three-dimensional video calls. Specifically, the video sending device first acquires the two-dimensional image and the depth image, and then compresses the two-dimensional image and the depth image and sends them to the server. The server decodes the received 2D image and depth image, generates a 3D image according to the decoded 2D image and depth image, compresses the 3D image and sends it to the video receiving device, so as to realize 3D video communication between users.

In the above video call solution, the server decodes the received 2D image and depth image, and encodes the generated 3D image, which increases the time delay for the video receiving device to obtain the 3D video.

Contents of the invention

Embodiments of the present application provide a 3D video call method and electronic equipment, which can reduce the time delay for acquiring 3D video during the video call process. In order to achieve the above purpose, the present application adopts the following technical solutions.

In a first aspect, an electronic device is provided. The electronic device includes: a face image acquisition module, a video encoding module and a network transmission module. Wherein, the face image acquisition module is used to obtain the face depth image and the two-dimensional image of the face; the depth image of the face is divided into a plurality of subunits including the first subunit and the second subunit; the two-dimensional image of the face Divided into a plurality of subunits including the third subunit and the fourth subunit; sending the first subunit and the third subunit to the video coding module; after sending the first subunit and the third subunit, sending to the video coding module The second subunit and the fourth subunit. Wherein, the first subunit corresponds to the third subunit, and the second subunit corresponds to the fourth subunit. The video coding module is used to obtain the first coding unit according to the first sub-unit and the third sub-unit, and send the first coding unit to the network transmission module; after obtaining and sending the first coding unit, according to the second sub-unit and the fourth sub-unit The subunit obtains the second encoding unit, and sends the second encoding unit to the network transmission module. A network transmission module, configured to send the first encoding unit to the second electronic device; after sending the first encoding unit, send the second encoding unit to the second electronic device.

Based on the electronic device described in the first aspect, the face image acquisition module of the electronic device divides the depth image of the face and the two-dimensional image of the face into a plurality of sub-units, sends a pair of sub-units to the video encoding module, and then sends the next A pair of sub-units, the pair of sub-units includes a sub-unit of the depth image of the face and a sub-unit of the two-dimensional image of the face corresponding to the sub-unit of the depth image of the face. In this way, the waiting time for the video encoding module to receive images can be shortened. After the video encoding module obtains a coding unit from a pair of subunits and sends it to the network transmission module, it performs the same processing on the next pair of subunits, which can shorten the waiting time for the network transmission module to receive the coding unit. After the network transmission module receives a coding unit and sends it to the second electronic device, it receives the next coding unit and sends it to the second electronic device, which can shorten the waiting time for the second electronic device to receive the coding unit, thereby reducing the number of second electronic devices. Get the time delay of 3D video.

In a possible design, the face image acquisition module can specifically be used to receive face depth information; receive face two-dimensional information; obtain face depth images according to face depth information, and obtain face 2D image. In this way, the face image acquisition module processes the face depth information to obtain a face depth image, so that a real 3D video can be displayed and a 3D video call can be realized.

In a possible design, the video encoding module may be specifically configured to: encode the third subunit to obtain a third coding unit; obtain the first coding unit according to the first subunit and the third coding unit; and, Encode the fourth sub-unit to obtain a fourth coding unit; obtain a second coding unit according to the second sub-unit and the fourth coding unit. That is to say, the video encoding module may first encode a sub-unit of the two-dimensional face image to obtain a coding unit, and then perform mixed coding on a sub-unit of the face depth image and the coding unit. In this way, encoding the sub-units of the depth image of the face and the sub-units of the two-dimensional image of the face into the same coding unit can reduce the complexity of transmission.

It should be noted that this application does not limit the order in which the video encoding module of the electronic device encodes the subunits of the two-dimensional face image and the subunits of the depth image of the face. For example, the video encoding module may first encode the first subunit Encoding is performed, and then the encoded first sub-unit and the third sub-unit are mixed-encoded to obtain the first coding unit.

In a possible design, the electronic device described in the first aspect may further include: a three-dimensional face generation module and a display module. Wherein, the face three-dimensional generating module is used for obtaining the first three-dimensional face sub-image according to the first subunit and the third sub-unit, and sending the first three-dimensional face sub-image to the display module; after obtaining and sending the first three-dimensional face sub-image After the sub-image, the second three-dimensional sub-image of human face is obtained according to the second sub-unit and the fourth sub-unit, and the second three-dimensional sub-image of human face is sent to the display module. The display module is used to superimpose the first three-dimensional sub-image of human face and the two-dimensional image of the scene; dimensional images are superimposed. In this way, the three-dimensional face generation module obtains a three-dimensional sub-image of a human face according to a pair of sub-units, and after sending it to the display module, performs the same processing on the next pair of sub-units, which can shorten the waiting time of the display module, thereby reducing The time delay for the electronic device to obtain the 3D image of the face can further reduce the time delay for the electronic device to obtain the 3D video.

In a second aspect, an electronic device is provided. The electronic device includes: a network transmission module, a video decoding module, a three-dimensional human face generation module and a display module. Wherein, the network transmission module is used to receive the first coding unit from the first electronic device, and send the first coding unit to the video decoding module; after receiving and sending the first coding unit, receive the second coding unit from the first electronic device unit, and send the second coding unit to the video decoding module. The video decoding module is used to obtain the first sub-unit and the third sub-unit according to the first coding unit; after obtaining the first sub-unit and the third sub-unit, obtain the second sub-unit and the fourth sub-unit according to the second coding unit . Wherein, the first subunit and the second subunit are subunits in the depth image of the face respectively, the third subunit and the fourth subunit are subunits in the two-dimensional image of the face respectively, and the first subunit corresponds to the third subunit unit, the second subunit corresponds to the fourth subunit. The three-dimensional face generation module is used to obtain the first three-dimensional sub-image of human face according to the first subunit and the third sub-unit, and send the first three-dimensional sub-image of human face to the display module; after obtaining and sending the first three-dimensional sub-image of human face Afterwards, the second three-dimensional sub-image of human face is obtained according to the second subunit and the fourth subunit, and the second three-dimensional sub-image of human face is sent to the display module. The display module is used to superimpose the first three-dimensional sub-image of human face and the two-dimensional image of the scene; dimensional images are superimposed.

Based on the electronic device described in the second aspect, after the network transmission module of the electronic device receives one coding unit and sends it to the video decoding module, it receives the next coding unit and sends it to the video decoding module, which can shorten the waiting time of the video decoding module. The video decoding module decodes a coding unit, obtains a subunit of a 3D face image and a subunit of a 2D face image, and sends them to the 3D face generation module, and performs the same processing on the next coding unit, which can shorten The waiting time of the 3D face generation module. The 3D face generation module obtains a 3D face sub-image according to a pair of sub-units, and sends it to the display module, and then performs the same processing on the next pair of sub-units, which can shorten the waiting time of the display module, thereby reducing the acquisition time of electronic equipment. The time delay of the 3D image of the face reduces the time delay of the electronic equipment to obtain the 3D video.

In a possible design, the video decoding module is further configured to parse the first coding unit to obtain the first sub-unit and the third coding unit; decode the third coding unit to obtain the third sub-unit; After the third coding unit is decoded, the second coding unit is parsed to obtain the second sub-unit and the fourth coding unit; the fourth coding unit is decoded to obtain the fourth sub-unit. In this way, the video decoding module can decode a pair of sub-units from one coding unit, which can reduce the complexity of acquiring the sub-units of the depth image of the face and the sub-units of the two-dimensional image of the face.

In a possible design, the electronic device provided in the second aspect may further include: a touch module. Wherein, the touch module is used for detecting adjustment actions. The display module is configured to adjust and display the angle of the human face in the three-dimensional image of the human face according to the adjustment action. In this way, the electronic device can display different angles of faces in the 3D video.

In a third aspect, a method for 3D video calling is provided. The three-dimensional video call method includes: acquiring a face depth image and a two-dimensional face image; dividing the face depth image into a plurality of subunits including a first subunit and a second subunit; dividing the two-dimensional face image into a plurality of subunits including A plurality of subunits of the third subunit and the fourth subunit. The first coding unit is obtained according to the first sub-unit and the third sub-unit; after the first coding unit is obtained, the second coding unit is obtained according to the second sub-unit and the fourth sub-unit. Send the first coding unit to the second electronic device; after sending the first coding unit, send the second coding unit to the second electronic device. Wherein, the first subunit corresponds to the third subunit, and the second subunit corresponds to the fourth subunit.

In a possible design, the acquisition of the face depth image and the face two-dimensional image may include: receiving face depth information; receiving face two-dimensional information; obtaining the face depth image according to the face depth information, and The two-dimensional face information is used to obtain a two-dimensional face image.

In a possible design, the 3D video call method described in the third aspect may further include: encoding the third subunit to obtain a third coding unit; according to the first subunit and the third coding unit, obtaining The first coding unit; and, encoding the fourth sub-unit to obtain a fourth coding unit; and obtaining a second coding unit according to the second sub-unit and the fourth coding unit.

In a possible design, the 3D video call method described in the third aspect may further include: obtaining the first 3D sub-image of human face according to the first subunit and the third subunit; After the image is obtained, the second three-dimensional sub-image of human face is obtained according to the second subunit and the fourth subunit. Superimpose the first 3D sub-image of human face with the 2D scene image; after superimposing the first 3D sub-image of human face with the 2D scene image, superimpose the second 3D sub-image of human face with the 2D scene image.

In addition, for the technical effect of the 3D video calling method described in the third aspect, reference may be made to the technical effect of the electronic device described in any implementation manner in the first aspect, which will not be repeated here.

In a fourth aspect, a method for 3D video calling is provided. The three-dimensional video calling method includes: receiving a first coding unit from a first electronic device; after receiving the first coding unit, receiving a second coding unit from the first electronic device. Obtain the first subunit and the third subunit according to the first coding unit; after obtaining the first subunit and the third subunit, obtain the second subunit and the fourth subunit according to the second coding unit; wherein, the first subunit The unit and the second subunit are subunits in the face depth image respectively, the third subunit and the fourth subunit are subunits in the two-dimensional image of the face respectively, the first subunit corresponds to the third subunit, and the second subunit A unit corresponds to a fourth subunit. The first three-dimensional sub-image of human face is obtained according to the first subunit and the third sub-unit; after the first three-dimensional sub-image of human face is obtained, the second three-dimensional sub-image of human face is obtained according to the second subunit and the fourth subunit. Superimpose the first 3D sub-image of human face with the 2D scene image; after superimposing the first 3D sub-image of human face with the 2D scene image, superimpose the second 3D sub-image of human face with the 2D scene image.

In a possible design, the 3D video calling method described in the fourth aspect may further include: parsing the first coding unit to obtain the first sub-unit and the third coding unit; decoding the third coding unit , to obtain the third subunit; after decoding the third coding unit, parse the second coding unit to obtain the second subunit and the fourth coding unit; decode the fourth coding unit to obtain the fourth subunit.

In a possible design, the 3D video calling method described in the fourth aspect may further include: detecting an adjustment action, and adjusting the angle of the human face displayed in the 3D image of the human face in response to the adjustment action.

In addition, for the technical effect of the 3D video call method described in the fourth aspect, reference may be made to the technical effect of the electronic device described in any one of the implementation manners in the second aspect, which will not be repeated here.

According to a fifth aspect, an electronic device is provided, and the electronic device includes: a processor, and the processor is coupled to a memory. Memory, used to store computer programs. The processor is configured to execute the computer program stored in the memory, so that the electronic device executes the three-dimensional video calling method described in any one possible implementation manner of the third aspect to the fourth aspect.

In a possible design, the electronic device described in the fifth aspect may further include a transceiver. The transceiver can be a transceiver circuit or an input/output port. The transceiver can be used for the electronic device to communicate with other devices.

In the present application, the electronic device described in the fifth aspect may be an electronic device, or a chip or a chip system disposed inside the electronic device.

In addition, for the technical effect of the electronic device described in the fifth aspect, reference may be made to the technical effect of the 3D video call method described in any one of the implementation manners from the third aspect to the fourth aspect, which will not be repeated here.

In a sixth aspect, a 3D video call system is provided. The 3D video call system includes the electronic device described in any possible implementation manner in the first aspect, and the electronic device described in any possible implementation manner in the second aspect.

According to a seventh aspect, a computer-readable storage medium is provided, the computer-readable storage medium stores a computer program or an instruction, and when the computer program or instruction is run on a computer, the computer executes any one of the third to fourth aspects. The three-dimensional video calling method described in a possible implementation manner.

According to an eighth aspect, a computer program product is provided, and the computer program product includes: a computer program or an instruction, when the computer program or instruction is run on a computer, the computer executes any possible implementation of the third aspect to the fourth aspect The three-dimensional video calling method described in the method.

Description of drawings

FIG. 1 is a schematic structural diagram of a three-dimensional video call system provided by an embodiment of the present application;

FIG. 2 is a first schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 3 is a block diagram of the software structure of the electronic device provided by the embodiment of the present application;

FIG. 4 is a second schematic structural diagram of the electronic device provided by the embodiment of the present application;

FIG. 5 is a schematic structural diagram of a face image acquisition module provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram III of the electronic device provided by the embodiment of the present application;

FIG. 7 is a schematic flowchart of a three-dimensional video call method provided in an embodiment of the present application;

FIG. 8 is a schematic diagram of a face depth image and a two-dimensional face image provided by an embodiment of the present application;

FIG. 9 is a first schematic diagram of the application of the first electronic device provided by the embodiment of the present application;

FIG. 10 is a second schematic diagram of the application of the first electronic device provided by the embodiment of the present application;

FIG. 11 is the second schematic flow diagram of the three-dimensional video call method provided by the embodiment of the present application;

FIG. 12 is a schematic structural diagram of a code stream provided by an embodiment of the present application;

FIG. 13 is a schematic diagram of the application of the second electronic device provided by the embodiment of the present application;

FIG. 14 is a schematic diagram of a three-dimensional face image provided by an embodiment of the present application.

Detailed ways

The 3D video call method and electronic equipment provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

The terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes other unlisted steps or units, or optionally also includes Other steps or elements inherent to the process, method, product or apparatus are included.

It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In the description of the present application, unless otherwise specified, the meaning of "plurality" refers to two or more. The "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, and B exists alone These three situations.

The application scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute limitations on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of the network architecture With the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

FIG. 1 is a schematic structural diagram of a 3D video call system to which the 3D video call method provided in the embodiment of the present application is applicable. To facilitate understanding of the embodiment of the present application, the 3D video calling system shown in FIG. 1 is taken as an example to describe in detail the 3D video calling system applicable to the embodiment of the present application. It should be noted that the solutions in the embodiments of the present application can also be applied to other 3D video call systems, such as a first electronic device pairing multiple second electronic devices, or multiple first electronic devices pairing multiple second electronic devices For the video call scene, the corresponding name can also be replaced by the name of the corresponding function in other 3D video call systems.

As shown in FIG. 1 , the 3D video call system includes at least two electronic devices, such as a first electronic device and a second electronic device. Wherein, the embodiment of the present application is described by taking the first electronic device as the sending end of the 3D video and the second electronic device as the receiving end of the 3D video as an example. Wherein, the electronic device specifically may be a mobile phone, a tablet computer, a vehicle-mounted device, an augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) device, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC) , netbooks, personal digital assistants (personal digital assistant, PDA), artificial intelligence (artificial intelligence) devices, wearable devices and other terminal devices with video call functions. Wearable devices can be smart watches, smart bracelets, smart glasses, smart Helmets etc. The embodiment of the present application does not impose any limitation on the specific type of the electronic device.

FIG. 2 is a first structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown in FIG. 2 , the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, and a battery 142 , antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, sensor module 190, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (subscriber identification module , SIM) card interface 195 etc.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU) wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Wherein, the controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments of the present application, the electronic device 100 can use the processor 110 to acquire the depth image of the face and the two-dimensional image of the face, divide the depth image of the face into multiple sub-units, and divide the two-dimensional image of the face into multiple sub-units. unit. Optionally, the electronic device 100 may use the processor 110 to obtain a three-dimensional face sub-image according to the face depth image and the two-dimensional face image. Specifically, the electronic device 100 may use the processor 110 to obtain a three-dimensional face sub-image according to a sub-unit of the face depth image and a sub-unit of the two-dimensional face image, and a sub-unit of the face depth image is related to the person corresponds to a subunit of the 2D image of the face.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger.

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 .

The wireless communication function of the electronic device 100 may be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like.

In some embodiments of the present application, the electronic device 100 may use the mobile communication module 150 to send encoded face depth images and face two-dimensional images to other electronic devices, and/or receive encoded face depth images and face two-dimensional images from other electronic devices. Face depth image and face 2D image. Exemplarily, the electronic device 100 can use the mobile communication module 150 to send the subunits of the encoded face depth image and the subunits of the two-dimensional face image to other electronic devices, and/or receive encoded The sub-units of the face depth image and the sub-units of the two-dimensional face image.

The wireless communication module 160 can provide wireless local area network (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite Wireless communication solutions such as global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared technology (IR).

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

In some embodiments of the present application, the electronic device 100 may use the GPU to superimpose the 3D sub-image of the face with the 2D image of the scene.

The display screen 194 is used to display images, videos and the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1. In some embodiments, the electronic device 100 may include 1 or N cameras 193 .

The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 , and the application processor.

Camera 193 is used to capture still images or video. The camera 193 may include a time of flight (time of flight, TOF) sensor, a three-dimensional structured light sensor, a color (red green blue, RGB) sensor, and the like.

In some embodiments of the present application, the electronic device 100 may use the camera 193 to collect a face depth image and a two-dimensional face image.

In some embodiments of the present application, the electronic device 100 may use a video codec to encode the face depth image and the two-dimensional face image, and/or obtain the face depth image and the two-dimensional face image through decoding. Exemplarily, the electronic device 100 may use a video codec to encode the sub-units of the face depth image and the sub-units of the two-dimensional face image, and/or obtain the sub-units of the face depth image and the face A subunit of a 2D image.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 .

The internal memory 121 may be used to store computer-executable program codes including instructions. The internal memory 121 may include an area for storing programs and an area for storing data. The internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor. In some embodiments of the present application, the internal memory 121 may be used to store an artificial intelligence algorithm model, and/or a 3D face generation algorithm model, and the like.

The audio module 170 includes a speaker, a receiver, a microphone, an earphone jack, and the like.

The audio module 170 is used to convert digital audio data into an analog audio electrical signal output, and is also used to convert an analog audio electrical signal input into digital audio data. The audio module 170 may include an analog/digital converter and a digital/analog converter.

In some embodiments, the electronic device 100 can implement the audio function through the audio module 170 and the application processor. Such as music playback, recording, etc.

The sensor module 190 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

In some embodiments of the present application, the electronic device 100 may use the touch sensor to detect an adjustment action, so as to adjust the angle of the face in the three-dimensional image of the face displayed on the display screen 194 .

It can be understood that, the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the software structure of the electronic device 100 is exemplarily described by taking an Android system with a layered architecture as an example.

FIG. 3 is a block diagram of a software structure of an electronic device provided by an embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, which are, from top to bottom, the application program layer, the application program framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.

The application layer can consist of a series of application packages.

As shown in Figure 3, the application package may include application programs such as camera, calendar, map, WLAN, music, short message, gallery, call, and navigation.

Wherein, the calling application can be used to realize three-dimensional video calling.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 3, the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

In some embodiments, the 3D video call can also be implemented as a module in the application framework layer of the electronic device, such as a 3D video call module.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

Content providers are used to store and retrieve data and make it accessible to applications. Said data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide communication functions of the electronic device 100 . For example, the management of call status (including connected, hung up, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompt text information in the status bar, make a prompt sound, and flash the indicator light, etc.

Android Runtime includes core library and virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

The core library consists of two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

A system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing, etc.

2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

FIG. 4 is a second structural schematic diagram of the electronic device provided by the embodiment of the present application.

The electronic device 400 shown in FIG. 4 may be a first electronic device, that is, the electronic device 400 may be a sending end of a 3D video. As shown in FIG. 4 , the first electronic device 400 provided by the embodiment of the present application may include a face image acquisition module 410 , a video encoding module 420 and a network transmission module 430 . Optionally, the first electronic device 400 may further include a face three-dimensional generation module 440 and a display module 450.

It should be noted that the modules shown in FIG. 4 can be implemented by electronic hardware, computer software, or a combination of computer software and electronic hardware. Exemplarily, when implemented by software, the module shown in FIG. 4 can be implemented as a call application in the application layer shown in FIG. 3 , or the module shown in FIG. 4 can also be implemented as an application program shown in FIG. 3 3D video call module in the framework layer. When implemented using hardware, the face image acquisition module 410, the video encoding module 420, and the three-dimensional face generation module 440 can be implemented as the processor 110 shown in Figure 2, and the network transmission module 430 can be implemented as the mobile communication module shown in Figure 2 150. The display module 450 may be implemented as the display screen 194 shown in FIG. 2 . When it is realized by combining computer software and electronic hardware, the above-mentioned way of implementing by using software and the way of implementing by using hardware can be combined, and the embodiments of the present application will not repeat them here.

Exemplarily, the face image acquisition module 410 may be configured to acquire a face depth image and a two-dimensional face image, divide the face depth image into a plurality of subunits including a first subunit and a second subunit, and divide the face depth image into a plurality of subunits including a first subunit and a second subunit. The dimensional image is divided into a plurality of subunits including a third subunit and a fourth subunit. Exemplarily, the face image acquisition module 410 can be used to send the first subunit and the third subunit to the following video encoding module 420; after sending the first subunit and the third subunit, send the following video encoding module 420 Send the second subunit and the fourth subunit. Wherein, the first subunit corresponds to the third subunit, and the second subunit corresponds to the fourth subunit.

Optionally, the face image acquisition module 410 can be used to send the first subunit and the third subunit to the following three-dimensional face generation module 440; after sending the first subunit and the third subunit, send the following person The face three-dimensional generation module 440 sends the second subunit and the fourth subunit.

Optionally, the face image acquisition module 410 may be used to acquire a two-dimensional image of the scene, and send the two-dimensional image of the scene to the three-dimensional face generation module 440 and/or the video encoding module 420 described below. Exemplarily, the scene two-dimensional image includes the scene image in the current video scene.

In some embodiments, the face image acquisition module 410 may be specifically configured to receive face depth information, receive face two-dimensional information, obtain a face depth image according to the face depth information, and obtain a face two-dimensional image according to the face two-dimensional information. dimensional image. Exemplarily, the face depth information may be collected by a high-precision depth camera, such as a TOF sensor, a three-dimensional structured light sensor, and the like.

Exemplarily, the two-dimensional face information may include face information and scene information in the current video scene. Or, for example, the two-dimensional face information only includes face information and does not include scene information. The face image acquisition module 410 can be used to receive the two-dimensional information of the scene, and obtain the scene image according to the two-dimensional information of the scene. Optionally, both the two-dimensional information of the face and the two-dimensional information of the scene may be collected by a two-dimensional camera, such as an RGB sensor.

In some other embodiments, the face image acquisition module 410 can be specifically used to collect face depth information, collect face two-dimensional information, obtain a face depth image according to the face depth information, and obtain a face image according to the two-dimensional information of the face. 2D image. That is to say, the face image collection module 410 may include a module for collecting depth information of a face and a module for collecting two-dimensional information of a face. Optionally, the face image acquisition module 410 may be specifically used to acquire scene two-dimensional information.

Exemplarily, FIG. 5 is a schematic structural diagram of a face image acquisition module provided in an embodiment of the present application. As shown in FIG. 5 , the face image acquisition module 410 may include: a face depth image acquisition submodule 411 , a two-dimensional image acquisition submodule 412 and an image signal processing (image signal processing, ISP) submodule 413 .

Among them, the face depth image collection sub-module 411 can be used to collect face depth information in the current video scene, and send it to the image signal processing sub-module 413 described below. Exemplarily, the face depth image acquisition sub-module 411 may be a high-precision depth camera, which may include but not limited to a TOF sensor and a three-dimensional structured light sensor. Among them, the TOF sensor can continuously send light pulses to the target object, and then receive the light returned from the target object, and obtain the distance from itself to the target object by detecting the flight (round trip) time of sending and receiving light pulses, and generate depth information. The 3D structured light sensor obtains the depth information of the surface of the target object by projecting structured light onto the surface of the target object and receiving the light reflected from the surface of the target object.

The two-dimensional image collection sub-module 412 can be used to collect two-dimensional face information in the current video scene, and send it to the image signal processing sub-module 413 described below. Exemplarily, the two-dimensional image acquisition sub-module 412 may be an RGB sensor, etc., which is not limited in this application. Optionally, the two-dimensional image collection sub-module 412 may be used to collect scene two-dimensional information in the current video scene, and send it to the image signal processing sub-module 413 described below.

The ISP sub-module 413 can be used to receive the face depth information from the face depth image acquisition sub-module 411 and the face two-dimensional information from the two-dimensional image acquisition sub-module 412, and obtain a face depth image according to the face depth information, A two-dimensional face image is obtained according to the two-dimensional face information. Optionally, the ISP submodule 413 may be configured to obtain a two-dimensional image of the scene according to the two-dimensional information of the scene.

Specifically, the ISP submodule 413 can be used to divide the face depth image into a plurality of subunits including the first subunit and the second subunit; divide the two-dimensional face image into a plurality of subunits including the third subunit and the fourth subunit multiple subunits. The ISP submodule 413 can be used to send the first subunit and the third subunit to the following video encoding module 420; after sending the first subunit and the third subunit, send the second subunit to the following video encoding module 420 and the fourth subunit. Optionally, the ISP submodule 413 may be configured to send the scene two-dimensional image to the following video encoding module 420 .

Optionally, the ISP submodule 413 can be used to send the first subunit and the third subunit to the following three-dimensional face generation module 440; after sending the first subunit and the third subunit, send the three-dimensional face generation module 440 Send the second subunit and the fourth subunit. The ISP sub-module 413 may be configured to send the two-dimensional image of the scene to the following three-dimensional face generation module 440 .

The video coding module 420 is configured to obtain the first coding unit according to the first subunit and the third subunit, and send the first coding unit to the network transmission module 430; after obtaining and sending the first coding unit, according to the second subunit and the The fourth subunit obtains the second encoding unit, and sends the second encoding unit to the network transmission module 430 . In this way, the video encoding module 420 receives and encodes multiple subunits included in the face depth image and multiple subunits included in the face two-dimensional image with a pair of subunits as the granularity pipeline, and the pair of subunits includes the face depth image. One subunit and one subunit of the two-dimensional face image corresponding to one subunit of the depth image of the face, so that the encoding unit corresponding to each pair of subunits is obtained and sent in a pipelined manner, which can reduce the acquisition time of the second electronic device. The time delay of the 3D image of the face improves the efficiency of obtaining 3D video.

Optionally, the video encoding module 420 is specifically configured to: encode the third sub-unit to obtain a third coding unit; obtain the first coding unit according to the first sub-unit and the third coding unit; The sub-units are encoded to obtain a fourth coding unit; according to the second sub-unit and the fourth coding unit, a second coding unit is obtained. That is to say, the video encoding module 420 can encode the sub-units of the face two-dimensional image to obtain the third coding unit, and then encode the sub-units of the face depth image and the third coding unit to obtain the first coding unit.

It should be noted that this embodiment of the present application does not limit the order in which the electronic device encodes the first subunit and the third subunit. For example, the first subunit may be encoded first, and then the encoded first subunit Perform mixed coding with the third sub-unit to obtain the first coding unit.

Optionally, the video encoding module 420 may encode the two-dimensional image of the scene, and send the encoded two-dimensional image of the scene to the network transmission module 430 .

The network transmission module 430 may be configured to send the encoded face depth image and the two-dimensional face image to the second electronic device. Specifically, the network transmission module 430 may be configured to send the first encoding unit to the second electronic device; after sending the first encoding unit, send the second encoding unit to the second electronic device. In this way, the network transmission module 430 can send the encoding units corresponding to multiple pairs of sub-units to the second electronic device in a pipelined manner, so that the second electronic device can obtain the three-dimensional face sub-image in a pipelined manner, which can reduce the time required for the second electronic device to obtain the three-dimensional face sub-image. Image delay, reducing the delay in obtaining 3D video.

Optionally, the network transmission module 430 may be configured to send the encoded two-dimensional image of the scene to the second electronic device.

Optionally, the three-dimensional face generating module 440 may be configured to obtain a three-dimensional face image according to the depth image of the face and the two-dimensional image of the face. Specifically, the three-dimensional face generation module 440 can be used to obtain the first three-dimensional sub-image of the human face according to the first subunit and the third sub-unit, and send the first three-dimensional sub-image of the human face to the following display module 450; After the first 3D sub-image of human face, the second 3D sub-image of human face is obtained according to the second subunit and the fourth subunit, and the second 3D sub-image of human face is sent to the display module 450 described below.

Optionally, the three-dimensional face generation module 440 may be configured to receive the two-dimensional image of the scene from the face image acquisition module 410 . Alternatively, the three-dimensional face generation module 440 may be used to obtain a two-dimensional image of the scene according to the two-dimensional image of the face, and send it to the display module 450 described below. Wherein, the two-dimensional face image includes a scene image and a face image.

Optionally, the three-dimensional face generation module 440 may be configured to send the two-dimensional image of the scene to the display module 450 .

Optionally, the display module 450 is configured to superimpose the first 3D sub-image of the face with the 2D image of the scene. After superimposing the first three-dimensional face sub-image and the two-dimensional scene image, superimposing the second three-dimensional face sub-image and the two-dimensional scene image. In this way, the display module 450 can superimpose multiple 3D sub-images of the face and the 2D scene image in a pipelined manner to display the 3D image of the face.

FIG. 6 is a third structural schematic diagram of an electronic device provided by an embodiment of the present application.

The electronic device 600 shown in FIG. 6 may be a second electronic device, that is, the electronic device 600 may be a receiving end of a 3D video. As shown in FIG. 6 , the second electronic device 600 provided by the embodiment of the present application may include a network transmission module 610 , a video decoding module 620 , a three-dimensional face generation module 630 and a display module 640 . Optionally, the second electronic device 600 may further include a touch module 650 .

It should be noted that the modules shown in FIG. 6 can be implemented by electronic hardware, computer software, or a combination of computer software and electronic hardware. Exemplarily, when implemented by software, the module shown in FIG. 6 can be implemented as a call application in the application layer shown in FIG. 3 , or, the module shown in FIG. 6 can also be implemented as the application shown in FIG. 3 The 3D video call module in the program framework layer. When implemented using hardware, the video decoding module 620 and the three-dimensional face generation module 630 can be implemented as the processor 110 shown in Figure 2, the network transmission module 610 can be implemented as the mobile communication module 150 shown in Figure 2, and the display module 640 can be implemented as shown in Figure 2 Realized as the display screen 194 shown in FIG. 2 , the touch module 650 may be realized as the sensor module 190 shown in FIG. 2 . When it is realized by combining computer software and electronic hardware, the above-mentioned way of implementing by using software and the way of implementing by using hardware can be combined, and the embodiments of the present application will not repeat them here.

Wherein, the network transmission module 610 may be configured to receive the encoded depth image of the face and the two-dimensional image of the face. Specifically, the network transmission module 610 may be configured to receive the first encoding unit from the first electronic device, and send the first encoding unit to the video decoding module 620; after receiving and sending the first encoding unit, receive the first encoding unit from the first electronic device The second coding unit, sending the second coding unit to the video decoding module 620 . Optionally, the network transmission module 610 may be configured to receive the encoded two-dimensional image of the scene and send it to the video decoding module 620 .

The video decoding module 620 can be used to decode the encoded face depth image and face two-dimensional image. Specifically, the video decoding module 620 can be used to obtain the first subunit and the third subunit according to the first coding unit, and send the first subunit and the third subunit to the following three-dimensional face generation module 630; after obtaining and sending After the first subunit and the third subunit, the second subunit and the fourth subunit are obtained according to the second coding unit, and the second subunit and the fourth subunit are sent to the 3D face generation module 630 described below. Wherein, the first subunit and the second subunit are subunits in the depth image of the face respectively, the third subunit and the fourth subunit are subunits in the two-dimensional image of the face respectively, and the first subunit corresponds to the third subunit unit, the second subunit corresponds to the fourth subunit.

Optionally, the video decoding module 620 may be specifically configured to parse the first coding unit to obtain the first sub-unit and the third coding unit; and decode the third coding unit to obtain the third sub-unit. After the third coding unit is decoded, the second coding unit is parsed to obtain the second sub-unit and the fourth coding unit; the fourth coding unit is decoded to obtain the fourth sub-unit. It should be noted that the decoding method of the video decoding module 620 corresponds to the encoding method of the video encoding module 420 , and this application does not limit the specific decoding method of the video decoding module 620 .

Optionally, the video decoding module 620 may be configured to decode the encoded two-dimensional image of the scene to obtain the two-dimensional image of the scene, and send the two-dimensional image of the scene to the three-dimensional face generating module 630 .

The three-dimensional face generation module 630 can be used to obtain a three-dimensional face image according to the depth image of the face and the two-dimensional image of the face. Specifically, the 3D face generation module 630 may be configured to obtain the first 3D sub-image of the face according to the first subunit and the third subunit, and send the first 3D sub-image of the face to the display module 640 . After the first 3D sub-image of human face is obtained and sent, the second 3D sub-image of human face is obtained according to the second subunit and the fourth subunit, and the second 3D sub-image of human face is sent to the display module 640 .

Optionally, the three-dimensional face generating module 630 may be used to obtain a two-dimensional image of a scene based on a two-dimensional image of a human face. Wherein, the two-dimensional face image includes a scene image and a face image.

Optionally, the three-dimensional face generation module 630 may be configured to send a two-dimensional scene image to the display module 640 described below.

The display module 640 is configured to superimpose the first 3D sub-image of the face with the 2D image of the scene. After superimposing the first three-dimensional face sub-image and the two-dimensional scene image, superimposing the second three-dimensional face sub-image and the two-dimensional scene image. In this way, the display module 640 can superimpose multiple 3D sub-images of the face and the 2D scene image in a pipelined manner to display the 3D image of the face.

Optionally, the touch module 650 is configured to detect adjustment actions.

Optionally, the display module 640 may be configured to adjust and display the angle of the human face in the overall three-dimensional image according to the adjustment action. That is to say, the electronic device 600 can adjust and display different angles of faces in the 3D video.

The three-dimensional video call method provided by the embodiment of the present application will be described in detail below with reference to FIGS. 7-14 .

FIG. 7 is a first schematic flowchart of a three-dimensional video call method provided by an embodiment of the present application.

As shown in Figure 7, the three-dimensional video call method includes the following steps:

S701. The first electronic device acquires a face depth image and a two-dimensional face image, divides the face depth image into multiple subunits including a first subunit and a second subunit, and divides the two-dimensional face image into multiple subunits including a second subunit. Three subunits and multiple subunits for a fourth subunit.

Specifically, the first subunit corresponds to the third subunit, and the second subunit corresponds to the fourth subunit. That is to say, the multiple sub-units of the depth image of the face are in one-to-one correspondence with the sub-units of the two-dimensional image of the face.

FIG. 8 is a schematic diagram of a face depth image and a two-dimensional face image provided by an embodiment of the present application. Take the face depth image and face two-dimensional image divided into four sub-units as an example. As shown in Figure 8, the face depth image is divided into subunit 1, subunit 2, subunit 3 and subunit 4, and the face two-dimensional image is divided into subunit 5, subunit 6, subunit 7 and subunit 8 . Wherein, subunit 1 corresponds to subunit 5 , subunit 2 corresponds to subunit 6 , subunit 3 corresponds to subunit 7 , and subunit 4 corresponds to subunit 8 .

It should be noted that FIG. 8 is only a way of dividing the face depth image and the face two-dimensional image into multiple sub-units proposed by the embodiment of the present application. For example, the face depth image and the face two-dimensional The three-dimensional image is divided into multiple subunits along the vertical direction, which is not limited in this application.

In a possible design manner, in the above S701, the first electronic device acquires the depth image of the face and the two-dimensional image of the face, which may include the following steps 1 to 3.

Step 1, the first electronic device receives face depth information.

Exemplarily, the face depth information may be collected by a high-precision depth camera, such as a TOF sensor, a three-dimensional structured light sensor, and the like. FIG. 9 is a first schematic diagram of the application of the first electronic device provided by the embodiment of the present application. Referring to FIG. 9 , the face image acquisition module 410 may receive face depth information.

Step 2, the first electronic device receives the two-dimensional face information.

Exemplarily, the two-dimensional face information may include face information. Optionally, the two-dimensional face information may also include scene information in the current video scene.

In some embodiments, the first electronic device may receive scene two-dimensional information. Wherein, the scene two-dimensional information may include scene information.

Exemplarily, both the two-dimensional face information and the two-dimensional scene information may be collected by a two-dimensional camera, such as an RGB sensor. Referring to FIG. 9 , the face image acquisition module 410 may receive two-dimensional face information. Optionally, the face image acquisition module 410 may also receive scene two-dimensional information.

Step 3, the first electronic device obtains a face depth image according to the face depth information, and obtains a two-dimensional face image according to the two-dimensional information of the face.

Referring to FIG. 9 , the face image acquisition module 410 converts the face depth information into a face depth image, and converts the two-dimensional face information into a two-dimensional face image. When the two-dimensional face information includes face information but does not include scene information, the two-dimensional face image includes face images; when the two-dimensional face information includes face information and scene information, the two-dimensional face image includes face images and scene images.

Optionally, the face image acquisition module 410 may convert the two-dimensional information of the scene into a two-dimensional image of the scene.

It should be noted that the embodiment of the present application does not limit the execution sequence of the above steps 1 to 3, and the depth image and the two-dimensional image of the face shall prevail.

In conjunction with FIG. 9, the face image acquisition module 410 divides the face depth image into a plurality of subunits including a first subunit and a second subunit, and divides a two-dimensional face image into a plurality of subunits including a third subunit and a fourth subunit. multiple subunits. Optionally, the face image acquisition module 410 transmits multiple sub-units of the face depth image to the video encoding module 420 and/or the face three-dimensional generation module 440 through the face depth image cache. The face image acquisition module 410 transmits multiple subunits of the face two-dimensional image and/or the scene two-dimensional image to the video encoding module 420 and/or the face three-dimensional generation module 440 through the face two-dimensional image cache.

In another possible design manner, in the above S701, the first electronic device acquires the depth image of the face and the two-dimensional image of the face, which may include the following steps 4 to 6.

Step 4, the first electronic device collects face depth information.

FIG. 10 is a second schematic diagram of the application of the first electronic device provided by the embodiment of the present application. Referring to FIG. 10 , the face image acquisition module 410 may include: a face depth image acquisition submodule 411 , a two-dimensional image acquisition submodule 412 and an image signal processing submodule 413 . Wherein, the face depth image collection sub-module 411 can collect face depth information, and send the face depth information to the image signal processing sub-module 413 .

It should be noted that the difference between the first electronic device shown in FIG. 10 and the first electronic device shown in FIG. 9 lies in that the structure of the face image acquisition module 410 is different, and other parts are the same. The following detailed explanations about the video encoding module 420 , the network transmission module 430 , the face 3D generation module 440 and the display module 450 are applicable to the first electronic device shown in FIG. 9 and FIG. 10 .

Step five, the first electronic device collects two-dimensional face information.

Referring to FIG. 10 , the 2D image collection submodule 412 can collect 2D face information and/or scene 2D information, and send the 2D face information and/or scene 2D information to the image signal processing submodule 413 . For the specific elaboration of the two-dimensional information of the face and the two-dimensional information of the scene, please refer to the above step two, which will not be repeated here.

Step 6, the first electronic device obtains the face depth image according to the face depth information, and obtains the face two-dimensional image according to the two-dimensional information of the face.

Referring to FIG. 10 , the image signal processing sub-module 413 can receive the face depth information from the face depth image acquisition sub-module 411 , and convert the face depth information into a face depth image. The image signal processing sub-module 413 can receive the two-dimensional face information and/or the two-dimensional information of the scene from the two-dimensional image acquisition sub-module 412, convert the two-dimensional face information into a two-dimensional image of the face, and convert the two-dimensional information of the scene into 2D image of the scene.

It should be noted that, for the specific elaboration on the two-dimensional image of the face and the two-dimensional image of the scene, reference may be made to the third step above. The embodiment of the present application does not limit the execution order of the above-mentioned steps 4 to 6, as long as the depth image of the face and the two-dimensional image of the face can be acquired.

10, the image signal processing submodule 413 in the face image acquisition module 410 divides the depth image of the face into a plurality of subunits including the first subunit and the second subunit, and divides the two-dimensional image of the face into subunits including the second subunit. Three subunits and multiple subunits for a fourth subunit. Optionally, the image signal processing sub-module 413 transmits multiple sub-units of the face depth image to the video encoding module 420 and/or the three-dimensional face generation module 440 through the face depth image cache. The image signal processing sub-module 413 transmits multiple sub-units of the two-dimensional face image and/or the two-dimensional scene image to the video encoding module 420 and/or the three-dimensional face generation module 440 through the two-dimensional face image cache.

In the embodiment of the present application, the first electronic device performs pipeline processing on the face depth image and the two-dimensional face image at a granularity of a pair of sub-units. Wherein, a pair of sub-units includes a sub-unit of the depth image of the face and a sub-unit of the two-dimensional image of the face corresponding to the sub-unit of the depth image of the face.

FIG. 11 is a second schematic flow diagram of a three-dimensional video calling method provided by an embodiment of the present application. Wherein, a sub-slice may include a pair of sub-units, or information related to the pair of sub-units. Referring to FIG. 8 and FIG. 11 , sub-slice 1 may include sub-unit 1 and sub-unit 5 , or sub-slice 1 may be a coding unit 1 a obtained after processing sub-unit 1 and sub-unit 5 , or a three-dimensional face sub-image 1 . The sub-slice 2 may include a sub-unit 2 and a sub-unit 6, or the sub-slice 2 may be a coding unit 2a obtained after processing the sub-unit 2 and the sub-unit 6, or a three-dimensional face sub-image 2. Similarly, sub-slice 3 may include sub-unit 3 and sub-unit 7 , or sub-slice 3 may be coding unit 3 a obtained after processing sub-unit 3 and sub-unit 7 , or face three-dimensional sub-image 3 . The sub-slice 4 may include a sub-unit 4 and a sub-unit 8 , or the sub-slice 4 may be a coding unit 4 a obtained after processing the sub-unit 4 and the sub-unit 8 , or a three-dimensional face sub-image 4 .

Referring to FIG. 11 , the video processing period is T, and the face image acquisition module 410 can obtain subunit 1 and subunit 5 within the first T time, and send them to the video encoding module 420 . Then, within the second T time, subunit 2 and subunit 6 are obtained and sent to the video encoding module 420 . During the third T time, subunit 3 and subunit 7 are obtained and sent to the video encoding module 420 . In the fourth T time, the subunit and subunit 8 are obtained and sent to the video encoding module 420 .

Similarly, the human face image acquisition module 410 can be a pair of subunits as the granularity, and send multiple subunits of the depth image of the human face and multiple subunits of the two-dimensional image of the human face to the three-dimensional face generation module 440 in a pipelined manner, as shown in FIG. 11 It is not shown, and will not be described in detail here.

It should be noted that, for the convenience of description, the time period T shown in FIG. 11 is equal to the maximum value of the processing time of each sub-slice by the module of the electronic device. The time of the sub-slices, the embodiment of the present application does not limit the size of the time period T. The modules of the electronic device include but are not limited to the modules shown in FIGS. 4-6 .

In some embodiments, the electronic device may process each sub-slice using a pipeline processing scheme. Exemplarily, if the time period T is greater than the processing time of each sub-slice by the module of the electronic device, the module of the electronic device may process sub-slice 1 within a T time, and then process sub-slice 2. Exemplarily, the face image acquisition module 410 can acquire subunit 1 and subunit 5 at the beginning of the first T time, and send them to the video encoding module 420, and complete the process at three quarters of the first T time . Then sub-unit 2 and sub-unit 6 are obtained and sent to the video coding module 420, and the process is completed at one-half of the second T time. Similarly, each module of the electronic device completes the processing of each sub-chip, which will not be listed here.

In other embodiments, the electronic device may process each sub-slice using a timing processing scheme. Exemplarily, if the time period T is longer than the processing time of each sub-slice by each module in the electronic device, one or more modules in the multiple modules of the electronic device may process each sub-slice regularly. For example, after the processing of sub-slice 1 is completed in three quarters of the first T time, sub-slice 2 is not processed immediately, but sub-slice 2 is not processed until the second T time starts. Exemplarily, the face image acquisition module 410 acquires subunit 1 and subunit 5 at the beginning of the first T time, and sends them to the video encoding module 420, and completes the process at three quarters of the first T time. After waiting for a period of time, when the second T time begins, subunit 2 and subunit 6 are obtained and sent to the video encoding module 420, and the process is completed at three quarters of the second T time. Wait for a while, and when the third T time starts, start processing the next sub-slice. Similarly, each module of the electronic device completes the processing of each sub-chip, which will not be listed here.

It should be noted that the embodiment of the present application does not limit whether each sub-module of the electronic device processes each sub-slice regularly, nor does it limit whether some sub-modules or all modules of the electronic device regularly process each sub-slice. Some modules process each sub-slice regularly, or each module in the electronic device processes each sub-slice regularly, or each module in the electronic device processes each sub-slice irregularly.

Optionally, if the time for one or more modules of the electronic device to process one or more sub-slices is longer than the time period T, the electronic device may process each sub-slice in the following manner, subject to the normal operation of the electronic device. For example, for the pipelined processing solution, if the time for processing M sub-slices is less than the first sub-slice threshold, the modules of the electronic device can process each sub-slice in a pipelined manner, wherein the first sub-slice threshold can be a preset processing The maximum time of M sub-slices; otherwise, the module of the electronic device can process some sub-slices in the M sub-slices, and discard the processing of another part of the sub-slices (for example, the processing result of the sub-slice corresponding to the previous frame image can be used replace), M is an integer greater than 1. For another example, for the timing processing scheme, take the time period T less than the time for processing sub-slice 1 by the face image acquisition module 410 as an example, the face image acquisition module 410 can abandon the processing of sub-slice 1 (for example, the last frame can be used The processing result of the sub-slice 1 corresponding to the image is replaced), and at the beginning of the second T time, the sub-slice 2 is processed regularly. The processing time of one or more sub-slices by one or more modules of the electronic device is longer than the time period T, which may be caused by failure of one or some modules of the electronic device, which is not limited in this application. Abandoning the processing of the sub-slices includes: terminating the processing of the sub-slices when the processing of the sub-slices has started but the processing result has not been obtained.

S702. The first electronic device obtains the first coding unit according to the first subunit and the third subunit; after obtaining the first coding unit, obtains the second coding unit according to the second subunit and the fourth subunit.

In a possible design manner, the above S702 may include the following steps 7 to 10.

In step seven, the first electronic device encodes the third subunit to obtain a third encoding unit.

Exemplarily, the third coding unit may be a video coding layer (video coding layer, VCL) type network abstraction layer (network abstraction layer, NAL) unit.

FIG. 12 is a schematic structural diagram of a code stream provided by an embodiment of the present application. As shown in FIG. 12 , the code stream may include a header (NALU Header), a (sequence paramater set, SPS) sequence parameter set, a (picture paramater set, PPS) picture parameter set, and at least one coding unit. The first electronic device may encode the subunit 5 into an encoding unit 1b, and the encoding unit 1b is a VCL type NAL unit.

Referring to FIG. 9 or FIG. 10 , the video coding module 420 may code the third sub-unit to obtain a third coding unit.

Step eight, the first electronic device obtains the first coding unit according to the first sub-unit and the third coding unit.

Exemplarily, the first coding unit may be a NAL unit of supplemental enhancement information (supplemental enhancement information, SEI) type.

Referring to FIG. 12 , the first electronic device may fill the subunit 1 into a preset field of the coding unit 1b to obtain the coding unit 1a, and the coding unit 1a is an SEI-type NAL unit. Optionally, the preset field may be a supplemental enhancement information SEI field.

Referring to FIG. 9 or FIG. 10 , the video encoding module 420 may encode the first sub-unit and the third coding unit to obtain the first coding unit.

It should be noted that this embodiment of the present application does not limit the order in which the electronic device encodes the first subunit and the third subunit. For example, the first subunit may be encoded first, and then the encoded first subunit Perform mixed coding with the third sub-unit to obtain the first coding unit.

In the embodiment of the present application, the first electronic device encodes the sub-units of the depth image of the face and the sub-units of the two-dimensional image of the face into the same coding unit, avoiding combining the sub-units of the depth image of the face with the sub-units of the two-dimensional image of the face The sub-units are transmitted independently, which can reduce the increased complexity caused by the transmission of multiple barcode streams and time synchronization.

In step nine, the first electronic device encodes the fourth subunit to obtain a fourth encoding unit.

Exemplarily, the fourth coding unit may be a VCL type NAL unit.

Referring to FIG. 12 , the first electronic device may encode the subunit 6 into a coding unit 2b, and the coding unit 2b is a VCL type NAL unit.

Referring to FIG. 9 or FIG. 10 , the video encoding module 420 may encode the fourth sub-unit to obtain a fourth coding unit.

Step ten, the first electronic device obtains the second encoding unit according to the second subunit and the fourth encoding unit.

Exemplarily, the second coding unit may be an SEI-type NAL unit.

Referring to FIG. 12 , the first electronic device may fill the subunit 2 into the preset field of the coding unit 2b to obtain the coding unit 2a, and the coding unit 2a is an SEI type NAL unit. Optionally, the preset field may be a supplemental enhancement information SEI field.

Referring to FIG. 9 or FIG. 10 , the video encoding module 420 may encode the second sub-unit and the fourth coding unit to obtain the second coding unit.

Similarly, with reference to FIG. 12 , the first electronic device can obtain the encoding unit 3b, the encoding unit 3a, the encoding unit 4b, and the encoding unit 4a in the manner described in the above steps 7 to 8, which will not be described in detail here.

In this embodiment of the present application, the video coding module of the first electronic device can perform pipeline coding on the sub-units of the face depth image and the sub-units of the two-dimensional face image received in a pipeline, thereby reducing the number of people obtained by the second electronic device. Time-lapse of 3D images of faces.

With reference to FIG. 9 or FIG. 10, the video encoding module 420 can encode the first subunit and the third subunit to obtain the first coding unit; after obtaining the first coding unit, perform encoding on the second subunit and the fourth subunit Encode to obtain the second coding unit. Optionally, the video coding module 420 may send the coded first coding unit and the second coding unit to the network transmission module 430 through the coding unit cache.

Referring to FIG. 11 , within the second T time, the video encoding module 420 can encode the subunit 1 and the subunit 5 to obtain the encoding unit 1 a and send it to the network transmission module 430 . In the third time T, the video encoding module 420 encodes the subunit 2 and the subunit 6 to obtain the encoding unit 2a, and sends it to the network transmission module 430 . Similarly, in the fourth time T, the video encoding module 420 can encode the subunit 3 and the subunit 7 to obtain the encoding unit 3 a and send it to the network transmission module 430 . During the fifth T time, the video encoding module 420 can encode the sub-unit 4 and the sub-unit 8 to obtain the encoding unit 4 a and send it to the network transmission module 430 .

S703. The first electronic device sends the first encoding unit to the second electronic device; after sending the first encoding unit, sends the second encoding unit to the second electronic device.

Referring to FIG. 9 or FIG. 10 , the network transmission module 430 may send the first encoding unit to the second electronic device; after sending the first encoding unit, send the second encoding unit to the second electronic device.

In this embodiment of the present application, the first electronic device may send the encoded face depth image sub-unit and the face two-dimensional image sub-unit to the second electronic device in a pipelined manner, so that the second electronic device may acquire the three-dimensional face image in a pipelined manner. The image can reduce the time delay for the second electronic device to obtain the three-dimensional image of the face, and improve the efficiency of obtaining the three-dimensional video.

Referring to FIG. 11 , within the third time T, the network transmission module 430 may send the encoding unit 1a to the second electronic device. During the fourth T time, the network transmission module 430 sends the encoding unit 2a to the second electronic device. During the fifth T period, the network transmission module 430 sends the encoding unit 3a to the second electronic device. During the sixth T time, the network transmission module 430 sends the encoding unit 4a to the second electronic device.

In some embodiments, the first electronic device may periodically send the first encoding unit and the second encoding unit to the second electronic device.

Referring to FIG. 9 or FIG. 10 , the network transmission module 430 may periodically send the first encoding unit and the second encoding unit to the second electronic device.

In some embodiments, the first electronic device can display the 3D video of the current video scene of the user of the first electronic device, and the first electronic device can analyze the multiple subunits of the depth image of the face and the multiple subunits of the two-dimensional image of the face Pipeline processing is performed to improve the efficiency of acquiring 3D video.

Optionally, the 3D video call method provided in the embodiment of the present application may further include: the first electronic device obtains the first 3D sub-image of human face according to the first subunit and the third subunit; After the image is obtained, the second three-dimensional sub-image of human face is obtained according to the second subunit and the fourth subunit.

Referring to FIG. 9 or FIG. 10 , the 3D face generation module 440 can obtain the first 3D sub-image of the face according to the first subunit and the third subunit, and send the first 3D sub-image of the face to the display module 450 . After the first 3D sub-image of human face is obtained and sent, the second 3D sub-image of human face is obtained according to the second subunit and the fourth subunit, and the second 3D sub-image of human face is sent to the display module 450 .

Optionally, the 3D face generation module 440 may use an artificial intelligence algorithm model or a 3D face generation algorithm model to obtain a 3D face sub-image according to the sub-units of the depth image of the face and the sub-units of the 2-D face image.

Referring to FIG. 8 and FIG. 9 , the three-dimensional face generation module 440 can generate a three-dimensional face sub-image 1 according to the subunit 1 and the subunit 5 and send it to the display module 450 . Then, a three-dimensional face sub-image 2 is generated according to the sub-unit 2 and the sub-unit 6 , and sent to the display module 450 . The three-dimensional face sub-image 3 is generated according to the sub-unit 3 and the sub-unit 7 , and sent to the display module 450 . Finally, a three-dimensional face sub-image 4 is generated according to the sub-unit 4 and the sub-unit 8 , and sent to the display module 450 . It should be noted that the three-dimensional human face generation module 440 shown in FIG. 10 can realize the same function, which will not be repeated here.

Optionally, the three-dimensional face generation module 440 may send the two-dimensional image of the scene to the display module 450 . Exemplarily, when the two-dimensional face image includes a scene image and a face image, the two-dimensional scene image may be obtained by the three-dimensional face generating module 440 according to the two-dimensional face image. Alternatively, the two-dimensional image of the scene may be sent by the face image acquisition module 410 to the three-dimensional face generation module 440 .

Optionally, the 3D video call method provided in the embodiment of the present application may further include: the first electronic device superimposes the first 3D sub-image of human face with the 2D image of the scene; After the two-dimensional image is superimposed, the second human face three-dimensional sub-image is superimposed on the scene two-dimensional image.

Referring to FIG. 9 or FIG. 10 , the display module 450 may superimpose the first 3D sub-image of the human face with the 2D scene image. After superimposing the first three-dimensional face sub-image and the two-dimensional scene image, superimposing the second three-dimensional face sub-image and the two-dimensional scene image. Optionally, the display subsystem of the display module 450 can receive multiple 3D sub-images of faces and 2D images of the scene through the preview buffer pipeline, and superimpose the 3D subsystem of the face with the 2D scene images in a pipeline to obtain the face The three-dimensional image is then transferred to a display screen for display.

Exemplarily, the display module 450 can superimpose the 3D sub-image 1 of the face with the 2D scene image, then superimpose the 3D sub-image 2 of the face with the 2D scene image, and superimpose the 3D sub-image 3 of the face with the 2D scene image. The three-dimensional image is superimposed, and finally, the three-dimensional sub-image 4 of the face is superimposed on the two-dimensional image of the scene, so as to obtain the three-dimensional image of the face.

In this way, the first electronic device can obtain multiple frames of three-dimensional face images by executing the above S701-S704 multiple times, so as to display the three-dimensional video of the user of the first electronic device. Processing the image and the two-dimensional image of the face can reduce the time delay for the first electronic device to acquire the three-dimensional image of the face, thereby improving the efficiency of displaying the three-dimensional video.

S704. The second electronic device receives the first encoding unit from the first electronic device; after receiving the first encoding unit, receives the second encoding unit from the first electronic device.

FIG. 13 is a schematic diagram of the application of the second electronic device provided by the embodiment of the present application. Referring to FIG. 13 , the network transmission module 610 may be configured to receive the first coding unit from the first electronic device, and send the first coding unit to the video decoding module 620 . After receiving and sending the first encoding unit, receive the second encoding unit from the first electronic device, and send the second encoding unit to the video decoding module 620 .

Referring to FIG. 11 , within the fourth time T, the network transmission module 610 acquires the encoding unit 1 a and sends the encoding unit 1 a to the video decoding module 620 . During the fifth T time, the encoding unit 2 a is acquired and sent to the video decoding module 620 . During the sixth T time, the encoding unit 3 a is acquired and sent to the video decoding module 620 . During the seventh T time, the encoding unit 4 a is acquired and sent to the video decoding module 620 .

S705, the second electronic device obtains the first subunit and the third subunit according to the first coding unit; after obtaining the first subunit and the third subunit, obtains the second subunit and the fourth subunit according to the second coding unit .

Referring to FIG. 13 , the video decoding module 620 may decode the first coding unit to obtain the first sub-unit and the third sub-unit. Optionally, the video decoding module 620 may transmit the first subunit to the 3D face generation module 440 through the face depth image buffer, and transmit the third subunit to the 3D face generation module 440 through the 2D face image buffer. Similarly, the video decoding module 620 may decode the second coding unit to obtain the second sub-unit and the fourth sub-unit. Optionally, the video decoding module 620 may transmit the second subunit to the 3D face generation module 440 through the face depth image cache, and transmit the fourth subunit to the 3D face generation module 440 through the 2D face image cache.

Referring to FIG. 11 , the video decoding module 620 decodes the encoding unit 1a to obtain subunit 1 and subunit 5 within the fifth time T, and sends them to the 3D face generation module 630 . In the sixth T time, decode the encoding unit 2a to obtain the subunit 2 and the subunit 6, and send them to the three-dimensional face generation module 630 . In the seventh T time, decode the encoding unit 3 a to obtain subunit 3 and subunit 7 , and send them to the face three-dimensional generating module 630 . In the eighth time T, the encoding unit 4a is decoded to obtain the subunit 4 and the subunit 8, and sent to the three-dimensional face generation module 630 .

In some embodiments, the above S705 may include the following steps eleven and fourteen.

Step eleven, the second electronic device parses the first coding unit to obtain the first sub-unit and the third coding unit.

Referring to FIG. 13 , the video decoding module 620 may analyze the first coding unit to obtain the first sub-unit and the third coding unit. Exemplarily, the video decoding module 620 may analyze the coding unit 1a to obtain the sub-unit 1 and the coding unit 1b.

In step 12, the second electronic device decodes the third encoding unit to obtain a third subunit.

Referring to FIG. 13 , the video decoding module 620 may decode the third coding unit to obtain the third sub-unit. Exemplarily, the video decoding module 620 may decode the coding unit 1 b to obtain the sub-unit 5 .

Step 13: After decoding the third coding unit, the second electronic device parses the second coding unit to obtain the second sub-unit and the fourth coding unit.

Referring to FIG. 13 , the video decoding module 620 may analyze the second coding unit to obtain the second sub-unit and the fourth coding unit. Exemplarily, the video decoding module 620 may analyze the coding unit 2a to obtain the sub-unit 2 and the coding unit 2b.

Step fourteen, the second electronic device decodes the fourth encoding unit to obtain a fourth subunit.

Referring to FIG. 13 , the video decoding module 620 may decode the fourth coding unit to obtain the fourth sub-unit. Exemplarily, the video decoding module 620 may decode the coding unit 2 b to obtain the sub-unit 6 .

Similarly, the second electronic device can obtain the subunit 3, the subunit 7, the subunit 4, and the subunit 8 in the manner described in the above steps 11 to 12, which will not be described in detail here.

Referring to FIG. 11 , the video decoding module 620 parses the coding unit 1a within the fifth T time to obtain subunit 1 and coding unit 1b, decodes coding unit 1b to obtain subunit 5, and then converts subunit 1 and subunit 5 Send to the face three-dimensional generation module 630. In the sixth T time, the encoding unit 2a is analyzed to obtain the subunit 2 and the encoding unit 2b, and the encoding unit 2b is decoded to obtain the subunit 6, and then the subunit 2 and the subunit 6 are sent to the face three-dimensional generation module 630 . In the seventh T time, the encoding unit 3a is analyzed to obtain the subunit 3 and the encoding unit 3b, and the encoding unit 3b is decoded to obtain the subunit 7, and then the subunit 3 and the subunit 7 are sent to the face three-dimensional generation module 630 . In the eighth T time, the encoding unit 4a is analyzed to obtain the subunit 4 and the encoding unit 4b, and the encoding unit 4b is decoded to obtain the subunit 8, and then the subunit 4 and the subunit 8 are sent to the face three-dimensional generation module 630 .

S706, the second electronic device obtains the first three-dimensional face sub-image according to the first subunit and the third subunit; after obtaining the first three-dimensional face sub-image, obtains the second person according to the second subunit and the fourth subunit 3D sub-image of the face.

Referring to FIG. 13 , the 3D face generation module 630 can obtain the first 3D subimage of the face according to the first subunit and the third subunit, and send the first 3D subimage of the face to the display module 640 . After the first 3D sub-image of human face is obtained and sent, the second 3D sub-image of human face is obtained according to the second subunit and the fourth subunit, and the second 3D sub-image of human face is sent to the display module 640 .

Referring to FIG. 11 , the 3D face generation module 630 can generate a 3D face sub-image 1 according to the subunit 1 and the subunit 5 within the sixth time T, and send it to the display module 640 . Then, within the seventh T time, generate a three-dimensional face sub-image 2 according to the sub-unit 2 and the sub-unit 6 , and send it to the display module 640 . In the eighth time T, a three-dimensional face sub-image 3 is generated according to the sub-unit 3 and the sub-unit 7 , and sent to the display module 640 . In the ninth time T, generate the three-dimensional face sub-image 4 according to the sub-unit 4 and the sub-unit 8 , and send it to the display module 640 .

Optionally, the three-dimensional face generation module 630 may send the two-dimensional image of the scene to the display module 640 . When the two-dimensional face image includes a scene image, the two-dimensional scene image may be obtained by the three-dimensional face generation module 630 according to the two-dimensional face image. When the two-dimensional face image does not include a scene image, the two-dimensional scene image may be sent to the three-dimensional face generation module 630 by the first electronic device.

S707. The second electronic device superimposes the first three-dimensional face sub-image and the two-dimensional scene image; after superimposing the first three-dimensional face sub-image and the two-dimensional scene image, the second three-dimensional face sub-image and the scene Two-dimensional images are superimposed.

13, the display module 640 can superimpose the first three-dimensional face sub-image and the two-dimensional scene image; after superimposing the first three-dimensional face sub-image and the two-dimensional scene image, the second three-dimensional face sub-image Overlay with the 2D image of the scene. Optionally, the display subsystem of the display module 640 can receive multiple 3D sub-images of faces and 2D images of the scene through the preview buffer pipeline, and superimpose the 3D subsystem of the face with the 2D scene images in a pipeline to obtain the face The three-dimensional image is then transferred to a display screen for display.

Exemplarily, the display module 640 may superimpose the 3D face image and the 2D scene image to obtain the 3D face image. Specifically, referring to FIG. 11 , during the seventh time T, the display module 640 may superimpose the 3D sub-image 1 of the face with the 2D image of the scene. In the eighth time T, the face 3D sub-image 2 is superimposed on the scene 2D image. In the ninth time T, the face 3D sub-image 3 is superimposed on the scene 2D image. In the tenth time T, the three-dimensional sub-image 4 of the face is superimposed on the two-dimensional image of the scene, so as to obtain the three-dimensional image of the face.

In a possible design manner, the 3D video call method provided in the embodiment of the present application may further include: detecting an adjustment action, and adjusting the angle of the face in the 3D image of the displayed face in response to the adjustment action.

Referring to FIG. 13 , the touch module 650 is configured to detect an adjustment action, and the display module 640 may be configured to adjust the angle of a human face displayed in a three-dimensional image of a human face in response to the adjustment action.

Exemplarily, the adjustment action may be a left angle or a right angle set by the user. For example, the electronic device can determine the frontal display angle of the three-dimensional image of the face as 0 degrees, and the display interface of the electronic device can include a left angle setting area and a right angle setting area, and the user can set the left angle setting area or the right angle setting area. Use the value in the Angle setting area to adjust the angle of the face display.

Exemplarily, the adjustment action may be a rotation action performed by the user on the touch screen. For example, a user places two fingers on the touch screen to swivel in a clockwise or counterclockwise direction. As shown in (a) in Figure 14, in the current 3D video, the frontal 3D image of the face is displayed, and the user uses two fingers to rotate clockwise on the touch screen to adjust the display angle of the face as shown in Figure 14 As shown in (b), the 3D image of the left side of the human face is displayed.

It should be noted that the 3D video call method provided in the above embodiments of the present application is described by taking the three-dimensional display of the face in the call video as an example. The 3D video call method provided in the embodiment of the present application can also display the face of the person in the call video Both the face and the scene are displayed in three dimensions, and the specific implementation method is similar to the above S701-S707. The above-mentioned face depth information may include the depth information of the face and the scene, and the face depth image may include the depth image of the face and the depth image of the scene. No more details here.

Based on the three-dimensional video call method shown in FIG. 7 , the first electronic device divides the depth image of the face and the two-dimensional image of the face into a plurality of subunits, and processes the depth image of the face and the two-dimensional image of the face with a pair of subunits as the granularity pipeline Encode the two-dimensional image of the face, and send it to the second electronic device in a pipeline, and the second electronic device receives and decodes it in a pipeline, and obtains the subunit of the depth image of the face and the subunit of the two-dimensional image of the face, and obtains the face in a pipeline 3D subimage. In this way, the time delay for the second electronic device to acquire the 3D image of the face can be reduced, thereby reducing the time delay for acquiring the 3D video.

An embodiment of the present application provides a three-dimensional video call system. The system includes the above-mentioned one or more first electronic devices, and one or more second electronic devices.

An embodiment of the present application provides a computer-readable storage medium. Computer programs or instructions are stored on the computer-readable storage medium. When the computer program or instructions are run on a computer, the computer executes the three-dimensional video recording described in the above-mentioned method embodiments. call method.

An embodiment of the present application provides a computer program product. The computer program product includes: a computer program or an instruction. When the computer program or instruction is run on a computer, the computer is made to execute the three-dimensional video calling method described in the method embodiment above.

The above-mentioned embodiments may be implemented in whole or in part by software, hardware (such as circuits), firmware, or other arbitrary combinations. When implemented using software, the above-described embodiments may be implemented in whole or in part in the form of computer program products. The computer program product comprises one or more computer instructions or computer programs. When the computer instruction or computer program is loaded or executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer may be an electronic device, a general computer, a special computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center that includes one or more sets of available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media. The semiconductor medium may be a solid state drive.

In this application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one item (piece) of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple .

It should be understood that, in various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not be used in the embodiments of the present application. The implementation process constitutes any limitation.

Those skilled in the art can appreciate that the units or modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit or module can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the above units or modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or modules can be combined Or it can be integrated into another system, or some units or modules can be ignored, or their corresponding functions are not performed. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units/modules may be in electrical, mechanical or other forms.

The units/modules described as separate components may or may not be physically separated, and the components displayed as units/modules may or may not be physical units/modules, that is, they may be located in one place, or may also be distributed to on multiple network elements/modules. Part or all of the units/modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit/module in each embodiment of the present application may be integrated into one processing unit/module, each unit/module may exist separately physically, or two or more units/modules may be integrated into one processing unit/module. unit/module.

If the functions are implemented in the form of software function units/modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other various media that can store program codes. .

In the embodiments of the present application, on the premise that there is no logical contradiction, the various embodiments may refer to each other, for example, the methods and/or terms between the method embodiments may refer to each other, such as the functions and/or terms between the device embodiments Or terms may refer to each other, for example, functions and/or terms between the apparatus embodiment and the method embodiment may refer to each other.

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (15)

1. An electronic device, comprising: the system comprises a face image acquisition module, a video coding module and a network transmission module; wherein,

   the face image acquisition module is used for acquiring a face depth image and a face two-dimensional image; dividing the face depth image into a plurality of sub-units including a first sub-unit and a second sub-unit; dividing the face two-dimensional image into a plurality of subunits including a third subunit and a fourth subunit; transmitting the first subunit and the third subunit to the video coding module; after the first subunit and the third subunit are sent, sending the second subunit and the fourth subunit to the video coding module; wherein the first subunit corresponds to the third subunit and the second subunit corresponds to the fourth subunit;

   The video coding module is used for obtaining a first coding unit according to the first subunit and the third subunit and sending the first coding unit to the network transmission module; after the first coding unit is obtained and sent, a second coding unit is obtained according to the second subunit and the fourth subunit, and the second coding unit is sent to the network transmission module;

   the network transmission module is used for sending the first coding unit to the second electronic equipment; and after the first coding unit is sent, sending the second coding unit to the second electronic equipment.
2. The electronic device of claim 1, wherein the electronic device comprises a memory device,

   the face image acquisition module is specifically used for receiving face depth information; receiving two-dimensional information of a human face; and obtaining the face depth image according to the face depth information and obtaining the face two-dimensional image according to the face two-dimensional information.
3. The electronic device according to claim 1 or 2, characterized in that,

   the video coding module is specifically configured to:

   encoding the third subunit to obtain a third encoding unit; obtaining a first coding unit according to the first subunit and the third coding unit; the method comprises the steps of,

   Encoding the fourth subunit to obtain a fourth encoding unit; and obtaining a second coding unit according to the second subunit and the fourth coding unit.
4. The electronic device of any one of claims 1-3, further comprising: the face three-dimensional generation module and the display module; wherein,

   the face three-dimensional generation module is used for obtaining a first face three-dimensional sub-image according to the first sub-unit and the third sub-unit and sending the first face three-dimensional sub-image to the display module; after the first face three-dimensional sub-image is obtained and sent, a second face three-dimensional sub-image is obtained according to the second sub-unit and the fourth sub-unit, and the second face three-dimensional sub-image is sent to the display module;

   the display module is used for superposing the first face three-dimensional sub-image and the scene two-dimensional image; and after the first face three-dimensional sub-image and the scene two-dimensional image are overlapped, the second face three-dimensional sub-image and the scene two-dimensional image are overlapped.
5. An electronic device, comprising: the system comprises a network transmission module, a video decoding module, a three-dimensional face generation module and a display module; wherein,

   The network transmission module is used for receiving a first coding unit from first electronic equipment and sending the first coding unit to the video decoding module; after receiving and transmitting the first coding unit, receiving a second coding unit from the first electronic device, and transmitting the second coding unit to the video decoding module;

   the video decoding module is used for obtaining a first subunit and a third subunit according to the first coding unit; after the first subunit and the third subunit are obtained, obtaining a second subunit and a fourth subunit according to the second coding unit; the first subunit and the second subunit are subunits in the face depth image respectively, the third subunit and the fourth subunit are subunits in the face two-dimensional image respectively, the first subunit corresponds to the third subunit, and the second subunit corresponds to the fourth subunit;

   the three-dimensional face generation module is used for obtaining a first face three-dimensional sub-image according to the first sub-unit and the third sub-unit and sending the first face three-dimensional sub-image to the display module; after the first face three-dimensional sub-image is obtained and sent, a second face three-dimensional sub-image is obtained according to the second sub-unit and the fourth sub-unit, and the second face three-dimensional sub-image is sent to the display module;

   The display module is used for superposing the first face three-dimensional sub-image and the scene two-dimensional image; and after the first face three-dimensional sub-image and the scene two-dimensional image are overlapped, the second face three-dimensional sub-image and the scene two-dimensional image are overlapped.
6. The electronic device of claim 5, wherein the electronic device comprises a memory device,

   the video decoding module is further configured to parse the first coding unit to obtain the first subunit and a third coding unit; decoding the third coding unit to obtain the third subunit; after decoding the third coding unit, analyzing the second coding unit to obtain the second subunit and a fourth coding unit; decoding the fourth coding unit to obtain the fourth subunit.
7. A three-dimensional video call method, comprising:

   acquiring a face depth image and a face two-dimensional image; dividing the face depth image into a plurality of sub-units including a first sub-unit and a second sub-unit; dividing the face two-dimensional image into a plurality of subunits including a third subunit and a fourth subunit; wherein the first subunit corresponds to the third subunit and the second subunit corresponds to the fourth subunit;

   Obtaining a first coding unit according to the first subunit and the third subunit; after the first coding unit is obtained, a second coding unit is obtained according to the second subunit and the fourth subunit;

   transmitting the first coding unit to a second electronic device; and after the first coding unit is sent, sending the second coding unit to the second electronic equipment.
8. The method of three-dimensional video call according to claim 7, wherein the acquiring the face depth image and the face two-dimensional image comprises:

   receiving face depth information; receiving two-dimensional information of a human face; and obtaining the face depth image according to the face depth information and obtaining the face two-dimensional image according to the face two-dimensional information.
9. The three-dimensional video call method according to claim 7 or 8, further comprising:

   encoding the third subunit to obtain a third encoding unit; obtaining a first coding unit according to the first subunit and the third coding unit; the method comprises the steps of,

   encoding the fourth subunit to obtain a fourth encoding unit; and obtaining a second coding unit according to the second subunit and the fourth coding unit.
10. The three-dimensional video call method according to any one of claims 7 to 9, further comprising:

    obtaining a first face three-dimensional sub-image according to the first sub-unit and the third sub-unit; after the first face three-dimensional sub-image is obtained, a second face three-dimensional sub-image is obtained according to the second sub-unit and the fourth sub-unit;

    superposing the first face three-dimensional sub-image and a scene two-dimensional image; and after the first face three-dimensional sub-image and the scene two-dimensional image are overlapped, the second face three-dimensional sub-image and the scene two-dimensional image are overlapped.
11. A three-dimensional video call method, comprising:

    receiving a first encoding unit from a first electronic device; receiving a second coding unit from the first electronic device after receiving the first coding unit;

    obtaining a first subunit and a third subunit according to the first coding unit; after the first subunit and the third subunit are obtained, obtaining a second subunit and a fourth subunit according to the second coding unit; the first subunit and the second subunit are subunits in the face depth image respectively, the third subunit and the fourth subunit are subunits in the face two-dimensional image respectively, the first subunit corresponds to the third subunit, and the second subunit corresponds to the fourth subunit;

    Obtaining a first face three-dimensional sub-image according to the first sub-unit and the third sub-unit; after the first face three-dimensional sub-image is obtained, a second face three-dimensional sub-image is obtained according to the second sub-unit and the fourth sub-unit;

    superposing the first face three-dimensional sub-image and a scene two-dimensional image; and after the first face three-dimensional sub-image and the scene two-dimensional image are overlapped, the second face three-dimensional sub-image and the scene two-dimensional image are overlapped.
12. The three-dimensional video call method of claim 11, further comprising:

    analyzing the first coding unit to obtain the first subunit and a third coding unit; decoding the third coding unit to obtain the third subunit; after decoding the third coding unit, analyzing the second coding unit to obtain the second subunit and a fourth coding unit; decoding the fourth coding unit to obtain the fourth subunit.
13. A three-dimensional video telephony system comprising an electronic device as claimed in any of claims 1 to 4, and an electronic device as claimed in any of claims 5 to 6.
14. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program or instructions which, when run on a computer, cause the computer to perform the three-dimensional video telephony method of any of claims 7 to 12.
15. A computer program product, the computer program product comprising: computer program or instructions which, when run on a computer, cause the computer to perform the three-dimensional video telephony method of any of claims 7 to 12.