CN112073664A - A video call method and display device - Google Patents

Abstract

In the embodiments provided in this application, in particular, it relates to a video calling method and a display device. The display device includes: a camera; a display for displaying a peer window and/or a local window, wherein the peer window refers to a window used to play image data sent by the mobile terminal, and the local window refers to a window. a window for playing the image data collected by the camera; a processor, where the processor is configured to: receive a push message sent by a server, where the push message is when the server receives a video call request sent by the mobile terminal sent later; when the mobile terminal is authorized by the display device, in response to the push message, a call process is automatically established, and the display is controlled to display the opposite end window but not the local window; When authorized by the display device, the video call is rejected in response to the push message.

Description

A video call method and display device

This application claims the priority of the Chinese patent application filed on June 11, 2019 with the application number of 201910502717.9 and the application title of "a video call method and display device", the entire contents of which are incorporated into this application by reference middle.

technical field

The embodiments of the present application relate to display technologies. More specifically, it relates to a video calling method and display device.

Background technique

Currently, terminal devices with cameras make people more socially connected. When a terminal device initiates a video call, the server forwards the call request to another terminal device, and the other terminal device agrees to the call request by clicking the control symbol, and then the two terminal devices can establish a video call, which can be seen in the existing video During the call establishment process, a user of another terminal device is required to operate, and the process of establishing a video call is cumbersome.

SUMMARY OF THE INVENTION

A first aspect of the present application provides a display device, including: a camera;

A display, used to display a peer window and/or a local window, wherein the peer window refers to a window used to play image data sent by the mobile terminal, and the local window refers to a window used to play the image data collected by the camera. window for image data;

a processor for:

Receive a push message sent by a server, where the push message is sent by the server after receiving a video call request sent by the mobile terminal; when the mobile terminal is authorized by the display device, respond to the Pushing a message, automatically establishing a call process, and controlling the display to display the opposite end window and not display the local window; when the mobile terminal is not authorized by the display device, in response to the push message, reject the video call.

In a second aspect, the present application provides a display device, comprising: a microphone; a speaker; a camera;

A display, used to display a peer window and/or a local window, wherein the peer window refers to a window used to play image data sent by the mobile terminal, and the local window refers to a window used to play the image data collected by the camera. window for image data;

a processor for:

Receive a push message sent by a server, where the push message is sent by the server after receiving a video call request sent by the mobile terminal; when the mobile terminal is authorized by the display device, respond to the Pushing a message, controlling the display to display the peer window but not the local window, controlling the speaker to output the audio data sent by the mobile terminal, controlling the camera to collect image data for sending to the mobile terminal, controlling the The microphone collects audio data for sending to the mobile terminal; when the mobile terminal is not authorized by the display device, the video call is rejected in response to the push message.

In a third aspect, the present application provides a video call method, which is applied to a display device, and is characterized in that, it includes:

receiving a push message sent by a server, where the push message is sent by the server after receiving a video call request sent by the mobile terminal;

When the mobile terminal is authorized by the display device, in response to the push message, automatically establish a call process;

The video call is rejected in response to the push message when the mobile terminal is not authorized by the display device.

In a fourth aspect, the present application provides a video call method, which is applied to a display device, and is characterized in that, it includes:

receiving a push message sent by a server, where the push message is sent by the server after receiving a video call request sent by the mobile terminal;

When the mobile terminal is authorized by the display device, in response to the push message, a call process is automatically established, and the display is controlled to display the opposite end window and not display the local window, wherein the opposite end window refers to using for playing the window of the image data sent by the mobile terminal, the local window refers to the window for playing the image data collected by the camera;

The video call is rejected in response to the push message when the mobile terminal is not authorized by the display device.

In a fifth aspect, the present application provides a video call method, which is applied to a server, and the method includes:

Send a push message to the display device according to the received video call request sent by the mobile terminal, so that the display device responds to the push message when the mobile terminal is authorized by the display device, automatically establishing a call process; and rejecting the video call in response to the push message when the mobile terminal is not authorized by the display device.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present application, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 exemplarily shows a schematic diagram of an operation scene between a display device and a control device according to an embodiment;

FIG. 2 exemplarily shows a block diagram of the hardware configuration of the control device 100 according to the embodiment;

FIG. 3 exemplarily shows a block diagram of the hardware configuration of the display device 200 according to the embodiment;

FIG. 4 exemplarily shows the hardware architecture block diagram of the display device 200 according to FIG. 3;

FIG. 5 exemplarily shows a schematic diagram of the functional configuration of the display device 200 according to the embodiment;

FIG. 6a exemplarily shows a schematic diagram of the software configuration in the display device 200 according to the embodiment;

FIG. 6b exemplarily shows a schematic diagram of the configuration of the application program in the display device 200 according to the embodiment;

FIG. 7 exemplarily shows a schematic diagram of the user interface in the display device 200 according to the embodiment;

Figure 8a exemplarily shows a schematic diagram of a user interface after the display device automatically establishes a call process under an embodiment;

Fig. 8b exemplarily shows a schematic diagram of the user interface of the mobile terminal after the display device automatically establishes a call process under the embodiment shown in Fig. 8a;

Fig. 8c exemplarily shows a schematic diagram of a user interface in which the display device switches the display of small windows to full-screen display in the embodiment shown in Fig. 8a;

Figure 9a exemplarily shows a schematic diagram of a user interface after the display device automatically establishes a call process under another embodiment;

Fig. 9b exemplarily shows a schematic diagram of the user interface of the mobile terminal after the display device automatically establishes a call process under the embodiment shown in Fig. 9a;

Fig. 10a exemplarily shows a schematic flow diagram under an embodiment;

FIG. 10b exemplarily shows a schematic flowchart before sending an authentication request under another embodiment;

Fig. 10c exemplarily shows a schematic flowchart of sending an authentication request under another implementation manner; Fig. 11 is a schematic structural diagram of a video call system according to some exemplary embodiments;

12 is a method flowchart of a video calling method provided by an embodiment of the present application;

13 is a method flowchart of a video calling method provided by another embodiment of the present application;

14 is a method flowchart of a video calling method provided by an embodiment of the present application;

15 is a method flowchart of a video calling method provided by another embodiment of the present application;

FIG. 16 is a structural block diagram of a video call apparatus provided by an embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the exemplary embodiments of the present application clearer, the technical solutions in the exemplary embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the exemplary embodiments of the present application. Obviously, , the described exemplary embodiments are only a part of the embodiments of the present application, but not all of the embodiments.

Video calling is one of the important means of socializing between people. After the two devices establish a video call process, the video call can be realized. The device that initiates the video call request among the two devices is the initiator device, which receives the video call request. is the invitee's device. The video calling method provided in this application is not limited by hardware, and can be used in devices other than dual-chip systems, or in devices with dual-chip dual-systems. The equipment of the system is expanded.

The concepts involved in the present application are first described below with reference to the accompanying drawings. It should be pointed out here that the following descriptions of various concepts are only for the purpose of making the content of the present application easier to understand, and do not limit the protection scope of the present application.

The term "module" as used in some embodiments of this application may refer to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware or/and software codes capable of executing the execution associated with the element function.

The term "remote control" used in some embodiments of this application refers to a component of an electronic device (such as the display device disclosed in this application), which can generally wirelessly control the electronic device within a short distance range. The component can generally use infrared and/or radio frequency (RF) signals and/or Bluetooth to connect with electronic devices, and may also include functional modules such as WiFi, wireless USB, Bluetooth, and motion sensors. For example, a hand-held touch remote control replaces most of the physical built-in hard keys in a general remote control device with a user interface in a touch screen.

The term "gesture" used in some embodiments of the present application refers to a user behavior that is used by a user to express an expected thought, action, purpose/or result through an action such as a hand shape change or hand movement.

The term "hardware system" used in some embodiments of the present application may refer to a system with computing, A solid component that controls, stores, imports, and exports functionality. In various embodiments of the present application, the hardware system is also generally referred to as a motherboard (motherboard) or a chip.

The term "operating system" used in some embodiments of the present application may refer to a computer system that is presented to a user after a processor reads codes in a memory, such as "Android OS", "Mac OS", "Windows OS" and so on.

FIG. 1 exemplarily shows a schematic diagram of an operation scenario between a display device and a control apparatus according to an embodiment. As shown in FIG. 1 , a user can operate the display apparatus 200 through the control apparatus 100 .

Wherein, the control device 100 may be a remote controller 100A, which can communicate with the display device 200 through infrared protocol communication, Bluetooth protocol communication, ZigBee protocol communication or other short-range communication methods, for wireless or other communication The display device 200 is controlled in a wired manner. The user can control the display device 200 by inputting user instructions through keys on the remote control, voice input, control panel input, and the like. For example, the user can control the display device 200 by inputting corresponding control commands through the volume up/down key, channel control key, up/down/left/right movement keys, voice input key, menu key, power-on/off key, etc. on the remote control. function.

The control device 100 may also be a smart device, such as a mobile terminal 100B, a tablet computer, a computer, a notebook computer, etc., which can be connected through a local area network (LAN, Local Area Network), a wide area network (WAN, Wide Area Network), a wireless local area network ((WLAN) , Wireless Local Area Network) or other networks and the display device 200, and control the display device 200 through an application program corresponding to the display device 200. For example, use an application program running on a smart device to control the display device 200 .The application can provide users with various controls through an intuitive user interface (UI, User Interface) on the screen associated with the smart device.

For example, software applications can be installed on both the mobile terminal 100B and the display device 200, so that the connection and communication between the two can be realized through a network communication protocol, thereby realizing the purpose of one-to-one control operation and data communication. For example, the mobile terminal 100B and the display device 200 can be made to establish a control instruction protocol, the remote control keyboard can be synchronized to the mobile terminal 100B, and the function of controlling the display device 200 can be realized by controlling the user interface on the mobile terminal 100B; The audio and video content displayed on the display device 200 is transmitted to the display device 200 to realize the synchronous display function.

As shown in FIG. 1 , the display device 200 can also perform data communication with the server 300 through various communication methods. In various embodiments of the present application, the display device 200 may be allowed to communicate with the server 300 through a local area network, a wireless local area network or other networks. The server 300 may provide various contents and interactions to the display device 200 .

For example, the display device 200 interacts with an Electronic Program Guide (EPG) by sending and receiving information, receiving software program updates, or accessing a remotely stored digital media library. The server 300 may be one group or multiple groups, and may be one or more types of servers. Other network service contents such as video-on-demand and advertising services are provided through the server 300 .

The display device 200, on the one hand, can be a liquid crystal display, an OLED (Organic Light Emitting Diode) display, and a projection display device; on the other hand, the display device can be a smart TV or a display system composed of a monitor and a set-top box. The specific display device type, size and resolution are not limited. Those skilled in the art can understand that the display device 200 can make some changes in performance and configuration as required.

The display device 200 may additionally provide an intelligent network television function that provides a computer-supported function in addition to the function of broadcasting and receiving television. Examples include Internet TV, Smart TV, Internet Protocol TV (IPTV), and the like. In some embodiments, the display device may not have broadcast reception television capabilities.

As shown in FIG. 1 , a camera may be connected or set on the display device for presenting the picture captured by the camera on the display interface of the display device or other display devices, so as to realize interactive chat between users. Specifically, the picture captured by the camera may be displayed on the display device in full screen, half screen, or in any optional area.

As an optional connection method, the camera is connected to the rear shell of the display through a connecting plate, and is fixedly installed in the upper middle of the rear shell of the display. It is sufficient that the image acquisition area is not blocked by the back cover. For example, the image acquisition area and the display device have the same display orientation.

As another optional connection method, the camera can be connected to the display back shell through a connecting plate or other conceivable connectors in a liftable manner. A lifting motor is installed on the connector. When the user wants to use the camera or has an application program to use the camera When the camera is not needed, it can be embedded behind the rear case to protect the camera from damage.

As an embodiment, the camera used in this application may be 16 million pixels, so as to achieve the purpose of ultra-high-definition display. In actual use, cameras with higher or lower pixels than 16 megapixels can also be used.

When a camera is installed on the display device, the content displayed in different application scenarios of the display device can be integrated in many different ways, so as to achieve functions that cannot be achieved by traditional display devices.

In some embodiments, a user may video chat with at least one other user while watching a video program. The presentation of the video program can be used as a background image, and the video chat window is displayed on the background image. Visually, this function can be called "watch while chatting".

In some embodiments, in a "watch while chat" scenario, while watching a live video or an online video, at least one video chat is performed across terminals.

In some embodiments, a user may video chat with at least one other user while entering the educational application to learn. For example, students can interact with teachers remotely while learning content in educational applications. Visually, this function can be called "learning while chatting".

In some embodiments, users video chat with players entering the game while playing a card game. For example, when a player enters a game application to participate in the game, he can realize remote interaction with other players. Visually, this function can be called "play while watching".

In some embodiments, the game scene is merged with the video picture, and the portraits in the video picture are cut out and displayed in the game picture, so as to improve the user experience.

In some embodiments, in somatosensory games (such as playing ball, boxing, running, dancing, etc.), the camera is used to obtain human postures and movements, limb detection and tracking, detection of human skeleton key point data, and then combined with The animation in the game is integrated to realize the game of scenes such as sports and dance.

In some embodiments, the user can perform video and voice interaction with at least one other user in the karaoke application. Visually, this function can be called "sing while watching". Preferably, when at least one user enters the application in a chatting scenario, multiple users can jointly complete the recording of a song.

In some embodiments, the user can turn on the camera locally to obtain pictures and videos, which can be called "looking in the mirror".

In some embodiments, more functions may be added or the above functions may be reduced. The present application does not specifically limit the function of the display device.

FIG. 2 exemplarily shows a block diagram of the configuration of the control apparatus 100 according to the exemplary embodiment. As shown in FIG. 3 , the control device 100 includes a controller 110 , a communicator 130 , a user input/output interface 140 , a memory 190 , and a power supply 180 .

The control apparatus 100 is configured to be able to control the display device 200, and to receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, so as to serve as an interactive intermediary between the user and the display device 200. effect. For example, the user operates the channel addition and subtraction keys on the control device 100, and the display device 200 responds to the channel addition and subtraction operation.

In some embodiments, the control apparatus 100 may be a smart device. For example, the control apparatus 100 may install various applications for controlling the display device 200 according to user requirements.

In some embodiments, as shown in FIG. 1 , the mobile terminal 100B or other intelligent electronic device may perform a similar function of the control apparatus 100 after installing the application for operating the display device 200 . For example, the user can install the application, various function keys or virtual buttons of the graphical user interface that can be provided on the mobile terminal 100B or other intelligent electronic devices, so as to realize the function of the physical key of the control apparatus 100 .

The controller 110 includes a processor 112, RAM 113 and ROM 114, a communication interface, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation among the internal components and the external and internal data processing functions.

The communicator 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110 . For example, the received user input signal is sent to the display device 200 . The communicator 130 may include at least one of communication modules such as a WIFI module 131 , a Bluetooth module 132 , and an NFC module 133 .

The user input/output interface 140, wherein the input interface includes at least one of input interfaces such as a microphone 141, a touch panel 142, a sensor 143, and a key 144. For example, the user can implement the user command input function through actions such as voice, touch, gesture, pressing, etc. The input interface converts the received analog signal into a digital signal, and converts the digital signal into a corresponding command signal, and sends it to the display device 200.

The output interface includes an interface for transmitting received user instructions to the display device 200 . In some embodiments, it may be an infrared interface or a radio frequency interface. For example, when the infrared signal interface is used, the user input instruction needs to be converted into an infrared control signal according to the infrared control protocol, and sent to the display device 200 through the infrared sending module. For another example, when a radio frequency signal interface is used, the user input command needs to be converted into a digital signal, and then modulated according to the radio frequency control signal modulation protocol, and then sent to the display device 200 by the radio frequency transmission terminal.

In some embodiments, the control device 100 includes at least one of a communicator 130 and an output interface. The control device 100 is configured with a communicator 130, such as: WIFI, Bluetooth, NFC and other modules, which can send user input instructions to the display device 200 through WIFI protocol, or Bluetooth protocol, or NFC protocol encoding.

The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller 110 . The memory 190 can store various control signal instructions input by the user.

The power supply 180 is used to provide operating power support for each element of the control device 100 under the control of the controller 110 . Can battery and related control circuit.

FIG. 3 exemplarily shows a hardware configuration block diagram of a hardware system in the display device 200 according to an exemplary embodiment.

When the dual hardware system architecture is adopted, the institutional relationship of the hardware system can be shown in FIG. 3 . For ease of expression, one hardware system in the dual hardware system architecture is referred to as the first hardware system or A system, A chip, and the other hardware system is referred to as the second hardware system or N system, N chip. The A chip includes the controller of the A chip and various interfaces, and the N chip includes the controller and various interfaces of the N chip. A relatively independent operating system can be installed in the A chip and the N chip. The operating system of the A chip and the operating system of the N chip can communicate with each other through a communication protocol. Exemplary: the framework layer of the operating system of the A chip and the The framework layer of the operating system can communicate to transmit commands and data, so that there are two independent but interrelated subsystems in the display device 200 .

As shown in FIG. 3 , the connection, communication and power supply can be implemented between the A chip and the N chip through a plurality of different types of interfaces. The interface type of the interface between the A chip and the N chip may include at least one of a general-purpose input/output interface (General-purpose input/output, GPIO), a USB interface, an HDMI interface, a UART interface, and the like. One or more of these interfaces may be used for communication or power transfer between the A chip and the N chip. For example, as shown in Figure 3, under the dual hardware system architecture, the N chip can be powered by an external power supply, while the A chip can be powered by the N chip instead of the external power supply.

In addition to the interface for connecting with the N chip, the A chip may also include an interface for connecting other devices or components, such as the MIPI interface shown in Figure 3 for connecting a camera (Camera), a Bluetooth interface, and so on.

Similarly, in addition to the interface for connecting with the N chip, the N chip may also include a VBY interface for connecting to the display TCON (Timer Control Register), for connecting to a power amplifier (Amplifier, AMP) and a speaker (Speaker). ) i2S interface; and IR/Key interface, USB interface, Wifi interface, Bluetooth interface, HDMI interface, Tuner interface, etc.

The dual hardware system architecture of the present application will be further described below with reference to FIG. 4 . It should be noted that FIG. 4 is only an exemplary illustration of the dual-hardware system architecture of the present application, and does not represent a limitation on the present application. In practical applications, both hardware systems may contain more or less hardware or interfaces as required.

FIG. 4 exemplarily shows a hardware architecture block diagram of the display device 200 according to FIG. 3 . As shown in FIG. 4 , the hardware system of the display device 200 may include an A chip and an N chip, and modules connected to the A chip or the N chip through various interfaces.

The N chip may include a tuner-demodulator 220, a communicator 230, an external device interface 250, a controller 210, a memory 290, a user input interface, a video processor 260-1, an audio processor 260-2, a display 280, an audio output interface 272. At least one of the power supplies. In other embodiments, the N chip may also include more or fewer modules.

Among them, the tuner and demodulator 220 is used to perform modulation and demodulation processing such as amplification, frequency mixing and resonance on the broadcast and television signals received by wire or wireless, so as to demodulate the user's data from a plurality of wireless or wire broadcast and TV signals. The audio and video signals carried in the frequencies of the selected TV channels, as well as additional information (eg EPG data signals). According to different TV signal broadcasting standards, the signal paths of the tuner and demodulator 220 can be various, such as: terrestrial broadcasting, cable broadcasting, satellite broadcasting or Internet broadcasting, etc.; and according to different modulation types, the adjustment method of the signal can be digitally modulated The tuner demodulator 220 can demodulate the analog signal and/or the digital signal according to the type of the received TV signal.

The tuner and demodulator 220 is also used for, according to user selection, and controlled by the controller 210, to respond to the television channel frequency selected by the user and the television signal carried by the frequency.

In some other exemplary embodiments, the tuner 220 may also be in an external device, such as an external set-top box or the like. In this way, the set-top box outputs the television audio and video signals after modulation and demodulation, and inputs them into the display device 200 through the external device interface 250 .

The communicator 230 is a component for communicating with external devices or external servers according to various communication protocol types. For example, the communicator 230 may include a WIFI module 231, a Bluetooth communication protocol module 232, a wired Ethernet communication protocol module 233, and other network communication protocol modules or near field communication protocol modules such as an infrared communication protocol module.

The display device 200 may establish a connection of control signals and data signals with an external control device or a content providing device through the communicator 230 . For example, the communicator may receive a control signal of the remote controller 100 according to the control of the controller.

The external device interface 250 is a component that provides data transmission between the N-chip controller 210 and the A-chip and other external devices. The external device interface can be connected to external devices such as set-top boxes, game devices, notebook computers, etc. in a wired/wireless manner, and can receive external devices such as video signals (such as moving images), audio signals (such as music), additional information (such as EPG) ) and other data.

The external device interface 250 may include: a high-definition multimedia interface (HDMI) terminal 251 , a composite video blanking and sync (CVBS) terminal 252 , an analog or digital component terminal 253 , a universal serial bus (USB) terminal 254 , a red, green, and blue (RGB) terminal RGB) terminal (not shown in the figure), etc. any one or more. This application does not limit the number and type of external device interfaces.

The controller 210 controls the operation of the display device 200 and responds to user operations by running various software control programs (eg, an operating system and/or various application programs) stored in the memory 290 .

As shown in FIG. 4, the controller 210 includes a read only memory RAM 213, a random access memory ROM 214, a graphics processor 216, a CPU processor 212, a communication interface 218, and a communication bus. Among them, the RAM 213 and the ROM 214, the graphics processor 216, the CPU processor 212, and the communication interface 218 are connected through a bus.

The ROM 213 is used to store various system startup instructions. For example, when a power-on signal is received, the power supply of the display device 200 starts, and the CPU processor 212 executes the system startup instruction in the ROM, and copies the temporary data generated by the operating system stored in the memory 290 to the RAM 214 to run or start the operating system. After the startup of the operating system is completed, the CPU processor 212 copies the temporary data generated by various application programs in the memory 290 to the RAM 214, and then runs or starts various application programs.

The graphics processor 216 is used to generate various graphic objects, such as: icons, operation menus, and user input instructions to display graphics and so on. It includes an operator, which performs operations by receiving various interactive instructions input by the user, and displays various objects according to the display properties. and includes a renderer, generating various objects obtained based on the arithmetic unit, and displaying the result of rendering on the display 280 .

CPU processor 212 for executing operating system and application program instructions stored in memory 290 . And various applications, data and content are executed according to various interactive instructions received from the external input, so as to finally display and play various audio and video content.

In some exemplary embodiments, CPU processor 212 may include multiple processors. The plurality of processors may include a main processor and multiple or one sub-processors. The main processor is used to perform some operations of the display device 200 in the pre-power-on mode, and/or an operation of displaying a picture in the normal mode. Multiple or one sub-processors for performing an operation in a state such as standby mode.

The communication interface may include a first interface 218-1 to an nth interface 218-n. These interfaces may be network interfaces connected to external devices via a network.

The controller 210 may control the overall operation of the display apparatus 200 . For example: In response to receiving a user command for selecting a UI object to be displayed on the display 280, the controller 210 may perform an operation related to the object selected by the user command.

Wherein, the object can be any one of the optional objects, such as a hyperlink or an icon. Actions related to the selected object, such as displaying actions linked to hyperlinked pages, documents, images, etc., or executing actions that correspond to an icon. The user command for selecting the UI object may be an input command through various input devices (eg, a mouse, a keyboard, a touchpad, etc.) connected to the display device 200 or a voice command corresponding to a voice spoken by the user.

The memory 290 includes storing various software modules for driving and controlling the display device 200 . For example, various software modules stored in the memory 290 include: a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules.

The basic module is a low-level software module used for signal communication between various hardwares in the display device 200 and sending processing and control signals to the upper-layer modules. The detection module is a management module used to collect various information from various sensors or user input interfaces, perform digital-to-analog conversion, and analyze and manage.

For example, the speech recognition module includes a speech analysis module and a speech instruction database module. The display control module is a module for controlling the display 280 to display image content, and can be used to play information such as multimedia image content and UI interface. The communication module is a module used for control and data communication with external devices. The browser module is a module for performing data communication between browser servers. Service modules are modules used to provide various services and various applications.

At the same time, the memory 290 is also used for storing received external data and user data, images of various items in various user interfaces, and visual effect diagrams of focal objects.

The user input interface is used to send the user's input signal to the controller 210, or transmit the signal output from the controller to the user. Exemplarily, the control device (such as a mobile terminal or a remote control) can send input signals input by the user, such as power switch signal, channel selection signal, volume adjustment signal, etc., to the user input interface, and then the user input interface forwards it to the controller; Alternatively, the control device may receive an output signal such as audio, video or data output from the user input interface through the controller processing, and display the received output signal or output the received output signal in the form of audio or vibration.

In some embodiments, the user may input user commands on a graphical user interface (GUI) displayed on the display 280, and the user input interface receives the user input commands through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

The video processor 260-1 is used to receive video signals, and perform video data processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to the standard codec protocol of the input signal, and can obtain A video signal that is displayed or played directly on the display 280 .

Exemplarily, the video processor 260-1 includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like.

Wherein, the demultiplexing module is used for demultiplexing the input audio and video data stream. For example, if MPEG-2 is input, the demultiplexing module demultiplexes it into video signals and audio signals.

The video decoding module is used to process the demultiplexed video signal, including decoding and scaling.

The image synthesizing module, such as an image synthesizer, is used for superimposing and mixing the GUI signal generated by the graphics generator according to the user's input or itself, and the zoomed video image, so as to generate an image signal that can be displayed.

The frame rate conversion module is used to convert the frame rate of the input video, such as converting the frame rate of the input 24Hz, 25Hz, 30Hz, 60Hz video to the frame rate of 60Hz, 120Hz or 240Hz. The video stream is related, and the output frame rate can be related to the update rate of the display. The input has the usual format, such as frame insertion.

The display formatting module is used to change the signal output by the frame rate conversion module into a signal conforming to a display format such as a display, such as converting the format of the signal output by the frame rate conversion module to output an RGB data signal.

In some embodiments, the video processor and the graphics processor may be integrated into the chip, and in some embodiments, the functional modules of the graphics processor and the video processor may be set as required.

The display 280 is configured to receive the image signal input from the video processor 260-1, and to display the video content and images and the menu manipulation interface. The display 280 includes a display component for presenting a picture and a driving component for driving image display. The displayed video content can come from the video in the broadcast signal received by the tuner and demodulator 220, or from the video content input from the communicator or the interface of the external device. The display 220 simultaneously displays a user manipulation interface UI generated in the display device 200 and used to control the display device 200 .

And, depending on the type of the display 280, a driving component for driving the display is also included. Alternatively, if the display 280 is a projection display, it may also include a projection device and a projection screen.

The audio processor 260-2 is used to receive the audio signal, perform decompression and decoding according to the standard codec protocol of the input signal, and perform audio data processing such as noise reduction, digital-to-analog conversion, and amplification processing, and obtain a signal that can be displayed in the speaker 272. Played audio signal.

The audio output interface 270 is used to receive the audio signal output by the audio processor 260-2 under the control of the controller 210. The audio output interface may include a speaker 272, or an external audio output terminal 274 of a generating device output to an external device, such as : External audio terminal or headphone output terminal, etc.

In some other exemplary embodiments, the video processor 260-1 may comprise one or more chips. The audio processor 260-2 may also include one or more chips.

And, in some other exemplary embodiments, the video processor 260-1 and the audio processor 260-2 may be separate chips, or may be integrated with the controller 210 in one or more chips.

The power supply is used to provide power supply support for the display device 200 with the power input from the external power supply under the control of the controller 210 . The power supply may include a built-in power supply circuit installed inside the display device 200 , or may be a power supply installed outside the display device 200 , such as a power interface that provides an external power supply in the display device 200 .

Similar to the N chip, as shown in FIG. 4 , the A chip may include a controller 310 , a communicator 330 , a detector 340 , and a memory 390 . A user input interface, video processor, audio processor, display, audio output interface may also be included in some embodiments. In some embodiments, there may also be a power supply that independently supplies power to the A chip.

The communicator 330 is a component for communicating with external devices or external servers according to various communication protocol types. For example, the communicator 330 may include a WIFI module 331, a Bluetooth communication protocol module 332, a wired Ethernet communication protocol module 333, and an infrared communication protocol module and other network communication protocol modules or near field communication protocol modules.

The communicator 330 of the A chip and the communicator 230 of the N chip also interact with each other. For example, the WiFi module 231 in the N-chip hardware system is used to connect to an external network and generate network communication with an external server and the like. The WiFi module 331 in the A chip is used to connect to the WiFi module 231 in the N chip hardware system, without directly connecting with the external network, etc. The A chip is connected to the external network through the N chip. Therefore, for the user, a display device as in the above embodiment displays a WiFi account to the outside.

The detector 340 is a component of the display device A chip used to collect signals from the external environment or interact with the outside. The detector 340 may include a light receiver 342, a sensor used to collect ambient light intensity, and can adapt to changes in display parameters by collecting ambient light; it may also include an image collector 341, such as a camera, a camera, etc., which can be used to collect external The environment scene, as well as the gestures used to collect user attributes or interact with the user, can adaptively change display parameters, and can also recognize user gestures to realize the function of interacting with the user.

The external device interface 350 provides components for data transmission between the controller 310 and the N chip or other external devices. The external device interface can be connected with external devices such as set-top boxes, game devices, notebook computers, etc. in a wired/wireless manner.

The controller 310 controls the operation of the display device 200 and responds to user operations by running various software control programs stored in the memory 390 (eg, using installed third-party applications, etc.) and interacting with the N chip.

As shown in FIG. 4, the controller 310 includes a read-only memory ROM 313, a random access memory RAM 314, a graphics processor 316, a CPU processor 312, a communication interface 318, and a communication bus. Among them, the ROM 313 and the RAM 314, the graphics processor 316, the CPU processor 312, and the communication interface 318 are connected through a bus.

ROM313 is used to store various system startup instructions. The CPU processor 312 executes the system startup instruction in the ROM, and copies the temporary data of the operating system stored in the memory 390 to the RAM 314 to run or start the operating system. After the startup of the operating system is completed, the CPU processor 312 copies the temporary data of various application programs in the memory 390 to the RAM 314, and then runs or starts various application programs.

The CPU processor 312 is used to execute the operating system and application program instructions stored in the memory 390, communicate with the N chip, transmit and interact with the N chip, such as signals, data, and instructions, and execute various interactive instructions received from external input. Various applications, data and content for the ultimate display and playback of various audio and video content.

The communication interface may include a first interface 318-1 to an nth interface 318-n. These interfaces may be network interfaces connected to external devices via a network, or may be network interfaces connected to the N chip via a network.

The controller 310 may control the overall operation of the display apparatus 200 . For example: In response to receiving a user command for selecting a UI object to be displayed on the display 280, the controller 210 may perform an operation related to the object selected by the user command.

The graphics processor 316 is used to generate various graphic objects, such as: icons, operation menus, and user input instructions to display graphics and so on. It includes an operator, which performs operations by receiving various interactive instructions input by the user, and displays various objects according to the display properties. and includes a renderer, generating various objects obtained based on the arithmetic unit, and displaying the result of rendering on the display 280 .

Both the A-chip graphics processor 316 and the N-chip graphics processor 216 can generate various graphics objects. Differently, if application 1 is installed on the A chip and application 2 is installed on the N chip, when the user is on the interface of application 1 and executes an instruction input by the user in application 1, the graphics object of the A chip is generated by the graphics processor 316 of the A chip. When the user is on the interface of the application 2 and performs an instruction input by the user in the application 2, the graphics object is generated by the graphics processor 216 of the N chip.

FIG. 5 exemplarily shows a functional configuration diagram of a display device according to an exemplary embodiment.

As shown in FIG. 5 , the memory 390 of the A chip and the memory 290 of the N chip are respectively used to store the operating system, application program, content and user data, etc., under the control of the controller 310 of the A chip and the controller 210 of the N chip Various operations that drive the system operation of the display apparatus 200 and respond to the user are performed. The A-chip memory 390 and the N-chip memory 290 may include volatile and/or nonvolatile memory.

For the N chip, the memory 290 is specifically used to store the running program for driving the controller 210 in the display device 200, and to store various application programs built in the display device 200, various application programs downloaded by the user from external devices, and related application programs. Related to various graphical user interfaces, as well as various objects related to graphical user interfaces, user data information, and various internal data supporting applications. The memory 290 is used to store system software such as an operating system (OS) kernel, middleware, and applications, as well as input video data and audio data, and other user data.

The memory 290 is specifically used to store drivers and related data such as the video processor 260-1, the audio processor 260-2, the display 280, the communication interface 230, the tuner and demodulator 220, and the input/output interface.

In some embodiments, memory 290 may store software and/or programs representing an operating system (OS) including, for example, a kernel, middleware, application programming interfaces (APIs) and/or applications. Illustratively, the kernel may control or manage system resources, or functions implemented by other programs (such as the middleware, APIs, or applications), and the kernel may provide interfaces to allow middleware and APIs, or applications, to access the controller , to control or manage system resources.

Exemplarily, the memory 290 includes a broadcast receiving module 2901, a channel control module 2902, a volume control module 2903, an image control module 2904, a display control module 2905, an audio control module 2906, an external command recognition module 2907, a communication control module 2908, a light receiving module module 2909, power control module 2910, operating system 2911, and other applications 2912, browser modules, and the like. The controller 210 executes various software programs in the memory 290, such as: broadcast TV signal reception and demodulation function, TV channel selection control function, volume selection control function, image control function, display control function, audio control function, external command Various functions such as identification function, communication control function, optical signal receiving function, power control function, software control platform supporting various functions, and browser function.

The memory 390 includes storing various software modules for driving and controlling the display device 200 . For example, various software modules stored in the memory 390 include: a basic module, a detection module, a communication module, a display control module, a browser module, and various service modules. Since the functions of the memory 390 and the memory 290 are relatively similar, reference may be made to the memory 290 for related parts, which will not be repeated here.

For example, the memory 390 includes an image control module 3904, an audio control module 2906, an external command recognition module 3907, a communication control module 3908, a light receiving module 3909, an operating system 3911, other application programs 3912, a browser module, and the like. The controller 210 executes various software programs in the memory 290, such as: image control function, display control function, audio control function, external command recognition function, communication control function, optical signal receiving function, power control function, supporting various Function software control platform, browser function and other functions.

Differently, the external instruction identification module 2907 of the N chip and the external instruction identification module 3907 of the A chip can identify different instructions.

Exemplarily, since an image receiving device such as a camera is connected to the A chip, the external command recognition module 3907 of the A chip may include a graphic recognition module 2907-1, and the graphic recognition module 3907-1 stores a graphic database. When the graphics command is generated, it corresponds to the command in the graphics database, so as to make command control to the display device. Since the voice receiving device and the remote control are connected to the N chip, the external command recognition module 2907 of the N chip may include a voice recognition module 2907-2. The graphic recognition module 2907-2 stores a voice database, and the voice receiving device receives the The external voice command or time corresponds to the command in the voice database, so as to perform command control on the display device. Similarly, the control device 100 such as a remote control is connected to the N chip, and the key command recognition module performs command interaction with the control device 100 .

FIG. 6a exemplarily shows a configuration block diagram of a software system in the display device 200 according to an exemplary embodiment.

For the N chip, as shown in Figure 6a, the operating system 2911 includes execution software for processing various basic system services and for implementing hardware-related tasks.

In some embodiments, a portion of the operating system kernel may contain a set of software to manage hardware resources of the display device and provide services for other programs or software codes.

In some other embodiments, a portion of the operating system kernel may contain one or more device drivers, and the device drivers may be a set of software codes in the operating system to help operate or control devices or hardware associated with the display device. Drivers may contain code to operate video, audio and/or other multimedia components. Examples include displays, cameras, Flash, WiFi, and audio drivers.

Among them, the accessibility module 2911-1 is used to modify or access the application program, so as to realize the accessibility of the application program and the operability of the displayed content thereof.

The communication module 2911-2 is used to connect with other peripherals via related communication interfaces and communication networks.

The user interface module 2911-3 is used to provide objects for displaying a user interface for each application program to access, which can realize user operability.

The control application 2911-4 is used to control process management, including runtime applications and the like.

The event transmission system 2914 can be implemented in the operating system 2911 or in the application program 2912 . In some embodiments, on the one hand, it is implemented in the operating system 2911, and at the same time, it is implemented in the application program 2912 to monitor various user input events, and will refer to the recognition results in response to various events or sub-events according to various events, Instead, a handler that implements one or more sets of predefined actions.

Among them, the event monitoring module 2914-1 is used to monitor user input interface input events or sub-events.

The event identification module 2914-1 is used to input the definitions of various events to various user input interfaces, identify various events or sub-events, and transmit them to the processing program for executing its corresponding one or more groups .

The event or sub-event refers to the input detected by one or more sensors in the display device 200 and the input of the external control device (such as the control device 100 , etc.). Such as: various sub-events of voice input, gesture input sub-events of gesture recognition, and sub-events of remote key command input of the control device, etc. Exemplarily, one or more sub-events in the remote control include various forms, including but not limited to one or a combination of key press up/down/left/right/, confirm key, key press and so on. And non-physical button operations, such as move, hold, release and other operations.

The interface layout management module 2913 directly or indirectly receives and monitors each user input event or sub-event from the event transmission system 2914, and is used to update the layout of the user interface, including but not limited to the position of each control or sub-control in the interface, and the container The size or position, level, etc. of the interface perform various operations related to the interface layout.

Since the functions of the operating system 3911 of the A chip and the operating system 2911 of the N chip are relatively similar, please refer to the operating system 2911 for related parts, which will not be repeated here.

As shown in FIG. 6b , the application layer of the display device contains various applications executable on the display device 200 .

The application layer 2912 of the N-chip may include, but is not limited to, one or more applications, such as video-on-demand applications, application centers, game applications, and the like. The application layer 3912 of the A-chip may include, but is not limited to, one or more applications, such as a live TV application, a media center application, and the like. It should be noted that the application programs contained on the A chip and the N chip respectively are determined according to the operating system and other designs, and the present invention does not need to specifically limit and divide the application programs contained on the A chip and the N chip.

Live TV app that can provide live TV from different sources. For example, a live TV application may provide a TV signal using input from cable, over-the-air, satellite services, or other types of live TV services. And, the live TV application may display the video of the live TV signal on the display device 200 .

Video-on-demand application that can provide video from different storage sources. Unlike live TV applications, video-on-demand provides a display of video from certain storage sources. For example, video-on-demand can come from the server side of cloud storage, from local hard disk storage containing existing video programs.

A media center application that can provide a variety of multimedia content playback applications. For example, a media center may provide services other than live TV or video-on-demand, where users can access various images or audio through a media center application.

The application center can provide storage of various applications. An application can be a game, an application, or some other application that is associated with a computer system or other device but can run on a display device. The application center can obtain these applications from various sources, store them in local storage, and then run them on the display device 200 .

In some embodiments, the operating system includes a kernel, a database, a framework layer and an application layer, all of which are applications in the application layer.

FIG. 7 exemplarily shows a schematic diagram of a user interface in the display device 200 according to an exemplary embodiment. As shown in FIG. 7 , the user interface includes a plurality of view display areas, for example, a first view display area 201 and a play screen 202, wherein the play screen includes layout of one or more different items. And the user interface also includes a selector indicating that an item is selected, and the position of the selector can be moved by user input to change the selection of a different item.

It should be noted that, multiple view display areas can present display images of different levels. For example, the first view display area may present the content of the video chatting item, and the second view display area may present the content of the application layer item (eg, webpage video, VOD display, application screen, etc.).

Optionally, the presentation of different view display areas has different priorities, and between view display areas with different priorities, the display priorities of the view display areas are different. For example, the priority of the system layer is higher than that of the application layer. When the user uses the acquisition selector and screen switching in the application layer, the screen display in the view display area of the system layer is not blocked; When the size and position of the view display area of the system layer are changed, the size and position of the view display area of the system layer are not affected.

Displays of the same level can also be presented. At this time, the selector can switch between the first view display area and the second view display area, and when the size and position of the first view display area changes, the second view The size and position of the display area can be changed accordingly.

Since the A chip and the N chip may be respectively installed with independent operating systems, there are two independent but interrelated subsystems in the display device 200 . For example, both the A chip and the N chip can be independently installed with Android (Android) and various APPs, so that each chip can realize a certain function, and the A chip and the N chip can cooperate to realize a certain function.

In the present application, a video call method is provided, so that an authorized account can directly establish a video connection with a display device without requiring the display device (as an invited device) to operate, which simplifies the process of establishing a video call. The solution provided by the present application can not only be used in a dual-chip display device or a multi-chip display device, but also can be used in a single-chip display device.

In some embodiments, a camera is set on the display device of the invitee of the video call, and the initiating device of the video call may be a mobile terminal or another display device. In some embodiments the display device is a smart TV or the display device is a system of a display and a set-top box.

In some embodiments, taking the initiating end of the video request as a mobile terminal as an example, a corresponding APP is installed on the mobile terminal, for example, at least one of Juhaohao, Xiaojujia, and Hijian. The video call request sent by the mobile terminal may be that the user opens the corresponding APP and clicks on the control corresponding to the video function in the APP, and the mobile terminal sends a video call request to the server. The video call request at least includes the unique identifier of the display device or the account signal of the TV terminal The one of the mobile terminal also includes account information on the mobile terminal, and the video call request is used to request the display device to transmit the image information obtained by the camera on the display device and the sound information obtained by the microphone to the mobile terminal. The unique identification of the display device or the account signal of the TV terminal may be used as identification information of the display device, that is, the first identification information. The account information on the mobile terminal may be used as identification information representing the mobile terminal, that is, the second identification information.

After receiving the video call request, the server sends a push message to the TV according to the first identification information, where the push message includes the first identification information and/or the second identification information, and the push message is used to trigger the display device to determine the mobile terminal legitimacy.

In some embodiments, after receiving the video call request, the server sends a push message to the TV according to the account of the TV terminal and/or the unique identifier of the display device, and the push message includes at least the unique identifier of the display device and the account of the TV terminal. , one of account information on the mobile terminal.

In some embodiments, the server may perform authentication before sending a push message to the display device, so that the terminal can directly establish a call process after making a judgment according to the push message. Specifically, the authentication process includes identifying whether the unique identifier of the display device in the video call request or the account signal of the TV terminal authorizes the account logged in on the mobile terminal to directly establish the call request, and then sending a push message and a successful authentication message .

In response to the push message, the display device automatically establishes the video call process.

In some embodiments, the display device automatically establishing a video call includes: the display device sends an authentication request to the server according to the unique identifier of the display device and/or the account currently logged in; the server responds to the authentication request. according to the preset correspondence relationship to determine whether the mobile terminal is authorized by the television device; the display device automatically establishes the video call process in response to the authentication success message.

In some embodiments, when the display device determines that the account of the TV terminal in the push request and/or the unique identifier of the display device are consistent with the current login account of the display device or the unique identifier of the current display device, the display device responds In the message of successful authentication, the video call process is automatically established.

In some embodiments, the server sends the push message only when the authentication succeeds, and does not send the push message when the authentication fails. Therefore, the display device can directly determine the account of the TV terminal and/or the unique information of the display device in the push request according to the push message. When the identifier is consistent with the current login account of the display device or the unique identifier of the current display device, the video call process is automatically established.

In some embodiments provided in this application, the first video application installed in the display device (for example: Xiaoju at home) may also include a second video application (for example, see you), and may also include other applications that use cameras ( For example: a game application whose main function is chess and card games, or a camera application whose main function is taking pictures, or a mirror application.)

In the process of using the function, the mobile terminal must first obtain the authorization of the display device. The authorization process may be that the two-dimensional code information that enables the mobile terminal to obtain the ID of the display device is displayed on the display device. After scanning the code, the mobile terminal obtains the ID of the display device and displays it in the UI. The interface presents the ID information of the display device. After the user selects the binding on the mobile terminal, the server binds the current account information on the mobile terminal with the ID of the display device. In some embodiments, the ID of the display device can be replaced by the account information logged on the display device, the server combines the current account information on the mobile phone with the account information logged in on the display device, and in some embodiments, the two-dimensional code may contain both Display the account information logged on the device and display the ID of the device.

In some embodiments, in the process of authorization of the display device, the account information logged in the display device may be required to be the same as the account information logged in on the mobile terminal, and then the ID of the display device is bound.

In some embodiments, the authentication correspondence established after binding may be stored on the server or in the display device, and the authentication correspondence may include: the correspondence between the current account information on the mobile phone and the ID of the display device , the current account information on the mobile phone and the corresponding relationship with the account information logged in on the display device, and at least one of the account information logged in on the display device and the corresponding relationship between the ID of the device and the current account information on the mobile phone.

As shown in Figure 10a, the process of a video call includes:

S001: The mobile terminal sends a video call request;

In some embodiments, the video call request includes first identification information and second identification information, and the video call request is used to request the display device to transmit to the mobile terminal the information obtained by the camera on the display device. image information, wherein the first identification information is identification information representing the display terminal invited in the video call, and the second identification information is identification information representing the mobile terminal that initiates the video call.

In some embodiments, during the authorization process, a mapping relationship between the first identification information and the second identification information is established, and the server pre-records the mapping relationship.

In some embodiments, the first identification information may be at least one of the unique identification (device ID1) of the display device or the first account information (customer ID1) stored on the mobile terminal; the second identification information may be the mobile terminal The account information (customer ID 2) currently logged in on the mobile terminal or the unique device identifier (device ID2) of the mobile terminal.

The first identification information may be the ID of the display device and/or the account information of the display device stored in the video call authorization process on the mobile terminal, or may be the ID of the display device and/or the account information of the display device manually input by the user.

Wherein, the video call request further includes the first video application identifier (for example, the identifier of the Xiaoju at home function), so that the display device determines which function needs to be called to establish the video call process.

In some embodiments, the display device is a TV, and the mobile terminal is a mobile phone or a tablet computer.

In some embodiments, the display device sends a video call request to the server, and the server informs the display device that there is a video call request. The display device and the mobile terminal are in different networks, that is, not in the same local area network. For example, the TV is in the local area network of the home, and the mobile phone is in the data network or another local area network.

S002: After receiving the video call request, the server sends a push message to the TV according to the first identification information;

In some embodiments, the push message includes first identification information and/or second identification information, and the push message is used to trigger the display device to judge the validity of the mobile terminal.

In some embodiments, judging the legitimacy of the mobile terminal refers to judging whether the display device authorizes the mobile terminal to directly connect to the video call.

The push message is used to inform the display device that there is a video call access request and trigger the display device to judge the validity of the mobile terminal.

In some embodiments, the push message includes the second identification information, so that the display device triggers the validity judgment.

In some embodiments, the push message further includes first identification information, so that the display device judges whether its own identification is consistent with the first identification information, and triggers the validity judgment;

In some embodiments, the push message includes the first identification information and the second identification information, so that the display device can determine whether its own identification is consistent with the first identification information and trigger the validity judgment at the same time.

In some embodiments, it may be possible to send a push message to the TV according to the first identification information (that is, the account of the TV end and/or the unique identification of the display device), where the push message at least includes the unique identification (deviceID1) of the display device, and the unique identification of the TV end. One of the account number (customer ID 1) and the account information (customer ID 2) on the mobile terminal.

In some embodiments, the push message may further include an identifier of the first application function, and the identifier of the first application function is used to enable the display device to determine that the call process of that function should be established.

In some embodiments, the first identification information includes first account information (that is, the account information of the TV terminal), and there is a long connection between the running display device and the system terminal. The long connection is established according to the account logged in on the display device, and the server According to the account information (customer ID 1) of the TV terminal uploaded by the mobile phone, it is judged whether there is a long connection corresponding to the TV account information (customer ID 1). The server pushes a message to the device corresponding to the TV account according to the persistent connection, where the push message includes at least one of the above parameters (the first identification information and the second identification information).

Correspondingly, if the display device is running and there is a long connection between the display device (such as display device A) and the server, the server determines the corresponding display device (such as display device A) according to the account information (customer ID 1) of the TV terminal uploaded by the mobile phone. Send the generated push message to the display device (such as display device A), if the display device (such as display device A) is turned off or in standby state, there is no long connection between the display device and the server, and the server cannot query and upload the mobile phone. The account information (customer ID 1) of the TV end corresponds to a long connection, so the mobile terminal should feed back the information that the display device is offline to indicate that the display device corresponding to the account information (customer ID 1) of the TV end is not online.

In some embodiments, the server determines whether the display device is online according to the polling information of the display device. If it is online, it will continue. If it is not online, it will feedback to the mobile terminal that the device is not online.

In some embodiments, the display device automatically establishes the call process in response to the push message.

In some embodiments, the display device automatically sends a confirmation message to the server in response to the push message; the server assigns a room address for the video call, and the display device receives the call room address sent by the server in response to the confirmation message; and according to The call room address initiates the video call.

In some embodiments, after receiving the video call request sent by the mobile terminal, the server determines whether the mobile terminal represented by the call request is authorized by the requested display device according to a pre-stored authorization relationship, and the server confirms the authorization after confirming the authorization. , sending a push message and a successful authentication message to the display device, the display device automatically establishes a call process, and the server confirms that there is no authorization, and feeds back a rejection message to the mobile terminal.

In some embodiments, the method further includes S003: the display device receives the push message, and sends an authentication request to the server when the first identification information corresponds to the identification information of the display device;

In some embodiments, the display device determines the legitimacy of the mobile terminal in response to the push message;

The judgment of the validity of the mobile terminal includes: the display device judges whether the first identification information corresponds to the identification information of the display device according to the first identification information, and if so, the display device according to the display The identification information of the device (including the unique identification of the display device and/or the account currently logged in) and/or the first identification information sends an authentication request to the server.

In some embodiments, the above steps further include: after the display device receives the push message, judging whether the unique identifier (device ID1) of the display device in the push message is consistent with the unique identifier (device ID3) of the display device and/or the TV Check whether the account (customer ID 1) of the terminal is consistent with the account (customer ID 3) logged in on the display device, and then send an authentication request to the server according to the judgment result. In some embodiments, the display device sends an authentication request to the server when the unique identifier and/or the account of the TV terminal are consistent. In some embodiments, the authentication request includes the unique identification (device ID3) of the display device and/or the currently logged-in account (customer ID3) and/or the first identification information, since the display device in the push message at this time Whether the unique identifier (device ID1) of the display device is consistent with the unique identifier (device ID3) of the display device and/or whether the account number (customer ID 1) on the TV side is consistent with the account logged in on the display device (customer ID 3), so The authentication request may also include the unique identifier (device ID1) of the display device in the push message or the account number (customer ID1) of the TV terminal.

In some embodiments, the display device includes a small gathering at home call function and a Hijian call function, and the above steps further include: the display device receives the push message, and the display device determines the information according to the identification of the small gathering at home function. Whether the push message corresponds to Xiaoju at home; if so, the display device determines whether the TV account and TV ID carried in the push message are consistent with the account and device ID currently logged in by the device, if not, Then follow the Hijian call flow to establish a call. Hi, during the call, the user operation of the terminal and the terminal is required to select the answer and then establish the call connection. The function of Xiaoju at home can directly establish a call connection without displaying the terminal user to operate. Among them, Xiaojujia (application A) and Hijian (application B) are two functions of an application package.

It also includes after the display device judges whether the TV account and the TV ID carried in the push message are consistent with the account and device ID currently logged in by the device: if they are inconsistent, ignore them, and if they are consistent, execute sending authentication. request to the server.

In some embodiments, the authentication request includes the ID of the display device and/or the account currently logged in by the display device. The device ID and account information of the mobile terminal may also be included. The server performs the authentication determination according to the above information and the preset authorization mapping relationship.

In some embodiments, the method further includes S004: in response to the authentication request, the server determines, according to the authentication request, whether the mobile terminal represented by the second identifier is authorized by the display device represented by the first identifier.

In some embodiments, if authorized, an authentication success message is sent to the display device.

In some embodiments, it further includes, if the authentication is unsuccessful, sending an unsuccessful message to a display device (television), and the display device feeds back an unauthorized message to the mobile terminal through the server, and the mobile terminal uses Unauthorized prompt information is presented by means of interface or voice or text.

If the authentication is successful, a message of successful authentication is sent to the display device.

In some embodiments, the authentication request includes identification information and/or first identification information of the display device, and the server determines according to a pre-stored authorization relationship that the mobile terminal represented by the second identification has obtained the first identification information. An identification characterizes the authorization of the display device.

In some embodiments, the display device determines whether it is in a call state, and if it is in a call state, it determines whether the current call is a small gathering at home or a Hi-Meet function, and if it is a Hi-Meet function, the display terminal feeds back the terminal busy message through the server to mobile terminal. If the current call is the Xiaoju at home function, the session ID is sent to the server, and the server sends the session ID to the mobile terminal, so that the mobile terminal automatically joins the session corresponding to the session ID. Exemplary: the corresponding scenario is that when the second mobile terminal uses the small gathering at home function to establish a call with the display terminal, the first mobile terminal initiates a call request to the same display terminal. Since the first mobile terminal is authorized by the TV, The first mobile terminal can directly join the call established by the online second mobile terminal and the display terminal, and the windows in the second mobile terminal are changed from two (the TV terminal window and the local window of the second mobile terminal) to three (the TV terminal window). terminal window, local window of the second mobile terminal, window of the first mobile terminal). The first mobile terminal directly displays three windows (a TV terminal window, a local window of the first mobile terminal, and a second mobile terminal window). The TV side is transformed from one window (the second mobile terminal window) to two windows (the first mobile terminal window and the second mobile terminal window).

In some embodiments, the authentication operation may be performed before the server sends the call request to the display terminal (television), and after receiving the first request from the mobile terminal, the server determines the first identification information according to the pre-stored record information Whether the represented terminal authorizes the mobile terminal represented by the second identification information to directly establish a call. After the authentication is successful, a push message is sent to the server. After receiving the push message, the display device judges whether the first identification information corresponds to the identification information of the display device again according to the first identification information. Establish, if not corresponding, discard the push message. The authentication by the server and the further determination by the TV according to the first identification information can increase the accuracy of establishing a direct call and reduce the probability of misoperation.

In some embodiments, the method further includes S005: the display device automatically establishes the call process in response to the authentication success message.

In some embodiments, in response to the authentication failure message, the display device causes the server to send a call rejection message to the mobile terminal.

In some embodiments, after receiving the authentication success message, the display device automatically establishes a call process to achieve a call with the mobile terminal, avoiding the inconvenience of requiring a user operation to complete the call.

In some embodiments, after receiving the authentication success message, the display device automatically returns a confirmation message to automatically establish a call process; the server allocates a call room address for this call process, and sends the call room address to the mobile phone respectively. terminal and the display device, so that the display device and the mobile terminal establish a video call according to the call room address

In some embodiments, the establishment of the call process includes establishing a link of a video call and establishing a floating window that only displays image data of the mobile terminal in the form of a small window on the floating layer of the display of the display device, and does not establish a local video of the display terminal. window, as shown in Figure 8a. The small window method refers to establishing a non-full-screen display window, the area occupied by the floating window is smaller than the area of the display interface of the display, and the floating window is set to not be able to obtain focus. . In some embodiments, the window of the widget is built on the OSD overlay. At this time, in response to the creation of the video process, the mobile terminal displays the local video window of the mobile terminal and the window displaying the video of the display device on the interface, as shown in FIG. 8b.

In some embodiments, the display device provides a switching interface (for example, the user triggers a "setting" button on the remote control to perform a switching operation), and switches the window of the small window to a full-screen state. The switching interface may be a control on the display interface, or is the key value of a specific key of the remote control. Switch the floating window of the small window to a full-screen state. At this time, the display device cancels the floating display window of the small window mode, and pulls up the thread that displays the local video window of the device, and displays the local video window and the displayed mobile terminal image data in a full-screen mode. The full-screen mode refers to a display mode in which the layer where the local window used to display the local image and the window used to display the image data of the mobile terminal is located occupies the entire display screen of the display device. Local image refers to the image obtained by the device through the camera. On the interface of the display device, the local video window of the display device and the video window of the mobile terminal are simultaneously displayed. As shown in Figure 8c.

In some embodiments, the local window corresponding to the full-screen mode and the newly created first window (the peer window) are two windows established in the video layer, which are respectively used to display the image data acquired by the local camera and the peer device (mobile terminal) to send the image data. At this time, both windows can acquire the focus, so that the user can control the volume or image respectively.

In some embodiments, when the display displays the local window and the first window on the video layer, a switch widget operation input by the user is received; in response to the switch widget operation input by the user, the local window and the first window displayed by the video layer are canceled , and create the second window and the third window in the suspension layer, wherein the second window is used by the user to play the video data collected by the camera, and the third window is used to play the video data sent by the mobile terminal.

In some embodiments, the display device provides two functions of video calling, such as Xiaojujia (first function module or first application) and Hijian (second function module or second application) so in some embodiments : The display device determines whether the display device is in a call state; the display device determines whether the display device is in a call state according to whether a video call link exists. If yes, then judge whether the application corresponding to the call is Hijian or Xiaojujia, and if it is Hijian, feed back the information that the terminal is busy to the server, and the server sends the message that the terminal is busy to the mobile terminal, If the small gathering is at home, it proves that there is a video call link, then feed back the information that the terminal is busy to the server, and send the session ID to the server, and the server receives the polling transmission information (the session ID and the current call room). address (the server address corresponding to the session ID)) to the mobile terminal, and the mobile terminal automatically answers and joins the session. In response to the joining of the mobile terminal, the display device adds a call window corresponding to the mobile terminal on the call interface, as shown in Figure 9a, at this time, the windows in the meeting session of the mobile terminal are all displayed on the interface of the mobile terminal, including the local preview of the mobile terminal. window, the video window of the display device, and the video window of the other party in the existing call, as shown in Figure 9b.

In some embodiments, if not in a video call state, the display device determines whether the camera is occupied by other programs, and if so, the display device prompts the user on the display screen whether to answer the video call request, if not Then feed back the information that the terminal is busy to the server, and the server sends the message that the terminal is busy to the mobile terminal; if received, the display device exits the application program that occupies the camera, such as the chess and card application (stop the process). ), the display device automatically connects to the call, and displays only the mobile video in a small window. If the camera is not occupied by other applications, the display device automatically establishes a call process, establishes a video link, and establishes a video call interface that presents the mobile terminal in the form of a small window on the terminal.

In some embodiments, the display device receives a push message sent by a server, and the push message is sent by the server after receiving a video call request sent by the mobile terminal; When the display device is authorized, the call process is automatically established in response to the push message; when the mobile terminal is not authorized by the display device, the display device rejects the video call in response to the push message.

Further, the receiving of the push message sent by the server includes: the display device receives the push message sent by the server and the message of successful authentication, wherein the message of the successful authentication of the push message is that the server responds to receiving the video call initiated by the mobile terminal. If the request is sent, the authentication successful message indicates that the server determines that the mobile terminal is authorized by the display device;

Further, when the mobile terminal is authorized by the display device, in response to the push message, the process of automatically establishing a call includes: the display device determines, according to the authentication success message, that the mobile terminal has obtained the authorization. The authorization of the display device is obtained; the display device automatically establishes a call process to realize the transmission of audio and video data between the display device and the mobile terminal.

Further, the process of automatically establishing a call includes: the display device automatically sends a confirmation message to the server; the display device receives the call room address sent by the server in response to the confirmation message; the display device starts the video according to the call room address call.

Further, the push message includes a first identifier, the video call request includes a first identifier and a second identifier, the first identifier is used to represent the display device invited in the video call, and the second identifier is used. Used to characterize the mobile terminal that initiates the video call; the receiving the push message sent by the server and the successful authentication message includes: the display device receives the push message sent by the server; the display device responds to all the push messages in the push message. The first identification and the identification of the display device are the same, and an authentication request is sent to the server; the display device receives the authentication success message sent by the server, and the authentication success message is that the server responds to the authentication request. , sent when it is determined according to a pre-stored authorization relationship that the mobile terminal represented by the second identification has obtained the authorization of the display device represented by the first identification.

Further, the automatic establishment of the call process includes: a display device judging whether the camera is occupied; if the camera is occupied, judging the application currently occupying the camera; the application currently occupying the camera by the display device is the video call application, then controlling the application. The server sends the address of the current call room to the mobile terminal so that the mobile terminal directly joins the current call according to the address of the call room; the application that displays the device currently occupying the camera is another application, and controls the The server feeds back a message indicating that the camera of the final display device is occupied to the mobile terminal; if the camera is not occupied, a call process is automatically established.

In some embodiments, the display device receives a push message sent by a server, and the push message is sent by the server after receiving a video call request sent by the mobile terminal; When the authorization of the display device is displayed, the call process is automatically established in response to the push message, and the opposite end window is controlled to be displayed on the display without displaying the local window, wherein the opposite end window refers to the image data used for playing the image data sent by the mobile terminal. window, the local window refers to the window used to play the image data collected by the camera; the display device rejects the video in response to the push message when the mobile terminal is not authorized by the display device call.

Further, the receiving of the push message sent by the server includes: the display device receives the push message sent by the server and the message of successful authentication, wherein the message of the successful authentication of the push message is that the server responds to receiving the video call initiated by the mobile terminal. If the request is sent, the authentication successful message indicates that the server determines that the mobile terminal is authorized by the display device.

Further, when the mobile terminal is authorized by the display device, in response to the push message, automatically establishing a call process, and controlling the display to display the opposite end window and not display the local window includes: the display device according to the the authentication success message, it is determined that the mobile terminal is authorized by the display device; the display device automatically establishes a call process to realize the transmission of audio and video data between the display device and the mobile finger end; and Creating a peer window on the display interface does not create a local window.

In some embodiments, the peer window includes a floating window created on a floating layer of the display interface, and the area of the floating window is smaller than the display area of the display screen.

In some embodiments, the display device receives a switching operation input by the user; the display device cancels the floating window in response to the switching operation input by the user, and creates a local video window and a first window at the video layer, wherein the first The window is set to the peer window.

In some embodiments, the server sends a push message to the display device according to the received video call request sent by the mobile terminal, so that the display device, when the mobile terminal is authorized by the display device, In response to the push message, a call process is automatically established; when the mobile terminal is not authorized by the display device, the video call is rejected in response to the push message.

Further, sending the push message to the display device by the server according to the received video call request sent by the mobile terminal includes: receiving the video call request sent by the mobile terminal, where the first call request includes a first identifier and a second identifier, The first identifier is used to represent the display device that is invited in the video call, and the second identifier is used to represent the mobile terminal that initiates the video call; according to the second identifier, a push message is sent and the authentication is successful. message to the display device, wherein the authentication successful message is determined according to the first identifier, the second identifier and the pre-stored authorization relationship, and the authentication successful message indicates that the mobile terminal has obtained the authorized push message of the display device.

Further, the server receives the confirmation message automatically returned by the display device; the server sends the call room address to the mobile terminal and the display device respectively, so that the display device automatically establishes a video call with the mobile terminal.

Further, sending the push message and the authentication success message to the display device by the server according to the second identifier includes: the server querying the first identifier and the second identifier in a pre-stored authorization relationship according to the first identifier and the second identifier. Whether the second identifier has an authorization relationship; if so, send a push message and a successful authentication message to the display device, and if not, reject the video call request.

Further, the sending a push message and a successful authentication message to the display device according to the second identifier includes: the server sends a push message to the display device according to the first identifier, and the push message includes the The first identification and the second identification; the server receives an authentication request sent by the display device, the authentication request includes the first identification, and the authentication request is the display device in the first identification and display Sent when the identifiers of the devices are the same; the server queries whether there is an authorization relationship between the first identifier and the second identifier according to the pre-stored authorization relationship between the first identifier and the second identifier; if so, the authentication is successful. The message is sent to the display device; if it does not exist, an authentication failure message is sent to the display device.

Please refer to FIG. 11 , which shows a schematic structural diagram of a video call system provided by an embodiment of the present application. The video calling system includes a first mobile terminal 1110 , a second mobile terminal 1120 , a smart TV 1130 and a server 1140 . The first mobile terminal 1110 establishes a connection with the server 1140 through a wired or wireless network, the second mobile terminal 1120 establishes a connection with the server 1140 through a wired or wireless network, and the smart TV 1130 establishes a connection with the server 1140 through a wired or wireless network.

The first mobile terminal 1110 is a device with a video call function, such as a smart phone.

The second mobile terminal 1120 is a device with a video call function, such as a smart phone.

The server 1140 is a background server, which can be a server cluster or a cloud computing center composed of one server or multiple servers.

In the embodiment of the present application, during the first video call between the first mobile terminal 1110 and the smart TV 1130 through the server 1140, the second mobile terminal 1120 requests to make a second video call with the smart TV 1130, so as to join the first video call. The number of the first mobile terminal 1110 and the second mobile terminal 1120 is at least one, and FIG. 11 only shows one first mobile terminal 1110 and one second mobile terminal 1120 for illustration.

Please refer to FIG. 12 , which shows a method flowchart of a video calling method provided by an embodiment of the present application. The video calling method can be applied to the video calling system shown in FIG. 11 . The video call method includes:

Step S201, when the first mobile terminal makes a first video call with the smart TV through the server, the second mobile terminal sends a first call request to the server, where the first call request is used to request a second video call with the smart TV.

Step S202, the server sends a first call request to the smart TV according to the first call request, where the first call request is used to instruct the second mobile terminal to call the smart TV.

Step S203, the smart TV obtains the first call information of the first video call according to the first call request, and sends the first call information to the server, where the first call information is the identification information of the first video call.

Step S204, the server sends the first call information to the second mobile terminal.

Step 205, the second mobile terminal accesses the first video call according to the first call information.

To sum up, in the video call method provided by the embodiments of the present application, in the process of the first video call between the first mobile terminal and the smart TV through the server, if the second mobile terminal needs to initiate a second video call with the smart TV, Then the server will send the first call request to the smart TV, the smart TV will send the first call information of the first video call to the server, and the server will send the first call information to the second mobile terminal, thereby instructing the second mobile terminal to connect. enter the first video call, in this way, even if the second mobile terminal is not invited by the first mobile terminal, it can access the first video call.

Please refer to FIG. 13 , which shows a method flowchart of a video calling method provided by another embodiment of the present application. The video calling method can be applied to the video calling system shown in FIG. 11 . The video call method includes:

Step 301 , the first mobile terminal sends a second call request to the server, where the second call request is used to request a first video call with the smart TV.

Wherein, before step 301, the method further includes: the first mobile terminal sends the second terminal identification of the first mobile terminal and the TV identification of the smart TV to the server; the server determines the first mobile terminal according to the second terminal identification and the TV identification Whether authorized by the smart TV; if the first mobile terminal is authorized by the smart TV, send third confirmation information to the first mobile terminal; the first mobile terminal triggers the execution of step 301 according to the third confirmation information.

Step 302, the server generates first call information according to the second call request, and sends the first call information to the smart TV and the first mobile terminal respectively.

When the server includes a load balancing network element and a first service network element, step 302 may be implemented as:

The load balancing network element allocates the first service network element to the first mobile terminal according to the second call request, where the second call request includes the second terminal identifier of the first mobile terminal and the TV identifier of the smart TV;

The first service network element generates a second call request according to the second call request, sends the second call request to the smart TV, the second call request includes the first call information, the second terminal identification and the TV identification, and sends the first call to the smart TV. The information is sent to the first mobile terminal.

Step 303, the smart TV sends an answering success message to the server according to the first call information.

When the server includes the load balancing network element and the first service network element, step 303 may be implemented as:

The smart TV sends the second terminal identification and the TV identification to the server, and the server is used to determine whether the first mobile terminal is authorized by the smart TV according to the second terminal identification and the TV identification; if the first mobile terminal is authorized by the smart TV, then The second confirmation information is sent to the smart TV; the smart TV sends an answering success message to the first service network element according to the second confirmation information.

Step 304, the server sends the answering success message to the first mobile terminal.

When the server includes the load balancing network element and the first service network element, step 304 can be implemented as:

The first serving network element sends the answering success message to the first mobile terminal.

Step 305, the first mobile terminal accesses the first video call according to the first call information.

Step 306, the smart TV accesses the first video call according to the first call information.

Step 307, when the first mobile terminal makes a first video call with the smart TV through the server, the second mobile terminal sends a first call request to the server, where the first call request is used to request a second video call with the smart TV.

Step 308, the server sends a first call request to the smart TV according to the first call request, where the first call request is used to instruct the second mobile terminal to call the smart TV.

When the server includes a load balancing network element, a first service network element and a second service network element, and the first mobile terminal conducts the first video call with the smart TV through the first service network element, step 308 may be implemented as:

The load balancing network element allocates a second service network element to the second mobile terminal according to the first call request, where the first call request includes the first terminal identifier of the second mobile terminal and the TV identifier of the smart TV;

The second service network element generates the first call request according to the first call request, and sends the first call request to the smart TV, where the first call request includes the second call information of the second video call, the first terminal identifier and the TV identifier.

Step 309, the smart TV obtains the first call information of the first video call according to the first call request, and sends the first call information to the server.

When the server includes a load balancing network element, a first service network element and a second service network element, and the first mobile terminal conducts the first video call with the smart TV through the first service network element, step 309 may be implemented as:

The smart TV sends the first terminal identification and the TV identification to the server, and the server is used to determine whether the second mobile terminal is authorized by the smart TV according to the first terminal identification and the TV identification; if the second mobile terminal is authorized by the smart TV, then Send the first confirmation information to the smart TV; the smart TV obtains the first call information according to the first confirmation information, and sends the first call information to the second service network element.

Step 310, the server sends the first call information to the second mobile terminal.

When the server includes a load balancing network element, a first service network element and a second service network element, and the first mobile terminal conducts the first video call with the smart TV through the first service network element, step 310 may be implemented as: the second service network element The network element sends the first call information to the second mobile terminal.

Step 311, the second mobile terminal accesses the first video call according to the first call information.

In this embodiment, the method further includes:

The second service network element sends the second call information to the load balancing network element;

The load balancing network element sends the second call information to the second mobile terminal;

The second mobile terminal sends the first polling request to the second service network element according to the second call information;

The second service network element sends the first polling response to the second mobile terminal, where the first polling response includes the first call information;

The second mobile terminal receives the first polling response, and acquires the first call information from the first polling response.

To sum up, in the video call method provided by the embodiments of the present application, in the process of the first video call between the first mobile terminal and the smart TV through the server, if the second mobile terminal needs to initiate a second video call with the smart TV, Then the server will send the first call request to the smart TV, the smart TV will send the first call information of the first video call to the server, and the server will send the first call information to the second mobile terminal, thereby instructing the second mobile terminal to connect. enter the first video call, in this way, even if the second mobile terminal is not invited by the first mobile terminal, it can access the first video call.

Wherein, steps 301-306 describe the process of establishing the first video call between the first mobile terminal and the smart TV, and the implementation process of which is described below with reference to FIG. 14 . Among them, the load balancing network element in the server is CSR (Call Signaling Router, video call signaling route), the service network element is CSE (Call Signaling Engine, video call signaling engine), and the server also includes system API (Application Programming Interface, application programming interface) and messaging platform.

(1) Check: The first mobile terminal sends a request to the API, carrying the deviceid (deviceid), the client identifier (customerid) of the first mobile terminal, and the deviceid and customerid of the smart TV.

(2) Ret OK: The API checks whether the deviceid of the customerid of the smart TV authorizes the video call of the deviceid of the customerid of the first mobile terminal. If authorized, return an ok message. If not authorized, return unauthorized information. If unauthorized information is returned, end the call. If the ok message is returned, go to step (3).

(3) homeInvite: The first mobile terminal initiates a call request to the CSR of the video call.

(4) homeInvite: The CSR receives the call request, selects a CSE, and forwards the call request to the CSE.

(5) Task: CSE receives the call request, assigns the unique session id of the call, and sends a push request to the messaging platform. The push request carries the unique session id, the url address of the CSE, the customerid of the smart TV, and the deviceid and customerid and calling number.

(6) Response: The CSE returns information to the CSR, carrying the unique session id and the url address of the serving network element CSE.

(7) Response: The CSR forwards the information returned by the CSE to the first mobile terminal.

(12) (14) (16) homeRing: The first mobile terminal receives the url address of the CSE, then polls the CSE, and waits for the smart TV to answer.

(8) Message: The message platform sends a message to the deviceid corresponding to the customer of the smart TV.

(9) Receive task: The message platform returns the message push result to the CSE, and the message push result includes whether to push the message to the smart TV. If the smart TV is not online, the CSE ends the call. The first mobile terminal obtains a reply of the end of the call in the polling of (12) (14) (16). The first mobile terminal ends the call.

(10) Check: The smart TV sends a request to the API, and the request carries the deviceid and customerid of the smart TV, and the deviceid and customerid of the first mobile terminal.

(11) Ret OK: The API checks whether the deviceid of the customerid of the first mobile terminal is authorized by the deviceid of the customerid of the smart TV. If authorized, return ok message. If not authorized, return unauthorized information.

(13) homeRing busy_type=0: The smart TV obtains the unique session id from the message push, the url address of the CSE, the customerid of the smart TV, the deviceid and customerid of the first mobile terminal, and the number. If not authorized, send unauthorized information to CSE. The first mobile terminal obtains a reply that the call is not authorized in the polling of (12) (14) (16). The first mobile terminal ends the call.

(15) homeReceive OK: If authorized, the smart TV sends a success message to the CSE. The first mobile terminal obtains a reply that the call is answered successfully in the polling of (12) (14) (16).

(17) homeTalking: After the first mobile terminal enters the call, it joins the unique room mapped by the unique session id, conducts a video call in the room, and polls the url address of the CSE for heartbeat messages to ensure that the call process continues.

(18) homeTalking: After the smart TV enters the call, it joins the unique room mapped by the unique session id, makes a video call in the room, and polls the url address of the CSE for heartbeat messages to ensure that the call process continues.

Wherein, steps 307-311 describe the process of accessing the first video call by the second mobile terminal, and the implementation process thereof will be described below with reference to FIG. 15 . The load balancing network element in the server is CSR, the service network element is CSE, and the server also includes a system API and a message platform.

(1) homeTalking: After the first mobile terminal enters the call, it joins the unique room mapped by the unique session id session1, conducts a video call in the room, and polls the url1 address of the CSE for heartbeat messages to ensure that the call process continues.

(2) homeTalking: After the smart TV enters the call, it joins the unique room mapped by the unique session id session1, conducts a video call in this room, and polls the url1 address of the CSE for heartbeat messages to ensure that the call process continues.

(3) Check: The second mobile terminal sends a request to the API, carrying the deviceid and customerid of the first mobile terminal, and the deviceid and customerid of the smart TV.

(4) Ret OK: The API checks whether the deviceid of the customerid of the smart TV has authorized the deviceid of the customerid of the second mobile terminal to make a video call. If authorized, return ok message. If not authorized, return unauthorized information. If the return message is not authorized, end the call. If the ok message is returned, go to step (5).

(5) homeInvite: The second mobile terminal initiates a call request to the CSR of the video call.

(6) homeInvite: CSR receives the call request, selects a CSE, and forwards the call request to the CSE.

(7) Task: CSE receives the call request, assigns the unique session id session2 of the call, and sends a push request to the messaging platform. The push request carries the unique session id session2, the url2 address of the CSE, the customerid of the smart TV, and the second mobile The deviceid and customerid of the terminal and the calling number.

(8) Response: The CSE returns information to the CSR, carrying the unique session id session2 and the url2 address of the CSE.

(9) Response: The CSR forwards the information returned by the CSE to the second mobile terminal.

(15) homeRing msg: The second mobile terminal receives the url2 address of the CSE, then polls the CSE, and waits for the smart TV to answer.

(10) Message: The message platform sends a message to the deviceid corresponding to the customer of the called smart TV.

(11) Receive task: The message platform returns the message push result to the CSE, and the message push result includes whether it is pushed to the called party. If the smart TV is not online, the CSE ends the call. The second mobile terminal obtains a reply of the end of the call in the polling of (15) (homeRingmsg). The second mobile terminal ends the call.

(12) Check: The smart TV sends a request to the API, and the request carries the deviceid and customerid of the smart TV, and the deviceid and customerid of the second mobile terminal.

(13) Ret OK: The API checks whether the deviceid of the customerid of the second mobile terminal is authorized by the deviceid of the customerid of the called smart TV. If authorized, return ok message. If not authorized, return unauthorized information.

(14) homeRing is_busy=1 busy_type=1 msg: The smart TV obtains the unique session id session2, the url2 address of the CSE, the customerid of the smart TV, the deviceid and customerid of the second mobile terminal, and the calling number from the message push. If not authorized, send unauthorized information to CSE. The second mobile terminal obtains a reply that the call is not authorized in the polling of (15). The second mobile terminal ends the call. If authorized, send an answering success message to the CSE, and the answering success message includes the unique session id session1 of the calling and the url1 address of the CSE that is being called. The second mobile terminal obtains a reply that the call is answered successfully in the polling of (15).

(16) homeReceive: The second mobile terminal obtains the unique session id session1 in the call and the url1 address of the CSE in the call from the polling message in (15), and initiates a call join request to the url1 address of the CSE, and the request carries The unique session id session1, the deviceid and customerid of the second mobile terminal, and the calling number.

(17) homeTalking: After the second mobile terminal enters the call, it joins the unique room mapped by the unique session id session1, conducts a video call in the room, and polls the url1 address of the CSE for heartbeat messages to ensure that the call process continues.

(18) homeTalking: The smart TV continues to hold the call, continues to make a video call in the unique room mapped by the unique session id session1, and polls the url1 address of the CSE for heartbeat messages to ensure that the call process continues. Synchronously obtain the information of newly added members.

(19) homeTalking: The first mobile terminal continues to hold the call, continues to make a video call in the unique room mapped by the unique session id session1, polls the url1 address of the CSE for heartbeat messages, and ensures that the call process continues. Query message to obtain the newly added member information synchronously.

Please refer to FIG. 16 , which shows a structural block diagram of a video call apparatus provided by an embodiment of the present application, and the video call apparatus can be applied to a server. The video calling device includes:

The receiving module 610 is configured to receive a first call request sent by the second mobile terminal when the first mobile terminal conducts a first video call with the smart TV through the server, and the first call request is used to request a second video call with the smart TV ;

a sending module 620, configured to send a first call request to the smart TV according to the first call request, where the first call request is used to instruct the second mobile terminal to call the smart TV;

The receiving module 610 is further configured to receive the first call information of the first video call sent by the smart TV according to the first call request;

The sending module 620 is further configured to send the first call information to the second mobile terminal, where the first call information is used to instruct the second mobile terminal to access the first video call.

In some embodiments, a display device includes: a camera;

A display, used to display a peer window and/or a local window, wherein the peer window refers to a window used to play image data sent by the mobile terminal, and the local window refers to a window used to play the image data collected by the camera. window for image data;

a processor for:

Receive a push message sent by a server, where the push message is sent by the server after receiving a video call request sent by the mobile terminal; when the mobile terminal is authorized by the display device, respond to the Pushing a message, automatically establishing a call process, and controlling the display to display the opposite end window and not display the local window; when the mobile terminal is not authorized by the display device, in response to the push message, reject the video call.

Further, the processor is also used for:

When the peer window is displayed on the monitor and the local window is not displayed, the switch operation input by the user is received;

In response to the switching operation input by the user, the floating window is canceled, and a local video window and a first window are created on the video layer, wherein the first window is set as the opposite terminal window.

Further, the processor is also used for:

When the display displays the local window and the first window on the video layer, receiving a user-input switching operation of small windows;

In response to the user input switching the small window operation, cancel the local window and the first window displayed by the video layer, and create the second window and the third window in the floating layer, wherein the second window is used by the user to play the camera The collected video data, the third window is used to play the video data sent by the mobile terminal.

In some embodiments, a display device includes: a microphone; a speaker; a camera;

A display, used to display a peer window and/or a local window, wherein the peer window refers to a window used to play image data sent by the mobile terminal, and the local window refers to a window used to play the image data collected by the camera. window for image data;

a processor for:

receiving a push message sent by a server, where the push message is sent by the server after receiving a video call request sent by the mobile terminal;

When the mobile terminal is authorized by the display device, in response to the push message, control the display on the display to display the peer window but not the local window, control the speaker to output the audio data sent by the mobile terminal, control The camera collects image data for sending to the mobile terminal, and controls the microphone to collect audio data for sending to the mobile terminal;

The video call is rejected in response to the push message when the mobile terminal is not authorized by the display device.

To sum up, in the video call device provided by the embodiments of the present application, during the process of the first video call between the first mobile terminal and the smart TV through the server, if the second mobile terminal needs to initiate a second video call with the smart TV, Then the server will send the first call request to the smart TV, the smart TV will send the first call information of the first video call to the server, and the server will send the first call information to the second mobile terminal, thereby instructing the second mobile terminal to connect. enter the first video call, in this way, even if the second mobile terminal is not invited by the first mobile terminal, it can access the first video call.

Based on the exemplary embodiments shown in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application. In addition, although the disclosures in this application are presented in terms of one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution individually.

It should be understood that the terms "first", "second", "third" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that data so used may be interchanged under appropriate circumstances, eg, can be implemented according to an order other than those presented in the illustrations or descriptions of the embodiments of the present application.

Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover but not exclusively include, for example, a product or device incorporating a series of components is not necessarily limited to those explicitly listed, but may include No other components are expressly listed or inherent to these products or devices.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of some embodiments of the present application. scope.

Claims (18)

1. A display device, comprising:

a camera;

the display is used for displaying an opposite-end window and/or a local window, wherein the opposite-end window is used for playing image data sent by the mobile terminal, and the local window is used for playing the image data acquired by the camera;

a processor to:

receiving a push message sent by a server, wherein the push message is sent by the server after receiving a video call request sent by the mobile terminal;

when the mobile terminal is authorized by the display device, responding to the push message, automatically establishing a conversation process, and controlling an opposite terminal window displayed on a display not to display a local window;

and when the mobile terminal is not authorized by the display equipment, responding to the push message and refusing the video call.

2. The display device of claim 1, the processor further to:

when the opposite terminal window is displayed on the display and the local window is not displayed, receiving switching operation input by a user;

and in response to a switching operation input by a user, canceling the floating window, and creating a local video window and a first window in a video layer, wherein the first window is set as the opposite-end window.

3. The display device of claim 2, the processor further to:

when a display displays a local window and a first window on a video layer, receiving a small window switching operation input by a user;

and in response to the small window switching operation input by the user, canceling the local window and the first window displayed on the video layer, and creating a second window and a third window on the suspension layer, wherein the second window is used for playing the video data acquired by the camera, and the third window is used for playing the video data sent by the mobile terminal.

4. A display device, comprising:

a microphone;

a speaker;

a camera;

the display is used for displaying an opposite-end window and/or a local window, wherein the opposite-end window is used for playing image data sent by the mobile terminal, and the local window is used for playing the image data acquired by the camera;

a processor to:

receiving a push message sent by a server, wherein the push message is sent by the server after receiving a video call request sent by the mobile terminal;

when the mobile terminal is authorized by the display device, responding to the push message, controlling a display to display an opposite terminal window on a display and not to display a local window, controlling the loudspeaker to output audio data sent by the mobile terminal, controlling the camera to collect image data used for sending to the mobile terminal, and controlling the microphone to collect audio data used for sending to the mobile terminal;

and when the mobile terminal is not authorized by the display equipment, responding to the push message and refusing the video call.

5. A video call method is applied to display equipment and is characterized by comprising the following steps:

receiving a push message sent by a server, wherein the push message is sent by the server after receiving a video call request sent by the mobile terminal;

when the mobile terminal is authorized by the display device, responding to the push message, and automatically establishing a conversation process;

and when the mobile terminal is not authorized by the display equipment, responding to the push message and refusing the video call.

6. The method of claim 5, comprising:

the receiving the push message sent by the server comprises:

receiving a push message and a message of successful authentication sent by a server, wherein the message of successful authentication is sent by the server in response to a video call request sent by a mobile terminal, and the message of successful authentication represents that the server determines that the mobile terminal is authorized by the display device;

when the mobile terminal is authorized by the display device, responding to the push message, and automatically establishing a call process comprises:

according to the message of successful authentication, determining that the mobile terminal is authorized by the display device;

and automatically establishing a conversation process to realize the transmission of audio and video data between the display equipment and the mobile finger end.

7. The method of claim 5, wherein: the automatic call establishment process comprises the following steps:

automatically sending a confirmation message to the server;

receiving a call room address sent by the server in response to the confirmation message;

and starting the video call according to the call room address.

8. The method of claim 5, wherein: the push message comprises a first identifier, the video call request comprises the first identifier and a second identifier, the first identifier is used for representing the display equipment invited in the video call, and the second identifier is used for representing the mobile terminal initiating the video call;

the receiving of the push message and the authentication success message sent by the server includes:

receiving a push message sent by a server;

responding to the first identification in the push message and the identification of the display equipment to be the same, and sending an authentication request to a server;

and receiving a message of successful authentication sent by a server, wherein the message of successful authentication is sent when the server responds to the authentication request and determines that the mobile terminal represented by the second identifier is authorized by the display equipment represented by the first identifier according to a prestored authorization relationship.

9. The video call method of claim 5, wherein: the automatic call establishment process comprises the following steps:

judging whether the camera is occupied or not;

if the camera is occupied, judging the application of the currently occupied camera;

if the application occupying the camera at present is the video call application, controlling the server to send the address of the current call room to the mobile terminal so that the mobile terminal directly joins the current call according to the address of the call room;

when the application occupying the camera is other applications, controlling the server to feed back a message representing that the camera of the terminal display equipment is occupied to the mobile terminal;

and if the camera is not occupied, automatically establishing a conversation process.

10. A video call method is applied to display equipment and is characterized by comprising the following steps:

receiving a push message sent by a server, wherein the push message is sent by the server after receiving a video call request sent by the mobile terminal;

when the mobile terminal is authorized by the display device, automatically establishing a call process in response to the push message, and controlling an opposite-end window displayed on a display not to display a local window, wherein the opposite-end window is a window used for playing image data sent by the mobile terminal, and the local window is a window used for playing the image data acquired by the camera;

and when the mobile terminal is not authorized by the display equipment, responding to the push message and refusing the video call.

11. The video call method of claim 10, wherein: the receiving the push message sent by the server comprises:

receiving a push message and a message of successful authentication sent by a server, wherein the message of successful authentication is sent by the server in response to a video call request sent by a mobile terminal, and the message of successful authentication represents that the server determines that the mobile terminal is authorized by the display device;

when the mobile terminal is authorized by the display device, responding to the push message, automatically establishing a communication process, and controlling an opposite terminal window displayed on a display not to display a local window comprises:

according to the message of successful authentication, determining that the mobile terminal is authorized by the display device;

automatically establishing a conversation process to realize the transmission of audio and video data between the display equipment and the mobile finger end; and creating a peer window on the display interface without creating a local window.

12. The video call method of claim 10, wherein:

the opposite-end window comprises a floating window which is created on a floating layer of the display interface, and the area of the floating window is smaller than the display area of the display screen.

13. The video call method of claim 12, wherein the method further comprises:

receiving switching operation input by a user;

and in response to a switching operation input by a user, canceling the floating window, and creating a local video window and a first window in a video layer, wherein the first window is set as the opposite-end window.

14. A video call method is applied to a server and comprises the following steps:

sending a push message to the display equipment according to a received video call request sent by the mobile terminal, so that the display equipment responds to the push message when the mobile terminal is authorized by the display equipment, and automatically establishing a call process; and when the mobile terminal is not authorized by the display equipment, responding to the push message and refusing the video call.

15. The method of claim 14, wherein the method further comprises:

the step of sending a push message to the display device according to the received video call request sent by the mobile terminal comprises the following steps:

receiving a video call request sent by a mobile terminal, wherein the first call request comprises a first identifier and a second identifier, the first identifier is used for representing the display equipment invited in the video call, and the second identifier is used for representing the mobile terminal initiating the video call;

and sending a push message and a message of successful authentication to the display device according to the second identifier, wherein the message of successful authentication is determined according to the first identifier, the second identifier and a prestored authorization relationship, and the message of successful authentication represents that the mobile terminal obtains an authorized push message of the display device.

16. The method of claim 14, wherein the method further comprises:

receiving a confirmation message automatically returned by the display equipment;

and respectively sending the call room addresses to the mobile terminal and the display equipment so as to enable the display equipment to automatically establish a video call with the mobile terminal.

17. The video call method of claim 15, wherein sending a push message and a successfully authenticated message to the display device according to the second identifier comprises:

inquiring whether the first identifier and the second identifier have an authorization relationship in a prestored authorization relationship according to the first identifier and the second identifier;

and if the video call request does not exist, rejecting the video call request.

18. The video call method of claim 15, wherein sending a push message and a successfully authenticated message to the display device according to the second identifier comprises:

sending a push message to the display device according to the first identifier, wherein the push message comprises the first identifier and the second identifier;

receiving an authentication request sent by display equipment, wherein the authentication request comprises the first identifier, and the authentication request is sent by the display equipment when the first identifier is the same as the identifier of the display equipment;

inquiring whether the first identifier and the second identifier have an authorization relationship in a prestored authorization relationship according to the first identifier and the second identifier;

if yes, sending a message of successful authentication to the display equipment;

and if not, sending a message of authentication failure to the display equipment.

Images