[go: up one dir, main page]

WO2023068849A1 - Dispositif d'affichage et son procédé de fonctionnement - Google Patents

Dispositif d'affichage et son procédé de fonctionnement Download PDF

Info

Publication number
WO2023068849A1
WO2023068849A1 PCT/KR2022/016054 KR2022016054W WO2023068849A1 WO 2023068849 A1 WO2023068849 A1 WO 2023068849A1 KR 2022016054 W KR2022016054 W KR 2022016054W WO 2023068849 A1 WO2023068849 A1 WO 2023068849A1
Authority
WO
WIPO (PCT)
Prior art keywords
content
display device
screen
neural network
network model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2022/016054
Other languages
English (en)
Korean (ko)
Inventor
이종인
김세현
양관식
최길수
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of WO2023068849A1 publication Critical patent/WO2023068849A1/fr
Priority to US18/640,479 priority Critical patent/US12537996B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/19—Recognition using electronic means
    • G06V30/191—Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
    • G06V30/19173—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/40—Document-oriented image-based pattern recognition
    • G06V30/42—Document-oriented image-based pattern recognition based on the type of document
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/4363—Adapting the video stream to a specific local network, e.g. a Bluetooth® network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/4363—Adapting the video stream to a specific local network, e.g. a Bluetooth® network
    • H04N21/43632—Adapting the video stream to a specific local network, e.g. a Bluetooth® network involving a wired protocol, e.g. IEEE 1394
    • H04N21/43635—HDMI
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/443—OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/466—Learning process for intelligent management, e.g. learning user preferences for recommending movies
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/466—Learning process for intelligent management, e.g. learning user preferences for recommending movies
    • H04N21/4662—Learning process for intelligent management, e.g. learning user preferences for recommending movies characterized by learning algorithms
    • H04N21/4666—Learning process for intelligent management, e.g. learning user preferences for recommending movies characterized by learning algorithms using neural networks, e.g. processing the feedback provided by the user

Definitions

  • Various embodiments relate to a display device and an operating method thereof, and more specifically, to a display device capable of providing a content execution environment based on content executed on a display device and an operating method thereof.
  • a display device that receives and displays content executed on an external device connected to the display device, characteristic information of the content is obtained from the received content, and the display device can control the content execution environment of the display device. And it aims to provide an operation method thereof.
  • a display device may include a display, an input/output interface, a communication interface, a memory storing one or more instructions, and a processor executing the one or more instructions stored in the memory.
  • the processor may display a screen of an image received from a connected electronic device by executing the one or more instructions.
  • the processor may determine whether content execution starts by analyzing the displayed video screen using a first neural network model by executing the one or more instructions.
  • the processor may call a second neural network model when it is determined that the execution of the content starts by executing the one or more instructions.
  • the processor may obtain attribute information of the content by analyzing a video screen of the content using the second neural network model by executing the one or more instructions.
  • the processor may control an execution environment of the content based on the acquired attribute information by executing the one or more instructions.
  • the processor identifies the electronic device from HDMI Consumer Electronics Control (CEC) received through High Definition Multimedia Interface (HDMI) communication from the electronic device by executing the one or more instructions, and the identified The first neural network model learned in correspondence with the electronic device may be obtained.
  • HDMI Consumer Electronics Control CEC
  • HDMI High Definition Multimedia Interface HDMI
  • HDMI Consumer Electronics Control
  • HDMI High Definition Multimedia Interface
  • the first neural network model receives a plurality of user interface (UI) screens that can be received from the electronic device as training data, and the content executable UI screen and the content non-executable UI screen are input. It can include models trained to classify UI screens.
  • UI user interface
  • the processor by executing the one or more instructions, inputs the displayed video screen to the first neural network model and analyzes it, thereby determining whether the content executable UI screen is converted to the content executable UI screen.
  • the processor inputs the displayed video screen to the first neural network model and analyzes it, thereby determining whether the content executable UI screen is converted to the content executable UI screen.
  • the second neural network model may include a model trained to receive a plurality of video screens as training data and detect a text area or a logo area from the video screens.
  • the processor extracts the text area or the logo area from the video screen by inputting the video screen to the second neural network model for analysis, and extracts the text area or the logo area from the video screen. Attribute information of the content may be obtained based on the logo area.
  • the processor transmits the text or the logo area extracted from the text area to a server by executing the one or more instructions, and receives attribute information of the content related to the text or the logo area from the server. By receiving, attribute information of the content may be obtained.
  • the processor controls at least one of picture quality setting, sound setting, viewing age limit setting, and display device environment setting suitable for the content based on the acquired attribute information by executing the one or more instructions.
  • Content execution environment can be controlled.
  • the processor determines whether the execution of the content is terminated after controlling the execution environment of the content, and uses the first neural network model when it is determined that the execution of the content is terminated. By analyzing the video screen, it is possible to determine whether a new content execution is started.
  • a method of operating a display device may include an operation of displaying a screen of an image received from a connected electronic device.
  • the operating method of the display device may include an operation of determining whether content execution starts by analyzing the displayed video screen using a first neural network model.
  • the operating method of the display device may include calling a second neural network model when it is determined that the execution of the content starts.
  • the operating method of the display device may include obtaining attribute information of the content by analyzing a video screen of the content using the second neural network model.
  • a method of operating a display device may include controlling an execution environment of the content based on the acquired attribute information.
  • the method of operating the display device may include receiving from a connected electronic device An operation of displaying a screen of an image, an operation of determining whether content execution starts by analyzing the displayed video screen using a first neural network model, an operation of calling a second neural network model when it is determined that the execution of the content starts, An operation of obtaining property information of the content by analyzing a video screen of the content using a second neural network model, and an operation of controlling an execution environment of the content based on the acquired property information.
  • the display device even when a display device receives and displays content executed in an external device connected to the display device, by providing a method of obtaining characteristic information of the content from the received content, It is possible to control the content execution environment of the display device according to the characteristics of the display device. Accordingly, the display device can provide users with a content experience that matches the characteristics of the content.
  • FIG. 1 is a reference diagram for explaining a system for controlling an execution environment of a display device according to an exemplary embodiment.
  • FIG. 2 shows an example of a system including a display device, an electronic device and a server computer according to an embodiment.
  • FIG. 3 is a block diagram illustrating a specific configuration of a display device according to an exemplary embodiment.
  • 4A is a reference diagram for explaining a method of learning a first neural network model corresponding to each electronic device according to an embodiment.
  • 4B is a reference diagram for explaining a method of acquiring a first neural network model corresponding to a first electronic device according to an embodiment.
  • FIG. 5 is a flowchart illustrating an example of a method of operating a display device according to an exemplary embodiment.
  • FIG. 6 is a reference diagram for explaining the operating method shown in FIG. 5 according to an embodiment.
  • FIG. 7 is a flowchart illustrating a process of a method of obtaining a first neural network model in a display device according to an embodiment.
  • FIG. 8 is a flowchart illustrating a process of a method of detecting a content execution start time in the display device 100 according to an embodiment.
  • 9 is a reference diagram for explaining a method of determining a content execution time according to an example.
  • FIG. 9 is a reference diagram for explaining a method of determining a content execution time according to an example.
  • FIG. 10 illustrates an example of a neural network model for classifying an image screen according to an exemplary embodiment.
  • FIG. 11 is a reference diagram for explaining a method of recognizing content after a content execution start point in a display device according to an exemplary embodiment.
  • FIG 12 illustrates an example of an object detection model according to an example.
  • FIG. 13 is a reference diagram for explaining a process of obtaining property information of content by analyzing a video screen according to an example.
  • FIG. 14 illustrates an example of a table including picture quality setting values and sound setting values set according to genres of content according to an embodiment.
  • 15 is a reference diagram for explaining an operation of a display device when viewing age information is received as content property information according to an exemplary embodiment.
  • 16 is a reference diagram for describing switching between a content execution timing determination mode and a content property recognition mode according to an exemplary embodiment.
  • the term "user” refers to a person who controls a function or operation of a computing device or electronic device using a control device, and may include a viewer, administrator, or installer.
  • FIG. 1 is a reference diagram for explaining a system for controlling an execution environment of a display device according to an exemplary embodiment.
  • the system may include a display device 100, an electronic device 200, and a server computer 300.
  • a display device 100 may be an electronic device that receives content from various sources and displays the received content.
  • the display device 100 includes a TV, a set-top box, a mobile phone, a tablet PC, a digital camera, a camcorder, a laptop computer, a desktop, an e-reader, a digital broadcasting terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP).
  • PDA personal digital assistant
  • PMP portable multimedia player
  • the display device 100 may be a fixed electronic device disposed at a fixed location or a mobile electronic device having a portable form, and may be a digital broadcasting receiver capable of receiving digital broadcasting.
  • the display device 100 may be controlled using IR (Infrared), BT (Bluetooth), Wi-Fi, and the like by various types of devices such as a remote controller or a mobile phone.
  • IR Infrared
  • BT Bluetooth
  • Wi-Fi Wireless Fidelity
  • various types of devices such as a remote controller or a mobile phone.
  • the electronic device 200 may execute content and transmit the content execution screen to the display device 100 through wired or wireless communication.
  • the electronic device 200 may include game consoles manufactured by various manufacturers.
  • the executed game content screen can be displayed on the display device 100 .
  • the display device 100 may control the environment of the display device 100 according to characteristics of content displayed on the display device 100 or content property information. For example, the display device 100 may perform image quality processing appropriate to properties of content displayed on the display device 100, set sound, or provide services such as age restriction.
  • the display device 100 when the display device 100 receives a screen as a result of content execution from the electronic device 200, it may be difficult for the display device 100 to directly obtain attribute information on content corresponding to the received screen from the electronic device 200. Therefore, in this situation, the display device 100 needs a method of acquiring information about content or property information of content received from the content execution screen received from the electronic device 200 .
  • the content execution screen received from the electronic device 200 In order to obtain attribute information about content from the content execution screen received from the electronic device 200 by the display device 100, the content execution screen received from the electronic device 200 must be analyzed. At this time, if the display device 200 attempts to acquire content attribute information by analyzing all image frames received from the electronic device 200, the accuracy of the analysis may decrease, and resources such as time and resources may be significantly wasted in the analysis.
  • the display device 100 systematizes the process of analyzing the content execution screen received from the electronic device 200, recognizes the content (30), and uses the attribute information of the recognized content to display the content execution environment of the display device 100. You want to control (50).
  • control of a content execution environment refers to providing a better user experience when the display device 100 displays a video screen received from the electronic device 200 in accordance with the characteristics of the video screen displayed on the display device 100.
  • the display device 100 may analyze the video screen received from the electronic device 200 to determine whether content execution starts. In order to obtain content attribute information, it is desirable to analyze a screen after a specific content is selected and executed by a user. Before the actual content selection and execution, various video screens, such as setting screens or UI screens for selecting content execution, may precede. You will not need to do any analysis to extract the properties of . Accordingly, it is preferable that the display device 100 searches for a point at which content execution starts before an operation for extracting attribute information of actual content, and analyzes for extracting content attribute information after the found point.
  • the display device 100 may use a first neural network model to analyze whether content execution starts from the video screen received from the electronic device 200 .
  • the first neural network model may be a first neural network model specialized for the electronic device 200 used to determine whether content execution starts from the video screens provided by the electronic device 200 by training based on the video screens provided by the electronic device 200. there is.
  • the display device 100 obtains property information of the content from the video screen received from the electronic device 200 by calling a second neural network model when it is determined that content execution starts by analyzing the video screen received from the electronic device 200. action can be performed.
  • the display device 100 may recognize a text area or a logo area from an image screen received from the electronic device 200 and obtain content property information based on the recognized text area or logo area.
  • Attribute information of the content may include, for example, metadata of the content, such as a title or genre of the content, or viewing age possibility information of the content.
  • the display device 100 may control an execution environment of content based on acquired content attribute information.
  • the content execution environment includes a picture quality processing part for processing or controlling video data displayed on the display device 100 while the display device 100 reproduces audio video data included in the content, and processing or controlling audio data output from the display device 100. It may include services such as a sound setting part for and a viewing age restriction.
  • the server computer 300 may communicate with the display device 100 through a communication network 70 .
  • the server computer 300 may receive a request from the display device 100 through the communication network 70 and transmit a response corresponding to the request to the display device 100 .
  • the display device 100 may store the first neural network model or the second neural network model used when analyzing the video screen previously received from the electronic device 200 in the display device 100 itself, but the display device 100 The first neural network model or the second neural network model may be requested and received from the server computer 300 . Also, the display device 100 may receive an updated version of the first neural network model or the second neural network model from the server computer 300 periodically or upon request.
  • the display device 100 when the display device 100 obtains property information about content by analyzing a video screen previously received from the electronic device 200 and recognizing a text area or a logo area, the text or logo area extracted from the recognized text area. It is possible to transmit information about to the server computer 300 and receive property information of content obtained based on the text or logo area transmitted from the server computer 300 .
  • the server computer providing the first neural network model or the second neural network model may be the same as or different from the server computer providing attribute information of the content obtained based on the text or logo area.
  • FIG. 2 shows an example of a system including a display device, an electronic device and a server computer according to an embodiment.
  • the system may include a display device 100, an electronic device 200, and a server computer 300.
  • the electronic device 200 is an electronic device that is connected to the display device 100 by wire or wirelessly to transmit and receive data and/or content.
  • the electronic device 200 may execute game content and transmit a content execution screen to the display device 100 .
  • the electronic device 200 may transmit other video contents and/or audio contents other than game contents.
  • the electronic device 200 may be any device capable of transmitting and receiving data by connecting to the display device 100 .
  • the electronic device 200 may include various types of electronic devices capable of providing content to the display device 100, such as, for example, a set-top box, a DVD player, a Blu-ray disc player, a PC, and a game machine.
  • the electronic device 200 and the display device 100 may transmit/receive content by being connected through various connection means.
  • Various connection means may include, for example, a cable, and the electronic device 200 and the display device 100 may include one or more ports for cable connection.
  • the one or more ports may include, for example, a digital input interface such as an HDMI port, DisplayPort, Type-C, or the like.
  • the electronic device 200 may be a device dedicated to game content, such as a game console.
  • the electronic device 200 is not limited to a game console, and may be any type of device that provides various contents such as game contents, movie contents, and video contents.
  • the electronic device 200 may include an input/output unit 210, a communication unit 220, a memory 230, and a control unit 240.
  • the input/output unit 210 may be connected to an external device through a wire to input or output data. According to an embodiment, the input/output unit 210 may be connected to the input/output unit 110 of the display device 100 by wire to transmit an execution screen of content executed in the electronic device 200 to the display device 100 .
  • the input/output unit 210 may include an HDMI port.
  • the input/output unit 210 may transmit device information about the electronic device 200 to the display device 100 through an HDMI photo call.
  • the communication unit 220 can wirelessly connect to an external device to input or output data. According to an embodiment, the communication unit 220 is wirelessly connected to the communication unit 110 of the display device 100 to transmit a video screen executed on the electronic device 200 to the display device 100 .
  • the memory 230 may include data processed by the controller 240 and applications used for processing by the controller 240 .
  • applications used for processing by the controller 240 For example, one or more game applications executed by the controller 240 and execution result data of the game applications may be stored.
  • the controller 240 may control the components of the electronic device 200 as a whole. Also, the controller 240 may execute a game application by executing instructions stored in the memory 230 .
  • a user input for controlling execution of the game content may be received from a remote device controller that controls the electronic device 200.
  • the electronic device 200 may directly receive user input from the remote device controller that controls the electronic device 200 from the remote device controller, or the remote device controller may be connected to the display device 100 to display the user input from the remote device controller. It can also be received through 100.
  • the display device 100 may refer to a device capable of displaying image content, video content, game content, graphic content, and the like by having a display.
  • the display device 100 may output or display images or content received from the electronic device 200 .
  • the display device 100 may include various types of electronic devices capable of receiving and outputting content, such as, for example, network TV, smart TV, Internet TV, web TV, IPTV, PC, and the like.
  • the display device 100 may be referred to as a display device in that it receives and displays content, and may also be referred to as a content receiving device, a sink device, an electronic device, a computing device, and the like.
  • the display device 100 may include an input/output unit 110, a communication unit 120, a video processing unit 130, a display 140, an audio processing unit 150, an audio output unit 160, a memory 170, and a control unit 180.
  • the input/output unit 110 may receive a video signal and/or an audio signal from the electronic device 200 according to a connected protocol under the control of the controller 180 .
  • the communication unit 120 may include one or more modules enabling wireless communication between the display device 100 and a wireless communication system or between the display device 100 and a network where other electronic devices are located.
  • the communication unit 120 may receive a video signal and/or an audio signal received from the electronic device 200 according to a wireless communication protocol under the control of the control unit 180 .
  • the communication unit 120 may connect to the server computer 300 under the control of the controller 180 to transmit a request to the server computer 300 and receive a response to the request from the server computer 300 .
  • the video processing unit 130 may process and output an image signal received from the input/output unit 110 or the communication unit 120 to the display 140.
  • the display 140 may display the video signal received from the video processing unit 130 on the screen.
  • the audio processor 150 may convert an audio signal received from the input/output unit 110 or the communication unit 120 into an analog audio signal and output the analog audio signal to the audio output unit 160.
  • the audio output unit 160 may output a received analog audio signal through a speaker.
  • the memory 170 may store programs related to the operation of the display device 100 and various data generated during operation of the display device 100 .
  • the memory 170 analyzes the function of the display device 100 disclosed in the present disclosure, that is, the video screen received from the electronic device 200 to detect a content execution start time, and the video screen at which the content execution start time is detected.
  • One or more instructions for realizing a function of acquiring attribute information of content to be executed by analyzing and controlling a content execution environment based on the acquired attribute information of the content may be stored.
  • the controller 180 may control the overall operation of the display device 100 by executing one or more instructions stored in the memory 170 .
  • the controller 180 displays a screen of an image received from an electronic device connected to the display device 100 by executing one or more instructions stored in the memory 170 and analyzes the displayed image screen using a first neural network model. It is determined whether content execution starts, and when it is determined that content execution starts, a second neural network model is called, and a video screen of the content is analyzed using the second neural network model to obtain property information of the content, An execution environment of the content may be controlled based on the acquired attribute information.
  • the control unit 180 identifies the electronic device from HDMI CEC received through HDMI communication from the electronic device by executing one or more instructions stored in the memory 170, and the learned electronic device corresponding to the identified electronic device.
  • a first neural network model may be obtained.
  • the first neural network model receives a plurality of user interface (UI) screens, which can be received from the electronic device, as training data, and outputs the content executable UI screen and the content executable UI screen. It can include models that have been trained to classify.
  • UI user interface
  • the controller 180 executes one or more instructions stored in the memory 170, converts the displayed video screen into the first neural network model and analyzes it, and converts the content-executable UI screen to the content-executable UI screen. It is determined whether or not the content is executed, and it is determined that the execution of the content starts when it is determined that the content is executed from the UI screen capable of executing the content to the UI screen in which the content cannot be executed.
  • the second neural network model may include a model trained to receive a plurality of video screens as training data and detect a text area or a logo area from the video screens.
  • the controller 180 executes one or more instructions stored in the memory 170 to extract the text area or the logo area from the video screen by inputting the image screen to the second neural network model and analyzing the image screen, Attribute information of the content may be obtained based on the text area or the logo area.
  • the controller 180 transmits text extracted from the text area or the logo area to a server by executing one or more instructions stored in the memory 170, and the server transmits the content related to the text or the logo area. Attribute information of the content may be acquired by receiving attribute information.
  • the controller 180 executes one or more instructions stored in the memory 170 to set at least one of image quality setting, sound setting, viewing age limit setting, and display device environment setting suitable for the content based on the acquired attribute information. By controlling, it is possible to control the execution environment of the content.
  • control unit 180 executes one or more instructions stored in the memory 170 to control the execution environment of the content, then determines whether the execution of the content is terminated, and upon determining that the execution of the content is terminated, the first neural network By analyzing the video screen using the model, it may be determined whether a new content execution starts.
  • the server computer 300 may serve to receive a request from the display device 100 and provide a response corresponding to the received request.
  • the server computer 300 may include a communication unit 310, a memory 320, and a control unit 330.
  • the communication unit 310 may communicate with the display device through a wired or wireless communication method.
  • the memory 320 may include data processed by the controller 330 and applications used for processing by the controller 330 .
  • the memory 320 may store one or more programs that perform text recognition or image recognition.
  • the memory 320 may include a database storing attribute information about content.
  • the controller 330 may control the components of the server computer 300 as a whole. Also, the controller 330 may execute an application by executing instructions stored in the memory 320 .
  • the controller 330 receives data such as text, text area, logo image, etc. extracted from the video screen from the display device 100, and performs text recognition or image recognition based on the received data, thereby corresponding to the video screen. You can get text or image.
  • the control unit 330 retrieves content property information corresponding to the acquired text or image from a content property storage database, thereby obtaining text, text area, or content property information corresponding to a logo image received from the display device 100. can be obtained.
  • the content attribute information may include various metadata about the content, such as category, genre, and viewing age information of the content.
  • FIG. 3 is a block diagram illustrating a specific configuration of a display device according to an exemplary embodiment.
  • the display device 100 may include an input/output unit 110, a communication unit 120, a video processing unit 130, a display 140, an audio processing unit 150, an audio output unit 160, a memory 170, a control unit 180, and a sensing unit 190.
  • the input/output unit 110 transmits video (eg, motion picture, etc.), audio (eg, voice, music, etc.), and additional information (eg, EPG, etc.) from the outside of the display device 100 under the control of the controller 180.
  • the input/output unit 110 may include one of an HDMI port (High-Definition Multimedia Interface port), a component jack, a PC port, and a USB port, or may include a combination of one or more.
  • the input/output unit 110 may further include DisplayPort (DP), Thunderbolt, and Mobile High-Definition Link (MHL).
  • DP DisplayPort
  • Thunderbolt Thunderbolt
  • MHL Mobile High-Definition Link
  • the communication unit 120 may include one or more modules enabling wireless communication between the display device 100 and a wireless communication system or between the display device 100 and a network where other electronic devices are located.
  • the communication unit 120 may include a broadcast receiving module 121, a mobile communication module 122, a wireless Internet module 123, and a short-distance communication module 124.
  • the broadcast reception module 121 may include a module for receiving a broadcast signal.
  • the mobile communication module 122 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network.
  • the wireless signal may include a voice call signal, a video call signal, or various types of data according to text/multimedia message transmission/reception.
  • the wireless Internet module 123 refers to a module for wireless Internet access, and may be built into or external to a device.
  • Wireless Internet technologies include wireless LAN (WLAN) (WiFi), wireless broadband (Wibro), world interoperability for microwave access (Wimax), high speed downlink packet access (HSDPA), and the like.
  • Wi-Fi wireless LAN
  • Wibro wireless broadband
  • Wimax wireless broadband
  • HSDPA high speed downlink packet access
  • P2P Wi-Fi peer to peer
  • the short distance communication module 124 refers to a module for short distance communication.
  • Bluetooth Bluetooth Low Energy (BLE), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like may be used as short-distance communication technologies.
  • BLE Bluetooth Low Energy
  • RFID Radio Frequency Identification
  • IrDA Infrared Data Association
  • UWB Ultra Wideband
  • ZigBee ZigBee
  • the video processing unit 130, the display 140, and the audio processing unit 150 are as described above with reference to FIG. 2.
  • the audio output unit 160 may output audio (eg, voice, sound) input through the communication unit 120 or the input/output unit 110 . Also, the audio output unit 165 may output audio stored in the memory 170 under the control of the controller 180 .
  • the audio output unit 160 may include at least one of a speaker 161, a headphone output terminal 162, or a Sony/Philips Digital Interface (S/PDIF) output terminal 163, or a combination thereof.
  • S/PDIF Sony/Philips Digital Interface
  • the sensing unit 190 senses a user's voice, a user's video, or a user's interaction, and may include a microphone 191, a camera unit 192, and a light receiver 193.
  • the microphone 191 receives the user's utterance.
  • the microphone 191 may convert the received voice into an electrical signal and output it to the controller 180 .
  • the user's voice may include, for example, a voice corresponding to a menu or function of the display apparatus 100 .
  • the camera unit 192 may receive an image (eg, continuous frames) corresponding to a user's motion including a gesture within the camera recognition range.
  • the controller 180 may select a menu displayed on the display device 100 or perform control corresponding to the motion recognition result by using the received motion recognition result.
  • the light receiving unit 193 receives an optical signal (including a control signal) received from an external control device.
  • the light receiving unit 193 may receive an optical signal corresponding to a user input (eg, touch, pressure, touch gesture, voice, or motion) from the control device.
  • a control signal may be extracted from the received optical signal by control of the control unit 180 .
  • the memory 170 may include a content execution environment control module 171 , a first neural network model database 172 , and a second neural network model database 173 .
  • the content execution environment control module 171 may include one or more instructions for managing and controlling environment information of the display device 100 suitable for the content displayed on the display 130 .
  • the content execution environment control module 171 refers to the first neural network model database 172 and the second neural network model database 173 and analyzes the video screen received from the electronic device 200 to detect the start time of content execution, and after the start time of content execution It may include one or more instructions for acquiring content property information by analyzing a received video screen and controlling an execution environment of the content based on the obtained content property information.
  • control of a content execution environment refers to providing a better user experience when the display device 100 displays a video screen received from the electronic device 200 in accordance with the characteristics of the video screen displayed on the display device 100.
  • the first neural network model database 172 may include a plurality of first neural network models used to analyze an image received from the electronic device 200 and determine whether content execution starts.
  • the first neural network model is applied to each electronic device. It may be desirable to learn correspondingly.
  • the first neural network model database 172 may include a plurality of first neural network models learned to correspond to each electronic device of the plurality of electronic devices. In FIG.
  • the first neural network model database 172 includes a first neural network model 410 learned using the user interface screen of the first electronic device, a first neural network model 420 learned using the user interface screen of the second electronic device, and a third neural network model database 172 . It may include the first neural network model 430 learned using the user interface screen of the electronic device.
  • a first neural network model corresponding to the first game console device may be acquired by training the first neural network model using user interface screens output from the first game console device, with respect to the first game console device;
  • a first neural network model corresponding to the second game console device may be acquired by learning the first neural network model using user interface screens output from the second game console device with respect to the second game console device.
  • the second neural network model 1300 may represent a neural network model used to acquire attribute information of content by analyzing an image received from the electronic device 200 .
  • the neural network model included in the first neural network model database 172 may be stored in a memory when the display device 100 is manufactured, or may be downloaded from a server and stored after the display device 100 is manufactured. In addition, such a neural network model may be periodically or non-periodically updated through a server.
  • the memory 170 is shown as storing the first neural network model database 172 and the second neural network model database 173, but the first neural network model database 172 and the second neural network model database 173 are necessarily stored in the display device. It doesn't have to be stored at 100.
  • the first neural network model database 172 and the second neural network model database 173 are present in the server computer, and the display device 100 transmits a query referring to the first neural network model or a query referring to the second neural network model to the server computer and retrieves the query from the server computer. You may receive a response to
  • 4A is a reference diagram for explaining a method of learning a first neural network model corresponding to each electronic device according to an embodiment.
  • a plurality of different electronic devices may be connected to the display device 100, and the display device 100 may receive and display images from each of the different electronic devices.
  • Each electronic device may use a user interface environment provided by the electronic device itself.
  • the format of the main screen 411 provided by the first electronic device 200a, the format of the main screen 421 provided by the second electronic device 200b, and the main screen 431 provided by the third electronic device 200c are all different from each other.
  • the display device 100 uses the neural network model learned using the UI screen provided by the connected electronic device.
  • more accurate results can be obtained by analyzing the image received from the corresponding electronic device.
  • the first neural network model corresponding to the first electronic device 200a may be obtained by learning using UI screens 411 provided from the first electronic device 200 as training data.
  • the second neural network model 420 corresponding to the second electronic device 200b may be obtained by learning using UI screens 421 provided from the second electronic device 200 as training data.
  • the third neural network model 430 corresponding to the third electronic device 200c may be obtained by learning using UI screens 431 provided from the third electronic device 200c as training data.
  • 4B is a reference diagram for explaining a method of acquiring a first neural network model corresponding to a first electronic device according to an embodiment.
  • the neural network model 410 corresponding to the first electronic device may be obtained by learning using UI screens provided by the first electronic device. Specifically, the neural network model 410 corresponding to the first electronic device can be obtained by classifying and learning UI screens 412 incapable of executing content and UI screens 413 in which content can be executed among UI screens provided by the first electronic device.
  • the first electronic device may provide various UI screens to execute content.
  • the first electronic device may include a setting UI screen for changing settings of the first electronic device, a menu UI screen for displaying and selecting items of content executable on the first electronic device, 1 A screen displaying a logo of an electronic device, a logo screen of a selected content producer, and a logo screen of selected content may be provided.
  • the setting UI screen or the logo display screen cannot be operated for content execution, so it is classified as a non-content executable UI screen 412, and the menu UI screen can operate content execution, so the content execution UI is possible. It can be classified as screen 413.
  • the neural network model corresponding to the first electronic device includes UI screens tagged as the UI screen 413 capable of executing content and UI screens tagged as the UI screen 412 unable to execute content among the UI screens provided by the first electronic device.
  • UI screens tagged as the UI screen 413 capable of executing content
  • UI screens tagged as the UI screen 412 unable to execute content among the UI screens provided by the first electronic device.
  • the neural network model corresponding to the second electronic device receives UI screens tagged as UI screens capable of executing content and UI screens tagged as UI screens incapable of executing content among UI screens provided by the second electronic device.
  • a probability of a UI screen in which content can be executed or a probability of a UI screen in which content cannot be executed can be output as a result.
  • a neural network model that outputs the category of the input UI screen as a result by receiving and learning tagged UI screens may use a Deep Neural Network (DNN) or the like.
  • DNN Deep Neural Network
  • FIG. 5 is a flowchart illustrating an example of a method of operating a display device according to an exemplary embodiment.
  • the display device 100 may display a screen of an image received from a connected electronic device.
  • the display device 100 may receive a video screen from the electronic device 200 after being wired or wirelessly connected to the electronic device 200 and display the received video screen on the display.
  • the video screen received by the display device 100 from the electronic device 200 may include, for example, a setting UI screen, a menu UI screen, a logo display screen, and an actual content video screen.
  • the display device 100 may determine whether content execution starts by analyzing the displayed video screen using the first neural network model.
  • the first neural network model may indicate a neural network model trained to determine whether the input screen is a content executable UI screen or a content nonexecutable UI screen by receiving and analyzing a displayed video screen. Specifically, the first neural network model receives and analyzes a video screen and outputs at least one of the probability that the input screen is a UI screen capable of executing content or the probability that the screen is a non-executable UI screen, so that the input screen is a UI screen capable of executing content. It can be determined whether it is a non-executable UI screen. For example, if the first neural network model analyzes the video screen and the probability that the video screen is a UI screen capable of executing content is 90% or more, it may be determined that the video screen is a UI screen capable of executing content.
  • the display device 100 may determine that content execution starts when the displayed video screen is switched from a content executable UI screen to a content executable UI screen. For example, the display device 100 may determine that content execution starts when the result of image analysis through the first neural network model is converted from a UI screen capable of executing content to a UI screen incapable of executing content.
  • the display device 100 may determine whether content execution starts by analyzing the displayed video screen using the first neural network model learned in correspondence with the connected electronic device 200 .
  • the display apparatus 100 determines whether content execution starts by analyzing the displayed video screen, and when it is not determined that content execution starts, the display device 100 may continuously analyze the video screen.
  • the display device 100 determines whether content execution starts by analyzing the displayed video screen, and when it is determined that content execution starts, it may proceed to operation 530.
  • the display device 100 may call a second neural network model.
  • the second neural network model may indicate a neural network model learned to extract a text area or a logo area from which attribute information of content may be derived from an input screen by receiving and analyzing a video screen.
  • the display device 100 may obtain attribute information of the content by analyzing the video screen using the second neural network model.
  • the display device 100 may acquire text or a logo from a text area or a logo area of an image screen by using a second neural network model, and obtain property information of content using the thus obtained text or logo. there is.
  • the display device 100 transmits the text or logo extracted from the text area or logo area of the video screen or the text area or logo area to the server using the second neural network model, and the server transmits the text or logo that is matched to the text or logo.
  • Attribute information of the content may be obtained.
  • Attribute information of the content may include metadata including a title of the content, a category of the content, and the like.
  • the display device 100 may control an execution environment of a content image based on the obtained content attribute information.
  • the display device 100 may set a picture quality suitable for the corresponding content, set a sound suitable for the corresponding content, or provide a viewing age restriction service suitable for the corresponding content, based on attribute information of the content.
  • FIG. 6 is a reference diagram for explaining the operating method shown in FIG. 5 according to an embodiment.
  • the display device 100 may receive an image 600 output from the electronic device 200 from the electronic device 200.
  • the display device 100 may receive the image 600 in frame units.
  • the display device 100 may analyze the received images and operate in the content execution time determination mode 610 until content execution start is detected.
  • the display device 100 may perform the content execution timing determination operation 630 using the first neural network model. For example, when the display device 100 inputs the received image frame to the first neural network model and analyzes it, it is determined that the k-1th frame is a content executable UI screen and the kth frame is a content executable UI screen. , the display device 100 may detect (640) that content execution has started. The display device 100 may analyze the frames received from the electronic device 200 by sampling them every frame or at regular time intervals, for example, at 100 ms intervals, until the content execution time is detected.
  • the display device 100 may call the second neural network model and perform the content property recognition operation 650 without using the first neural network model any longer.
  • the display device 100 may maintain the content property recognition mode 620 in which the content property recognition operation 650 is performed using the second neural network model until content property recognition succeeds.
  • the display device 100 may sample and analyze frames received from the electronic device 200 at regular intervals until content property recognition succeeds.
  • the display device 100 detects a text area or logo area from the video screen using a second neural network model, and obtains content property information based on the text or logo extracted from the detected text area or logo area. can do.
  • the display device 100 succeeds in recognizing the content property by analyzing the image using the second neural network model (660), it can control the execution environment of the content based on the content property (670).
  • FIG. 7 is a flowchart illustrating a process of a method of obtaining a first neural network model in a display device according to an embodiment.
  • the electronic device 200 and the display device 100 may be connected.
  • the display device 100 and the electronic device 200 may be connected through wired communication.
  • the display device 100 and the electronic device 200 may be connected through an HDMI cable, and at this time, the input/output unit 110 of the display device 100 and the input/output unit 210 of the electronic device 100 may communicate according to an HDMI communication protocol.
  • the display device 100 and the electronic device 200 may be connected through wireless communication.
  • the display device 100 and the electronic device 200 may be connected through wireless communication such as Bluetooth, BLE, ZigBee, Wi-Fi, etc.
  • the display device 100 and the electronic device 200 may communicate according to each communication protocol.
  • the display device 100 may receive device information of the electronic device 200 from the electronic device 200.
  • the display device 100 may receive device information about the electronic device 200 using an HDMI-CEC or HDMI Info frame.
  • the device information may include at least one of the device type, manufacturer, business name, and model name of the electronic device 200, but is not limited thereto.
  • the display device 100 may receive device information about the electronic device 200 according to the connected wireless communication protocol.
  • the device information may include at least one of the device type, manufacturer, business name, and model name of the electronic device 200, but is not limited thereto.
  • the display device 100 may identify the electronic device 200 from device information received from the electronic device 200.
  • the display device 100 may obtain a first neural network model corresponding to the identified electronic device 200.
  • the display device 100 may include a first neural network model database 172 including first neural network models learned for each of one or more electronic devices connectable to the display device 100 .
  • the display device 100 may acquire a neural network model corresponding to the identified electronic device from the first neural network model database 172 .
  • the display device 100 may obtain a first neural network model corresponding to the first electronic device.
  • FIG. 8 is a flowchart illustrating a process of a method of detecting a content execution start time in the display device 100 according to an embodiment. The operation shown in FIG. 8 may be performed after the operation shown in FIG. 7 .
  • the electronic device 200 may transmit an image to the display device 100.
  • the display device 100 may display the image received from the electronic device 200 on the screen and analyze the displayed image screen using the first neural network model.
  • the first neural network model may be obtained through, for example, the operation shown in FIG. 7 .
  • the display device 100 may determine whether the content executable UI screen is switched to the content executable UI screen according to the video screen analysis.
  • the display device 100 analyzes the image received from the electronic device 200 frame by frame. By inputting the received image frame to the first neural network model, it is possible to determine whether the input image frame is a UI screen in which content can be executed or a UI in which content cannot be executed. .
  • the display device 100 may determine whether the video screen is switched from a UI screen capable of executing content to a UI screen incapable of executing content, by using a result output from the first neural network model. If a content executable UI screen is followed by a content executable UI screen, it may be determined that the user selects a certain content from the content executable UI screen and the corresponding content is executed, that is, content execution has started.
  • FIG. 9 is a reference diagram for explaining a method of determining a content execution time according to an example.
  • the display device 100 sequentially receives video screens, that is, a setting UI 910, a menu UI screen 920, and a device logo display screen 930 from an electronic device 200 connected to the display device 100.
  • the display device 100 inputs the setting UI screen 910 received from the electronic device 200 to the first neural network model 400 corresponding to the electronic device 200, and thus obtains a result that the corresponding input screen is a content-executable UI screen.
  • the display device 100 then inputs the received menu UI screen 920 to the first neural network model 400 corresponding to the electronic device 200, thereby obtaining a result that the corresponding input screen is a content executable UI screen.
  • the transition from the setting UI screen 910 to the menu UI screen 920 is a transition from a content non-executable UI screen to a content executable UI screen, it may be determined that the condition of operation 830 is not satisfied.
  • the display device 100 inputs the received device logo output screen 930 to the first neural network model 400 corresponding to the electronic device 200, thereby obtaining a result that the corresponding input screen is a content-executable UI screen.
  • the transition from the menu UI screen 920 to the device logo display screen 930 is a transition from a content executable UI screen to a content non-executable UI screen, it can be determined that the condition of operation 830 is satisfied.
  • step 820 to analyze the video received from the electronic device 200.
  • operation 840 may be performed.
  • the display device 100 may determine that the content execution start time has been detected.
  • the display device 100 may call a second neural network to recognize the executed content according to the detection of content execution start time.
  • the second neural network may represent a neural network trained to detect a text area or a logo area by analyzing an image screen.
  • FIG. 10 illustrates an example of a neural network model for classifying an image screen according to an exemplary embodiment.
  • a deep neural network may perform learning through training data.
  • the trained deep neural network may perform an inference operation, which is an operation for object recognition.
  • the deep neural network can be designed in a variety of ways according to a model implementation method (eg, CNN (Convolution Neural Network), etc.), result accuracy, result reliability, processor processing speed and capacity, and the like.
  • FIG. 10 is a diagram showing the structure of a convolutional neural network according to an embodiment.
  • the convolutional neural network 1000 has a structure in which an input image 1010 is input, and output data 1030 is output after passing through N convolutional layers 1020 .
  • the convolutional neural network 1000 may be a deep convolutional neural network including two or more convolutional layers.
  • the display device 100 may extract "features" such as a border, a line, and a color from an input image using the convolutional neural network 1000 .
  • Each of the N convolution layers 1020 included in the convolution neural network 1000 may receive data, process the received data, and generate output data.
  • the neural network may generate a first feature map 1021 by convolving an image input to a convolution layer with one or more kernels or filters.
  • the generated first feature map is subsampled to obtain a second feature map 1022, the second feature map 1022 is input to the second convolution layer, and the second feature map input from the second convolution layer is one.
  • a third feature map 1023 may be generated by convolution with the above kernels or filters.
  • the initial convolutional layers of the convolutional neural network 1000 may be operated to extract low-level features such as edges or gradients from an input image. As the later convolutional layers progress, more complex features can be extracted.
  • One or more convolutional layers that receive and output feature maps within the convolutional neural network 1000 may be hidden layers (eg, hidden convolutional layers).
  • other processing operations may be performed in addition to an operation of convolving by applying one or more kernels to a feature map.
  • operations such as an activation function and pooling may be performed.
  • the image processing device may apply an activation function to change the values of the feature map extracted as a result of performing the convolution operation into a non-linear value of "existence or absence" of the feature of the content executable UI screen.
  • the ReLu function may be used, but is not limited thereto.
  • FIG. 11 is a reference diagram for explaining a method of recognizing content after a content execution start point in a display device according to an exemplary embodiment.
  • the display device 100 may analyze a content video screen using a second neural network model. According to an example, the display device 100 may be acquired according to the operation shown in FIG. 8 .
  • the second neural network model is a neural network used to detect one or more objects from an input image.
  • two stage methods such as Faster R-CNN, R_FCN [Region-based Fully Convolutional Networks] and FPN-FRCN.
  • single stage methods algorithms such as YOLO, SSD [Single Shot Mutibox Detector], or RetinaNet.
  • the second neural network model includes an object detection model that detects an object including text or a logo from an input screen by learning a plurality of input images including text and a plurality of input images including logo images. can do.
  • FIG 12 illustrates an example of an object detection model according to an example.
  • the object detection model may detect one or more objects from an input image using one or more neural networks, and output object information including object classes and object positions corresponding to the one or more detected objects.
  • object detection determines where objects are located in a given image (object localization) and determines which category each object belongs to (object classification). Therefore, object detection models generally have three steps: object candidate region selection (informative region selection), feature extraction from each candidate region (feature extraction), and object candidate region classification by applying a classifier to the extracted features. can be rough. Depending on the detection method, localization performance can be improved through post-processing such as bounding box regression.
  • R-CNN which is an object detection method combining region proposal and CNN according to an example of an object detection model, is shown.
  • an object detection model 1200 may include a region proposal module 1210, a CNN 1220, a classifier module 1230, and a bounding box regression module 1240.
  • a region proposal module 1210 extracts a candidate region from the input image 200 .
  • the number of candidate regions may be a certain number, for example, 2000.
  • R-CNN uses selective-search, one of region proposal algorithms.
  • a Convolutional Neural Network (CNN) 1220 extracts a fixed-length feature vector from the region generated by the region proposal module 1210. Since CNNs (eg. AlexNet, VggNet, etc.) receive inputs of a certain size, it is necessary to warp the various rectangular-shaped regions given by the region proposal algorithm for the image to a certain size regardless of size or aspect ratio. . The CNN receives the warped region and extracts the result of the layer before the classifier module.
  • CNNs eg. AlexNet, VggNet, etc.
  • the classifier module 1230 receives a fixed-length feature vector as an input and performs classification. For example, the classifier module 1230 may classify whether an object corresponds to text or a logo.
  • the bounding-box regression module 1240 receives a fixed-length feature vector as an input and calculates four numbers (x, y, w, h) representing a box.
  • the position of the object can be specified by four numbers (x, y, w, h) representing the box.
  • R-CNN performs object detection by performing localization of an object through region proposal extraction and recognizing an object class through classification of extracted features.
  • a process of reducing localization errors may be performed by performing bounding box regression.
  • the training of the object detection model 1200 is to transform the pre-learned CNN to suit the object detection task, and the classification layer (eg, the output layer) in the existing pre-learned CNN is newly selected as "object's object” for object detection. number + background", and weight initialization is performed only for that part.
  • the object information 1250 includes information on one or more objects, and each object information may be displayed as (object class, location).
  • object class may represent logo or text.
  • the display device 100 may determine whether a text area or a logo area is extracted from the content video screen.
  • operation 1110 may be performed to analyze the next screen.
  • operation 1130 may be performed.
  • the display device 100 may obtain content property information based on the detected logo text area or logo area.
  • the display device 100 may extract text from the text area and obtain content attribute information based on the extracted text.
  • the display device 100 may extract text from the text area using a technology such as OCR (Optical Character Recognition).
  • OCR Optical Character Recognition
  • the display device 100 may transmit text extracted from the text area to the server 300 that manages information about contents, and may receive attribute information of the contents corresponding to the text from the server 300 .
  • the server receives text from the display device 100 and searches for content corresponding to the text to find information on the corresponding content
  • the server receives information about the content, for example, content category, genre, and viewing age possibility information. Attribute information such as, etc. may be extracted and the extracted attribute information of content may be transmitted to the display device 100 .
  • the server 300 may transmit a search failure result to the display device 100.
  • FIG. 13 is a reference diagram for explaining a process of obtaining property information of content by analyzing a video screen according to an example.
  • the display device 100 sequentially receives video screens 930 to 980 from the electronic device 200 connected to the display device 100 .
  • the display device 100 inputs the image screen 930 received from the electronic device 200 to the second neural network model 1300 trained to detect an object region, for example, a text region or a logo region, from the image, thereby detecting the logo region from the corresponding video screen 930. 931 can be obtained.
  • the display device 100 may then transmit the detected logo area image to the server 300 .
  • the server 300 may perform an operation such as image search using the logo area image received from the display device 100 to analyze what content the corresponding logo area image is related to. If the logo area image 931 is a logo image related to a specific electronic device rather than a specific content, the server may transmit a search failure result to the display device 100 .
  • the display device 100 Since the display device 100 receives a search failure result from the server 300, it continuously analyzes the image received from the electronic device 200.
  • the display device 100 inputs the video screen 940 received from the electronic device 200 to the second neural network model 1300. Since the video screen 940 is a black screen, the second neural network model can output that no object has been detected as an object detection result. there is.
  • the display device 100 may obtain a logo area detection result 951 from the corresponding video screen 950 by inputting the video screen 950 received from the electronic device 200 to the second neural network model 1300 .
  • the display device 100 may then transmit the detected logo area image 951 to the server 300 .
  • the server 300 may perform an operation such as image search using the logo area image received from the display device 100 to analyze what content the corresponding logo area image is related to. If the logo area image 951 is a logo image related to a specific content producer rather than a specific content, the server may transmit a search failure result to the display device 100 .
  • the display device 100 inputs the video screen 960 received from the electronic device 200 to the second neural network model 1300. Since the video screen 960 is a black screen, the second neural network model can output that no object has been detected as an object detection result. there is.
  • the display device 100 may obtain a text area detection result 971 from the corresponding video screen 970 by inputting the video screen 970 received from the electronic device 200 to the second neural network model 1300 .
  • the display device 100 may then extract text from the text area image 971 by analyzing the detected text area image 971 using a technology such as OCR.
  • the display device 100 may transmit the extracted text to the server 300 .
  • the server 300 may search for content corresponding to the text received from the display device 100 . As a result of searching for content corresponding to the text, the server 300 may transmit attribute information about the searched content to the display device 100 .
  • the display device 100 extracts text from the text area image and transmits the extracted text to the server 300, it is not limited thereto.
  • the display device 100 may transmit the entire text area image to the server 300, and the server 300 may extract text from the text area image extracted from the display device 100.
  • the display device 100 may control the content execution environment based on the property information of the content obtained in operation 1130. Controlling the execution environment of content may include setting image quality or sound suitable for content properties, or providing a user viewing age restriction service.
  • the display apparatus 100 may set picture quality or sound according to the genre or category of content.
  • the display device 100 may output an appropriate guide message, output a message requesting password input, or control viewing restriction based on the viewable age information.
  • FIG. 14 illustrates an example of a table including picture quality setting values and sound setting values set according to genres of content according to an embodiment.
  • the display device 100 may store a picture quality/sound setting table 1400 including picture quality setting values and sound setting values set differently according to the genre of the game content. .
  • the picture quality/sound setting table 1400 includes, for example, general basic 1410 as a genre of game content, RPG (Role Playing Game) 1420, a role-playing game that users enjoy playing characters in the game, player's point of view, and my point of view.
  • RPG Role Playing Game
  • RTS Real-time strategy, real-time strategy game
  • the display device 100 may map different image quality values according to attributes of game contents of each genre included in the table 1400 .
  • the 1st quality value for the basic genre 1410, the 2nd quality value for the RPG genre 1420, the 3rd quality value for the FPS genre 1430, the 4th quality value for the RTS genre 1440, and the 5th quality value for the sports genre 1450 can be mapped.
  • first-person shooter games such as FPS require higher realism than other games because the character's point of view and the player's point of view in the game must be the same. You can set 3 picture quality values.
  • the display device 100 may map different sound values according to the properties of the game content of each genre included in the table 1400. 1st sound value for basic genre 1410, 2nd sound value for RPG genre 1420, 3rd sound value for FPS genre 1430, 4th sound value for RTS genre 1440, 5th sound value for sports genre 1450 can be mapped.
  • the display device 100 refers to the table 1400, and when it is recognized that the genre of the game content is RPG as content attribute information, the display device 100 extracts a second picture quality value and a second sound value corresponding to the RPG genre, , the image quality of the display device 100 and the sound may be set according to the second picture quality value and the second sound value.
  • 15 is a reference diagram for explaining an operation of a display device when viewing age information is received as content property information according to an exemplary embodiment.
  • the display device 100 may provide a viewing age restriction service based on the received viewing age possibility information. For example, when the display device 100 receives information indicating that the viewing age of the content is 19 years old as content property information, the display device 100 may output the user interface 1500 as shown in FIG. 15 . .
  • User interface 1500 reads "The content displayed is for those over the age of 19. To continue viewing, please enter your password for adult authentication! An input window for entering a 4-digit adult authentication password may be provided. The user can display the content by inputting a password in the input window of the user interface 1500.
  • 16 is a reference diagram for describing switching between a content execution timing determination mode and a content property recognition mode according to an exemplary embodiment.
  • the display device 100 may determine a content execution start time using a first neural network model corresponding to an electronic device connected to the display device 100 in a content execution time determination mode 610 .
  • the display device 100 may enter 1610 the content property recognition mode 620 to obtain property information of the executed content.
  • the display device 100 extracts a logo area or text area from the video screen by analyzing the video screen received from the electronic device 200 using the second neural network model in the content property recognition mode 620, and based on the extracted logo area or text area, Attribute information of content being executed can be obtained. According to the attribute information of the content thus obtained, the content execution environment may be controlled and the content execution timing determination mode 610 may be entered 1620 again.
  • the display device 100 may control the content execution environment according to the attribute information of the content thus obtained and enter 630 into a content execution termination determination mode 630 .
  • the content execution termination determination mode 630 it is possible to detect whether content execution is terminated by analyzing a video screen received from the electronic device 200 and using a third neural network model trained to detect a screen indicating the content execution termination.
  • the display device 100 may enter 1640 the content execution timing determination mode 610 to monitor whether execution of the next new content starts.
  • Detecting the start of content execution in the content execution timing determination mode 610 or detecting content properties in the content property recognition mode 620 is performed in a relatively short time period, so the video screen to be analyzed is sampled at shorter time intervals. An analysis may be appropriate.
  • determining the end of content execution in the content execution termination determination mode 630 may be generally performed during a relatively long time period, that is, while content execution continues. Therefore, in this case, it may be possible to sample and analyze an image screen to be analyzed at a longer time interval.
  • Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • the disclosed embodiments may be implemented as a S/W program including instructions stored in computer-readable storage media.
  • a computer is a device capable of calling instructions stored in a storage medium and performing operations according to the disclosed embodiments according to the called instructions, and may include electronic devices according to the disclosed embodiments.
  • a computer-readable storage medium may be provided in the form of a non-transitory storage medium.
  • 'non-temporary' only means that the storage medium does not contain a signal and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.
  • control method according to the disclosed embodiments may be provided by being included in a computer program product.
  • Computer program products may be traded between sellers and buyers as commodities.
  • a computer program product may include a S/W program and a computer-readable storage medium in which the S/W program is stored.
  • the computer program product may include a product (eg, downloadable app) in the form of a S/W program that is distributed electronically through a device manufacturer or an electronic market (eg, Google Play Store, App Store).
  • a product eg, downloadable app
  • the storage medium may be a storage medium of a manufacturer's server, an electronic market server, or a relay server temporarily storing SW programs.
  • a computer program product may include a storage medium of a server or a storage medium of a device in a system composed of a server and a device.
  • the computer program product may include a storage medium of the third device.
  • the computer program product may include a S/W program itself transmitted from the server to the device or the third device or from the third device to the device.
  • one of the server, the device and the third apparatus may execute the computer program product to perform the method according to the disclosed embodiments.
  • two or more of the server, the device, and the third device may execute the computer program product to implement the method according to the disclosed embodiments in a distributed manner.
  • a server may execute a computer program product stored in the server to control a device communicatively connected to the server to perform a method according to the disclosed embodiments.
  • the third apparatus may execute a computer program product to control a device communicatively connected to the third apparatus to perform a method according to the disclosed embodiment.
  • the third device may download the computer program product from the server and execute the downloaded computer program product.
  • the third device may perform the method according to the disclosed embodiments by executing a computer program product provided in a preloaded state.
  • unit may be a hardware component such as a processor or a circuit, and/or a software component executed by the hardware component such as a processor.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Human Computer Interaction (AREA)
  • Medical Informatics (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

Selon des modes de réalisation, un dispositif électronique et un procédé de fonctionnement associé sont divulgués. Le dispositif d'affichage divulgué comprend : une unité d'affichage ; une interface d'entrée/sortie ; une interface de communication ; une mémoire dans laquelle une ou plusieurs instructions sont stockées ; et un processeur pour exécuter la ou les instructions stockées dans la mémoire. En exécutant la ou les instructions, le processeur : affiche un écran d'image reçu en provenance d'un dispositif électronique connecté ; détermine si l'exécution de contenu commence en analysant l'écran d'image affiché à l'aide d'un premier modèle de réseau neuronal ; appelle un second modèle de réseau neuronal s'il est déterminé que l'exécution de contenu commence ; acquiert des informations d'attribut concernant le contenu en analysant l'écran d'image du contenu à l'aide du second modèle de réseau neuronal ; et commande l'environnement d'exécution du contenu sur la base des informations d'attribut acquises.
PCT/KR2022/016054 2021-10-20 2022-10-20 Dispositif d'affichage et son procédé de fonctionnement Ceased WO2023068849A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/640,479 US12537996B2 (en) 2021-10-20 2024-04-19 Display device and operation method thereof

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020210140486A KR20230056452A (ko) 2021-10-20 2021-10-20 디스플레이 장치 및 그 동작 방법
KR10-2021-0140486 2021-10-20

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/640,479 Continuation US12537996B2 (en) 2021-10-20 2024-04-19 Display device and operation method thereof

Publications (1)

Publication Number Publication Date
WO2023068849A1 true WO2023068849A1 (fr) 2023-04-27

Family

ID=86059536

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2022/016054 Ceased WO2023068849A1 (fr) 2021-10-20 2022-10-20 Dispositif d'affichage et son procédé de fonctionnement

Country Status (2)

Country Link
KR (1) KR20230056452A (fr)
WO (1) WO2023068849A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025014073A1 (fr) * 2023-07-12 2025-01-16 삼성전자주식회사 Dispositif électronique et son procédé de commande

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20250123596A (ko) * 2024-02-08 2025-08-18 삼성전자주식회사 디스플레이 및 그 제어 방법

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150076629A (ko) * 2013-12-27 2015-07-07 삼성전자주식회사 디스플레이 장치, 서버 장치 및 이들을 포함하는 디스플레이 시스템과 그 컨텐츠 제공 방법들
KR20190031032A (ko) * 2017-09-15 2019-03-25 삼성전자주식회사 컨텐트를 실행하는 방법 및 장치
KR20190125095A (ko) * 2018-04-27 2019-11-06 삼성전자주식회사 컨텐츠에 부가 정보를 표시하는 방법 및 디바이스
KR20200037602A (ko) * 2018-10-01 2020-04-09 주식회사 한글과컴퓨터 인공 신경망 선택 장치 및 방법
KR20200072456A (ko) * 2018-06-20 2020-06-22 라인플러스 주식회사 이미지에서 추출된 키워드를 이용하여 이미지를 필터링하기 위한 방법과 시스템 및 비-일시적인 컴퓨터 판독 가능한 기록 매체

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150076629A (ko) * 2013-12-27 2015-07-07 삼성전자주식회사 디스플레이 장치, 서버 장치 및 이들을 포함하는 디스플레이 시스템과 그 컨텐츠 제공 방법들
KR20190031032A (ko) * 2017-09-15 2019-03-25 삼성전자주식회사 컨텐트를 실행하는 방법 및 장치
KR20190125095A (ko) * 2018-04-27 2019-11-06 삼성전자주식회사 컨텐츠에 부가 정보를 표시하는 방법 및 디바이스
KR20200072456A (ko) * 2018-06-20 2020-06-22 라인플러스 주식회사 이미지에서 추출된 키워드를 이용하여 이미지를 필터링하기 위한 방법과 시스템 및 비-일시적인 컴퓨터 판독 가능한 기록 매체
KR20200037602A (ko) * 2018-10-01 2020-04-09 주식회사 한글과컴퓨터 인공 신경망 선택 장치 및 방법

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025014073A1 (fr) * 2023-07-12 2025-01-16 삼성전자주식회사 Dispositif électronique et son procédé de commande

Also Published As

Publication number Publication date
US20240267592A1 (en) 2024-08-08
KR20230056452A (ko) 2023-04-27

Similar Documents

Publication Publication Date Title
WO2020101453A1 (fr) Dispositif électronique et procédé de reconnaissance d'une scène audio
WO2021261836A1 (fr) Appareil de détection d'image et procédé de fonctionnement de celui-ci
WO2018026162A1 (fr) Appareil électronique et son procédé de fonctionnement
WO2019054791A1 (fr) Procédé et appareil d'exécution de contenu
WO2015194693A1 (fr) Dispositif d'affichage de vidéo et son procédé de fonctionnement
WO2014038828A1 (fr) Procédé et appareil d'exécution d'une application dans un dispositif
EP3430811A1 (fr) Appareil électronique et son procédé de fonctionnement
WO2019198951A1 (fr) Dispositif électronique et procédé de fonctionnement de celui-ci
WO2020256185A1 (fr) Dispositif d'affichage
WO2020111567A1 (fr) Dispositif électronique et procédé d'utilisation associé
WO2019135433A1 (fr) Dispositif d'affichage et système comprenant ce dernier
WO2023068849A1 (fr) Dispositif d'affichage et son procédé de fonctionnement
WO2021060575A1 (fr) Serveur à intelligence artificielle et procédé de fonctionnement associé
WO2022045613A1 (fr) Procédé et dispositif d'amélioration de la qualité vidéo
WO2023017989A1 (fr) Appareil d'affichage et procédé de fonctionnement de celui-ci
WO2017160062A1 (fr) Procédé et dispositif de reconnaissance de contenu
WO2022050433A1 (fr) Dispositif d'affichage pour régler la sensibilité de reconnaissance d'un mot de départ de reconnaissance de la parole et son procédé de fonctionnement
WO2022131432A1 (fr) Dispositif électronique et son procédé de commande
WO2021060570A1 (fr) Appareil électroménager et serveur
WO2020230923A1 (fr) Dispositif d'affichage permettant de fournir un service de reconnaissance de la parole, et son procédé de fonctionnement
WO2020235724A1 (fr) Dispositif d'affichage
WO2024177352A1 (fr) Dispositif d'affichage et son procédé de fonctionnement
WO2019203370A1 (fr) Dispositif multimédia et procédé de commande de celui-ci
WO2025254240A1 (fr) Dispositif d'affichage et son procédé de fonctionnement
WO2025192838A1 (fr) Dispositif d'affichage et procédé de fonctionnement associé

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22884071

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22884071

Country of ref document: EP

Kind code of ref document: A1