KR102801405B1 - Electronic device, system and control method thereof - Google Patents

Abstract

An electronic device is disclosed. The electronic device includes a camera, a display, a communication interface including circuitry, and a processor controlling the display to display a live image received through the camera and controlling the communication interface to transmit the displayed live image to an external display device.

Description

{ ELECTRONIC DEVICE, SYSTEM AND CONTROL METHOD THEREOF }

The present invention relates to an electronic device, a system and a control method therefor, and more particularly, to an electronic device including a camera, a system and a control method therefor.

Recently, with the advancement of electronic technology, various types of electronic devices are being developed and distributed. In particular, electronic devices are being developed and distributed to reduce physical discomfort for users who are experiencing such discomfort.

An electronic magnifier is an electronic device that is necessary for many visually impaired users, as it enlarges objects and text to a size that can be recognized by users with low vision who have difficulty recognizing objects or text.

However, since conventional electronic magnifiers have integrated cameras and displays for recognizing objects, letters, etc., there was the inconvenience of having to move the object to be magnified to the position of the camera equipped in the electronic magnifier.

Additionally, there was a limitation that it was impossible to enlarge objects, letters, etc. beyond the size of the display equipped with the electronic magnifier.

Accordingly, there was a demand for a magnifier that conveniently enlarges and displays an object without having to move the object to be enlarged.

The present disclosure is in accordance with the aforementioned necessity, and an object of the present disclosure is to provide an electronic device, system, and control method for easily photographing an object to be enlarged using an electronic device equipped with a camera and displaying the enlarged object and letters using a relatively large screen.

According to one embodiment of the present disclosure to achieve the above-described purpose, the electronic device includes a camera, a display, a communication interface including a circuit, and a processor controlling the display to display a live image received through the camera and controlling the communication interface to transmit the displayed live image to an external display device, wherein the processor controls the display to display a UI guiding a change in a shooting direction of the camera on the live image when a text area including text in the live image is located in an edge area of the live image, and controls the communication interface to transmit the live image including the UI to the external display device.

Here, the processor can control the display to enlarge and display the text area included in the live image so that the text area is greater than a threshold ratio within a screen of the display, and control the communication interface to transmit the live image including the enlarged text area to the external display device.

Additionally, the processor can control the display to display a UI for setting a magnification ratio, and control the display to display the text area by enlarging it based on a magnification ratio set through the UI for setting the magnification ratio.

In addition, the electronic device further includes a speaker, and the processor can control the speaker to identify the text included in the text area, apply TTS (Text to Speech) to the identified text to obtain voice information corresponding to the text, and output sound based on the voice information.

Additionally, the processor can control the communication interface to transmit the acquired voice information to the external display device.

Additionally, the processor can control the communication interface to identify the text included in the text area and transmit information including the identified text to the external display device.

In addition, the processor may control the display to identify the text included in the text area and display an edit UI including the identified text when an edit mode is executed according to a user command, control the display to display the edited text when editing is performed on the text on the edit UI, and control the communication interface to transmit a screen image of the display including the edited text to the external display device.

Additionally, the processor can control the communication interface to stop transmitting the live image and transmit the captured image to the external display device when the displayed live image is captured according to a user command while the live image is being transmitted to the external display device.

Additionally, the external display device may be a device having a display having a screen size larger than that of the display.

According to one embodiment of the present disclosure to achieve the above-described purpose, a system including an electronic device and a display device includes an electronic device which, when a text area including text in a received live image received through a camera is located in an edge area of the live image, displays the live image including a UI that guides a change in a shooting direction of the camera, and transmits the live image including the UI to the display device, and a display device which displays the live image including the UI received from the electronic device.

Here, the display device can display the text area included in the live image by enlarging it so that the text area is greater than a threshold ratio within the screen of the display device.

In addition, the display device can display a UI for setting a magnification ratio, and display the text area by enlarging it based on the magnification ratio set through the UI for setting the magnification ratio.

In addition, the display device can identify an output layout based on the identified text when the text included in the text area is identified, and display the text according to the identified output layout.

According to one embodiment of the present disclosure for achieving the above-described purpose, a method for controlling an electronic device includes the steps of displaying a live image received through a camera, displaying a UI guiding a change in a shooting direction of the camera on the live image when a text area including text in the live image is located in an edge area of the live image, and transmitting the live image including the UI to an external display device.

Here, the step of displaying the live image may include the step of displaying the text area included in the live image by enlarging the text area so that the text area is greater than a threshold ratio within the screen, and the step of transmitting may include the step of transmitting the live image including the enlarged text area to the external display device.

In addition, the control method may further include a step of displaying a UI for setting a magnification ratio, and the step of displaying the live image may include a step of displaying the text area by enlarging it based on a magnification ratio set through the UI for setting the magnification ratio.

In addition, the control method may further include a step of identifying the text included in the text area, a step of applying TTS (Text to Speech) to the identified text to obtain voice information corresponding to the text, and a step of outputting a sound based on the voice information.

Here, the control method may further include a step of transmitting the acquired voice information to the external display device.

In addition, the control method may further include a step of identifying the text included in the text area and displaying an edit UI including the identified text when an edit mode is executed according to a user command, a step of displaying the edited text when editing is performed on the text on the edit UI, and a step of transmitting a screen image of the display including the edited text to the external display device.

Additionally, the transmitting step may include a step of stopping the transmission of the live image and transmitting the captured image to the external display device when the displayed live image is captured according to a user command while the live image is being transmitted to the external display device.

According to various embodiments of the present disclosure, an object to be enlarged can be easily captured using a camera equipped in a portable electronic device.

Text contained in a live image can be enlarged and displayed to the user.

It can reduce the misrecognition rate of text and apply TTS to text to output sound corresponding to the text.

It is possible to convert text-based information contained in an information medium into sound and output it, so that people with low vision or the visual impairment can easily acquire the information through sound.

FIG. 1 is a drawing schematically illustrating the configuration of a system according to one embodiment of the present disclosure.
FIG. 2 is a drawing schematically illustrating the configuration of an electronic device according to an embodiment of the present disclosure.
FIG. 3 is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure.
FIG. 4 is a drawing for explaining a text area according to one embodiment of the present disclosure.
FIG. 5 is a drawing for explaining an enlarged text area according to one embodiment of the present disclosure.
FIG. 6 is a drawing for explaining an enlargement ratio according to one embodiment of the present disclosure.
FIG. 7 is a diagram for explaining voice information according to one embodiment of the present disclosure.
FIG. 8 is a drawing for explaining text provided through a display device according to one embodiment of the present disclosure.
FIG. 9 is a drawing for explaining an editing mode according to one embodiment of the present disclosure.
FIG. 10 is a drawing for explaining a captured image according to one embodiment of the present disclosure.
FIG. 11 is a drawing for explaining an output layout according to one embodiment of the present disclosure.
FIG. 12 is a flowchart for explaining a method for controlling an electronic device according to an embodiment of the present disclosure.

The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

The terms used in the embodiments of the present disclosure are selected from commonly used terms that are as much as possible while considering the functions of the present disclosure, but this may vary depending on the intention of a technician working in the relevant field, precedents, the emergence of new technologies, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meanings thereof will be described in detail in the description of the relevant disclosure. Therefore, the terms used in the present disclosure should be defined based on the meanings of the terms and the overall contents of the present disclosure, rather than simply the names of the terms.

The embodiments of the present disclosure can be variously transformed and have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, and it should be understood that all transformations, equivalents, or substitutes included in the scope of the disclosed idea and technology are included. In describing the embodiments, if it is determined that a specific description of a related known technology may obscure the main point, the detailed description is omitted.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "consist of" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood to not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

In the present disclosure, a “module” or “part” performs at least one function or operation, and may be implemented by hardware or software, or by a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “parts” may be integrated into at least one module and implemented by at least one processor (not shown), except for a “module” or “part” that needs to be implemented by a specific hardware.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present disclosure in the drawings, parts that are not related to the description are omitted, and similar parts are designated by similar drawing reference numerals throughout the specification.

FIG. 1 is a drawing schematically illustrating the configuration of a system according to one embodiment of the present disclosure.

As illustrated in FIG. 1, a system (1000) according to one embodiment of the present disclosure includes an electronic device (100) and a display device (200).

Here, each of the electronic device (100) and the display device (200) may include at least one of, for example, a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a PDA, a portable multimedia player (PMP), an MP3 player, a medical device, a camera, a virtual reality (VR) implementation device, or a wearable device. Here, the wearable device may include at least one of an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, a contact lens, or a head-mounted device (HMD)), a fabric or clothing-integrated type (e.g., an electronic garment), a body-attached type (e.g., a skin pad or a tattoo), or a bio-implantable circuit. In some embodiments, each of the electronic device (100) and the display device (200) may include at least one of, for example, a television, a digital video disk (DVD) player, an audio, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box (e.g., Samsung HomeSync ^TM , Apple TV ^TM , or Google TV ^TM ), a game console (e.g., Xbox ^TM , PlayStation ^TM ), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

In another embodiment, each of the electronic device (100) and the display device (200) may be configured to be connected to various medical devices (e.g., various portable medical measuring devices (such as blood sugar meters, heart rate monitors, blood pressure monitors, or body temperature monitors), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), a camera, or an ultrasound device), a navigation device, a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), an automobile infotainment device, electronic equipment for ships (e.g., a navigation device for ships, a gyro compass, etc.), avionics, a security device, a head unit for a vehicle, an industrial or household robot, a drone, an ATM of a financial institution, a POS (point of sales) of a store, or an Internet of Things device (e.g., a light bulb, various sensors, a sprinkler device, a fire alarm, a thermostat, a streetlight, a toaster, exercise equipment, a hot water tank, It may include at least one of the following (heater, boiler, etc.).

Each of the electronic device (100) and the display device (200) according to an embodiment of the present disclosure may display various types of content. According to an embodiment, the electronic device (100) may be implemented as a user terminal device, and the display device (200) may be implemented as a TV, but is not limited thereto. For example, the display device (200) is not limited to a device having a display function, such as a video wall, a large format display (LFD), a digital signage, a digital information display (DID), a projector display, and the like, and may be applied. In addition, the display device (100) may be implemented as various forms of displays, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), a quantum dot (QD) display panel, a quantum dot light-emitting diodes (QLED), a micro light-emitting diodes (μLED), a mini LED, and the like. Meanwhile, each of the electronic device (100) and the display device (100) may be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which multiple display modules are physically connected, etc.

In the following, for convenience of explanation, it is assumed that the electronic device (100) is implemented as a user terminal device and the display device (200) is implemented as a TV. The electronic device (100) and the display device (200) according to one embodiment of the present disclosure perform communication, and the display device (200) can display various types of content received from the electronic device (100).

For example, the display device (200) may receive content data from the electronic device (100) and display content based on the received content data. As another example, the electronic device (100) and the display device (200) may perform wireless communication, and the display device (200) may display the same screen as the screen displayed on the electronic device (100) through a display equipped with its own display through screen mirroring.

Referring to FIG. 1, an electronic device (100) can obtain and display a live image (10) of the surroundings through a camera equipped in the electronic device (100). A display device (200) that communicates with the electronic device (100) can independently display the live image (10) displayed through the electronic device (100) through a screen mirroring function.

An electronic device (100) according to one embodiment can obtain a live image (10) of a book, document, newspaper, etc. containing text, and transmit the obtained live image (10) to a display device (200).

Here, the display device (200) may have a relatively larger display than the display provided in the electronic device (100), and the display device (200) may display a live image received from the electronic device (100) through the relatively larger display.

Below, various embodiments of enlarging a specific area in a live image (10) acquired through an electronic device (100) and displaying the enlarged specific area through a display device (200) will be described.

FIG. 2 is a drawing schematically illustrating the configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2, the electronic device (100) can identify a text area containing text within a live image received in real time from a camera.

For example, the electronic device (100) may perform OCR (Optical Character Recognition) on at least one object, etc. included in the live image (10) to identify a text area including text. However, this is only an example embodiment, and the electronic device (100) may identify a text area including text within the live image (10) in various ways.

For example, the electronic device (100) may input a live image (10) into an artificial intelligence model to identify a text area including text in the live image (10). Here, the artificial intelligence model according to the embodiment may be a judgment model learned based on a plurality of images based on an artificial intelligence algorithm, and may be a model based on a neural network. The learned judgment model may be designed to simulate the structure of a human brain on a computer and may include a plurality of network nodes having weights that simulate neurons of a human neural network. The plurality of network nodes may each form a connection relationship to simulate synaptic activity of neurons that transmit and receive signals through synapses. In addition, the learned judgment model may include, for example, a machine learning model, a neural network model, or a deep learning model developed from a neural network model. In the deep learning model, a plurality of network nodes may be located at different depths (or layers) and may transmit and receive data according to a convolution connection relationship.

For example, the artificial intelligence model may be a CNN (Convolution Neural Network) model learned based on images. CNN is a multilayer neural network with a special connection structure designed for speech processing, image processing, etc. Meanwhile, the artificial intelligence model is not limited to CNN. For example, the artificial intelligence model may be implemented as at least one DNN (Deep Neural Network) model among RNN (Recurrent Neural Network), LSTM (Long Short Term Memory Network), GRU (Gated Recurrent Units), or GAN (Generative Adversarial Networks).

An electronic device (100) according to an embodiment of the present disclosure may identify a text region predicted to include text before recognizing the text. As an example, the electronic device (100) may perform OCR (Optical Character Recognition) on a live image (10) to identify a text region predicted to include text. As another example, the electronic device (100) may input the live image (10) into an artificial intelligence model to identify a text region predicted to include text within the live image (10).

Referring to FIG. 2, an electronic device (100) according to an embodiment of the present disclosure can display a UI (20) guiding a change in the shooting direction of a camera on a live image (10) when a text area is located in an edge area of a live image (10).

Here, the edge area may mean an edge area or a corner area on a live image. For example, if the text area (10) is located in a corner area of the live image (10) as illustrated in FIG. 2, a part of the text included in the text area (10) may not be displayed on the display provided in the electronic device (100), or a part of the text may be cut off and only the remainder may be displayed. In this case, the probability that the electronic device (100) misrecognizes the text included in the text area (10) increases, and there is a concern that incorrect information or feedback (e.g., visual or auditory feedback) related to the text may be provided to the user.

For example, as illustrated in FIG. 2, if the electronic device (100) identifies that a portion of the text ‘Samsung ran more ~’ in a newspaper article included in a live image (10) is cut off or if text is detected in an edge area of the live image (10), there is a concern that the recognition rate for the text in the newspaper article may be reduced.

According to an embodiment of the present disclosure, if the electronic device (100) identifies that the text area is located in the edge area of the live image (10), it can display a UI (20) that guides a change in the shooting direction of the camera. In this case, if the shooting direction of the camera is changed, the electronic device (100) can display a changed live image (10') received through the camera in real time. Here, the changed live image (10') may be an image in which the text area is located in the center area rather than the edge area. Here, the center area may mean an area in which the entire text included in the text area (10) is displayed through a display provided in the electronic device (100), and in which the probability of misrecognition of the corresponding text by the electronic device (100) is significantly reduced.

Meanwhile, as illustrated in FIG. 1, a display device (200) that performs wired or wireless communication with an electronic device (100) can display a live image (10) received from the electronic device (100) through a display equipped in the display device (200).

A display device (200) according to an embodiment of the present disclosure may display a live image including a UI (20) for guiding a change in a shooting direction received from an electronic device (100). Meanwhile, in FIG. 2, it is assumed and described that the UI (20) for guiding a change in a shooting direction includes text such as ‘WARNING: Text on edge detected.’, but this is for convenience of explanation and is not limited thereto. For example, the UI (20) for guiding a change in a shooting direction may include an arrow indicating a shooting direction, and may include various types of objects for notifying that a part of the text is cut off in the live image (10).

FIG. 3 is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 3, an electronic device (100) according to an embodiment of the present disclosure may include a camera (110), a display (120), a communication interface (130), and a processor (140).

The camera (110) can scan an area surrounding the electronic device (100) and convert it into an electrical signal and transmit it to the processor (140).

A camera (110) according to one embodiment may include a plurality of photodiodes that convert an optical signal into an electrical signal, and the plurality of photodiodes may be arranged in two dimensions. The camera (170) may include a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor in which the plurality of photodiodes are arranged in two dimensions.

According to one embodiment, a camera (110) may scan a surrounding area corresponding to a shooting direction, for example, an angle, and may convert an optical signal obtained by the camera (110) into an electrical signal and transmit the signal to a processor (140) in real time. Subsequently, the processor (140) may control the display (120) to display a live image (10) corresponding to the electrical signal received from the camera (110).

Meanwhile, in FIGS. 1 and 2, for convenience of explanation, it is assumed that the camera (110) scans a newspaper article containing a large number of text areas, but this is only an example, and it is obvious that the camera (110) can scan an area surrounding the camera (110) according to user control.

The display (120) may be implemented as a display including a self-luminous element, or a display including a non-luminous element and a backlight. For example, it may be implemented as various forms of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (120) may also include a driving circuit, a backlight unit, etc., which may be implemented in forms such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (120) may be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc. A display (120) according to one embodiment of the present disclosure can display various types of content under the control of a processor (140). In particular, the display (120) can display a live image (10) received through a camera (110) under the control of the processor (140).

A communication interface (130) according to an embodiment of the present disclosure receives various types of content. For example, the communication interface (130) may receive various types of content from an external device (e.g., a source device), an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through a communication method such as AP-based Wi-Fi (Wireless Local Area Network), Bluetooth, Zigbee, wired/wireless Local Area Network (LAN), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES/EBU (Audio Engineering Society/European Broadcasting Union), Optical, coaxial, etc. Here, the content may include a video signal, an article, text information, a posting, etc.

In particular, the communication interface (130) according to one embodiment of the present disclosure can perform communication with the display device (200). The communication interface (130) according to one embodiment can transmit a live image (10) to the display device (200) under the control of the processor (140). For example, the electronic device (100) performs mirroring with the display device (200) through the communication interface (130), and the live image (10) displayed through the display (120) provided in the electronic device (100) can be displayed in the same manner on the display device (200).

The processor (140) controls the overall operation of the electronic device (100). Here, the processor (140) may be composed of one or more processors. Specifically, the processor (140) may perform the operation of the electronic device (100) according to various embodiments of the present disclosure by executing at least one instruction stored in the memory (130).

According to an embodiment of the present invention, the processor (140) may be implemented as a digital signal processor (DSP) for processing a digital image signal, a microprocessor, a GPU (Graphics Processing Unit), an AI (Artificial Intelligence) processor, an NPU (Neural Processing Unit), or a TCON (Time Controller). However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a MCU (Micro Controller Unit), an MPU (Micro Processing Unit), a controller, an application processor (AP), a communication processor (CP), or an ARM processor, or may be defined by the relevant terms. In addition, the processor (140) may be implemented as a SoC (System on Chip), an LSI (Large Scale Integration) having a processing algorithm built in, or may be implemented in the form of an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

According to an embodiment of the present disclosure, a processor (140) may control a display (120) to display a UI guiding a change in the shooting direction of a camera (110) on a live image (10) when a text area including text in the live image (10) is located in an edge area of the live image (10), as illustrated in FIG. 2.

According to one embodiment, a processor (140) may control a communication interface (130) to transmit a live image (10) including a UI to a display device (200).

A processor (140) according to one embodiment of the present disclosure can display a text area included in a live image (10) by enlarging the text area so that the text area is greater than a critical ratio within the screen of the display. A detailed description thereof will be given with reference to FIGS. 4 and 5.

FIG. 4 is a drawing for explaining a text area according to one embodiment of the present disclosure.

Referring to FIG. 4, the processor (140) may identify a text area within the live image (10). For example, the processor (140) may perform OCR on at least one object included in the live image (10) to identify whether the object includes text. As another example, the processor (140) may input the live image (10) into an artificial intelligence model to obtain information on whether a text area is included in the live image (10), the location of the text area, whether the text is located in an edge area, and the like. However, this is merely an example and is not limited thereto. For example, the processor (140) may identify a text area within the live image (10) by using a known algorithm for identifying a text area within an image.

In one embodiment, the processor (140) may identify a text area containing text within a live image (10) and transmit information about the text area to the display device (200) so that the text area is displayed on the display device (200). For example, the processor (140) may transmit the live image (10) to the display device (200), and then the display device (200) may identify a text area within the received live image (10) and display the identified text area.

As another example, the processor (140) may identify a text area within the live image (10) and transmit an image corresponding to the identified text area to the display device (200). Subsequently, the display device (200) may display the received image. Here, the display provided in the display device (200) may be relatively larger in size than the display (120) provided in the electronic device (100). Therefore, if the text area is displayed through the display device (200), the user can recognize the text relatively more conveniently and easily through the display device (200) than when the text is provided through the display (120) provided in the electronic device (100).

FIG. 5 is a drawing for explaining an enlarged text area according to one embodiment of the present disclosure.

Referring to FIG. 5, the processor (140) can identify a text area within a live image (10) and display the identified text area by enlarging it to a threshold ratio or greater within the screen of the display (120).

For example, the processor (140) may analyze at least one object included in the live image (10) and classify the object as text, a picture, or a photograph. Then, the processor (140) may identify an area including an object classified as text as a text area.

According to one embodiment, the processor (140) may enlarge and display a text area so that the text area is larger than a threshold ratio within the screen. Here, the threshold ratio may mean the size of the display (120). For example, the processor (140) may enlarge and display only the text area within the live image (10) so that it corresponds to the entire screen of the display (120) based on the size of the display (120). Referring to FIG. 5, the processor (140) may control the display (120) to display only the text area, excluding an area including a photograph, a picture, etc., in the live image (10).

However, this is only an example and is not limited thereto. For example, the processor (140) may identify an area including a specific object within the live image (10) and enlarge and display the area. For example, the processor (140) may identify an area including a photo within the live image (10) and enlarge and display only the photo, excluding text, etc. As another example, the processor (140) may identify a specific text (e.g., a specific word) within the live image (10) and enlarge and display only the specific text.

The threshold ratio according to an embodiment of the present disclosure may be a ratio set according to a user command. For example, the processor (140) may enlarge and display a text area at a ratio corresponding to the user command. For example, if a specific ratio (or magnification) is selected from 1 to 8 times according to the user command, the processor (140) may enlarge and display the text area based on the selected ratio (or magnification). Here, the specific numbers are merely examples for the convenience of explanation, and it goes without saying that the range of selectable ratios may be variously changed. For example, the processor (140) may identify a ratio corresponding to a user's pinch touch input or spread touch input, and enlarge the text area using the identified ratio as the threshold ratio. Meanwhile, the ratio according to various embodiments of the present disclosure may include the meaning of an enlargement ratio.

FIG. 6 is a drawing for explaining an enlargement ratio according to one embodiment of the present disclosure.

Referring to FIG. 6, a processor (140) according to an embodiment of the present disclosure may control a display to display a UI for setting a magnification ratio. When a user command for selecting a magnification ratio of one to four times (e.g., 1X to 4X) is received through the UI for setting a magnification ratio, the processor (140) according to an embodiment of the present disclosure may magnify a text area to a magnification ratio corresponding to the user command.

Next, the processor (140) can transmit a live image (10) including the enlarged text area to the display device (200).

When the magnification ratio is set according to a user command, the processor (140) can magnify the text area based on the set magnification ratio, and the text area that the display device (200) receives from the electronic device (100) and displays can also be automatically magnified. For example, since the display device (200) performs mirroring communication with the electronic device (100) to display the text area, when the electronic device (100) magnifies the text area that it is displaying according to the set magnification ratio, the text area that the display device (200) is displaying can also be magnified at the same time.

Returning to FIG. 3, each of the electronic device (100) and the display device (200) according to one embodiment of the present disclosure may be equipped with a speaker (not shown).

Here, the speaker has the function of outputting an audio signal. For example, the speaker may include at least one speaker unit (or audio amplifier).

Specifically, the speaker may include at least one of a woofer speaker, a midrange speaker, and a tweeter speaker. However, the present invention is not limited thereto, and the speaker may output sound signals of various ranges by using a plurality of speakers.

In particular, a processor (140) according to an embodiment of the present disclosure can identify text included in a text area, apply TTS (Text to Speech) to the identified text to obtain voice information corresponding to the text, and control a speaker to output sound based on the voice information. A detailed description thereof will be given with reference to FIG. 7.

FIG. 7 is a diagram for explaining voice information according to one embodiment of the present disclosure.

Referring to FIG. 7, a processor (140) according to an embodiment of the present disclosure may identify text included in a text area. For example, the processor (140) may perform OCR on the text area or input the text area into an artificial intelligence model to identify text included in the text area.

Next, the processor (140) can obtain voice information, for example, a sound signal, corresponding to the identified text using the TTS engine. Next, the processor (140) can control the speaker to output the sound signal.

Accordingly, the electronic device (100) can provide the user with a voice corresponding to the text included in the live image (10). People with low vision or visual impairments who have difficulty recognizing text can acquire information included in information media that mainly consists of text, such as newspapers and books, through the voice output by the electronic device (100).

Meanwhile, referring to FIG. 7, the electronic device (100) may output a sound corresponding to text, and the display device (200) may output a sound corresponding to text.

For example, the electronic device (100) can apply TTS to the identified text to obtain voice information corresponding to the text and transmit the obtained voice information to the display device (200). Then, the display device (200) can output sound through a speaker provided in the display device (200) based on the received voice information.

As another example, the display device (200) may identify text included in a text area and apply TTS to the identified text to obtain voice information corresponding to the text. In this case, the display device (200) may obtain voice information corresponding to the text by using a TTS engine on its own and output a sound corresponding to the obtained voice information.

FIG. 8 is a drawing for explaining text provided through a display device according to one embodiment of the present disclosure.

Referring to FIG. 8, an electronic device (100) according to an embodiment of the present disclosure may identify text included in a text area and transmit information including the identified text to a display device (200). As an example, the electronic device (100) may transmit a live image (10) to the display device (200) in real time, or may transmit only an image corresponding to a text area included in the live image (10) to the display device (200). As another example, the electronic device (100) may identify text included in a text area and transmit information including the identified text to the display device (200). Here, the information including the text may include text, a font of the text, a size of the text, and the like.

A display device (200) according to one embodiment of the present disclosure can display text based on received text information. In this case, the display device (200) may not display other objects such as photographs and pictures included in the live image (10). Meanwhile, this is only one embodiment and is not limited thereto.

A display device (200) according to one embodiment of the present disclosure can enlarge text to a specific magnification based on received text information and display the enlarged text.

Referring to FIG. 8, the display device (200) may receive information about text included in a live image (10) from the electronic device (100) and display the text based on the information about the received text. For example, the display device (200) may display ‘Samsung ran more than ~’. Here, the display device (200) may enlarge the text to a specific magnification and then display it.

For convenience of explanation, FIG. 8 assumes a case where the display device (200) enlarges and displays text so that a specific number of characters (e.g., three alphabets) occupy the entire screen of the display provided in the display device (200). However, this is only an example and is not limited thereto. For example, the display device (200) may enlarge text so that one character occupies the entire screen of the display provided in the display device (200). Since a person with low vision can receive text through a display device (200) having a relatively large screen, such as a TV, rather than through an electronic device (100) having a relatively small screen, such as a smart phone or a user terminal device, there is an effect in which even a person with low vision can easily recognize text.

FIG. 9 is a drawing for explaining an editing mode according to one embodiment of the present disclosure.

Referring to FIG. 9, an electronic device (100) according to an embodiment of the present disclosure may provide an edit mode for editing identified text.

For example, when the edit mode is executed according to a user command, the processor (140) can identify text included in the text area and display an edit UI including the identified text. Here, the edit UI can include a plurality of UIs for controlling addition of new text, modification or deletion of identified text, change of font of text, change of size of text, etc.

According to one embodiment of the present disclosure, when editing of text is performed on the editing UI, the processor (140) can control the display (120) to display the edited text in real time. For convenience of explanation, FIG. 9 assumes a case where the ‘NEWS’ text (30) is added through the editing UI. However, this is only an example, and the processor (140) can of course delete or modify some of the identified text according to a user command for the editing UI.

Meanwhile, according to one embodiment of the present disclosure, the processor (140) may control the communication interface (130) to transmit a screen image of the display (120) including newly added ‘NEWS’ text (30) according to a user command in addition to the edited text, for example, the text identified from the live image (10), to the display device (200).

According to one embodiment, the display device (200) and the electronic device (100) are performing mirroring communication, and when editing of text is performed through the electronic device (100), the display device (200) can receive and display a screen image of the electronic device (100) including the edited text in real time.

Referring to FIG. 9, the display device (200) can receive and display a screen image of the display (120) including the text ‘NEWS’ (30) from the electronic device (100). For convenience of explanation, FIG. 9 illustrates a case in which the display device (100) enlarges the screen image received from the electronic device (100) to a certain magnification (e.g., a magnification set according to a user command) and then displays it.

FIG. 10 is a drawing for explaining a captured image according to one embodiment of the present disclosure.

Referring to FIG. 10, a processor (140) according to one embodiment of the present disclosure may stop transmitting a live image (10) if the displayed live image (10) is captured according to a user command while the live image (10) is being transmitted to a display device (200).

For example, if the hand of the user holding the electronic device (100) is not fixed, there is a problem that it is difficult for the user or the processor (140) to recognize text, etc. because the live image (10) received through the camera (110) is constantly changing.

The processor (140) can capture a displayed live image (10) according to a user command. In this case, the processor (140) can stop transmission of the live image (10) being transmitted in real time to the display device (200). Subsequently, the processor (140) can transmit the communication interface (130) to transmit the captured image to the display device (200). Here, the captured image may be called a frozen image, a fixed image, etc., but for the convenience of explanation, it will be collectively called a captured image.

The display device (200) can display a captured image received from the electronic device (100). In this case, even if the live image (10) transmitted by the camera (110) equipped in the electronic device (100) is changed, the electronic device (100) and the display device (200) can display the captured image. However, this is just an example, and the electronic device (100) can display the live image (10) transmitted from the camera (110), and the display device (200) can also display the captured image.

A display device (200) according to one embodiment can identify text included in a captured image and display the identified text by enlarging it to a critical ratio.

A display device (200) according to one embodiment of the present disclosure can display a UI for setting a magnification ratio. For example, the display device (200) can magnify and then display a text area or identified text based on a magnification ratio set according to a user command for the UI for setting a magnification ratio.

Additionally, the display device (200) may apply TTS to the identified text to obtain voice information corresponding to the text.

FIG. 11 is a drawing for explaining an output layout according to one embodiment of the present disclosure.

Referring to FIG. 11, a display device (200) according to an embodiment of the present disclosure can identify a text area within a live image (10) received from an electronic device (100) and identify text included in the identified text area.

Next, the display device (200) can identify an output layout based on the identified text. For example, the display device (200) can identify whether the number of texts of the identified text is greater than or equal to a threshold value. Next, if the number of texts of the identified text is greater than or equal to the threshold value, the display device (200) can identify an output layout suitable for displaying a relatively large amount of text.

For example, the display device (200) can enlarge text so that only one to ten characters are displayed on the screen so that a person with low vision can easily recognize the text. On the other hand, if the number of texts is greater than a threshold value, there is a problem that it takes a long time for the user to recognize all the texts. The display device (200) according to one embodiment can divide the screen in half and enlarge and display a certain number of texts on each of the left and right sides of the screen to maximize the enlargement ratio of the text while minimizing the margins within the screen.

Referring to Fig. 11, the output layout may include a One page layout or a Two page layout. However, this is only an example and is not limited thereto. For example, it may include a Two page layout that divides the screen into left and right halves, or it may include a Two page layout that divides the screen into top and bottom halves.

As another example, when the display device (200) identifies multiple text areas within the live image (10), it can identify a Two page layout as the output layout. Then, it can display the text included in the first text area in the left layout on the screen, and display the text included in the second text area in the right layout on the screen.

FIG. 12 is a flowchart for explaining a method for controlling an electronic device according to an embodiment of the present disclosure.

A method for controlling an electronic device according to an embodiment of the present disclosure first displays a live image received through a camera (S1210).

Next, if a text area containing text within a live image is located in an edge area of the live image, a UI guiding a change in the shooting direction of the camera is displayed on the live image (S1220).

Next, a live image including the UI is transmitted to an external display device (S1230).

The step S1210 of displaying a live image according to one embodiment of the present disclosure may include the step of displaying the text area included in the live image by enlarging the text area so that the text area is greater than a threshold ratio within the screen, and the step S1230 of transmitting may include the step of transmitting the live image including the enlarged text area to an external display device.

A control method according to one embodiment further includes a step of displaying a UI for setting a magnification ratio, and the step S1210 of displaying a live image may include a step of displaying a text area by enlarging it based on a magnification ratio set through the UI for setting the magnification ratio.

A control method according to one embodiment of the present disclosure may further include a step of identifying text included in a text area, a step of applying TTS (Text to Speech) to the identified text to obtain voice information corresponding to the text, and a step of outputting sound based on the voice information.

A control method according to one embodiment of the present disclosure may further include a step of transmitting acquired voice information to an external display device.

A control method according to one embodiment of the present disclosure may further include, when an edit mode is executed according to a user command, a step of identifying text included in a text area and displaying an edit UI including the identified text, a step of displaying edited text when editing of the text is performed on the edit UI, and a step of transmitting a screen image of a display including the edited text to an external display device.

The step S1230 of transmitting according to one embodiment of the present disclosure may include a step of stopping transmission of the live image and transmitting the captured image to the external display device when the displayed live image is captured according to a user command while the live image is being transmitted to the external display device.

However, it goes without saying that the various embodiments of the present disclosure can be applied not only to electronic devices, but also to all types of electronic devices equipped with a display.

Meanwhile, the various embodiments described above may be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described in this specification may be implemented by the processor itself. According to a software implementation, embodiments such as the procedures and functions described in this specification may be implemented by separate software modules. Each of the software modules may perform one or more functions and operations described in this specification.

Meanwhile, computer instructions for performing processing operations of the electronic device (100) according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in the non-transitory computer-readable medium are executed by a processor of a specific device, they cause the specific device to perform processing operations in the electronic device (100) according to various embodiments described above.

A non-transitory computer-readable medium is not a medium that stores data for a short period of time, such as a register, cache, or memory, but a medium that permanently stores data and can be read by a device. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray disks, USBs, memory cards, and ROMs.

Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be individually understood from the technical idea or prospect of the present disclosure.

100: Electronic devices 110: Camera
120: Display 130: Communication Interface
140: Processor 200: Display Device
1000: System

Claims (20)

camera;
display;
a communication interface including a circuit; and
Controlling the display to display a live image received through the camera;
A processor for controlling the communication interface to transmit the displayed live image to an external display device;
The above processor,
Dividing the above live image into multiple areas, and identifying a text area among the multiple areas,
Enlarging the text area so that it exceeds a critical ratio within the screen of the display and providing it through the display,
Controlling the communication interface to transmit the live image including the enlarged text area to the external display device;
When the text area including the text is located in the edge area of the live image, the display is controlled to display a UI guiding a change in the shooting direction of the camera on the live image,
Controlling the communication interface to transmit the UI to the external display device so that the UI is displayed together with the live image;
The above multiple areas are,
An electronic device comprising the above text area and image area. delete In the first paragraph,
The above processor,
Control the display to display a UI for setting the magnification ratio,
An electronic device that controls the display to enlarge and display the text area based on a magnification ratio set through a UI for setting the magnification ratio. In the first paragraph,
including speakers;
The above processor,
An electronic device that identifies the text included in the text area, applies TTS (Text to Speech) to the identified text to obtain voice information corresponding to the text, and controls the speaker to output sound based on the voice information. In paragraph 4,
The above processor,
An electronic device that controls the communication interface to transmit the acquired voice information to the external display device. In the first paragraph,
The above processor,
Identify the text contained in the above text area,
An electronic device that controls the communication interface to transmit information including the identified text to the external display device. In the first paragraph,
The above processor,
When the edit mode is executed according to a user command, the display is controlled to identify the text included in the text area and display an edit UI including the identified text.
When editing of the text is performed on the above editing UI, the display is controlled to display the edited text,
An electronic device that controls the communication interface to transmit a screen image of the display including the edited text to the external display device. In the first paragraph,
The above processor,
An electronic device that controls the communication interface to stop transmitting the live image and transmit the captured image to the external display device when the displayed live image is captured according to a user command while the live image is being transmitted to the external display device. In the first paragraph,
The above external display device,
An electronic device having a display having a screen size larger than that of the above display. In a system including an electronic device and a display device,
Divides the received live image through the camera into multiple areas,
Identify a text area among the above multiple areas,
Enlarge and display the text area so that the text area is larger than a critical ratio within the screen of the display device,
Transmitting the live image including the enlarged text area to an external device;
When the text area including the text is located in the edge area of the live image, a UI guiding a change in the shooting direction of the camera is displayed on the live image,
An electronic device that transmits the UI to the display device so that the UI is displayed together with the live image; and
A display device for displaying the live image and the UI received from the electronic device;
The above multiple areas are,
A system comprising the above text area and image area. delete ◈Claim 12 was abandoned upon payment of the registration fee.◈ In Article 10,
The above display device,
Displays a UI for setting the magnification ratio,
A system that enlarges and displays the text area based on a magnification ratio set through a UI for setting the magnification ratio. ◈Claim 13 was abandoned upon payment of the registration fee.◈ In Article 10,
The above display device,
When the text contained in the text area is identified, the output layout is identified based on the identified text,
A system for displaying the text according to the identified output layout. In a method for controlling an electronic device,
A step of displaying a live image received through a camera; and
A step of transmitting the above live image to an external display device:
The above displaying steps are:
A step of dividing the above live image into a plurality of areas and identifying a text area among the plurality of areas;
A step of providing the text area by enlarging it so that the text area is greater than a critical ratio within the screen of the display; and
A step of displaying a UI guiding a change in the shooting direction of the camera on the live image when the text area including the text is located in the edge area of the live image;
The above transmitting step is,
a step of transmitting the live image including the enlarged text area to the external display device; and
A step of transmitting the UI to the external display device so that the UI is displayed together with the live image;
The above multiple areas are,
A control method including the above text area and image area. delete ◈Claim 16 was abandoned upon payment of the registration fee.◈ In Article 14,
A step of displaying a UI for setting a magnification ratio is further included;
The step of providing an enlarged text area is as follows:
A control method, comprising: a step of displaying the text area by enlarging it based on a magnification ratio set through a UI for setting the magnification ratio; ◈Claim 17 was abandoned upon payment of the registration fee.◈ In Article 14,
A step of identifying the text contained in the text area;
A step of applying TTS (Text to Speech) to the identified text to obtain voice information corresponding to the text; and
A control method further comprising a step of outputting a sound based on the above voice information. ◈Claim 18 was abandoned upon payment of the registration fee.◈ In Article 17,
A control method further comprising a step of transmitting the acquired voice information to the external display device. ◈Claim 19 was abandoned upon payment of the registration fee.◈ In Article 14,
When the edit mode is executed according to a user command, a step of identifying the text included in the text area and displaying an edit UI including the identified text;
When editing of the text is performed on the above editing UI, a step of displaying the edited text; and
A control method further comprising the step of transmitting a screen image of the display including the edited text to the external display device. ◈Claim 20 was abandoned upon payment of the registration fee.◈ In Article 14,
The above transmitting step is,
A control method, comprising: a step of: when the displayed live image is captured according to a user command while the live image is being transmitted to the external display device, stopping the transmission of the live image and transmitting the captured image to the external display device;

Images