TWI898735B - Electronic apparatus and method for dynamically displaying image thereof - Google Patents

Abstract

An electronic apparatus and a method for dynamically displaying image thereof are provided. The method is adapted to the electronic apparatus including a LED array and an image sensor and includes the following steps. The method, applicable to the electronic devices including a light-emitting diode array and an image sensor, includes the following steps. A scene image is captured through the image sensor. Object recognition is performed on the scene image to obtain scene information. A target image is determined based on the scene information. The light-emitting diode array is controlled to display the target image.

Description

Electronic device and method for dynamically displaying images therewith

The present invention relates to an electronic device, and more particularly to an electronic device and a method for dynamically displaying images thereon.

To meet users' demands for visual and audio effects, manufacturers are increasingly integrating LEDs into electronic devices to create diverse and colorful lighting effects. Existing LED display modules offer predetermined and fixed lighting effects, making them difficult to adapt to user behavior and provide effective, real-time user interaction. Furthermore, it's difficult for users to modify or adjust the lighting effects presented by LEDs, and a single lighting effect can't satisfy all user needs and preferences. Furthermore, the significant performance burden of dynamically controlling the display of a large number of LEDs in real time significantly limits the speed at which the lighting pattern can change, limiting the flexibility and diversity of the lighting effects.

In view of this, the present invention proposes an electronic device and a method for dynamically displaying images thereof, which can solve the above technical problems.

An embodiment of the present invention provides a method for dynamically displaying images, applicable to an electronic device comprising a light-emitting diode array and an image sensor. The method comprises the following steps: capturing a scene image using the image sensor; performing object recognition on the scene image to obtain scene information; determining a target image based on the scene information; and controlling the light-emitting diode array to display the target image.

An embodiment of the present invention provides an electronic device comprising a light-emitting diode array, an image sensor, a storage device, a controller, and a processor. The light-emitting diode array comprises a plurality of light-emitting diodes, and the storage device stores a plurality of instructions. The controller is coupled to the light-emitting diode array. The processor is coupled to the image sensor, the storage device, and the controller, and is configured to execute the aforementioned instructions to perform the following operations. A scene image is captured by the image sensor. Object recognition is performed on the scene image to obtain scene information. A target image is determined based on the scene information. The controller is used to control the light-emitting diode array to display the target image.

Based on the above, in an embodiment of the present invention, scene information is acquired through object recognition within a scene image. A target image is then generated based on this scene information. A LED array is then controlled by a controller to display the target image. Consequently, the LED array on an electronic device can dynamically display a target image related to the scene information and dynamically adjust the displayed content in real time to respond to scene changes, significantly enhancing the user experience.

Some embodiments of the present invention are described in detail below with reference to the accompanying drawings. Reference symbols in the following description will identify identical or similar elements when the same symbols appear in different drawings. These embodiments are only a portion of the present invention and do not disclose all possible implementations of the present invention. Rather, these embodiments are merely examples of the methods and apparatus within the scope of the present invention.

Figure 1 is a block diagram of an electronic device according to one embodiment of the present invention. Referring to Figure 1 , electronic device 100 includes a light-emitting diode (LED) array 110, an image sensor 120, a controller 130, a storage device 140, and a processor 150. Electronic device 100 may be, for example, a tablet computer, a laptop computer, a desktop computer, or an all-in-one computer, although the present invention is not limited thereto.

The LED array 110 includes multiple light-emitting diodes (LEDs). In some embodiments, these LEDs may include multi-color (RGB) LEDs, single-color LEDs, or a combination thereof. In some embodiments, the LEDs in the LED array 110 may be arranged in an M*N array to form a dot matrix LED panel, where M is an integer greater than 1 and N is an integer greater than 1. The present disclosure does not limit the number of LEDs in the LED array 110.

In some embodiments, based on the Microsoft HID Lighting and Illumination (HID) standard, the operating system of electronic device 100 can directly control the lighting elements of LED array 110. Based on the HID Lighting and Illumination standard, each LED in LED array 110 can be assigned a logical position, and this logical position serves as the LED's identification data. A program module in the operating system can directly control the LEDs in LED array 110. Lighting effect control of LED array 110 can be performed by the program module running in user mode.

The image sensor 120 is used to capture scene images. In some embodiments, the image sensor 120 may be an RGB image sensor or an infrared image sensor, etc., which is not limited in this disclosure. In some embodiments, the image sensor 120 may include a lens, an image sensing element, and other components. The lens may include an optical lens for controlling the light path. The image sensing element is used to provide an image sensing function. The image sensing element may include a photosensitive element, such as a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) element, or other elements, which is not limited in this disclosure. The lens can converge imaging light onto the image sensing element to achieve the purpose of capturing images.

Controller 130 is coupled to LED array 110. In some embodiments, controller 130 supports the HID standard and is used to control the lighting of LED array 110. The functions of controller 130 can be implemented in various logic blocks, modules, and circuits within one or more controllers, microcontrollers, microprocessors, application-specific integrated circuits (ASICs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), and/or other processing units. In some embodiments, controller 130 can be implemented by a Raspberry Pi RP2040. In some embodiments, controller 130 can communicate with processor 150 via a USB interface and control the lighting of LED array 110 according to the HID standard.

In some embodiments, the controller 130 may include a programmable I/O (PIO) component. The controller 130 may transmit data to the LED array 110 via the PIO component to control the plurality of LEDs in the LED array 110.

The storage device 140 is used to store data and software modules (such as operating systems, applications, and drivers) for access by the processor 150. It can be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard drive, or a combination thereof.

Processor 150 is coupled to controller 130, image sensor 120, and storage device 140. It can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, multiple microprocessors, one or more microprocessors incorporating a digital signal processor core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), any other type of integrated circuit, a state machine, an Advanced RISC Machine (ARM)-based processor, or the like. Processor 150 can access and execute instructions or program code stored in storage device 140 to implement the method for dynamically displaying images in the embodiments of the present invention.

It should be noted that different LEDs in LED array 110 correspond to different logical positions. Therefore, based on HID specifications, processor 150 can individually and directly control the lighting parameters of these LEDs based on their logical positions. These lighting parameters may include color parameters, brightness parameters, lighting frequency parameters, or other lighting control parameters.

In some embodiments, the LED array 110 can be disposed on a housing of the electronic device 100. The housing can be a top cover of a laptop or a chassis of a desktop computer, and the present disclosure is not limited thereto.

For example, Figure 2A is a schematic diagram of an electronic device according to an embodiment of the present invention. Referring to Figure 2A , in some embodiments, electronic device 100 may be a laptop computer. LED array 110 may be disposed on the laptop computer's upper cover B1. Alternatively, in other embodiments, LED array 110 may be disposed on the laptop computer's keyboard bezel.

For example, FIG2B is a schematic diagram of an electronic device according to an embodiment of the present invention. Referring to FIG2B , in some embodiments, the electronic device 100 may be a desktop computer. The LED array 110 may be mounted on a side surface of the desktop computer's chassis C1. Alternatively, in other embodiments, the LED array 110 may be mounted on the top surface of the desktop computer's chassis C1.

Figure 2C is a schematic diagram of an electronic device according to an embodiment of the present invention. Referring to Figure 2C , in some embodiments, LED array 110 may be mounted on a peripheral electronic device P1 of electronic device 100. Peripheral electronic device P1 may be connected to the host computer of electronic device 100 via a USB interface. Peripheral electronic device P1 may be, for example, a USB hub or a docking station, but the present disclosure is not limited thereto.

FIG3 is a flow chart of a method for dynamically displaying images according to an embodiment of the present invention. The method flow in FIG3 can be implemented by the components of electronic device 100 in FIG1 . Referring to FIG1 and FIG3 together, the following describes the steps of the method for dynamically displaying images according to this embodiment using the components of electronic device 100 in FIG1 .

In step S310 , the processor 150 captures a scene image via the image sensor 120 . For example, the processor 150 may use a front-facing camera to capture the scene image facing the user. Alternatively, the processor 150 may use a rear-facing camera to capture the scene image facing the back of the electronic device 100 .

In some embodiments, the processor 150 may control the image sensor 120 to capture scene images in response to executing a specific application. In some embodiments, the processor 150 may control the image sensor 120 to capture scene images in response to entering a specific operating state.

In step S320, the processor 150 performs object recognition on the scene image to obtain scene information. In some embodiments, the scene information may be the object type of the scene object in the scene image. For example, the object type may be "dog," "flower," "face," "cat," "cup," "hand," or other object types, although the present disclosure is not limited thereto. In some embodiments, the scene information may be a scene category determined based on the object type of the scene object in the scene image. For example, the scene category may be "outdoor," "indoor," "office," "cafe," "conference room," or "restaurant," etc., although the present disclosure is not limited thereto. For example, when the processor 150 recognizes that the scene image includes a whiteboard or a projection screen, the processor 150 may determine that the scene category is “conference room”.

In some embodiments, processor 150 may utilize a convolutional neural network (CNN) model to perform object recognition on a scene image. Processor 150 may then obtain the object type of a target scene object output by the CNN model. The scene information may include the object type of the target scene object.

In some embodiments, the trained convolutional neural network model is a machine learning model pre-constructed through deep learning based on a training image set and can be stored in the storage device 140. In other words, the model parameters of the trained convolutional neural network model (e.g., the number of neural network layers and the weights of each neural network layer) have been determined through pre-training and stored in the storage device 140. When a scene image is input to the trained convolutional neural network model, the trained convolutional neural network model first performs feature extraction to generate a feature vector. These feature vectors are then fed into a classifier within a trained convolutional neural network model. The classifier then performs classification based on these feature vectors, thereby identifying object information of scene objects within the scene image. The trained convolutional neural network model can be, for example, an R-CNN model, Fast R-CNN model, Faster R-CNN model, YOLO model, or SSD model for object detection, though this disclosure is not limited thereto.

In some embodiments, the object information output by the convolutional neural network model may include object type, object size, and object position. For example, the processor 150 may input a scene image into the convolutional neural network model for object recognition and obtain the object type "flower." Therefore, the processor 150 may use the object type "flower" as the scene information. Alternatively, when the scene image includes a specific gesture of the user, the processor 150 may use the convolutional neural network model for object recognition and obtain the object type "specific gesture." Therefore, the processor 150 may use the object type "specific gesture" as the scene information.

In some embodiments, the processor 150 may obtain multiple scene objects identified by the convolutional neural network model. That is, the processor 150 may identify one or more scene objects from the scene image. The processor 150 may select a target scene object from the multiple scene objects to obtain the object type of the target scene object. In some embodiments, the processor 150 may select the target scene object based on the object size of each scene object in the scene image. For example, when the processor 150 detects multiple scene objects from the scene image, the processor 150 may select the scene object with the largest object size as the target scene object. The object size may be the size of the bounding box output by the convolutional neural network model. In some embodiments, the processor 150 may select a target scene object based on the object position of each scene object in the scene image. For example, when the processor 150 detects multiple scene objects from the scene image, the processor 150 may select the scene object closest to the center of the scene image as the target scene object.

In step S330, the processor 150 determines a target image based on the scene information. In some embodiments, the processor 150 may select the target image from a plurality of preset images. Alternatively, in some embodiments, the processor 150 may utilize an image generation model to generate the target image.

Figure 4 is a flowchart of determining a target image according to an embodiment of the present invention. Referring to Figure 4 , in operation 41, the processor 150 may input the scene image Img1 into a convolutional neural network model to identify the object type 42 of the target scene object. Subsequently, in operation 43, the processor 150 may select a target image ImgT1 from a plurality of preset images based on the object type of the target scene object. The processor 150 may select the target image ImgT1 from a plurality of preset images in database db1. In some embodiments, each preset image in database db1 may be associated with an image label. When the object type 42 of the target scene object matches the image label of a preset image, the processor 150 may select the preset image as the target image ImgT1. For example, when the object type 42 is "dog," the processor 150 may obtain the target image ImgT1 associated with the image label "dog" from the database db1.

FIG5 is a flowchart of determining a target image according to an embodiment of the present invention. Referring to FIG5 , in operation 41, the processor 150 may input the scene image Img1 into a convolutional neural network model to identify the object type 42 of the target scene object. Subsequently, in operation 46, the processor 150 may determine a text prompt 47 based on the object type 42 of the target scene object. For example, when the object type 42 is "dog," the processor 150 may determine a text prompt 47 to be "dog." When the object type 42 includes "dog" and "flower," the processor 150 may determine a text prompt 47 to be "dog and flower." In operation 48, the processor 150 may generate the target image ImgT1 based on the text prompt 47 using an image generation model. The image generation model can be a text-to-image model, such as, but not limited to, a Contrastive Language–Image Pre-Training (CLIP) model. The image generation model can generate a target image ImgT1 based on an input text prompt 47.

In some embodiments, the processor 150 may also combine other scene information with the object type 42 of the target scene object to generate the text prompt 47. The other scene information may be, for example, sensor data, user input text, current time, or weather information.

Returning to Figure 3, in step S340, processor 150 utilizes controller 130 to control LED array 110 to display a target image. In other words, LED array 110 may display a target image related to the content of the scene being captured. For example, if a dog appears in the scene image, the target image displayed by LED array 110 may include an image of the dog.

In some embodiments, the processor 150 may determine the emission parameters of each LED in the LED array 110 based on the target image. Each LED in the LED array 110 is mapped to a corresponding one of the multiple pixels in the target image. In some embodiments, the emission parameters of each LED include a red channel value, a green channel value, and a blue channel value. Specifically, the processor 150 may determine the emission color corresponding to each of the multiple RGB LEDs in the LED array 110 based on the pixel data of the target image. The emission color of each RGB LED in the LED array 110 may be set to the pixel data of a corresponding pixel in the target image. That is, the red channel value of each LED corresponds to the red pixel value of the corresponding pixel. The green channel value of each LED corresponds to the green pixel value of a corresponding pixel. The blue channel value of each LED corresponds to the blue pixel value of a corresponding pixel.

In some embodiments, processor 150 may transmit the emission parameters of each LED to controller 130, causing controller 130 to control the lighting of each LED based on the emission parameters. In other words, controller 130 may control each LED in LED array 110 to emit light of a corresponding color. In some embodiments, processor 150 may utilize a HID API to transmit the emission parameters of each LED to controller 130. After parsing the HID report, controller 130 may transmit the RGB data of each LED to LED array 110. Based on this, the LEDs in the LED array 110 will emit corresponding colored light according to the RGB data provided by the controller 130 to display the target image.

Figure 6 is a flow chart for controlling a LED array to display a target image according to an embodiment of the present invention. Referring to Figure 6 , in step S601, processor 150 determines the lighting parameters of each LED in LED array 110 based on the target image. Each LED is an RGB LED, and each LED's lighting parameters include red, green, and blue channel values. In step S602, processor 150 transmits the lighting parameters of each LED to a controller 130 via a Human Interface Device (HID) report. Furthermore, the processor 150 may map each LED to a pixel in the target image according to its logical position, and associate the logical position of each LED with the corresponding red channel value, green channel value, and blue channel value in the HID report.

In step S603, the controller 130 parses the HID report to obtain the RGB data for each LED, namely, the red, green, and blue channel data for each LED. The red, green, and blue channel data can be, for example, 8-bit binary data. In step S604, the controller 130 transmits the RGB data for each LED to the LED array 110. In step S604, the LED array 110 emits light based on the RGB data from each LED to display the target image.

Figure 7 is a schematic diagram of a controller and LED array according to one embodiment of the present invention. Referring to Figure 7 , processor 150 can connect to controller 130 via a USB interface and transmit the lighting parameters of each LED to controller 130 via HID reports. Controller 130 communicates with processor 150 via USB as a Human Interface Device (HID).

In this embodiment, LED array 110 may include eight LED arrays 110_1 through 110_8. Controller 130 includes a programmable input/output (PIO) component 131. Controller 130 controls LED arrays 110_1 through 110_8 via PIO component 131. PI component 131 of controller 130 can be used to send data to LED array 110 via general-purpose input/output (GPIO) pins GPIO1 through GPIO8.

Controller 130 is connected to eight 8x8 LED arrays 110_1 through 110_8 via eight GPIO pins (GPIO1 through GPIO8). Each LED array 110_1 through 110_8 may include 64 LEDs. The RGB data transmitted by controller 130 is passed sequentially within each LED array 110_1 through 110_8, from the first LED to the last, ensuring that all LEDs receive the correct control signals.

In some embodiments, the PIO component 131 may include a state machine, a first-in, first-out (FIFO) input module, and a FIFO output module. The FIFO input module is responsible for receiving external data and storing this data in a first-in, first-out manner. The state machine is responsible for controlling the transmission of data from the FIFO input module to the FIFO output module, processing it according to predefined logic and states. The FIFO output module receives processed data from the state machine and stores and outputs this data in a first-in, first-out manner. In this way, each LED array 110_1-110_8 can receive RGB data and pass it to each LED in sequence, ultimately controlling the color and brightness of all LEDs. By controlling the lighting of each LED through controller 130, the burden on processor 150 can be reduced, thereby achieving a high refresh rate. This allows for efficient LED matrix control, making it ideal for applications requiring both high performance and a high refresh rate.

In summary, in embodiments of the present invention, scene information is acquired through object recognition within a scene image. A target image is then derived based on this scene information. A LED array is then controlled by a controller to display the target image. Consequently, the LED array on an electronic device can dynamically display a target image related to the scene information and dynamically adjust the displayed content in real time to respond to scene changes, significantly enhancing the user experience. The image content displayed by the LED array can meet the user's personalized needs, providing a unique lighting and visual experience.

Although the present invention has been disclosed above by way of embodiments, they are not intended to limit the present invention. Any person having ordinary skill in the art may make slight modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of the attached patent application.

100: Electronic device 110, 110_1-110_8: LED array 120: Image sensor 130: Controller 131: PIO device 140: Storage device 150: Processor B1: Cover C1: Chassis db1: Database GPIO1-GPIO8: GPIO pins Img1: Scene image ImgT1: Target image P1: Peripheral electronic device S310, S320, S330, S340, S601-S604: Steps

Figure 1 is a block diagram of an electronic device according to an embodiment of the present invention. Figure 2A is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 2B is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 2C is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 3 is a flow chart of a method for dynamically displaying an image according to an embodiment of the present invention. Figure 4 is a flow chart of determining a target image according to an embodiment of the present invention. Figure 5 is a flow chart of determining a target image according to an embodiment of the present invention. Figure 6 is a flow chart of controlling a light-emitting diode array to display a target image according to an embodiment of the present invention. Figure 7 is a schematic diagram of a controller and a light-emitting diode array according to an embodiment of the present invention.

S310~S340: Steps

Claims (20)

A method for dynamically displaying an image is applicable to an electronic device including a light-emitting diode array and an image sensor. The method comprises: capturing a scene image through the image sensor; performing object recognition on the scene image to obtain scene information; determining a target image based on the scene information; and controlling the light-emitting diode array to display the target image. The light-emitting diode array includes a plurality of light-emitting diodes, and the step of controlling the light-emitting diode array to display the target image includes: controlling the light-emitting diode array to display the target image through a human-machine interface (HMI). The device transmits the luminous parameters of each of the plurality of light-emitting diodes to a controller through a High Identification Device (HID) report, so that the controller controls each of the plurality of light-emitting diodes to emit light according to the luminous parameters of each of the plurality of light-emitting diodes. A method for dynamically displaying images as described in claim 1, wherein the step of performing object recognition on the scene image to obtain the scene information includes: performing object recognition on the scene image using a convolutional neural network model; and obtaining an object type of a target scene object output by the convolutional neural network model, wherein the scene information includes the object type of the target scene object. In the method for dynamically displaying images as described in claim 2, the step of obtaining the object type of the target scene object output by the convolutional neural network model includes: obtaining multiple scene objects recognized by the convolutional neural network model; and selecting the target scene object from the multiple scene objects to obtain the object type of the target scene object. The method for dynamically displaying images as described in claim 2, wherein the step of determining the target image based on the scene information includes: selecting the target image from multiple preset images based on the object type of the target scene object. A method for dynamically displaying images as described in claim 2, wherein the step of determining the target image based on the scene information includes: determining a text prompt based on the object type of the target scene object; and generating the target image using an image generation model based on the text prompt. In the method for dynamically displaying images as described in claim 1, the step of controlling the LED array to display the target image includes: determining the light-emitting parameters of each of the plurality of LEDs in the LED array according to the target image. The method for dynamically displaying images as described in claim 6, wherein the controller controls the light-emitting diode array through a programmable input/output (PIO) element. A method for dynamically displaying images as described in claim 6, wherein each of the plurality of light-emitting diodes is mapped to one of the plurality of pixels of the target image. The method for dynamically displaying images as described in claim 8, wherein each of the plurality of light-emitting diodes is an RGB light-emitting diode, and the light-emitting parameters of each of the plurality of light-emitting diodes include a red channel value, a green channel value, and a blue channel value. The method for dynamically displaying images as described in claim 1, wherein the light-emitting diode array is disposed on a housing of the electronic device. An electronic device includes: a light-emitting diode array having a plurality of light-emitting diodes; an image sensor; a controller coupled to the light-emitting diode array; a storage device storing a plurality of instructions; and a processor coupled to the image sensor, the controller, and the storage device, configured to: capture a scene image through the image sensor; perform object recognition on the scene image to obtain scene information; determine a target image based on the scene information; and control the light-emitting diode array to display the target image using the controller, wherein the processor is further configured to: display the target image through a human-machine interface (HMI) The controller transmits the luminous parameters of each of the plurality of light-emitting diodes to the controller through a High Identification Device (HID) report, so that the controller controls each of the plurality of light-emitting diodes to emit light according to the luminous parameters of each of the plurality of light-emitting diodes. An electronic device as described in claim 11, wherein the processor is further configured to: perform object recognition on the scene image using a convolutional neural network model; and obtain an object type of a target scene object output by the convolutional neural network model, wherein the scene information includes the object type of the target scene object. The electronic device of claim 12, wherein the processor is further configured to: obtain a plurality of scene objects recognized by the convolutional neural network model; and select the target scene object from the plurality of scene objects to obtain the object type of the target scene object. The electronic device of claim 12, wherein the processor is further configured to: select the target image from a plurality of preset images based on the object type of the target scene object. The electronic device of claim 12, wherein the processor is further configured to: determine a text prompt based on the object type of the target scene object; and generate the target image using an image generation model based on the text prompt. The electronic device of claim 11, wherein the processor is further configured to: determine a light emitting parameter of each of the plurality of light emitting diodes in the light emitting diode array based on the target image. The electronic device of claim 16, wherein the controller controls the LED array via a programmable input/output (PIO) element. The electronic device of claim 16, wherein each of the plurality of light-emitting diodes is mapped to one of the plurality of pixels of the target image. The electronic device of claim 18, wherein each of the plurality of light-emitting diodes is an RGB light-emitting diode, and the light-emitting parameters of each of the plurality of light-emitting diodes include a red channel value, a green channel value, and a blue channel value. The electronic device as described in claim 11, wherein the light-emitting diode array is disposed on a housing of the electronic device.