TW201902226A - Frame rate adjustment method and image processing device - Google Patents

Abstract

The present invention provides a method for frame rate adjustment and an image processing apparatus. wherein the image processing apparatus includes a first circuitry, a second circuitry, a third circuitry, and a fourth circuitry. The first circuitry determines a frame miss rate according to a current frame rate and a target frame rate of an image signal. The second circuitry decreases the target frame rate to the current frame rate when the frame miss rate is greater than a first threshold. The third circuitry increases the target frame rate to an upper-limit frame rate which is determined according to the frame rendering time or memory bandwidth capability, when the frame miss rate is less than a second threshold which is smaller than the first threshold. The fourth circuitry applies the decreased or increased target frame rate for an image to be displayed.

Description

   Frame rate adjustment method and image processing device   

The present invention relates to frame rate adjustment, and in particular, to a frame rate adjustment method and an image processing device capable of providing a better user experience.

In the field of graphics and multimedia, the frame rate generally refers to the frame rate of the visual content provided by the graphics engine. Depending on the source of the visual content, the frame rate is in the range of 24Hz to more than 100Hz, or 0Hz in the case of static graphics. Optionally, certain types of visual content have a varying frame rate, or the frame rate may vary with the display of different content types. For example, movie material on a television generally has a frame rate of 24 or 25 Hz, and Video material (such as video games) may have a frame rate of 30, 50, or 60 Hz.

Although higher frame rates can provide better visual quality, they generally tend to cause more power consumption. In order to reduce power consumption, the traditional design of frame rate adjustment configures different target frame rates for different application scenarios. However, in the same application scenario, even if there are various events with different display requirements, the target frame rate is static.

Instead of reducing power consumption, there is another traditional design of frame rate adjustment, which is designed to increase the system's capability by increasing the thermal throttling point to avoid frame loss, thereby providing a better user experience. However, this leads to higher power consumption and overheating of the system, which can easily cause the system to enter an unstable state. In the end, the user experience may not improve as expected, but the cost of high power consumption is inevitable.

Therefore, a more robust frame rate adjustment method is desired.

Therefore, the main object of the present invention is to provide a frame rate adjustment method and an image processing apparatus.

An image processing device according to at least one embodiment of the present invention includes: a first circuit for determining a frame loss rate according to a current frame rate and a target frame rate of an image signal; a second circuit for When the frame loss rate is greater than the first threshold, the target frame rate is reduced to the current frame rate; a third circuit is used to change the target frame rate when the frame loss rate is less than the second threshold. The frame rate is increased to an upper frame rate, wherein the second threshold value is smaller than the first threshold, and the upper frame rate is determined according to a frame rendering time or a memory bandwidth capability; and a fourth circuit for The displayed image applies this reduced or increased target frame rate.

A frame rate adjusting method according to at least one embodiment of the present invention is executed by a controller in an image processing apparatus, and the method includes: according to a current frame rate of an image signal generated by the image processing apparatus And the target frame rate to determine the frame loss rate; when the frame loss rate is greater than the first threshold, reduce the target frame rate to the current frame rate; when the frame loss rate is less than the second threshold, Increasing the target frame rate to an upper frame rate, where the second threshold is less than the first threshold, and the upper frame rate is determined based on frame rendering time or memory bandwidth capability; and processing the image The image displayed by the device should apply the reduced or increased target frame rate.

In the embodiment of the present invention, by dynamically adjusting the frame rate, a smooth display of the image can be obtained, thereby providing a better user experience.

10‧‧‧ Controller

20‧‧‧ Cooperative Controller

30‧‧‧Storage Equipment

40‧‧‧display equipment

50‧‧‧ input device

210‧‧‧ Detection Module

220‧‧‧FPS stabilization module

230‧‧‧Graphic Service Module

S310 ~ S390‧‧‧step

The embodiments of the present invention can be more easily understood by reading the following detailed description and the examples made with reference to the accompanying drawings, in which: FIG. 1 is a block diagram of an example of an image processing apparatus according to an embodiment of the present invention; FIG. 2 is a schematic diagram of a software architecture of a dynamic frame rate adjustment method according to an embodiment of the present invention; FIG. 3 is a schematic flowchart of a dynamic frame rate adjustment method according to an embodiment of the present invention; Schematic illustration of frame rendering time.

The following description is for the purpose of illustrating the general principles of the invention and is not meant to be limiting. It should be understood that the embodiments of the present invention may be implemented in software, hardware, firmware, or any combination thereof. As used herein, the terms "including," "including," and / or "containing" indicate the presence of stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or A plurality of other features, wholes, steps, operations, elements, components and / or combinations thereof.

Please note that the term “frame rate” used herein may refer to the rate of the frame of the visual content provided. In a specific embodiment, the frame rate may be expressed as Frames Per Second (FPS).

FIG. 1 is a schematic block diagram of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 100 includes a controller 10, a co-controller 20, a storage device 30, a display device 40, and an input device 50. The cooperative controller 20, the storage device 30, and the input device 50 are all coupled to the controller 10, and the cooperative controller 20 and the storage device 30 are coupled to each other and coupled to the display device 40. For example, the image processing apparatus 100 may be a mobile phone, a tablet PC (Personal Computer), a notebook computer, a game console, or any electronic device with a display function.

The controller 10 may be a general-purpose processor, a CPU (Central Processing Unit, central processing unit), an MCU (Micro-Control Unit, micro-control unit), a DSP (Digital Signal Processor, digital signal processor), an application processor, etc. The controller 10 includes various circuits for providing data processing and calculation, controlling the cooperative controller 20 for image rendering, storing data to the storage device 30, retrieving data from the storage device 30, and / or receiving input from Signals and other functions of the device 50.

The cooperative controller 20 can be used to offload the image processing tasks of the controller 10, thereby allowing the controller 10 to process other processing tasks. The cooperative controller 20 may include one or a plurality of appropriately configured graphics processors, such as a suitably configured discrete or integrated graphics processor (such as a GPU (Graphics Processing Unit)), or a specific device (such as Digital TV or handheld / mobile device) processor. Specifically, the cooperative controller 20 includes various circuits for providing image processing and calculation, receiving control signals from the controller 10, storing data to the storage device 30, retrieving data from the storage device 30, and / or, sending a A series of frame materials (such as text messages, graphics, images, etc.) to the storage device 30 or display device 40 and other functions. For example, the frame material may be sent directly to the display device 40, or sent to the storage device 30 from which the display device 40 may retrieve it.

Those having ordinary knowledge in the field of the invention will understand that the circuits in the controller 10 and the cooperative controller 20 generally include a plurality of transistors configured to control the operation of the circuit according to the functions and operations described herein. It can be further understood that the specific structure or interconnection of the transistor is generally determined by a compiler, such as an RTL (Register Transfer Language, RTL) compiler. The processor operates the RTL compiler on a script very similar to assembly language code to compile the script into a form for layout or fabrication of the final circuit. In fact, RTL is known for its role and use in facilitating the design of electronic and digital systems.

The storage device 30 is a non-volatile machine-readable storage medium, including: a memory, such as a flash memory or a RAM (Random Access Memory), or a magnetic storage device, such as a hard disk or a magnetic tape. , Or optical discs, or any combination thereof, is used to store data, such as frame data, and program codes for communication protocols, applications (such as game applications), and methods of embodiments of the present invention.

Specifically, the controller 10 or the cooperative controller 20 can load and execute program code of the method in the embodiment of the present invention.

Optionally, the image processing apparatus 100 may further include a frame rate controller coupled to at least one of the controller 10, the cooperative controller 20, the storage device 30, and the display device 40, and is dedicated to controlling the frame rate. Specifically, the frame rate controller may include various circuits for providing a function of frame rate control, for example, providing frame rate control by loading and executing program code from the storage device 30 of the method of the embodiment of the present invention.

The display device 40 may be a CRT (Cathode Ray Tube) display, an LCD (Liquid-Crystal Display), an LED (Light-Emitting Diode) display, or an EPD (Electronic Paper Display, electronic Paper display), etc., for providing a display function based on frame data received or read from the cooperative controller 20 or the storage device 30. Optionally, the display device 40 may further include one or a plurality of touch sensors disposed thereon or below, for sensing an approach of a touch, a contact, or an object such as a finger or a stylus.

The input device 50 may include one or a plurality of buttons, a keyboard, a mouse, a touch pad, a video camera, a microphone, and / or a speaker, etc., serving as an MMI (Man-Machine Interface) for interacting with a user.

It can be understood that, the elements described in the embodiment in FIG. 1 are for illustrative purposes only, and are not meant to limit the embodiments of the present invention. For example, the display device 40 may be implemented by a display external to the image processing apparatus 100, or the image processing apparatus 100 may further include: a GPS (Global Positioning System) for providing position information; a power source (such as a battery) , For providing energy to other components; and / or, a wireless transceiver for providing wireless communication functions, and so on.

FIG. 2 is a schematic diagram of a software architecture of a dynamic frame rate adjustment method according to an embodiment of the present invention. The software architecture includes: a detection module 210, an FPS stabilization module 220, and a graphics service module 230, where each module can be a software module implemented using program code and controlled by a controller (such as the controller 10, a cooperative controller 20, or frame rate controller), so as to cooperate with each other to perform dynamic frame rate adjustment.

The detection module 210 is responsible for collecting statistical information from hardware components of the graphics processing apparatus 100. The hardware components include: a controller 10, a cooperative controller 20, and a storage device 30. Specifically, the statistical information includes: a frame rate currently used for image rendering, a frame rendering time required by the controller 10 and the cooperative controller 20, and a current and maximum OPP (Operating) of the controller 10 and the cooperative controller 20 Performance Points), and the throughput and maximum available bandwidth of the storage device 30 (such as memory throughput and maximum available memory bandwidth).

The FPS stabilization module 220 is responsible for dynamically adjusting the target frame rate according to the statistical information provided by the detection module 210. Specifically, the FPS stabilization module 220 determines the frame loss rate by subtracting the current frame rate from the target frame rate and dividing the subtraction result by the target frame rate, and comparing the frame loss rate with at least two thresholds. Compare and adjust the target frame rate according to the comparison result and the statistical information provided by the detection module 210.

The graphics service module 230 is responsible for applying the target frame rate to a graphics service, such as a game application, which generates an image signal to be displayed by the image processing apparatus 100.

FIG. 3 is a schematic flowchart of a dynamic frame rate adjustment method according to an embodiment of the present invention. In this embodiment, the method is applied to an image processing apparatus (eg, the image processing apparatus 100). Specifically, the controller of the image processing apparatus executes the method, such as the controller 10, the cooperative controller 20, or a frame rate controller other than the controller 10 and the cooperative controller 20. The controller includes: A circuit for providing data processing and calculation functions to perform the method of the embodiment of the present invention.

In step S310, the image processing apparatus starts a periodic polling timer to count a predetermined period of time.

In step S320, the periodic polling timer expires and the method proceeds to the next step.

In step S330, the graphics processing device determines a frame loss rate according to the current frame rate and the target frame rate. Specifically, the frame loss rate is determined by subtracting the current frame rate from the target frame rate and dividing the subtraction result by the target frame rate.

In one embodiment, the current frame rate refers to the number of frames processed by the Android display manager per second. In another embodiment, the current frame rate refers to the number of frames processed by Surfaceflinger per second. Alternatively, the current frame rate may be determined by the time interval between two consecutive events of the frame stored in the queue buffer.

In step S340, the image processing device determines whether the frame loss rate is greater than the first threshold. If it is greater than that, the method proceeds to step S350. In one embodiment, the first threshold may be configured as 0.02.

In step S350, the image processing device reduces the target frame rate to the current frame rate.

Following step S340, if the frame loss rate is less than or equal to the first threshold, the method proceeds to step S360.

In step S360, the image processing device determines whether the frame loss rate is less than or equal to the second threshold, and if it is less than or equal to the second threshold, the method proceeds to step S370. Specifically, the second threshold is configured to be within a range of 0 to the first threshold. For example, the first threshold may be configured as 0.02 and the second threshold may be configured as 0.01.

In step S370, the image processing device increases the target frame rate to an upper frame rate, where the upper frame rate is determined according to a frame rendering time and / or a memory bandwidth capability. Specifically, the frame rendering time may refer to a time period required for a controller (such as a CPU, a GPU) to process the frame.

FIG. 4 is a schematic diagram of a frame rendering time according to an embodiment of the present invention. As shown in Figure 4, the time period representing the frame rendering time can refer to the actual processing time of the CPU / GPU (labeled as CPU / GUP frame time in Figure 4) plus the frame rendering task is preempted by other threads time.

Memory bandwidth capability refers to the result of dividing the maximum available memory bandwidth by the memory bandwidth used by each frame, where the memory bandwidth used by each frame is equal to the memory throughput divided by the current frame rate.

Optionally, the frame rendering time may refer to the result of multiplying the boost headroom by the time period required for the controller to process the frame. The enhancement margin may be determined by dividing the first performance index by a second performance index, where the first performance index indicates a performance level at which the image processing device operates under the current system capability, and the second performance index indicates that the image processing device is at Performance level of operation at maximum system capacity. The current system capability may refer to the current OPP or DVFS (Dynamic Voltage and Frequency Scaling) of the image processing device, and the maximum system capability may refer to the maximum OPP or DVFS of the image processing device.

For example, the current OPP / DVFS can be represented by the frequency and voltage currently applied to the controller (such as CPU or GPU), such as {frequency = 300MHz, voltage = 1V}, and the maximum OPP / DVFS can be configured by the controller (such as CPU or GPU), such as {frequency = 1GHz, voltage = 1.3V}.

Correspondingly, the first performance index may be represented by a score (such as 1020) obtained by using a benchmark to test an image processing device using the current OPP / DVFS, and the second performance index may be represented by using the benchmark The score (such as 3000) obtained by the image processing device for testing is expressed.

In one embodiment, the inverse of the frame rendering time is used to determine the upper frame rate.

In another embodiment, the upper frame rate is determined by selecting a minimum value among the reciprocal of the frame rendering time and the memory bandwidth capability. For example, the upper frame rate is equal to the minimum of the memory bandwidth capability, the reciprocal of the CPU frame rendering time, and the GPU frame rendering time.

Then go to step S360. If the frame loss rate is greater than the second threshold, the method proceeds to step S380.

In step S380, the image processing apparatus maintains the target frame rate.

Following steps S350, S370, and S380, the method proceeds to step S390.

In step S390, the image processing device applies the adjusted target frame rate to the displayed image, and the method ends.

In view of the above embodiments, it can be understood that the embodiment of the present invention uses the frame loss rate as the main reference for determining whether to adjust the target frame rate and considers various system prompts (such as FPS, CPU / GPU frame rendering time, and Memory bandwidth capability) to determine how to adjust the target frame rate to achieve dynamic frame rate adjustment. Advantageously, the method of the embodiment of the present invention can obtain a smooth display of an image, thereby providing a better user experience.

Although the present application has been described by way of examples and various embodiments, it should be understood that the present application is not limited thereto. Without departing from the scope and spirit of the present application, those with ordinary knowledge in the field to which the invention belongs can still make various changes and modifications. Therefore, the scope of this application should be defined and protected by the scope of the following patent applications and their equivalents.

The use of sequential terms such as "first," "second," and "third" in the scope of a patent application to modify a claimed element does not in itself mean the priority of a claimed element, which is higher in priority or order Another, or chronological order of performing the actions of a method, is only used as a label to distinguish one claimed element with a certain name from another element with the same name (but using sequential terminology).

The above description is only the preferred embodiments of the present invention, and is not intended to limit the present invention. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention shall be included in the protection of the present invention. Within range.

Claims (10)

    An image processing device includes: a first circuit for determining a frame loss rate according to a current frame rate and a target frame rate of an image signal; and a second circuit for when the frame loss rate is greater than At a first threshold, reducing the target frame rate to the current frame rate; a third circuit for increasing the target frame rate to an upper frame rate when the frame loss rate is less than the second threshold Wherein the second threshold value is smaller than the first threshold value, the upper frame rate is determined according to the frame rendering time or the memory bandwidth capability; and a fourth circuit for applying the reduction or Increased target frame rate.         The image processing device according to item 1 of the patent application scope, further comprising: a fifth circuit for maintaining the target frame when the frame loss rate is less than the first threshold value and greater than the second threshold value. rate.         The image processing device according to item 1 of the patent application range, wherein the frame loss rate is obtained by subtracting the current frame rate from the target frame rate and dividing the result of the subtraction by the target frame rate. And ok.         The image processing device according to item 1 of the patent application range, wherein the upper frame rate is determined by using a reciprocal of the frame rendering time.         The image processing device according to item 1 of the scope of patent application, wherein the frame rendering time refers to a time period required for the central processing unit or the graphics processing unit to process the frame.         The image processing device according to item 1 of the scope of patent application, further comprising: a sixth circuit for determining a first performance index of the image processing device operating under the current system capability, and the image processing device The second performance number operating at the maximum system capacity; wherein the frame rendering time represents: the result of dividing the first performance index by the second performance index multiplied by the time period required for the central processing unit or graphics processing unit to process the frame .         The image processing device according to item 1 of the scope of patent application, wherein the upper frame rate is determined by selecting a minimum value among a reciprocal of a frame rendering time and a memory bandwidth capability.         The image processing device according to item 7 of the scope of patent application, wherein the memory bandwidth capability represents a result of dividing a maximum available bandwidth by a memory bandwidth used per frame.         A frame rate adjusting method is executed by a controller in an image processing apparatus. The method includes: determining a frame according to a current frame rate and a target frame rate of an image signal generated by the image processing apparatus. Frame loss rate; when the frame loss rate is greater than the first threshold, reduce the target frame rate to the current frame rate; when the frame loss rate is less than the second threshold, increase the target frame rate to An upper frame rate, wherein the second threshold value is smaller than the first threshold, the upper frame rate is determined according to a frame rendering time or a memory bandwidth capability; and applying the reduction to an image displayed by the image processing device Or increased target frame rate.         The frame rate adjusting method according to item 9 of the scope of patent application, further comprising: maintaining the target frame rate when the frame loss rate is less than the first threshold value and greater than the second threshold value.