WO2023154109A1

WO2023154109A1 - Methods and systems for upscaling video graphics

Info

Publication number: WO2023154109A1
Application number: PCT/US2022/052726
Authority: WO
Inventors: Steven Jackson; Chen Li; Hongyu Sun
Original assignee: Innopeak Technology, Inc.
Priority date: 2022-02-10
Filing date: 2022-12-13
Publication date: 2023-08-17
Also published as: CN118843882A

Abstract

The present invention is directed to video processing methods and systems. In a specific embodiment, an input video characterized by a resolution of m rows and n columns is upscaled to an output video characterized by a resolution of 2m rows and 2n columns. An intermediate image based on the input video is generated with m rows and n columns of blocks. Sampling is performed for a first pair of non-adjacent pixels of each block, and a second pair of non-adjacent pixels are calculated using the first pair of non-adjacent pixels and neighboring blocks. There are other embodiments as well.

Description

METHODS AND SYSTEMS FOR UPSCALING VIDEO GRAPHICS CROSS-REFERENCES TO RELATED APPLICATIONS [0001] This application claim priority to U.S. Provisional Application No.63/308,860, entitled “CHECKERBOARD SUPER SAMPLING SUBSTITUTION THROUGH GRAPHICS API INTERCEPTION”, filed February 10, 2022, which is commonly owned and incorporated by reference herein for all purposes. TECHNICAL FIELD [0002] The present invention is directed to video processing methods and systems. BACKGROUND OF THE INVENTION [0003] With the increasing popularity of video streaming platforms and video games, the demand for displaying videos and video game graphics on all types of display devices continues to grow rapidly. The resolutions of display devices also continue to grow with advancements in display technologies. In these applications, a challenge arises when the resolution of a video frame is lower than the resolution of the display device. In such cases, the video frame needs the be upscaled to be displayed on the higher resolution display. [0004] Conventional video processing systems include those implementing machine learning to perform the video upscaling process. However, those systems tend to consume substantially more power for deblocking due to machine learning and AI up-scale requirements. Such systems may also require dedicated hardware acceleration to run at desirable performance levels. Further, these conventional systems may have limited graphics processing unit (GPU) and application programming interface (API) support and may require custom integration via the relevant software development kit (SDK). Other conventional systems have high memory storage and bandwidth requirements and other performance issues as well. [0005] From the above, it can be seen that improved methods and systems for video processing are highly desirable. BRIEF SUMMARY OF THE INVENTION [0006] The present invention is directed to video processing methods and systems. In a specific embodiment, an input video characterized by a resolution of m rows and n columns is upscaled to an output video characterized by a resolution of 2m rows and 2n columns. An intermediate image based on the input video is generated with m rows and n columns of blocks. Sampling is performed for a first pair of non-adjacent pixels of each block, and a second pair of non-adjacent pixels are calculated using the first pair of non-adjacent pixels and neighboring blocks. There are other embodiments as well. [0007] Embodiments of the present invention can be implemented in conjunction with existing systems and processes. For example, the video processing methods and systems according to the present invention can be used in a wide variety of applications, including video streaming, client-side media players, and online media platforms. Additionally, various techniques according to the present invention can be adopted into existing systems via architectures that support multi-sample anti-aliasing (MSAA) textures and include extension ARB_sample_locations. There are other benefits as well. [0008] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions. One general aspect includes a method for video upscaling. The method includes receiving an input video, wherein the input video includes a first frame and a second frame, the first frame being characterized by a dimension of m rows by n columns of pixels. The method also includes storing the first frame and the second frame at a first buffer. The method also includes providing a first stencil image at a second buffer, the first stencil image including m rows by n columns of pixel blocks, the pixel blocks including a first block and a second block and a third block, the first block being adjacent to the second block, the first block including two top pixels and two bottom pixels. The method also includes writing values to a first top pixel and a first bottom pixel of the first block, the first top pixel being diagonally positioned relative to the first bottom pixel. The method also includes calculating a first pixel value of a second top pixel of the first block, the first pixel value being based at least on the first top pixel and the first bottom pixel. The method also includes generating a first output image associated with the first stencil image, the first output image being characterized by a dimension of 2m rows by 2n columns of pixels. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. [0009] Implementations may include one or more of the following features. The method where the first buffer may include a frame buffer and the second buffer may include a stencil buffer. The first pixel value is further based on adjacent pixels of the second block and the third block. The method may include performing sampling to generate the values of the first top pixel and the first bottom pixel. The method may include calculating a pixel value of a second bottom pixel of the first block, the pixel value being based at least on the first top pixel and the first bottom pixel. The method may further include providing a second stencil image at the second buffer, the second stencil image including a fourth block positioned at a same location as the first block, the fourth block including a third top pixel and a third bottom pixel, the third top pixel being positioned at a same location of the second top pixel, the third bottom pixel being positioned at a same location of the second bottom pixel. The method may also include calculating a second pixel value of a fourth top pixel of the second block, the second pixel value being based at least on the third top pixel and the third bottom pixel. The method may additionally include generating a second output image associated with the second stencil image. The method may include storing the first output image. The method may include enabling a super sampling mode. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium. [0010] One general aspect includes a method for processing video. The method includes receiving a request for an output video, the output video being characterized by a dimension of 2m rows by 2n columns of pixels. The method also includes providing an input video, the input video including a first frame and a second frame, the first frame being characterized by a dimension of m rows by n columns of pixels. The method also includes storing the first frame and the second frame at a first buffer. The method also includes providing a first stencil image at a second buffer, the first stencil image including m rows by n columns of pixel blocks, the pixel blocks including a first block and a second block and a third block, the first block being adjacent to the second block and the second block, the first block including two top pixels and two bottom pixels. The method also includes writing values to a first top pixel and a first bottom pixel of the first block, the first top pixel being diagonally positioned relative to the first bottom pixel. The method also includes calculating a first pixel value of a second top pixel of the first block, the pixel value being based at least on the first top pixel and the first bottom pixel. The method also includes generating a first output image associated with the first stencil image, the first output image being characterized by a dimension of 2m rows by 2n columns of pixels. The method also includes providing the output video, wherein the output video may include the first output image. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. [0011] [0008] Implementations may include one or more of the following features. The method where the first buffer may include a data structure for storing video data at a dimension of m rows by n columns of pixels. The method further may include rendering the input video at a dimension of m rows by n columns of pixels. The method further may include receiving a request to enable an upscaling process 125. The method further may include performing sampling to obtain the values for the first top pixel and the first bottom pixel. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium. [0012] The video processing system includes a first data buffer that may include a first data structure to store video at a dimension of m rows by n columns of pixels, the first data buffer being configured to store an input video, the input video including a first frame and a second frame, the first frame being characterized by a dimension of m rows by n columns of pixels. The system also includes a second data buffer, that may include a second data structure, to store video at a dimension of 2m rows by 2n columns of pixels, the second data structure being configured to store a first stencil image at a second buffer, the first stencil image including m rows by n columns of pixel blocks, the pixel blocks including a first block and a second block and a third block, the first block being adjacent to the second block and the second block, the first block including two top pixels and two bottom pixels. The system also includes a data bus. The system also includes a processor coupled to the first data buffer and the second data buffer via a data bus. The process is configured to write values to a first top pixel and a first bottom pixel of the first block, the first top pixel being diagonally positioned relative to the first bottom pixel. The process is also configured to calculate a first pixel value of a second top pixel of the first block, the pixel value being based at least on the first top pixel and the first bottom pixel, and generate an output video, which may include the first stencil image, wherein the output video is characterized by a dimensions of 2m rows by 2n columns of pixels. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. [0013] Implementations may include one or more of the following features. The video processing system where the first data buffer may include a random access memory. The video processing system may include a graphical processing unit configured to generate the input video. The video processing system may include a communication interface coupled to a display, the display being characterized by a resolution of at least 2m rows by 2n columns of pixels. The video processing system may include a storage configured to store instruction executable by the processor. The video processing system may include interpolating the first stencil image. The video processing system may include generating a second stencil image based on the second frame. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium. [0014] It is to be appreciated that embodiments of the present invention provide many advantages over conventional techniques. Among other things, embodiments of the present invention can reduce power consumption and increase frame rate while maintaining a desired resolution in video applications. The present invention can also remove the maintenance workload for certain aspects of video processing from the application/game developer. Additionally, the present invention provides a new performance improvement technique via image up-scaling that can be implemented with application programming interfaces (APIs), such as OpenGL, and on mobile platforms. [0015] Embodiments of the present invention can be implemented in conjunction with existing systems and processes. For example, the video processing methods and systems according to the present invention can be used in a wide variety of applications, including video streaming, client-side media players, and online media platforms. Additionally, various techniques according to the present invention can be adopted into existing systems via architectures that support multi-sample anti-aliasing (MSAA) textures and include extension ARB_sample_locations. There are other benefits as well. [0016] The present invention achieves these benefits and others in the context of known technology. However, a further understanding of the nature and advantages of the present invention may be realized by reference to the latter portions of the specification and attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS [0009] Figure 1 is a simplified block diagram illustrating a video processing system according to embodiments of the present invention. [0010] Figure 2 is a simplified block diagram illustrating software modules for video processing according to embodiments of the present invention. [0011] Figure 3 is a simplified flow diagram illustrating a method of video processing according to embodiments of the present invention. [0012] Figure 4 is a simplified diagram video texture blocks according to embodiments of the present invention. [0013] Figure 5 is a simplified diagram illustrating a super sampling block according to embodiments of the present invention. [0014] Figure 6 is a simplified diagram illustrating super sampling processing with pixel blocks according to embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION [0015] The present invention is directed to video processing methods and systems. In a specific embodiment, an input video characterized by a resolution of m rows and n columns is upscaled to an output video characterized by a resolution of 2m rows and 2n columns. An intermediate image based on the input video is generated with m rows and n columns of blocks. Sampling is performed for a first pair of non-adjacent pixels of each block, and a second pair of non-adjacent pixels are calculated using the first pair of non-adjacent pixels and neighboring blocks. There are other embodiments as well. [0016] The following abbreviations are used hereafter: • CBSS: Checkerboard Super Sampling • CPU: Central Processing Unit • GPU: Graphics Processing Unit • API: Application Programming Interface • OpenGL: Open Graphics Library • DX11/DX12: DirectX 11, DirectX 12 • MSAA: Multisample Anti-Aliasing • VR: Virtual Reality • AI: Artificial Intelligence [0017] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions. [0018] In various embodiments, the present invention provides a CBSS mechanism, which is a method for increasing the performance of the rendering process on the GPU at a specified output resolution. For example, the process may be transparent to the application layer, and does not require any additional integration and maintenance by application developers. In an implementation, CBSS includes five stages: (1) FBO Replace, (2) Checkerboard Stencil, (3) Main Color Pass, (4) Resolve, and (5) Upscale. For example, the process uses an MSAA texture for a scene render process to achieve the checkerboard render pattern. When scene rendering is complete, the checkerboard holes are then filled by calculating the most appropriate value. The resolved image may be manually up-scaled from the MSAA texture to a non-MSAA texture for displaying of the frame. [0019] The following description is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a wide range of embodiments. Thus, the present invention is not intended to be limited to the embodiments presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. [0020] In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without necessarily being limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention. [0021] The reader’s attention is directed to all papers and documents which are filed concurrently with this specification and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification, (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features. [0022] Furthermore, any element in a claim that does not explicitly state “means for” performing a specified function, or “step for” performing a specific function, is not to be interpreted as a “means” or “step” clause as specified in 35 U.S.C. Section 112, Paragraph 6. In particular, the use of “step of” or “act of” in the Claims herein is not intended to invoke the provisions of 35 U.S.C.112, Paragraph 6. [0023] Please note, if used, the labels left, right, front, back, top, bottom, forward, reverse, clockwise and counterclockwise have been used for convenience purposes only and are not intended to imply any particular fixed direction. Instead, they are used to reflect relative locations and/or directions between various portions of an object. [0024] Figure 1 is a simplified block diagram illustrating a video processing system 100 according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. [0025] As shown, the video processing system 100 includes a first data buffer 110 and a second data buffer 120 coupled via a data bus 130 to a processor 140. The elements of system 100 can be configured together to perform a video upscaling process on an input video to produce an output video for a higher resolution, as further described below. [0026] The first data buffer 110 can include a first data structure 115 to store video at a dimension of m rows by n columns of pixels, where m and n are integers greater than zero. This first data buffer 110 can be configured to store an input video at the first data structure 115, including at least a first frame and a second frame. The first frame can be characterized by a dimension of m rows by n columns of pixels. In a specific example, the first data buffer 110 includes a random access memory (RAM), or other similar memory storage device 142. [0027] Similarly, the second data buffer 120 includes a second data structure 122 to store video at a dimension of 2m rows by 2n columns of pixels. This second data structure 122 can be configured to store a first stencil image at the second data structure, which includes m rows by n columns of pixel blocks. The pixel blocks can include a first block and a second block and a third block. In an example, the first block is spatially positioned adjacent to the second block and the third block, In a specific example, the first block includes two top pixels and two bottom pixels. The system 100 can also generate a second stencil image based on the second frame. In other examples, the system 100 can generate a plurality of stencil images based on a plurality of frames of the input video. These stencil images can also be stored in the second data structure 122 of the second data buffer 120. [0028] The system 100 can include a storage 142 (e.g., RAM, hard drive, flash drive, etc.) coupled to the processor 140 and configured to store instructions executable by the processor 140. In an example, the processor 140 is configured to write values to a first top pixel and a first bottom pixel of the first block, calculate a first pixel value of a second top pixel of the first block based at least on the first top pixel and the first bottom pixel, and generate an output video, second data structure 122, being characterized by a dimension of 2m rows by 2n columns of pixels and including the first stencil image. The first top pixel is diagonally positioned relative to the first bottom pixel. Also, the processor 140 can be configured to interpolate, via the second data structure 122, the first stencil image, the second stencil image, or any number of a plurality of stencil images for an input video having a plurality of frames. [0029] This system 100 can also include a graphical processing unit (GPU) 150 and a communication interface 160 coupled to the processor 140. The GPU 150 can be coupled to the first data buffer and configured to generate the input video. And, the communication interface 160 can be coupled to a display 170 that is characterized by a resolution of at least 2m rows by 2n columns of pixels. Other embodiments of this system include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. [0030] Figure 2 is a simplified block diagram illustrating software modules for video processing according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. [0031] As shown, this block diagram 200 includes an application/game software module 210 coupled together with an application programming interface (API) module 220 and a video output module 230. Depending on the needs of the application/game software module 210, the API module 220 can be configured to enable services for video streaming, client-side video players, online media platforms, and the like. The application/game module 210 can use these services to display desired media content at the video output module 230 through a render module that generates the media content to be displayed in a display device 250. [0032] In this case, the block diagram includes a first render module 242 and a second render module 244 configured in separate pathways between the API module 220 and the video output module 230. The API module 220 can choose between rendering an input video at the first render module 242 or the second render module 244. In a specific example, the first render module 242 is configured as a full-resolution 3D render module, and the second render module 244 is configured as a ½ render module with a checkerboard super-sampling (CBSS) mechanism. [0033] According to an example, CBSS is a technique for up-scaling a half-resolution checkerboard frame to a standard resolution. The CBSS functionality acts as a layer between the application and driver intercepting API calls (e.g., OpenGL, or the like) and modifying the actions performed. In particular, CBSS can replace a main color pass frame buffer with a multi-sample anti-aliasing (MSAA) variant with half-width and half-height of the original. Upon 3D scene render completion, CBSS up-scales a half-resolution rendered frame to a standard resolution output frame without additional data. Further details of the CBSS functionality and other video processing techniques are discussed with reference to Figures 3- 6. [0034] Figure 3 is a simplified flow diagram illustrating a method 300 of video processing according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. For example, one or more steps may be added, removed, replaced, modified, rearranged, repeated, and/or overlapped, and they should limit the scope of claims. [0035] As shown, the method 300 includes receiving a request for an output video, step 302. This output video can be characterized by a dimension of 2m rows by 2n columns of pixels, where m and n are integers greater than zero. Once the request is received, the method includes providing an input video, step 304. This input video can include at least a first frame and a second frame. The first frame can be characterized by a dimension of m rows by n columns of pixels. [0036] The first and the second frames can be stored at a first buffer. The first buffer can include a data structure (e.g., a frame buffer, or the like) for storing video data at a dimension of m rows by n columns of pixels. In an example, the method can also include rendering the input video at a dimension of m rows by n columns of pixels. This rendering process can be used to produce the first and second frames. [0037] In other examples, the input video can include a plurality of frames, and each such frame can be characterized by a dimension of m rows by n columns of pixels. These frames can also be stored at the first buffer in preparation for processing and/or transmission. Also, the method can include receiving a request to enable an upscaling process. [0038] In step 308, the method includes providing a first stencil image at a second buffer (e.g., a stencil buffer, or the like). The first stencil image can include m rows by n columns of pixel blocks. These pixel blocks can include at least a first block and a second block and a third block. In an example. the first block can be spatially positioned adjacent to the second block and the third block. In a specific example, the first block includes at least two top pixels and at least two bottom pixels. For example, a 4xMSAA frame buffer object (FBO) may be introduced to replace the FBO for the main color pass process, where the non-MSAA will be the default main color pass texture. It is to be appreciated that the use of MSAA frame buffer is advantageous to capture the additional texel samples over the default non- MSAA texture. Further details of a frame buffer object replacement process according to an example of the present invention are discussed with reference to Figure 4. [0039] In step 310, the method includes writing values to a first top pixel and a first bottom pixel of the first block. In this case, the first top pixel is diagonally positioned relative to the first bottom pixel. In an example, the method can also include a sampling process to obtain the values for the first top pixel and the first bottom pixel. Further details of a checkerboard stencil pass process according to an example of the present invention is discussed with reference to Figure 5. [0040] In step 312, the method includes calculating a first pixel value of a second top pixel of the first block. This pixel value can be based at least on the first top pixel and the first bottom pixel. In an example, the pixel value can be further based on adjacent pixels of the second block and the third block. Further, the method can include calculating a pixel value of a second bottom pixel of the first block. This pixel value can also be based at least on the first top pixel and the first bottom pixel. Further details of a resolve pass process according to an example of the present invention are discussed with reference to Figure 6. [0041] In an example, the method can include providing a second stencil image at the second buffer. This second stencil image can include a fourth block positioned at a same location as the first block, and this fourth block can include a third top pixel and a third bottom pixel. The third top pixel can be spatially positioned at a same location of the second top pixel, and the third bottom pixel can be spatially positioned at a same location of the second bottom pixel. With the second stencil image, the method can include calculating the second pixel value of a fourth top pixel of the second block. This second pixel value can be based at least on the third top pixel and the third bottom pixel. [0042] In step 314, the method includes generating a first output image associated with the first stencil image. This first output image can be characterized by a dimension of 2m rows by 2n columns of pixels. In the case with the second stencil image, the method can include generating a second output image associated with the second stencil image. In other examples, additional stencil images can be provided and additional output images associated with these stencil images can be generated. The method can also include storing the first output image, the second output image, or any additional output image in an output buffer, the video output module, or the like. Further details of an up-scale pass process according to an example of the present invention is discussed below. [0043] The method can include enabling a super sampling mode. Following the previous steps, the method can include providing the output video comprising the first output image. In an example, this output video can be sent by a GPU to a display device using a communication device (see Figure 1). Or, the output video can be sent to a video output module (see Figure 2) that is configured to transmit the video to a display device. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium. [0044] Figure 4 is a simplified diagram of video texture blocks 400 according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. [0045] As shown, the graph 401 on the left-side represents a 4xMSAA (640x360) video texture block, while the graph 402 on the right-side represents a non-MSAA (1280x720) video texture block. As discussed previously, the CBSS mechanism can introduce an MSAA frame buffer object (FBO) to replace the FBO for the main color pass process (i.e., default frame render process). This 4xMSAA FBO can contain a 4xMSAA texture 401 with half- width and half-height as the original main color pass texture used by an application/game. The 4xMSAA texture 401 has the same number of samples (in this case, the total is 921,600) as the original main color pass texture. Thus, the maximum number of possible fragments in each texture will have a 1-to-1 relationship, which can be seen by comparison of the left-side graph 401 to the right-side graph 402. Here, 4xMSAA 401 represents the CBSS texture, and the non-MSAA 402 represents the default main color pass texture that is replaced. The color values of the 4xMSAA texture 401 are used to represent the different samples as there are four samples in any given MSAA pixel. [0046] Figure 5 is a simplified diagram illustrating a super sampling block 500 according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. As shown, the video texture block 510, denoted as T(x, y) with cells containing samples S0-S3, is used in an example of the present invention in a checkerboard stencil pass process. In an example, this checkerboard stencil pass technique is responsible for writing values to the stencil buffer in a checkerboard pattern. Considering the 4xMSAA texture 401, samples opposite each other will be stenciled out. For example, in the super sampling block 500, either the pair of (S1, S2) or (S0, S3) will be stenciled on any given frame. In an example, the CBSS mechanism (e.g., as illustrated in Figure 2) can be configured with at least two modes: (1) using the same checkerboard pattern for each frame, and (2) using an alternate checkerboard pattern for each frame. For proper functionality with the main color pass process, the system using the CBSS mechanism will require super sampling to be enabled. In some cases, a depth-based method may be employed for blocking out the checkerboard pattern. For example, a depth value may be contained at the time when a sample fragment is written and stored in a depth buffer. By comparing the depth value between the sample fragments, the samples to be blocked out can be determined. For instance, the sample that has a greater depth value may be discarded as it is further away from the viewer. [0047] Considering the previous example, outside of binding the 4xMSAA 401 render target instead of a standard non-MSAA 402 buffer, the main color pass process should otherwise remain unchanged. Preparations made to the render target depth buffer (see method 300 in Figure 3) in the checkboard stencil pass 510 can allow the main color pass process to maintain standard functionality. [0048] According to an example, the present invention implements a resolve pass process that is responsible for filling the checkboard holes created by the checkboard stencil pass process 510 and the resulting stencil testing. Rather than performing a full-screen resolve by invoking every sample fragment, this example implementation involves preparing a depth test that matches the values written in the checkerboard stencil pass process 510. This implementation would only invoke sample fragments not modified during the main color pass process (i.e., sample fragments with image holes). [0049] Figure 6 is a simplified diagram illustrating super sampling processing with pixel blocks 600 according to embodiments of the present invention. This diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. [0050] As shown in the pixel block configuration 600, there are four pixel blocks arranged in a 2x2 configuration with the blocks denoted by T(x, y), T(x+1, y), T(x, y-1), and T(x+1, y- 1). Each of these pixel blocks includes four quadrants denoted by Q(0), Q(1), Q(2), and Q(3). In an example, resolving each quadrant Q([0 -3]) can use neighboring pixel data for the resolve process because the diagonal sample data is already rendered. For the alternating checkerboard implementation, resolving the quadrants can include reading all available diagonal sample values, performing an interpolation process, and comparing the result to the rendered value from the previous frame to calculate the current frame value. For the non- alternating checkboard implementation, resolving the quadrants can include reading all available diagonal sample values, performing an interpolation process as the calculated value for the current frame. [0051] In an example, the present invention implements an up-scale pass process that is responsible for scaling the half-resolution resolved frame to a target resolution. In a specific example, this scaling can be from a 4xMSAA 401 main color pass frame to a color back buffer frame. In this case, each pixel in the 4xMSAA 401 texture has four samples which will be expanded to a 2x2 pixel region in the color back buffer. In other examples, the techniques described previously can apply to scaling from different frame sizes and blocks of different dimensions. Those of ordinary skill in the art will recognize variations, modifications, and alternatives to these methods and devices for video processing. [0052] It is to be appreciated that embodiments of the present invention provide many advantages over conventional techniques. Among other things, embodiments of the present invention can reduce power consumption and increase frame rate while maintaining a desired resolution in video applications. The present invention can also remove the maintenance workload for certain aspects of video processing from the application/game developer. Additionally, the present invention provides a new performance improvement technique via image up-scaling that can be implemented with application programming interfaces (APIs), such as OpenGL, Vulkan, and on mobile platforms. [0060] While the above is a full description of the specific embodiments, various modifications, alternative constructions and equivalents may be used. Therefore, the above description and illustrations should not be taken as limiting the scope of the present invention which is defined by the appended claims.

Claims

WHAT IS CLAIMED IS: 1. A method for video upscaling, the method comprising: receiving an input video, the input video including a first frame and a second frame, the first frame being characterized by a dimension of m rows by n columns of pixels; storing the first frame and the second frame at a first buffer; providing a first stencil image at a second buffer, the first stencil image including m rows by n columns of pixel blocks, the pixel blocks including a first block and a second block and a third block, the first block being adjacent to the second block and the third block, the first block including two top pixels and two bottom pixels; writing values to a first top pixel and a first bottom pixel of the first block, the first top pixel being diagonally positioned relative to the first bottom pixel; calculating a first pixel value of a second top pixel of the first block, the first pixel value being based at least on the first top pixel and the first bottom pixel; and generating a first output image associated with the first stencil image, the first output image being characterized by a dimension of 2m rows by 2n columns of pixels.

2. The method of claim 1 wherein the first buffer comprises a frame buffer and the second buffer comprises a stencil buffer.

3. The method of claim 1 wherein the first pixel value is further based on adjacent pixels of the second block and the third block.

4. The method of claim 1 further comprising performing sampling to generate the values of the first top pixel and the first bottom pixel.

5. The method of claim 1 further comprising calculating a pixel value of a second bottom pixel of the first block, the pixel value being based at least on the first top pixel and the first bottom pixel.

6. The method of claim 5 further comprising: providing a second stencil image at the second buffer, the second stencil image including a fourth block positioned at a same location as the first block, the fourth block including a third top pixel and a third bottom pixel, the third top pixel being positioned at a same location of the second top pixel, the third bottom pixel being positioned at a same location of the second bottom pixel; calculating a second pixel value of a fourth top pixel of the second block, the second pixel value being based at least on the third top pixel and the third bottom pixel; and generating a second output image associated with the second stencil image.

7. The method of claim 1 further comprising storing the first output image.

8. The method of claim 1 further comprising enabling a super sampling mode.

9. A method for processing video, the method comprising: receiving a request for an output video, the output video being characterized by a dimension of 2m rows by 2n columns of pixels; providing an input video, the input video including a first frame and a second frame, the first frame being characterized by a dimension of m rows by n columns of pixels; storing the first frame and the second frame at a first buffer; providing a first stencil image at a second buffer, the first stencil image including m rows by n columns of pixel blocks, the pixel blocks including a first block and a second block and a third block, the first block being adjacent to the second block and the third block, the first block including two top pixels and two bottom pixels; writing values to a first top pixel and a first bottom pixel of the first block, the first top pixel being diagonally positioned relative to the first bottom pixel; calculating a first pixel value of a second top pixel of the first block, the pixel value being based at least on the first top pixel and the first bottom pixel; generating a first output image associated with the first stencil image, the first output image being characterized by a dimension of 2m rows by 2n columns of pixels; and providing the output video comprising the first output image.

10. The method of claim 9 wherein the first buffer comprises a data structure for storing video data at a dimension of m rows by n columns of pixels.

11. The method of claim 9 further comprises rendering the input video at a dimension of m rows by n columns of pixels.

12. The method of claim 9 further comprises receiving a request to enable an upscaling process.

13. The method of claim 9 further comprises performing sampling to obtain the values for the first top pixel and the first bottom pixel.

14. A video processing system comprising: a first data buffer comprising a first data structure to store video at a dimension of m rows by n columns of pixels, the first data buffer being configured to store an input video, the input video including a first frame and a second frame, the first frame being characterized by a dimension of m rows by n columns of pixels; a second data buffer comprising a second data structure to store video at a dimension of 2m rows by 2n columns of pixels, the second data buffer being configured to store a first stencil image, the first stencil image including m rows by n columns of pixel blocks, the pixel blocks including a first block and a second block and a third block, the first block being adjacent to the second block and the third block, the first block including two top pixels and two bottom pixels; a data bus; and a processor coupled to the first data buffer and the second data buffer via the data bus, the process is configured to: write values to a first top pixel and a first bottom pixel of the first block, the first top pixel being diagonally positioned relative to the first bottom pixel; calculate a first pixel value of a second top pixel of the first block, the pixel value being based at least on the first top pixel and the first bottom pixel; and generate an output video comprising the first stencil image, the output video being characterized by a dimension of 2m rows by 2n columns of pixels.

15. The video processing system of claim 14 wherein the first data buffer comprises a random access memory.

16. The video processing system of claim 14 further comprising a graphical processing unit configured to generate the input video.

17. The video processing system of claim 14 further comprising a communication interface coupled to a display, the display being characterized by a resolution of at least 2m rows by 2n columns of pixels.

18. The video processing system of claim 14 further comprising a storage configured to store instruction executable by the processor.

19. The video processing system of claim 14 further comprising interpolating the first stencil image.

20. The video processing system of claim 14 further comprising generating a second stencil image based on the second frame.