RS20180385A1 - Method for detecting packet loss artifacts in an image in a transmitted video signal - Google Patents

Abstract

The process of detecting packet loss artifacts in an image in a transmitted video signal has the novelty that by pre-processing, filtering vertical and horizontal gradients and horizontal and vertical edges, right angles (210) are obtained by overlapping the obtained sharp horizontal edges (240) and sharp vertical edges (245) , that is, macro-blocks (330) are found with detected packet loss artifacts. In addition to the right angle detection step (210), the method is characterized by the way in which the image content differs from the packet loss artifact, which is the tracking of how many consecutive frames the macro-block (330) with the detected packet loss artifact appears at the same location and comparing with the predefined maximum number F of frame repeats at the same location in step (150) of distinguishing the contents from the packet loss artifacts. If packet loss artifacts are found at the same location, that is, if more than F consecutive frames appear in the same macro-block (200), the macro-block (330) with the detected packet loss artifact is "locked" and the algorithm records it as content . The macro-block (200) remains "locked" as long as the packet loss artifact appears in it in consecutive frames, and is "unlocked" again only when no artifact appears in the specified location in the predefined number A of consecutive frames.

Description

A method for detecting image packet loss artifacts in a transmitted video signal

Field of technology to which the invention relates

The invention generally belongs to the field of digital signal processing and digital video communications. The invention primarily belongs to the field of video signal transmission over various communication, i.e. packet, IP networks and "free to air" video signal transmission. The invention also relates to the field of image reproduction due to packet loss, after distortions caused by packet loss itself. The invention is closely related to testing various networks for digital video signal transmission, digital TV receivers (Set Top Box, STB), as well as digital television.

The designations according to the International Patent Classification (IPC) are: H04N7/12, H04N1/409, H04N5/94, H04V1/66.

Technical problem

When transmitting video signals over communication channels, the received signal can be attenuated due to propagation itself, and noise can be superimposed on it. In addition to the signal being distorted by superimposed noise, part of the signal can also be lost due to a certain number of signal packets not arriving at the receiver. Uncompressed digital video signals take up a large amount of resources when stored on a specific data storage medium, as well as when transmitted over a network, which is why signals are compressed according to various video compression standards (for example, MPEG-2, N.264, and N.265) before storage and distribution. Such a compression process introduces distortions into the video and reduces its quality. In addition, when transmitting video signals over heterogeneous networks, packets that carry video information can be lost (Packet Loss). Therefore, if these video packets do not reach the intended receiver, additional artifacts and additional degradation in video quality are introduced due to packet transmission. In applications where it is essential that all packets reach the receiver, retransmission of lost packets may be required to reduce the amount of artifact due to packet loss. However, this increases latency due to the additional time required for retransmission. In applications where retransmission is not required, there is no additional latency, but then the packet loss results in a loss of information at the receiver itself. This loss of information is visually manifested as a loss of packet artifact and significantly affects the perceived quality of the video signal.

The presented invention solves the technical problem of detecting packet loss artifacts on a decoded video frame ("no reference" artifact detection method). When the amount of packet loss artifacts is above a certain limit, the DTV receiver most often freezes part of the image or the entire image, or displays a black screen, or reproduces a degraded (distorted) image when the amount of artifacts is below a certain threshold level. The invention also solves the problem of testing different networks (mobile, IP, etc.) for video signal transmission, in order to increase the quality of the service provided to the end user (information on the amount of artifacts is taken into account to optimize network parameters).

State of the art

Previous solutions describe methods and systems that describe the essence with certain characteristics, while the individual steps described in the presented method do not coincide with the state of the art, and therefore the given steps are key to the presented method.

Patent US 8488915 of the Georgia Tech Research Corporation was published on May 1, 2006, under the title "Automatic video quality measurement system and method based on spatial-temporal coherence metrics". It describes a method for measuring video artifacts using block metrics, where vertical edges that are longer than a predefined value are detected by filtering vertical edges in a certain frame. However, unlike the method described above, in this case, compression artifacts are detected based on the distribution of vertical edges. This is followed by horizontal edge filtering, which detects horizontal edges longer than a predefined value and based on them, network artifacts are found. The patent emphasizes that images with prominent horizontal and vertical edges are further processed in order to obtain only those pixels that belong to edges that are exclusively vertical or horizontal, and that are also longer than a given size.

Patent application US 20060181740, published on December 8, 2004, entitled "Block artifact phenomenon eliminating device and eliminating method thereof", describes a method for removing block artifacts in a decoded image frame, where edge detection is performed and pixels belonging to the block edge, pixels that are part of the object edge and pixels that are not within any edge are determined, based on which appropriate filters are used to remove artifacts in the observed block. In the given patent application, unlike the method being protected, it is considered whether the direction of the edge to which the pixel belongs is 0 or 90 degrees, while the intersection of the edges is not considered in order to detect right angles. Therefore, the patent application determines whether a pixel with a certain orientation is located in the vicinity of the block edge. Based on the absolute value of the edge response for a certain pixel, and based on a predefined threshold, it is determined whether a given pixel belongs to an edge.

Patent US 5621467 published on February 16, 1995 by "Thomson Multimedia S.A", entitled "Temporal-spatial error concealment apparatus and method for video signal processors", describes a video decompression system and a method for generating a block made up of pixels with data for a block that is missing or corrupted. The dominant gradient of the image is determined and spatial interpolation is performed accordingly. A block is then formed with appropriate coefficients, according to predefined criteria. Dominant image features such as edges and gradients are also evaluated. The dominant gradient of the super-block is determined and its angle is recorded. The angle with the largest sum of coefficients is estimated to be dominant.

Patent application US 8130828 of "Microsoft" Corporation, published on April 7, 2006, entitled "Adjusting quantization to preserve non-zero AC coefficients", describes a method of identifying one or more coefficients of the alternating components of each of multiple macroblocks in a video image. For a macroblock of four blocks, a quantization threshold is identified, which involves determining the number of coefficients, the largest amplitude of the coefficients in each of the four blocks, and the smallest amplitude of the coefficient among the largest amplitudes recorded.

The state of the art also includes Ivan Glavota's thesis, defended on September 21, 2016 at the Faculty of Electrical Engineering, Computer Science and Information Technologies in Osijek, entitled "Detecting packet loss in video transmission over a network". The described procedure demonstrates the detection of regular and irregular packet loss artifacts, finding horizontal and vertical edges of macro-blocks, as well as calculating the repetition of image elements from the previous frame, with a specific condition for the existence of artifacts. In post-processing, the detected propagation artifacts are horizontally connected. At the end of the algorithm, the percentage of the frame area containing packet loss artifacts is calculated.

Exposition of the essence of the invention

The presented invention presents several steps of the method that are specific, therefore introducing novelty and will be described in detail below. The aim is to find the most efficient way of finding packet loss artifacts. The essence of the invention is that artifacts are detected in a decoded video frame, without the presence of a reference signal, in several ways (since using only one of the methods leads to the non-detection of a certain number of existing artifacts): by detecting right angles, by detecting paired parallel sharp horizontal edges, by detecting paired parallel sharp vertical edges and by merging undetected macro-blocks containing artifacts, which are located between two or more consecutive groups of macro-blocks with detected packet loss artifacts, whereby the merging takes place only in the horizontal direction (in one row of macro-blocks). Using one of the methods for finding artifacts only finds some of the artifacts, while using all four methods ensures the discovery of those artifacts that are not detected by using only one of the methods, which makes artifact detection more efficient.

far more accurate when losing packets.

Brief description of the invention images

The following images complete the description of the invention:

Figure 1: Process with steps for detecting packet loss artifacts.

Figure 2: Illustration of ideal right angles and angles with deviation.

Figure 3: Connecting macro-blocks to artifacts in one row.

Figure 4: Artifact detection using a) paired parallel sharp horizontal edges and b) paired parallel sharp vertical edges.

Figure 5: Display of detected packet loss artifacts in different ways.

Detailed description of the invention

Some of the standards whose compression principle is based on the processing of macro-blocks 200 are JPEG, N.261, MPEG-1, H.262/MPEG-2, N.263 and MPEG-4. In the transmission of signals compressed in this way, the size of macro-blocks is most often 16x16 pixels (segments of dimensions 16x16). There are different types of packet loss artifacts. Their visual appearance depends on the error correction technique (error concealment) applied during post-processing at the decoder. The artifact is usually found at the boundaries of macro-blocks if it occurred in an / frame (l-frame) or anywhere in the frame if it was caused by the propagation of artifacts from / frames, which is obtained in the case of the predicted R and 8 frames (P-frames, B-frames). In the intended R and V frames, the artifact tends to distort and lose its original characteristics, making it more difficult to detect.

Figure 1 shows the steps of the method to be protected. Step 120 of detecting right angles, step 130 of detecting paired parallel sharp horizontal edges, step 135 of detecting paired parallel sharp vertical edges, and step 140 of merging macroblocks in the horizontal direction represent ways of finding packet loss artifacts.

The packet loss artifact detection process begins with step 100 of acquiring the current video frame, and continues with preprocessing in step 110 of detecting sharp horizontal edges and step 115 of detecting sharp vertical edges. After finding sharp horizontal edges and sharp vertical edges, step 120 of detecting right angles finds right angles and thus detects certain packet loss artifacts that have pronounced right angles. In order to find other artifacts, a step 130 of detecting paired parallel sharp horizontal edges longer than or equal to a predefined number L pixels, spaced apart by a predefined number D1 pixels (can be 16, 32 or 48 pixels), is applied, followed by a step 135 of detecting paired parallel sharp vertical edges longer than or equal to a predefined number M pixels, spaced apart by a predefined number D2 pixels (can be 16 or 32 pixels). The predefined number L pixels is at least eight pixels, while the predefined number M pixels is at least four pixels. After that, a fourth way of finding artifacts is achieved by a step 140 of merging macroblocks that are undetected, with their number being at most four, and located between two consecutive groups of macroblocks with detected packet loss artifacts. The merging takes place only in one row of macro-blocks, that is, in the horizontal direction. Post-processing is performed by step 150 of distinguishing content from packet loss artifacts, in order to reject macro-blocks with incorrectly detected artifacts that do not actually contain artifacts, but are part of the content that has some artifact characteristics. Based on these characteristics, the algorithm previously incorrectly assumed that they are artifacts, while post-processing actually teaches the algorithm that they are not artifacts, but content.

In step 150 of distinguishing content from packet loss artifacts, the number of consecutive occurrences of packet loss artifacts at the location of the same macroblock in consecutive frames is first checked. If the number of consecutive occurrences of artifacts at the same macroblock location is greater than a predefined number F for which the frame is "locked", that macroblock is marked with a LOCK indicator. In this case, the algorithm has incorrectly detected the content of the image as a packet loss artifact, and that macroblock is no longer taken as a macroblock in which the artifact exists (the algorithm has "learned" that it is content). The macroblock then remains marked with a LOCK indicator as long as a packet loss artifact appears in it. After no artifact is detected in a macroblock marked with the LOCK indicator for a predefined number A of consecutive frames for which the frame is "unlocked", the same macroblock will be marked with the UNLOCK indicator, so the algorithm will again consider that macroblock as a location where a packet loss artifact may appear. The procedure ends with step 160 of calculating the percentage of frame coverage by packet loss artifacts, where the given percentage is calculated as the ratio of the number of macroblocks with detected packet loss artifacts to the total number of macroblocks in the entire frame.

Figure 2 shows ideal right angles 210 and angles with a deviation of 250 of several pixels S, which are formed by the intersection 230 of sharp horizontal edges 240 and sharp vertical edges 245. Figure 2 shows cases of found right angles 210 within macro-blocks 200, where packet loss artifacts are detected. The ideal right angle 210 is shown on the left side of Figure 2 and is located in the lower left macro-block 200, while the angle with a deviation of 250 of several pixels is located in the macro-block 200 next to it (the next macro-block 200 to the right). Sharp horizontal edges 240 and sharp vertical edges 245 are located within macroblocks 200 in Figure 2 and paired parallel sharp horizontal edges 240 and sharp vertical edges 245 are observed for artifact detection. In the case of finding an angle with a deviation of 250 of several pixels, if it is defined by edges that have a deviation greater than two pixels, a packet loss artifact will not be detected in a given macroblock 200. Ideal right angles 210 and angles with a deviation of 250 of several pixels are detected with the aim of finding the edge macroblocks 200 of the packet loss artifact itself, which leads to the finding of rectangular objects separated from the background, and it is assumed that these are packet loss artifacts.

Figure 3 for a given frame illustrates the linking of undetected artifacts with detected artifacts in a row of macroblocks 200, i.e. in the horizontal direction. Between one or more macroblocks 330 with detected artifacts, packet loss artifacts are identified in the same row of macroblocks 200 if their number is less than or equal to a predefined value K, which is at most four macroblocks 200. Linking in a row also takes place when the edges of the detected artifacts are not located on the edges of the macroblocks 200, but are shifted so that they are located inside individual macroblocks 300 with undetected artifacts.

Figure 4 a) depicts the case of packet loss artifact detection based on parallel paired sharp horizontal edges 240 longer than a predefined value L, which is eight pixels. Artifact detection is performed at the edges of macroblocks 200 and within individual macroblocks 200, for the currently viewed frame. Figure 4 b) depicts packet loss artifact detection based on paired parallel sharp vertical edges 245 equally long or longer than a predefined value M, which is four pixels.

Figure 5 shows an example of packet loss artifact detection using all four detection methods proposed in this application. Shown are artifacts 410 detected using right angles 210, artifacts 420 detected using parallel paired sharp horizontal edges 240, artifacts 430 detected using parallel paired sharp vertical edges 245, and artifacts 440 detected by connecting in the horizontal direction. For the detection of artifacts 420 detected using sharp horizontal edges 240 and artifacts 430 detected using sharp vertical edges 245, paired parallel sharp horizontal edges 240 and parallel paired sharp vertical edges 245, longer than or equal to a predefined value L, respectively M, are observed. In post-processing, the macroblocks 300 with undetected artifacts are merged (filled) between two or more macroblocks 330 with detected packet loss artifacts. Thus, Figure 5 shows the shaded macroblocks with detected artifacts, as those that certainly contain packet loss artifacts, where all four methods proposed in this application were used for artifact detection.

Method of industrial and other application of the invention

The application of the presented invention is in the industry, when testing digital TV receivers, in order to more accurately separate the detected packet loss artifacts from the video content. Therefore, by performing preprocessing, determining the necessary characteristics of the packet loss artifacts and setting restrictions, macro-blocks that have artifacts in one or more rows are more accurately indicated. The goal of the proposed method is to improve the quality of the video signal and timely indicate the state of the signal transmission over the communication network. The application is also in the concealment of packet loss artifacts in video frames, whereby approximation and elimination of errors due to packet loss are performed. The goal is to detect as many existing packet loss artifacts as possible, which can then be attempted to be concealed by one of the error concealment methods. It is in the interest of providers to optimize the required bandwidth, and this is facilitated by the correct amount of existing packet loss artifacts.

Claims (8)

Patent claims: 1. A method for detecting packet loss artifacts in a transmitted video signal, wherein the step 100 of acquiring the current video frame takes its luminance component, characterized in that the method continues with preprocessing in a step (110) of detecting sharp horizontal edges and a step (115) of detecting sharp vertical edges, and that the step (110) of detecting sharp horizontal edges (240) and the step (115) of detecting sharp vertical edges (245) are performed in parallel or one after the other; that the step (120) of detecting right angles (210) finds the right angles (210) of the detected edges; that in step (130) of detecting sharp horizontal edges (240), sharp horizontal edges (240) longer than or equal to a predefined number L pixels and at distances determined by the dimensions of the macro-blocks (200) which do not necessarily have to be one height of the macro-block (200), and that in step (135) of detecting sharp vertical edges (245), sharp vertical edges (245) longer than or equal to a predefined number M pixels and at distances determined by the dimensions of the macro-blocks (200), but not necessarily one width of the macro-block (200); that the step (140) of merging macro-blocks (200) in the horizontal direction connects macro-blocks (330) with detected packet loss artifacts if the distance between macro-blocks (330) with detected packet loss artifacts is less than or equal to K macro-blocks (200), thereby considering them as macro-blocks (330) with detected packet loss artifacts; that in step (150) of distinguishing content from packet loss artifacts, a macro-block (330) with detected packet loss artifacts is not accepted if it appears at the same position in more than F consecutive frames, and that the same position is taken as the position at which the artifact appeared only when no artifact is detected at the said position in a predefined number A of frame repetitions. and that the method ends with a step (160) of calculating the percentage of frame coverage by packet loss artifacts. 2. The method defined according to claim 1, characterized in that the right angles are defined by the intersection (230) of the found corresponding sharp horizontal edges (240) and sharp vertical edges (245). 3. The method defined according to claim 1 and 2, characterized in that right angles with a deviation from the ideal right angle (210) have a tolerance of up to a predefined number of 5 pixels, where the given number S is at most 2 pixels (250). 4. The method defined in claim 1, characterized in that the distances for the parallel paired sharp horizontal edges (240) and the parallel paired sharp vertical edges (245) are a multiple of 16 pixels. 5. The method defined according to claim 1, characterized in that the number M is 4 pixels, while the number L is equal to 8 pixels. 6. The method defined according to claim 1, characterized in that during horizontal merging, macroblocks (300) with undetected artifacts are declared macroblocks (330) with detected packet loss artifacts if they are located between two macroblocks (330) with detected packet loss artifacts that are arranged in one row and whose mutual distance is not greater than a predefined value K. 7. The method defined according to claim 1, characterized in that packet loss artifacts are detected not only at the edges of macro-blocks (200), but anywhere in the image. 8. The method defined according to claim 1, characterized in that the percentage of frame coverage by packet loss artifacts is obtained based on the ratio of the number of macro-blocks (330) with detected packet loss artifacts and the total number of macro-blocks (200) in the entire frame.