WO2025009857A1 - Image encoding/decoding method, and recording medium for storing bitstream - Google Patents

Abstract

An image encoding/decoding method according to the present invention comprises the steps of: determining a reference line of the current block; determining an intra prediction mode of the current block; filtering reference samples included in the reference line; and acquiring a prediction block of the current block on the basis of the filtered reference samples and the intra prediction mode. At this time, the filtering is performed by a first filter type and/or a second filter type, the first filter type uses a neighboring reference sample included in the reference line, and the second filter type uses an angular neighboring reference sample included in a neighboring reference line that is adjacent to the reference line.

Description

Video encoding/decoding method and recording medium for storing bitstream

The present disclosure relates to a method and device for processing an image signal.

Recently, the demand for high-resolution, high-quality images such as HD (High Definition) images and UHD (Ultra High Definition) images is increasing in various application fields. As the image data becomes higher in resolution and quality, the amount of data increases relatively compared to existing image data. Therefore, when transmitting the image data using media such as existing wired and wireless broadband lines or storing it using existing storage media, the transmission and storage costs increase. In order to solve these problems that occur as image data becomes higher in resolution and quality, high-efficiency image compression technologies can be utilized.

There are various technologies such as inter-picture prediction technology that predicts pixel values included in the current picture from pictures before or after the current picture, intra-picture prediction technology that predicts pixel values included in the current picture using pixel information in the current picture, and entropy coding technology that assigns short codes to values with high frequency of appearance and long codes to values with low frequency of appearance, etc., and using these image compression technologies, image data can be effectively compressed and transmitted or stored.

Meanwhile, as the demand for high-resolution images increases, the demand for stereoscopic image content as a new image service is also increasing. Discussions are underway on video compression technology to effectively provide high-resolution and ultra-high-resolution stereoscopic image content.

The present disclosure aims to provide a method for filtering reference samples for performing intra prediction and a device for performing the same.

The present disclosure aims to provide a method for adaptively determining a filter type to be applied to a reference sample according to an intra prediction mode and a device for performing the same.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

A video encoding/decoding method according to the present invention may include: a step of determining a reference line of a current block; a step of determining an intra prediction mode of the current block; a step of filtering reference samples included in the reference line; and a step of obtaining a prediction block of the current block based on the filtered reference samples and the intra prediction mode. In this case, the filtering may be performed by at least one of a first filter type or a second filter type, and the first filter type may use a neighboring reference sample included in the reference line, and the second filter type may use an angular neighboring reference sample included in a neighboring reference line adjacent to the reference line.

In the image decoding/encoding method according to the present disclosure, one of the first filter type or the second filter type may be selected based on the intra prediction mode of the current block.

In the video decoding/encoding method according to the present disclosure, when the intra prediction mode is a non-directional mode, the reference samples may be filtered based on the first filter type, and when the intra prediction mode is a directional mode, the reference samples may be filtered based on the second filter type.

In the image decoding/encoding method according to the present disclosure, the angular neighboring reference sample may exist at a position projected to the neighboring reference line, according to the directionality of the intra prediction mode, from a reference sample including the reference line.

In the image decoding/encoding method according to the present disclosure, when the projected position indicates a fractional position, the angular neighboring reference sample can be derived by interpolating two integer position reference samples located on both sides of the projected position within the neighboring reference line.

In the image decoding/encoding method according to the present disclosure, when the projected position indicates a fractional position, an integer position reference sample closest to the projected position within the neighboring reference sample line may be set as the angular neighboring reference sample.

In the image decoding/encoding method according to the present disclosure, the neighboring reference line may be a reference line having an index that is 1 larger or smaller than the reference line.

In the image decoding/encoding method according to the present disclosure, when the index of the reference line is not the maximum value, the neighboring reference line may have an index that is 1 larger than the reference line, and when the index of the reference line is the maximum value, the neighboring reference line may have an index that is 1 smaller than the reference line.

In the image decoding/encoding method according to the present disclosure, the filtered reference sample can be obtained by weighting the reference sample within the reference line and the angular reference sample within the neighboring reference line.

In the image decoding/encoding method according to the present disclosure, the weight assigned to the reference sample may have a larger value than the weight assigned to the angular reference sample.

In the image decoding/encoding method according to the present disclosure, the filtered reference sample can be obtained by weighting a reference sample within the reference line, a first angular reference sample within a first neighboring reference line adjacent to the reference line, and a second angular reference sample within a second neighboring reference line adjacent to the reference line.

In the image decoding/encoding method according to the present disclosure, the reference line may exist between the first neighboring reference line and the second neighboring reference line.

In the image decoding/encoding method according to the present disclosure, when performing filtering according to the second filter type, whether a plurality of neighboring reference lines are used can be determined based on the index of the reference line.

A computer-readable recording medium storing a bitstream encoded by an image encoding method according to the present disclosure can be provided.

According to the present disclosure, by filtering reference samples, prediction accuracy can be increased, and thus encoding/decoding performance can be improved.

According to the present disclosure, by adaptively selecting a filter type applied to a reference sample according to the characteristics of an intra prediction mode, prediction accuracy can be improved, and thereby image compression performance can be improved.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art to which the present disclosure belongs from the description below.

FIG. 1 is a block diagram illustrating an image encoding device according to an embodiment of the present disclosure.

FIG. 2 is a block diagram illustrating an image decoding device according to an embodiment of the present disclosure.

FIG. 3 illustrates an image encoding/decoding method performed by an image encoding/decoding device according to the present disclosure.

FIGS. 4 and 5 illustrate examples of multiple intra prediction modes according to the present disclosure.

FIG. 6 illustrates an intra prediction method based on a planar mode according to the present disclosure.

FIG. 7 illustrates an intra prediction method based on DC mode according to the present disclosure.

FIG. 8 illustrates an intra prediction method based on a directional mode according to the present disclosure.

Figure 9 illustrates a method for deriving samples of fractional positions.

Figures 10 and 11 illustrate that the tangent value for angle is scaled by a factor of 32 for each intra prediction mode.

Figure 12 is a diagram illustrating an intra prediction aspect when the directional mode is one of modes 34 to 49.

Figure 13 is a diagram illustrating an example of generating an upper reference sample by interpolating left reference samples.

Figure 14 shows an example in which intra prediction is performed using reference samples arranged in a 1D array.

Figure 15 is a diagram explaining the encoding/decoding order when a single tree structure is used.

Figure 16 is a diagram explaining the order of encoding/decoding when a dual tree structure is used.

Figure 17 is a flowchart illustrating a method for predicting a chroma block using a restored luma block.

Figures 18 to 20 illustrate examples of downsampling a luma block.

Figure 21 is a drawing to explain an example related to the location where down sampling is applied.

Figure 22 is a diagram illustrating an example of filtering reference samples.

Figure 23 is a diagram illustrating an example of filtering reference samples.

Figure 24 illustrates directional modes to which angular filters are applied.

The present invention can have various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention. In describing each drawing, similar reference numerals are used for similar components.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and/or includes a combination of a plurality of related described items or any one of a plurality of related described items.

When it is said that a component is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. On the other hand, when it is said that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Hereinafter, with reference to the attached drawings, a preferred embodiment of the present invention will be described in more detail. Hereinafter, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

FIG. 1 is a block diagram showing an image encoding device according to an embodiment of the present invention.

Referring to FIG. 1, a video encoding device (100) may include a picture segmentation unit (110), a prediction unit (120, 125), a transformation unit (130), a quantization unit (135), a reordering unit (160), an entropy encoding unit (165), an inverse quantization unit (140), an inverse transformation unit (145), a filter unit (150), and a memory (155).

Each component shown in FIG. 1 is independently depicted to indicate different characteristic functions in the image encoding device, and does not mean that each component is composed of separate hardware or a single software configuration unit. That is, each component is listed and included as a separate component for convenience of explanation, and at least two components among each component may be combined to form a single component, or one component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are also included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

In addition, some components may not be essential components that perform essential functions in the present invention, but may be optional components that are merely used to improve performance. The present invention may be implemented by including only essential components for implementing the essence of the present invention, excluding components that are merely used to improve performance, and a structure that includes only essential components, excluding optional components that are merely used to improve performance, is also included in the scope of the present invention.

The picture splitting unit (110) can split an input picture into at least one processing unit. At this time, the processing unit may be a prediction unit (PU), a transform unit (TU), or a coding unit (CU). The picture splitting unit (110) can split one picture into a combination of multiple coding units, prediction units, and transform units, and select one combination of coding units, prediction units, and transform units based on a predetermined criterion (e.g., a cost function) to encode the picture.

For example, a picture can be split into multiple coding units. In order to split a coding unit in a picture, a recursive tree structure such as a quad tree structure can be used. A coding unit that is split into other coding units with one image or the largest coding unit as the root can be split with as many child nodes as the number of split coding units. A coding unit that cannot be split any further according to a certain restriction becomes a leaf node. That is, if it is assumed that only a square split is possible for a coding unit, a coding unit can be split into at most four different coding units.

Hereinafter, in the embodiments of the present invention, the encoding unit may be used to mean a unit that performs encoding, or may be used to mean a unit that performs decoding.

A prediction unit may be divided into at least one square or rectangular shape of the same size within one coding unit, or may be divided such that one prediction unit among the divided prediction units within one coding unit has a different shape and/or size from another prediction unit.

When generating a prediction unit that performs intra prediction based on a coding unit, if it is not the minimum coding unit, intra prediction can be performed without dividing it into multiple prediction units NxN.

The prediction unit (120, 125) may include an inter prediction unit (120) that performs inter prediction and an intra prediction unit (125) that performs intra prediction. It may be determined whether to use inter prediction or intra prediction for a prediction unit, and specific information (e.g., intra prediction mode, motion vector, reference picture, etc.) according to each prediction method may be determined. At this time, the processing unit where the prediction is performed and the processing unit where the prediction method and specific contents are determined may be different. For example, the prediction method and prediction mode, etc. may be determined in the prediction unit, and the prediction may be performed in the transformation unit. The residual value (residual block) between the generated prediction block and the original block may be input to the transformation unit (130). In addition, the prediction mode information, motion vector information, etc. used for the prediction may be encoded together with the residual value in the entropy encoding unit (165) and transmitted to the decoding device. When using a specific encoding mode, it is also possible to encode the original block as is and transmit it to the decoding unit without generating a prediction block through the prediction unit (120, 125).

The inter prediction unit (120) may predict a prediction unit based on information of at least one picture among the previous picture or the subsequent picture of the current picture, and in some cases, may predict a prediction unit based on information of a part of an encoded region within the current picture. The inter prediction unit (120) may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.

The reference picture interpolation unit can receive reference picture information from the memory (155) and generate pixel information below an integer pixel from the reference picture. In the case of luminance pixels, a DCT-based 8-tap interpolation filter (DCT-based Interpolation Filter) with different filter coefficients can be used to generate pixel information below an integer pixel in units of 1/4 pixels. In the case of a chrominance signal, a DCT-based 4-tap interpolation filter (DCT-based Interpolation Filter) with different filter coefficients can be used to generate pixel information below an integer pixel in units of 1/8 pixels.

The motion prediction unit can perform motion prediction based on a reference picture interpolated by the reference picture interpolation unit. Various methods such as FBMA (Full search-based Block Matching Algorithm), TSS (Three Step Search), and NTS (New Three-Step Search Algorithm) can be used to derive a motion vector. The motion vector can have a motion vector value of 1/2 or 1/4 pixel units based on the interpolated pixel. The motion prediction unit can predict the current prediction unit by using a different motion prediction method. Various methods such as the Skip method, the Merge method, the AMVP (Advanced Motion Vector Prediction) method, and the Intra Block Copy method can be used as the motion prediction method.

The intra prediction unit (125) can generate a prediction unit based on reference pixel information surrounding the current block, which is pixel information within the current picture. If the surrounding block of the current prediction unit is a block on which inter prediction is performed and the reference pixel is a pixel on which inter prediction is performed, the reference pixel included in the block on which inter prediction is performed can be used as a replacement for the reference pixel information of the surrounding block on which intra prediction is performed. That is, if the reference pixel is not available, the unavailable reference pixel information can be used as a replacement for at least one reference pixel among the available reference pixels.

In intra prediction, the prediction mode can have a directional prediction mode that uses reference pixel information according to the prediction direction and a non-directional mode that does not use directional information when performing prediction. The mode for predicting luminance information and the mode for predicting chrominance information can be different, and the intra prediction mode information used for predicting luminance information or the predicted luminance signal information can be utilized to predict chrominance information.

When performing intra prediction, if the size of the prediction unit and the size of the transformation unit are the same, intra prediction for the prediction unit can be performed based on the pixels on the left side of the prediction unit, the pixels on the upper left side, and the pixels on the upper side. However, when performing intra prediction, if the sizes of the prediction unit and the transformation unit are different, intra prediction can be performed using reference pixels based on the transformation unit. In addition, intra prediction using NxN division can be used only for the minimum coding unit.

The intra prediction method can generate a prediction block after applying an AIS (Adaptive Intra Smoothing) filter to a reference pixel according to a prediction mode. The type of AIS filter applied to the reference pixel may be different. In order to perform the intra prediction method, the intra prediction mode of a current prediction unit can be predicted from the intra prediction modes of prediction units existing in the vicinity of the current prediction unit. When the prediction mode of the current prediction unit is predicted using mode information predicted from the surrounding prediction units, if the intra prediction modes of the current prediction unit and the surrounding prediction units are the same, information that the prediction modes of the current prediction unit and the surrounding prediction units are the same can be transmitted using predetermined flag information, and if the prediction modes of the current prediction unit and the surrounding prediction units are different, entropy encoding can be performed to encode the prediction mode information of the current block.

In addition, a residual block including residual value information, which is a difference value between the prediction unit that performed the prediction and the original block of the prediction unit based on the prediction unit generated in the prediction unit (120, 125), can be generated. The generated residual block can be input to the transformation unit (130).

In the transformation unit (130), the residual block including the residual value information of the prediction unit generated through the original block and the prediction unit (120, 125) can be transformed using a transformation method such as DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), or KLT. Whether to apply DCT, DST, or KLT to transform the residual block can be determined based on the intra prediction mode information of the prediction unit used to generate the residual block.

The quantization unit (135) can quantize the values converted to the frequency domain in the transformation unit (130). The quantization coefficients can vary depending on the block or the importance of the image. The values produced by the quantization unit (135) can be provided to the dequantization unit (140) and the reordering unit (160).

The rearrangement unit (160) can perform rearrangement of coefficient values for quantized residual values.

The rearrangement unit (160) can change a two-dimensional block-shaped coefficient into a one-dimensional vector shape by using a coefficient scanning method. For example, the rearrangement unit (160) can change the two-dimensional block-shaped coefficient into a one-dimensional vector shape by scanning from the DC coefficient to the coefficient of the high-frequency region by using a zig-zag scan method. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional block-shaped coefficient in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficient in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, it is possible to determine which scan method among the zig-zag scan, the vertical scan, and the horizontal scan is to be used.

The entropy encoding unit (165) can perform entropy encoding based on the values produced by the rearrangement unit (160). Entropy encoding can use various encoding methods such as, for example, Exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding).

The entropy encoding unit (165) can encode various information such as residual value coefficient information of an encoding unit, block type information, prediction mode information, division unit information, prediction unit information, transmission unit information, motion vector information, reference frame information, block interpolation information, and filtering information from the rearrangement unit (160) and the prediction unit (120, 125).

The entropy encoding unit (165) can entropy encode the coefficient values of the encoding unit input from the rearrangement unit (160).

In the inverse quantization unit (140) and the inverse transformation unit (145), the values quantized in the quantization unit (135) are inversely quantized and the values transformed in the transformation unit (130) are inversely transformed. The residual values generated in the inverse quantization unit (140) and the inverse transformation unit (145) can be combined with the predicted prediction units through the motion estimation unit, motion compensation unit, and intra prediction unit included in the prediction unit (120, 125) to generate a reconstructed block.

The filter unit (150) may include at least one of a deblocking filter, an offset correction unit, and an ALF (Adaptive Loop Filter).

A deblocking filter can remove block distortion caused by boundaries between blocks in a restored picture. In order to determine whether to perform deblocking, it is possible to determine whether to apply a deblocking filter to the current block based on pixels included in several columns or rows included in the block. When applying a deblocking filter to a block, a strong filter or a weak filter can be applied depending on the required deblocking filtering strength. In addition, when applying a deblocking filter, horizontal filtering and vertical filtering can be processed in parallel when performing vertical filtering and horizontal filtering.

The offset correction unit can correct the offset from the original image on a pixel basis for the image on which deblocking has been performed. To perform offset correction for a specific picture, a method can be used in which the pixels included in the image are divided into a certain number of regions, the regions to be offset are determined, and the offset is applied to the regions, or a method can be used in which the offset is applied by considering the edge information of each pixel.

Adaptive Loop Filtering (ALF) can be performed based on the value compared between the filtered restored image and the original image. After dividing the pixels included in the image into a predetermined group, one filter to be applied to the group is determined, and filtering can be performed differentially for each group. Information related to whether to apply ALF can be transmitted by luminance signal for each coding unit (CU), and the shape and filter coefficient of the ALF filter to be applied can be different for each block. In addition, the same shape (fixed shape) of the ALF filter can be applied regardless of the characteristics of the target block.

The memory (155) can store a restored block or picture produced through the filter unit (150), and the stored restored block or picture can be provided to the prediction unit (120, 125) when performing inter prediction.

FIG. 2 is a block diagram showing an image decoding device according to an embodiment of the present invention.

Referring to FIG. 2, the image decoding device (200) may include an entropy decoding unit (210), a reordering unit (215), an inverse quantization unit (220), an inverse transformation unit (225), a prediction unit (230, 235), a filter unit (240), and a memory (245).

When a video bitstream is input from a video encoding device, the input bitstream can be decoded in the opposite procedure to that of the video encoding device.

The entropy decoding unit (210) can perform entropy decoding in a procedure opposite to that of performing entropy encoding in the entropy encoding unit of the video encoding device. For example, various methods such as Exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding) can be applied in response to the method performed in the video encoding device.

The entropy decoding unit (210) can decode information related to intra prediction and inter prediction performed in the encoding device.

The reordering unit (215) can perform reordering based on the method in which the bitstream that has been entropy-decoded by the entropy decoding unit (210) is reordered by the encoding unit. The coefficients expressed in the form of a one-dimensional vector can be re-restored into coefficients in the form of a two-dimensional block and reordered. The reordering unit (215) can perform reordering by receiving information related to the coefficient scanning performed by the encoding unit and performing reverse scanning based on the scanning order performed by the corresponding encoding unit.

The inverse quantization unit (220) can perform inverse quantization based on the quantization parameters provided from the encoding device and the coefficient values of the rearranged block.

The inverse transform unit (225) can perform inverse transform, i.e., inverse DCT, inverse DST, and inverse KLT, on the transforms performed by the transform unit, i.e., DCT, DST, and KLT, on the quantization result performed by the image encoding device. The inverse transform can be performed based on the transmission unit determined by the image encoding device. In the inverse transform unit (225) of the image decoding device, a transform technique (e.g., DCT, DST, KLT) can be selectively performed according to a plurality of pieces of information, such as a prediction method, the size of the current block, and the prediction direction.

The prediction unit (230, 235) can generate a prediction block based on prediction block generation related information provided from the entropy decoding unit (210) and previously decoded block or picture information provided from the memory (245).

As described above, when performing intra prediction in the same manner as the operation in the video encoding device, if the size of the prediction unit and the size of the transformation unit are the same, intra prediction for the prediction unit is performed based on the pixels on the left side of the prediction unit, the pixels on the upper left side, and the pixels on the upper side. However, if the size of the prediction unit and the size of the transformation unit are different when performing intra prediction, intra prediction can be performed using reference pixels based on the transformation unit. In addition, intra prediction using NxN division only for the minimum coding unit can be used.

The prediction unit (230, 235) may include a prediction unit determination unit, an inter prediction unit, and an intra prediction unit. The prediction unit determination unit may receive various information such as prediction unit information input from the entropy decoding unit (210), prediction mode information of an intra prediction method, and motion prediction-related information of an inter prediction method, and may distinguish a prediction unit from a current encoding unit and determine whether the prediction unit performs inter prediction or intra prediction. The inter prediction unit (230) may perform inter prediction on the current prediction unit based on information included in at least one of a previous picture or a subsequent picture of the current picture including the current prediction unit, by using information necessary for inter prediction of the current prediction unit provided from the image encoding device. Alternatively, inter prediction may be performed based on information on a portion of a pre-restored area within the current picture including the current prediction unit.

In order to perform inter prediction, it is possible to determine whether the motion prediction method of the prediction unit included in the encoding unit is Skip Mode, Merge Mode, AMVP Mode, or Intra Block Copy Mode based on the encoding unit.

The intra prediction unit (235) can generate a prediction block based on pixel information in the current picture. If the prediction unit is a prediction unit that has performed intra prediction, the intra prediction can be performed based on intra prediction mode information of the prediction unit provided by the image encoding device. The intra prediction unit (235) can include an AIS (Adaptive Intra Smoothing) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter is a part that performs filtering on the reference pixels of the current block and can determine whether to apply the filter according to the prediction mode of the current prediction unit and apply it. The AIS filter can be performed on the reference pixels of the current block using the prediction mode and AIS filter information of the prediction unit provided by the image encoding device. If the prediction mode of the current block is a mode that does not perform AIS filtering, the AIS filter may not be applied.

The reference pixel interpolation unit can interpolate the reference pixel to generate a reference pixel of a pixel unit less than an integer value when the prediction mode of the prediction unit is a prediction unit that performs intra prediction based on the pixel value interpolated by the reference pixel. When the prediction mode of the current prediction unit is a prediction mode that generates a prediction block without interpolating the reference pixel, the reference pixel may not be interpolated. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is the DC mode.

The restored block or picture may be provided to a filter unit (240). The filter unit (240) may include a deblocking filter, an offset correction unit, and an ALF.

Information on whether a deblocking filter has been applied to a corresponding block or picture from a video encoding device and, if a deblocking filter has been applied, information on whether a strong filter or a weak filter has been applied can be provided. A deblocking filter of a video decoding device can receive information related to a deblocking filter provided from a video encoding device and perform deblocking filtering on a corresponding block in the video decoding device.

The offset correction unit can perform offset correction on the restored image based on information such as the type of offset correction applied to the image during encoding and the offset value.

ALF can be applied to an encoding unit based on ALF application information provided from an encoding device, ALF coefficient information, etc. This ALF information can be provided by being included in a specific parameter set.

The memory (245) can store a restored picture or block so that it can be used as a reference picture or reference block, and can also provide the restored picture to an output unit.

As described above, in the following embodiments of the present invention, for convenience of explanation, the term coding unit is used as an encoding unit, but it may also be a unit that performs not only encoding but also decoding.

In addition, the current block represents a block to be encoded/decoded, and may represent a coding tree block (or coding tree unit), an encoding block (or encoding unit), a transform block (or transform unit), or a prediction block (or prediction unit), depending on the encoding/decoding step. In this specification, a 'unit' represents a basic unit for performing a specific encoding/decoding process, and a 'block' may represent a pixel array of a predetermined size. Unless otherwise distinguished, 'block' and 'unit' may be used with the same meaning. For example, in the embodiment described below, an encoding block (coding block) and an encoding unit (coding unit) may be understood to have the same meaning.

FIG. 3 illustrates an image encoding/decoding method performed by an image encoding/decoding device according to the present disclosure.

Referring to FIG. 3, a reference line for intra prediction of the current block can be determined (S300).

The current block can use one or more of a plurality of pre-defined reference line candidates in the video encoding/decoding device as reference lines for intra prediction. Here, the plurality of pre-defined reference line candidates can include neighboring reference lines adjacent to the current block to be decoded and N non-neighboring reference lines that are 1 to N samples away from the boundary of the current block. N can be 1, 2, 3, or an integer greater than or equal to 1. For convenience of explanation, it is assumed hereafter that the plurality of reference line candidates available to the current block are composed of neighboring reference line candidates and three non-neighboring reference line candidates, but the present invention is not limited thereto. That is, it goes without saying that the plurality of reference line candidates available to the current block can include four or more non-neighboring reference line candidates.

The video encoding device can determine an optimal reference line candidate from among a plurality of reference line candidates and encode an index for specifying the optimal reference line candidate. The video decoding device can determine a reference line of a current block based on an index signaled through a bitstream. The index can specify one of the plurality of reference line candidates. The reference line candidate specified by the index can be used as a reference line of the current block.

The number of indexes signaled to determine the reference line of the current block may be 1, 2 or more. For example, when the number of the signaled indexes is 1, the current block can perform intra prediction using only a single reference line candidate specified by the signaled index among the plurality of reference line candidates. Alternatively, when the number of the signaled indexes is 2 or more, the current block can perform intra prediction using a plurality of reference line candidates specified by a plurality of indexes among the plurality of reference line candidates.

Referring to FIG. 3, the intra prediction mode of the current block can be determined (S310).

The intra prediction mode of the current block can be determined from among a plurality of intra prediction modes pre-defined in the video encoding/decoding device. The pre-defined plurality of intra prediction modes will be examined with reference to FIGS. 4 and 5.

FIG. 4 illustrates an example of multiple intra prediction modes according to the present disclosure.

Referring to FIG. 4, a plurality of intra prediction modes pre-defined in the video encoding/decoding device may be composed of a non-directional mode and a directional mode. The non-directional mode may include at least one of a planar mode or a DC mode. The directional mode may include directional modes 2 to 66.

The directional mode can be further extended than that shown in Fig. 4. Fig. 5 shows an example in which the directional mode is extended.

In Fig. 5, it is exemplified that modes -1 to -14 and modes 67 to 80 are added. These directional modes may be referred to as wide-angle intra prediction modes. Whether to use the wide-angle intra prediction mode may be determined depending on the shape of the current block. For example, if the current block is a non-square block whose width is greater than its height, some directional modes (e.g., modes 2 to 15) may be converted to wide-angle intra prediction modes between modes 67 to 80. On the other hand, if the current block is a non-square block whose height is greater than its width, some directional modes (e.g., modes 53 to 66) may be converted to wide-angle intra prediction modes between modes -1 to -14.

The range of available wide-angle intra prediction modes can be adaptively determined according to the width and height ratio of the current block. Table 1 shows the range of available wide-angle intra prediction modes according to the width and height ratio of the current block.

Width/Height Range of available wide-angle intra prediction modes W/H = 16 67~80 W/H = 8 67~78 W/H = 4 67~76 W/H = 2 67~74 W/H = 1 doesn't exist W/H = 1/2 -1~-8 W/H = 1/4 -1~-10 W/H = 1/8 -1~-12 W/H = 1/16 -1~-14

Among the above plural intra prediction modes, K candidate modes (most probable modes, MPMs) can be selected. A candidate list including the selected candidate modes can be generated. An index indicating one of the candidate modes belonging to the candidate list can be signaled. The intra prediction mode of the current block can be determined based on the candidate mode indicated by the index. For example, the candidate mode indicated by the index can be set as the intra prediction mode of the current block. Alternatively, the intra prediction mode of the current block can be determined based on a value of the candidate mode indicated by the index and a predetermined differential value. The differential value can be defined as a difference between a value of the intra prediction mode of the current block and a value of the candidate mode indicated by the index. The differential value can be signaled through a bitstream. Alternatively, the difference value may be a pre-defined value in the video encoding/decoding device. Alternatively, the intra prediction mode of the current block may be determined based on a flag indicating whether a mode identical to the intra prediction mode of the current block exists in the candidate list. For example, if the flag is a first value, the intra prediction mode of the current block may be determined from the candidate list. In this case, an index indicating any one of a plurality of candidate modes belonging to the candidate list may be signaled. The candidate mode indicated by the index may be set as the intra prediction mode of the current block. On the other hand, if the flag is a second value, any one of the remaining intra prediction modes may be set as the intra prediction mode of the current block. The remaining intra prediction mode may mean a mode excluding a candidate mode belonging to the candidate list among the pre-defined plurality of intra prediction modes. If the flag is a second value, an index indicating any one of the remaining intra prediction modes may be signaled. An intra prediction mode indicated by a signaled index can be set as the intra prediction mode of the current block. The intra prediction mode of the chroma block can be selected from intra prediction mode candidates of a plurality of chroma blocks. For this purpose, index information indicating one of the intra prediction mode candidates of the chroma block can be explicitly encoded and signaled through the bitstream. Table 2 illustrates intra prediction mode candidates of the chroma block.

Index Intra prediction mode candidates for chroma blocks Luma Mode: 0 Luma Mode: 50 Lumamode:18 Lumamode:1 Other 0 66 0 0 0 0 1 50 66 50 50 50 2 18 18 66 18 18 3 1 1 1 66 1 4 DM

In the example of Table 2, DM (Direct Mode) means setting the intra prediction mode of the luma block existing at the same position as the chroma block to the intra prediction mode of the chroma block. Meanwhile, the luma block existing at the same position as the chroma block can be determined based on the position of the upper left sample or the position of the center sample of the chroma block. For example, if the intra prediction mode (luma mode) of the luma block is 0 (planar mode) and the index points to 2, the intra prediction mode of the chroma block can be determined as the horizontal mode (18). For example, if the intra prediction mode (luma mode) of the luma block is 1 (DC mode) and the index points to 0, the intra prediction mode of the chroma block can be determined as the planar mode (0).

As a result, the intra prediction mode of the chroma block may also be set to one of the intra prediction modes illustrated in FIG. 4 or FIG. 5. The intra prediction mode of the current block may also be used to determine the reference line of the current block, in which case step S310 may be performed before step S300.

Referring to FIG. 3, intra prediction can be performed for the current block based on the reference line and intra prediction mode of the current block (S320).

Hereinafter, the intra prediction method for each intra prediction mode will be examined in detail with reference to FIGS. 6 to 8. However, for convenience of explanation, it is assumed that a single reference line is used for intra prediction of the current block, but even when multiple reference lines are used, the intra prediction method described below can be applied in the same/similar manner.

FIG. 6 illustrates an intra prediction method based on a planar mode according to the present disclosure.

Referring to FIG. 6, T represents a reference sample located at the upper right corner of the current block, and L represents a reference sample located at the lower left corner of the current block. P1 can be generated through horizontal interpolation. For example, P1 can be generated by interpolating T with a reference sample located on the same horizontal line as P1. P2 can be generated through vertical interpolation. For example, P2 can be generated by interpolating L with a reference sample located on the same vertical line as P2. The current sample in the current block can be predicted through a weighted sum of P1 and P2, as in the following mathematical expression 1.

In mathematical expression 1, the weights α and β can be determined by considering the width and height of the current block. Depending on the width and height of the current block, the weights α and β may have the same value or different values. If the width and height of the current block are the same, the weights α and β can be set to the same value, and the prediction sample of the current sample can be set to the average value of P1 and P2. If the width and height of the current block are not the same, the weights α and β can have different values. For example, if the width is larger than the height, a smaller value can be set for the weight corresponding to the width of the current block, and a larger value can be set for the weight corresponding to the height of the current block. Conversely, if the width is larger than the height, a larger value can be set for the weight corresponding to the width of the current block, and a smaller value can be set for the weight corresponding to the height of the current block. Here, the weight corresponding to the width of the current block can mean β, and the weight corresponding to the height of the current block can mean α.

FIG. 7 illustrates an intra prediction method based on DC mode according to the present disclosure.

Referring to FIG. 7, the average value of the surrounding samples adjacent to the current block can be calculated, and the calculated average value can be set as the predicted value of all samples in the current block. Here, the surrounding samples can include the upper reference sample and the left reference sample of the current block. However, depending on the shape of the current block, the average value can be calculated using only the upper reference sample or the left reference sample. For example, if the width of the current block is greater than the height, the average value can be calculated using only the upper reference sample of the current block. Alternatively, if the ratio of the width and height of the current block is greater than or equal to a predetermined threshold, the average value can be calculated using only the upper reference sample of the current block. Alternatively, if the ratio of the width and height of the current block is less than or equal to a predetermined threshold, the average value can be calculated using only the upper reference sample of the current block. On the other hand, if the width of the current block is less than the height, the average value can be calculated using only the left reference sample of the current block. Alternatively, if the ratio of the width and height of the current block is less than or equal to a predetermined threshold, the average value can be calculated using only the left reference sample of the current block. Alternatively, if the ratio of the width to height of the current block is greater than or equal to a predetermined threshold, the average value can be calculated using only the left reference sample of the current block.

FIG. 8 illustrates an intra prediction method based on a directional mode according to the present disclosure.

If the intra prediction mode of the current block is a directional mode, projection can be performed as a reference line according to the angle of the directional mode. If a reference sample exists at the projected position, the reference sample can be set as a prediction sample of the current sample. If the reference sample does not exist at the projected position, a sample corresponding to the projected position can be generated using one or more neighboring samples neighboring the projected position. For example, a sample corresponding to the projected position can be generated by performing interpolation based on two or more neighboring samples neighboring in both directions with respect to the projected position. Alternatively, one neighboring sample neighboring the projected position can be set as a sample corresponding to the projected position. In this case, among a plurality of neighboring samples neighboring the projected position, a neighboring sample closest to the projected position can be used. The sample corresponding to the projected position can be set as a prediction sample of the current sample.

Referring to FIG. 8, in the case of the current sample B, if projection is performed with a reference line according to the angle of the intra prediction mode at the corresponding position, a reference sample exists at the projected position (i.e., a reference sample at an integer position, R3). In this case, the reference sample at the projected position can be set as a prediction sample of the current sample B. In the case of the current sample A, if projection is performed with a reference line according to the angle of the intra prediction mode at the corresponding position, a reference sample (i.e., a reference sample at an integer position) does not exist at the projected position. In this case, a sample (r) at a fractional position can be generated by performing interpolation based on neighboring samples (e.g., R2 and R3) adjacent to the projected position. The generated sample (r) at the fractional position can be set as a prediction sample of the current sample A.

Figure 9 illustrates a method for deriving samples of fractional positions.

In the example of Fig. 9, the variable h represents the vertical distance (i.e., vertical distance) between the position of the prediction sample A and the reference line, and the variable w represents the horizontal distance (i.e., horizontal distance) between the position of the prediction sample A and the fractional position sample. In addition, the variable θ represents a predefined angle according to the directionality of the intra prediction mode, and the variable x represents the fractional position.

The variable w can be derived as shown in the following mathematical expression 2.

Afterwards, by removing the integer positions from the variable w, the fractional positions can finally be derived.

The fractional position sample can be generated by interpolating adjacent integer position reference samples. For example, the fractional position reference sample at the x position can be generated by interpolating the integer position reference sample R2 and the integer position reference sample R3.

In deriving fractional position samples, a scaling factor can be used to avoid real number operations. For example, when the scaling factor f is set to 32, the distance between neighboring integer reference samples can be set to 32 instead of 1, as in the example shown in (b) of Fig. 8.

Additionally, the tangent value for angle θ, which is determined by the directionality of the intra prediction mode, can also be scaled up using the same scaling factor (e.g., 32).

Figures 10 and 11 illustrate that the tangent value for angle is scaled by a factor of 32 for each intra prediction mode.

Figure 10 shows the scaled results of tangent values for the non-wide angle intra prediction mode, and Figure 11 shows the scaled results of tangent values for the wide angle intra prediction mode.

When the tangent value (tanθ) for the angle value of the intra prediction mode is positive, intra prediction can be performed using only one of the reference samples belonging to the upper line of the current block (i.e., upper reference samples) or the reference samples belonging to the left line of the current block (i.e., left reference samples). On the other hand, when the tangent value for the angle value of the intra prediction mode is negative, both the reference samples located at the upper side and the reference samples located at the left side are used.

At this time, to simplify the implementation, the reference samples may be arranged in a 1D array form by projecting the left reference samples upward or the upper reference samples to the left, and intra prediction may be performed using the reference samples in the 1D array form.

Figure 12 is a diagram illustrating the intra prediction aspect when the directional mode is one of modes 34 to 49.

If the intra prediction mode of the current block is one of modes 34 to 49, intra prediction is performed using not only the upper reference samples of the current block but also the left reference samples. At this time, as in the example illustrated in Fig. 12, the reference sample located on the left side of the current block can be copied to the position of the upper line, or the reference samples located on the left can be interpolated to generate the reference sample of the upper line.

For example, in case of obtaining a reference sample for position A at the top of the current block, projection can be performed from position A of the top line to the left line of the current block, considering the directionality of the intra prediction mode of the current block. If the projected position is a, a value corresponding to position a can be copied, or a fractional position value corresponding to a can be generated and set as the value of position A. For example, if position a is an integer position, the value of position A can be generated by copying the integer position reference sample. On the other hand, if position a is a fractional position, the reference sample located above position a and the reference sample located below position a can be interpolated, and the interpolated value can be set as the value of position A. Meanwhile, the direction projected from position A at the top of the current block to the left line of the current block can be parallel to and opposite to the direction of the intra prediction mode of the current block.

Figure 13 is a diagram illustrating an example of generating an upper reference sample by interpolating left reference samples.

In Fig. 13, variable h represents the horizontal distance between position A of the upper line and position a of the left line. Variable w represents the vertical distance between position A of the upper line and position a of the left line. In addition, variable θ represents a predefined angle according to the directionality of the intra prediction mode, and variable x represents a fractional position.

The variable h can be derived as shown in the following mathematical expression 3.

Afterwards, by removing the integer position from the variable h, the fractional position can be finally derived.

In deriving the fractional position samples, a scaling factor can be used to avoid real number operations. For example, the tangent value for variable θ can be scaled using the scaling factor f1. Here, since the direction projected to the left line is parallel and opposite to the directional prediction pattern, the scaled tangent value shown in Figs. 10 and 11 can also be used.

When the scaling factor f1 is applied, Equation 3 can be transformed and used as Equation 4 below.

In the same way as above, a 1D reference sample array can be constructed using only the reference samples belonging to the upper line. As a result, intra prediction for the current block can be performed using only the upper reference samples constructed as a 1D array.

Figure 14 shows an example in which intra prediction is performed using reference samples arranged in a 1D array.

As in the example illustrated in Fig. 14, by projecting the left reference samples to generate the upper reference samples, prediction samples of the current block can be obtained using only the reference samples belonging to the upper line.

Contrary to what is illustrated in FIGS. 12 and 14, the upper reference sample may be projected onto the left line, thereby constructing a 1D reference sample array using only the reference samples belonging to the left line. Specifically, for modes 19 to 33 among the directional modes in which the tangent value (tanθ) for the angle of the directional mode is negative, the reference samples belonging to the upper line may be projected onto the left line, thereby generating the left reference sample.

As described above, a picture can be encoded/decoded in block units. For example, a picture can be divided into blocks of a predetermined size. A block of a predetermined size can be called a coding tree block (or coding tree unit) or a reference block. Information indicating the size of the reference block can be signaled through a bitstream. For example, information indicating the size of the coding tree block can be encoded through a sequence parameter set or a picture header.

Thereafter, based on at least one of the tree structures defined in the reference block, the reference block can be added into blocks of various sizes. Then, pre/decode processing such as prediction, transformation, quantization, and/or entropy encoding can be performed on each of the divided blocks. Each of the divided blocks can be a coding block, a prediction block, or a transformation block.

Meanwhile, in the case of a color picture, luma pictures and chroma pictures are encoded/decoded respectively. At this time, in the case of a chroma picture, it is common for it to have similar characteristics to a luma picture. That is, there is a tendency for the characteristics of chroma samples in a chroma picture to be similar to the characteristics of luma samples in the same position in a luma picture. By utilizing this characteristic, the division structure for the reference block can be independently determined only for one reference component among the color components, and the tree structure determined in the existing component can be identically applied to the other components. In this way, a division structure in which the tree division structure of the reference component is applied as it is to the other components can be called a single tree structure.

Meanwhile, in the embodiments described below, it is assumed that the reference component is a luma component. That is, it is assumed that a tree partitioning structure is independently applied to the luma component, while the tree partitioning structure of the luma component is applied as is to the chroma component.

Information about the tree partition structure in the reference component can be explicitly encoded and signaled. For example, the tree partition structure information for the luma reference block can be encoded and signaled. On the other hand, for the chroma reference block, encoding/decoding of the tree partition structure information can be omitted, and the tree partition structure information in the luma reference block can be used in the same manner.

Figure 15 is a diagram explaining the encoding/decoding order when a single tree structure is used.

As in the example illustrated in Fig. 15, when a single tree structure is applied, the tree partitioning structure for the luma reference block can be applied as is to the chroma reference block. Accordingly, the partitioning form for the chroma reference block is the same as the partitioning form for the luma reference block.

Meanwhile, encoding/decoding can be performed in an alternating order of luma components and chroma components. For example, in Fig. 15, the numbers written in each leaf node block indicate the encoding/decoding order.

Meanwhile, a partition structure in which the tree structure for each color component is independently determined may be referred to as a dual tree structure. When a dual tree structure is applied, information on the tree partition structure for the luma component and information on the tree partition structure for the chroma component may be independently encoded and signaled.

Figure 16 is a diagram explaining the order of encoding/decoding when a dual tree structure is used.

When a dual tree structure is applied, after the encoding/decoding for one component is completed, the encoding/decoding for the next component can begin.

For example, after encoding/decoding a luma component picture is completed, encoding/decoding a chroma component picture can be performed.

Alternatively, the order of encoding/decoding between components can be set on a per-reference block basis. For example, after encoding/decoding for a luma component reference block is completed, encoding/decoding for a chroma component reference block can be performed.

In the example illustrated in Fig. 16, the numbers written within the blocks indicate the encoding/decoding order. In the example of Fig. 16, it is illustrated that the reference block of the luma component is encoded/decoded before the reference block of the chroma component.

Using the restored luma block, prediction can be performed on the chroma block. The prediction model using different color components, such as the above, can be called a cross-component linear model (CCLM). When the CCLM is applied, the process of deriving the intra prediction mode of the chroma block based on the intra prediction mode of the luma block can be omitted.

Figure 17 is a flowchart illustrating a method for predicting a chroma block using a restored luma block.

Referring to Fig. 17, first, for prediction of a chroma block, prediction parameters can be derived (S1710). At this time, the prediction parameters can be derived in different ways depending on the image format of the picture. The image format indicates a chroma subsampling rate and can be determined as one of 4:4:4, 4:2:2, or 4:2:0.

If the video format is not 4:4:4, the luma block is downsampled to the same size as the chroma block.

Figures 18 to 20 illustrate examples of downsampling a luma block.

For convenience of explanation, the video format is assumed to be 4:2:0.

When the video format is 4:2:0, the size of the chroma block corresponding to the 4x4 sized luma block is 2x2, as in the example illustrated in Fig. 18. In this case, by applying a down-sampling filter to the luma block, the 4x4 sized luma block can be reduced to 2x2. The following mathematical expression 5 shows how the down-sampling filter is applied.

In mathematical expression 5, Downsampled_Luma means a sample value in a down-sampled luma block, and Luma means a value of a luma sample before down-sampling. For example, Luma[0][0] can represent the position of the upper left sample in a luma block before down-sampling. Since the size of the down-sampled luma block is 2x2, the variables w and h representing the coordinates of the sample can have values ranging from 0 to 1, respectively.

When applying the down-sampling filter according to mathematical expression 5, the value of the down-sampled luma sample can be obtained by applying a cross-shaped down-sampling filter to the luma samples. For example, the value of the down-sampled luma sample at the (0, 0) position can be obtained by applying a down-sampling filter to the luma sample at the (0, 0) position, the upper luma sample at the (0, 0) position, the left luma sample at the (0, 0) position, the lower luma sample at the (0, 0) position, and the right luma sample at the (0, 0) position.

A downsampling filter of a different form than that illustrated in Fig. 19 may also be applied. For example, a 1D filter, a rectangular or square filter may be applied to obtain a downsampled luma sample. The 1D filter may have a size of 1x3 or 3x1, the rectangular filter may have a size of 2x3 or 3x2, and the square filter may have a size of 2x2 or 3x2.

The shape of the filter may be predefined in the encoder and decoder.

Alternatively, the shape of the filter can be adaptively determined based on at least one of the size/shape of the current block, the intra prediction mode applied to the luma block, whether the positions of the chroma samples match the positions of the luma samples, or the image format.

Alternatively, information indicating one of multiple filter candidates may be encoded and signaled.

Alternatively, depending on the downsampling location, the filter type may be different. For example, a 1D filter or a rectangular filter may be applied to luma samples located at the boundary of a luma block, while a cross-shaped filter may be applied to luma samples not located at the boundary of a luma block.

As illustrated in Figure 19, a downsampling filter can be applied to positions where both the x-axis coordinate and the y-axis coordinate are even numbers.

The application position of the down-sampling filter may also be set differently from that shown in Fig. 19. Fig. 20 shows various examples of the application position of the down-sampling filter.

After defining multiple candidates for the position to which downsampling is applied, one of the multiple candidates can be selected. For example, after defining the examples of (a) to (d) of Fig. 20 as multiple candidates, index information indicating one of the multiple examples can be encoded and signaled.

Alternatively, one of the multiple candidates may be selected based on whether the location of the chroma sample matches the location of the luma sample.

A downsampling filter can also be applied to reference samples around the luma block. Here, the reference sample can represent a previously restored sample. Specifically, a downsampling filter can be applied to at least one of an upper reference region adjacent to the upper side of the luma block or a left reference region adjacent to the left side, to obtain a downsampled luma reference sample.

A number of downsampled luma reference samples equal to the number of reference samples included in the reference area of the chroma block can be obtained.

Meanwhile, the reference area of the luma block can be called a luma reference area, and the reference area of the chroma block can be called a chroma reference area.

The inter-component prediction mode can be classified into an upper inter-component prediction mode, a left inter-component prediction mode, and an upper and left inter-component prediction mode, depending on the configuration of the reference region. When the upper inter-component prediction mode is selected, the reference region of each of the luma block and the chroma block consists of only the upper reference region. When the left inter-component prediction mode is selected, the reference region of each of the luma block and the chroma block consists of only the left reference region. When the upper and left inter-component prediction modes are selected, the reference region of each of the luma block and the chroma block can consist of an upper reference region and a left reference region.

Information indicating which of the upper inter-component prediction mode, the left inter-component prediction mode, and the upper and left inter-component prediction modes is applied to the current block can be explicitly encoded and signaled. For example, index information indicating the type of the inter-component prediction mode can be encoded and signaled.

Alternatively, one of the top inter-component prediction mode, the left inter-component prediction mode, and the top and left inter-component prediction mode may be selected based on at least one of the size/shape of the current block, whether the current block touches a CTU or picture boundary, or an intra prediction mode applied to the luma block.

For convenience of explanation, in the embodiments described below, it is assumed that the reference areas of each of the luma block and the chroma block include an upper reference area and a left reference area.

The shape of the down-sampling filter applied to the reference area of the luma block may be the same as the down-sampling filter applied to the luma block. Alternatively, the shape of the down-sampling filter applied to the reference area of the luma block may be different from the down-sampling filter applied to the luma block. Alternatively, the shape of the down-sampling filter applied to the upper reference area of the luma block may be different from the shape of the down-sampling filter applied to the left reference area of the luma block.

Meanwhile, the location where downsampling is applied within the reference region may be predefined in the encoder and decoder.

In other words, in a decoder, in the same way as the encoder, it may decide on its own where in the reference region downsampling is applied.

Figure 21 is a drawing to explain an example related to the location where down sampling is applied.

When the video format is 4:2:0, a 1x1 sized chroma block corresponds to a 2x2 sized luma block. Accordingly, a down-sampling filter can be applied to one of the four luma reference samples to derive a down-sampled luma reference sample corresponding to the chroma reference sample.

When four luma reference samples corresponding to one chroma reference sample are represented as A to D, down-sampling can be performed on each of positions A to D within the reference area, and then the cost for each position can be calculated. Here, the cost for a specific position can be derived based on the sum of the differences between the down-sampled luma reference sample obtained by applying a down-sampling filter centered on the position and the chroma reference sample corresponding to the position, or the sum of the absolute values of the differences. In this way, the cost derived based on the sum of the absolute values of the differences can be referred to as SAD (Sum of Difference).

Afterwards, the location with the lowest cost is determined as the optimal location, and the prediction parameter derivation process described below can be performed using the downsampled luma samples at the optimal location.

Alternatively, information indicating one of the multiple positions to which the down-sampling filter can be applied can be encoded and signaled. For example, in the example illustrated in FIG. 21, an index indicating one of positions A to D can be encoded and signaled. To this end, the encoder can obtain prediction parameters for each of the multiple positions to which the down-sampling filter can be applied, and encode and signal an index indicating a position used to derive an optimal prediction parameter among the multiple prediction parameters. Here, the optimal prediction parameter can be derived by a cost or RDO (Rate Distortion Optimization) of each of the prediction parameters.

Meanwhile, determining the optimal down-sampling application location within the upper reference area may be independent of determining the optimal down-sampling application location within the left reference area. In this case, the optimal down-sampling application location within the upper reference area and the optimal down-sampling application location within the left reference area may be different.

Using the downsampled luma reference samples and the reference samples of the chroma block, prediction parameters for the chroma block can be derived. The prediction parameters can include weights α and offsets β. The prediction parameters can be derived using the least square method, etc.

Alternatively, the weight α offset β can be derived based on the linearity of the maximum and minimum values of the downsampled luma reference samples and the maximum and minimum values of the chroma reference samples.

At this time, prediction parameters can be derived only by using chroma reference samples at predefined locations and corresponding down-sampled luma reference samples. In this case, the process of deriving prediction parameters can be simplified, and the complexity in the encoder and decoder can be reduced. For example, prediction parameters can be derived by using chroma reference samples at locations exemplified in the following mathematical expression 6.

In the above example, W and H represent the width and height of the chroma block, respectively. According to the above example, prediction parameters can be derived using four chroma reference samples and four corresponding down-sampled luma reference samples.

Prediction parameters can also be obtained using reference samples at different locations from the above example. For example, the locations of reference samples can be determined as in the following mathematical expressions 7 and 8.

After defining multiple candidates for the locations of reference samples, one of the multiple candidates can be selected. For example, each of the examples of mathematical expressions 6 to 8 listed above is set as a location candidate, and then reference samples can be selected according to one of the multiple location candidates.

Information for selecting one of the multiple location candidates can be encoded and signaled. For example, an index pointing to one of the multiple location candidates can be encoded and signaled.

Alternatively, one of the plurality of location candidates can be adaptively selected based on at least one of the size/shape of the current block, the color format, or whether the location of the chroma sample matches the location of the luma sample.

For example, if the current block is a square shape, the prediction parameter can be derived using the location candidate of Equation 6. On the other hand, if the current block is a non-square shape, the prediction parameter can be derived using the location candidate of Equation 7 or Equation 8. For example, if the current block is a non-square shape with a width greater than its height, the location candidate of Equation 7 can be used, and if the current block is a non-square shape with a height greater than its width, the location candidate of Equation 8 can be used.

Once the prediction parameters are derived, a prediction sample of a chroma block can be obtained based on the downsampled luma sample (S1720). For example, a prediction sample of a chroma block can be obtained according to the following mathematical expression 9.

In mathematical expression 9, PredChroma represents a prediction sample of a chroma block, and Downsampled_Luma represents a downsampled luma sample at a location corresponding to the chroma prediction sample.

Meanwhile, when the video format is 4:4:4, the above-described down sampling process can be omitted. That is, when the video format is 4:4:4, the process of performing down sampling on the restored samples in the luma block and the process of performing down sampling on the reference samples of the luma block can be omitted.

As another example, regardless of the image format, the reference region of the luma block may not be subjected to a downsampling filter. That is, instead of using the minimum and maximum values of the downsampled luma reference samples, the minimum and maximum values of the luma reference samples may be used when deriving the prediction parameters.

Reference samples for in-screen prediction can be filtered, and prediction samples of the current block can be derived using the filtered reference samples.

Meanwhile, whether to perform filtering on the reference samples can be determined based on whether a preset condition is satisfied. Here, the preset condition can be related to at least one of the index of the reference line, the size of the current block, the color component, or the intra prediction mode of the current block.

For example, filtering of reference samples can be performed only when the index of the reference line of the current block is 0, i.e., only when the reference line of the current block is an adjacent reference line. Meanwhile, the reference line may also be referred to as a reference sample line.

For example, filtering the reference sample may be performed only if the size of the current block is greater than or equal to a threshold. Here, the size of the current block may indicate at least one of whether the width of the current block, the height, or the number of samples in the current block (i.e., the product of the width and the height) is greater than or equal to the threshold.

For example, filtering the reference sample can only be performed if the predicted color component is a luma component.

For example, filtering a reference sample can be performed only if the intra prediction mode of the current block is one of the predefined intra prediction modes.

If it is decided to filter reference samples, filtering can be performed for each of the reference samples using adjacent reference samples.

Figure 22 is a diagram illustrating an example of filtering reference samples.

For convenience of explanation, the number of reference samples is assumed to be (2W + 2H + 1), where W and H represent the width and height of the current block, respectively.

Additionally, we assume that the coordinates of the upper left sample in the current block are (0, 0), and we call the buffer where the reference samples are stored as ref.

Even if it is decided to perform filtering on reference samples for the current block, whether or not filtering is performed may differ for each reference sample. For example, reference sample filtering may not be performed on the rightmost reference sample among the upper reference samples (i.e., position e) and the bottommost reference sample among the left reference samples (i.e., position a).

Reference samples can be filtered using adjacent reference samples adjacent to the reference sample. For example, left reference samples (i.e., reference samples included in region b) can be filtered using the reference sample adjacent to the top and the reference sample adjacent to the bottom. For example, Equation 10 shows an example in which the left reference sample is filtered.

The upper reference samples (i.e., the reference samples included in the d region) can be filtered using the reference samples adjacent to the right and the reference samples adjacent to the left. As an example, mathematical expression 11 shows an example in which the upper reference samples are filtered.

The upper left reference sample (i.e., location c) can be filtered using the reference samples adjacent to the bottom and the reference samples adjacent to the right. As an example, Equation 12 shows an example in which the upper left reference sample is filtered.

As in the examples described above, whether filtering is performed and the filter type can be determined depending on the location of the reference sample.

For convenience of explanation, a filter type that filters a reference sample by using reference samples belonging to the same reference line as the reference sample, as in the examples of Equations 10 to 12, may be referred to as a single line filter.

As another example, whether to perform filtering on the current block or at least one of the filter types may be adaptively determined based on the intra prediction mode of the current block.

Figure 23 is a diagram illustrating an example of filtering reference samples.

When the intra prediction mode of the current block is the planar mode or the DC mode, reference samples can be filtered based on a single line filter according to the embodiment described with reference to FIG. 22 (i.e., mathematical expressions 10 to 12). That is, when the intra prediction mode of the current block is the planar mode or the DC mode, reference samples can be filtered based on at least one adjacent reference sample belonging to the same reference line.

On the other hand, if the intra prediction mode of the current block is a directional mode, reference samples can be filtered using an angular filter, as in the example illustrated in Fig. 23. That is, if the intra prediction mode of the current block is a directional mode, reference samples can be filtered using reference samples belonging to neighboring reference lines.

The angular filter may filter reference samples belonging to the current reference line by using reference samples belonging to neighboring reference lines adjacent to the current reference line. Here, the current reference line may mean a reference line selected by a reference line index of the current block.

From the reference sample belonging to the current reference sample line, a reference sample belonging to a neighboring reference line can be selected according to the directionality according to the intra prediction mode of the current block. The selected reference sample may be called an angular neighboring reference sample or a directional neighboring reference sample. That is, the angular neighboring reference sample may be located on an angular line according to the intra prediction mode from the reference sample belonging to the current reference line.

By inputting the reference sample belonging to the current reference line and the angular neighboring reference sample into the filter, a filtered reference sample can be obtained. The filtered reference sample can be obtained by a weighted sum operation of the reference sample belonging to the current reference line and the angular neighboring reference sample.

At this time, when projection is performed from a reference sample belonging to the current reference line to a neighboring reference line, if the projected position does not exist at an integer position, a fractional position reference sample can be generated, and the generated fractional position reference sample can be set as an angular neighboring reference sample. Here, the fractional position reference sample can be obtained by interpolating integer position reference samples located on both sides of the projected position within the neighboring reference line.

Alternatively, for simplicity, the angular filter can be set to apply only for certain directional modes where the positions indicated by the angular lines are integer positions rather than fractional positions.

Figure 24 illustrates directional modes to which angular filters are applied.

As in the example illustrated in Fig. 24, an angular filter can be applied to at least one directional mode among the lower left 45 degree direction (a), the horizontal direction (b), the upper left 45 degree direction (c), the vertical direction (d), or the upper right 45 degree direction (e).

Alternatively, for simplicity, if the projected position is a fractional position, the integer position reference pixel closest to the projected position within the neighboring reference line can be set as the angular neighboring reference sample. That is, the integer position reference sample closest to the projected position can be set as the angular reference sample without interpolating the reference samples.

That is, if the intra prediction mode of the current block is a directional mode, one of the examples shown in (a) to (e) of Fig. 24 may be used to filter reference samples. In this case, quantization may be required to determine the filtering direction for each directional mode.

For example, in the example illustrated in FIG. 5, for directional modes having indices less than 10, filtering based on an angular filter may be performed in the lower left 45 degree diagonal direction (i.e., (a) of FIG. 24). For directional modes having indices greater than or equal to 10 and less than 26, filtering based on an angular filter may be performed in the horizontal direction (i.e., (b) of FIG. 24). For directional modes having indices greater than or equal to 26 and less than 42, filtering based on an angular filter may be performed in the upper left 45 degree diagonal direction (i.e., (c) of FIG. 24). For directional modes having indices greater than or equal to 42 and less than 56, filtering based on an angular filter may be performed in the vertical direction (i.e., (d) of FIG. 24). For directional modes with an index of 58, filtering based on an angular filter can be performed in the upper right diagonal direction (i.e., (e) of Fig. 24).

As another example, for simplicity, based on the directional mode in the upper left 45 degree direction (i.e., number 34 in Fig. 5), the directional modes can be divided into modes existing to the left of the reference mode (i.e., directional modes less than 34) and modes existing above the reference mode (i.e., directional modes greater than 34). In this case, filtering based on an angular filter may be performed in the horizontal direction (i.e., (b) of Fig. 24) for the modes existing to the left of the reference mode, and filtering based on an angular filter may be performed in the vertical direction (i.e., (d) of Fig. 24) for the modes existing above the reference mode.

Instead of allowing filtering based on angular filters for all directional modes, it is also possible to perform filtering based on angular filters only for some directional modes. That is, filtering based on angular filters can be performed only if the intra prediction mode of the current block is one of the predefined directional modes.

At this time, the number or range of directional prediction modes in which filtering based on the angular filter is performed can be set differently depending on the size of the current block.

Meanwhile, when calculating the weighted sum, the weight assigned to the reference sample at the filtering position (i.e., the reference sample belonging to the current reference line) may be set to have a value equal to or greater than the weight assigned to the angular neighboring reference sample. For example, the set of the weight assigned to the reference sample at the position to be filtered and the weight assigned to the angular neighboring reference sample may be {1/2, 1/2} or {3/4, 1/4}.

Depending on the reference line, the weights assigned to the reference samples at the positions to be filtered and the weights assigned to the angular neighboring reference samples can be adaptively determined based on at least one of the index of the reference line or the intra prediction mode.

Alternatively, information indicating the set of weights to be assigned to reference samples at the locations to be filtered and the set of weights to be assigned to angular neighboring reference samples may be encoded and signaled.

In the examples illustrated in FIGS. 23 and 24, filtering is performed on reference samples when the reference line of the current block is an adjacent reference line (i.e., a reference line with index 0). As in the illustrated examples, filtering may be set to be performed only when the reference line of the current block is an adjacent reference line. In this case, the adjacent reference line adjacent to the current block may be a reference line whose index is 1 greater than that of the reference line of the current block.

Additionally, depending on the intra prediction mode of the current block, either a single-line filter or an angular filter may be applied to filter the reference samples. For example, if the intra prediction mode of the current block is a non-directional mode, the reference samples may be filtered based on a single-line filter, and if the intra prediction mode of the current block is a directional mode, the reference samples may be filtered based on an angular filter.

Alternatively, even if the reference line is not an adjacent reference line, the reference samples may be filtered based on the angular filter. In this case, the neighboring reference line may be a reference line whose index is 1 greater than that of the current reference line. On the other hand, if the index of the current reference line is the maximum value, the neighboring reference line may be a reference line whose index is 1 less than that of the current reference line. Alternatively, if the index of the current reference line is the maximum value, reference sample filtering may be set not to be applied to the current block.

Alternatively, the neighboring reference line can be a reference line whose index is 1 less than that of the current reference line. If the index of the current reference line is the minimum (i.e., 0), the neighboring reference line can be a reference line whose index is 1 greater than that of the current reference line. Alternatively, if the index of the current reference line is the minimum, reference sample filtering can be set not to be applied to the current block.

Alternatively, if the reference line is an adjacent reference line (i.e., a reference line with an index of 0), the reference samples can be filtered using a single line filter, and if the reference line is not an adjacent reference line (i.e., a reference line with an index greater than 0), the reference samples can be filtered based on an angular filter.

Reference samples can also be filtered using multiple neighboring reference lines. For example, a first angular neighboring reference sample can be selected from a first reference line whose index is 1 greater than that of the current reference line, and a second angular neighboring reference sample can be selected from a second reference line whose index is 1 less than that of the current reference line. Then, by inputting the reference sample at the position to be filtered within the current reference line, the first angular neighboring reference sample, and the second angular neighboring reference sample into the filter, a filtered reference sample can be obtained.

Alternatively, a first angular neighboring reference sample may be selected from a first reference line whose index is 1 greater than that of the current reference line, and a second angular neighboring reference sample may be selected from a second reference line whose index is 2 greater than that of the current reference line.

Alternatively, a first angular neighboring reference sample may be selected from a first reference line whose index is 1 less than that of the current reference line, and a second angular neighboring reference sample may be selected from a second reference line whose index is 2 less than that of the current reference line.

Meanwhile, the weight assigned to the reference sample at the position to be filtered may have a value greater than the weight assigned to each of the first angular neighboring reference sample and the second angular neighboring reference sample. For example, the weight assigned to each of the reference sample at the position to be filtered, the first angular neighboring reference sample, and the second angular neighboring reference sample may be {1/2, 1/4, 1/4}.

Depending on the index of the reference line, at least one of the number or positions of neighboring reference lines for angular filtering may be different. For example, if the index of the reference line of the current block indicates a minimum value (i.e., 0, i.e., an adjacent reference line) or a maximum value, filtering based on the angular filter can be performed using one adjacent reference line adjacent to the current reference line. On the other hand, if the index of the reference line of the current block belongs to a predetermined range (e.g., a value excluding the minimum value and the maximum value), filtering based on the angular filter can be performed using two adjacent reference lines adjacent to the current reference line.

Alternatively, if the index of the current reference line is the minimum, filtering based on an angular filter can be performed using two neighboring reference lines having indices greater than those of the current reference line. On the other hand, if the index of the current reference line is the maximum, filtering based on an angular filter can be performed using two neighboring reference lines having indices less than those of the current reference line. If the index of the current reference line is not the minimum or maximum, the current reference line can exist between the first neighboring reference line and the second neighboring reference line.

Alternatively, information can be encoded and signaled to the decoder indicating what type of filtering to use.

The names of the syntaxes used in the above-described embodiments are merely named for convenience of explanation.

It is within the scope of the present disclosure to apply the embodiments described with a focus on the decoding process or the encoding process to the encoding process or the decoding process. It is also within the scope of the present disclosure to change the embodiments described in a certain order to a different order from the described order.

Although the above-described disclosure has been described based on a series of steps or a flow chart, this does not limit the chronological order of the invention, and may be performed simultaneously or in a different order as needed. In addition, each of the components (e.g., units, modules, etc.) constituting the block diagram in the above-described disclosure may be implemented as a hardware device or software, or a plurality of components may be combined to be implemented as a single hardware device or software. As an example, the hardware device may include at least one of a processor for performing a calculation, a memory for storing data, a transmitter for transmitting data, and a receiver for receiving data.

The above-described disclosure may be implemented in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., singly or in combination.

In addition, according to the present disclosure, a computer-readable recording medium storing a bitstream generated by the above-described encoding method can be provided. The bitstream can be transmitted by an encoding device, and a decoding device can receive the bitstream and decode an image.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions such as ROMs, RAMs, flash memories, and the like. The hardware devices may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa.

Embodiments of the present disclosure can be applied to electronic devices that encode or decode images.

Claims (15)

Step of determining the reference line of the current block; A step for determining the intra prediction mode of the current block; A step of filtering reference samples included in the above reference line; and A step of obtaining a prediction block of the current block based on the filtered reference samples and the intra prediction mode, The above filtering is performed by at least one of the first filter type or the second filter type, The above first filter type uses neighboring reference samples included in the above reference line, An image decoding method, characterized in that the second filter type uses angular neighboring reference samples included in neighboring reference lines adjacent to the reference line. In the first paragraph, A video decoding method, characterized in that one of the first filter type or the second filter type is selected based on the intra prediction mode of the current block. In the second paragraph, If the intra prediction mode is a non-directional mode, the reference samples are filtered based on the first filter type, An image decoding method, characterized in that when the intra prediction mode is a directional mode, the reference samples are filtered based on the second filter type. In the first paragraph, An image decoding method, characterized in that the above angular neighboring reference sample exists at a position projected to the neighboring reference line according to the directionality of the intra prediction mode from the reference sample including the reference line. In the fourth paragraph, An image decoding method, characterized in that when the projected position indicates a fractional position, the angular neighboring reference sample is derived by interpolating two integer position reference samples located on both sides of the projected position within the neighboring reference line. In the fourth paragraph, An image decoding method, characterized in that when the projected position indicates a fractional position, the integer position reference sample closest to the projected position within the neighboring reference sample line is set as the angular neighboring reference sample. In the first paragraph, An image decoding method, characterized in that the above-mentioned neighboring reference line is a reference line whose index is 1 larger or smaller than that of the above-mentioned reference line. In Article 7, If the index of the above reference line is not the maximum, the neighboring reference line has an index that is 1 greater than the above reference line, An image decoding method, characterized in that when the index of the above reference line is the maximum value, the neighboring reference line has an index that is 1 smaller than that of the above reference line. In the first paragraph, An image decoding method, characterized in that the filtered reference sample is obtained by weighting the reference sample within the reference line and the angular reference sample within the neighboring reference line. In Article 9, An image decoding method, characterized in that the weight assigned to the above reference sample has a larger value than the weight assigned to the angular reference sample. In the first paragraph, An image decoding method, characterized in that the filtered reference sample is obtained by weighting a reference sample within the reference line, a first angular reference sample within a first neighboring reference line adjacent to the reference line, and a second angular reference sample within a second neighboring reference line adjacent to the reference line. In Article 11, An image decoding method, characterized in that the reference line exists between the first neighboring reference line and the second neighboring reference line. In Article 11, An image decoding method, characterized in that, when performing filtering according to the second filter type, whether a plurality of neighboring reference lines are used is determined based on the index of the reference line. Step of determining the reference line of the current block; A step for determining the intra prediction mode of the current block; A step of filtering reference samples included in the above reference line; and A step of obtaining a prediction block of the current block based on the filtered reference samples and the intra prediction mode, The above filtering is performed by at least one of the first filter type or the second filter type, The above first filter type uses neighboring reference samples included in the above reference line, A video encoding method, characterized in that the second filter type uses angular neighboring reference samples included in neighboring reference lines adjacent to the reference line. Step of determining the reference line of the current block; A step for determining the intra prediction mode of the current block; A step of filtering reference samples included in the above reference line; and A step of obtaining a prediction block of the current block based on the filtered reference samples and the intra prediction mode, The above filtering is performed by at least one of the first filter type or the second filter type, The above first filter type uses neighboring reference samples included in the above reference line, A computer-readable recording medium storing a bitstream generated by a video encoding method, characterized in that the second filter type uses angular neighboring reference samples included in neighboring reference lines adjacent to the reference line.

Images