KR102746357B1 - Pyramid history map generating method for calculating feature map in deep learning based on convolution neural network and feature map generating method - Google Patents

Abstract

A method for generating a pyramid history map (B) for a feature map (A) is configured to include the steps of: determining a reference value corresponding to each convolution value based on whether pixel values of a first frame and a second frame included in a window area corresponding to each convolution value of the feature map (A) match, and generating a first layer including a plurality of reference values; dividing the first layer into a plurality of blocks, determining a block value corresponding to each block based on the reference value in each of the divided blocks, and generating a second layer including a plurality of block values; and generating a third layer based on the plurality of block values of the second layer.

Description

{PYRAMID HISTORY MAP GENERATING METHOD FOR CALCULATING FEATURE MAP IN DEEP LEARNING BASED ON CONVOLUTION NEURAL NETWORK AND FEATURE MAP GENERATING METHOD}

The present invention relates to a method for generating a pyramid history map and a method for generating a feature map using the pyramid history map. Specifically, the present invention relates to a method for generating a pyramid history map for calculating a feature map in deep learning based on a convolutional neural network (CNN) and a method for generating a feature map using the pyramid history map.

When searching for images or scenes in the past, search results were presented by finding similar pictures or using text words with captions (annotations). This required a large database of images or captions.

Recently, deep learning has been developed and has greatly improved the accuracy of existing image recognition, and has shown an accuracy level that can classify the category of objects in an image or recognize what objects are present. In addition, it has developed to the level of creating sentences that describe images or scenes in words.

Deep learning uses a method to extract image features using a convolutional neural network. In order to search and analyze video images, all images must be analyzed with convolutional layers, but there is a problem that the amount of calculation increases. In addition, as the depth increases, the amount of calculation increases multiplier.

With respect to a method for extracting features of such images, prior art Korean Patent No. 10-1349672 relates to a method for high-speed extraction of image features and a device supporting the same, and discloses a method for obtaining information on the size of a sliding window and step distance, detecting duplicate images in image analyzed data based on the sliding window information, and analyzing only data in a newly designated area, excluding the duplicate image portion among newly analyzed image data designated by the sliding window based on the detected duplicate images.

The present invention provides a method for generating a pyramid history map that enables the feature map of a previous frame to be reused in the current frame by comparing the previous frame with the current frame, and a method for generating a feature map using the pyramid history map.

The present invention provides a method for generating a pyramid history map that can efficiently reuse feature maps calculated from previous frames and reduce the amount of computation for feature maps by encoding overlapping parts between previous frames and current frames in a three-level pyramid manner, and a method for generating feature maps using the pyramid history map.

However, the technical tasks that this embodiment seeks to accomplish are not limited to the technical tasks described above, and other technical tasks may exist.

As a technical means for achieving the above-described technical task, one embodiment of the present invention can provide a method for generating a pyramid history map (B) for a feature map (A), the method including: a step of determining a reference value corresponding to each convolution value based on whether pixel values of a first frame and a second frame included in a window area corresponding to each convolution value of the feature map (A) match, and generating a first layer including a plurality of reference values; a step of dividing the first layer into a plurality of blocks, and determining a block value corresponding to each block based on the reference value in each of the divided blocks, and generating a second layer including a plurality of block values; and a step of generating a third layer based on the plurality of block values of the second layer.

The step of generating the first layer may include a step of determining a reference value corresponding to the convolution value as the first value when pixel values of the first frame and the second frame included in the window area corresponding to each convolution value all match, and otherwise determining the reference value as the second value.

The step of generating the second layer may include a step of determining the block value of the block as the third value when all reference values within each block are the first value, and otherwise determining the block value as the fourth value.

The step of generating the third layer may include a step of determining the value of the third layer as the fifth value when all block values in the second layer are the third value, and otherwise determining the value of the third layer as the sixth value.

The method may further include a step of reconstructing a pyramid history map (B) for the above feature map (A) to generate a pyramid history map (B') for the feature map (A') of the next depth.

The step of generating the pyramid history map (B') may include: a step of determining a reference value of the pyramid history map (B') based on a reference value of the pyramid history map (B), and generating a first layer of the pyramid history map (B') including a plurality of reference values; a step of dividing the first layer of the pyramid history map (B') into a plurality of blocks, and determining a block value corresponding to each block based on the reference value in each of the divided blocks, and generating a second layer of the pyramid history map (B') including a plurality of block values; and a step of generating a third layer of the pyramid history map (B') based on the plurality of block values of the second layer of the pyramid history map (B').

The step of generating the first layer of the above pyramid history map (B') is such that, when the window area is a k*l (k, l are natural numbers) matrix and the position of each second value of the first layer of the pyramid history map (B) is represented as (x, y) (x, y are integers greater than or equal to 0), a reference value within a rectangular area of [x-k+1, x]*[y-l+1, y] of the first layer of the pyramid history map (B') can be determined as the second value.

If the first frame and the second frame are m*n matrices (m, n are natural numbers) and the window area is a k*l (k, l are natural numbers) matrix, the first layer may be a (m-k+1)*(n-l+1) matrix.

The second layer may be a matrix of the first size, and the third layer may be a matrix of the second size smaller than the first size.

The above first frame and the above second frame may be consecutive frames in one image.

Another embodiment of the present invention provides a method for generating a pyramid history map (B) for a feature map (A), the method comprising: a step of determining a reference value corresponding to each convolution value based on whether a difference between pixel values of a first frame and a second frame included in a window area corresponding to each convolution value of the feature map (A) is less than a reference value, and generating a first layer including a plurality of reference values; a step of dividing the first layer into a plurality of blocks, and determining a block value corresponding to each block based on the reference value in each of the divided blocks, and generating a second layer including a plurality of block values; and a step of generating a third layer based on the plurality of block values of the second layer.

Another embodiment of the present invention provides a method for generating a feature map using a pyramid history map, which is generated based on pixel values of a first frame and a second frame and includes a first layer, a second layer, and a third layer, the method comprising: calculating a convolution value applied with a kernel to the first frame to generate a feature map (a) of the first frame; determining an overlapping area between the feature map (a) of the first frame and the feature map (b) of the second frame by referring to the pyramid history map; using a convolution value within the overlapping area of the feature map (a) as a convolution value within the overlapping area of the feature map (b); and calculating a convolution value applied with the kernel to the second frame for a remaining area of the feature map (b) excluding the overlapping area, to generate a feature map (b) of the second frame.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the problem solving means of the present invention described above, a method for generating a pyramid history map that enables reuse of information of a previous frame through comparison between a previous frame and a current frame, and a method for generating a feature map using the pyramid history map can be provided.

A method for generating a pyramid history map that can efficiently reuse a feature map calculated in a previous frame and reduce the amount of computation of the feature map by encoding the overlapping portion between the previous frame and the current frame in a three-level pyramid manner, and a method for generating a feature map using the pyramid history map can be provided.

When an image is operated through a neural network such as a CNN (Convolution Neural Network) for recognition of faces, objects, and object classification in an image, the amount of calculation can be drastically reduced by encoding overlapping areas in a pyramid format, thereby enabling real-time processing or obtaining results from a deeper neural network faster than before. A method for generating a pyramid history map and a method for generating a feature map using the pyramid history map can be provided.

FIG. 1 is a flowchart illustrating a method for generating a pyramid history map according to one embodiment of the present invention.
FIG. 2 is a drawing for explaining a window area corresponding to a convolution value according to one embodiment of the present invention.
FIG. 3 is a diagram for explaining a method for generating a first layer of a pyramid history map according to one embodiment of the present invention.
FIG. 4 is a drawing for explaining a method for generating a second layer and a third layer of a pyramid history map according to one embodiment of the present invention.
FIG. 5 is a drawing for explaining a method for reconstructing a pyramid history map according to one embodiment of the present invention.
FIG. 6 is a diagram for explaining a method for generating a first layer of a pyramid history map according to another embodiment of the present invention.
FIG. 7 is a flowchart illustrating a method for generating a feature map using a pyramid history map according to one embodiment of the present invention.
FIG. 8 is a drawing for explaining a method for generating a feature map of a first frame according to one embodiment of the present invention.
FIG. 9 is a drawing for explaining a method for determining an overlapping area between a first frame and a second frame according to one embodiment of the present invention.

Hereinafter, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are assigned similar drawing reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element in between. Also, when a part is said to "include" a component, this should be understood to mean that it may further include other components, unless specifically stated to the contrary, and does not preclude the presence or possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Some of the operations or functions described as being performed by a terminal or device in this specification may instead be performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by a server may also be performed by a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

FIG. 1 is a flowchart illustrating a method for generating a pyramid history map according to one embodiment of the present invention.

Referring to FIG. 1, in step S102, pixel values of the first frame and the second frame are compared to generate a first layer.

The first frame and the second frame may be frames of an image or video composed of a plurality of pixels. The first frame or the second frame may be a rectangular frame having a size of m pixels horizontally and n pixels vertically (m and n are natural numbers), in which case it may be expressed as an m*n matrix.

The first frame and the second frame may be consecutive frames in one video. For example, the first frame and the second frame may be two consecutive frames in a video in which one second consists of 30 frames, and the first frame and the second frame may overlap some or all of the pixels.

In step S102, comparing pixel values of the first frame and the second frame may be performed for each window area corresponding to each convolution value of the feature map, and a reference value corresponding to each convolution value may be determined based on whether pixel values of the first frame and the second frame included in each window area match, and a first layer including a plurality of reference values may be generated.

A feature map is the result of extracting feature values by applying a convolution kernel to a frame, and is composed of a matrix containing multiple convolution values.

The window area corresponding to each convolution value of the feature map refers to an area among the pixel areas of the frame to which the convolution kernel is applied when calculating each convolution value. The window area corresponding to the convolution value will be described below with reference to FIG. 2.

FIG. 2 is a drawing for explaining a window area corresponding to a convolution value according to one embodiment of the present invention.

Referring to Fig. 2, a 3*3 sized convolution kernel (203) can be applied to a 5*5 sized frame (201) to generate a 3*3 sized feature map (205). If the convolution kernel (203) is applied to the first region (209) of the frame (201), 1*1 + 1*0 + 1*1 + 0*0 + 1*1 + 1*0 + 0*0 + 1*1 = 4 is obtained, so the first convolution value (207) of the feature map (205) becomes 4. In the same manner, the convolution value is calculated by moving the convolution kernel (203) by 1 pixel within the frame (201).

At this time, the window area corresponding to the first convolution value (207) is the first area (209) of the frame (201), and the window area corresponding to the second convolution value (211) is the second area (213) of the frame (201).

Next, a method for generating the first layer of the pyramid history map will be described in detail with reference to FIG. 3.

FIG. 3 is a diagram for explaining a method for generating a first layer of a pyramid history map according to one embodiment of the present invention.

Referring to FIG. 3, the pixel values of the first frame (301) and the second frame (303) having a size of 8*8 can be compared to generate the first layer (305) having a size of 6*6. The size of the first layer (305) is the same as the size of the feature map to be generated using the pyramid history map, and is determined according to the size of the frame and the size of the convolution kernel used when generating the feature map. If the size of the frame is m*l and the size of the convolution kernel used when generating the feature map is k*l, the size of the first layer becomes (m-k+1)*(n-l+1) (m, l, k, n are natural numbers). The first layer (305) illustrated in FIG. 3 is the first layer of the pyramid history map that can be used when generating a feature map by applying a convolution kernel having a size of 3*3 to the first frame (301) and the second frame (303).

The first layer (305) includes a plurality of reference values, and the reference values can be determined as the first value or the second value. For example, the first layer (305) illustrated in FIG. 3 includes 6*6=36 reference values, and the first value is 1 and the second value is 0. In the following, it will be explained assuming that the first value is 1 and the second value is 0.

The reference value of the first layer (305) can be determined as 1 when the pixel values of the first frame and the second frame included in the window area corresponding to each convolution value all match, and 0 otherwise. The other case may be a case where at least one pixel value does not match.

For example, when comparing the pixel values included in the window area (307) of the first frame (301) corresponding to the first convolution value and the pixel values included in the window area (309) of the second frame (303), they all match as 12, 20, 30, 8, 11, 2, 5, 7, and 3. Therefore, the reference value corresponding to the first convolution value can be determined as 1.

In the same manner, the window area can be moved by 1 pixel within the first frame (301) and the second frame (303) and the pixel values included in the windows of both frames can be compared to determine the reference value.

In the case of the first frame (301) and the second frame (303) illustrated in Fig. 3, all pixel values are the same except for the pixel value within the rectangular area (313) at the upper right. Therefore, when the window area is moved by 1 pixel and reference values are determined for all window areas, the reference value of the rectangular area (315) at the upper right of the first layer (305) is determined as 0, and the reference values of the other areas are determined as 1.

Returning to FIG. 1 again, in step S104, the first layer is divided into a plurality of blocks, and the block value of each block is determined to generate the second layer. Then, in step S106, the third layer is generated based on the block value of the second layer. Hereinafter, with reference to FIG. 4, a method of generating the second layer and the third layer will be described.

FIG. 4 is a drawing for explaining a method for generating a second layer and a third layer of a pyramid history map according to one embodiment of the present invention.

Referring to FIG. 4, a first layer (410) may be divided into a plurality of blocks, for example, four blocks (411, 413, 415, 417), a block value corresponding to each block may be determined based on a reference value within each divided block, and a second layer (430) including a plurality of block values may be generated.

For example, if all reference values in each block are 1, the block value of the corresponding block can be determined as the third value, and if at least one reference value in the block is 0, the block value can be determined as the fourth value. For example, FIG. 4 illustrates a case where the third value is 1 and the fourth value is 0. In the following, it will be explained assuming that the third value is 1 and the fourth value is 0.

Referring to FIG. 4, since all reference values in the first block (411) of the first layer (410) are 1, the block value (431) of the first block (411) is determined as 1. Meanwhile, since the second block (413) includes an area where the reference value is 0, the block value (433) of the second block (413) is determined as 0. Similarly, the block value (435) of the third block (415) is determined as 1, and the block value (437) of the fourth block (417) is determined as 0.

As described above, a second layer (430) including a plurality of block values can be generated based on the reference values of the first layer. The second layer can be a matrix of the first size, for example, a matrix of the size of 2*2 as illustrated in FIG. 4.

Next, a method of generating a third layer (450) based on the block value of the second layer (430) is described.

If all block values of the second layer (430) are 1, the third layer can be determined as the fifth value, and if at least one block value is 0, the third layer can be determined as the sixth value to generate the third layer. For example, the fifth value can be 1 and the sixth value can be 0. In the following, it will be explained assuming that the fifth value is 1 and the sixth value is 0.

In the embodiment illustrated in Fig. 4, since the second layer (430) includes a block with a block value of 0, the value of the third layer (450) is determined to be 0.

As described above, a first layer (410), a second layer (430), and a third layer (450) constituting a pyramid history map can be created.

When a pyramid history map (B) for a feature map (A) is generated according to a method for generating a pyramid history map according to an embodiment illustrated in FIG. 1, the generated history map (B) can be reconstructed to generate a pyramid history map (B') for a feature map (A') of the next depth.

FIG. 5 is a drawing for explaining a method for reconstructing a pyramid history map according to one embodiment of the present invention.

Referring to FIG. 5, the first layer (510) of the pyramid history map (B) for the feature map (A) can be reconstructed to generate the first layer (530) of the pyramid history map (B') for the feature map (A') of the next depth. The embodiment illustrated in FIG. 5 is a case where a convolution kernel of size 3*3 is used, and if the size of the first layer (510) is 6*6, the size of the reconstructed first layer (530) becomes 4*4.

The reference value of the reconstructed first layer (530) can be determined based on the reference value of the first layer (510). For example, as illustrated in FIG. 5, if the reference value of the first location (511) of the first layer (510) is 0, the reference value of the rectangular area (531) of the reconstructed first layer (530) that includes 2 spaces to the left and 2 spaces above from the first location can be determined as 0.

In other words, if the reference value of the 4th row, 3rd column location (511) in the 1st layer (510) is 0, the reference value of the rectangular area (531) from the 2nd row, 1st column to the 4th row, 3rd column of the reconstructed 1st layer (530) can be determined as 0. This can be understood by considering the case where a reference value of 0 is included in the kernel when moving the 1st layer (510) by 1 pixel by applying a 3*3 kernel.

For an arbitrary location, if the convolution kernel is a k*l matrix, and the reference value is 0 at the location of (x, y) in the x row and y column of the first layer (510), the reference value within the rectangular area ([x-k+1, x]*[y-l+1, y]) from the (x-k+1) row and (y-l+1) column to the x row and y column of the reconstructed first layer (530) can be determined as 0.

As described above, based on the reference value of the first layer (510) for the feature map (A), the first layer (530) for the feature map (A') of the next depth can be reconstructed. The method for generating the second layer and the third layer of the pyramid history map (B') is the same as the method for generating the second layer and the third layer of the pyramid history map (B), and therefore, the description is omitted.

As explained above, the pyramid history map for the feature map of the next depth can be generated by simply reconstructing the pyramid history map generated for the feature map of the previous depth, so there is no need to generate the pyramid history map by comparing pixels for all feature maps of different depths.

FIG. 6 is a diagram for explaining a method for generating a first layer of a pyramid history map according to another embodiment of the present invention.

Referring to FIG. 6, the second frame (603) has differences in pixel values from the first frame (601) in the first pixel area (605) and the second pixel area (607). The first pixel area (605) has a small difference in pixel values (a difference value of less than 2), and the second pixel area (607) has a large difference in pixel values.

According to the method for generating a pyramid history map according to one embodiment of the present invention described in FIG. 1, since the reference value is determined based on whether the pixel values of the first frame (601) and the second frame (603) match, the reference value of the area (609) corresponding to the first pixel area (605) of the first layer (608) generated according to one embodiment is determined to be 0.

A method for generating a pyramid history map according to another embodiment of the present invention differs in that a reference value is determined based on whether the difference between pixel values of the first frame (601) and the second frame (603) is less than a reference value. The reference value may be determined based on the average and variance of pixel values of the first frame (601) or the second frame (603).

In the case of the first layer (611) generated according to another embodiment, if the reference value is 2 or more, the reference value of the area (613) corresponding to the first pixel area (605) is determined as 1.

According to a method for generating a pyramid history map according to another embodiment of the present invention, even if there is a difference in pixel values between two frames, if the difference is slight, it is determined that the two frames overlap with respect to the corresponding area. This is because even if an area with a slight difference in pixel values is treated as an overlapping area, it does not affect feature extraction of the frame.

The method for generating a pyramid history map according to one embodiment and another embodiment of the present invention has been described above, and the method for utilizing the pyramid history map will be described below.

FIG. 7 is a flowchart illustrating a method for generating a feature map using a pyramid history map according to one embodiment of the present invention.

Referring to Fig. 7, a feature map of the first frame is generated in step S701. The feature map generation in step S701 is the same as a conventional feature map generation method. That is, a feature map is generated by applying a convolution kernel to the entire area of the first frame and calculating a convolution value.

In step S703, an overlapping area between the first frame and the second frame is determined, and in step S705, the convolution value of the feature map of the first frame for the overlapping area is reused in the feature map of the second frame. Therefore, the amount of computation required to generate the feature map of the second frame can be reduced by the amount of reuse of the convolution value.

In step S707, a feature map of the second frame is generated by calculating convolution values for the remaining areas excluding the overlapping areas.

Hereinafter, a method for generating a feature map of the first frame through a convolution operation is described with reference to FIG. 8, and then, a method for determining an overlapping area between the first frame and the second frame and reusing the convolution value using a pyramid history map is described with reference to FIG. 9.

FIG. 8 is a drawing for explaining a method for generating a feature map of a first frame according to one embodiment of the present invention.

Referring to Fig. 8, a convolution kernel (803) is applied to the first frame (801) to generate a feature map (805). The first convolution value of the feature map (805), 61, is obtained through 9 multiplications and 8 additions, as in 12*1 + 20*0 + 30*1 + 8*0 + 11*1 + 2*0 + 5*1 + 7*0 + 3*1 = 61. Accordingly, a total of 9*36 = 324 multiplications and 8*36 = 288 additions must be performed to generate the feature map (805).

FIG. 9 is a drawing for explaining a method for determining an overlapping area between a first frame and a second frame according to one embodiment of the present invention.

Referring to FIG. 9, a process of determining an overlapping area between a feature map (805) of a first frame and a feature map (903 or 915) of a second frame is illustrated by sequentially referencing the third layer (901), the second layer (905), and the first layer (909) of the pyramid history map.

First, with reference to the third layer (901), if the value of the third layer is 1, the entire area of the feature map (805) of the first frame and the feature map (903) of the second frame can be determined as an overlapping area. In this case, all convolution values of the feature map (805) of the first frame can be reused in the feature map (903) of the second frame.

If the value of the third layer (901) is 0, the second layer (905) is referenced. With reference to the second layer (905), the area (907) of the second frame corresponding to the block whose block value of the second layer is 1 is determined as an overlapping area, and the convolution value of the feature map (805) of the first frame can be reused for the overlapping area.

Next, for blocks in which the block value of the second layer (905) is 0, an area corresponding to an area (911) in which the reference value of the first layer (909) is 1 can be additionally determined as an overlapping area by referring to the first layer (909).

The convolution value of the feature map (805) of the first frame can be reused for the overlapping area (913) determined by sequentially referring to the third layer (901), the second layer (905), and the first layer (909). Accordingly, since the convolution operation only needs to be performed for the remaining areas excluding the overlapping area, the amount of calculation required to generate the feature map of the second frame can be reduced, and the feature map can be generated efficiently.

The pyramid history map according to the present invention can also be utilized in a system that recognizes objects/people and recognizes actions of the objects or people by utilizing a 3D convolutional neural network. For motion recognition, when 3 to 5 frames are viewed as one block, the block at a specific point in time is compared with the next block to construct the pyramid history map as described above, so that the amount of redundant calculations for pixels in a part that has no change compared to the previous block can be reduced.

The pyramid history map according to the present invention has the advantage that the larger the screen size, the greater the reduction in the amount of calculation. This is because pixel-level calculations are required to recognize people/objects using a convolutional neural network.

For example, if we apply UHD 4k, the required calculation amount when applying a 3x3 convolution kernel to (4096) x (2160) pixels is approximately 8 million times. If about 3 convolution kernels are used, 24 million calculations are required. If the depth is 10, 240 million calculations are required. If the previous frame and the current frame overlap by 1/3, the calculation amount can be reduced by approximately 1/3, reducing the calculation amount to 80 million times.

The method for generating a pyramid history map described through FIG. 1 and the method for generating a feature map using the pyramid history map described through FIG. 7 can also be implemented in the form of a recording medium including a program stored on a computer-executable medium or instructions executable by a computer. The computer-readable medium can be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium can include both computer storage media and communication media. The computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. The communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism, and includes any information delivery media.

The above description of the present invention is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential characteristics of the present invention. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and likewise, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

410: Layer 1
430: 2nd layer
450: 3rd layer

Claims (17)

A method for generating a pyramid history map (B) for a feature map (A) performed by a program stored in a medium executed by a computer,
A step of determining a reference value corresponding to each convolution value based on whether the pixel values of the first frame and the second frame included in the window area corresponding to each convolution value of the feature map (A) match, and generating a first layer including a plurality of reference values;
A step of dividing the first layer into a plurality of blocks, determining a block value corresponding to each block based on a reference value within each divided block, and generating a second layer including a plurality of block values; and
A step of generating a third layer based on the plurality of block values of the second layer
A method for creating a pyramid history map, comprising:
In paragraph 1,
The step of generating the above first layer is:
A step of determining a reference value corresponding to the convolution value as the first value if the pixel values of the first frame and the second frame included in the window area corresponding to each of the above convolution values are all the same, and otherwise determining the reference value as the second value.
A method for creating a pyramid history map, comprising:
In the second paragraph,
The step of creating the above second layer is:
If all reference values within each block above are the first value, the block value of the block is determined as the third value, and in other cases, the block value is determined as the fourth value.
A method for creating a pyramid history map, comprising:
In the third paragraph,
The steps to create the third layer are
If all block values in the second layer are the third value, a step of determining the value of the third layer as the fifth value, and otherwise determining the value of the third layer as the sixth value
A method for creating a pyramid history map, comprising:
In paragraph 4,
A step of reconstructing a pyramid history map (B) for the above feature map (A) to generate a pyramid history map (B') for the feature map (A') of the next depth.
Including more,
The steps for generating the above pyramid history map (B') are:
A step of determining a reference value of the pyramid history map (B') based on a reference value of the pyramid history map (B), and generating a first layer of the pyramid history map (B') including a plurality of reference values;
A step of dividing the first layer of the pyramid history map (B') into a plurality of blocks, determining a block value corresponding to each block based on a reference value within each divided block, and generating a second layer of the pyramid history map (B') including a plurality of block values; and
A step of generating a third layer of the pyramid history map (B') based on the plurality of block values of the second layer of the pyramid history map (B')
A method for creating a pyramid history map, comprising:
In paragraph 5,
The step of generating the first layer of the above pyramid history map (B') is:
A method for generating a pyramid history map, wherein the window area is a k*l (k, l are natural numbers) matrix, and when the position of each second value of the first layer of the pyramid history map (B) is represented as (x, y) (x, y are integers greater than or equal to 0), a reference value within a rectangular area of [x-k+1, x]*[y-l+1, y] of the first layer of the pyramid history map (B') is determined as the second value.
In paragraph 1,
A method for generating a pyramid history map, wherein if the first frame and the second frame are m*n matrices (m, n are natural numbers) and the window area is a k*l (k, l are natural numbers) matrix, the first layer is a (m-k+1)*(n-l+1) matrix.
In paragraph 1,
A method for generating a pyramid history map, wherein the second layer is a matrix of the first size.
In Article 8,
A method for generating a pyramid history map, wherein the third layer is a matrix of a second size smaller than the first size.
In paragraph 1,
A method for generating a pyramid history map, wherein the first frame and the second frame are consecutive frames in one image.
A method for generating a pyramid history map (B) for a feature map (A) performed by a program stored in a medium executed by a computer,
A step of determining a reference value corresponding to each convolution value based on whether the difference between the pixel values of the first frame and the second frame included in the window area corresponding to each convolution value of the feature map (A) is less than a reference value, and generating a first layer including a plurality of reference values;
A step of dividing the first layer into a plurality of blocks, determining a block value corresponding to each block based on the reference value within each divided block, and generating a second layer including a plurality of block values; and
A step of generating a third layer based on the plurality of block values of the second layer
A method for creating a pyramid history map, comprising:
In Article 11,
A method for generating a pyramid history map, wherein the reference value is determined based on the average and variance of pixel values of the first frame or the second frame.
A method for generating a feature map using a pyramid history map including a first layer, a second layer, and a third layer, the pyramid history map being generated based on pixel values of a first frame and a second frame performed by a program stored in a medium executed by a computer,
A step of generating a feature map (a) of the first frame by calculating a convolution value by applying a kernel to the first frame;
A step of determining an overlapping area between the feature map (a) of the first frame and the feature map (b) of the second frame by referring to the pyramid history map;
A step of using the convolution value within the overlapping area of the above feature map (a) as the convolution value within the overlapping area of the above feature map (b); and
A step of generating a feature map (b) of the second frame by calculating a convolution value that applies the kernel to the second frame for the remaining area excluding the overlapping area of the feature map (b)
Including
The above pyramid history map is,
A step of determining a reference value corresponding to each convolution value based on whether the pixel values of the first frame and the second frame included in the window area corresponding to each convolution value of the feature map (a) match, and generating the first layer including a plurality of reference values;
A step of dividing the first layer into a plurality of blocks, determining a block value corresponding to each block based on the reference value within each divided block, and generating the second layer including a plurality of block values, and
A method for generating a feature map, the method comprising: generating a third layer based on the plurality of block values of the second layer.
In Article 13,
A method for generating a feature map, wherein the step of determining the overlapping area determines the overlapping area by sequentially referring to the third layer, the second layer, and the first layer of the pyramid history map.
delete In Article 14,
The step of generating the first layer includes a step of determining a reference value corresponding to the convolution value as the first value if the pixel values of the first frame and the second frame included in the window area corresponding to each convolution value all match, and otherwise determining the reference value as the second value.
The step of generating the second layer includes the step of determining the block value of the block as the third value if all reference values within each block are the first value, and otherwise determining the block value as the fourth value.
A method for generating a feature map, wherein the step of generating the third layer includes the step of determining the value of the third layer as the fifth value when all block values in the second layer are the third value, and otherwise determining the value of the third layer as the sixth value.
In Article 16,
The step of determining the above overlapping area is:
Referring to the third layer, if the value of the third layer is the fifth value, a step of determining the entire area of the feature map (a) and the feature map (b) as an overlapping area;
If the value of the third layer is the sixth value, a step of determining the area of the feature map (a) and the feature map (b) corresponding to the block whose block value is the third value as the overlapping area by referring to the second layer; and
A step of determining an area of the feature map (a) and the feature map (b) corresponding to an area where the reference value is the first value as the overlapping area by referring to the first layer for a block where the block value is the fourth value.
A method for generating a feature map, comprising:

Images