CN115496818B - Semantic graph compression method and device based on dynamic object segmentation - Google Patents

Abstract

The invention discloses a semantic map compression method and device based on dynamic object segmentation. The method first divides the simulation scene into two parts, the static background and the dynamic object, and draws the simulation scene to obtain a semantic map; all dynamic objects in the semantic map are segmented out. Get the semantic submap, and use the adjacent pixels of the dynamic object semantic submap to fill the static background semantic map; finally use the coding algorithm to encode all the dynamic object semantic submap and the filled static background semantic map respectively. This method separates the static background from the dynamic objects, and splits a semantic map into a static background semantic map and multiple dynamic object semantic sub-maps, which reduces the pixel mutation in the semantic map and increases the continuity of data distribution. The compression ratio of semantic graph encoding has been significantly improved.

Description

A Semantic Graph Compression Method and Device Based on Dynamic Object Segmentation

technical field

The invention relates to the field of image data compression, in particular to a semantic map compression method and device based on dynamic object segmentation.

Background technique

As the core technology of computer vision and image understanding, object detection algorithm is an important research direction in the field of computer vision and the basis of other complex vision tasks. The application scenarios of target detection algorithms include image description, scene understanding, image segmentation, target tracking, event detection, etc. In addition, they are also widely used in judgment scenarios such as automatic driving, medical images, and drone navigation.

The target detection algorithm based on machine learning needs to provide a large amount of real-valued data during the training process, and these real-valued data mainly include semantic segmentation graphs, that is, semantic graphs. The semantic map divides a picture or video into multiple blocks according to the similarities and differences of categories, and realizes the classification at the pixel level. The truth value of the semantic map is generally manually annotated or automatically generated by a simulation program. Due to the high degree of customization of the simulation algorithm and the ability to quickly generate massive amounts of data, it has become one of the important methods for generating semantic graph truth datasets. Traditional simulation semantic graphs are generally not coded, or directly coded using compression algorithms such as run-length coding, which does not fully consider the characteristics of the simulation scene, resulting in low compression rates and limited storage and data transmission efficiency.

Contents of the invention

In order to solve the deficiencies of the prior art and realize more efficient semantic map compression, the present invention adopts the following technical solutions:

A semantic map compression method based on dynamic object segmentation, comprising the following steps:

S1, initializing a simulation scene, the simulation scene is composed of a static background and a dynamic object;

S2, update and draw the simulation scene, and obtain the two-dimensional bounding volume of the semantic map and all dynamic objects in the coordinate system of the semantic map;

S3, using the two-dimensional bounding volume to segment the semantic data of the dynamic object to form several dynamic object semantic subgraphs; for the remaining static background semantic graph, use the pixels adjacent to each dynamic object semantic subgraph to fill the corresponding image area;

S4, use the encoding algorithm to encode all the dynamic object semantic sub-graphs and the filled static background semantic graph respectively.

Further, all objects in the simulation scene in S1 are assigned an ID, and objects with the same type of semantics have the same ID; each ID corresponds to a unique color, and different IDs correspond to different colors.

Further, the update of the simulated scene in S2 includes update of poses of all dynamic objects and rendering perspectives; each pixel in the semantic graph corresponds to an object uniquely, and is colored with the color corresponding to the object ID.

Further, in S2, the two-dimensional bounding volume of the dynamic object in the semantic map coordinate system should include all the pixels of the dynamic object on the semantic map.

Further, in S3, the semantic data of the dynamic object is segmented using the two-dimensional bounding volume in S2 to form several semantic subgraphs, which are specifically implemented through the following substeps:

(1) According to the size of the two-dimensional bounding volume of each dynamic object, initialize the corresponding semantic submap data, where each element of the semantic submap is initialized to (R:0, G:0, B:0);

(2) Traverse each pixel of the semantic map, and for each pixel, perform the following processing:

Determine whether the pixel coordinates of the pixel are within the range of the two-dimensional bounding volume of a dynamic object, if so, calculate the relative coordinates of the pixel in the semantic sub-image coordinate system of the dynamic object, and write the In the corresponding element of the semantic subgraph of the dynamic object, otherwise ignore the pixel and continue to traverse the next pixel to obtain the semantic subgraph of a dynamic object.

Further, in the S3, for the rest of the static background semantic map, the pixels adjacent to each dynamic object semantic sub-map are used to fill the corresponding image area through the following sub-steps:

(1) Find the edge pixels around the semantic submap of the dynamic object. If it exceeds the range of the semantic map, the edge pixels are considered invalid;

(2) Count all valid edge pixels, and fill the corresponding image area of the dynamic object semantic submap in the semantic map according to the values of these edge pixels.

Further, the encoding result of each dynamic object semantic subgraph in S4 should be accompanied by corresponding two-dimensional bounding volume information.

A semantic graph compression device based on dynamic object segmentation, the device includes one or more processors for realizing the above-mentioned semantic graph compression method based on dynamic object segmentation.

A computer-readable storage medium stores a program on it. When the program is executed by a processor, a semantic map compression method based on dynamic object segmentation is realized.

The beneficial effects of the present invention are as follows:

The invention discloses a semantic map compression method based on dynamic object segmentation. The method firstly divides the simulation scene into two parts, the static background and the dynamic object, and draws the simulation scene to obtain the semantic map; all the dynamic objects in the semantic map are separated to obtain the semantic map. Submap, and use the adjacent pixels of the dynamic object semantic submap to fill the static background semantic map; finally use the coding algorithm to encode all the dynamic object semantic submap and the filled static background semantic map respectively. This method separates the static background from the dynamic objects, and splits a semantic map into a static background semantic map and multiple dynamic object semantic sub-maps, which reduces the pixel mutation in the semantic map and increases the continuity of data distribution. The compression ratio of semantic graph encoding has been significantly improved.

Description of drawings

Fig. 1 is a step diagram of a semantic graph compression method based on dynamic object segmentation in an exemplary embodiment.

Fig. 2 is a schematic diagram of a static background and dynamic objects in a simulation scene in an exemplary embodiment.

Fig. 3 is a schematic diagram of segmentation and compression of a dynamic object in a semantic graph in an exemplary embodiment.

Fig. 4 is a schematic diagram of a semantic map compression device based on dynamic object segmentation in an exemplary embodiment.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, rather than all Example. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts are within the protection scope of the present invention.

In one embodiment, as shown in Figure 1, a semantic map compression method and device based on dynamic object segmentation is proposed. The method first divides the simulation scene into two parts, the static background and the dynamic object, and draws the simulation scene to obtain the semantic map. ;Segment all the dynamic objects in the semantic map to get the semantic submap, and use the adjacent pixels of the dynamic object semantic submap to fill the static background semantic map; finally use the encoding algorithm to encode all the dynamic object semantic submap and the filled static background Semantic map.

The method specifically includes the following steps:

Step 1, initialize the simulation scene, which consists of static background and dynamic objects.

As shown in Figure 2, in this embodiment, the simulation scene is a digital twin scene of a park, which is composed of several types of models such as oblique photography model, art handmade model and procedural generation model, and is divided into static background and dynamic objects. kind. The static background includes the sky, terrain, road surface, curbs, street lights, buildings, plants, etc., and the dynamic objects include non-motor vehicles, motor vehicles, pedestrians, etc.

The image rendering interface of the simulation program uses the open source and open graphics rendering library OpenGL (Open Graphics Library). OpenGL is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. The OpenGL interface is usually used to interact with the graphics processing unit (GPU) to achieve hardware-accelerated rendering, including nearly 350 different function calls, used to draw from simple graphics bits to complex three-dimensional scenes; OpenGL is often used in CAD , virtual reality, scientific visualization programs and video game development, including 7 major functions: building 3D models, graphics transformation, color mode, lighting and material settings, texture mapping, image enhancement functions and bitmap display extensions and double buffering functions .

All objects in the simulation scene are assigned an ID. Objects with the same type of semantics have the same ID. The same type of semantics refers to the same type of object in the target detection algorithm; each ID corresponds to a unique color, and different IDs Corresponding colors are different. The corresponding relationship between ID and color of different objects is shown in Table 1.

A semantic map simulation camera is also placed in the simulation scene, and the semantic map is generated by scene rendering. Set the field of view of the semantic map simulation camera to 72 degrees, the aspect ratio to 16:9, and the resolution of the target image to 1280*720. The semantic graph simulation camera also includes a motion control module to control its motion logic in the scene: preset a motion trajectory, each point of the motion trajectory contains a set of pose data, and the semantic graph simulation camera follows the motion trajectory Move at a constant speed according to the preset speed, and draw and generate a semantic map according to a fixed time interval.

Table 1 Correspondence between ID and color of different objects

Step 2, update and draw the simulation scene, and obtain the semantic map and the two-dimensional bounding volume of all dynamic objects in the semantic map coordinate system.

The two-dimensional bounding volume here can be a two-dimensional bounding box, such as a rectangular box, or other regular shapes, such as a circle, an ellipse, etc., or other irregular shapes.

Semantic segmentation is an important part of image understanding in image processing and computer vision technology. Semantic segmentation classifies each pixel in the image, determines the category of each point, such as belonging to the background, edge or body, etc., and distinguishes the instance segmentation. Semantic segmentation does not separate instances of the same class; it only cares about the class of each pixel, and if there are two objects of the same class among the input objects, the segmentation itself will not distinguish them as separate objects. The semantic segmentation algorithm will output a semantic segmentation image, referred to as a semantic map.

In this embodiment, the semantic map divides a picture or video into multiple blocks according to the similarities and differences of categories, so as to realize the classification at the pixel level; each pixel in the semantic map obtained by drawing is uniquely corresponding to an object, using The color corresponding to the object ID is colored.

In this embodiment, the update of the simulation scene includes the pose update of all dynamic objects and semantic map simulation cameras, and the update frequency is set to 60HZ; all dynamic objects, including non-motor vehicles, motor vehicles and pedestrians, are bone skin models , its pose update includes the following two steps:

(1) Receive the dynamic object status data from the server, parse out the position and rotation data, and set it to the corresponding dynamic object according to the object identifier string;

(2) According to the current simulation moment and the animation state of the dynamic object, perform GPU-based animation skin calculation and update each vertex of the dynamic object model.

The semantic map simulation camera updates its position and posture by sampling the preset motion trajectory, and the specific steps are as follows:

(1) According to the current simulation time and the preset semantic map simulation camera movement speed, calculate the path length of the current movement;

(2) If the path length is greater than the preset motion track length, stop the simulation process and exit the program; otherwise use the path length to sample the preset motion track, and set the obtained position and attitude to the semantic map simulation camera.

Wait for the poses of all dynamic objects and semantic map simulation cameras to be updated, and start the drawing operation. Before drawing each scene object, first call the glUniform1i function to pass the object ID as a uniform variable to the pixel shader, and then call the glDrawElements function to draw the vertex data of the model. In the pixel shader, the output pixel color PixelColor and the calculation formula of the object ID are as follows:

PixelColor.r=ID*20

PixelColor.g=ID*20

PixelColor.b=ID*20

In addition to obtaining the drawn semantic map data, this embodiment will also generate a two-dimensional bounding box of all dynamic objects in the semantic map coordinate system, and the area covered by the two-dimensional bounding box includes all pixels of the dynamic object on the semantic map ; For each dynamic object, the specific generation algorithm of its two-dimensional bounding box is as follows:

(1) Traverse all vertices of the dynamic object, calculate the maximum and minimum values in the three axes respectively, and construct an axially aligned three-dimensional bounding box (AABB, Axis-Aligned Bounding Box), which can contain All vertices of the dynamic object;

(2) Traverse the 8 3D corners of the 3D bounding box, and transform them into 2D corners in the semantic map coordinate system through the model matrix, view matrix, perspective matrix, and viewport transformation matrix in turn;

(3) Traverse the two-dimensional corner points obtained in the previous step, calculate the maximum and minimum values in the two axes respectively, and construct an axially aligned two-dimensional bounding box containing all two-dimensional corner points, the two-dimensional bounding box is the two-dimensional bounding box of the dynamic object.

Step 3, use the two-dimensional bounding volume in step 2 to segment the semantic data of the dynamic object to form several semantic submaps; for the rest of the static background semantic map, use the pixels adjacent to the semantic submap of each dynamic object to fill the corresponding image area.

In this embodiment, as shown in Figure 3, the semantic data of all dynamic objects is segmented from the original semantic graph using the two-dimensional bounding box of the dynamic object to form a semantic sub-graph; the semantic graph data and the semantic sub-graph data use two-dimensional The array is stored, and the elements of the two-dimensional array are the RGB color values of the pixels;

According to the size of the two-dimensional bounding volume of each dynamic object, initialize the corresponding semantic subgraph data, where each element of the semantic subgraph is initialized to (R:0, G:0, B:0); traverse the original semantic graph For each pixel, for each pixel, the following processing is performed:

Determine whether the pixel coordinates of the pixel are within the range of the two-dimensional bounding volume of a dynamic object, if so, calculate the relative coordinates of the pixel in the semantic sub-image coordinate system of the dynamic object, and write the In the corresponding element of the semantic submap of the dynamic object; otherwise, the pixel is ignored, and the next pixel is traversed to obtain the semantic submap of a dynamic object.

For each dynamic object semantic subgraph, find all the edge pixels of the dynamic object semantic subgraph in the circle around the original semantic map, if it exceeds the range of the semantic map, it will be considered invalid; select the mode of all valid edge pixels as the specified The adjacent pixels are used to fill the corresponding image area of the dynamic object semantic submap in the original semantic map. Wherein, the edge pixels can be one or more circles; when specifying adjacent pixels, in addition to selecting the mode of all effective edge pixels, the average number and median of all effective edge pixels can also be selected.

In this embodiment, if there is no dynamic object in the simulation scene, the above steps of segmenting the dynamic object and filling the original semantic map are not performed.

Step 4: Encode all dynamic object semantic subgraphs and filled static background semantic graphs respectively using an encoding algorithm.

In this embodiment, run-length encoding is used to compress and encode all dynamic object semantic subgraphs and filled static background semantic graphs; run-length encoding (RLE, Run-Length Encoding) is a form of lossless data compression, in which data runs (in A sequence of occurrences of the same data value in many consecutive data elements) is stored as individual data values and counts, rather than as raw runs. This works best for data that contains many such runs, such as simple graphic images such as icons, line drawings, and animations. For files that don't have many repeating characters, run-length encoding can actually increase the file size.

In addition, run-length coding is very sensitive to transmission errors. If an error occurs in one of the symbols, it will affect the correctness of the entire coding sequence, making it impossible to restore the original data to the run-length coding. Control within one row and one column. All in all, run-length encoding is very suitable for compressing highly repetitive, computer-generated images, especially binary images, and is also suitable for image data such as semantic maps.

All dynamic object semantic submaps and filled static background semantic maps are compressed using run-length encoding, and the encoding compression process is consistent; here we take the filled static background semantic map (abbreviated as background semantic map) as an example to describe its compression process:

Create a variable Header, initialized to the first pixel of the background semantic map; create a variable Count, initialized to 0; create an empty one-dimensional array to store the compressed result data, where the array elements are 24-bit integer values; traverse For each pixel of the background semantic map, perform the following operations for each pixel: determine whether the color value of the current pixel is the same as the Header, and if so, add one to the Count; otherwise, write the current Count value into the compressed result array, and write the Header The RGB three channel values of the color value are combined into an integer and written into the compressed result array, and at the same time, the color value of the current pixel is assigned to the Header, the Count value is reset to 0, and the next pixel is traversed.

In addition to the compressed data of the run-length encoding, the compressed result of the semantic subgraph of the dynamic object also needs to store its two-dimensional bounding volume information on the original semantic graph. The final compressed result data consists of three parts:

(1) The run-length encoded compressed array of the background semantic map;

(2) The run-length encoded compressed array of all dynamic object semantic subgraphs;

(3) 2D bounding volume information of all dynamic object semantic subgraphs.

In this embodiment, the complete compressed result data is transmitted to the algorithm side through the network, and the algorithm side restores the original semantic map through the corresponding decompression method: first, decompress the background semantic map data according to the run-length code compression array of the background semantic map, and then according to Decompress the data of all dynamic object semantic subgraphs from the run-length encoding compressed array of all dynamic object semantic subgraphs, and finally overwrite all dynamic object semantic subgraph data to the background semantic graph data according to the two-dimensional bounding volume information of all dynamic object semantic subgraphs Corresponding to the position, the original semantic map data is obtained;

The specific method of decompressing the original data according to the run-length encoding compressed array is as follows:

(1) Create a variable Count and initialize it to 0; read the first array element and assign it to Count;

(2) Read the subsequent Count array elements, split each array element into three RGB channel values to form a pixel color value, and write it into the original data;

(3) If all array elements have been traversed, complete the decompression step and exit the program; otherwise read the last array element, assign it to Count, and continue to step 2.

After completing the decompression of all dynamic object semantic subgraphs and background semantic graphs, perform the final semantic map stitching step: traverse the two-dimensional bounding volume information of all dynamic object semantic subgraphs, and for the dynamic object semantic subgraph corresponding to the current two-dimensional bounding volume , traverse all the pixels of the dynamic object semantic submap, use the two-dimensional bounding volume to transform the coordinates of the current pixel in the dynamic object semantic submap to the coordinates of the background semantic map, and write the current pixel to the pixel at the corresponding position of the background semantic map.

The final application effect of the semantic map compression method based on dynamic object segmentation is shown in Table 2.

Table 2 is the comparison of compression ratios for image compression using existing three methods and the method of the present invention

RLE TRLE FastPFor Ours Compression ratio 22.4% 14.6% 9.3% 5.9%

Table 2 compares four encoding methods: traditional run-length encoding (RLE, Run-Length Encoding), accelerated run-length encoding (TRLE, Turbo Run-Length Encoding), fast integer sequence compression (FastPFor) and semantic map compression based on dynamic object segmentation. the compression rate. It can be seen that due to the full use of the characteristics of the simulation scene, the separation of dynamic and static objects, and the maximization of the continuity of semantic map data for subsequent compression coding, the compression rate of the semantic map compression method based on dynamic object segmentation is the lowest. Beneficial effects of the invention.

Corresponding to the aforementioned embodiments, the present invention also provides an embodiment of a semantic map compression device based on dynamic object segmentation, as shown in Figure 4, the device includes one or more processors for implementing the above-mentioned laser radar Point cloud generation method.

以软件实现为例，作为一个逻辑意义上的装置，是通过其所在任意具备数据处理能力的设备的处理器将非易失性存储器中对应的计算机程序指令读取到内存中运行形成的。从硬件层面而言，除了处理器、内存、网络接口、以及非易失性存储器之外，实施例中装置所在的任意具备数据处理能力的设备通常根据该任意具备数据处理能力的设备的实际功能，还可以包括其他硬件，对此不再赘述。The embodiment of the device for compressing semantic graph based on dynamic object segmentation of the present invention can be applied to any device with data processing capability, and any device with data processing capability can be a device or device such as a computer. The device embodiments can be implemented by software, or by hardware or a combination of software and hardware.

Taking software implementation as an example, as a device in a logical sense, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for operation by the processor of any device capable of data processing. From the perspective of hardware, in addition to the processor, memory, network interface, and non-volatile memory, any device with data processing capabilities in which the device in the embodiment is usually based on the actual function of any device with data processing capabilities , may also include other hardware, which will not be repeated here.

For the implementation process of the functions and effects of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method for details, and will not be repeated here.

As for the device embodiment, since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. It can be understood and implemented by those skilled in the art without creative effort.

An embodiment of the present invention also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the semantic map compression method based on dynamic object segmentation in the above-mentioned embodiment is implemented.

The computer-readable storage medium may be an internal storage unit of any device capable of data processing described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (SmartMedia card, SMC), an SD card, a flash memory card (Flash card) and the like equipped on the device. Further, the computer-readable storage medium may also include both an internal storage unit of any device capable of data processing and an external storage device. The computer-readable storage medium is used to store the computing program and other programs and data required by any device capable of data processing, and may also be used to temporarily store outputted or to-be-outputted data.

The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be described in the foregoing embodiments Modifications to the technical solutions, or equivalent replacement of some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (8)

1. A semantic graph compression method based on dynamic object segmentation is characterized by comprising the following steps:

s1, initializing a simulation scene, wherein the simulation scene consists of a static background and a dynamic object;

s2, updating and drawing the simulation scene to obtain a semantic graph and two-dimensional bounding volumes of all dynamic objects under a semantic graph coordinate system;

s3, segmenting the semantic data of the dynamic object by using the two-dimensional bounding volume to form a plurality of dynamic object semantic subgraphs; filling the corresponding image area with the adjacent pixels of each dynamic object semantic subgraph in the rest static background semantic graph;

in the step S3, the semantic data of the dynamic object is segmented by using the two-dimensional bounding volume in the step S2 to form a plurality of semantic subgraphs, and the semantic subgraphs are specifically realized by the following substeps:

(1) Initializing corresponding semantic subgraph data according to the size of a two-dimensional bounding volume of each dynamic object, wherein each element of the semantic subgraph is initialized to be (R: 0, G:0, B: 0);

(2) Traversing each pixel of the semantic graph, and for each pixel, performing the following processing:

judging whether the pixel coordinate of the pixel is located in the range of a two-dimensional surrounding body of a certain dynamic object, if so, calculating the relative coordinate of the pixel under the semantic sub-image coordinate system of the dynamic object, writing the relative coordinate into a corresponding element of the semantic sub-image of the dynamic object, otherwise, ignoring the pixel, and continuously traversing the next pixel to obtain the semantic sub-image of the certain dynamic object;

and S4, respectively coding all the dynamic object semantic subgraphs and the filled static background semantic graph by using a coding algorithm.

2. The semantic graph compression method based on dynamic object segmentation according to claim 1, wherein all objects of the simulation scene in S1 are assigned with an ID, and IDs of objects having the same semantic class are the same; each ID uniquely corresponds to one color, and different IDs correspond to different colors.

3. The semantic graph compression method based on dynamic object segmentation according to claim 1, wherein the updating of the simulation scene in S2 includes pose updating of all dynamic objects and rendering view angles; each pixel in the semantic graph uniquely corresponds to one object, and the color corresponding to the object ID is used for coloring.

4. The method according to claim 1, wherein in S2, the two-dimensional bounding volume of the dynamic object in the semantic map coordinate system contains all pixels of the dynamic object on the semantic map.

5. The semantic map compression method based on dynamic object segmentation according to claim 1, wherein in the step S3, the remaining static background semantic map fills the corresponding image region with the adjacent pixels of each dynamic object semantic sub-map, and is implemented by the following sub-steps:

(1) Searching edge pixels around the dynamic object semantic subgraph, and if the edge pixels exceed the range of the semantic graph, determining the edge pixels to be invalid;

(2) And counting all effective edge pixels, and filling the corresponding image area of the dynamic object semantic subgraph in the semantic graph according to the values of the edge pixels.

6. The semantic graph compression method based on dynamic object segmentation according to claim 1, wherein the encoding result of each dynamic object semantic subgraph in S4 is accompanied by corresponding two-dimensional bounding volume information.

7. A semantic graph compression device based on dynamic object segmentation is characterized by comprising one or more processors and being used for realizing the semantic graph compression method based on dynamic object segmentation in any one of claims 1 to 6.

8. A computer-readable storage medium, on which a program is stored, which, when executed by a processor, implements the semantic graph compression method based on dynamic object segmentation according to any one of claims 1 to 6.

Images