[go: up one dir, main page]

TWI799181B - Method of establishing integrate network model to generate complete 3d point clouds from sparse 3d point clouds and segment parts - Google Patents

Method of establishing integrate network model to generate complete 3d point clouds from sparse 3d point clouds and segment parts Download PDF

Info

Publication number
TWI799181B
TWI799181B TW111108862A TW111108862A TWI799181B TW I799181 B TWI799181 B TW I799181B TW 111108862 A TW111108862 A TW 111108862A TW 111108862 A TW111108862 A TW 111108862A TW I799181 B TWI799181 B TW I799181B
Authority
TW
Taiwan
Prior art keywords
layer
point
nodes
hidden
point cloud
Prior art date
Application number
TW111108862A
Other languages
Chinese (zh)
Other versions
TW202336703A (en
Inventor
林春宏
林晏瑜
Original Assignee
國立臺中科技大學
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 國立臺中科技大學 filed Critical 國立臺中科技大學
Priority to TW111108862A priority Critical patent/TWI799181B/en
Application granted granted Critical
Publication of TWI799181B publication Critical patent/TWI799181B/en
Publication of TW202336703A publication Critical patent/TW202336703A/en

Links

Images

Landscapes

  • Image Analysis (AREA)
  • Monitoring And Testing Of Transmission In General (AREA)

Abstract

A method of establishing an integrate network model for 2D images to generate complete 3D point clouds from sparse 3D point clouds and to segment parts is discloses, wherein an input of the integrate network model is a plurality of sparse 3D point clouds, and the method includes the followings steps: make the sparse 3D point clouds pass through an encoding layer for extracting a plurality of features contained therein; calculate attention weights of the extracted features; transcode the extracted features to generate a plurality of transcoded data; decode the transcoded data; and outputting a plurality of complete 3D point clouds and a plurality of results of part segmentation.

Description

由三維稀疏點雲生成三維完整點雲與零件切割之整合模型的建立方法A method for establishing an integrated model of 3D complete point cloud and part cutting from 3D sparse point cloud

本發明係與生成三維點雲的技術有關；特別是指一種由三維稀疏點雲生成三維完整點雲與零件切割之整合模型的建立方法。The present invention is related to the technology of generating three-dimensional point cloud; in particular, it refers to a method for establishing an integrated model for generating three-dimensional complete point cloud and part cutting from three-dimensional sparse point cloud.

由於單獨點雲到點雲的零件切割模型設計與點雲點輸入與輸出的數量有關。例如習用的PointNet模型係以局部特徵與全域特徵進行並列（concatenation），其零件切割效果為最佳，但該模型要求輸入與輸出的點數量必須固定。因此，若欲以稀疏的點雲資料生成更完整的點雲資料，亦即當輸入與輸出的點數量不同時，該模型便無法使用，不具實用性。Because the part cutting model design from individual point cloud to point cloud is related to the number of point cloud point input and output. For example, the commonly used PointNet model is based on the concatenation of local features and global features, and its part cutting effect is the best, but this model requires that the number of input and output points must be fixed. Therefore, if it is desired to generate more complete point cloud data from sparse point cloud data, that is, when the number of input and output points is different, the model cannot be used and is not practical.

有鑑於此，本發明將提出一種由三維稀疏點雲生成三維完整點雲與零件切割之整合模型的建立方法，能夠由三維稀疏點雲生成三維完整點雲，並能用以進行零件切割。In view of this, the present invention will propose a method for establishing an integrated model of 3D complete point cloud and part cutting from 3D sparse point cloud, which can generate 3D complete point cloud from 3D sparse point cloud and can be used for part cutting.

本發明提供一種由三維稀疏點雲生成三維完整點雲與零件切割之整合模型的建立方法，其中該方法的輸入為複數三維稀疏點雲；該方法包括以下步驟：A. 使該些三維稀疏點雲通過一編碼層，萃取其所包含的複數特徵；B. 計算該些特徵的關注權重；C. 對該些特徵進行轉碼，產生複數轉碼後資料；D.使該些轉碼後資料通過一解碼層以進行解碼；以及E. 輸出生成的複數三維完整點雲資料與複數零件切割結果。The present invention provides a method for establishing an integrated model of a three-dimensional complete point cloud and part cutting generated from a three-dimensional sparse point cloud, wherein the input of the method is a complex three-dimensional sparse point cloud; the method includes the following steps: A. making these three-dimensional sparse points The cloud passes through a coding layer to extract the complex features contained in it; B. Calculate the attention weight of these features; C. Transcode these features to generate complex transcoded data; D. Make these transcoded data Decoding through a decoding layer; and E. outputting generated complex 3D complete point cloud data and complex part cutting results.

於一實施例中，步驟A所述的該編碼層具有一第一模組及一第二模組；該第一模組為習用的3D-LMNet模型，該第二模組則係選自改良點雲資料生成模型（G3D）的六個模型中最佳者之編碼層。In one embodiment, the encoding layer described in step A has a first module and a second module; the first module is a conventional 3D-LMNet model, and the second module is selected from improved The encoding layer of the best of the six models of the point cloud data generation model (G3D).

於一實施例中，步驟D所述的該解碼層具有一第一模組及一第二模組；該第一模組包含有兩個獨立的多層感知機（MLP），分別用以生成點雲與零件切割；該第二模組亦包含有兩個獨立的多層感知機，分別用以生成點雲與零件切割。In one embodiment, the decoding layer described in step D has a first module and a second module; the first module includes two independent multi-layer perceptrons (MLP), which are used to generate point Cloud and part cutting; the second module also includes two independent multi-layer perceptrons, which are used to generate point cloud and part cutting respectively.

於一實施例中，該解碼層的該第一模組所包含的該二多層感知機各有四層隱藏層，而且都有相同的隱藏節點數；其中該第一至三層皆有128個隱藏節點；用以生成點雲的該一多層感知機的第四層具有n×3個隱藏節點，用以零件切割的該一多層感知機的第四層具有的隱藏節點數個數等於n乘上零件類別編號之數量。In one embodiment, the two multi-layer perceptrons included in the first module of the decoding layer each have four hidden layers, and all have the same number of hidden nodes; wherein the first to third layers all have 128 hidden nodes; the fourth layer of this multi-layer perceptron used to generate point clouds has n×3 hidden nodes, and the number of hidden nodes that the fourth layer of this multi-layer perceptron used for part cutting has Equal to n multiplied by the number of part category numbers.

於一實施例中，該解碼層的該第二模組所包含用以生成點雲的該一多層感知機具有三層隱藏層，第一層隱藏層有512個節點，第二層隱藏層有256個節點，第三層具有n×3個隱藏節點；該解碼層的該第二模組所包含用以零件切割的該一多層感知機則具有五層隱藏層，第一層隱藏層有512個節點，第二層隱藏層有256個節點，第三與四層隱藏層均有128個節點，第五層具有的隱藏節點數個數等於n乘上零件類別編號之數量。In one embodiment, the multi-layer perceptron included in the second module of the decoding layer for generating point clouds has three hidden layers, the first hidden layer has 512 nodes, and the second hidden layer There are 256 nodes, and the third layer has n×3 hidden nodes; the multi-layer perceptron included in the second module of the decoding layer for part cutting has five hidden layers, and the first hidden layer There are 512 nodes, the second hidden layer has 256 nodes, the third and fourth hidden layers both have 128 nodes, and the fifth layer has a number of hidden nodes equal to n times the number of part category numbers.

於一實施例中，更包含有另一步驟於前述步驟D之後：計算一損失函數In one embodiment, another step is further included after the aforementioned step D: calculating a loss function

於一實施例中，該損失函數包含有一適應性之生成點雲損失函數，其公式為：

Figure 02_image001
其中
Figure 02_image003
，式中
Figure 02_image005
與
Figure 02_image007
表示如下：
Figure 02_image009
與
Figure 02_image011
式中
Figure 02_image013
是真實資料點的標籤
Figure 02_image015
與預測點的零件標籤
Figure 02_image017
之相似值，表示如下：
Figure 02_image019
； 其中
Figure 02_image021
式中
Figure 02_image005
與
Figure 02_image023
表示如下：
Figure 02_image025
與
Figure 02_image027
式中
Figure 02_image029
是預測點的零件標籤
Figure 02_image031
與真實資料點的標籤
Figure 02_image033
之相似值，表示如下：
Figure 02_image035
。 In one embodiment, the loss function includes an adaptive point cloud generation loss function, the formula of which is:
Figure 02_image001
in
Figure 02_image003
, where
Figure 02_image005
and
Figure 02_image007
Expressed as follows:
Figure 02_image009
and
Figure 02_image011
In the formula
Figure 02_image013
is the label of the ground truth point
Figure 02_image015
Part labels with predicted points
Figure 02_image017
The similarity value is expressed as follows:
Figure 02_image019
; in
Figure 02_image021
In the formula
Figure 02_image005
and
Figure 02_image023
Expressed as follows:
Figure 02_image025
and
Figure 02_image027
In the formula
Figure 02_image029
is the part label of the predicted point
Figure 02_image031
Labels with ground truth data points
Figure 02_image033
The similarity value is expressed as follows:
Figure 02_image035
.

於一實施例中，該損失函數包含有一適應性空間關係之交叉熵的零件切割損失函數，其公式如下：

Figure 02_image037
其中
Figure 02_image039
式中
Figure 02_image041
是第
Figure 02_image005
個預測(prediction)點的零件標籤為
Figure 02_image017
之信心分數(confidence)，
Figure 02_image005
與
Figure 02_image043
是第
Figure 02_image045
個真實點與預測點集合中距離最接近的點
Figure 02_image005
間的距離權重值，分別表示如下：
Figure 02_image009
與
Figure 02_image047
Figure 02_image013
是真實(ground truth)資料點的標籤
Figure 02_image015
與預測點的零件標籤
Figure 02_image017
之相似值，表示如下：
Figure 02_image019
； 其中
Figure 02_image049
式中
Figure 02_image041
是第
Figure 02_image005
個預測(prediction)點的零件標籤為
Figure 02_image017
之信心分數(confidence)，
Figure 02_image005
與
Figure 02_image051
是第
Figure 02_image045
個預測點與真實資料集合中距離最近的點
Figure 02_image005
間之距離權重值，分別表示如下：
Figure 02_image025
與
Figure 02_image053
Figure 02_image029
是零件標籤為
Figure 02_image031
與真實(ground truth)資料點的標籤
Figure 02_image033
之相似值，表示如下：
Figure 02_image055
。 In one embodiment, the loss function includes a part-cutting loss function of cross-entropy adaptive spatial relationship, the formula of which is as follows:
Figure 02_image037
in
Figure 02_image039
In the formula
Figure 02_image041
is the first
Figure 02_image005
The part label of a prediction point is
Figure 02_image017
Confidence score (confidence),
Figure 02_image005
and
Figure 02_image043
is the first
Figure 02_image045
The closest point in the set of real points and predicted points
Figure 02_image005
The distance weight values between are expressed as follows:
Figure 02_image009
and
Figure 02_image047
Figure 02_image013
is the label of the ground truth data point
Figure 02_image015
Part labels with predicted points
Figure 02_image017
The similarity value is expressed as follows:
Figure 02_image019
; in
Figure 02_image049
In the formula
Figure 02_image041
is the first
Figure 02_image005
The part label of a prediction point is
Figure 02_image017
Confidence score (confidence),
Figure 02_image005
and
Figure 02_image051
is the first
Figure 02_image045
The nearest point in the predicted point and the real data set
Figure 02_image005
The distance weight values between them are expressed as follows:
Figure 02_image025
and
Figure 02_image053
Figure 02_image029
is the part labeled as
Figure 02_image031
Labels with ground truth data points
Figure 02_image033
The similarity value is expressed as follows:
Figure 02_image055
.

於一實施例中，本方法所建立之該整合模型更具有物件分類之任務，且該解碼層具有一第一模組，該第一模組具有三個獨立的多層感知機，分別用以生成點雲、零件切割，與物件分類；其中各該多層感知機皆具有四層相同隱藏節點數的隱藏層，第一至三層有128個隱藏節點，用以生成點雲的該一多層感知機之第四層具有n×3個隱藏節點，用以零件切割的該一多層感知機之第四層的隱藏節點個數為n乘上零件類別編號之數量，至於用以物件分類的該一多層感知機的第四層的隱藏節點個數等於物件類別編號之個數。In one embodiment, the integrated model established by this method further has the task of object classification, and the decoding layer has a first module, and the first module has three independent multi-layer perceptrons, which are used to generate Point cloud, part cutting, and object classification; each of the multi-layer perceptrons has four hidden layers with the same number of hidden nodes, and the first to third layers have 128 hidden nodes, which are used to generate the multi-layer perceptron of the point cloud The fourth layer of the machine has n×3 hidden nodes, the number of hidden nodes of the fourth layer of the multi-layer perceptron used for part cutting is n multiplied by the number of part category numbers, and the number of hidden nodes used for object classification The number of hidden nodes in the fourth layer of a multi-layer perceptron is equal to the number of object class numbers.

於一實施例中，該解碼層更具有一第二模組，該第二模組具有三個獨立的多層感知機，分別用以生成點雲、零件切割，與物件分類；用以生成點雲與物件分類的該二多層感知機同樣有三層隱藏層，第一層隱藏層有512個節點，第二層隱藏層有256個節點，其中用以生成點雲的該一多層感知機之第三層具有n×3個隱藏節點，而用以物件分類的該一多層感知機的第三層的隱藏節點個數等於物件類別編號之個數；用以零件切割的多層感知機係具有五層隱藏層，第一層隱藏層有512個節點，第二層隱藏層有256個節點，第三與四層隱藏層均有128個節點，第五層的節點個數為n乘上零件類別編號之數量。In one embodiment, the decoding layer further has a second module, the second module has three independent multi-layer perceptrons, which are respectively used to generate point clouds, part cutting, and object classification; to generate point clouds The two-layer perceptron for object classification also has three hidden layers, the first hidden layer has 512 nodes, and the second hidden layer has 256 nodes. Among them, the multi-layer perceptron for generating point clouds The third layer has n × 3 hidden nodes, and the number of hidden nodes in the third layer of the multi-layer perceptron used for object classification is equal to the number of object category numbers; the multi-layer perceptron used for part cutting has Five hidden layers, the first hidden layer has 512 nodes, the second hidden layer has 256 nodes, the third and fourth hidden layers have 128 nodes, and the number of nodes in the fifth layer is n multiplied by parts The number of category numbers.

藉此，本發明所提供之方法所建立的整合模型能夠由三維稀疏點雲生成三維完整點雲，解決前述習用PointNet模型的問題。Thus, the integrated model established by the method provided by the present invention can generate a three-dimensional complete point cloud from a three-dimensional sparse point cloud, and solve the aforementioned problems of the conventional PointNet model.

為能更清楚地說明本發明，茲舉較佳實施例並配合圖式詳細說明如後。請參照圖1及圖2，本發明提供的一種由三維稀疏點雲生成三維完整點雲與零件切割之整合模型的建立方法包含有六個步驟，其中該方法的輸入為複數三維稀疏點雲。如步驟S1所述，本方法首先使該些三維稀疏點雲通過一編碼層，以萃取其所包含的複數特徵。本發明所採用的編碼層具有兩個模組（Encode 1 & Encode 2），分別稱作第一及第二模組，其中編碼層的該第一模組係採用習用的3D-LMNet模型，此一模型是一種生成點雲的模型，其編碼層的隱藏層之節點數不多，因此本發明使用其編碼層做為本發明編碼層的第一個模組；另外，本發明編碼層的該第二模組係從改良點雲資料生成模型（G3D）的六個模型中，取其中最佳的一個模型之編碼層。接著，於步驟S2，本發明提出之方法會去計算萃取得到的該些特徵的關注權重；步驟S3則對該些特徵進行轉碼，藉此產生複數轉碼後資料。隨後，在步驟S4使該些轉碼後資料通過一解碼層以進行解碼；於步驟S5計算損失函數；最後，在步驟S6輸出生成的複數三維完整點雲資料與複數零件切割結果。In order to illustrate the present invention more clearly, preferred embodiments are given and detailed descriptions are given below in conjunction with drawings. Please refer to FIG. 1 and FIG. 2 , a method for establishing an integrated model of 3D complete point cloud and part cutting provided by the present invention includes six steps, wherein the input of the method is complex 3D sparse point cloud. As described in step S1, the method first passes the 3D sparse point clouds through an encoding layer to extract complex features contained therein. The encoding layer used in the present invention has two modules (Encode 1 & Encode 2), called the first and second modules respectively, wherein the first module of the encoding layer adopts the conventional 3D-LMNet model, here A model is a model for generating point clouds, and the number of nodes in the hidden layer of the encoding layer is small, so the present invention uses its encoding layer as the first module of the encoding layer of the present invention; in addition, the encoding layer of the present invention The second module is to select the coding layer of the best model from the six models of the improved point cloud data generation model (G3D). Next, in step S2, the method proposed by the present invention calculates the attention weights of the extracted features; and in step S3, transcodes these features to generate complex transcoded data. Subsequently, in step S4, the transcoded data are passed through a decoding layer for decoding; in step S5, a loss function is calculated; finally, in step S6, the generated complex 3D complete point cloud data and complex part cutting results are output.

本發明相對該編碼層，亦具有一解碼層，即使用於步驟S4中的該解碼層，此處對該解碼層加以說明。該解碼層同樣則有兩個模組（Decode 1 & Decode 2），分別稱作第一及第二模組，其中該解碼層的該第一模組包含有兩個獨立的多層感知機（MLP），分別用以生成點雲與零件切割；該兩個MLP各有四層隱藏層，而且都有相同的隱藏節點數，第一至三層有128個隱藏節點，用以生成點雲的該一MLP的第四層具有n×3個隱藏節點，用以零件切割的該一MLP的第四層所具有的隱藏節點數量為n×label（零件類別編號）。該解碼層的該第二模組也包含有兩個獨立的多層感知機（MLP），分別用以生成點雲與零件切割，其中用以生成點雲的MLP有三層隱藏層，第一層隱藏層有512個節點，第二層隱藏層有256個節點，第三層具有n×3個隱藏節點；用以零件切割的MLP有五層隱藏層，第一層隱藏層有512個節點，第二層隱藏層有256個節點，第三與四層隱藏層均有128個節點，第五層為n×label（零件類別編號）個節點。前述的該編碼層及該解碼層的整體模組如圖3所示。The present invention also has a decoding layer relative to the encoding layer, that is, the decoding layer used in step S4, and the decoding layer is described here. The decoding layer also has two modules (Decode 1 & Decode 2), called the first and second modules respectively, wherein the first module of the decoding layer includes two independent multi-layer perceptrons (MLP ), which are used to generate point cloud and part cutting respectively; the two MLPs each have four hidden layers, and both have the same number of hidden nodes. The first to third layers have 128 hidden nodes, which are used to generate the point cloud. The fourth layer of an MLP has n×3 hidden nodes, and the number of hidden nodes in the fourth layer of the MLP used for part cutting is n×label (part category number). The second module of the decoding layer also includes two independent multi-layer perceptrons (MLP), which are used to generate point clouds and part cutting respectively. The MLP used to generate point clouds has three hidden layers, and the first layer hides layer has 512 nodes, the second hidden layer has 256 nodes, the third layer has n×3 hidden nodes; the MLP used for part cutting has five hidden layers, the first hidden layer has 512 nodes, the The second hidden layer has 256 nodes, the third and fourth hidden layers both have 128 nodes, and the fifth layer has n×label (part category number) nodes. The aforementioned overall modules of the encoding layer and the decoding layer are shown in FIG. 3 .

此處對後續步驟S5應用到的損失函數進行說明，其中，關於生成點雲的損失函數係計算預測點與真實點最接近的點之Chamfer距離，其值的範圍與3D模型的尺寸大小有關，而關於零件切割的損失函數係計算交叉熵(cross-entropy)，其值的範圍為0至1之間。本發明提出之方法所建立的整合模型在計算損失函數時，係將前述兩個損失函數採加總方式處理。Here, the loss function applied in the subsequent step S5 is described, wherein the loss function for generating the point cloud is to calculate the Chamfer distance between the predicted point and the closest point to the real point, and the range of its value is related to the size of the 3D model. The loss function for part cutting is to calculate cross-entropy, and its value ranges from 0 to 1. When the integrated model established by the method proposed by the present invention calculates the loss function, the aforementioned two loss functions are summed up.

本發明所提出關於生成點雲的損失函數係一種基於Chamfer距離的適應性生成點雲損失函數(the adaptive loss function based on chamfer distance of generating a point cloud)損失值。此損失值係將每個點所計算的Chamfer距離值取自然指數函數(natural exponential function)，並將此距離值歸納至0至1之間。本研究將對生成點雲之預測(predicted)點雲的點集合

Figure 02_image057
，以及真實(ground truth)點雲的點集合
Figure 02_image059
，分別提出從真實點至預測點與從預測點至真實點的損失函數。 The loss function proposed by the present invention for generating a point cloud is a Chamfer distance-based adaptive loss function (the adaptive loss function based on chamfer distance of generating a point cloud) loss value. This loss value is to take the Chamfer distance value calculated by each point as a natural exponential function (natural exponential function), and summarize the distance value between 0 and 1. This research will generate point cloud prediction (predicted) point cloud point collection
Figure 02_image057
, and the point set of the real (ground truth) point cloud
Figure 02_image059
, respectively propose the loss functions from the real point to the predicted point and from the predicted point to the real point.

本發明以真實資料點為基準，逐一尋找座標距離最接近的預測點，係從真實點第

Figure 02_image045
個點且
Figure 02_image061
，逐一尋找預測點集合
Figure 02_image057
中與第
Figure 02_image045
個真實點距離最接近的點，表示成
Figure 02_image005
且
Figure 02_image063
。然後再將所有點計算Chamfer距離的交叉熵值，最後進行加總，本發明稱之為
Figure 02_image065
損失函數，其公式如下： The present invention takes the real data points as the benchmark, and searches for the prediction points with the closest coordinate distance one by one, and starts from the first point of the real point.
Figure 02_image045
points and
Figure 02_image061
, looking for the set of prediction points one by one
Figure 02_image057
middle and first
Figure 02_image045
The closest point to the real point, expressed as
Figure 02_image005
and
Figure 02_image063
. Then calculate the cross entropy value of Chamfer distance for all points, and finally add up, the present invention is called
Figure 02_image065
The loss function, whose formula is as follows:

Figure 02_image003
式中
Figure 02_image005
與
Figure 02_image007
表示如下：
Figure 02_image009
與
Figure 02_image011
式中
Figure 02_image013
是真實資料點的標籤
Figure 02_image015
與預測點的零件標籤
Figure 02_image017
之相似值，表示如下：
Figure 02_image019
Figure 02_image003
In the formula
Figure 02_image005
and
Figure 02_image007
Expressed as follows:
Figure 02_image009
and
Figure 02_image011
In the formula
Figure 02_image013
is the label of the ground truth point
Figure 02_image015
Part labels with predicted points
Figure 02_image017
The similarity value is expressed as follows:
Figure 02_image019

接著，本發明亦以預測點為基準，逐一尋找座標距離最接近的真實資料點，係從預測點第

Figure 02_image045
個點且
Figure 02_image067
，逐一尋找真實點集合
Figure 02_image059
中與第
Figure 02_image045
個預測點距離最接近的點，表示成
Figure 02_image005
且
Figure 02_image069
。然後再將所有點計算Chamfer距離的交叉熵值，最後進行加總，本發明稱之為
Figure 02_image071
損失函數，其公式如下： Next, the present invention also uses the predicted point as a benchmark to find the real data point with the closest coordinate distance one by one, starting from the predicted point
Figure 02_image045
points and
Figure 02_image067
, find the set of real points one by one
Figure 02_image059
middle and first
Figure 02_image045
The point closest to the prediction point, expressed as
Figure 02_image005
and
Figure 02_image069
. Then calculate the cross entropy value of Chamfer distance for all points, and finally add up, the present invention is called
Figure 02_image071
The loss function, whose formula is as follows:

Figure 02_image021
式中
Figure 02_image005
與
Figure 02_image023
表示如下：
Figure 02_image025
與
Figure 02_image027
式中
Figure 02_image029
是預測點的零件標籤
Figure 02_image031
與真實資料點的標籤
Figure 02_image033
之相似值，表示如下：
Figure 02_image035
Figure 02_image021
In the formula
Figure 02_image005
and
Figure 02_image023
Expressed as follows:
Figure 02_image025
and
Figure 02_image027
In the formula
Figure 02_image029
is the part label of the predicted point
Figure 02_image031
Labels with ground truth data points
Figure 02_image033
The similarity value is expressed as follows:
Figure 02_image035

結合前述兩種損失函數，本發明提出基於Chamfer距離的適應性之生成點雲損失函數AG3DL_CF表示如下：Combining the aforementioned two loss functions, the present invention proposes the adaptive point cloud generation loss function AG3DL_CF based on the Chamfer distance as follows:

Figure 02_image001
Figure 02_image001

另外，本發明提出適應性空間關係之交叉熵的零件切割損失函數(the adaptive loss function of the parts segmentation based on cross-entropy of spatial relationship)。此一損失函數係將每個點計算Chamfer距離值，然後再乘上距離的權重值，也就是說距離值越大則損失值越大，距離值越小則損失值越小。本發明將對生成點雲之預測(predicted)點雲的點集合

Figure 02_image057
，以及真實(ground truth)點雲的點集合
Figure 02_image059
，分別提出從真實點至預測點與從預測點至真實點的損失函數。 In addition, the present invention proposes the adaptive loss function of the parts segmentation based on cross-entropy of spatial relationship. This loss function calculates the Chamfer distance value for each point, and then multiplies the weight value of the distance, that is to say, the larger the distance value, the larger the loss value, and the smaller the distance value, the smaller the loss value. The present invention will generate the point set of the predicted (predicted) point cloud of the point cloud
Figure 02_image057
, and the point set of the real (ground truth) point cloud
Figure 02_image059
, respectively propose the loss functions from the real point to the predicted point and from the predicted point to the real point.

其中，從真實點至預測點的損失函數公式如下：Among them, the loss function formula from the real point to the predicted point is as follows:

Figure 02_image039
式中
Figure 02_image041
是第
Figure 02_image005
個預測(prediction)點的零件標籤為
Figure 02_image017
之信心分數(confidence)，
Figure 02_image005
與
Figure 02_image043
是第
Figure 02_image045
個真實點與預測點集合中距離最接近的點
Figure 02_image005
間的距離權重值，分別表示如下：
Figure 02_image009
與
Figure 02_image047
Figure 02_image013
是真實(ground truth)資料點的標籤
Figure 02_image015
與預測點的零件標籤
Figure 02_image017
之相似值，表示如下：
Figure 02_image019
Figure 02_image039
In the formula
Figure 02_image041
is the first
Figure 02_image005
The part label of a prediction point is
Figure 02_image017
Confidence score (confidence),
Figure 02_image005
and
Figure 02_image043
is the first
Figure 02_image045
The closest point in the set of real points and predicted points
Figure 02_image005
The distance weight values between are expressed as follows:
Figure 02_image009
and
Figure 02_image047
Figure 02_image013
is the label of the ground truth data point
Figure 02_image015
Part labels with predicted points
Figure 02_image017
The similarity value is expressed as follows:
Figure 02_image019

至於從預測點至真實點的損失函數則是公式如下：As for the loss function from the predicted point to the real point, the formula is as follows:

Figure 02_image073
式中
Figure 02_image041
是第
Figure 02_image005
個預測(prediction)點的零件標籤為
Figure 02_image017
之信心分數(confidence)，
Figure 02_image005
與
Figure 02_image051
是第
Figure 02_image045
個預測點與真實資料集合中距離最近的點
Figure 02_image005
間之距離權重值，分別表示如下：
Figure 02_image025
與
Figure 02_image053
Figure 02_image029
是零件標籤為
Figure 02_image031
與真實(ground truth)資料點的標籤
Figure 02_image033
之相似值，表示如下：
Figure 02_image055
Figure 02_image073
In the formula
Figure 02_image041
is the first
Figure 02_image005
The part label of a prediction point is
Figure 02_image017
Confidence score (confidence),
Figure 02_image005
and
Figure 02_image051
is the first
Figure 02_image045
The nearest point in the predicted point and the real data set
Figure 02_image005
The distance weight values between them are expressed as follows:
Figure 02_image025
and
Figure 02_image053
Figure 02_image029
is the part labeled as
Figure 02_image031
Labels with ground truth data points
Figure 02_image033
The similarity value is expressed as follows:
Figure 02_image055

結合前述兩種損失函數，本發明提出適應性空間關係之交叉熵的零件切割損失函數APSL_CESR表示如下：Combining the above two loss functions, the present invention proposes the part cutting loss function APSL_CESR of the cross-entropy of the adaptive spatial relationship as follows:

Figure 02_image075
Figure 02_image075

由於本發明提供之方法所建立的整合模型，其損失函數係由生成點雲與零件切割的損失函數組成，故其整體的損失函數可表示為：Since the integrated model established by the method provided by the present invention has a loss function composed of point cloud generation and part cutting loss functions, the overall loss function can be expressed as:

Figure 02_image077
+
Figure 02_image079
式中
Figure 02_image081
與
Figure 02_image083
分別是生成點雲與零件切割的損失函數，
Figure 02_image085
與
Figure 02_image087
分別是生成點雲與零件切割損失函數的權重。
Figure 02_image077
+
Figure 02_image079
In the formula
Figure 02_image081
and
Figure 02_image083
are the loss functions for point cloud generation and part cutting, respectively,
Figure 02_image085
and
Figure 02_image087
are the weights of the generated point cloud and part cutting loss functions, respectively.

本發明提出之方法所建立的整合模型，除了生成點雲與零件切割之外，還可以更具有物件分類的任務。即使加上物件分類的任務，該編碼層的架構仍同前述，這是因為三個任務的編碼層具有相同的特徵。但是三個任務的輸出型態不同，而且代表的物理意義也不同。因此，解碼層的部分將具有三個獨立的多層感知機(MLP)，並提出兩個不同的模組。The integrated model established by the method proposed in the present invention can not only generate point clouds and cut parts, but also have the task of classifying objects. Even adding the task of object classification, the structure of the coding layer is still the same as above, because the coding layers of the three tasks have the same characteristics. However, the output types of the three tasks are different, and the physical meanings represented are also different. Therefore, the part of the decoding layer will have three independent multi-layer perceptrons (MLP), and propose two different modules.

其中該解碼層的第一模組（Decode 1）將生成點雲、零件切割與物件分類分成三個獨立的多層感知機(MLP)，三個MLP各有四層相同隱藏節點數的隱藏層，第一至三層有128個隱藏節點，用以生成點雲的多層感知機之第四層具有n×3個隱藏節點，用以零件切割的多層感知機之第四層具有n×label(零件類別編號)個隱藏節點，至於用以物件分類的多層感知機的第四層則是具有class (物件類別編號)個隱藏節點。Among them, the first module (Decode 1) of the decoding layer divides point cloud generation, part cutting and object classification into three independent multi-layer perceptrons (MLP), and each of the three MLPs has four hidden layers with the same number of hidden nodes. The first to third layers have 128 hidden nodes, the fourth layer of the multilayer perceptron used to generate point clouds has n×3 hidden nodes, and the fourth layer of the multilayer perceptron used for part cutting has n×label (part class number) hidden nodes, and the fourth layer of the multi-layer perceptron for object classification has class (object class number) hidden nodes.

其中該解碼層的第二模組（Decode 2）中，用以生成點雲與物件分類的多層感知機同樣有三層隱藏層，第一層隱藏層有512個節點，第二層隱藏層有256個節點，用以生成點雲的多層感知機之第三層具有n×3個隱藏節點，而用以物件分類的多層感知機的第三層則是具有class (物件類別編號)個隱藏節點。至於用以零件切割的多層感知機係具有五層隱藏層，第一層隱藏層有512個節點，第二層隱藏層有256個節點，第三與四層隱藏層均有128個節點，第五層的節點個數為n×label。此一具有物件分類任務的整合模型之解碼層架構如圖4所示。Among them, in the second module (Decode 2) of the decoding layer, the multi-layer perceptron used to generate point clouds and object classification also has three hidden layers. The first hidden layer has 512 nodes, and the second hidden layer has 256 nodes. nodes, the third layer of the multilayer perceptron for point cloud generation has n×3 hidden nodes, and the third layer of the multilayer perceptron for object classification has class (object class number) hidden nodes. As for the multi-layer perceptron system used for part cutting, it has five hidden layers, the first hidden layer has 512 nodes, the second hidden layer has 256 nodes, the third and fourth hidden layers both have 128 nodes, and the second hidden layer has 128 nodes. The number of nodes in the fifth layer is n×label. The decoding layer architecture of this integrated model with object classification task is shown in FIG. 4 .

至於此一具有物件分類任務的整合模型所採用的損失函數，其公式則如下所示：As for the loss function used in this integrated model with object classification task, its formula is as follows:

Figure 02_image089
+
Figure 02_image079
+
Figure 02_image091
式中
Figure 02_image081
、
Figure 02_image083
與
Figure 02_image093
分別是生成點雲、零件切割以及物件分類的損失函數，
Figure 02_image085
、
Figure 02_image087
與
Figure 02_image095
分別是生成點雲、零件切割以及物件分類損失函數的權重。
Figure 02_image089
+
Figure 02_image079
+
Figure 02_image091
In the formula
Figure 02_image081
,
Figure 02_image083
and
Figure 02_image093
are the loss functions for point cloud generation, part cutting, and object classification, respectively,
Figure 02_image085
,
Figure 02_image087
and
Figure 02_image095
are the weights of generating point cloud, part cutting and object classification loss functions respectively.

以上所述僅為本發明較佳可行實施例而已，舉凡應用本發明說明書及申請專利範圍所為之等效方法變化，理應包含在本發明之專利範圍內。The above description is only a preferred embodiment of the present invention, and all equivalent method changes made by applying the description of the present invention and the scope of the patent application should be included in the scope of the patent of the present invention.

S1、S2、S3、S4、S5、S6:步驟S1, S2, S3, S4, S5, S6: steps

圖1是本發明由三維稀疏點雲生成三維完整點雲與零件切割之整合模型的建立方法之流程圖； 圖2是本發明前述方法建立之整合模型的架構圖； 圖3是本發明的編碼層及解碼層的整體模組圖；以及 圖4是本發明具有物件分類任務的整合模型之解碼層架構圖。 Fig. 1 is the flow chart of the establishment method of the integrated model of three-dimensional complete point cloud and parts cutting generated by the present invention from three-dimensional sparse point cloud; Fig. 2 is the structural diagram of the integrated model that aforementioned method of the present invention establishes; Fig. 3 is an overall module diagram of the coding layer and the decoding layer of the present invention; and FIG. 4 is a structure diagram of a decoding layer of an integrated model with an object classification task according to the present invention.

S1、S2、S3、S4、S5、S6:步驟 S1, S2, S3, S4, S5, S6: steps

Claims (4)

一種由三維稀疏點雲生成三維完整點雲的模型建立方法，其中該方法的輸入為複數三維稀疏點雲，該方法包括：A.使該些三維稀疏點雲通過一編碼層，萃取其所包含的複數特徵；B.取得該些特徵經計算得到的關注權重；C.取得該些特徵轉碼後產生的複數轉碼後資料；D.該些轉碼後資料通過一解碼層接受解碼；E.施用一損失函數於解碼後的該些轉碼後資料；以及F.輸出生成的複數三維完整點雲資料；其中該損失函數包含有一適應性之生成點雲損失函數，其公式為：L AG3DL_CF =L AG3DL_CF1+L AG3DL_CF2其中
Figure 111108862-A0305-02-0015-3
，式中i*與d i*表示如下：
Figure 111108862-A0305-02-0015-1
與
Figure 111108862-A0305-02-0015-2
；式中y i 是真實資料點的標籤p(i)與預測點的零件標籤
Figure 111108862-A0305-02-0015-4
之相似值，表示如下：
Figure 111108862-A0305-02-0015-5
 其中
Figure 111108862-A0305-02-0015-6
；式中i*與
Figure 111108862-A0305-02-0015-7
表示如下：
Figure 111108862-A0305-02-0015-8
與
Figure 111108862-A0305-02-0015-9
； 式中
Figure 111108862-A0305-02-0016-10
是預測點的零件標籤
Figure 111108862-A0305-02-0016-11
與真實資料點的標籤p(i*)之相似值，表示如下：
Figure 111108862-A0305-02-0016-12
 A method for building a model of a three-dimensional complete point cloud generated from a three-dimensional sparse point cloud, wherein the input of the method is a complex three-dimensional sparse point cloud, and the method includes: A. making these three-dimensional sparse point clouds pass through a coding layer to extract the contained B. Obtain the attention weights calculated by these features; C. Obtain the complex transcoded data generated after these features are transcoded; D. The transcoded data are decoded through a decoding layer; E .Apply a loss function to the decoded transcoded data; and F. Output the generated complex three-dimensional complete point cloud data; wherein the loss function includes an adaptive loss function for generating point cloud, the formula of which is: L AG 3 DL_CF = L AG 3 DL_CF 1 + L AG 3 DL_CF 2 where
Figure 111108862-A0305-02-0015-3
, where i * and d i * are expressed as follows:
Figure 111108862-A0305-02-0015-1
and
Figure 111108862-A0305-02-0015-2
; where y i is the label p ( i ) of the real data point and the part label of the predicted point
Figure 111108862-A0305-02-0015-4
The similarity value is expressed as follows:
Figure 111108862-A0305-02-0015-5
 in
Figure 111108862-A0305-02-0015-6
; where i * and
Figure 111108862-A0305-02-0015-7
Expressed as follows:
Figure 111108862-A0305-02-0015-8
and
Figure 111108862-A0305-02-0015-9
; where
Figure 111108862-A0305-02-0016-10
is the part label of the predicted point
Figure 111108862-A0305-02-0016-11
The similar value to the label p ( i *) of the real data point is expressed as follows:
Figure 111108862-A0305-02-0016-12
 如請求項1所述之方法，其中步驟D所述的該解碼層具有一第一模組及一第二模組；該第一模組包含有一多層感知機(MLP)，用以生成點雲；該第二模組亦包含有另一多層感知機，用以生成點雲。 The method as described in claim 1, wherein the decoding layer described in step D has a first module and a second module; the first module includes a multi-layer perceptron (MLP) for generating point clouds ; The second module also includes another multi-layer perceptron for generating point clouds. 如請求項2所述之方法，其中該解碼層的該第一模組所包含的該多層感知機有四層隱藏層；其中該第一至三層有128個隱藏節點；該第四層具有n×3個隱藏節點，其中n為輸入該第四層的節點個數。 The method as described in claim 2, wherein the multi-layer perceptron included in the first module of the decoding layer has four hidden layers; wherein the first to third layers have 128 hidden nodes; the fourth layer has n×3 hidden nodes, where n is the number of nodes input to the fourth layer. 如請求項2所述之方法，其中該解碼層的該第二模組所包含用以生成點雲的該多層感知機具有三層隱藏層，第一層隱藏層有512個節點，第二層隱藏層有256個節點，第三層具有n×3個隱藏節點，其中n為輸入第三層的節點個數。The method as described in claim 2, wherein the multi-layer perceptron included in the second module of the decoding layer for generating point clouds has three hidden layers, the first hidden layer has 512 nodes, and the second layer The hidden layer has 256 nodes, and the third layer has n×3 hidden nodes, where n is the number of nodes input to the third layer.
TW111108862A 2022-03-10 2022-03-10 Method of establishing integrate network model to generate complete 3d point clouds from sparse 3d point clouds and segment parts TWI799181B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
TW111108862A TWI799181B (en) 2022-03-10 2022-03-10 Method of establishing integrate network model to generate complete 3d point clouds from sparse 3d point clouds and segment parts

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
TW111108862A TWI799181B (en) 2022-03-10 2022-03-10 Method of establishing integrate network model to generate complete 3d point clouds from sparse 3d point clouds and segment parts

Publications (2)

Publication Number Publication Date
TWI799181B true TWI799181B (en) 2023-04-11
TW202336703A TW202336703A (en) 2023-09-16

Family

ID=86948631

Family Applications (1)

Application Number Title Priority Date Filing Date
TW111108862A TWI799181B (en) 2022-03-10 2022-03-10 Method of establishing integrate network model to generate complete 3d point clouds from sparse 3d point clouds and segment parts

Country Status (1)

Country Link
TW (1) TWI799181B (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190147245A1 (en) * 2017-11-14 2019-05-16 Nuro, Inc. Three-dimensional object detection for autonomous robotic systems using image proposals
CN112967219A (en) * 2021-03-17 2021-06-15 复旦大学附属华山医院 Two-stage dental point cloud completion method and system based on deep learning network
TW202129593A (en) * 2020-01-16 2021-08-01 國立臺中科技大學 Convolutional neural network detection method and system for animals
TW202134999A (en) * 2020-02-12 2021-09-16 國立臺中科技大學 Sea level image detection system
CN113901938A (en) * 2021-10-21 2022-01-07 深圳市中钞科信金融科技有限公司 Identity recognition method and system based on tooth 3D point cloud
CN113989547A (en) * 2021-10-15 2022-01-28 天津大学 Classification structure and method of 3D point cloud data based on graph convolutional deep neural network
CN114048845A (en) * 2022-01-14 2022-02-15 深圳大学 Point cloud repairing method, device, computer equipment and storage medium
CN114067075A (en) * 2021-09-27 2022-02-18 中国地质大学（武汉） Point cloud completion method and device based on generation of countermeasure network
CN114127785A (en) * 2021-04-15 2022-03-01 商汤国际私人有限公司 Point cloud completion method, network training method, device, equipment and storage medium

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190147245A1 (en) * 2017-11-14 2019-05-16 Nuro, Inc. Three-dimensional object detection for autonomous robotic systems using image proposals
TW202129593A (en) * 2020-01-16 2021-08-01 國立臺中科技大學 Convolutional neural network detection method and system for animals
TW202134999A (en) * 2020-02-12 2021-09-16 國立臺中科技大學 Sea level image detection system
CN112967219A (en) * 2021-03-17 2021-06-15 复旦大学附属华山医院 Two-stage dental point cloud completion method and system based on deep learning network
CN114127785A (en) * 2021-04-15 2022-03-01 商汤国际私人有限公司 Point cloud completion method, network training method, device, equipment and storage medium
CN114067075A (en) * 2021-09-27 2022-02-18 中国地质大学（武汉） Point cloud completion method and device based on generation of countermeasure network
CN113989547A (en) * 2021-10-15 2022-01-28 天津大学 Classification structure and method of 3D point cloud data based on graph convolutional deep neural network
CN113901938A (en) * 2021-10-21 2022-01-07 深圳市中钞科信金融科技有限公司 Identity recognition method and system based on tooth 3D point cloud
CN114048845A (en) * 2022-01-14 2022-02-15 深圳大学 Point cloud repairing method, device, computer equipment and storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
期刊 余佳靜、林春宏及黃馨逸 融合二維與三維卷積類神經網路技術之汽車零件的自動偵測與分類 國立臺中科技大學資訊工程系碩士論文 臺灣碩博士論文加值系統(https://ndltd.ncl.edu.tw/) 2020年 https://hdl.handle.net/11296/kzhk36 *

Also Published As

Publication number Publication date
TW202336703A (en) 2023-09-16

Similar Documents

Publication Publication Date Title
CN114758129B (en) A semantic segmentation method for outdoor scenes based on RandLA-Net based on local feature enhancement
CN109377530B (en) A Binocular Depth Estimation Method Based on Deep Neural Network
CN112819833B (en) A large scene point cloud semantic segmentation method
CN110245655B (en) A Single-Stage Object Detection Method Based on Lightweight Image Pyramid Network
CN115294282A (en) Monocular depth estimation system and method for enhancing feature fusion in three-dimensional scene reconstruction
CN115457498A (en) Urban road semantic segmentation method based on double attention and dense connection
CN114332592B (en) A marine environmental data fusion method and system based on attention mechanism
CN115223082A (en) A spatiotemporal multi-scale Transformer-based aerial video classification method
CN111144329A (en) A lightweight and fast crowd counting method based on multi-label
CN115713546B (en) Lightweight target tracking method for mobile terminal devices
CN114092773B (en) Signal processing method, signal processing device, electronic apparatus, and storage medium
CN116756314B (en) GCN-based aspect-level multi-mode emotion analysis method
CN114821438A (en) Video human behavior identification method and system based on multipath excitation
CN116704506A (en) A Cross-Context Attention-Based Approach to Referential Image Segmentation
CN110196928B (en) Fully parallelized end-to-end multi-turn dialogue system and method with domain scalability
CN116704187B (en) Real-time semantic segmentation method, system and storage medium for semantic alignment
CN113747168A (en) Training method of multimedia data description model and generation method of description information
WO2023010981A1 (en) Encoding and decoding methods and apparatus
CN115713632A (en) Feature extraction method and device based on multi-scale attention mechanism
Yi et al. An effective lightweight crowd counting method based on an encoder–decoder network for internet of video things
CN118799665A (en) A three-dimensional object detection method and system based on cross-modal decoupling knowledge transfer
CN116485867A (en) A Depth Estimation Method for Structured Scenes for Autonomous Driving
CN116912579A (en) Scene graph generation method based on multi-level attention mechanism
TWI799181B (en) Method of establishing integrate network model to generate complete 3d point clouds from sparse 3d point clouds and segment parts
CN117522882A (en) Point cloud segmentation method based on geometric feature aggregation and self-adaptive fusion