CN114240957A - Method, system and storage medium for planning rib cutting path - Google Patents

Abstract

The invention relates to a method for planning a rib cutting path, which comprises the following steps: acquiring three-dimensional point cloud data and visible light picture data of the rib rows; generating RGBD point cloud data according to the three-dimensional point cloud data and the visible light picture data; determining the range of rib row gaps according to the RGBD point cloud data; extracting the set of rib row slit points from within the range of the rib row slit and fitting the set of rib row slit points to a rib row cutting path. According to the technical scheme, the positions of rib row gaps between ribs are accurately positioned according to RGBD point cloud data of the rib rows by effectively utilizing the physiological characteristics of the rib rows, and rib row cutting paths are planned. Therefore, the machine can replace manual operation with low efficiency, and industrial intelligent upgrading is promoted. Meanwhile, the full-automatic rib cutting path planning process is beneficial to saving labor cost and further improving the economic benefit of the slaughterhouse.

Description

Method, system and storage medium for planning rib cutting path

Technical Field

The present invention relates generally to the field of new energy power generation. More particularly, the present invention relates to a method, system and readable storage medium for planning a rib cutting path.

Background

Currently, in the slaughtering industry, slaughtering and cutting of animals is a complete process. The finer the animal is divided by parts, the higher the price can be sold and the higher the profit can be obtained. Consequently, slaughterhouses often need to separate the rib rows of larger body animals into individual ribs. At present, most slaughterhouses divide rib rows by using a manual cutting method, even in some automatic slaughterhouses, the rib rows can only be roughly divided, and the rib rows cannot be automatically divided into single ribs by a machine. Meanwhile, the efficient and accurate rib row segmentation cannot be finished at present due to the proficiency of butchers and the number of butchers. The current situation of the industry still depends on manual rib row segmentation, standard operation cannot be carried out, and then the working efficiency is low, and the requirement of industrial capacity upgrading cannot be met. Therefore, how to realize rapid and standardized rib row division becomes a bottleneck problem faced by the technical personnel in the field.

Disclosure of Invention

In order to at least solve the problems, the invention provides a method for planning a rib row cutting path, which accurately and effectively determines the position, the coordinate and the direction of a rib row gap according to the image point cloud characteristics of the rib row, and feeds the position, the coordinate and the direction back to a manipulator end to execute segmentation, thereby being beneficial to the manipulator to finish automatic intelligent accurate segmentation of the rib row, being convenient for realizing rapid and standardized segmentation of the rib row and further improving the slaughtering efficiency.

In a first aspect, the present invention provides a method of planning a rib row cutting path, comprising the steps of: acquiring three-dimensional point cloud data and visible light picture data of the rib rows; generating RGBD point cloud data according to the three-dimensional point cloud data and the visible light picture data; determining the range of rib row gaps according to the RGBD point cloud data; extracting rib row gap point sets from the range of the rib row gaps; and fitting the set of rib row slit points to a rib row cutting path.

In one embodiment, further comprising: converting the rib cutting path to a machine recognizable set of coordinates.

In one embodiment, the determining the range of the rib row slit from the point cloud data of RGBD comprises: and determining the range of the rib row gaps by utilizing a semantic segmentation model according to the point cloud data of the RGBD.

In one embodiment, extracting the rib row slit point from within the range of the rib row slit comprises: and performing normal vector analysis on the rib row according to the point cloud data of the RGBD to extract the rib row gap point set from the range of the rib row gap.

In one embodiment, the performing normal vector analysis on the rib according to the point cloud data of RGBD to extract the set of rib slit points from the range of the rib slit comprises: determining a nearest neighbor point set within a preset radius range of each point on the rib row according to the RGBD point cloud data of the rib row; determining a normal vector of each point in the nearest neighbor point set; and taking the points with the normal vector included angle smaller than 90 degrees as a rib row gap point set.

In one embodiment, said fitting the set of rib row slit points to the rib row cutting path comprises: and dividing the rib row gap point set into a single group of rib row gap point sets, and respectively fitting the single group of rib row gap point sets into a single rib row cutting path.

In one embodiment, fitting the set of rib row slit points to a rib row cutting path comprises: fitting the set of rib row slit points to a rib row cutting path by fitting a spatial circle.

In one embodiment, said fitting the rib row slit points to the rib row cutting path further comprises: and determining the positions of the starting point and the ending point of the rib row cutting path, and determining the coordinates of the rib row cutting path.

In a second aspect, the present invention provides a system for planning a rib cutting path, the system comprising: one or more processors; storage means for storing one or more programs; when executed by the one or more processors, cause the one or more processors to implement a method in accordance with any of the above-described invention and embodiments thereof.

In a third aspect, the invention also provides a computer-readable storage medium comprising computer program instructions which, when executed by one or more processors, cause the implementation of a method according to any one of the above-described invention and embodiments thereof.

Different from the prior art that the rib rows are manually segmented, the method and the device accurately position the rib row gaps between the ribs according to RGBD point cloud data of the rib rows by effectively utilizing the physiological characteristics of the rib rows, and complete the planning of the rib row cutting path. Therefore, the automatic rib row cutting machine can assist a manipulator to realize automatic rib row cutting, so that manual operation with low efficiency is replaced, and industrial intelligent upgrading is promoted. Meanwhile, the full-automatic rib cutting path planning process is beneficial to saving labor cost and further improving the economic benefit of the slaughterhouse.

Drawings

The above and other objects, features and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar or corresponding parts and in which:

FIG. 1 is a simplified flow diagram illustrating a method of planning a rib row cutting path in accordance with an embodiment of the present invention;

FIG. 2 is a flow chart illustrating normal vector analysis of a rib row according to an embodiment of the present invention; and

fig. 3 is a block diagram illustrating a system for planning a rib row cutting path according to an embodiment of the present invention.

Detailed Description

Embodiments will now be described with reference to the accompanying drawings. It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, this application sets forth numerous specific details in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Moreover, this description is not to be taken as limiting the scope of the embodiments described herein.

The rib row is the slice spareribs of animal thorax position, and the piece head of this slice spareribs is great, is difficult to wholly sell, also is not favorable to the further fine processing and the packing of rib row simultaneously. Slaughter houses therefore often require that the spareribs are divided into sub-spareribs (one spareribs) which are then individually processed, packaged or sold. At present, the rib row can only be cut along the gap of the rib row manually, so that the rib row is cut, the efficiency is low, and meanwhile, the machine cannot accurately cut the rib row.

The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

Fig. 1 is a simplified flow diagram illustrating a method 100 of planning a rib row cutting path according to an embodiment of the present invention. FIG. 2 is a flow diagram illustrating a normal vector analysis 200 of a rib row according to an embodiment of the invention. A method 100 for planning a rib cutting path according to an embodiment of the present invention will be described in detail with reference to fig. 1 and 2.

As shown in fig. 1, the present invention provides a method 100 for planning a rib row cutting path, which may include the following steps S102-S108. At step S102, three-dimensional point cloud data and visible light picture data of the rib row are acquired. In one embodiment, the point cloud refers to a data set of points in a coordinate system, which is a set of three-dimensional points that can be obtained by different sensors, such as a depth camera, a lidar scanner, and the like. The points may contain rich information and may include, for example, information about three-dimensional coordinates X, Y, Z, color, classification value, intensity value, time, and the like. In one application scenario, the three-dimensional point cloud data and the visible light picture can be acquired by a depth camera. The acquisition process may be as follows: the rib row is first clamped on an operating table so that the rib row is fixed below the depth camera. And then opening depth flow data of the depth camera, and acquiring three-dimensional point cloud data and visible light picture data.

At step S104, RGBD point cloud data is generated from the three-dimensional point cloud data and the visible light picture data. Since the point cloud may also have rgb values (red, green, blue, three primary colors of red, green and blue) for each point, the point cloud may be a colored point cloud. In one embodiment, the depth three-dimensional point cloud data acquired by the depth camera at the same time can be fused with the rgb image to form a 6-channel point cloud of RGBD (rgb + depth, three primary colors of red, green and blue + depth), and the point cloud has both color image information and spatial position information. In one embodiment, the three-dimensional point cloud data and the rgb image may be fused by the following method. For example: the method mainly solves the camera parameters by solving the minimum distance from the corresponding straight line end point in the point cloud to the plane, and further realizes the fusion of the three-dimensional point cloud and the two-dimensional image. In order to facilitate later training of the segmentation model, at least 1000 pieces of RGBD image data of the rib rows can be collected through the depth camera at the early stage.

At step S106, the range of the rib row slit is determined from the point cloud data of the RGBD described above. In one embodiment, the range of the rib row gaps can be determined by a semantic segmentation model according to the point cloud data of the RGBD. In one application scenario, the semantic segmentation model may be a pointent + + model (a deep feature learning model that measures a set of points in space). Before determining the range of the rib row gaps through the pointenet + + model, the collected RGBD image data of at least 1000 rib rows may be segmented and labeled by using cloudcompare software (a three-dimensional point cloud processing software). The specific positions of the rib rows (such as rib row gaps, ribs and the like) are marked, and then the marked data are input into a pointet + + model for training. Then, according to the RGBD point cloud information, the specific spatial point cloud positions (such as rib gaps) of the ribs are identified through a pointent + + model, and the rib point cloud is separated from the background.

After the range of the rib row slit is identified, a set of rib row slit points is extracted from the range of the rib row slit and is fitted to the rib row cutting path at step S108. In an application scenario, a rib may be subjected to normal vector analysis according to the point cloud data of RGBD, so as to extract the rib gap point set from the range of the rib gap. In one embodiment, the process of performing normal vector analysis on the rib row according to the point cloud data of RGBD can be as shown in fig. 2: at step S202, a nearest neighbor point set within a preset radius range of each point on the rib row is determined according to the point cloud data of the RGBD of the rib row. In one embodiment, a nearest neighbor point set within a specified Radius range of each point in the rib point cloud can be found based on Radius-NN (Radius-nearest neighbor search, which specifies a specific Radius, traverses each point in the point cloud, and finds a nearest neighbor point set with a spatial distance smaller than the Radius from the current point as the current point). At step S204, the normal vector of each point in the nearest neighboring point set is determined (i.e. the normal vector of the point set is solved), so as to obtain a normal vector map of the whole rib row, thereby implementing the normal vector analysis of the rib row by point cloud. In step S206, points in the normal vector diagram with an included angle smaller than 90 degrees are taken as a rib row gap point set (i.e., points on two ribs facing each other). Here, for a regular surface R (u, v) in a three-dimensional space, the normal vector of the tangent plane (Ru, Rv) at the point (u, v) is the normal vector of the surface at the point (u, v). The point cloud is a point sample of the curved surface, and the normal vector of the sampled curved surface is the normal vector of the point cloud. And finding out points with the included angle of the normal vector smaller than 90 degrees (the direction is upward) in the nearest neighbor point set based on the obtained normal vector of each point in the nearest neighbor point set, wherein the point set formed by the points is the target rib row gap point set.

Having obtained the set of rib row slit points as described above, in one embodiment, the set of rib row slit points may be fit to a rib row cutting path. However, since this point set may be a set of points for all rib row slits, it is necessary to further divide the rib row slit point set into a point set for each rib row slit. In one embodiment, the rib row gap point set can be divided into a single group of rib row gap point sets by an Euclidean clustering segmentation method. That is, the nearest n neighbors of each rib gap point are first computed by a kd-tree (short for k-dimensional tree, a tree data structure that stores instance points in k-dimensional space for fast retrieval). Then judging whether the Euclidean distance from each adjacent point to the rib row gap point is within a threshold value; if so, the neighbor points are added to the set. After traversing all rib gap points, obtaining each point set based on the Euclidean clustering segmentation method, namely the point set of each rib gap, which will be called as a single-group rib gap point set hereinafter.

After the single set of rib row slit point sets are obtained as described above, since the points in the rib row slit point sets are discrete, in order to make the rib row cutting path smoother, in an application scenario, the obtained single set of rib row slit point sets may be respectively fitted to the single rib row cutting path. In one embodiment, a single set of rib slit point sets may be fitted to the rib cutting path by fitting an equation to a spatial circle. Specifically, a single set of rib slit point sets can be obtained By a method of random sampling consistency (parameters of a mathematical model are estimated from a set of observed data containing outliers in an iterative manner), and parameters of a spatial arc equation (combined with a (x-x0) ^2+ B (y-y0) ^2+ C (z-z0) ^2 ^ 1) are solved By a gradient descent algorithm, so that a spatial arc is obtained to smooth a single rib cutting path. In one implementation scenario, the arc start point and the arc end point of the single rib cutting path may also be determined according to the single set of rib slit point set coordinates. Finally, the coordinates of the cutting path of the single rib row are output in a uniform sampling mode. Of course, the coordinates of all the single rib cutting paths may be collected and output together. The rib cutting path is formed based on RGBD data, which is a three-dimensional, solid path. That is, it includes not only the coordinates in the up-down direction, the left-right direction, but also the coordinates in the front-back direction. Reflected in the cutting path, i.e. the rib cutting path includes not only the direction of the cut but also the depth of the cut. Therefore, the rib row cutting path formed by the technical scheme of the invention is more practical and accurate.

After the coordinates of the single rib row cutting path are obtained as described above, in order to allow the machine to perform the division of the rib row by the coordinates, in one application scenario, the rib row cutting path may also be converted into a coordinate system recognizable by the machine. And finally, the information of the path coordinates and the orientation of the normal vector is converted and sent to the manipulator through a coordinate system calibrated by the hand eyes to carry out the segmentation of the rib row. In practical application, in order to realize the coordinate system conversion of the hand-eye calibration, in the implementation process of the invention, a binocular structured light RGBD depth camera, a manipulator with a cutter (the introduction of the cutter is not expanded here) and a calibration plate are required. Before the work is actually started, the hand-eye calibration is first performed on the depth camera and the manipulator, and an open source calibration program may be used here. For example, eye to hand (eye to eye) is a scheme that a coordinate system from a calibration plate to a depth camera is calibrated, then coordinates of the calibration plate under a manipulator coordinate system are calibrated, and then a relation matrix between the manipulator coordinates and the depth camera coordinate system is converted through conversion of a coordinate matrix. The coordinates of the single rib row cutting path can be converted to coordinates that can be recognized by the robot.

The following describes the operation steps in the actual operation. Before the operation starts, the hand-eye calibration of the depth camera and the manipulator with the cutter needs to be carried out through the calibration plate. An open source calibration procedure may be used for hand-eye calibration, and the calibration scheme may be eye to hand. Firstly, calibrating a coordinate system from a calibration plate to a camera, and then calibrating coordinates of the calibration plate under the manipulator coordinate system, so as to convert a relation matrix between the manipulator coordinates and the camera coordinate system through the conversion of a coordinate matrix. After the calibration of the eyes and hands is finished, the data acquisition of the rib row is needed. The rib row is first clamped on the console and fixed under the RGBD depth camera for binocular structured light. And then opening the depth flow data of the depth camera, and acquiring the three-dimensional point cloud data and the visible light picture data of the rib row through the depth camera. At least 1000 pieces of RGBD image data of the rib row need to be acquired by the depth camera, and then segmentation and labeling of color point cloud are carried out on the acquired at least 1000 pieces of RGBD images by virtue of cloudcompare software so as to mark specific positions (such as rib row gaps) of the rib row. And inputting the marked data into a pointenet + + model for training.

The depth three-dimensional point cloud data acquired by the camera at the same time is fused with an rbg image (visible light picture data) to form a 6-channel point cloud of RGBD, and the point cloud has color image information and spatial position information at the same time. And then, according to RGBD point cloud information, identifying the specific spatial point cloud position of the rib row through a pointenet + + model, and separating the point cloud of the rib row from the background. And then carrying out normal vector analysis on the ribs by using the point cloud data: firstly, searching a nearest neighbor point set in a specified Radius range of each point on the rib row point cloud based on Radius-NN, analyzing and solving a normal vector of the nearest neighbor point set, and thus obtaining a normal vector graph of the whole rib row.

Then, based on the above normal vector map, finding out the point whose normal vector included angle is smaller than 90 degrees (direction is upward) in the nearest neighbor point set, where this point set is the target rib row gap point set, and this rib row gap point set may be the set of points of all gaps (more than one gap), so it is necessary to further divide the rib row gap point set into a single group of rib row gap point sets, where the rib row gap point set is divided into a single group of rib row gap point sets by the method of the euclidean clustering segmentation method. And then solving parameters of a space arc equation by using a gradient descent algorithm through a random sampling consistency method for the single-group rib gap point set, so as to obtain a space arc to fit the single-group rib gap point set, and further obtain a smooth rib cutting path. And determining the start point position and the end point position of the rib cutting path. Then, by uniform sampling, the rib row cutting path coordinates are output. And finally, converting the information of the coordinates of the cutting path of the rib row and the orientation of the normal vector through a coordinate system calibrated by hands and eyes, and sending the converted information to a manipulator to segment the rib row.

While the method 100 for planning the cutting path of the rib row is described above with reference to fig. 1 and 2, it should be understood by those skilled in the art that the above process is illustrative and not restrictive, and may be modified according to actual requirements. Fig. 3 is a block diagram illustrating a system for planning a rib row cutting path according to an embodiment of the present invention. A system for planning a rib cutting path according to an embodiment of the present invention will be described in detail with reference to fig. 3.

As shown in FIG. 3, the present invention provides a system for planning a rib cutting path, the system being in the form of a general purpose computing device, including but not limited to: at least one processor, at least one memory, a communication bus connecting different system components. The communication bus represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. The memory may include readable media in the form of volatile memory, such as Random Access Memory (RAM) and/or cache memory, and may further include Read Only Memory (ROM). The memory may also include program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

Furthermore, the invention also provides a computer-readable storage medium comprising computer program instructions which, when executed by one or more processors, cause a method according to any one of the embodiments of the invention and the above described implementations to be carried out.

The invention provides a rib row segmentation method, which finds out the position of a gap between each rib among rib rows through three-dimensional point cloud data and fits the coordinates of the gap position into a planned rib row cutting path. The rib row cutting path is used as a lower cutter position for cutting the rib row by a mechanical hand, so that the rib row is cut in a standardized and efficient manner by a machine. The requirement of industrial capacity upgrading is met.

It should be noted that while the operations of the method of the present invention are depicted in the drawings in a particular order, this is not intended to require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Rather, the steps depicted in the flowcharts may change the order of execution. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.

It should be understood that the terms "first", "second", "third" and "fourth", etc. used in the claims, the specification and the drawings of the present invention are only used for distinguishing different objects, and are not used for describing a specific order. The terms "comprises" and "comprising," when used in the specification and claims of this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification and claims of this application, the singular form of "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the term "and/or" as used in the specification and claims of this specification refers to any and all possible combinations of one or more of the associated listed items and includes such combinations.

Although the embodiments of the present invention are described above, the descriptions are only examples for facilitating understanding of the present invention, and are not intended to limit the scope and application scenarios of the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (10)

1. A method of planning a rib cutting path, comprising the steps of:

acquiring three-dimensional point cloud data and visible light picture data of the rib rows;

generating RGBD point cloud data according to the three-dimensional point cloud data and the visible light picture data;

determining the range of rib row gaps according to the RGBD point cloud data;

extracting rib row gap point sets from the range of the rib row gaps; and

fitting the set of rib row slit points to a rib row cutting path.

2. The method of claim 1, further comprising:

converting the rib cutting path to a machine recognizable set of coordinates.

3. The method of claim 1, wherein determining the range of the rib row gaps from the RGBD point cloud data comprises:

and determining the range of the rib row gaps by utilizing a semantic segmentation model according to the point cloud data of the RGBD.

4. The method of claim 3, wherein extracting the rib row slit point from within the range of the rib row slit comprises: and performing normal vector analysis on the rib row according to the point cloud data of the RGBD to extract the rib row gap point set from the range of the rib row gap.

5. The method of claim 4, wherein performing normal vector analysis on the rib row according to the point cloud data of RGBD to extract the set of rib row gap points from the range of the rib row gap comprises:

determining a nearest neighbor point set within a preset radius range of each point on the rib row according to the RGBD point cloud data of the rib row;

determining a normal vector of each point in the nearest neighbor point set;

and taking the points with the normal vector included angle smaller than 90 degrees as a rib row gap point set.

6. The method of claim 1, wherein fitting the set of rib row slit points to a rib row cutting path comprises: and dividing the rib row gap point set into a single group of rib row gap point sets, and respectively fitting the single group of rib row gap point sets into a single rib row cutting path.

7. The method of claim 1, wherein fitting the set of rib row slit points to a rib row cutting path comprises:

fitting the set of rib row slit points to a rib row cutting path by fitting a spatial circle.

8. The method of claim 7, wherein fitting the rib row slit points to a rib row cutting path further comprises: and determining the positions of the starting point and the ending point of the rib row cutting path, and determining the coordinates of the rib row cutting path.

9. A system for planning a rib cutting path, the system comprising: one or more processors;

storage means for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.

10. A computer-readable storage medium comprising computer program instructions which, when executed by one or more processors, cause performance of the method of any one of claims 1-8.

Images