JP2018179859A - Image processing apparatus, image processing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To control an image processing apparatus, an image processing method and a computer for controlling a picking operation by a manipulator in 3D picking based on statistical data regarding success or failure of gripping at a position registered in advance even if the workpieces are stacked in bulk. Provide a program. SOLUTION: A work space in which bulk-loaded workpieces are mounted is three-dimensionally measured, and a plurality of work models stored in advance are registered. The gripping data composed of the gripped position and the posture is set, and the statistical data regarding the success or failure of the gripping is stored for each gripping data. For the acquired 3D measurement data, a 3D search process is executed to specify the position and orientation of the work using a plurality of registered work models, and whether or not the work with a large detection score interferes with other works. Is determined. The picking operation is controlled by selecting the gripping data based on the gripping success rate in the statistical data from the gripping data determined not to interfere with other workpieces. [Selection diagram] Fig. 4

Description

  The present invention is an image processing apparatus and an image processing method for controlling picking operation by a manipulator in 3D picking based on statistical data on success or failure of gripping at a position registered in advance even in a work in a bulk state. And a computer program.

  In the factory, 3D picking technology has been improved so that picking operations can be performed automatically by manipulators. In 3D picking, first, a work space in which workpieces to be picked are bulk-laminated is three-dimensionally measured by a three-dimensional measurement sensor to acquire three-dimensional measurement data. Next, using an image processing apparatus, a three-dimensional search process using a workpiece model registered in advance from among three-dimensional measurement data is executed to detect the position and orientation of the workpiece. With respect to the workpiece whose position and orientation have been detected, the gripping position of the manipulator for gripping the position on the workpiece to be gripped by the manipulator and the position on the workpiece to be gripped is determined using the previously registered gripping data, The robot controller actually operates the manipulator or moves the gripper of the manipulator on 3D-CAD of the simulator of the image processing apparatus. As described above, the gripping portion of the manipulator is guided to a position where the workpiece can be gripped, and the positional relationship between the workpiece and the gripping portion of the manipulator is detected and registered. As a result, movement control of the gripping portion of the manipulator can be reliably performed to a position at which the workpiece can be gripped.

  However, the workpiece can not be gripped 100% reliably, and in some circumstances it may not be possible to grip the workpiece. Therefore, it is considered to use statistical data on success or failure of past gripping for controlling the operation of the manipulator and the gripping unit in order to increase the probability that the workpiece can be gripped even a little.

  For example, in Patent Document 1, a work taking-out apparatus is put in the form of a database of the success or failure of the past work taking out, and the ease of grasping is estimated based on the measurement feature and information regarding the success or failure of taking out. It is disclosed. According to Patent Document 1, at least one of an open / close amount of a hand, a gripping force, and an operation speed can be automatically adjusted to take out a work that may be difficult to grasp.

JP, 2013-052490, A

  However, in the method disclosed in Patent Document 1, since the operation is controlled so as to take out a workpiece that is easy to grasp based on the information made into a database in work units, when there are a plurality of grippable parts in the same workpiece However, there is a problem that it is impossible to change which position to hold. In addition, it is not possible to change the gripping priority of the grippable part based on statistical information.

  The present invention has been made in view of such circumstances, and the picking operation by the manipulator in 3D picking is based on statistical data on success or failure of gripping at a position registered in advance even if the work is in a bulk state. It is an object of the present invention to provide an image processing apparatus connected to a robot controller for controlling the image processing apparatus, an image processing method, and a computer program.

  In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention is a robot controller for controlling picking operations for gripping workpieces stacked in bulk with a gripper of a manipulator and moving the gripping workpieces to a predetermined position In the image processing apparatus connected to the three-dimensional measurement of the work space in which the workpieces loaded in bulk are mounted to obtain a three-dimensional measurement data, and a plurality of work models stored in advance Grasping data composed of a workpiece model registration unit to be registered and a position to be gripped and a posture of the gripping unit of the manipulator to be gripped at the position as relative position data with respect to a model coordinate system of the workpiece model And storing statistical data on success or failure of gripping of the workpiece for each of the gripping data. A measurement data storage unit, and a search process execution unit that executes a three-dimensional search process that specifies the position and orientation of a workpiece using a plurality of registered workpiece models on three-dimensional measurement data acquired by three-dimensional measurement; An interference determination unit that determines whether or not a workpiece whose detection score output as a result of the three-dimensional search processing is larger than a predetermined threshold interferes with another workpiece, and a grip determined not to interfere with the other workpiece The data processing apparatus is characterized by further comprising a gripping data selection unit that selects gripping data based on a success rate of gripping in the statistical data among the data.

  Further, in the image processing apparatus according to the second invention, in the first invention, the sensing unit captures a workpiece in the work space to obtain image data, and the image data captured and acquired It is preferable to provide a three-dimensional data extraction unit that extracts three-dimensional data of a work in the work space to be subjected to the three-dimensional search processing as three-dimensional measurement data based on the above.

  Further, in the image processing apparatus according to the third invention, in the first or second invention, the statistical data storage unit stores statistical data regarding success or failure of gripping of the workpiece for each partial area where the workpiece is detected, and the gripping Preferably, the data selection unit selects, from among the grip data determined to not cause interference in the interference determination unit, grip data corresponding to a workpiece present in a partial region with the highest success rate of gripping in the statistical data.

  In the image processing apparatus according to the fourth invention, in any one of the first to third inventions, the statistical data storage unit stores the statistical data including the number of times of selection of gripping data, and the interference determination It is preferable that the unit performs the interference determination in order from the grip data having a large number of selections in the statistical data.

  Next, in order to achieve the above object, in the image processing method according to the fifth aspect of the present invention, a picking operation is performed in which workpieces stacked in bulk are gripped by the gripping portion of the manipulator and the griped workpieces are moved to a predetermined position. In the image processing method that can be executed by an image processing apparatus connected to the robot controller, the image processing apparatus three-dimensionally measures a work space in which work pieces stacked in bulk are mounted to perform three-dimensional measurement. The first step of acquiring data, the second step of registering a plurality of workpiece models stored in advance, and gripping as relative position data with respect to the model coordinate system of the workpiece model A third step of setting gripping data composed of the position and the posture of the gripping portion of the manipulator gripping at the position; The fourth step of storing statistical data on the success or failure of grasping of the workpiece, and the third order of specifying the position and posture of the workpiece using a plurality of registered workpiece models for the three-dimensional measurement data acquired by three-dimensional measurement A fifth step of executing an original search process, and a sixth step of determining whether or not a workpiece whose detection score output as a result of the three-dimensional search process is larger than a predetermined threshold interferes with another workpiece And a seventh step of selecting gripping data based on a gripping success rate in the statistical data from among gripping data determined not to interfere with another workpiece.

  The image processing method according to the sixth aspect of the present invention is the image processing method according to the fifth aspect, wherein the first step is an eighth step of capturing an image of a workpiece in the work space to obtain image data. It is preferable to include a ninth step of extracting, as three-dimensional measurement data, three-dimensional data of a work in the work space to be subjected to the three-dimensional search processing based on the image data.

  In the image processing method according to the seventh invention, in the fifth or sixth invention, the fourth step stores statistical data regarding success or failure of gripping of the workpiece for each partial area where the workpiece is detected; In the seventh process, it is preferable to select the gripping data corresponding to the workpiece present in the partial region having the highest gripping success rate in the statistical data among the gripping data determined not to interfere in the sixth step. .

  In the image processing method according to the eighth invention, in any one of the fifth to seventh inventions, the fourth step stores the statistical data including the number of times of selection of gripping data, and the sixth step It is preferable that in the process of 3, the interference determination is performed in order from the grip data having a large number of selections in the statistical data.

  Next, in order to achieve the above object, a computer program according to a ninth aspect of the present invention is for controlling a picking operation of gripping workpieces stacked in bulk by a gripping portion of a manipulator and moving the gripping workpieces to a predetermined position. In a computer program that can be executed by an image processing apparatus connected to a robot controller, the image processing apparatus performs three-dimensional measurement on a work space in which workpieces stacked in bulk are mounted to obtain three-dimensional measurement data. The sensing means to be acquired, the work model registration means for registering a plurality of work models stored in advance, the position to be held and the position held as relative position data with respect to the model coordinate system of the work model For setting the grip data composed of the posture of the grip portion of the manipulator Setting means, statistical data storage means for storing statistical data on success or failure of gripping of the workpiece for each gripping data, and using a plurality of workpiece models registered for three-dimensional measurement data acquired by three-dimensional measurement Search process execution means for executing a three-dimensional search process that specifies the position and orientation of the subject, whether or not a workpiece whose detection score output as a result of the three-dimensional search process is larger than a predetermined threshold interferes with other workpieces It is characterized in that it functions as a grip data selection means for selecting grip data based on a success rate of gripping in the statistical data among interference determination means to be determined and grip data determined to not interfere with other work. Do.

  A computer program according to the tenth invention is the computer program according to the ninth invention, wherein the sensing means is a work imaging means for imaging a workpiece in the work space to acquire image data, and image data acquired by imaging. It is preferable to function as three-dimensional data extraction means for extracting three-dimensional data of a work in the work space to be subjected to the three-dimensional search process as three-dimensional measurement data.

  A computer program according to an eleventh aspect of the present invention is the computer program according to the ninth or tenth aspect, wherein the statistical data storage means functions as a means for storing statistical data regarding success or failure of gripping of a workpiece for each partial area where the workpiece is detected. Means for selecting the gripping data corresponding to the work existing in the partial region having the highest success rate of gripping in the statistical data among the gripping data determined by the interference determining means not to cause interference by the gripping data selecting means It is preferable to function as

  A computer program according to a twelfth aspect of the present invention is the computer program according to any one of the ninth to eleventh aspects, wherein the statistical data storage means functions as means for storing the statistical data including the number of times of selection of grip data. It is preferable to cause the interference determination unit to function as a unit that performs the interference determination in order from the grip data having a large number of selections in the statistical data.

  In the first invention, the fifth invention, and the ninth invention, the image processing apparatus three-dimensionally measures the work space in which the bulk-stacked workpieces are mounted, and acquires three-dimensional measurement data. A plurality of workpiece models stored in advance are registered, and position data to be held relative to the model coordinate system of the workpiece model are the position to be held and the posture of the holding portion of the manipulator held at the position Set the configured grip data. Statistical data on success or failure of gripping of a workpiece is stored for each gripping data, and the three-dimensional measurement data acquired by three-dimensional measurement is used to identify the position and orientation of the workpiece using a plurality of registered workpiece models. Execute search processing. It is determined whether or not a workpiece whose detection score output as a result of the three-dimensional search processing is larger than a predetermined threshold interferes with another workpiece, and among gripping data determined not to interfere with the other workpiece, Grasping data is selected based on the success rate of gripping in statistical data. As a result, the success rate of gripping the workpiece by the gripper of the manipulator can be increased, and the operation rate of the picking device can be increased, and the tact time of the operation can be improved.

  In the second invention, the sixth invention and the tenth invention, the work in the work space is acquired to obtain image data, and the work in the work space to be subjected to the three-dimensional search processing based on the obtained image data. Since three-dimensional data of the above are extracted as three-dimensional measurement data, it is possible to acquire the bulk stacking state of the work in the work space as three-dimensional measurement data without using an expensive three-dimensional shape sensor.

  In the third invention, the seventh invention, and the eleventh invention, statistical data relating to the success or failure of gripping of the workpiece for each partial area in which the workpiece is detected is stored, and statistical data among gripping data determined not to interfere Since the gripping data corresponding to the workpiece existing in the partial region where the success rate of gripping is highest is selected, the partial region where the success rate of gripping is high and the workpiece model of the workpiece detected as a gripping object in the partial region are set By using the gripping data, the success rate of gripping the workpiece by the gripping unit of the manipulator can be increased, and the operation rate of the picking device can be enhanced, and the tact time of the work can be improved.

  In the fourth, eighth, and twelfth inventions, the statistical data is stored including the number of times of selection of gripping data, and interference determination is sequentially performed from the gripping data having the largest number of selection in the statistical data. The success rate of gripping the workpiece by the manipulator gripping part can be increased by using gripping data that has been determined and selected many times in the past as being high, and the operation rate of the picking device can be increased and the tact time of the work It is possible to improve the

  According to the present invention, it is determined whether or not a workpiece whose detection score is greater than a predetermined threshold interferes with another workpiece, and among gripping data determined not to interfere with the other workpiece, gripping in statistical data Select gripping data based on the success rate of As a result, the success rate of gripping the workpiece by the gripper of the manipulator can be increased, and the operation rate of the picking device can be increased, and the tact time of the operation can be improved.

It is a block diagram showing an example of composition of a picking system using an image processing device concerning an embodiment of the invention. FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. It is an illustration figure which shows the structure of the imaging part of the image processing apparatus which concerns on embodiment of this invention. FIG. 1 is a functional block diagram of an image processing apparatus according to Embodiment 1 of the present invention. It is an illustration figure of the position and attitude of a grasping part memorized as grasping data of an image processing device concerning Embodiment 1 of the present invention. It is a schematic diagram which shows an example of the success or failure judgment of the holding | grip in the robot controller connected with the image processing apparatus which concerns on Embodiment 1 of this invention. It is an illustration figure of the work model of the image processing device concerning an embodiment of the invention. It is explanatory drawing of the three-dimensional search process of the image processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the process sequence after the three-dimensional search process of CPU of the image processing apparatus concerning Embodiment 1 of this invention. It is a flowchart which shows the procedure of the priority determination process of the holding | grip data of CPU of the image processing apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows the procedure of the priority decision processing of the holding data with a high holding success rate of CPU of the image processing apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows the procedure of the priority decision processing of the partial area which shows the workpiece | work of CPU of the image processing apparatus which concerns on Embodiment 2 of this invention. It is a flowchart which shows the procedure of the priority decision processing of the partial area | region where a gripping success rate is high of CPU of the image processing apparatus which concerns on Embodiment 2 of this invention. It is a flowchart which shows the procedure of a process of a picking system using the image processing apparatus which concerns on embodiment of this invention.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be specifically described based on the drawings. In the present embodiment, in the picking operation of the manipulator, it is used to guide the gripping portion of the manipulator to a position and posture with high probability of success in gripping.

  FIG. 1 is a block diagram showing a configuration example of a picking system using an image processing apparatus according to an embodiment of the present invention. In the image processing apparatus 1, an imaging unit 2 for imaging workpieces stacked in bulk is connected as a sensing unit, setting of image processing is performed by an input device such as a keyboard 111 and a mouse 112, and setting is performed by a display device 113. And check the operating status.

  FIG. 2 is a block diagram showing the configuration of the image processing apparatus 1 according to the embodiment of the present invention. As shown in FIG. 2, the image processing apparatus 1 according to the present embodiment includes at least a CPU 11, a memory 12, a storage device 13 such as a hard disk, an I / O interface 14, a video interface 15, a communication interface 16, and a portable disk drive. 17 and an internal bus 18 connecting the above-described hardware.

  The CPU 11 is connected to the above-described hardware units of the image processing apparatus 1 via the internal bus 18 and controls the operation of the above-described hardware units, and the computer program 100 stored in the storage device 13 Perform various software-like functions according to The memory 12 is composed of volatile memory such as SRAM, SDRAM, etc. The load module is expanded when the computer program 100 is executed, and stores temporary data and the like generated when the computer program 100 is executed.

  The storage device 13 is configured by a built-in fixed storage device (hard disk), a ROM, and the like. The computer program 100 stored in the storage device 13 is downloaded by the portable disk drive 17 from a portable recording medium 90 such as a DVD, CD-ROM or the like in which information such as a program and data is recorded. To memory 12 and executed. Of course, it may be a computer program downloaded from an external computer connected to the network via the communication interface 16.

  The I / O interface 14 is connected to input devices such as a keyboard 111 and a mouse 112 and receives data input. Further, the video interface 15 is connected to a display device 113 such as a CRT monitor or an LCD, and displays setting data of the image processing apparatus 1, an operation state of a manipulator, and the like.

  The communication interface 16 is connected to the internal bus 18, and by being connected to an external network such as the Internet, LAN, or WAN, data can be exchanged with the robot controller 4, the imaging unit 2, the external computer, etc. It has become.

  Returning to FIG. 1, the imaging unit 2 three-dimensionally images the workpieces stacked in bulk in the work space. FIG. 3 is an exemplary view showing a configuration of the imaging unit 2 of the image processing apparatus 1 according to the embodiment of the present invention.

  As shown in FIG. 3, the imaging unit 2 according to the present embodiment has a total of four cameras 22 (22a), one for each of the front and rear and left and right directions with respect to the moving direction of the work, with one projector 21 as the center. To 22d). This makes it possible to acquire three-dimensional measurement data for grasping the position and posture of each workpiece in the work space without creating a blind spot regardless of the state of workpieces stacked in bulk. Of course, it is sufficient if at least one or more cameras 22 are provided, as long as the configuration of only the cameras 22 excluding the projector 21 can generate a three-dimensional model.

  The light source of the projector 21 may be, for example, a halogen lamp that emits white light, or a white LED (light emitting diode) that emits white light. A pattern generation unit (not shown) is provided in the vicinity of the emission port to generate a fringe pattern. The pattern generation unit may be, for example, a DMD (digital micro mirror device), LCOS (registered trademark) (Liquid Crystal on Silicon: reflective liquid crystal element), a mask or the like.

  The cameras 22a to 22d are disposed at four positions in the front-rear and left-right directions with respect to the movement direction of the workpieces stacked in bulk. The cameras 22a to 22d each include an imaging device such as a monochrome CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor) image sensor. A light reception signal corresponding to the light reception amount is output for each pixel of the imaging device. Then, an image is generated based on the light reception signal.

  Referring back to FIG. 1, the image processing apparatus 1 is connected in data communication with a robot controller 4 that controls the operation of the manipulator 5. Since the position and orientation of the work W on the work space are specified by the image processing apparatus 1, in the image processing apparatus 1, the robot controller 4 controls the operation of the manipulator 5 and the holding unit 50 to hold the work W. Necessary information is output to the robot controller 4. The robot controller 4 generates an operation signal of the manipulator 5 to control the operation of the manipulator 5 and the gripping unit 50.

  A pendant 6 is connected to the robot controller 4 so as to be capable of performing data communication, and receives input of setting data related to operations of the manipulator 5 and the gripping unit 50.

Embodiment 1
The operation of the image processing apparatus 1 having the above configuration will be described. FIG. 4 is a functional block diagram of the image processing apparatus 1 according to the first embodiment of the present invention. In FIG. 4, the sensing unit 1001 three-dimensionally measures the work space in which the work W stacked in bulk is mounted, for example, via the imaging unit 2, and acquires three-dimensional measurement data. In the present embodiment, the sensing unit 1001 includes a work imaging unit 1002 and a three-dimensional data extraction unit 1003.

  The work imaging unit 1002 captures an image of the work W in the work space with the imaging unit 2 and acquires image data. Then, three-dimensional data is generated on the basis of the acquired image data. The three-dimensional data extraction unit 1003 extracts three-dimensional data of the workpiece W in the work space to be subjected to the three-dimensional search processing based on the acquired image data, and sets it as three-dimensional measurement data. Here, "three-dimensional data" refers to three-dimensional data of a single workpiece W not stacked in bulk, and "three-dimensional measurement data" refers to three-dimensional data of the entire workpiece W stacked in bulk. Shall be meant. That is, the three-dimensional search process is executed with the three-dimensional data as key information and the three-dimensional measurement data as a search target.

  The work model registration unit 1004 registers one or a plurality of three-dimensional data (hereinafter referred to as a work model) to be used in the three-dimensional search process out of the generated three-dimensional data. The registered work model is the key information of the three-dimensional search process. The three-dimensional data (work model) may be acquired from CAD data of the work W.

  The gripping data setting unit 1005 sets, as relative position data, gripping data composed of the position to be gripped and the posture of the gripping unit 50 of the manipulator 5 to be gripped at the position with respect to the model coordinate system of the workpiece model Do. Specifically, with respect to three-dimensional measurement data acquired by three-dimensional measurement while sequentially changing the posture of the workpiece W, setting of a model coordinate system and holding data of one or a plurality of registered work models are performed. The setting is repeated, and a plurality of grip data are stored in the storage device 13 in association with each work model.

  FIG. 5 is a view showing an example of the position and the attitude of the grip unit 50 stored as grip data of the image processing apparatus 1 according to the first embodiment of the present invention. FIG. 5A is a perspective view illustrating a workpiece model of the workpiece W to be gripped, and FIGS. 5B to 5G are positions and orientations of the gripping unit 50 stored as gripping data. It is the perspective view which illustrated.

  As shown in FIG. 5A, when the workpiece W in which a plurality of gripping positions exist is a gripping target, the position and the posture of the gripping unit 50 differ according to the gripping position. FIG. 5B and FIG. 5C show the position and posture of the gripping portion 50 when the flange W1 in the long axis direction of the workpiece W is a gripping target. Similarly, FIGS. 5 (d) and 5 (e) show the position and orientation of the gripping portion 50 when the annular portion W2 of the workpiece W is a gripping object, and FIGS. 5 (f) and 5 (g) show them. The position and the posture of the gripping portion 50 when the flange portion W3 in the short axis direction of the workpiece W is a gripping target are shown.

  As described above, even if workpieces stacked in bulk are detected at any position and posture by setting gripping data in advance according to which part of the workpiece W is gripped from which direction. , It is possible to detect a portion to be gripped.

  Returning to FIG. 4, the statistical data storage unit 1006 stores statistical data on success or failure of gripping of the workpiece W for each gripping data. That is, the workpiece W is picked based on the grip data selected each time, and the result data of whether or not the grip is successful is stored in the storage device 13 as statistical data. Further, the statistical data storage unit 1006 also stores, in the storage device 13, statistical data as to whether or not interference will be made by the interference determination unit 1008 described later for each grip data.

  The method for determining the success or failure of gripping is not particularly limited. FIG. 6 is a schematic view showing an example of gripping success / failure determination in the robot controller 4 connected to the image processing apparatus 1 according to the first embodiment of the present invention. FIG. 6A is a schematic view showing the configuration of the gripping portion 50 of the manipulator 5. The two gripping members 501 are moved to detect the distance B1 between the gripping members 501 and 501.

  Assuming that the width of the portion to be gripped of the workpiece W to be gripped is B2, it can be understood that it is a gripping process if B1> B2 (state of FIG. 6A). And when it is B1 = B2, it turns out that it has succeeded in grasping work W (state of Drawing 6 (b)). However, if B1 <B2, it can be determined that gripping has failed (the state of FIG. 6C).

  Of course, the method of judging the success or failure of gripping is not limited to this. For example, a weight sensor or a pressure sensor may be attached to the gripping portion, and it may be determined whether the gripping of the workpiece W has succeeded according to the output value of the sensor.

  Alternatively, an electrode may be simply provided at the tip of the holding member 501 of the holding unit 50, and a warning signal may be output when electricity is supplied. The electrodes are in contact only when the state of FIG. 6C is reached, and a warning signal is output.

  Alternatively, the held workpiece W may be released at the destination, and the position may be imaged by another camera. If the workpiece W does not exist at the predetermined position or the positions are different, it can be determined that the gripping has failed if it is determined from analysis of the image data acquired by imaging.

  Similarly, another camera may capture an image of the held work W while moving to the destination. If it is determined from the analysis of the image data acquired by imaging that the workpiece W does not exist during movement, it can be determined that gripping has failed.

  Furthermore, in the case where the gripping portion 50 is a method using suction by suction of air, it is possible to judge success or failure of gripping based on the difference in air flow rate. Of course, only the data acquisition by the sensor or the like may be executed by the instruction of the robot controller 4, and the image processing apparatus 1 may execute the gripping success / failure judgment.

  Returning to FIG. 4, the search processing execution unit 1007 specifies the position and orientation of the workpiece W using the registered one or more workpiece models for the three-dimensional measurement data acquired by three-dimensional measurement. Execute search processing. The search process execution unit 1007 calculates a detection score for each work W and outputs it as a result of the three-dimensional search process.

  FIG. 7 is a view showing an example of a work model of the image processing apparatus 1 according to the embodiment of the present invention. As shown in FIG. 7, feature points are extracted on the surface of the workpiece model. The feature point means a point indicating a feature required in the three-dimensional search process, for example, two feature points of feature points on the contour representing the contour of the shape, and feature points on the surface representing the surface shape or What is necessary is just to be comprised by one of the feature points.

  FIG. 8 is an explanatory diagram of a three-dimensional search process of the image processing apparatus 1 according to the embodiment of the present invention. As shown to Fig.8 (a), the workpiece | work W is piled up in the work space. In this state, the workpieces 81 and 82 are detected by executing a three-dimensional search process using the workpiece model shown in FIG.

  In the three-dimensional search process, a detection score is calculated based on a ratio indicating how much the feature point corresponding to the work model exists. For example, for the height image, the ratio of the score of feature points having an error equal to or less than a predetermined distance is used. When many invalid pixels are included, the detection score is low because there is no corresponding feature point. Thus, the higher the detection score, the more reliable as a result of the three-dimensional search process. Moreover, when it is a detection score below a fixed threshold value, since it is highly possible that it is misdetecting, it can also be remove | excluded from holding | grip object.

  Returning to FIG. 4, the interference determination unit 1008 determines whether or not a workpiece W having a detection score output as a result of the three-dimensional search processing larger than a predetermined threshold interferes with another workpiece W. The interference check with another work W is performed only for the work W which is highly reliable and is detected as a gripping target. Thereby, the workpiece | work W which interferes can be excluded from holding object.

  The gripping data selection unit 1009 selects, from among the gripping data determined to not interfere with another work W, the gripping data having the highest gripping success rate in the statistical data. Further, the gripping data selection unit 1009 excludes gripping data having a low gripping success rate from gripping data candidates to be selected among gripping data determined to not interfere with another workpiece W based on statistical data. In this case, the gripping data selection unit 1009 selects, from among the gripping data excluding the excluded gripping data, the gripping data having the highest detection score obtained by the three-dimensional search processing.

  That is, the grip data selection unit 1009 performs the interference determination on the grip data for which the detection score higher than the predetermined threshold is obtained by the three-dimensional search process, and the statistical data storage unit 1006 for the grip data determined not to interfere. By referring to the statistical data stored in the storage device 13, the user selects grip data having a high probability of actually successfully gripping. The robot controller 4 shown in FIG. 1 controls the operation of the manipulator 5 and the gripping unit 50 based on the gripping data set in the workpiece model of the workpiece W detected as a gripping target.

  FIG. 9 is a flowchart showing the processing procedure after the three-dimensional search processing of the CPU 11 of the image processing apparatus 1 according to Embodiment 1 of the present invention. As shown in FIG. 9, the CPU 11 of the image processing apparatus 1 executes the three-dimensional search processing as described above (step S901), and specifies the position and orientation of the workpiece W using one or more workpiece models, It is determined whether a workpiece W has been detected (step S902).

  When the CPU 11 determines that the work W has been detected (step S 902: YES), the CPU 11 selects the work W having the highest priority from the detected one or more works W (step S 903). An interference determination with another work W in the case of using the grip data with the highest degree is executed (step S904). The CPU 11 determines whether there is interference with another work W (step S905).

  When the CPU 11 determines that there is no interference with another work W (step S 905: NO), the CPU 11 transmits an instruction to the robot controller 4 to grip the selected work W with the selected gripping data ( Step S911). When the CPU 11 determines that there is an interference with another work W (step S 905: YES), the CPU 11 determines whether there is other grip data (step S 906).

  If the CPU 11 determines that there is another grip data (step S 906: YES), the CPU 11 performs an interference determination with another work W when using the grip data with the second highest priority (step S 907). ), The process returns to step S905 and the above-described process is repeated. If the CPU 11 determines that there is no other grip data (step S 906: NO), the CPU 11 determines whether there is another work W (step S 908).

  If the CPU 11 determines that there is another work W (step S 908: YES), the CPU 11 selects the work W having the next highest priority (step S 909), returns the process to step S 904, and performs the process described above repeat. If the CPU 11 determines that there is no other work W (step S 908: NO), and if it determines that the work W is not detected (step S 902: NO), the CPU 11 determines that gripping is not possible. Then (step S910), the process ends.

  In the above-described process, priority is given to the presence or absence of grip data, but the presence or absence of another work W may be prioritized. In this case, steps S906 and S907 and steps S908 and S909 are interchanged.

  Also, the priority of the grip data can be determined in various ways. For example, the statistical data storage unit 1006 stores statistical data including the number of times of selection of gripping data, and the interference determination unit 1008 makes interference determination by increasing the priority of gripping data having a large number of times of selection in statistical data. It may be a method to do. FIG. 10 is a flowchart showing a procedure of priority determination processing of grip data of the CPU 11 of the image processing apparatus 1 according to the first embodiment of the present invention.

  As shown in FIG. 10, the CPU 11 of the image processing apparatus 1 initializes the counter for each grip data to '0' (step S1001), and temporarily sets the priority of the grip data (step S1002). The CPU 11 executes the process shown in FIG. 9 (step S1003), and adds the counter of the selected grip data (step S1004).

  The CPU 11 changes the priority of the grip data according to the value of the counter (step S1005), and determines whether or not to initialize (step S1006). When the CPU 11 determines that initialization is not to be performed (step S1006: NO), the CPU 11 returns the process to step S1003 and repeats the above-described process. If the CPU 11 determines to initialize (step S1006: YES), the CPU 11 ends the process.

  Also, statistical data on success or failure of gripping of gripping data can be created similarly by using a success / failure counter. FIG. 11 is a flowchart showing a procedure of priority determination processing of gripping data with a high gripping success rate of the CPU 11 of the image processing apparatus 1 according to the first embodiment of the present invention.

  As shown in FIG. 11, the CPU 11 of the image processing apparatus 1 initializes a success / failure counter for each grip data (step S1101), and temporarily sets the priority of the grip data (step S1102). The CPU 11 executes the process shown in FIG. 9 (step S1103), and outputs the selected grip data to the robot controller 4 (step S1104).

  The CPU 11 acquires grip success / failure data regarding success or failure of gripping with the selected grip data from the robot controller 4 (step S1105), and determines whether gripping has succeeded (step S1106). If the CPU 11 determines that gripping has succeeded (step S1106: YES), the CPU 11 updates the success / failure counter of the selected gripping data (step S1107).

  If the CPU 11 determines that gripping has failed (step S1106: NO), the CPU 11 updates the success / failure counter of the selected gripping data (step S1108), and determines whether to end the process (step S1109) ). When the CPU 11 determines that the process is not ended (step S1109: NO), the CPU 11 returns the process to step S1103 and repeats the above-described process. If the CPU 11 determines that the process is to be ended (step S1109: YES), the CPU 11 ends the process.

  As described above, according to the first embodiment, it is determined whether or not a workpiece W having a detection score larger than a predetermined threshold interferes with another workpiece W, and it is determined that the workpiece W does not interfere with the other workpiece W. Among gripping data, the gripping data with the highest gripping success rate in statistical data is selected. By controlling the operation of the manipulator 5 and the gripping unit 50 based on the selected gripping data, the success rate of the gripping unit 50 of the manipulator 5 gripping the workpiece W can be increased, and the operation rate of the picking device can be increased. And the tact time of the work can be improved.

Second Embodiment
The configuration and functional block diagram of the image processing apparatus 1 according to the second embodiment of the present invention are the same as in the first embodiment, and thus the detailed description will be omitted by attaching the same reference numerals. The second embodiment is different from the first embodiment in that statistical data is generated based on whether or not there is gripping data that does not interfere with the partial region, instead of generating statistical data based on the gripping data. It is different.

  In the second embodiment, the processing procedure after the three-dimensional search processing shown in FIG. 9 is similar to the case where the presence or absence of another work W is prioritized. Therefore, in FIG. 9, description will be made on the assumption that steps S906 and S907 and steps S908 and S909 have been interchanged.

  The gripping data selection unit 1009 in FIG. 4 selects the gripping data set in the workpiece model of the workpiece W detected as a gripping object among the gripping data determined that the gripping data that does not interfere with the partial area is present . The robot controller 4 shown in FIG. 1 receives a control signal for controlling the operation of the manipulator 5 and the gripping unit 50 based on the gripping data set in the workpiece model of the workpiece W detected as the gripping object, and the gripping unit 50 Control the operation of

  The priority of the partial area indicating the workpiece W to be gripped can be determined by the following procedure. FIG. 12 is a flowchart showing the procedure of the process of determining the priority of the partial area indicating the work W of the CPU 11 of the image processing apparatus 1 according to the second embodiment of the present invention.

  As shown in FIG. 12, the CPU 11 of the image processing apparatus 1 initializes the counter for each partial area to '0' (step S1201), and temporarily sets the priority of the partial area (step S1202). The CPU 11 executes the process shown in FIG. 9 (step S1203), and determines whether or not the selected grip data exists in the partial area (step S1204). If it is determined that the CPU 11 exists in the partial area (step S1204: YES), the CPU 11 adds a counter of the partial area in which the grip data exists (step S1205). If the CPU 11 determines that the partial area does not exist (step S1204: NO), the CPU 11 skips step S1205.

  The CPU 11 replaces the priorities of the partial areas according to the value of the counter for each partial area (step S1206), and determines whether to end the process (step S1207). If the CPU 11 determines that the process is not completed (step S1207: NO), the CPU 11 returns the process to step S1203 and repeats the above-described process. If the CPU 11 determines that the process ends (step S1207: YES), the CPU 11 ends the process.

  Also, statistical data on the success or failure of gripping for each partial area can be created similarly by using a success / failure counter. FIG. 13 is a flowchart showing a procedure of priority determination processing of a partial region with a high gripping success rate of the CPU 11 of the image processing apparatus 1 according to the second embodiment of the present invention.

  As shown in FIG. 13, the CPU 11 of the image processing apparatus 1 initializes a success / failure counter for each partial area (step S1301), and performs temporary setting of the priority of partial areas (step S1302). The CPU 11 executes the processing shown in FIG. 9 (step S1303), and outputs gripping data set in the workpiece model of the workpiece W detected as a gripping target in the selected partial region and partial region to the robot controller 4 ( Step S1304).

  The CPU 11 acquires, from the robot controller 4, gripping success / failure data regarding success or failure of gripping with gripping data set in the workpiece model of the workpiece W detected as a gripping target in the selected partial region and partial region (step S1305) It is determined whether or not the gripping has succeeded (step S1306). If the CPU 11 determines that the gripping is successful (step S1306: YES), the CPU 11 updates the success / failure counter of the selected partial area (step S1307).

  If the CPU 11 determines that gripping has failed (step S1306: NO), the CPU 11 determines whether to end the process (step S1308). If the CPU 11 determines that the process is not completed (step S1308: NO), the CPU 11 returns the process to step S1303 and repeats the above-described process. If the CPU 11 determines that the process is to be ended (step S1308: YES), the CPU 11 ends the process.

  As described above, according to the second embodiment, it is possible to use the gripping data set in the partial region where the success rate of gripping is high and the workpiece model of the workpiece W detected as the gripping target in the partial region. The success rate in which the gripping unit 50 grips the work can be increased, and the operation rate of the picking device can be increased, and the tact time of the work can be improved.

  The present invention is not limited to the above-described embodiment, and various changes, improvements, and the like can be made within the scope of the present invention. For example, only statistical data may be stored in another external computer or in a cloud environment.

  FIG. 14 is a flowchart showing the procedure of processing of the picking system using the image processing apparatus 1 according to the embodiment of the present invention. FIG. 14 is different from the above-described embodiment in that statistical data is stored in the statistical data external storage device 7.

  In FIG. 14, the image processing apparatus 1 selects grip data by the above-described processing (step S1411), and outputs the grip data to the robot controller 4 (step S1412).

  The robot controller 4 receives an input of grip data from the image processing apparatus 1 (step S1441), and controls the operation of the manipulator 5 and the grip unit 50 (step S1442). Then, the processing described above is executed to judge the success or failure of gripping (step S1443), and the selected gripping data and the gripping success or failure data regarding the success or failure of gripping are output to the statistical data external storage device 7 (step S1444).

  The statistical data external storage device 7 receives the input of the gripping data and the gripping success / failure data from the robot controller 4 (step S1471), and updates the success / failure counter associated with the gripping data or the partial area (step S1472). Based on the updated success / failure counter, the priority of the grip data or the partial area is updated (step S1473), and the updated priority is output to the image processing apparatus 1 (step S1474).

  The image processing device 1 receives the input of the priority from the statistical data external storage device 7 (step S1413), and applies it to the next picking operation. This relieves the positional constraints of storing statistical data, and facilitates the handling as big data.

  In the embodiment described above, the case where one image processing apparatus 1 is provided is described, but the invention is not particularly limited to one, and a plurality of image processing apparatuses 1 may be provided. In this case, since the same processing is performed by a plurality of image processing apparatuses 1, common statistical data may be used as the statistical data.

  For example, as the statistical data shared by the plurality of image processing apparatuses 1, the storage data 13 of one or more of the image processing apparatuses 1 is used as statistical data as holding data and holding success / failure results obtained by the plurality of image processing apparatuses 1. Remember. The priority order is specified based on the stored statistical data, and is used in a plurality of image processing apparatuses 1. Of course, it is needless to say that a statistical data shared storage device for collecting and storing only statistical data may be provided outside.

  In addition, gripping data and gripping success / failure results obtained by each image processing apparatus 1 are stored separately and independently in each storage device 13, and only a plurality of priorities obtained based on each statistical data are stored. The image processing apparatus 1 may be used respectively.

1 Image Processing Device 2 Imaging Unit 4 Robot Controller 5 Manipulator 11 CPU
50 gripping parts W work

Claims (12)

In an image processing apparatus connected to a robot controller for controlling a picking operation for gripping a bulk-stacked workpiece with a gripper of a manipulator and moving the griped workpiece to a predetermined position,
A sensing unit for three-dimensionally measuring a work space in which workpieces stacked in bulk are mounted and acquiring three-dimensional measurement data;
A work model registration unit for registering a plurality of work models stored in advance;
A grasping data setting unit for setting grasping data including a grasped position and an attitude of the grasping unit of the manipulator grasped at the position as relative position data with respect to a model coordinate system of the work model; ,
A statistical data storage unit for storing statistical data on success or failure of gripping of a work for each of the gripping data;
A search processing execution unit that executes a three-dimensional search process that specifies the position and orientation of a work using a plurality of registered work models on three-dimensional measurement data acquired by three-dimensional measurement;
An interference determination unit that determines whether a workpiece whose detection score output as a result of the three-dimensional search processing is larger than a predetermined threshold interferes with another workpiece;
An image processing apparatus comprising: a grip data selection unit configured to select grip data based on a gripping success rate in the statistical data among grip data determined not to interfere with another work. The sensing unit is
A work imaging unit for imaging a work in the work space to obtain image data;
And a three-dimensional data extraction unit for extracting, as three-dimensional measurement data, three-dimensional data of a work in the work space to be subjected to the three-dimensional search processing based on image data acquired by imaging. The image processing apparatus according to claim 1, wherein The statistical data storage unit stores statistical data regarding success or failure of gripping of the workpiece for each partial area in which the workpiece is detected,
The gripping data selection unit is configured to select gripping data corresponding to a work existing in a partial region having the highest success rate of gripping among the statistical data, among the gripping data determined to not cause interference in the interference determining unit. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized in that: The statistical data storage unit stores the statistical data including the number of times of selection of grip data,
The image processing apparatus according to any one of claims 1 to 3, wherein the interference determination unit performs the interference determination sequentially from grip data having a large number of selections in the statistical data. An image processing method that can be executed by an image processing apparatus connected to a robot controller for controlling a picking operation for gripping a bulk-stacked workpiece with a gripper of a manipulator and moving the griped workpiece to a predetermined position In
The image processing apparatus is
The first step of obtaining three-dimensional measurement data by three-dimensionally measuring the work space in which workpieces stacked in bulk are mounted.
A second step of registering a plurality of work models stored in advance;
A third step of setting, as relative position data, gripping data composed of a gripping position and an attitude of the gripping portion of the manipulator gripping at the position with respect to a model coordinate system of the workpiece model; ,
A fourth step of storing statistical data on success or failure of gripping of the workpiece for each of the gripping data;
A fifth step of executing three-dimensional search processing for specifying the position and orientation of a work using a plurality of registered work models on three-dimensional measurement data acquired by three-dimensional measurement;
A sixth step of determining whether or not a workpiece whose detection score output as a result of the three-dimensional search processing is larger than a predetermined threshold interferes with another workpiece;
And a seventh step of selecting gripping data based on a gripping success rate in the statistical data from among gripping data determined not to interfere with another workpiece. The first step is
An eighth step of imaging the work in the work space to acquire image data;
And a ninth step of extracting, as three-dimensional measurement data, three-dimensional data of a work in the work space to be subjected to the three-dimensional search processing based on image data acquired by imaging. The image processing method according to claim 5. The fourth step stores statistical data on success or failure of gripping of the workpiece for each partial area where the workpiece is detected,
In the seventh step, among the grip data determined not to interfere in the sixth step, the grip data corresponding to the workpiece present in the partial region having the highest success rate of gripping in the statistical data is selected. The image processing method according to claim 5 or 6, characterized in that The fourth step stores the statistical data including the number of times of selection of grip data,
The image processing method according to any one of claims 5 to 7, wherein, in the sixth step, the interference determination is performed sequentially from grip data having a large number of selections in the statistical data. A computer program that can be executed by an image processing apparatus connected to a robot controller for controlling a picking operation for gripping a bulk-stacked workpiece with a gripper of a manipulator and moving the griped workpiece to a predetermined position ,
The image processing apparatus
Sensing means for acquiring three-dimensional measurement data by three-dimensionally measuring the work space in which workpieces loaded in bulk are mounted
Work model registration means for registering a plurality of work models stored in advance,
Grasping data setting means for setting grasping data composed of a position to be grasped and an attitude of the grasping portion of the manipulator grasped at the position as relative position data with respect to a model coordinate system of the workpiece model;
Statistical data storage means for storing statistical data on success or failure of gripping of a work for each of the gripping data,
Search processing execution means for executing three-dimensional search processing for specifying the position and posture of a work using a plurality of registered work models on three-dimensional measurement data acquired by three-dimensional measurement;
Interference determination means for determining whether or not a workpiece whose detection score output as a result of the three-dimensional search processing is larger than a predetermined threshold interferes with another workpiece, and grip data determined not to interfere with the other workpiece And a computer program that causes a computer to function as a gripping data selection unit that selects gripping data based on a success rate of gripping in the statistical data. The sensing means is
Work imaging means for imaging a workpiece in the work space to acquire image data, and three-dimensional work in the work space to be subjected to the three-dimensional search processing based on the image data acquired by imaging The computer program according to claim 9, characterized in that it functions as three-dimensional data extraction means for extracting data as three-dimensional measurement data. The statistical data storage means functions as a means for storing statistical data regarding success or failure of gripping of the workpiece for each partial area where the workpiece is detected,
The gripping data selecting means is a means for selecting the gripping data corresponding to the work existing in the partial region having the highest gripping success rate in the statistical data among the gripping data determined to not cause interference by the interference determining means. The computer program according to claim 9 or 10, characterized in that it functions. The statistical data storage means functions as means for storing the statistical data including the number of times of selection of gripping data,
The computer program according to any one of claims 9 to 11, wherein the interference determination means is caused to function as means for performing an interference determination in order from the grip data having a large number of selections in the statistical data.