JP2012030320A - Work system, working robot controller, and work program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a work system that efficiently recognizes components for work even if a position or the like of the component on each tray is changed.SOLUTION: The work system recognizes a plurality of trays 4 movably disposed, on an image with low spatial resolution in a wide field of view, and thereafter carries out image processing with high spatial resolution in a narrow field of view, thereby identifying the components 3 mounted on the trays 4 and recognizing arrangement of the trays 4. During the tray recognition step, there is no need to process a large amount of images due to the low spatial resolution, and during the component recognition step, the large amount of image processing is not required due to the narrow field of view. Therefore, three-dimensional positions and postures of the components 3 to be mounted on the trays 4 can be efficiently recognized. The two-step image recognition enables a work robot 2 to efficiently perform work.

Description

  The present invention relates to a work system, a work robot control device, and a work program, and more particularly, to a work robot that performs a work operation using parts placed on a plurality of trays, and control thereof.

  In a system for producing a product using a work robot (for example, line production, cell production, etc., hereinafter a work system), a work instruction is appropriately given to the work robot, and the work robot produces a product according to the work instruction.

  For example, in Patent Document 1, object models of work objects and work models corresponding to the work objects are stored in advance and collated with an object model that stores work objects recognized by the robot. A work model is selected and the robot is made to work.

  Moreover, in patent document 2, when recognizing a work target, a mark is imaged in advance, and the three-dimensional position of the work target is corrected and adjusted. In this way, the work is performed while grasping the three-dimensional position of the work object.

  In addition, as in Patent Document 3, when recognizing a work object, each of the randomly stacked work objects is taken out one by one and placed in the work area, and its three-dimensional position / orientation is recognized to simplify image analysis. Some of them are Also, as disclosed in Patent Document 4, when a work is performed by a plurality of work robots, there is a technique in which a work plan between the work robots is changed in accordance with work reliability to improve work efficiency.

JP 2008-68348 A JP 2010-64198 A JP-A-9-239682 Japanese Patent Laid-Open No. 10-44074

  For the operation of the work robot in the work system, for example, the three-dimensional position of the work target needs to be appropriately taught in accordance with the environment in the factory where the system is placed and the individual products to be produced. Therefore, the operation needs to be controlled appropriately.

  On the other hand, supply of parts necessary for product production is performed by placing the parts in a predetermined three-dimensional position and placing a tray or the like on which the parts are placed at a predetermined position. Since the work robot is previously taught about the three-dimensional position and orientation of the parts, the work robot can perform operations such as gripping the parts and combining them.

  Even after the supplied parts are combined, it is possible to produce a product by further combining the parts by previously teaching the work robot where to place the intermediate assembly created by the combination.

  In such a work system, it is necessary that the three-dimensional position / orientation, etc., of the parts and the intermediate assembly obtained by combining them be constant. When the value exceeds the allowable range, it is impossible to grasp the work target, and there is a problem that the entire system is temporarily stopped and the restoration work must be performed manually. The tray on which the component is placed is also required to be accurately positioned at a predetermined position so that the work robot can recognize it, and the three-dimensional position / posture of the tray has changed due to the supplementation of the component. In some cases, it was necessary to perform accurate positioning each time and teach the work robot.

  Further, when trying to recognize the three-dimensional position / orientation of a part without positioning, the entire work area is imaged and the part is found in the image, and the three-dimensional position of the part (three-dimensional) It is necessary to recognize the position) and the three-dimensional position / posture by image processing. In doing so, when imaging and image processing are performed with a spatially high resolution over the entire work area in order to improve recognition accuracy, there is a problem that it takes a long time to complete recognition.

  The present invention has been made to solve the above-described problems, and even if the three-dimensional position / orientation of a part is changed, the operation can be performed by efficiently recognizing the three-dimensional position / orientation of the part. An object is to provide a working robot system and related technology.

  The work system according to the first aspect of the present invention is a work system in which a part to be worked is taken out from a plurality of trays movably arranged in a predetermined work area, and a predetermined work is performed using the part. A plurality of trays present in the work area are collectively collected with a relatively wide field of view, corresponding storage means for storing tray correspondence data indicating the correspondence between each tray and the type of each component placed on the tray. Based on the imaging result of the first imaging mode in the first imaging mode by the imaging unit, and the imaging unit having the first imaging mode for imaging the second tray and the second imaging mode for individually imaging the plurality of trays with a relatively narrow field of view. First recognition means for performing first image processing with a relatively low spatial resolution, identifying each tray and specifying its arrangement position, and imaging by the imaging unit according to the specified arrangement position of each tray Direction And changing the types of components on each tray based on the tray correspondence data, and imaging control means for causing the imaging unit to image the components placed on the tray for each tray in the second imaging mode. On the other hand, the second image processing with a relatively high spatial resolution is performed based on the imaging result of each component in the second imaging mode, so that the second three-dimensional position / posture of each component on each tray is recognized. It comprises a recognition means, and a work robot that accesses each part on each tray for which the three-dimensional position / orientation is specified in a predetermined order and performs a predetermined work using each part.

  The invention according to claim 2 is the work system according to claim 1, wherein the predetermined work is interrupted during execution to interrupt the predetermined work, and the three-dimensional position of each part on each tray It is further characterized by further comprising an interrupt control means for restarting the predetermined work after the posture recognition is executed again.

  A third aspect of the present invention is the work system according to the first or second aspect, wherein the imaging unit is attached to the work robot.

  Invention of Claim 4 is a work system in any one of Claims 1-3, Comprising: The said imaging part, the 1st imaging part which images in the said 1st imaging mode, an imaging position, and an imaging direction Is variable and includes a second imaging unit that performs imaging in the second imaging mode.

  According to a fifth aspect of the present invention, in the work system according to the fourth aspect, the work robot includes a movable arm portion connected to one end of the hand portion that holds the component, and the second imaging unit. Is attached to the arm portion.

  A sixth aspect of the present invention is the work system according to any one of the first to fifth aspects, wherein the correspondence storage unit further stores work data for instructing work contents preset for each of the parts. The work robot performs work on the part based on the specified part and the work data.

  The invention according to claim 7 is the work system according to any one of claims 1 to 6, wherein the correspondence storage means further stores part shape data indicating a three-dimensional shape for each part, and The two recognizing means recognizes the three-dimensional position / posture of the part based on the specified part and the part shape data.

  The invention according to claim 8 is the work system according to any one of claims 1 to 7, wherein the correspondence storage means further stores tray shape data indicating a three-dimensional shape for each tray, and 1 recognition means recognizes the arrangement position of the tray based on the tray image and the tray shape data imaged in the first imaging mode.

  A ninth aspect of the present invention is the work system according to any one of the first to eighth aspects, wherein at least some of the plurality of trays can mount a plurality of types of components. And

  The work robot control device according to claim 10 is a work for taking out a part to be worked from a plurality of trays movably arranged in a predetermined work area and performing a predetermined work using the parts. A robot control apparatus, wherein the plurality of trays existing in the work area are collectively imaged with a relatively wide field of view, and the plurality of trays are individually imaged with a relatively narrow field of view. An image pickup unit having a second image pickup mode is attached to the work robot, and the control device stores tray correspondence data indicating a correspondence relationship between each tray and the type of each component placed on the tray. A first image processing is performed to identify each tray and to specify its arrangement position based on the storage means and the first image processing with a relatively low spatial resolution based on the imaging result in the first imaging mode by the imaging unit. The imaging unit causes the imaging unit to capture an image of the component placed on the tray for each tray in the second imaging mode while changing the imaging direction of the imaging unit according to the identification unit and the specified arrangement position of each tray. The second image processing with a relatively high spatial resolution is performed based on the imaging result of each component in the second imaging mode while limiting the types of components on each tray based on the control means and the tray correspondence data. Second recognition means for recognizing the three-dimensional position / orientation of each component on each tray, and an operation command for taking out a component on each tray with the specified three-dimensional position / orientation and performing a predetermined operation. It is provided with the operation command means given to a work robot.

  The invention of claim 11 is a program invention corresponding to the invention of claim 10.

  According to the first to eleventh aspects of the present invention, the plurality of trays can be moved by recognizing the identification of the plurality of trays and their arrangement positions and referring to the correspondence between the tray and the components placed on the tray. Even when the three-dimensional position / posture of the mounted component changes, the component's three-dimensional position / posture can be efficiently grasped based on the recognized tray placement position. Can be recognized, and the work operation using the parts can be performed by the work robot.

  In the first image processing in which the trays are collectively imaged to recognize each tray, the second image processing has a wide field of view and low spatial resolution, and the trays are individually imaged to recognize the components therein. However, since it has a narrow field of view and high spatial resolution, excessive data processing becomes unnecessary. Compared with the case where one-step image processing is performed with a wide field of view and high spatial resolution, the two-step processing of the present invention reduces the amount of data and thus reduces the processing time and increases the efficiency.

  In particular, according to the invention of claim 2, since the position of the component is re-recognized by the interruption process when the status of the tray or the component on the tray changes, in addition to the loading and unloading of the tray and the component, an unexpected misalignment In such a case, it is possible to prevent the erroneous operation from being continued using the old position data.

  In particular, according to the fourth and fifth aspects of the present invention, the first imaging unit for imaging the tray and the second imaging unit for imaging the component are provided, and the second imaging unit is provided on the arm portion of the work robot, thereby imaging. Since the roles of the objects can be shared, and the second imaging unit can move according to the operation of the arm, it is possible to efficiently image the parts.

  In particular, according to the invention of claim 9, by placing a plurality of types of parts on one tray, for example, it is possible to assign parts to the tray for each process step and visually check the progress of the process. Therefore, it is suitable for adjustment of timing such as replenishment of parts.

It is a notional top view showing a work system provided with a work robot concerning a 1st embodiment. It is a figure which shows the 1st specific example of the working robot concerning 1st Embodiment. It is a figure which shows the 2nd specific example of the working robot concerning 1st Embodiment. It is a functional block diagram of the work system concerning a 1st embodiment. It is a figure which shows operation | movement of the work system concerning 1st Embodiment. It is a figure which shows operation | movement of the work system concerning 1st Embodiment. It is a figure which shows operation | movement of the work system concerning 1st Embodiment. It is a figure which shows operation | movement of the work system concerning 1st Embodiment. It is a figure which shows operation | movement of the work system concerning 1st Embodiment. It is a figure which shows operation | movement of the work system concerning 1st Embodiment. It is a figure which shows operation | movement of the work system concerning 1st Embodiment. It is a figure which shows operation | movement of the work system concerning 1st Embodiment. It is a figure which shows the example of tray identification data TI and tray corresponding | compatible data TC. It is a figure which shows the example of the identification marker of a tray. It is a flowchart which shows the basic operation | movement of 1st Embodiment. It is a flowchart which shows the interruption operation | movement of 1st Embodiment.

<A. System configuration>
FIG. 1 is a plan view conceptually showing the configuration of the work system WS according to the first embodiment of the present invention. As shown in FIG. 1, the work system WS includes a plurality of trays 4 that are movably distributed and horizontally arranged in the work area 1, and a work robot 2 that performs work operations in the work area 1. In addition, a plurality of parts 3 used for part assembly work operations are distributed and placed on the respective trays 4 according to a predetermined rule.

  For example, as shown in FIG. 2 described later, the plurality of trays 4 are supported by a plurality of legs on the top plate portion on which the component 3 is placed, and the trays 4 move on the floor surface. A wheel or the like may be provided at the lower end of each leg so as to be possible. As will be described later, the correspondence between each tray 4 and the component 3 placed thereon is determined in advance, but the position and orientation of each component 3 on each tray 4 are arbitrary. That is, the component 3 is placed on the tray 4 in a so-called “separated state”, and it is not necessary to position the component 3 on the tray 4 with high accuracy. Further, the arrangement of the plurality of trays 4 is not limited to the case of FIG. 1, and various arrangement variations are possible. Furthermore, the component 3 placed on the tray 4 can be appropriately replenished from the outside as the work robot 2 performs the work operation.

  In FIG. 1, the components 3 placed on each tray 4 are a single type, but several types of components may be mixed in one tray 4. Further, the total number of trays is not fixed, but can be changed according to the work contents.

  The work robot 2 operates using the component 3 placed on the tray 4. For example, a plurality of parts 3 are respectively taken out from the tray 4 and combined with each other to generate a product or an intermediate assembly thereof, and the generated assembly is placed at a predetermined location in the work area 1. The place to be placed may be a dedicated place, or may be an empty tray 4 or an empty part of a tray with few parts 3 placed at that time.

  In FIG. 1, a predetermined table is provided as an assembly area 11 in which the work robot 2 performs assembly work. In this assembly area 11, for example, the first part is placed thereon, and is gripped and fixed by one hand portion H (see FIGS. 2 and 3) of the work robot 2, and the second hand portion H is the second portion. It is used as an application for assembling a part to a first part. In the case of work that can be assembled on both hands H (that is, in the air) of the work robot 2 without placing the first part, the assembly area 11 may not be particularly provided. If the assembly area 11 is not provided, an empty space in the work area 1 may be found by the overall recognition work described later, and the work operation may be performed there.

  When a table or the like is used as the assembly area 11, the posture of the component 3 can be changed as appropriate in the assembly area 11 so that the assembly work can be easily performed.

  FIG. 2 is a diagram showing the work robot 2 according to the first embodiment of the present invention. In FIG. 1, the work robot 2 is viewed from above.

  As shown in FIG. 2, the work robot 2 includes a whole camera 5 and a stereo camera 6 (6A, 6B) as an imaging unit. The whole camera 5 and the stereo camera 6 are digital cameras, and each pixel is a pixel. Image information is output as a digital signal sequence.

  As an example in FIG. 2, the whole camera 5 is attached to the head of the work robot 2, and the stereo camera 6 (6 </ b> A, 6 </ b> B) is attached to each of the two arms 7 (7 </ b> A, 7 </ b> B) of the work robot 2. . The robot head to which the whole camera 5 is attached is movably connected to the casing (base) of the work robot 2 via the neck, and the whole camera 5 has a horizontal rotation and vertical expansion and contraction. The viewing direction (imaging direction) can be changed. The attachment position of the stereo camera 6 is not limited to the arm portion 7 of the work robot 2 and may be attached to the hand portion H. When the stereo camera 6 is sufficiently small with respect to the finger structure of the hand unit H, it may be attached to the finger of the hand unit H. When attaching to the hand part H, if the hand part H has a part corresponding to, for example, the back of a human hand, it may be attached to that part. By doing in this way, there exists an effect that the stereo camera 6 can be brought closer to an imaging target object.

  Each of the pair of stereo cameras 6A and 6B can individually change the imaging position and the visual field direction (imaging direction), but this is because of the arm portions 7A and 7B to which the stereo cameras 6A and 6B are fixed. It is the movement that occurs with the movement. By acquiring images from the stereo cameras 6A and 6B, it is possible to obtain three-dimensional position information of the object to be imaged. One stereo camera 6 may be used as long as an arbitrary part in the work area 1 can be imaged.

  The whole camera 5 as the first imaging unit is a wide field of view (wide field of view) camera for ensuring a wide field of view and recognizing the arrangement positions of the plurality of trays 4 in the work area 1. The stereo camera 6 as the second imaging unit has a narrower field of view than the whole camera 5, but moves in accordance with the operation of the arm unit 7 provided in the work robot 2, and the three-dimensional of the component 3 placed on the tray 4. It is a narrow field of view (narrow field of view) camera for recognizing position and orientation. Here, the three-dimensional position / posture is defined as a set of coordinate values including at least one (typically both) of the spatial position and the spatial posture of the object. The condition that the field of view of the entire camera 5 is wider than the field of view of the stereo camera 6 is hereinafter referred to as “field condition”.

  The number of imaging pixels of the entire camera 5 and the stereo camera 6 may be the same or different from each other. However, the whole camera 5 captures an imaging result (wide field image) when the work area 1 is imaged with a relatively wide field of view, and an imaging result (narrow field) when each tray 4 is imaged with a relatively narrow field of view by the stereo camera 6. And the number of image sensors in each camera so that the spatial resolution of the stereo camera 6 is higher (the smaller distance can be distinguished) than the spatial resolution of the entire camera 5. The optical system is determined. Such a condition that “the spatial resolution of the wide-field image is made smaller than the spatial resolution of the narrow-field image” is hereinafter referred to as “spatial resolution condition”. The magnitude of the distance in the real space corresponding to one pixel in the image is shown, and “high spatial resolution” is defined in the sense that a short distance in the real space can be distinguished on the pixel array.

  For example, in the case where one pixel in the captured image corresponds to 3 mm in real space and the case in which one pixel corresponds to 1 mm in real space, the latter has higher spatial resolution.

  Even when the number of image pickup devices of the whole camera 5 and the stereo camera 6 is the same, the whole camera 5 picks up an object from a relatively long distance, so the spatial resolution is low, and the stereo camera 6 is a target from a relatively short distance. Since an object is imaged, its spatial resolution is high, so that the spatial resolution condition is satisfied. This is the reason why the number of imaging pixels of each camera may be the same. Further, if the number of imaging pixels of the stereo camera 6 is made larger than the number of imaging pixels of the whole camera 5, the above spatial resolution condition is automatically satisfied due to the difference in perspective of the imaging object.

  Furthermore, if the whole camera 5 captures the image itself using an image sensor having a large number of pixels, and if the number of pixels to be calculated is limited by performing pixel thinning or the like at a later stage of image recognition, substantially “ This is equivalent to performing processing at “low spatial resolution”, and the time required for image recognition is shorter than when processing is performed at high spatial resolution throughout. Therefore, the spatial resolution condition only needs to be achieved at the stage of recognition of an object after imaging, and is not necessarily satisfied from the first imaging stage.

  The whole camera 5 is used for identifying the tray 4 and recognizing the arrangement position, and the stereo camera 6 is used for recognizing the three-dimensional position / posture of the component 3 in each tray 4.

  Among these, image recognition using the whole camera 5 is considered to be “wide field of view” because it is necessary to capture a wide range at a time, but it is sufficient if recognition at the tray size is possible, so low spatial resolution and Is done.

  On the other hand, image recognition using the stereo camera 6 needs to be performed with a high spatial resolution in order to recognize the component 3 in each tray 4, but one tray 4 is within the field of view (view angle) at each time point. Since it is enough to enter, keep a narrow field of view.

  Thus, by sharing the respective roles between the entire camera 5 and the stereo camera 6, it is possible to reduce the overall time required for the entire image recognition and improve the work efficiency.

  As a comparative example, using a single stereo camera with a wide field of view and high spatial resolution, the entire work area 1 is imaged all at once, and not only the tray 4 is recognized but also the components in each tray 4 using the imaging result. Consider the case where even 3 recognition is performed. In this case, even the space where the tray 4 is not arranged in the work area 1 is imaged with high spatial resolution and image processing is performed, so that the amount of each calculation becomes large and the processing time is also long. Become. On the other hand, in the system WS of the present embodiment, the image processing is simplified for areas that do not require fine image processing, so that the amount of calculation and processing time can be reduced without reducing the accuracy of image recognition. is there.

  However, it is also possible to provide these two-stage imaging and image recognition functions with a single camera (not shown) provided in any part of the work robot 2. That is, the image recognition of the components on each tray 4 is performed by sequentially changing the shooting direction of the stereo camera 6 in the direction of each tray 4 specified by the analysis result of the imaging result of the entire camera 5. Therefore, it is not necessary to simultaneously capture images with the whole camera 5 and the stereo camera 6. Therefore, if one multifocal camera or zoom camera is used to perform wide-field imaging at the whole imaging stage and narrow-field imaging at the individual imaging stage of each tray 4, substantially the whole camera 5 and the stereo camera can be used. The camera 6 can be used to satisfy both the viewing condition and the spatial resolution condition. However, in that case, in response to the need to move the field of view with the function of the stereo camera 6, the single shared camera has a movable mechanism in itself so that the imaging direction can be changed. It is provided or attached to the movable part of the work robot as in FIG.

  In FIG. 2, the arm portion 7 (7A, 7B) is a double arm, and the working robot 2 is shown in which the left and right arm portions 7A, 7B are interlocked to combine the components 3, but one component 3 is fixed. A working robot that operates with the arm portion of one arm by preparing a jig such as the above may be used. The number of arms may be more than two, and the shape is not limited to that shown in the figure.

  Further, as shown in FIG. 3, the whole camera 5 may not be provided in the work robot 20 but may be provided in another place in the work system WS so that the entire work area 1 can be imaged. For example, the camera may be movable on a rail installed on the work system WS. Although only the whole camera 5 is separated from the work robot 20 in FIG. 3, the stereo camera 6 as the second imaging unit may be provided as such. In addition, in FIG. 3, although the assembly area as shown in FIG. 2 is not provided, you may provide.

  FIG. 4 is a functional block diagram of the work system WS according to the embodiment of the present invention, which is drawn together with the relationship of hardware elements.

  As shown in FIG. 4, the work system WS includes a control computer COM that is communicably connected to the work robot 2 and the imaging unit 12. The imaging unit 12 corresponds to a combination of a first imaging unit (entire camera 5) and a second imaging unit (stereo camera 6). The work robot 2 is, for example, an articulated robot, and as illustrated in FIGS. 2 and 3, a plurality of movable arms (arms) are connected in series, and a workpiece (work object) is held at the tip thereof. Or the hand part H suitable for various operations, such as assembly | attachment of a workpiece | work, is provided. The hand unit H may also include an end effector (robot work tool) such as a screw tightening tool.

  The control computer COM includes a CPU and a storage device. When the control computer COM executes a program stored and installed in advance in the storage device, first control means 8A (tray recognition) as each functional means. The second recognition means 8B (part recognition), the correspondence storage means 9, the imaging control means 10, and the operation command means 200 can be realized by hardware resources and software of the control computer COM.

  The whole camera 5 is used in a first imaging mode in which a plurality of trays 4 that are movably arranged in the work area 1 are collectively imaged. The first recognizing means 8A performs the first image processing based on the imaging result of the entire camera 5 in the first imaging mode, and identifies each tray 4 and specifies its arrangement position.

  In the storage device of the control computer COM, as illustrated in FIG. 13, the correspondence between the tray identification data TI for mutually identifying each tray 4 and the type of each tray 4 and each component 3 placed on it. Tray correspondence data TC indicating the relationship is stored in advance. However, in the example of FIG. 13, these are created as one integrated table.

  Among these, the tray identification data TI is data in which the tray number of each tray 4 is associated with the geometric shape data (tray shape data) of the tray. The tray correspondence data TC is data indicating correspondence between which type of component 3 is placed in which tray 4, and each component scheduled to be placed in that tray. Part numbers and geometric shape data (part shape data) for each part are registered in correspondence with each other. The tray shape data and component shape data are, for example, point cloud data (model point cloud) in which each point has three-dimensional position information.

  The shapes of the trays 4 are not the same, and are formed so as to have features that can be distinguished from each other in appearance. However, a shape that is geometrically similar to the “shape” here is treated as a different shape. Therefore, the identification of the tray 4 can be identified by comparing the position of the feature point of each tray 4, for example, the position of each point group indicating each vertex of the imaged tray 4 with the position of the model point group registered in advance. The position can be specified. Further, for example, as shown in FIG. 14, different shape markers M 1, M 2,... For each tray 4 are attached to the top of the frame FM of the tray 4, so that each of the markers M 1, M 2, etc. The trays 4 can be identified with each other. Note that a cut portion is formed asymmetrically in the frame FM of the tray 4 shown in FIG. 14 for identification of the two-dimensional arrangement direction.

  In addition, the trays can be distinguished from each other by giving different colors to the trays 4 or attaching barcodes with different codes to the trays 4 at predetermined positions. The tray shape data in the table of FIG. 13 is data including information on the appearance characteristics of such a tray.

  This tray identification data TI forms part of the function of the first recognition means 8A. Therefore, the first recognition means is based on information obtained by capturing images of the trays 4 in the work area 1 with the whole camera 5 in a lump or by capturing the work area 1 with a wide field of view divided into several sub-areas. By referring to the tray identification data TI by the 8A, mutual identification and arrangement position specification of each tray 4 can be performed.

  The imaging control unit 10 gives control signals to the imaging unit 12 and the work robot 2, and changes the imaging direction by the stereo camera 6 according to the arrangement position of each tray 4 specified by the first recognition unit 8 </ b> A. Every time, the control is performed such that the component 3 placed on the tray 4 is imaged by the stereo camera 6 in the second imaging mode.

  The second recognizing means 8B searches the tray correspondence data TC using the tray number of the tray 4 being imaged at that time as a key, and the type (part number) of the parts 3 corresponding to the tray 4 and the parts 3 thereof. Refer to the part shape data. Based on this, the type of component such as which component 3 on the tray 4 being imaged is specified, and the three-dimensional position / orientation of each component 3 is determined using a stereo method or the like. Identify. In the case where the component 3 has a unique color, the component shape data may include such color information.

  In the record of each tray 4 of the tray correspondence data TC, only the parts 3 in the planned range to be placed on the tray 4 are listed. Therefore, by identifying the tray 4 by the first recognition unit 8A, The types of the parts 3 on the tray 4 that are captured by the stereo camera 6 are limited. That is, for example, when there are 10 types of parts 3 and one or more of the four types of parts are scheduled to be placed on the first tray, the tray correspondence data TC Only the data for the four types of parts 3 are registered in the record of the first tray. Therefore, when the second recognizing means 8B identifies each component 3, it is sufficient to perform matching between the image obtained by imaging and the shapes of the four types of components 3, and all the ten types of components 3 can be obtained. The processing speed is improved as compared with the comparative example in which matching is performed with the shape.

  Such image processing (second image processing) in the second recognition unit 8B is performed with higher spatial resolution than image processing (first image processing) in the first recognition unit 8A. As a result, the identification of each component 3 and the determination of the three-dimensional position / posture are performed with high accuracy. However, since the types of components 3 to be matched are limited to each tray 4 as described above, such high accuracy is achieved. It is possible to prevent an increase in processing time as a whole while using the image processing in FIG.

  When the identification of each part 3 of each tray 4 and the identification of the three-dimensional position / orientation are completed by the second recognition means 8B, the operation command means 200 according to the work routine program stored in advance in the storage device of the control computer COM. Gives an operation command to the work robot 2. There, the work robot 2 accesses each part 3 on each tray 4 in a predetermined order, and sequentially removes each part 3 from the tray 4 in a predetermined order to perform a predetermined operation such as part assembly. In such an operation, the arrangement information of each component 3 of each tray 4 obtained by the second recognition unit 8B is used.

  In this work routine, work data is also referred to. The work data is data indicating the correspondence of which work operation is preset for which component 3, and is stored in advance in the correspondence storage means 9 of the control computer COM. There may be a plurality of work operations that can be set for one component 3, and in that case, the work robot 2 can select which work operation to select. The work operation contents associated with each other include, for example, the part 3 to be combined, the posture at the time of the work operation, the place to be transferred for combining, the place to transfer the intermediate assembly generated by the combination, and the like.

  Even when an intermediate assembly after assembling several parts 3 is temporarily placed on any tray 4, the recognition result of the second recognition means 8B is used. That is, when the second recognizing unit 8B recognizes the three-dimensional position / orientation of the component 3 in the tray 4, information on the position of the empty space on the tray 4 can be obtained incidentally. This empty space information is stored for each tray 4, and when the intermediate assembly is required to be placed, the empty space information is referred to, so that any tray can be used as the intermediate assembly placement location. 4 empty spaces are available. Further, by recognizing an empty tray on which no part 3 is placed and storing the position of the tray 4, such an empty tray can also be used as a place for placing an intermediate assembly.

  The operation input means 300 shown in FIG. 4 is provided in association with the control computer COM, and is used for data input, operation instruction input, and the like by the operator to the control computer COM.

<B. Operation>
<B-1. Basic operation>
Next, the operation of the work system according to the present embodiment will be described. 5 to 8 are top views of examples of the work system. As shown in FIGS. 5 to 8, the work area 1 includes trays 40, 41, 42, 43, an assembly area 11, and arm portions 7 (7 </ b> A, The work robots 2 having 7B) are respectively arranged. Components 30, 31, and 32 are placed on the trays 40, 41, and 42, respectively.

  FIG. 15 is a flowchart showing the operation of the work robot system WS. Of these, the first routine R1 is a preparation routine for recognizing the arrangement status of each tray and parts, and the second routine R2 is a result of such recognition. This is an actual movable routine for causing the work robot 2 to work based on the above.

  First, the first imaging mode is activated, and the trays 40 to 43 distributed over the entire work area 1 are imaged with a wide field of view by the entire camera 5 (see FIGS. 2 and 3) attached to the work robot 2 as shown in FIG. (FIG. 15: Step S1), the first recognition means 8A (see FIG. 4) identifies each of the trays 40 to 43 while referring to the tray identification data TI (FIG. 13). The arrangement position is recognized (step S2). Here, the three-dimensional postures of the trays 40 to 43 may also be recognized. In addition, since the height of each tray 40 to 43 from the floor is usually specified in advance, the two-dimensional position (or two-dimensional position / posture) of each tray 40 to 43 at that height is usually determined. It may be recognition.

  When the entire camera 5 cannot capture all of the trays 40 to 43 in the work area 1 at a time, the work area 1 is conceptually divided into a plurality of sub areas, and each sub area has a tray group existing therein. You may image every. That is, in this step S1, the work area 1 is collectively imaged with a wide field of view as compared to the second imaging mode described later.

  As described above, the trays 40 to 43 can be identified by providing a difference in the shapes of the trays or by applying different colors, and the identification can be easily performed by two-dimensional image analysis or the like. Also, the arrangement positions of the trays 40 to 43 can be matched using the model point group (tray shape data in the tray identification data TI) stored in the storage device of the control computer COM. From image analysis or the like, it is also possible to easily recognize the area where the trays 40 to 43 are placed with an accuracy that can specify the area without the comparison with the model point group.

  Next, a part of the trays 40 to 43 is selected (step S3). The trays to be selected may be one by one, or a small number of spatially adjacent trays. Here, the size and arrangement position of each recognized tray and the visual field of the stereo camera 6 are compared, and when only one tray falls within the visual field of the stereo camera 6, one tray is selected one by one. A plurality of trays can be selected when the trays can be accommodated. As a selection order of the trays 40 to 43, for example, a rule for clockwise rotation around the central vertical axis of the work robot 2 can be determined. Here, it is assumed that two trays 40 and 41 are selected.

  In general, imaging by the first imaging unit (first imaging mode) is “collective”, whereas imaging by the second imaging unit (second imaging mode) is “individual”. One having a relatively large number of trays acquired in one imaging can be defined as “collective (capturing)”, and one having a relatively small number of trays can be defined as “individual (capturing)”.

  As shown in FIG. 6, one arm portion 7 </ b> A of the arm portions 7 is moved above the tray 40 by the control of the imaging control means 10 (see FIG. 4) based on the recognized arrangement positions of the trays 40 and 41. Thus, the imaging direction is changed, and the component 30 placed on the tray 40 is imaged together with the tray 40 by one stereo camera 6A provided in the arm portion 7A. If necessary, the work robot 2 itself is controlled to translate to the tray 40 side. Similarly, the imaging direction is changed by moving the arm portion 7B above the tray 41, and the component 31 placed on the tray 41 is imaged together with the tray 41 by the stereo camera 6B provided in the arm portion 7B. . Since the arrangement positions of the trays 40 and 41 are specified in advance, the stereo cameras 6A and 6B with a narrow field of view are controlled so as to efficiently capture the region where the components 30 and 31 are present, and the components 30 and 31 are efficiently An image can be taken.

  On the other hand, in the second recognition unit 8B, the correspondence relationship between the trays 40 and 41 and the components 30 and 31 placed on the trays 40 and 41 is grasped with reference to the tray correspondence data TC. For example, it is the component 30 that is placed on the tray 40 and the correspondence relationship that either one of the two types of components 31 and 32 is placed on the tray 41 is registered in the tray correspondence data TC. Assume. In this case, the image of the component on the tray 40 is collated with the component shape data for the component 30, and the image is recognized as the component 30, and the three-dimensional position and orientation of the component 30 on the tray 40 are recognized. The Further, the image of the component on the tray 41 is collated with the component shape data of each of the components 31 and 32 to recognize that the image is the component 31, and the three-dimensional position / posture of the component 31 on the tray 41. Is recognized. The result recognized in this way is stored in the storage device of the control computer COM. The above process corresponds to step S5 in FIG.

  In the next step S6, it is checked whether or not there is a tray that has not been recognized yet, and if it remains, the process returns to step S3 to select a new tray. In the illustrated example, the trays 42 and 43 are selected, and steps S4 and S5 are executed as in the previous time. Since no components are placed on the tray 43, it is recognized that the tray 43 is empty, and that fact is stored.

  The above steps S3 to S6 are repeated until all the trays are imaged and the parts are recognized. When the parts are imaged and recognized for all the trays, the process proceeds to step S7.

  In step S7, the types of the parts 30 to 32 placed on the respective trays 40 to 42, the three-dimensional positions, and the like are added to the part variables in the work routine program (work data) stored in the storage device of the control computer COM. By substituting information on posture, the work routine program is realized.

  For specific types of parts placed on a tray, if the operation for that type of part is not set in the work routine program, the handling of that part is skipped and other trays are supported. Recognize the correspondence with work operations only for the parts to be performed. By doing in this way, even when parts irrelevant to the work operation to be performed are arranged in the work area 1, the work operation can be advanced without delay.

  In FIG. 5, a work operation in which the component 30 of the tray 40 and the component 31 of the tray 41 are combined with each other and a work operation for conveying the intermediate assembly generated thereby to the tray 43 are set. (The tray used is indicated by a thick frame). Handling is skipped on the assumption that a different work operation is set for the part 32 or no work operation is set.

  When the above first routine R1 (preparation routine) is completed, the routine proceeds to a second routine R2 (actual movable routine).

  In the second routine R2, the execution of the embodied work routine program is started, and the operation command means 200 gives a drive command signal to the work robot 2, so that the work robot 2 is operated in the order defined by the program. 30 and 31 are accessed, and are sequentially gripped by using the gripping mechanism (hand portion H) of the arm portion 7 and taken out from the trays 40 and 41. In doing so, information on the three-dimensional positions and orientations of the components 30 and 31 that have already been recognized is used. Then, the work robot 2 is controlled so as to transport the parts 30 and 31 to the assembly area 11 according to the work operation content specified by the work routine program (see FIG. 6).

  Next, as shown in FIG. 7, when the parts 30 and 31 necessary for the assembly area 11 are transported to the assembly area 11, the postures of the parts 30 and 31 are appropriately changed for combination, and a combined intermediate assembly 50 is generated. To do. Next, as shown in FIG. 8, the intermediate assembly 50 is transported to the tray 43 according to the specified work operation content.

  For example, parts of various sizes including screws, nuts, etc. are conceivable. By combining various parts generated as intermediate assemblies, it is possible to produce industrial products with complicated processes. It becomes.

  Among the above, the recognition operation (step S2) of the trays 40 to 43 may be performed only when the arrangement positions of the trays 40 to 43 are changed for replenishment of the components 30 to 32, but in this embodiment. In such a work system WS, since the trays 40 to 43 are not fixed, it is conceivable that the arrangement positions of the trays 40 to 43 change during the work operation. Therefore, in such a case, the arrangement positions of the trays 40 to 43 may be re-recognized for each work operation without replenishment of the components 30 to 32 or the like. Further, by providing a sensor (not shown) that can detect that the arrangement positions of the trays 40 to 43 are changed, the time when the arrangement positions of the trays 40 to 43 are changed regardless of the operation operation stage. Thus, it is possible to recognize the arrangement positions of the trays 40 to 43 again. Even in the case where the position of a component is shifted due to an impact or the like during work, the tray or the component can be recognized again by providing an impact sensor and detecting it.

  In addition, by providing a notification device or the like (not shown) for notifying that it is necessary to replenish components when the components in the tray are completely transported, the work operation can proceed without delay.

  FIG. 16 is a flowchart showing an interrupt operation that occurs during the repetition of the assembly work by the work robot 2 in response to these circumstances, and this interrupt routine is also installed in the control computer COM.

Step S11 corresponds to the second routine R2 of FIG. 15, but during its execution: • tray movement, loading or unloading;
-Carrying in parts (replenishment);
-Unloading an assembly that is a work product of the work robot 2, such as the above intermediate assembly;
Etc. occurs, the assembly work is interrupted and stopped in response to detection of the occurrence of the situation (steps S12 and S13). This occurrence detection may be automatic detection by the sensor as described above, and notifies the control computer COM that the operator operates a button or the like using the operation input means 300 of FIG. May be.

  When such an interruption occurs, a routine corresponding to the first routine R1 (preparation routine) in FIG. 15 is executed again, and the robot work program is instantiated (updated) again based on the result (step S14). ). In step S15, the interrupt process is canceled and the work program by the work robot 2 is resumed.

<B-2. Component allocation method: first format>
As a specific method of assigning the components placed on the tray, there is a method of assigning the components separately to the same type of components. That is, it is the same method as the method shown in FIGS.

  In this method, for example, one type of component is assigned to each tray, such as component 30 for tray 40, component 31 for tray 41, component 32 for tray 42, and the like.

  If the tray can be identified, the corresponding part can be specified, so that it is possible to specify shape data to be matched when recognizing the three-dimensional position / orientation of the part. Therefore, selection of shape data or the like is not necessary, and matching efficiency can be improved and overall work efficiency can be improved. Even if there are several types of parts assigned to the tray, it is only necessary to select an appropriate one from the shape data of the several kinds of parts, so that the matching efficiency can be improved and the overall work efficiency can be improved. It becomes possible.

  In addition, the same or different parts are appropriately replenished and reused in a tray that has no parts used for the work operation, so that the work operation can be performed efficiently using the work area.

<B-3. Parts assignment method: second format>
In addition, as a specific method for assigning the components placed on the tray, there is a method for assigning the components separately for each component used in the same process step.

  In the case of this method, as shown in FIG. 9, first, all the parts 30, 31, 33 to 35 are allocated to the tray 44. Each of the trays 41 and 42 is assigned to store an intermediate assembly generated in each subsequent process step. Here, as an example, the second process stage is assigned to the tray 41 and the third process stage is assigned to the tray 42.

  As the first process step, the parts 30 and 31, the parts 33 and 34 are respectively transported to the assembly area 11, and are combined with each other, and the generated intermediate assemblies 50 and 51 are transported to the tray 41.

  Next, as shown in FIGS. 10 and 11, as a second process step, the intermediate assemblies 50 and 51 and the part 35 are transported to the assembly area 11 and combined to generate an intermediate assembly 52. Then, as shown in FIG. 12, the intermediate assembly 52 is conveyed to the tray 42.

  By assigning a tray to each part used in each process step in this way, the progress of the process can be visually confirmed depending on whether or not a part is placed on each tray, and the work system is operating properly. It is possible to determine whether or not the replenishment timing is appropriate.

<C. Effect>
As described above, according to the embodiment of the present invention, the plurality of trays 40, 41, 42, 43 are identified and their arrangement positions are recognized, and are further placed on the trays 40, 41, 42, 43 and the trays. By referring to the correspondence relationship with the parts 30, 31, 32, the three-dimensional positions and postures of the parts 30, 31, 32 to be moved and placed on the trays 40, 41, 42, 43 are changed. Even in such a case, by efficiently grasping the placed parts 30, 31, 32 based on the recognized positions of the trays 40, 41, 42, 43, the three-dimensional positions of the parts 30, 31, 32 The posture can be recognized, and the work robot 2 can perform work operations using the parts 30, 31, and 32.

  Moreover, the whole camera 5 as a 1st imaging part which images the tray 4 and the stereo camera 6 as a 2nd imaging part which images the component 3 are provided, and the stereo camera 6 is provided in the arm part 7 of the work robot 2. Thus, the roles of the imaging target can be divided, and the stereo camera 6 can move according to the operation of the arm portion 7, so that the part 3 can be imaged efficiently.

  In particular, the first image processing based on the imaging result of the whole camera 5 is “wide field of view and low spatial resolution”, and the second image processing based on the imaging result of the stereo camera 6 is “narrow field of view and high spatial resolution”. For this reason, the first image processing can simultaneously recognize a large number of trays with a wide field of view. However, since the spatial resolution is not increased, the time required for the image processing can be shortened. In the second image processing, recognition is performed with a high spatial resolution. However, since only the components on a small number of trays are recognized with a narrow field of view, the time required for the image processing is short here.

  Further, when a plurality of types of parts 30, 31, 33, 34, 35 are placed on one tray 44, for example, the parts 30, 31, 33, 34, 35 are placed on the trays 44, 41, 42 for each process step. Since it is possible to visually check the progress of the process, it is suitable for timing adjustment such as replenishment of parts.

<D. Modification>
If the tray height is fixed and the parts are only flat, the height value of the parts is fixed if the two-dimensional position and orientation of the parts are recognized. The three-dimensional position and orientation of the part can be specified. Accordingly, in the above embodiment, a stereo camera is used as the second imaging unit, but a system using a two-dimensional camera can also be constructed.

  Moreover, in the said embodiment, although each camera which comprises an imaging part is a still camera, a movie camera may be sufficient.

  The present invention can be used for a line production system automated by a work robot, a cell production system, and the like.

DESCRIPTION OF SYMBOLS 1 Work area 2,20 Work robot 3,30-35 Parts 4,40-44 Tray 5 Whole camera (1st imaging part)
6, 6A, 6B Stereo camera (second imaging unit)
7, 7A, 7B Arm portion 8A First recognition means 8B Second recognition means 9 Corresponding storage means 10 Imaging control means 11 Assembly area 12 Imaging section 50 to 52 Intermediate assembly 200 Operation command means 300 Operation input means FM frame M1, M2 Marker H Hand part WS Work robot system

Claims (11)

A work system for taking out a part to be worked from a plurality of trays movably arranged in a predetermined work area and performing a predetermined work using the part,
Corresponding storage means for storing tray correspondence data indicating the correspondence between each tray and the type of each component placed thereon;
A first imaging mode that collectively images the plurality of trays existing in the work area with a relatively wide field of view; and a second imaging mode that individually images the plurality of trays with a relatively narrow field of view. An imaging unit having
First recognition means for performing first image processing with a relatively low spatial resolution based on a result of imaging in the first imaging mode by the imaging unit, identifying each tray and specifying its arrangement position;
An imaging control means for causing the imaging unit to image the component placed on the tray for each tray in the second imaging mode, while changing the imaging direction by the imaging unit according to the specified position of each tray;
By performing the second image processing with a relatively high spatial resolution based on the imaging result of each component in the second imaging mode, while limiting the types of components on each tray based on the tray correspondence data, Second recognition means for recognizing the three-dimensional position / posture of each part on each tray;
A work robot that accesses each part on each tray for which the three-dimensional position / orientation is specified in a predetermined order and performs a predetermined work using each part;
Characterized by comprising,
Work system. The work system according to claim 1,
Interrupt control means for interrupting the predetermined work during execution of the predetermined work, interrupting the predetermined work, re-recognizing the three-dimensional position / posture of each part on each tray, and restarting the predetermined work ,
Further comprising:
Work system. The work system according to claim 1 or 2,
The imaging unit is attached to the work robot,
Work system. The work system according to any one of claims 1 to 3,
The imaging unit
A first imaging unit that performs imaging in the first imaging mode;
A second imaging unit that has a variable imaging position and imaging direction and performs imaging in the second imaging mode;
Characterized by comprising,
Work system. The work system according to claim 4,
The working robot is
A movable arm unit connected to one end of the hand unit holding the component;
With
The second imaging unit is attached to the arm unit or the hand unit,
Work system. A work system according to any one of claims 1 to 5,
The correspondence storage means further stores work data for instructing a work content set in advance for each component,
The working robot is
Based on the identified part and the work data, the work for the part is performed,
Work system. The work system according to any one of claims 1 to 6,
The correspondence storage means further stores component shape data indicating a three-dimensional shape for each component,
The second recognizing means recognizes a three-dimensional position / posture of the part based on the identified part and the part shape data.
Work system. A work system according to any one of claims 1 to 7,
The correspondence storage means further stores tray shape data indicating a three-dimensional shape for each tray,
The first recognizing unit recognizes an arrangement position of the tray based on the tray image captured in the first imaging mode and the tray shape data.
Work system. A work system according to any one of claims 1 to 8,
At least some of the plurality of trays can be loaded with a plurality of types of components.
Work system. A work robot control apparatus that takes out a part to be worked from a plurality of trays movably arranged in a predetermined work area and performs a predetermined work using the part,
A first imaging mode that collectively images the plurality of trays existing in the work area with a relatively wide field of view; and a second imaging mode that individually images the plurality of trays with a relatively narrow field of view. An imaging unit having an attachment to the work robot;
The control device is
Corresponding storage means for storing tray correspondence data indicating the correspondence between each tray and the type of each component placed thereon;
First recognition means for performing first image processing with a relatively low spatial resolution based on a result of imaging in the first imaging mode by the imaging unit, identifying each tray and specifying its arrangement position;
An imaging control means for causing the imaging unit to image the component placed on the tray for each tray in the second imaging mode, while changing the imaging direction by the imaging unit according to the specified position of each tray;
By performing the second image processing with a relatively high spatial resolution based on the imaging result of each component in the second imaging mode, while limiting the types of components on each tray based on the tray correspondence data, Second recognition means for recognizing the three-dimensional position / posture of each part on each tray;
Action command means for giving a work robot an action command for taking out a part on each tray whose three-dimensional position / posture is specified and performing a predetermined work;
Characterized by comprising,
Work robot controller. When the computer is installed and executed, the computer is caused to function as the control device according to claim 10.
Work program.

Images