JP2024051079A - Navigation system, information processing method, and program - Google Patents

Abstract

To present information favorable in producing a device to an assembly worker.SOLUTION: A navigation system (production navigation system 10) includes: a display information generation unit 72 functioning as a display control unit, an analysis control unit, and an output unit; and a storage unit (accumulation unit 73). The display control unit is means for controlling a region to display information for assisting a worker, the region corresponding to a plurality of placement sections (part placement sections 21) where the worker places articles to be picked up. The analysis control unit is means for analyzing a motion of the worker on the basis of video including the worker captured by an imaging unit. The storage unit is means for storing work results based on a result analyzed by the analysis control unit. The output unit is means for outputting information including work time on the basis of the work results stored in the storage unit.SELECTED DRAWING: Figure 16

Description

The present invention relates to a navigation system, an information processing method , and a program.

In recent years, the following trends have been observed in factories that manufacture various types of equipment.
- The number of parts handled per person is large.
-New products are launched one after another in a short period of time.
・Production lines are frequently changed for each product.
-Production using temporary workers has become commonplace.
This tendency is more pronounced in factories that produce a wide variety of equipment in small quantities.
However, as this trend continues, the possibility of assembly omissions, assembly errors, etc. is increasing. Therefore, there is a demand in factories to realize a mechanism to prevent assembly omissions, assembly errors, work omissions, etc.

One mechanism for preventing missed parts in an assembly is the parts kitting system. In parts kitting, a parts distributor arranges parts from a parts storage warehouse into dedicated kit boxes with designated storage locations for each part, ensuring that all parts are supplied to assembly workers without missing any. The parts storage warehouse is designed to inform workers of the location of the parts to be picked by lighting an LED installed at the pick-up port. With this system, assembly workers can prevent missed parts by checking that no parts remain in the kit box after completing assembly.

One example of a device related to kit work is the device described in Patent Document 1. The device described in Patent Document 1 improves workability by transporting parts to the vicinity of a work table equipped with a sensor, and when the sensor detects the removal of a part, moving the next part to the vicinity of the work table (see Patent Document 1, for example).

As a mechanism for preventing assembly errors, there is a system in which work instructions instructing assembly methods are placed around the assembly worker until the worker becomes proficient at the assembly work, and the assembly worker assembles while checking the work instructions.

In addition, one mechanism for preventing work from being overlooked is to have each worker fill out a list of work items called a checklist based on the work they have performed, and then stamp or sign the list to keep a record of the work.

Japanese Patent Application Laid-Open No. 10-29139

However, the above-mentioned conventional mechanism for preventing assembly omissions requires a parts distributor, which increases the number of personnel required.
In addition, the parts storage warehouse is configured to inform the worker of the location of the parts to be collected by lighting an LED provided at the collection port. However, the parts storage warehouse is configured in such a way that the lighting status of the LED is difficult to understand depending on the work environment, and there is a problem that simply lighting the LED does not inform the worker of information such as how many parts to collect and where and how to install the parts.
Furthermore, conventional mechanisms for preventing missed assembly parts had the problem that they required a great deal of preparatory labor when launching a new product, such as arranging parts in the order in which they would be used, and creating the kit boxes used to prevent missed assembly parts.
Furthermore, with the conventional system for preventing assembly omissions, there were many design changes, such as part changes, that occurred during the start-up period, so the kit boxes had to be remade each time, which meant it took a long time to complete manufacturing preparations.

In addition, the mechanism for preventing assembly errors described above has the problem that because the work instructions are stored in paper files, it takes time to search for the desired page and to maintain and manage the work instructions, and there is a demand for a support system to prevent assembly errors.

The aforementioned system for preventing oversight of tasks has the problem that the checklists are paper-based, which requires the time-consuming task of writing the work history on paper and stamping or signing it, and the maintenance of the checklists requires a lot of man-hours, so there is a demand for a support system to prevent oversight of tasks.

The present invention has been made to solve the above-mentioned problems, and has as its main object to provide a navigation system, an information processing method , and a program that present suitable information to an assembly worker when producing a device.

In order to achieve the above-mentioned object, the first invention is a navigation system comprising a display control unit that controls the display of information assisting the worker in areas corresponding to a plurality of placement units on which the worker places items to be collected, an analysis control unit that analyzes the movements of the worker based on a moving image including the worker captured by a photographing unit, a memory unit that stores work results based on the analysis results by the analysis control unit, and an output unit that outputs information including work time based on the work results stored in the memory unit .

In addition, the second invention is an information processing method comprising the steps of: controlling the display of information assisting the worker in areas corresponding to a plurality of mounting sections on which the worker places items to be collected; analyzing the movements of the worker based on a moving image including the worker captured by a photographing section; storing work results based on the results of the analysis in a memory section; and outputting information including work time based on the work results stored in the memory section.
Moreover, the third invention is a program configured to cause a computer to function as a display control unit that controls a computer to display information assisting the worker in areas corresponding to a plurality of placement units on which the worker places items to be collected, an analysis control unit that analyzes the movements of the worker based on a moving image including the worker captured by a photographing unit, a memory unit that stores work results based on the analysis results by the analysis control unit, and an output unit that outputs information including work time based on the work results stored in the memory unit .

The present invention makes it possible to provide assembly workers with information that is appropriate for producing a device.

1 is a diagram showing an overall configuration of a system according to an embodiment; FIG. 1 is a diagram showing a configuration of a production navigation system according to an embodiment. FIG. 2 is a diagram showing the configuration of the production navigation system according to the embodiment. FIG. 1 is a diagram showing a configuration of a main part of a production navigation system according to an embodiment of the present invention; FIG. 2 is a diagram showing the configuration of the main parts of the production navigation system according to the embodiment; 4 is a flowchart showing the overall operation of the production navigation system according to the embodiment. 5 is a flowchart showing an operation during part navigation of the production navigation system according to the embodiment. FIG. 1 is a diagram showing an example of a display screen. FIG. 2 shows an example of a display screen. FIG. 3 shows an example of a display screen. FIG. 4 shows an example of a display screen. FIG. 5 shows an example of a display screen. FIG. 6 shows an example of a display screen. FIG. 11 is a diagram showing a display example during part navigation in the production navigation system according to the embodiment; FIG. 13 is a diagram showing a display example during part navigation in the production navigation system according to the embodiment (2). 1 is a diagram showing an overview of a hand position detection operation in a production navigation system according to an embodiment. FIG. FIG. 11 is a diagram showing an example of image analysis in the production navigation system according to the embodiment. 11 is a diagram showing a procedure for determining a hand position when the hand position is detected in the production navigation system according to the embodiment. FIG. FIG. 1 is a diagram showing an example of settings for image analysis in a production navigation system according to an embodiment. FIG. 2 shows an example of settings for image analysis in the production navigation system according to the embodiment. 4 is a flowchart showing an operation during image analysis of the production navigation system according to the embodiment. 4 is a flowchart showing a data management operation of the production navigation system according to the embodiment. FIG. 4 is a diagram showing an example of a management information file. FIG. 13 is a diagram showing an example of a main screen. FIG. 13 is a diagram showing an example of a production status screen. FIG. 13 is a diagram showing an example of a production volume performance screen. FIG. 13 is a diagram showing an example of a task comparison screen. FIG. 13 is a diagram showing an example of a work performance (by day) screen. FIG. 13 is a diagram showing an example of a work performance (by process) screen. FIG. 13 is a diagram showing an example of a motion analysis screen. FIG. 13 is a diagram showing an example of an end screen.

Below, an embodiment of the present invention (hereinafter referred to as "this embodiment") will be described in detail with reference to the drawings. Note that each figure is merely a schematic illustration to allow a sufficient understanding of the present invention. Therefore, the present invention is not limited to the illustrated examples. In addition, in each figure, common or similar components are given the same reference numerals, and duplicate explanations thereof will be omitted.

[Embodiment]
<Overall system configuration>
The overall configuration of the system according to this embodiment will be described below with reference to Fig. 1. Fig. 1 is a diagram showing the overall configuration of the system according to this embodiment.

As shown in FIG. 1, a production navigation system 10 and a support system 60 are installed in a factory that produces some kind of device.
The production navigation system 10 is a system that supports the assembly work of a product by presenting various information to an assembly worker (hereinafter, sometimes simply referred to as "worker"). The production navigation system 10 can output captured images (still images or videos) of the worker's working status, audio information recorded from the worker's voice, and the like to the support system 60. Note that Fig. 1 shows a control device 11, a management unit 71, a display information creation unit 72, a storage unit 73, and the like, which will be described later.
The support system 60 is a system that supports the production navigation system 10 .

The production navigation system 10 and the support system 60 are connected to each other via wire or wireless communication. The support system 60 may be installed outside the factory.

The support system 60 has a data collection device 61, a data processing device 62, and a support device 63.

The data collection device 61 stores in a storage unit the photographed information, audio information, and the like output from the production navigation system 10. The data collection device 61 also stores in the storage unit various types of information (e.g., work instruction information, work process information, guidance information, projection position information of marks, and the like) and various programs (e.g., a control program for a display mechanism, a control program for image analysis, and the like) and can provide these to the production navigation system 10.
The data processing device 62 analyzes the work contents, etc. based on the photographic information, audio information, etc. stored in the data collecting device 61.
The support device 63 is used for analyzing the work content of the worker, creating work instructions, etc. The support device 63 is configured by a personal computer (PC). The support device 63 is operated by a manager, a support staff member, etc. (hereinafter referred to as "manager, etc.") and is used for creating analysis data (for example, see Figures 19B to 19E described later) etc.

The data collection device 61 has functional means such as an acquisition unit 81, a storage unit 82, an analysis unit 83, and a provision unit 84. The acquisition unit 81 is a functional means for acquiring work data during product assembly work from the production navigation system 10. The work data includes images captured by the production navigation system 10 during product assembly work. The storage unit 82 is a functional means for storing work data during product assembly work. The analysis unit 83 is a functional means for analyzing the progress and progress of work based on the work data during product assembly work. The provision unit 84 is a functional means for providing navigation information to the production navigation system 10, which is displayed as display information in the production navigation system 10. One or more of these functional means may also be provided in the data processing device 62, support device 63, control device 11, etc. of the support system 60.

<Production navigation system configuration>
The configuration of the production navigation system 10 will be described below with reference to Figures 2 and 3. Figures 2 and 3 are diagrams showing the configuration of the production navigation system 10.

As shown in FIG. 2, the production navigation system 10 has a control device 11, a shelf 12, a workbench 13, a photographing unit 14 (14a, 14b), a display mechanism 15, a display 16 (16a, 16b), a microphone 17, a speaker 18, a code input unit 19, and a motion capture camera 20.

The control device 11 controls the operation of the imaging unit 14, display mechanism (display unit) 15, display 16, microphone 17, speaker 18, etc. The control device 11 is connected to these devices so as to be able to communicate with them via wired or wireless communication. The control device 11 is also connected to a data collection device 61 of the support system 60 so as to be able to communicate with them via wired or wireless communication. The control device 11 is configured as a personal computer (PC) and operates based on a control program stored in advance. The control device 11 determines the display information to be displayed on the display mechanism 15 according to the position of a specific part of the worker photographed by the imaging unit 14.

As shown in FIG. 1, the control device 11 has functional means such as a management unit 71, a display information creation unit 72, and a storage unit 73. The management unit 71 is a functional means for managing work results based on work data stored in the storage unit 73. The display information creation unit 72 is a functional means for creating display information representing work results at any time. The storage unit 73 is a functional means for storing work data acquired during product assembly work. One or more of these functional means may also be provided in the data collection device 61, data processing device 62, support device 63, etc. of the support system 60.

Returning to FIG. 2, the shelf 12 has a large number of part placement sections 21 (here, four vertical rows and six horizontal rows, a total of 24 parts) and functions as a parts warehouse. The shelf 12 is provided with a number of partitions 22. Each partition 22 is arranged in a substantially vertical direction, separating the individual part placement sections 21, and is arranged so that its tip protrudes forward from the front surface of the shelf 12. Each part placement section 21 has a tray 40 placed thereon, and the tray 40 stores a single type of part (e.g., screws, brackets, rollers, etc.). The control device 11 manages part placement information indicating which type of part 41 is placed on which part placement section 21, and can identify which type of part 41 has been picked up when a worker picks up a part 41 from the part placement section 21. Note that an assembly tool such as an electric screwdriver may be placed on the part placement section 21.

A command input position can be preset arbitrarily on the shelf 12. In the illustrated example, for example, the position of the component placement unit 21 that is first from the bottom and first from the right is set as the special command input unit 23 used for inputting special commands. Commands include, for example, reset, return (confirm), menu selection, and assembly confirmation. When the worker holds his/her hand over the command input position (in the illustrated example, the special command input unit 23) (i.e., when the worker holds his/her hand out as if to cover an object), the control device 11 determines the display information according to the command corresponding to the input position.

The workbench 13 is a table that a worker uses for assembling parts, etc. The workbench 13 has a flat table surface (upper surface). The table surface is positioned so as to extend horizontally across the entire width of the shelf 12.

The photographing unit 14 (14a, 14b) photographs the working status of the worker. The photographing unit 14 is arranged by the support frame 44 that supports the mirror 32 at a position above the head height of a worker of average height (for example, above the top part placement unit) and in front of the front surface of the shelf 12. The photographing unit 14 is composed of a USB camera. The USB camera 14 as the photographing unit outputs the photographed image to the control device 11. The number of USB cameras 14 is two or more. The two or more USB cameras 14 are arranged separately in the left-right direction. In the illustrated example, two USB cameras 14 are attached to the left and right of the mirror 32. However, the number of USB cameras 14 may be three or more. The USB cameras 14 are not limited to being arranged on the support frame 44, but may be arranged in various places (for example, outside the shelf 12 or inside each part placement unit 21).

The display mechanism 15 is a display unit that displays various information, marks, and the like. In this embodiment, the display mechanism 15 is composed of a projector 31, a mirror 32, and plate-like members 33 and 34. However, instead of these devices, the display mechanism 15 can also be composed of devices such as an EL display or a liquid crystal display arranged at the positions of the plate-like members 33 and 34.

The display 16 (16a, 16b) displays various information to the worker. The display 16 is preferably configured by a touch panel display. The display 16a displays, for example, a screen IM20 shown in FIG. 18 described later. On the other hand, the display 16b displays, for example, main screens IM21 to IM25 shown in FIGS. 19A to 19E described later and a screen IM31 shown in FIG. 20. In addition, for example, one of the displays 16a and 16b may be used as a motion capture display that displays a moving image of the worker captured by the motion capture camera 20, and the other may be used as a data analysis display that displays the results of an analysis of the work content of the worker based on the moving image. In this embodiment, the control device 11 is described as controlling the display operation of the two displays 16a and 16b. However, the production navigation system 10 may be configured to provide individual control devices for each of the two displays 16a and 16b.
The microphone 17 records the voice of the worker, etc.
The speaker 18 issues an alarm sound and voice guidance information.

The code input unit 19 is where the code is input. In the illustrated example, the code input unit 19 is configured with a barcode scanner and is attached to the workbench 13. The code input unit 19 is not limited to a barcode scanner, and can be configured with various input means such as an IC card reader or a keyboard.

The motion capture camera 20 captures an almost complete image of the worker, including at least the upper body of the worker. The motion capture camera 20 is placed, for example, to the side of the workbench 13. The control device 11 can analyze the worker's movement line, facial direction, and the like, based on the video images captured by the motion capture camera 20. Note that in this embodiment, the production navigation system 10 is described as including a display 16 and a motion capture camera 20. However, the production navigation system 10 can function without these. Also, it is not necessary to provide one display 16 and one motion capture camera 20 for each shelf 12, and one may be provided for multiple shelves 12.

In addition, a rail 45 that functions as a hanging part for hanging an assembly tool 46 such as an electric screwdriver is provided at a position forward of the position of the USB camera 14 on the support frame 44 that supports the mirror 32. The worker can easily perform the assembly work by using the assembly tool 46 hung on the rail 45. In addition, the control device 11 can perform a process to change the torque of the electric screwdriver according to the type of the part 41, for example, when the part 41 is a screw and an electric screwdriver is used as the assembly tool 46. This allows the production navigation system 10 to support the assembly work. In addition, when the production navigation system 10 is provided with multiple electric screwdrivers and the worker makes a mistake in selecting the electric screwdriver to be used, the control device 11 can display an error message as display information to alert the worker so that the worker does not proceed to the next process. This also allows the production navigation system 10 to support the assembly work. The type of part 41 can be identified based on, for example, position information of the part placement unit 21 (picking port), which is the picking position of the part 41 analyzed from the video image captured by the USB camera 14, and position information of the part placement unit 21 (picking port) that instructs the worker to pick the part 41 as display information.

As shown in FIG. 3, in this embodiment, the display mechanism 15 projects various display images from the projector 31 onto the plate-like members 33, 34 and the surface of the workbench 13 via the mirror 32. In other words, the display mechanism 15 uses the plate-like members 33, 34 and the surface of the workbench 13 as projection units, and projects various display images onto the plate-like members 33, 34 and the surface of the workbench 13 using projection mapping technology. The projector 31 corrects the display information and projects it onto the plate-like members 33, 34 and the surface of the workbench 13 in a shape that approximates a rectangle.

The projector 31 is disposed on the shelf 12. The mirror 32 is disposed by the support frame 44 at a position above the head height of a worker of average height (for example, above the top part placement section) and in front of the front surface of the shelf 12. Specifically, the mirror 32 is disposed above the workbench 13 and diagonally upward from the shelf 12, so as to extend in the left-right direction (horizontal direction) across the entire width of the shelf 12. The mirror 32 is disposed so that its reflective surface faces the projector 31. The mirror 32 is rotatably supported by a support mechanism so that the reflection angle can be changed in the vertical direction. The mirror 32 can be fixed at any reflection angle.

The plate-like members 33 and 34 are portions that function as projection units on which the image (display information) projected from the projector 31 is formed. The plate-like members 33 and 34 are disposed at either or both of the upper and lower positions of the component placement unit 21. However, the plate-like members 33 and 34 are preferably disposed at a position above the component placement unit 21 so that the worker's hand does not block the image (display information) projected from the projector 31 (i.e., so that the projected image is not reflected on the back of the worker's hand and the shadow of the worker's hand is not reflected at the position where the image should be projected). This allows the projector 31 to project the image (display information) at a position where the worker's hand does not get in the way. In particular, when projecting, for example, an arrow-shaped mark Mk (see FIG. 8), it is preferable to display (project) the image (display information) at a position above the component placement unit 21. At least the upper surface (exposed surface) of the plate-like members 33 and 34 is a flat surface. The plate-like member 33 is configured to be larger than the plate-like member 34. The plate-shaped member 33 is suitable for presenting information that requires a relatively large area (e.g., assembly guidance information Gu (see FIG. 8), work instruction information, graphic information, process flow information, etc.). The plate-shaped member 34 is suitable for presenting information that requires only a relatively small area (e.g., marks Mk (see FIG. 8), etc.).

In the illustrated example, the plate-like members 33 and 34 are arranged at an angle on the front surface of the shelf 12 so as to extend in the left-right direction (horizontal direction) across the entire width of the shelf 12. However, the plate-like member 33 can also be configured with a width that is shorter than the width of the shelf 12.

The plate-like members 33, 34 are preferably white in color so that the display image can be clearly projected and so that the USB camera 14 (see FIG. 2) can clearly capture the worker's hands (specifically, so that they can be extracted from the background). The workbench 13 is also preferably white in color.

The plate-like members 33, 34 are arranged so that the lower they are, the more they protrude forward (towards the work space) depending on the height at which they are placed (see FIG. 4A). The plate-like members 33, 34 are also arranged at an incline so that the front portion is lower than the rear portion. In particular, the plate-like members 34 are arranged at an incline so that the lower they are, the more they are laid down (fallen) depending on the height at which they are placed (see FIG. 4B). This allows the display mechanism 15 to effectively project the display information reflected by the mirror 32 onto the plate-like members 33, 34.

In this configuration, the display mechanism 15 reflects the display image with the mirror 32, ensuring a sufficient distance between the projector 31 and the plate-like members 33, 34 and between the projector 31 and the surface of the workbench 13 to project the display image. Such a display mechanism 15 can project the display image from the projector 31 over a wide area at a narrow projection angle. As a result, the production navigation system 10 can ensure a projection distance for properly projecting the display information onto the projection unit (the plate-like members 33, 34 and the surface of the workbench 13), making it possible to avoid placing the projector 31 in a high position such as on a ceiling. Therefore, the production navigation system 10 can be used even in places where the ceiling is not high, for example.

In addition, the production navigation system 10 can project the display information onto the surface of the workbench 13 using the projector 31, allowing a relatively large image to be presented in a position that is easy for the worker to see, thereby improving work efficiency.

The display mechanism 15 can also position the projector 31 on the shelf 12 by reflecting the display image with the mirror 32. Because the top of the shelf 12 is a relatively unused space, the display mechanism 15 can effectively use this space. The display mechanism 15 can also position the projector 31, which is a relatively heavy object, away from the worker's work area, ensuring the safety of the worker. The mirror 32 is made of a material that is difficult to break, such as reinforced plastic. This also enables the production navigation system 10 to ensure the safety of the worker.

Furthermore, the production navigation system 10 can display the display information in multiple locations simultaneously by increasing the number of plate-like members 34. In other words, the production navigation system 10 can easily and inexpensively increase the number of locations where the display information is displayed, simply by increasing the number of plate-like members 34.

Figures 4A and 4B each show the configuration of the main parts of the production navigation system 10.

As shown in FIG. 4A, in this embodiment, the shelf 12 is provided with four component placement sections 21. Hereinafter, when distinguishing between the four component placement sections 21, they will be referred to as "component placement sections 21a, 21b, 21c, 21d" in order from the top (see FIG. 4B). The front portion of the shelf 12 is inclined so that the lower side protrudes further forward than the upper side. Also, each component placement section 21 is inclined so that the front portion is lower than the rear portion. The space below the lowest component placement section 21 is dead space that is difficult to access from the front portion of the shelf 12, and the control device 11 is disposed in that space. In this way, the production navigation system 10 makes full use of the space in the shelf 12.

In front of the shelf 12, a work space SP1 is provided for assembly work. Meanwhile, behind the shelf 12, an aisle space SP2 is provided for distributing trays 40 containing parts 41. A distributor or worker can distribute (insert) the tray 40 containing parts 41 from the aisle space SP2 into the part placement section 21, and by sliding the tray 40 within the part placement section 21, the tray 40 can be positioned near the front of the shelf 12.

The projector 31 is positioned at the rear of the top of the shelf 12. This allows the production navigation system 10 to prevent the shelf 12 from tipping forward.

As shown in FIG. 4B, the component placement section 21 is preferably adapted to be able to place multiple trays 40 on each level. In other words, the depth length of the component placement section 21 is preferably longer than the depth length of two or more trays 40. This allows the production navigation system 10 to remove the front tray 40 that has become empty, and position the new tray 40 that was placed behind the empty tray 40 in a position near the front of the shelf 12. The depth length of the component placement section 21 is shorter in the upper levels. Therefore, the depth length of the topmost component placement section 21a is the shortest.

Each of the partition plates 22a, 22b, 22c, and 22d that divide the part placement section 21 is shaped to have an inclined surface 22aa, 22ab, 22ac, and 22ad with the upper portion cut diagonally. The shape of each partition plate 22 differs between those arranged on the upper side of the shelf 12 and those arranged on the lower side of the shelf 12. For example, in the illustrated example, the length of each partition plate 22 in the depth direction is longer for those arranged on the lower side. Also, the length of each inclined surface 22aa, 22ab, 22ac, and 22ad in the vertical direction is longer for those arranged on the lower side. Note that each of the partition plates 22a, 22b, 22c, and 22d may be shaped to have an R surface formed in an arc shape instead of the inclined surfaces 22aa, 22ab, 22ac, and 22ad.

Figure 5 is a flowchart showing the overall operation of the production navigation system 10. Figure 6 is a flowchart showing the operation of the production navigation system 10 during part navigation. Figures 7A to 7F are diagrams showing examples of display screens. Figures 7A to 7F respectively show that the top display screen is displayed (projected) on the plate-like member 33, the middle display screen is displayed (projected) on the plate-like member 34 (any of the four plate-like members 34), and the bottom display screen is displayed (projected) on the surface of the workbench 13. Note that the display (projection) operation of each display screen is mainly performed by the display information creation unit 72 (see Figure 1) or the management unit 71 (see Figure 1) of the control device 11. Note that the flows shown in Figures 5 and 6 and the display screens shown in Figures 7A to 7F can be changed as appropriate depending on the operation.

As shown in FIG. 5, the production navigation system 10 first displays (projects) the top display screen IM11a, the middle display screen IM11b, and the bottom display screen IM11c shown in FIG. 7A onto the plate-like member 33, the plate-like member 34, and the surface of the workbench 13, respectively, and accepts a mode selection (S105). However, in the example shown in FIG. 7A, the top display screen IM11a and the middle display screen IM11b are blank.

In the example shown in FIG. 7A, three modes, "task learning mode", "expert mode", and "demo mode", are displayed selectably on the bottom display screen IM11c. The "task learning mode" is a mode that displays navigation information including a work video and instruction information with detailed explanations on various matters as display information. The "expert mode" is a mode that displays navigation information including no work video and instruction information with only explanations on relatively important matters as display information. The "demo mode" is a mode that displays a temporary screen navigation as display information. In S105, the worker selects a mode by holding his/her hand over any of the three modes displayed on the bottom display screen IM11c. Here, the explanation is given assuming that the worker selects the "task learning mode".

Next, when a mode is selected, the production navigation system 10 displays (projects) as start screens, for example, the top display screen IM12a, the middle display screen IM12b, and the bottom display screen IM12c shown in FIG. 7B, onto the plate-shaped member 33, the plate-shaped member 34, and the surface of the workbench 13, respectively (S110).

In the example shown in FIG. 7B, the display information "Finished" is displayed on the top display screen IM12a. Also, the display information "Work ID", "Order ID", and "Progress % (progress rate)" are displayed on the bottom display screen IM12c.

Next, the production navigation system 10 accepts input of the worker ID and the order ID (S115, S120). The input is accepted, for example, by the worker inputting a code from the code input unit 19 (see FIG. 2). The code is input, for example, by having a label bearing barcode information representing the personal ID given to each worker and the product ID given to each product to be manufactured, affixed to a portable medium, and having the barcode information read from the label by the code input unit 19 (see FIG. 2). The label can be created at any time. The barcode information can have any content depending on the operation.

Next, the production navigation system 10 judges whether or not the process is the final process (S125). If the judgment in S125 judges that the process is not the final process ("No"), the production navigation system 10 performs part navigation (S130).

The part navigation is performed, for example, along the flow shown in FIG.
As shown in Figure 6, the production navigation system 10 first displays (projects) the top display screen IM13a, the middle display screen IM13b, and the bottom display screen IM13c shown in Figure 7C, respectively, onto the plate-shaped member 33, the plate-shaped member 34, and the surface of the workbench 13, and displays the position and quantity of parts to be collected using arrow-shaped marks Mk (S205).

As shown in FIG. 7C, the middle display screen IM13b displayed on the plate-shaped member 34 at this time includes a mark Mk as navigation information. In the example shown in FIG. 7C, the mark Mk represents information (e.g., arrow information) indicating the position of the part placement unit 21 (collection port) on which the parts 41 to be collected are placed, and information on the number of parts 41 to be collected (e.g., "xn (pieces)"). The worker collects the number of parts 41 indicated by the mark Mk from the part placement unit 21 (collection port) indicated by the mark Mk. At this time, it is preferable that the worker repeatedly puts his/her hand into the indicated part placement unit 21 (collection port) the number of times indicated by the mark Mk to collect the parts 41. Note that the shape of the "information indicating the position of the part placement unit 21 (collection port) on which the parts 41 to be collected are placed" is not limited to "arrow information" and may be any shape that points in any direction, and may include, for example, a triangular shape.

Next, the production navigation system 10 detects the removal of the part 41 based on the video image captured by the USB camera 14 (S210), and determines whether the removed part 41 was picked from the part placement unit 21 (picking port) with the correct address indicated by the mark Mk (S215). Note that "address" here means, for example, the position of each part placement unit 21 on the shelf 12 shown in FIG. 2.

If it is determined in S215 that the removed part 41 was not picked from the part placement section 21 (picking port) with the correct address (if "No"), the production navigation system 10 displays error information (S220).

At this time, the production navigation system 10 displays (projects) the top display screen IM14a, the middle display screen IM14b, and the bottom display screen IM14c shown in FIG. 7D, for example, on the plate-shaped member 33, the plate-shaped member 34, and the surface of the workbench 13. In the example shown in FIG. 7D, each display screen IM14a, IM14b, and IM14c displays a red alarm image on the entire surface as error display information. The worker redoes the collection of the parts 41 in accordance with the error display information. The redoing of the collection of the parts 41 is performed by returning the parts 41 collected in S210 to the original part placement unit 21, and collecting the number of parts 41 indicated by the mark Mk from the part placement unit 21 (collection port) with the correct address indicated by the mark Mk.

Next, the production navigation system 10 detects the re-picking operation of the part 41 based on the video image captured by the USB camera 14 (S225). After this, the process returns to S215.

On the other hand, if it is determined in S215 that the removed part 41 was picked from the part placement section 21 (picking port) with the correct address (if "Yes"), the production navigation system 10 writes the code information, quantity information, etc. of the picked part 41 into the database (storage section) (S230).

Next, the production navigation system 10 judges whether the number of collected parts 41 has reached the number indicated by the mark Mk (S235). If it is judged in S235 that the number of collected parts 41 has not reached the number indicated by the mark Mk (if "No"), the process returns to S210. On the other hand, if it is judged in S235 that the number of collected parts 41 has reached the number indicated by the mark Mk (if "Yes"), the production navigation system 10 displays the relevant video, instruction manual, etc. as navigation information (S240).

At this time, the production navigation system 10 displays (projects), for example, the top display screen IM15a, the middle display screen IM15b, and the bottom display screen IM15c shown in FIG. 7E onto the plate-like member 33, the plate-like member 34, and the surface of the workbench 13, respectively. In the example shown in FIG. 7E, the bottom display screen IM15c displayed on the surface of the workbench 13 contains, as navigation information, assembly guidance information Gu composed of videos, instructions, etc. The worker performs assembly work in accordance with the assembly guidance information Gu. After this, the process returns to S125 in FIG. 5.

Returning to FIG. 5, if the judgment in S125 judges that the process is the final process (if "Yes"), the production navigation system 10 displays (projects) the top display screen IM16a, the middle display screen IM16b, and the bottom display screen IM16c shown in FIG. 7F as the final screens onto the plate-like member 33, the plate-like member 34, and the surface of the workbench 13, respectively, and ends the process.

In this type of production navigation system 10, the control device 11 is configured to determine the display information according to the detection information of the worker's hands. In addition, the control device 11 is configured to present error display information if the worker inserts his or her hand into the wrong part placement section 21 (collection port).

Figure 8 shows an example of display information projected onto plate-like members 33 and 34. In the illustrated example, assembly guidance information Gu explaining the assembly work is projected onto plate-like member 33, and a mark Mk indicating the collection port for the part to be collected is projected onto plate-like member 34. Note that mark Mk can also be projected onto plate-like member 33.

The mark Mk can indicate not only the location of the collection port, but also the order in which the parts are to be collected. The mark Mk can also indicate the name or code of the parts to be collected. The mark Mk can also indicate not only parts, but also the collection ports of various assembly tools.

9 shows an example of display information projected onto the workbench 13. In the illustrated example, assembly guidance information Gu is projected onto the workbench 13.
The projector 31 can project, as the display information, at least one of a moving image showing the assembly work of a product, a manual image, information on the number of parts 41, and information on the assembly process of the product, for example.

Figure 10 is a diagram showing an overview of the hand position detection operation in the production navigation system 10. The control device 11 detects the position of the worker's hands using, for example, the principles of trigonometry. Specifically, as shown in Figure 10, the control device 11 detects the position of the worker's hands based on the AND condition of the left captured image taken by the left USB camera 14a and the right captured image taken by the right USB camera 14b. In doing so, the control device 11 analyzes the position of the worker's hands using, for example, the process shown in Figure 11.

Figure 11 is a diagram showing an example of image analysis by the production navigation system 10. As shown in Figure 11, the control device 11 (see Figure 2) has a function of performing image analysis on images captured by the USB camera 14. By performing image analysis, the control device 11 can, for example, identify the hand movements of the worker. As a result, the control device 11 can also identify information such as whether the worker has picked up a part and the number of parts picked up.

An example of image analysis will be described below with reference to FIG. 11. Note that in order to distinguish the worker's hands from the background, the production navigation system 10 uses a white-based color for the background (e.g., the plate-like members 33, 34, the surface of the workbench 13, the partition plate 22, etc.). This allows the control device 11 to suitably detect the position of the worker's hands.

11A, the control device 11 acquires in advance a master image captured by the USB camera 14. The master image is an image in which the worker is not captured. A plurality of master images are prepared in advance corresponding to arbitrary portions of the shelf 12.
Next, as shown in Fig. 11(b), when the assembly work is started, the control device 11 acquires an image of the work in progress from the USB camera 14. Note that in Fig. 11(b), the dashed line area indicates the area from which the master image in Fig. 11(a) was acquired.
Next, as shown in FIG. 11(c), an image of the worker's hands is extracted from the image of the worker in action (see FIG. 11(b)).

Next, as shown in FIG. 11(d), the control device 11 performs a binarization process on the extracted image of the worker's hand (see FIG. 11(c)).
Next, as shown in FIG. 11(e), the control device 11 performs a matching process between the image after the binarization process (see FIG. 11(d)) and the master image (see FIG. 11(a)), removes noise (the part overlapping with the plate-like member 34) from the obtained image, and then detects the center of gravity of the worker's hand from the obtained image.
After that, as shown in Figure 11 (f), the control device 11 detects the position of the part placement section 21 (collection port) in which the worker's hand is placed, based on the detected center of gravity of the worker's hand (see Figure 11 (e)).

Figure 12 is a diagram showing the determination procedure when detecting the hand position in the production navigation system 10. The control device 11 sets squares in the left-right (horizontal) and up-down (vertical) directions corresponding to each part placement section 21 provided on the shelf 12, and determines the hand position in each square according to the position of the worker's hand relative to each square. For example, in this embodiment, the shelf 12 has six columns of part placement sections 21 in the left-right (horizontal) direction and four rows in the up-down (vertical) direction (see Figure 2). Therefore, the control device 11 sets six columns of squares in the left-right (horizontal) direction and four rows in the up-down (vertical) direction.

Then, as in image Ph14a in FIG. 12, the control device 11 adds a mark (a black dot in the illustrated example) to the square where the position of the worker's hand is detected from the left image captured by the left USB camera 14a through the process shown in FIG. 11. Similarly, as in image Ph14b in FIG. 12, the control device 11 adds a mark (a black dot in the illustrated example) to the square where the position of the worker's hand is detected from the left image captured by the right USB camera 14b through the process shown in FIG. 11. Then, the control device 11 determines the position of the worker's hand (i.e., the part placement section 21 (collection port) where the worker's hand is placed) based on the AND condition between images Ph14a and Ph14b.

In order to achieve good image analysis, the control device 11 performs image analysis using multiple types of filters (for example, two types).
Fig. 13 shows an example of filter settings for skin color. Fig. 13(a) shows a captured image of an operator's hand being inserted into a collection port and an example of filter settings for the captured image. Fig. 13(b) shows example captured images of a woman's hand and a man's hand. For example, the control device 11 uses a blue filter to identify skin color. The filter is set to a desired gradation value so that the hand shown in Fig. 13(b) can be identified.

Figure 14 shows an example of filter settings for the color of work clothes. Figure 14 (a) shows a captured image of a worker's hand being inserted into a collection port and an example of filter settings for the captured image. Figure 14 (b) shows example captured images of women's work clothes and men's work clothes. Here, the colors of women's work clothes are yellow and light green, or yellow and dark green, while the colors of men's work clothes are black and dark green. In this case, the control device 11 uses a red filter to identify the color of the work clothes. The filter is set to the desired gradation value so that the work clothes shown in Figure 14 (b) can be identified.

The filter to be used may be automatically adjusted (set) according to the worker's skin color or the color of his or her work clothes.

FIG. 15 is a flowchart showing the operation of the production navigation system 10 during image analysis.
15, the control device 11 performs the processes of S305 to S350 on the left photographed image captured by the left USB camera 14a. In parallel with the processes of S305 to S350, the control device 11 performs the processes of S405 to S450 on the right photographed image captured by the right USB camera 14b.

Specifically, the control device 11 acquires the left photographed image (S305) and deletes unnecessary images (images of parts other than the worker's hand) from the left photographed image (S310). Then, the control device 11 masks the left camera image with a red filter (S322), binarizes the obtained image (S324), extracts the difference from the master image (S326), and deletes noise (parts overlapping with the plate-like member 34) from the obtained image (S328). Similarly, the control device 11 masks the left photographed image with a blue filter (S332), binarizes the obtained image (S334), extracts the difference from the master image (S336), and deletes noise (parts overlapping with the plate-like member 34) from the obtained image (S338). After this, the control device 11 combines the image obtained by applying the red filter and the image obtained by applying the blue filter (S340), detects the center of gravity of the worker's hand from the obtained image (S345), and determines the position of the collection port where the hand is inserted (S350).

The control device 11 also acquires a right-hand image (S405) and deletes unnecessary images (images of parts other than the worker's hand) from the right-hand image (S410). Then, the control device 11 masks the right camera image with a red filter (S422), binarizes the obtained image (S424), extracts the difference from the master image (S426), and deletes noise (parts overlapping with the plate-shaped member 34) from the obtained image (S428). Similarly, the control device 11 masks the left-hand image with a blue filter (S432), binarizes the obtained image (S434), extracts the difference from the master image (S436), and deletes noise (parts overlapping with the plate-shaped member 34) from the obtained image (S438). After this, the control device 11 combines the image obtained by applying the red filter and the image obtained by applying the blue filter (S440), detects the center of gravity of the worker's hand from the obtained image (S445), and determines the position of the collection port where the hand is inserted (S450).

After this, if the determination result is the same collection port, the control device 11 writes the position of the collection port into the database (S500). In this way, the control device 11 performs image analysis.

The control device 11 of the production navigation system 10 can perform data management operations such as storing work data acquired during product assembly work in the storage unit 73, managing work results in the management unit 71 based on the stored work data, and creating display information representing the work results at any time in the display information creation unit 72. At that time, the management unit 71 can manage the work data and work results based on the code input from the code input unit 19. The management unit 71 can also assign semi-finished product codes to semi-finished products, which are intermediate products, at any time, and register the work data in the management information file of the storage unit 73 by associating it with the finished product code and semi-finished product code assigned to the finished product, which is the final product.

The data management operation of the production navigation system 10 to realize such processing will be described below with reference to FIG. 16. FIG. 16 is a flowchart showing the data management operation of the production navigation system 10.

As shown in FIG. 16, the control device 11 of the production navigation system 10 determines whether work has started (S605), and if it is determined that work has started ("Yes"), starts accumulating work data (S610).

Next, the control device 11 determines whether or not there is an instruction to assign a semi-finished product code (S615). If it is determined in S615 that there is no instruction to assign a semi-finished product code (if "No"), the process proceeds to S625. On the other hand, if it is determined in S615 that there is an instruction to assign a semi-finished product code (if "Yes"), the control device 11 assigns an arbitrary semi-finished product code to the semi-finished product and registers it in the management information file D11 (see FIG. 17) stored in the database (storage unit) 11a (see FIG. 17) (S620).

Figure 17 is a diagram showing an example of a management information file D11. In the example shown in Figure 17, the management information file D11 is configured to register information such as a semi-finished product code, worker ID, order ID, station ID, work date, start time, end time, elapsed time, and finished product code.

Returning to FIG. 16, if the determination in S615 is "No" or after S620, the control device 11 determines whether or not there is an instruction to display the work results (S625). If it is determined in S625 that there is no instruction to display the work results (if "No"), the process proceeds to S635. On the other hand, if it is determined in S625 that there is an instruction to display the work results (if "Yes"), the control device 11 analyzes the work results and displays the analysis results of the work results on the display 16 (S630).

If the determination in S625 is "No" or after S630, the control device 11 determines whether or not an instruction to end the work has been given (S635). If the determination in S635 determines that an instruction to end the work has not been given (if "No"), the process returns to S615. On the other hand, if the determination in S635 determines that an instruction to end the work has been given (if "Yes"), the control device 11 ends the accumulation of work data (S640) and causes the work performance analysis/end screen IM41 (see FIG. 21) to be displayed on the display 16 (S645). This ends the processing of the series of routines.

Such a production navigation system 10 can effectively detect the picking up of parts. The production navigation system 10 can also detect the picking up of parts by detecting parts, not just hand movements. The production navigation system 10 can also detect when a worker has picked up multiple parts at once.

Such a production navigation system 10 can prevent a worker from judging that he or she has picked a part even if he or she has not actually picked a part. Furthermore, if the production navigation system 10 detects that a part has been picked incorrectly, it can point out the mistake by sounding an alarm from the speaker 18 or displaying an alarm on the plate-like members 33 and 34.

The control device 11 of the production navigation system 10 can also display, for example, analysis results IM21 to IM25 (see Figures 19A to 19E) of work performance that correspond to items BT11 to BT15 of the main screen IM20 shown in Figure 18 on the display 16. Figure 18 is a diagram showing an example of the main screen IM20. Figures 19A to 19E are respectively diagrams showing examples of screens IM21 to IM25 of the analysis results of work performance.

In the example shown in FIG. 18, the main screen IM20 contains items BT11 to BT15. "Item BT11" is a field for specifying the display of production status. "Item BT12" is a field for specifying the display of production results for each order ID of the target worker specified as the target of work results. "Item BT13" is a field for specifying the display of the results of comparing the work content of the target worker with that of an experienced worker. "Item BT14" is a field for specifying the display of the target worker's work results by day. "Item BT15" is a field for specifying the display of the target worker's work results by process. If no person is specified as the target worker, the person working at the current work site (work station) will be used. However, the production navigation system 10 can also specify any person as the target worker by inputting the ID code of that person using the code input unit 19 (see FIG. 2) or a keyboard, etc.

The main screen IM20 is displayed on the display 16a, which is, for example, a touch panel display. When, for example, one of items BT11 to BT15 is specified on the display 16a, the production navigation system 10 displays a screen showing the analysis results of the work performance corresponding to the specified item on the display 16b.

For example, when "item BT11" in FIG. 18 is specified, the production navigation system 10 displays screen IM21 shown in FIG. 19A on the display 16b. Screen IM21 shows the work results of the production status. In the example shown in FIG. 19A, screen IM21 contains the contents of "Today's target number," "Planned" number, "Actual" number, and "Difference" number.

For example, when "item BT12" in FIG. 18 is selected, the production navigation system 10 displays screen IM22 shown in FIG. 19B on the display 16b. Screen IM22 shows the production results for each order ID of the target worker as bar graph data. In the figure, the horizontal axis indicates the product order ID, and the vertical axis indicates the production time (seconds).

For example, when "item BT13" in FIG. 18 is specified, the production navigation system 10 displays the screen IM23 shown in FIG. 19C on the display 16b. The screen IM23 shows the result of comparing the work content of the target worker (comparator) and the skilled worker. The screen IM23 shows a comparison between the moving image of the target worker (comparator) working, captured by the USB cameras 14a and 14b, and the moving image of the skilled worker working, which was prepared in advance. In FIG. 19C, the four rectangular parts on the comparator side are areas where the moving image of the target worker (comparator) working is displayed. The four rectangular parts on the skilled worker side are areas where the moving image of the skilled worker working is displayed. In addition, three rows of square images are arranged on the right side of the area where each moving image is displayed. The image in the top row of squares shows the position of the hand detected from the left captured image (image captured by the left USB camera 14a). The image in the bottom square represents the hand position detected from the right captured image (image captured by the right USB camera 14b). The image in the middle square represents the worker's hand position determined based on the AND condition between the hand position detected from the left captured image and the hand position detected from the right captured image.

For example, when "item BT14" in FIG. 18 is specified, the production navigation system 10 displays screen IM24 shown in FIG. 19D on the display 16b. Screen IM24 shows the daily work performance of the target worker as bar graph data. In the figure, the horizontal axis shows the date, and the vertical axis shows the production time (minutes). The bar graph data is data that includes process information that shows each process of the product assembly work, and information on the time required for each process. Each bar graph data represents the range between the minimum and maximum work times for a certain product produced by the target worker. The rectangles in each bar graph data represent the average value of the minimum and maximum work times. The longer the bar graph, the greater the variation in process time for that process.

For example, when "item BT15" in FIG. 18 is specified, the production navigation system 10 displays screen IM25 shown in FIG. 19E on the display 16b. Screen IM25 shows the work performance of the target worker by process as bar graph data. In the figure, the horizontal axis shows the process number, and the vertical axis shows the production time (seconds). Each bar graph data represents the range between the minimum and maximum work times for a certain product produced by the target worker. One of the two rectangles in each bar graph data represents the average of the minimum and maximum work times, and the other represents the previous work time. The longer the bar graph, the greater the variation in process time for that process.

The control device 11 can create graph data for screen IM24 (FIG. 19D) or screen IM25 (FIG. 19E) for each semi-finished product and display it on the display 16b. The control device 11 can also create graph data for each worker and display it on the display 16b. By referring to the graph data, managers can understand which process is taking the most time.

The results of the analysis of work performance performed by the control device 11 of the production navigation system 10 are provided to the data collection device 61 (see FIG. 1) of the support system 60. As a result, the results of the analysis of work performance can be viewed not only by the control device 11 of the production navigation system 10, but also by the data collection device 61, data processing device 62, support device 63, etc. that constitute the support system 60 (see FIG. 1).

The control device 11 of the production navigation system 10 can also analyze the movements of the worker based on the video of the worker captured by the motion capture camera 20 during product assembly work. For example, as shown in FIG. 20, the display information creation unit 72 of the control device 11 of the production navigation system 10 can identify the movement distance of the worker's hands based on the video of the worker captured by the motion capture camera 20 during product assembly work, and create the line graph data shown in FIG. 20 as display information. FIG. 20 is a diagram showing an example of a movement analysis screen IM31 as a result of analyzing the movements of the worker. In FIG. 20, the line graph data represents the correlation between frame information of the video and movement distance information of the worker's head and hands. The movement analysis screen IM31 is displayed, for example, on the display 16b.

In the example shown in FIG. 20, the motion analysis screen IM31 includes a video display section Mo1 that displays video, a command section Co1 that displays commands that instruct the operation of the video, and a graph display section Gr1 that displays line graph data. The video in the video display section Mo1 displays additional information in the form of small circles that show the analysis results of the worker's movement line and facial direction, etc. The line graph data in the graph display section Gr1 has the horizontal axis indicating the number of frames of the video, and the vertical axis indicating the distance traveled by the head and hands. The vertical lines in the graph display section Gr1 indicate the divisions between each process, and the numbers indicate the process numbers.

In addition, analysis of the worker's movements, such as the movement analysis screen IM31 shown in FIG. 20, can be performed not only by the control device 11 of the production navigation system 10, but also by the data collection device 61, data processing device 62, support device 63, etc. that constitute the support system 60 (see FIG. 1).

The control device 11 of the production navigation system 10 can also create a screen (for example, an end screen IM41 (see FIG. 21)) that displays elapsed time, etc., based on the management information file D11 (see FIG. 17). FIG. 21 is a diagram showing an example of the end screen IM41. In the example shown in FIG. 21, the end screen IM41 includes a standard time column that displays the standard time, a productivity column that displays the production rate, an elapsed time column that displays the elapsed time, a start time column that displays the start time, and an end time column that displays the end time.

<Major functions of the production navigation system>
The production navigation system 10 has a control function, an image (moving image) capturing function, a voice recognition function, and a voice instruction function.
The production navigation system 10 has a guidance function, an image acquisition function during part picking and an assessment function for the same, an image acquisition function during part installation and an assessment function for the same, a work flow acquisition function and an analysis function, etc.
The production navigation system 10 can output captured image (video) information, input audio information, etc. to a data collection device 61 of the support system 60, and have this information analyzed and processed as big data by a data processing device 62 or a support device (PC) 63.
The production navigation system 10 can be expanded later with image analysis functions and voice recognition functions.
The production navigation system 10 can guide the worker in the assembly work by presenting various information to the worker.
The production navigation system 10 can analyze the type and number of parts picked by the worker, the work content performed by the worker, and the like.
The production navigation system 10 can sound an alarm and present alarm information.
As shown in FIG. 21, the production navigation system 10 can display the progress of assembly work on the display 16 at any desired timing, such as when assembly is completed or during the assembly process.

<Main functions of the support system>
The data collection device 61 can acquire and store photographic information and audio information from the control device 11 of the production navigation system 10. A manager or the like can view the photographic information stored in the data collection device 61 on a support device (PC) 63 to check the status of the assembly work site, such as the production progress, from a remote location.

<Main features of the production navigation system>
(1) Regarding Projection Mapping (a) The production navigation system 10 can present (project) the positions and quantities of parts to be picked by the worker. For example, the production navigation system 10 can project arrow information indicating the position of the part placement section 21 (picking port) on which the parts 41 to be picked are placed, an arrow-shaped mark Mk (see FIG. 8) including information on the number of parts, etc. This allows the production navigation system 10 to easily inform the worker how many parts 41 should be picked from which collection port, simply by glancing at the mark Mk (see FIG. 8).
(b) The production navigation system 10 can determine the display information to be displayed on the display mechanism 15 according to the position of a specific part of the worker photographed by the USB camera 14. For example, when the worker picks up a part, the production navigation system 10 can automatically reduce the count of the parts placed on the shelf according to the number of picked parts (picked parts). The production navigation system 10 can also present (project) display information for the next process. The production navigation system 10 can also determine the display information according to the command corresponding to the input position when the worker's hand is held over a command input position such as the special command input unit 23 (see FIG. 2).
(c) When a worker picks up a part, the production navigation system 10 can present (project) the position and quantity of the next part.
(d) When the worker places his/her hand in the wrong part placement unit 21, the production navigation system 10 displays attention-calling information such as error display information (see FIG. 7D ). This allows the production navigation system 10 to display (project) attention-calling information even when the worker picks up the wrong part.
(e) The production navigation system 10 can change the position at which the attention-calling information is presented depending on the position of each part.
(f) The production navigation system 10 can project, as display information, at least one of a moving image showing the assembly work of the product, a manual image, information on the number of parts 41, and product assembly process information from the projector 31. This allows the production navigation system 10 to efficiently support the assembly work of the product.
(g) The projector 31 corrects the display information and projects it in a shape that approximates a rectangle onto the plate-like members 33, 34 and the surface of the workbench 13. This enables the production navigation system 10 to present the worker with display information that is highly visible and has little distortion.
(h) The production navigation system 10 can selectively project from the projector 31, as display information, a task learning mode that displays navigation information that includes a task video and instruction information with detailed explanations on various matters, and a skilled mode that displays navigation information that does not include a task video and includes instruction information with only explanations on relatively important matters. This allows the production navigation system 10 to change navigation in accordance with the level of mastery of the product assembly task.
(i) The projector 31 can display (project) arrow information indicating the position of the component placement unit 21 on which the component 41 to be picked is placed, as display information, above the component placement unit 21. This allows the projector 31 to display (project) the arrow information at a position that does not interfere with hand detection.
(j) In the production navigation system 10, the number of locations on which display information is displayed can be increased easily and inexpensively by simply increasing the number of plate-like members 34.
(k) The production navigation system 10 changes the angle of the plate-like member 34 depending on the presentation (projection) position. The lower the plate-like member 34 is positioned, the more the plate-like member 34 is laid down. This makes it possible to improve the visibility of the displayed information.
(l) The color of the plate-like members 33, 34 onto which the display information is projected is preferably white or similar. This allows the display information to be clearly presented (projected) and improves the visibility of the display information.
(m) When the collection and assembly of parts is completed, the production navigation system 10 can display on the plate members 33, 34, the display 16, etc., what percentage of the entire process has been completed.
(n) The production navigation system 10 can project information such as worker ID, temporary ID, work progress information, location and quantity of parts to be collected, work procedure (including image diagrams, etc.), required assembly tools, etc.
(o) When the production navigation system 10 detects the collection of necessary parts, it can reduce the part quantity and manage the part quantity.
(p) When the production navigation system 10 detects the collection of a necessary part, it can present the location and quantity of the next part as display information.
(q) If the worker makes a mistake in picking the part, the production navigation system 10 can project warning information. The production navigation system 10 can also detect when the worker returns the part.
(r) When presenting attention-calling information, the production navigation system 10 can adjust the attention-calling information to an optimal position and present it according to the position of the part.
(s) The production navigation system 10 can use the surface of the workbench 13 as a projection unit.
(t) The production navigation system 10 can present the location of the next part to the worker when it detects that a part has been picked up. The production navigation system 10 can also present the location of the next part to the worker when all work procedures related to a desired task have been completed.
(u) The production navigation system 10 may be configured to switch the display information after the assembly using the electric screwdriver is completed.

(2) Regarding the configuration of the shelves (a) The shelves 12 are provided with a plurality of partition plates 22. The partition plates 22 are shaped to have inclined surfaces 22aa, 22ab, 22ac, 22ad (see FIG. 4B) or an R-surface (not shown) at the upper portion. This allows the production navigation system 10 to separate adjacent part placement sections 21 with each partition plate 22, and also ensures ease of insertion of the worker's hand into the part placement section 21.
(b) The plate-like members 34 are arranged at an incline so that the lower they are arranged, the more they are laid down (fallen down) depending on the height at which they are arranged (see FIG. 4B ). This allows the production navigation system 10 to project the display information reflected by the mirror 32 onto the plate-like members 33 in a good condition.

(3) Regarding image analysis of hand position (a) The production navigation system 10 can perform image analysis of the hand position by performing any image processing (e.g., image acquisition processing, binarization processing, center of gravity detection processing, position detection processing, etc.) based on the captured image.
(b) The production navigation system 10 can improve the accuracy of distinguishing between hands and the background by making the colors of the background parts (e.g., the colors of the plate-like members 33, 34, the surface of the workbench 13, the partition board 22, etc.) white-based colors.

(4) Image Detection (a) The production navigation system 10 can perform image judgments satisfactorily even if there are changes in the environment (external brightness).
(b) The production navigation system 10 can suppress a decrease in detection accuracy due to differences in clothing (for example, a decrease in detection accuracy for whitish clothing).
(c) The production navigation system 10 can accurately detect the position of the hand by analyzing the captured image, even if the hand is inserted into the collection port from an oblique direction.
(d) The production navigation system 10 can identify the amount of insertion of the worker's hand into the collection port (amount of movement in the depth direction) by analyzing the captured image.
(e) The production navigation system 10 can use a commercially available general-purpose program as the image analysis program.

(5) Image Analysis (a) Even if an operator mistakenly picks up a part, the production navigation system 10 can detect the mistake by image analysis. In this regard, the conventional technology could only identify whether or not the operator's hand passed through the sensor provided at the collection port (i.e., whether or not the operator's hand was inserted into the collection port), so there was a possibility that the operator would erroneously detect that a part had been picked even if the operator had not actually picked up a part. In addition, in the conventional technology, when an operator picks up multiple parts, the device cannot detect this, and as a result, there was a possibility that the count number of stored parts would be incorrect. In contrast, the production navigation system 10 according to this embodiment can identify whether or not the operator actually picked up a part and how many parts the operator picked up, by analyzing the moving image when the parts were picked. As a result, the production navigation system 10 can eliminate the mistaken detection of parts that have not been picked as having been picked and the mistaken count number of stored parts.
(b) The production navigation system 10 can identify that a worker has picked up multiple parts by image analysis. In this regard, in the prior art, for example, when a worker picks up two of the same parts, the worker needs to insert and remove his/her hand into and from the picking opening twice. In contrast, the production navigation system 10 according to the present embodiment can identify that two parts have been picked by the worker simply inserting and removing his/her hand into and from the picking opening once.
(c) The production navigation system 10 can confirm the completion of parts assembly by image analysis. In the prior art, whether or not parts assembly is complete is managed only by the worker's declaration, forcing the worker to perform the process. In contrast, the production navigation system 10 according to the present embodiment can automatically manage whether or not parts assembly is complete, without the worker having to declare it.
(d) The production navigation system 10 performs image analysis using multiple types (e.g., two types) of filters, and can detect the skin color of the worker, the worker's work clothes, etc. The production navigation system 10 can then, for example, binarize the captured image and detect the center of gravity (in the illustrated example, the base of the index finger and middle finger), thereby identifying the position of the worker's hand.

(6) Consolidated Data Management (a) The support system 60 can collectively store the captured images in the data collection device 61.
(b) The support system 60 allows a manager or the like to analyze the work content at each work site based on the captured images stored in the data collection device 61, thereby enabling short-term work improvement (for example, reducing the variation in work time and adjusting line balance). In other words, the support system 60 allows the manager or the like to check assembly operations, etc. from a remote location. In this regard, in the conventional technology, for example, the manager or the like had to go to the assembly work site to check the assembly operations, etc. In contrast, the production navigation system 10 according to this embodiment allows the manager or the like to check the completion of assembly, etc. from a remote location without having to go to the assembly work site.
(c) The support system 60 can check the operation of a process that has variations in work time from a remote location.

(7) Regarding Voice Confirmation (a) The production navigation system 10 can also confirm the completion of the assembly operation by acquiring voice information of the worker via the microphone 17.

(8) Regarding selection of assembly tools: (a) The production navigation system 10 can confirm that tightening has been completed with an electric screwdriver.
(b) When the torques of the electric screwdrivers are different, the production navigation system 10 can instruct the worker to select the desired electric screwdriver. Furthermore, when multiple electric screwdrivers are provided and the worker makes a mistake in selecting the electric screwdriver to use, the production navigation system 10 can display an error message as display information to alert the worker so that the worker will not proceed to the next process.
(c) The production navigation system 10 can automatically adjust the torque of the electric screwdriver by changing the current value according to the type of screw.

(9) Regarding parts management, work management, etc. (a) The production navigation system 10 can provide functions such as work instructions, production status, and warnings at a low cost by presenting information through projection mapping using the projector 31 and by performing image processing on images captured by the USB camera 14.
(b) The production navigation system 10 has a management unit 71 (see FIG. 1), a display information creation unit 72 (see FIG. 1), and a storage unit 73 (see FIG. 1), and by using these, a data management method can be realized in which work data acquired during product assembly work is stored in the storage unit 73, the management unit 71 manages work results based on the stored work data, and the display information creation unit 72 creates display information representing the work results at any timing. In addition, when the support system 60 is provided with the management unit 71, the display information creation unit 72, and the storage unit 73 within the system, the support system 60 can realize a similar data management method by using these. As a result, for example, the production navigation system 10 and the support system 60 can check whether there are any missing parts at the work site, whether screw tightening and adjustment work, test results, etc. are good or not.
(c) The production navigation system 10 can improve efficiency by implementing stricter work history and IT.
(d) The production navigation system 10 can display the work time at the assembly site in real time, which makes it possible to improve work in a short period of time (for example, reducing the variation in work time and adjusting the line balance).
(e) The production navigation system 10 can assign a semi-finished product code to a semi-finished product, which is an intermediate product, at any time and input the code from the code input unit 19 (see FIG. 2) or a keyboard. The production navigation system 10 and the support system 60 can manage the work data and the work results based on the input code. For example, the production navigation system 10 and the support system 60 can read out the work data related to the associated product at any time by associating the work data with the finished product code and the semi-finished product code assigned to the finished product, which is the final product, and registering the data in the management information file D11 (see FIG. 17) in the storage unit 73 (see FIG. 1). This allows the production navigation system 10 and the support system 60 to read out and verify information on which worker assembled which semi-finished product, and video images of the work during the assembly of the semi-finished product, from the database, for example, based on the finished product code of the finished product, which is the final product. In addition, the production navigation system 10 and the support system 60 can read out from the database, for example, a working video image of the worker with that ID based on the worker ID, and verify the quality of the work performed by the worker with that ID.
(f) The production navigation system 10 and the support system 60 can display on a display a video image of a non-skilled worker working and a video image of a skilled worker working so that they can be compared (see FIG. 19C). This allows the production navigation system 10 and the support system 60 to easily verify issues with the work performed by each worker (for example, factors that require time in each process, the appropriateness of the work content, etc.).
(g) The production navigation system 10 and the support system 60 can analyze the movements of a worker based on video images of the worker captured by the motion capture camera 20 during product assembly work (see FIG. 20).
(h) The production navigation system 10 eliminates the need for parts kitting, which can eliminate the need for parts distribution work and parts distributors. Therefore, the production navigation system 10 can reduce production costs.
(i) The production navigation system 10 can automatically record work history as electronic data by using information from sensors, electric drivers, etc. and analyzing the work content of the worker. This allows the production navigation system 10 to reduce the amount of work required to create checklists (i.e., reduce the effort required to enter the work history for each worker on a checklist (list of work items) and to stamp or sign it). It can also promote a paperless system. This also allows the production navigation system 10 to reduce production costs.
(j) The production navigation system 10 can immediately respond even if the worker is replaced.
(k) The display information creation unit 72 (see FIG. 1) can create and display, as the display information, graph data including process information representing each process of the assembly work of the product and time information required for each process (see, for example, FIG. 19B, FIG. 19D, and FIG. 19E). The display information creation unit 72 (see FIG. 1) can also create and display, as the display information, work image information that displays a work video image of an expert and a work video image of an unexpert so as to be comparable (see, for example, FIG. 19C). Furthermore, the display information creation unit 72 (see FIG. 1) can create, as the display information, navigation information for a work acquisition mode that includes a work video and instruction information with detailed explanations on various matters, and navigation information for a skilled mode that does not include a work video and instruction information with only explanations on relatively important matters, and selectably display them (see FIG. 7A). Furthermore, the display information creation unit 72 (see FIG. 1) can identify the distance the worker's hands have moved based on the video of the worker captured by the motion capture camera 20 during product assembly work, and create and display, as display information, graph data showing the correlation between frame information of the video and information on the distance the worker's hands have moved (see FIG. 20). By providing such a display information creation unit 72 (see FIG. 1) within the system, the production navigation system 10 and the support system 60 can efficiently support product assembly work.

(10) Functionality Expandability (a) The production navigation system 10 can be supplemented with a USB camera 14, etc. This enables the production navigation system 10 to check, for example, whether or not a nameplate is missing.
(b) The production navigation system 10 can be provided with an additional microphone 17 and speaker 18. This allows the production navigation system 10 to improve the accuracy of voice recognition and the appropriateness of voice instructions.
(c) The production navigation system 10 can update various programs (for example, the control program of the production navigation system 10, the image analysis program, the voice recognition program, etc.) as needed.
(d) The production navigation system 10 may be configured to provide separate lighting devices around the workbench 13. This allows the production navigation system 10 to acquire captured images with good brightness even if the brightness around the workbench 13 varies depending on the location, time of day, etc., and improves the accuracy of analyzing the captured images.

As described above, the production navigation system 10 according to this embodiment can provide assembly workers with information that is suitable for producing equipment.

The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention.
For example, the above-described embodiment has been described in detail to clearly explain the gist of the present invention. Therefore, the present invention is not necessarily limited to having all of the components described. In addition, the present invention can be modified by adding other components to certain components, or by replacing some components with other components. In addition, the present invention can be modified by deleting some components.

In addition, for example, the flowcharts shown in Figures 5, 6, 15, and 16 can be appropriately changed depending on the operation. In addition, for example, the contents of the display screen, graph data, etc. can be appropriately changed in configuration depending on the operation.
[Additional Notes]
The invention described in the claims originally attached to this application is as follows: (1)
a display control unit that controls to display information to assist the worker in an area corresponding to a plurality of placement units on which the worker places the items to be collected;
an analysis control unit that analyzes the motion of the worker based on a moving image including the worker captured by a capturing unit;
A storage unit that stores the work results based on the analysis result by the analysis control unit;
and an output unit that outputs information including a work time based on the work performance stored in the storage unit.
Navigation system.
(2)
The display control unit displays assembly guidance information related to the item.
The navigation system described in (1) above.
(3)
The storage unit further stores information for identifying the worker in association with the work record.
The navigation system described in (1) above.
(4)
The work performance includes a start time and an end time.
The navigation system described in (1) above.
(5)
The analysis control unit analyzes the motion of the worker by detecting a center of gravity position of the worker's hand based on the moving image.
The navigation system described in (1) above.
(6)
the analysis control unit detects a center of gravity position of the worker's hand based on a difference between the moving image and a master image that does not include the worker, thereby analyzing the motion of the worker.
The navigation system described in (5) above.
(7)
The navigation system further includes an assembly tool control unit that controls an assembly tool,
the assembly tool control unit acquires a work history using the assembly tool based on information from the assembly tool.
The navigation system described in (1) above.
(8)
A step of controlling the display of information to assist the worker in an area corresponding to a plurality of placement units on which the worker places the items to be collected;
Analyzing the movement of the worker based on a moving image including the worker captured by a capturing unit;
storing the work performance based on the analysis result in a storage unit;
and outputting information including a work time based on the work performance stored in the storage unit.
Information processing methods.
(9)
Computer,
a display control unit that controls to display information to assist the worker in an area corresponding to a plurality of placement units on which the worker places the items to be collected;
an analysis control unit that analyzes the motion of the worker based on a moving image including the worker captured by a capturing unit;
A storage unit that stores the work results based on the analysis result by the analysis control unit;
an output unit that outputs information including a work time based on the work performance stored in the storage unit;
A program to function as a

10 Production navigation system 11 Control device (PC)
11a Database (storage unit)
12 Shelf (parts warehouse)
13 Work table 14 (14a, 14b) Photography unit (USB camera)
15 Display mechanism (display unit)
16 (16a, 16b) Display 17 Microphone 18 Speaker 19 Code input section 20 Motion capture camera 21 (21a, 21b, 21c, 21d) Part placement section 22 (22a, 22b, 22c, 22d) Partition plate 22aa, 22ab, 22ac, 22ad Inclined surface 23 Command input section 31 Projector 32 Mirror 33, 34 Plate-shaped member (projection section)
40 Tray 41 Part 44 Support frame 45 Rail (hanging part)
46 Assembly tools (electric screwdriver)
60 Support system 61 Data collection device 62 Data processing device 63 Support device (PC)
71 Management unit 72 Display information creation unit (display control unit, analysis control unit, output unit)
73 Storage unit (memory unit)
81 Acquisition section 82 Storage section 83 Analysis section 84 Provision section DP Image data D1 Master image acquisition D2 Photographed image acquisition D3 Binarization processing D11 Management information file Gu Guidance Mk Mark P0 Center of gravity detection R1, R2, R3 Position detection P1 Position

Claims (9)

A display control unit that controls to display information to assist the worker in an area corresponding to a plurality of placement units on which the worker places the items to be collected;
an analysis control unit that analyzes the motion of the worker based on a moving image including the worker captured by a capturing unit;
A storage unit that stores work results based on the analysis results by the analysis control unit;
and an output unit that outputs information including a work time based on the work performance stored in the storage unit.
Navigation system. The display control unit displays assembly guidance information related to the item.
The navigation system of claim 1. The storage unit further stores information for identifying the worker in association with the work record.
The navigation system of claim 1 . The work performance includes a start time and an end time.
The navigation system of claim 1 . The analysis control unit analyzes the motion of the worker by detecting a center of gravity position of the worker's hand based on the moving image.
The navigation system of claim 1 . the analysis control unit detects a center of gravity position of the worker's hand based on a difference between the moving image and a master image that does not include the worker, thereby analyzing the motion of the worker.
6. The navigation system according to claim 5. The navigation system further includes an assembly tool control unit that controls an assembly tool,
the assembly tool control unit acquires a work history using the assembly tool based on information from the assembly tool.
The navigation system of claim 1. A step of controlling the display of information to assist the worker in an area corresponding to a plurality of placement units on which the worker places the items to be collected;
Analyzing the movement of the worker based on a moving image including the worker captured by a capturing unit;
storing the work performance based on the analysis result in a storage unit;
and outputting information including a work time based on the work performance stored in the storage unit.
Information processing methods. Computer,
a display control unit that controls to display information to assist the worker in an area corresponding to a plurality of placement units on which the worker places the items to be collected;
an analysis control unit that analyzes the motion of the worker based on a moving image including the worker captured by a capturing unit;
A storage unit that stores work results based on the analysis results by the analysis control unit;
an output unit that outputs information including a work time based on the work performance stored in the storage unit;
A program to function as a

Images