JP7629735B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

Patent Literature 1 discloses a tracking device that tracks a plurality of objects by using a distance measuring device that captures an image of a target space in which the plurality of objects exist and measures the three-dimensional positions of the plurality of objects.
[Prior Art Literature]
[Patent Documents]
[Patent Document 1] JP 2019-205060 A

In a first aspect of the present invention, an information processing device is provided. The information processing device includes, for example, a point cloud data acquisition unit that acquires three-dimensional point cloud data of a first object arranged in a three-dimensional space. The information processing device includes, for example, a modeling unit that constructs a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit. The information processing device includes, for example, an instruction receiving unit that receives a first instruction from a first user to specify a designated position that is a point or area arranged on the surface of the three-dimensional model of the first object arranged in the virtual space. The information processing device includes, for example, a projection control unit that controls a projection device that projects an image to project the first image at a projection position that is a point or area corresponding to the designated position on the surface of the first object.

In the above information processing device, the projection control unit may control the projection device so that, when the first object moves in three-dimensional space, the first image projected onto the first object follows the movement of the first object. In the above information processing device, the instruction receiving unit may receive a second instruction from the first user, the second instruction being for identifying an attribute of the first image. In the above information processing device, the projection control unit may (i) adjust at least one of the projection direction of the first image and the shape and size of the first image so that, when the first object moves in three-dimensional space, the first image projected onto the first object follows the movement of the first object when the attribute of the first image indicated by the second instruction is the first attribute. The projection control unit may (ii) not adjust the projection direction of the first image and the shape and size of the first image even when the first object moves in three-dimensional space when the attribute of the first image indicated by the second instruction is the second attribute.

In the above information processing device, the instruction receiving unit may receive an instruction from the first user, which is (i) a relative position of the indicated position with respect to a reference point of the three-dimensional model of the first object, or (ii) a third instruction for fixing the relative position of the indicated position with respect to a reference point of the virtual space. In the above information processing device, the projection control unit may adjust at least one of the projection direction of the first image and the shape and size of the first image so that the first image projected onto the first object follows the movement of the first object when the first object moves in the three-dimensional space when the instruction receiving unit receives the third instruction and the attribute of the first image indicated by the second instruction is the first attribute. The projection control unit may not adjust the projection direction of the first image and the shape and size of the first image even if the first object moves in the three-dimensional space when the instruction receiving unit receives the third instruction and the attribute of the first image indicated by the second instruction is the second attribute.

In the above information processing device, the instruction receiving unit may receive a fourth instruction from the first user, which is related to starting or stopping the projection of the first image. In the above information processing device, the projection control unit may determine whether or not to project the first image onto the first object based on the fourth instruction. The above information processing device may include an indication mark control unit that controls the movement of a mark indicating a specific point in the virtual space based on a control signal output by a pointing device or a controller used by the first user to specify an indication position in the virtual space. In the above information processing device, the indication mark control unit may control the movement of the mark based on first setting information that specifies how the mark moves near the boundary of the three-dimensional model of the first object.

The information processing device may include an image data acquisition unit that acquires image data of the first object. The information processing device may include an object estimation unit that estimates the type of the first object based on the image data acquired by the image data acquisition unit. The information processing device may include a setting extraction unit that extracts second setting information corresponding to the type of the first object estimated by the object estimation unit by referring to a setting storage unit that stores information indicating the type of each of the one or more objects in association with second setting information that specifies the movement of the mark at each of one or more positions of a three-dimensional model of each object corresponding to each of one or more positions on the surface of each object. In the information processing device, the indication mark control unit may control the movement of the mark based on the second setting information extracted by the setting extraction unit.

In the above information processing device, the point cloud data acquisition unit may acquire three-dimensional point cloud data of the first object from a measuring device that outputs three-dimensional point cloud data of the first object based on reflected light obtained by irradiating the first object with light. In the above information processing device, the measuring device may have an imaging unit that images the first object. The above information processing device may include a calibration unit that determines the positional relationship between the measuring device and the projection device. In the above information processing device, the projection device may project a calibration image including at least three line segments whose relative positions are known into three-dimensional space. In the above information processing device, the imaging unit may image the calibration image projected into three-dimensional space by the projection device. In the above information processing device, the calibration unit may determine the positional relationship between the measuring device and the projection device based on the calibration image imaged by the imaging unit.

In a second aspect of the present invention, an information processing method is provided. The information processing method includes, for example, a point cloud data acquisition step of acquiring three-dimensional point cloud data of a first object arranged in a three-dimensional space. The information processing method includes, for example, a modeling step of constructing a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired in the point cloud data acquisition step. The information processing method includes, for example, an instruction receiving step of receiving a first instruction from a first user for specifying a designated position, which is a point or area arranged on the surface of the three-dimensional model of the first object arranged in the virtual space. The information processing method includes, for example, a projection control step of controlling a projection device that projects an image to project the first image at a projection position, which is a point or area corresponding to the designated position on the surface of the first object.

In a third aspect of the present invention, a program is provided. The program may be a program for causing a computer to function as the information processing device according to the first embodiment. The program may be a program for causing a computer to execute the information processing method according to the second embodiment. A computer-readable medium for storing the program may be provided. The computer-readable medium may be a non-transitory computer-readable medium. The computer-readable medium may be a computer-readable recording medium.

Note that the above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions.

1 shows an example of a system configuration of a work support system 100. An example of a work object 102 is shown diagrammatically. An example of a support image 20 is shown diagrammatically. 13 shows an example of an operation screen 400 for operating the support image 20. 2 illustrates an example of the internal configuration of the lidar 124. 2 shows an example of an internal configuration of a work chamber control unit 162. An example of a calibration image 700 is shown diagrammatically. 2 shows an example of the internal configuration of a support terminal control unit 164. 13 shows an example of the internal configuration of the point cloud data output unit 540. 13 shows an overview of information processing in the variation index calculation unit 926. 13 shows an overview of information processing in the first correction unit 932. 13 shows an overview of information processing in the second correction unit 934 and the third correction unit 936. 13 shows an example of an image 1300 generated based on the value of the variability index. An example of the system configuration of a computer 3000 is shown in schematic form.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. In the drawings, the same reference numbers are used for the same or similar parts, and duplicate explanations may be omitted.

[Overview of work support system 100]
1 shows an example of a system configuration of a work support system 100. In this embodiment, the details of the work support system 100 will be described using as an example a case in which a worker 32 performs a specific task on a work target 102 inside a workroom 110 and a supporter 34 supports the worker 32 from a remote location. Although the details of the specific task are not limited, examples of the task include document creation, product manufacturing, product maintenance, and product repair.

Specifically, according to this embodiment, the support person 34 operates the support terminal 140 located away from the work room 110 to project the support image 20 for supporting the work on the work object 102 from the projector 126 located in the work room 110 toward an appropriate position on the surface of the work object 102 or in its vicinity. This allows the worker 32 to perform the work by referring to (i) the position where the support image 20 is projected, (ii) at least one of the color, shape, and pattern of the support image 20, and (iii) information presented by the support image 20.

For example, when a customer (an example of a worker 32) visiting a store prepares a document such as a contract or application form (an example of a work target 102) with the assistance of a clerk (an example of a supporter 34) working at another store away from the store, the customer may not be able to recognize the location where the seal should be placed. In such a case, when the clerk working at the other store accesses the work support system 100 using a terminal provided at the other store and specifies the location of the seal on the screen of the terminal, an image is projected onto the customer's document at the location where the seal should be placed to highlight the location. For example, a red donut-shaped figure is projected onto the customer's document at the location where the seal should be placed. The clerk then tells the customer, "Please put your seal in the red circle," and the customer can easily recognize the location where the seal should be placed.

According to this embodiment, the work support system 100 constructs a three-dimensional model of the periphery of the work object 102 in a virtual space to support the supporter 34 in specifying the projection position of the support image 20. This allows the supporter 34 to use the three-dimensional model of the work object 102 to instruct the support image 20 to be projected onto any point or area on or near the surface of the work object 102. For example, the supporter 34 can rotate the three-dimensional model of the work object 102 in the virtual space. Therefore, the supporter 34 can easily specify any position on the work object 102 while viewing the work object 102 from directly above.

In addition, when the work support system 100 detects the work object 102 or estimates the type of the work object 102 by image recognition processing, the work support system 100 can perform the above image recognition processing by comparing a teacher image obtained by capturing an image of the work object 102 from a specific direction with an image obtained by observing a three-dimensional model of the work object 102 from a specific direction. Therefore, the load of the image recognition processing is reduced and the accuracy of the image recognition processing is improved compared to when the above image recognition processing is performed by comparing a two-dimensional image of the work object 102 with a teacher image of the work object 102.

Furthermore, according to this embodiment, the work support system 100 (i) irradiates light onto one or more objects arranged in at least a part of the interior of the work room 110, and (ii) uses a measuring device that outputs three-dimensional point cloud data of at least a part of the interior of the work room 110 based on the light that is scattered or reflected back from each of a plurality of points on the surface of the one or more objects (sometimes referred to as reflected light), to construct a three-dimensional model of the periphery of the work target 102 in a virtual space. The work support system 100 calculates the depth of each point based on the light intensity of the reflected light from each of the above-mentioned plurality of points. At this time, the work support system 100 calculates an index (sometimes referred to as a variation index) that indicates the degree of variation in the light intensity of the reflected light at each point.

The inventors have found that the degree of variation in the intensity of reflected light at each point depends greatly on the material of the object. Although the mechanism is not limited to the following, it is presumed that the degree of variation in the intensity of reflected light depends greatly on the material of the object, since the reflectance or absorptance of light, and the characteristics related to diffuse reflection and/or regular reflection, vary depending on the material of the object.

According to this embodiment, based on the above knowledge, the work support system 100 uses the above-mentioned variation index when detecting the work object 102 from the three-dimensional point cloud data of at least a part of the interior of the workroom 110. As a result, even if it is difficult to detect the work object 102 using, for example, an RGB image or a distance image, the work support system 100 can relatively easily detect the work object 102 from the three-dimensional point cloud data of at least a part of the interior of the workroom 110.

For example, when the workroom 110 is dark, when strong light is shining around the work object 102, when the area around the work object 102 is covered by the shadow of another object, or when the pattern of the work object 102 is similar to the pattern around the work object 102, it is difficult to detect the boundary of the work object 102 from the RGB image of the work object 102 placed on the work table 112. On the other hand, when the work object 102 and the work table 112 are made of different materials, the above-mentioned variation index value differs between the points on the surface of the work object 102 and the points on the surface of the work table 112. Therefore, by using the above-mentioned variation index instead of or together with the RGB image, the boundary of the work object 102 can be detected relatively easily even in such cases.

Similarly, for example, if the thickness of the work object 102 is very small, it is difficult to detect the boundary of the work object 102 from the distance image of the work object 102 placed on the work table 112. On the other hand, by using the above-mentioned variation index instead of or in addition to the distance image, the boundary of the work object 102 can be detected relatively easily even in such a case.

[Overview of the system configuration of the work support system 100]
In this embodiment, the work support system 100 includes a work room 110, a support terminal 140, and a support server 160. In this embodiment, a work table 112 is provided inside the work room 110, and a work area 114 is set around the work table 112. In this embodiment, a display 122, a lidar 124, and a projector 126 are also provided inside the work room 110.

In this embodiment, the work object 102 is a tangible object that is the target of work performed by the worker 32. It is preferable that a material with a large diffuse reflectance compared to the regular reflectance is arranged on the surface of the work object 102. The surface of the work object 102 may be arranged with paper, a thin film of metal or metal oxide, a thin film of synthetic polymer or natural polymer, a plant leaf, a fungus, or the like. Examples of fungi include mold and mushrooms. The work object 102 may include at least one of paper, a thin film of metal or metal oxide, a thin film of synthetic polymer or natural polymer, a thin film of resin, a plant, a plant leaf, food, and a fungus. The work object 102 may be paper, a thin film of metal or metal oxide, a thin film of synthetic polymer or natural polymer, a thin film of resin, or a combination or aggregate thereof. The work object 102 may be a plant leaf or a food.

The thickness of the work object 102 may be less than 3 mm, less than 1 mm, less than 0.5 mm, less than 0.3 mm, or less than 0.1 mm. The work object 102 may be a collection of objects less than 3 mm thick, a collection of objects less than 1 mm thick, a collection of objects less than 0.5 mm thick, a collection of objects less than 0.3 mm thick, or a collection of objects less than 0.1 mm thick.

When the work object 102 is paper, a thin film of metal or metal oxide, a thin film of synthetic or natural polymer, or a combination or aggregate of these, the thickness of the work object 102 is very small. Therefore, for example, it is relatively difficult to detect or recognize the work object 102 based on the depth information shown by the distance image. However, as described below, according to one embodiment of the work support system 100, even if the thickness of the work object 102 is very small, the work object 102 can be detected or recognized by utilizing the difference between the light reflection or diffusion characteristics on the surface of the work object 102 and the light reflection or diffusion characteristics on the surfaces of objects surrounding the work object 102.

Furthermore, when the work object 102 is a plant leaf, the leaf surface may become discolored or bacteria may grow on the leaf surface due to disease. However, for example, the color of the leaf varies greatly depending on the imaging conditions (e.g., the brightness at the time of imaging), so it is relatively difficult to estimate the health state of the leaf or distinguish between healthy and diseased leaves based on the RGB image of the leaf. However, according to one embodiment of the work support system 100, it is possible to estimate the health state of the leaf and distinguish between healthy and diseased leaves by utilizing the difference between the light reflection or diffusion characteristics of the surface of a healthy leaf and the light reflection or diffusion characteristics of the surface of a diseased leaf.

Furthermore, when the work object 102 is food, the properties of the food may change due to spoilage or fermentation. However, the color change due to spoilage or fermentation is not necessarily large, and it is relatively difficult to detect the progress of spoilage or fermentation of the food based on an RGB image of the food. However, according to one embodiment of the work support system 100, the progress of spoilage or fermentation of food can be detected by utilizing the difference between the light reflection or diffusion characteristics of the surface of food that is not spoiled or fermented and the light reflection or diffusion characteristics of the surface of food that is spoiled or fermented.

In this embodiment, the support terminal 140 includes a support terminal 142 and a controller 144 that functions as part of the input device of the support terminal 142. In this embodiment, the support terminal 140 also includes a support terminal 146 and a pointing device 148 that functions as part of the input device of the support terminal 146.

In this embodiment, the support server 160 includes a workroom control unit 162 and a support terminal control unit 164. The various devices arranged in the workroom 110, the support terminal 140, and the support server 160 can send and receive information to each other via the communication network 10.

[Overview of each part related to the work support system 100]
In this embodiment, the communication network 10 may be a transmission path for wired communication, a transmission path for wireless communication, or a combination of a transmission path for wireless communication and a transmission path for wired communication. The communication network 10 may include a wireless packet communication network, the Internet, a P2P network, a dedicated line, a VPN, a power line communication line, etc.

The communication network 10 may include (i) a mobile communication network such as a mobile phone network, or (ii) a wireless communication network such as a wireless MAN (e.g., WiMAX (registered trademark)), a wireless LAN (e.g., WiFi (registered trademark)), Bluetooth (registered trademark), Zigbee (registered trademark), or NFC (Near Field Communication). Wireless LAN, Bluetooth (registered trademark), Zigbee (registered trademark), and NFC may be examples of short-range wireless communication.

In this embodiment, the support image 20 may include a graphic or may include a character string. In this embodiment, the support image 20 may include a message for the worker 32.

In this embodiment, the work chamber 110 and the work table 112 are used by the worker 32 to perform work. For example, the work object 102 is placed on the work table 112. It is preferable that the material of the work table 112 is different from the material of the work object 102. In this embodiment, the work area 114 includes at least a portion of the work object 102. The work area 114 may be a space including the entire work object 102 and at least a portion of the work table 112.

In this embodiment, the display 122 presents information to the worker 32. The display 122 may present information to the worker 32 by image and/or sound. For example, the display 122 outputs video and/or audio of the supporter 34. This allows the worker 32 to carry out work while talking with the supporter 34, as in a video conference or web conference.

In this embodiment, the LIDAR 124 acquires three-dimensional point cloud data of the interior of the work room 110. The LIDAR 124 may acquire three-dimensional point cloud data of the interior of the work area 114. For example, the LIDAR 124 irradiates light onto one or more objects disposed within the work area 114 and receives reflected light of the light. The LIDAR 124 outputs three-dimensional point cloud data of one or more objects disposed within the work area 114 based on the reflected light. Details of the LIDAR 124 will be described later.

In this embodiment, the projector 126 projects the support image 20 onto or near the surface of the work object 102. The projector 126 may project the support image 20 onto or near the surface of the work object 102 in accordance with instructions from the support server 160. The projector 126 may switch the projection of the support image 20 ON/OFF in accordance with instructions from the support server 160.

In this embodiment, the support terminal 140 is used by the supporter 34. The support terminal 140 may function as a user interface when the supporter 34 accesses the work support system 100. For example, the support terminal 140 is used by the supporter 34 to indicate the projection position of the support image 20. Examples of the support terminal 140 include a personal computer and a mobile terminal. Examples of the mobile terminal include a mobile phone, a smartphone, a PDA, a tablet, a notebook computer or laptop computer, and a wearable computer.

In this embodiment, the support terminal 142 includes equipment that provides the user with an XR experience, such as VR (virtual reality) goggles. XR may be a general term for VR (virtual reality), AR (augmented reality), MR (mixed reality), SR (alternative reality), etc. The support person 34 can use the controller 144 to specify the projection position of the support image 20 on the image of the virtual space displayed on the display of the support terminal 142.

In this embodiment, the support terminal 146 does not include equipment that provides an XR experience to the user. Even if the support terminal 146 does not include equipment that provides an XR experience to the user, the support terminal 146 can display an image of a virtual space on its display. In addition, the supporter 34 can use the pointing device 148 to specify the projection position of the support image 20 on the image of the virtual space displayed on the display of the support terminal 146.

In this embodiment, the support server 160 controls the projector 126 based on instructions from the supporter 34 to project the support image 20 at a position specified by the supporter 34. The support server 160 also recognizes each of the one or more objects arranged inside the work area 114 based on the three-dimensional point cloud data of the work area 114 output by the rider 124. For example, the support server 160 detects the work target 102 based on the three-dimensional point cloud data of the work area 114 output by the rider 124.

In one embodiment, the support server 160 acquires, for example, three-dimensional point cloud data of one or more objects placed in the working area 114. The support server 160 constructs a virtual space including a three-dimensional model of one or more objects placed in the working area 114 based on the three-dimensional point cloud data. The support server 160 also accepts various instructions from the supporter 34 via the support terminal 140. For example, the support server 160 accepts an instruction (sometimes referred to as a first instruction) from the support terminal 140 used by the supporter 34 to specify a point or area (sometimes referred to as a designated position) arranged on the surface of the three-dimensional model. The support server 160 controls the projector 126 to project the support image 20 onto a point or area (sometimes referred to as a projection position) corresponding to the designated position within the working area 114.

In another embodiment, the support server 160 acquires, for example, from the lidar 124, three-dimensional point cloud data of one or more objects placed in the work area 114. The three-dimensional point cloud data of the work area 114 includes three-dimensional position information indicating the three-dimensional positions of each of a plurality of points on the surface of one or more objects placed inside the work area 114. The three-dimensional point cloud data of the work area 114 also includes variation information indicating the degree of variation in the intensity of reflected light from each of a plurality of points on the surface of one or more objects placed inside the work area 114. The support server 160 detects the work object 102 based on the three-dimensional point cloud data of the work area 114. For example, the support server 160 detects at least a portion of the boundary of the work object 102 based on the three-dimensional position information and the variation information.

In this embodiment, the workroom control unit 162 controls the operation of each device arranged in the workroom 110. The workroom control unit 162 also acquires various information from each device arranged in the workroom 110. Details of the workroom control unit 162 will be described later.

In this embodiment, the support terminal control unit 164 controls the support terminal 140. The support terminal control unit 164 processes requests from the support terminal 140. For example, the support terminal control unit 164 accepts a request from the support terminal 140, and in response to the request from the support terminal 140, acquires, generates, or extracts information indicated by the request. The support terminal control unit 164 transmits the above information to the support terminal 140.

The support terminal control unit 164 also receives requests from the support terminal 140 to obtain information from each device arranged in the workroom 110, or requests to control the operation of each device arranged in the workroom 110. The support terminal control unit 164 outputs these requests to the workroom control unit 162. The support terminal control unit 164 may obtain information indicating the results of the above requests from the workroom control unit 162, and transmit the information to the support terminal 140. Details of the support terminal control unit 164 will be described later.

[Specific configuration of each part of the work support system 100]
Each part of the work support system 100 may be realized by hardware, software, or hardware and software. Each part of the work support system 100 may be realized, at least in part, by a single server or by multiple servers. Each part of the work support system 100 may be realized, at least in part, on a virtual machine or a cloud system. Each part of the work support system 100 may be realized, at least in part, by a personal computer or a mobile terminal. Examples of mobile terminals include mobile phones, smartphones, PDAs, tablets, notebook computers or laptop computers, and wearable computers. Each part of the work support system 100 may store information using a distributed ledger technology such as a blockchain or a distributed network.

When at least some of the components constituting the work assistance system 100 are realized by software, the components realized by the software may be realized by starting software or a program that defines the operation of the components in an information processing device of a general configuration. The information processing device of the general configuration described above may include (i) a data processing device having a processor such as a CPU or GPU, a ROM, a RAM, a communication interface, etc., (ii) input devices such as a keyboard, a pointing device, a touch panel, a camera, a voice input device, a gesture input device, various sensors, a GPS receiver, etc., (iii) output devices such as a display device, a voice output device, a vibration device, etc., and (iv) storage devices such as memory, HDD, SSD, etc. (including external storage devices).

In the information processing device of the above general configuration, the above data processing device or storage device may store the above software or program. The above software or program, when executed by a processor, causes the above information processing device to execute the operation defined by the software or program. The above software or program may be stored in a non-transitory computer-readable recording medium. The above software or program may be a program for causing a computer to function as the work support system 100 or a part thereof. The above software or program may be a program for causing a computer to execute an information processing method in the work support system 100 or a part thereof.

The above information processing method may be an information processing method in the support server 160. The above information processing method may be an information processing method in the workroom control unit 162. The above information processing method may be an information processing method in the support terminal control unit 164. The above information processing method may be an information processing method in the rider 124. The above information processing method may be an information processing method in the projector 126.

In one embodiment, the information processing method includes, for example, a point cloud data acquisition step of acquiring three-dimensional point cloud data of a first object arranged in a three-dimensional space. The information processing method includes, for example, a modeling step of constructing a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired in the point cloud data acquisition step. The information processing method includes, for example, an instruction receiving step of receiving a first instruction from a first user for identifying a designated position, which is a point or area arranged on the surface of the three-dimensional model of the first object arranged in the virtual space. The information processing method includes, for example, a projection control step of controlling a projection device that projects an image to project the first image at a projection position, which is a point or area corresponding to the designated position on the surface of the first object.

In another embodiment, the information processing method includes, for example, a point cloud data acquisition step of acquiring three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged. The information processing method includes, for example, an object detection step of detecting at least a part of a boundary of a first object included in one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired in the point cloud data acquisition step. In the information processing method, the point cloud data acquisition step includes, for example, a step of acquiring three-dimensional point cloud data of the three-dimensional space from a measuring device that outputs three-dimensional point cloud data of the three-dimensional space based on reflected light obtained by irradiating light on one or more objects. In the information processing method, the three-dimensional point cloud data of the three-dimensional space includes, for example, three-dimensional position information indicating the three-dimensional positions of each of a plurality of points on the surface of one or more objects. In the information processing method, the three-dimensional point cloud data of the three-dimensional space includes, for example, variation information indicating the degree of variation in the intensity of reflected light from each of the plurality of points. In the information processing method, the object detection step includes, for example, a step of detecting at least a part of the boundary of the first object based on the three-dimensional position information and the variation information.

The support image 20 may be an example of a first image. The supporter 34 may be an example of a first user. The work target 102 may be an example of a first object or one or more objects. The work target 102 may be an example of a first object included in one or more objects. The work room 110 may be an example of a three-dimensional space. The work table 112 may be an example of a first object or one or more objects. The work area 114 may be an example of a three-dimensional space. One or more objects arranged inside the work area 114 may be an example of a first object. The lidar 124 may be an example of a measuring device. The projector 126 may be an example of a projection device. The controller 144 may be an example of a controller used by the first user. The pointing device 148 may be an example of a controller used by the first user. The support server 160 may be an example of an information processing device or a detection device. The work room control unit 162 may be an example of an information processing device or a detection device.

[Overview of a method for supporting work using the work support system 100]
An overview of a method for supporting work using the work support system 100 will be described with reference to Figures 2, 3, and 4. In this embodiment, an example of a method for supporting work using the work support system 100 will be described, taking as an example a case where the work object 102 is a document such as a contract or an application form. As described above, when the work object 102 is an object with a small thickness such as a document or a film, it is difficult to detect the work object 102 based on a distance image.

Figure 2 shows an example of a work object 102. Figure 3 shows an example of a support image 20. Figure 4 shows an example of an operation screen 400 for operating the support image 20.

As shown in FIG. 2, a title 210 and an encoded code 212 are printed on the surface of the work object 102. The title 210 may be an example of identification information for identifying the type of work object 102. The encoded code 212 is generated by encoding the identification information for identifying the type of work object 102. Examples of the encoded code 212 include a barcode and a QR code (registered trademark).

A writing field 214, a signature field 216, and a seal field 218 are printed on the surface of the work object 102. The writing field 214 is used by the worker 32 to write a character string. The signature field 216 is used by the worker 32 to sign. The seal field 218 is used by the worker 32 to affix a seal. Writing in the writing field 214, signing in the signature field 216, and affixing a seal in the seal field 218 may be examples of work.

Next, an overview of the procedure for projecting the support image 20 onto the surface of the work object 102 will be described with reference to Figures 3 and 4. As shown in Figure 3, in this embodiment, an arrow-shaped marker 322 is projected at the upper left position of the signature field 216 of the work object 102 so that the supporter 34 can inform the worker 32 of the position of the signature field 216 on the work object 102. In addition, a pointer 324 having a frame shape slightly larger than the encoded code 212 is displayed at the position of the encoded code 212 of the work object 102 so that the supporter 34 can inform the worker 32 of the position of the work object 102 on the workbench 112.

In this embodiment, the marker 322 indicates the position of the signature section 216 on the work object 102. Therefore, when the work object 102 moves, it is preferable that the projection position of the marker 322 also moves to follow the movement of the work object 102. Furthermore, when the work object 102 moves in the real space, the change in the real space is also reflected in the virtual space. At this time, the position of the marker 322 in the virtual space may also move to follow the movement of the work object 102 in the virtual space.

In this embodiment, the pointer 324 indicates the position where the work target 102 is placed on the workbench 112. Therefore, it is preferable that the projection position of the pointer 324 does not change even if the work target 102 moves. The projection position of the pointer 324 in real space is changed, for example, when the supporter 34 executes an operation to change the position of the pointer 324 in the virtual space.

Therefore, in this embodiment, when the worker 32 moves the work object 102, the support server 160 controls the projector 126 so that the marker 322 follows the movement of the work object 102. On the other hand, even when the worker 32 moves the work object 102, the support server 160 controls the projector 126 so that the projection of the pointer 324 continues without changing the current projection conditions (e.g., at least one of the projection direction of the pointer 324 and the shape and size of the pointer 324). The support server 160 does not need to instruct the projector 126 to change the projection conditions of the pointer 324.

Figure 4 shows an example of an operation screen 400 for the supporter 34 to input instructions to the support server 160. An example of a manner in which the support server 160 supports the supporter 34 is described using Figure 4.

For example, if the supporter 34 wants to inform the worker 32 of the position of the signature field 216, the supporter 34 uses the support terminal 140 to access the support server 160. When the supporter 34 accesses the support server 160, the operation screen 400 shown in FIG. 4 is displayed on the display device of the support terminal 140. When the supporter 34 inputs various instructions into the operation screen 400 using the support terminal 140, the support terminal 140 transmits information indicating the content of the instructions of the supporter 34 to, for example, the support terminal control unit 164. The support terminal control unit 164 executes various processes according to the instructions (sometimes referred to as requests) of the supporter 34, and outputs information indicating the execution results to the workroom control unit 162 or the support terminal 140.

According to this embodiment, the assistant 34 uses the controller 144 or the pointing device 148 to specify a point or area (sometimes called a specified position) on the surface of the three-dimensional model of the work object 102 placed in the virtual space on the operation screen 400. As a result, for example, a marker 322 is projected onto a point or area (sometimes called a projection position) on the surface of the work object 102 placed in the real space.

Specifically, the support person 34 uses the controller 144 or the pointing device 148 to operate the cursor 402 displayed on the operation screen 400 to determine the designated position of the object 404 corresponding to the marker 322. Similarly, the support person 34 can determine various settings related to the object 404. In this embodiment, the cursor 402 indicates a specific point in the virtual space displayed in the virtual space display area 440. In this embodiment, the object 404 corresponds to the support image 20 in the virtual space that is projected into the real space. The shape, color, pattern, and attributes of the object 404 correspond to the shape, color, pattern, and attributes of the support image 20.

For example, the support person 34 can use the operation screen 400 to input instructions regarding (i) the position of the support image 20 on the work object 102, (ii) the size of the support image 20 on the work object 102, (iii) at least one of the shape, color, and pattern of the support image 20, and (iv) at least one of the attributes of the support image 20. Examples of the attributes of the support image 20 include (i) whether the support image 20 moves in conjunction with the movement of the work object 102 when the work object 102 moves, and (ii) whether the support image 20 is projected onto the surface of the work object 102 even during the period when the support person 34 is operating the support image 20 on the operation screen 400.

As shown in FIG. 4, in this embodiment, the operation screen 400 has a menu bar 420, a virtual space display area 440, and an object setting area 460. In this embodiment, an example of the operation screen 400 will be described using as an example a case in which the support image 20 is not projected onto the surface of the work target 102 while the supporter 34 is operating the support image 20 on the operation screen 400.

In this embodiment, the menu bar 420 displays one or more operation menus. Examples of operations include operations related to files, operations related to 3D models, operations related to the viewpoint when viewing a 3D model on the operation screen 400, operations related to images projected in real space (e.g., marker 322, pointer 324, etc.), and operations related to various settings. Examples of operations related to 3D models include operations for reconstructing or updating a 3D model based on the latest point cloud data in real space.

In this embodiment, a three-dimensional model of the work area 114 is displayed in the virtual space display area 440. For example, a three-dimensional model 442 of the workbench 112 and a three-dimensional model 444 of the work target 102 are displayed in the virtual space display area 440. Then, for example, the supporter 34 uses the controller 144 or the pointing device 148 to operate the position of the cursor 402 displayed on the operation screen 400, thereby determining the designated position of the object 404 corresponding to the marker 322.

According to this embodiment, a function for supporting the specification of a position by the cursor 402 in the virtual space display area 440 is provided. For example, in one embodiment, the support terminal control unit 164 supports the movement of the cursor 402 by the supporter 34 in the vicinity of the work target 102. In another embodiment, the support terminal control unit 164 supports the operation of the object 404 by the supporter 34 in the vicinity of the object 404. Details of the function for supporting the specification of a position by the cursor 402 will be described later.

The object setting area 460 is used to input various settings related to the object 404 placed in the virtual space. In this embodiment, the object setting area 460 is provided with a pull-down list 462 for selecting the attributes of the object 404, a pull-down list 464 for selecting the shape, color, and pattern of the object 404, and a button 466 for inputting instructions for projecting the support image 20 onto the work target 102 and for reflecting changes related to the support image 20.

According to this embodiment, the supporter 34 can specify the designated position by using the three-dimensional model of the work object 102. Therefore, the process for associating the designated position with the projection position is simplified compared to the case of specifying the designated position on a two-dimensional image. For example, the supporter 34 can rotate the three-dimensional model of the work object 102 on the screen of the support terminal 140, or enlarge a part of the three-dimensional model on the screen of the support terminal 140. The support server 160 may also detect an object pointed to by the worker 32 using a part of his/her body. Examples of the part of the body of the worker 32 include the face, eyes, arms, hands, and fingers. For example, the support server 160 displays an image or an enlarged image of the object in a part of the screen on which the three-dimensional model of the work object 102 is displayed. The support server 160 may display the image or the enlarged image of the object in a pop-up window.

An instruction regarding the position of the support image 20 on the work object 102 may be an example of a first instruction. An instruction regarding the attributes of the support image 20 may be an example of a second instruction. The cursor 402 may be an example of a mark.

Figure 5 shows an example of the internal configuration of the LIDAR 124. In this embodiment, the LIDAR 124 includes a distance measurement unit 520, an image capture unit 530, and a point cloud data output unit 540. In this embodiment, the distance measurement unit 520 includes an irradiation unit 522 and a light receiving unit 524.

In this embodiment, the distance measuring unit 520 outputs information for deriving the distance and angle from the reference point of the distance measuring unit 520 for each of a plurality of points on the surface of one or more objects arranged in the working area 114. The distance measuring unit 520 outputs, for example, information indicating the intensity of reflected light from each of a plurality of points on the surface of one or more objects arranged in the working area 114 (sometimes referred to as intensity information). The distance measuring unit 520 may output information for deriving the above-mentioned variation index for each of a plurality of points on the surface of one or more objects arranged in the working area 114.

In this embodiment, the irradiation unit 522 irradiates light onto one or more objects arranged in the working area 114. Examples of the light include ultraviolet light, visible light, and near-infrared light. The light may be a laser. The light may be pulsed light. The irradiation unit 522 includes, for example, a plurality of light-emitting elements arranged in an array, and an optical system that directs the light emitted by the plurality of light-emitting elements toward the scanning position. Each of the plurality of light-emitting elements is configured to emit light at a predetermined angle with respect to a reference plane of the irradiation unit 522.

The irradiation unit 522 outputs, for example, information indicating the time when the pulsed light was emitted and the scanning position at that time. The information indicating the scanning position may be the angle between a vector indicating the traveling direction of each light and a normal vector of a reference plane of the irradiation unit 522 that passes through a reference point arranged on the reference plane. Note that the angle between two vectors indicates an angle of 180° or less made by the two vectors when at least one of the two vectors is translated and the starting point of one vector is superimposed on the starting point of the other vector.

In this embodiment, the light receiving unit 524 is disposed near the irradiation unit 522. The light irradiated by the irradiation unit 522 to the one or more objects is scattered or reflected by the surface of the one or more objects and returns to the light receiving unit 524. The light receiving unit 524 receives the light that is scattered or reflected by the surface of the one or more objects and returns (sometimes referred to as reflected light from the object).

The light receiving unit 524 outputs the fluctuation in the received light intensity of the reflected light. The light receiving unit 524 outputs, for example, information indicating the time and the received light intensity of the reflected light at that time.

In this embodiment, the imaging unit 530 images the working area 114. The imaging unit 530 may image one or more objects arranged in the working area 114. The imaging unit 530 may acquire color information indicating the color of each of a plurality of points on the surface of the one or more objects arranged in the working area 114.

In this embodiment, the point cloud data output unit 540 generates color point cloud data 560 based on the information output by the distance measurement unit 520 and the information output by the imaging unit 530. As shown in FIG. 5, the color point cloud data 560 stores, for each of a plurality of points on the surface of one or more objects arranged in the working area 114, identification information of each point, three-dimensional position information of each point, color information of each point, and variation information of each point in association with each other. The color point cloud data 560 may further store intensity information indicating the intensity of reflected light from each of a plurality of points on the surface of one or more objects arranged in the working area 114 in association with each other.

Specifically, for example, the point cloud data output unit 540 calculates a depth indicating the distance from a reference point of the distance measuring unit 520 for each of a plurality of points on the surface of one or more objects arranged in the working area 114 based on the information output by the distance measuring unit 520. The point cloud data output unit 540 may calculate a depth indicating the distance from a reference point of the distance measuring unit 520 for each of the above-mentioned plurality of points based on the time when the distance measuring unit 520 irradiates light and the time when the light receiving unit 524 receives the reflected light of the light. The point cloud data output unit 540 may derive the three-dimensional position of each of the above-mentioned plurality of points based on the above-mentioned depth.

The point cloud data output unit 540 also calculates a variation index value that indicates the degree of variation in the intensity of reflected light from each of the above-mentioned multiple points based on the depth fluctuation in a unit period having a predetermined length. Examples of the above-mentioned index include variance, deviation, and standard deviation.

The point cloud data output unit 540 generates color point cloud data 560 by associating the identification information of each point, the three-dimensional position information indicating the three-dimensional position of each point, the variation information indicating the value of the variation index of each point, and the color information of each point. The point cloud data output unit 540 may output the color point cloud data 560 to the support server 160. This allows the LIDAR 124 to output three-dimensional point cloud data of one or more objects placed in the working area 114 based on the reflected light received by the light receiving unit 524.

[An example of another embodiment]
In the present embodiment, an example of the work support system 100 has been described by taking as an example a case in which variation information is generated inside the LIDAR 124 based on the intensity information output by the light receiving unit 524. However, the work support system 100 is not limited to this embodiment. In other embodiments, the support server 160 may acquire the intensity information output by the light receiving unit 524 and generate variation information based on the intensity information.

The point cloud data output unit 540 may be an example of a depth calculation unit or an index calculation unit. The color point cloud data 560 may be an example of three-dimensional point cloud data. The distance between each of the multiple points on the surface of one or more objects arranged in the working area 114 and the reference point of the distance measurement unit 520 may be an example of the distance between each of the multiple points on the surface of one or more objects arranged in the working area 114 and the irradiation unit 522 or the light receiving unit 524.

Figure 6 shows an example of the internal configuration of the workroom control unit 162. In this embodiment, the workroom control unit 162 includes a point cloud data acquisition unit 622, an object detection unit 632, a modeling unit 634, a type estimation unit 636, an instruction reception unit 642, a projection control unit 644, a calibration unit 650, and a storage unit 660.

In this embodiment, the point cloud data acquisition unit 622 acquires the three-dimensional point cloud data output by the LIDAR 124. For example, the point cloud data acquisition unit 622 acquires color point cloud data 560 related to one or more objects disposed within the working area 114.

The object detection unit 632 detects at least one of the one or more objects (e.g., the work target 102) arranged inside the working area 114 based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622. As described above, the three-dimensional point cloud data includes (i) three-dimensional position information or depth information and (ii) at least one of color information, variation information, and intensity information regarding each of a plurality of points on the surface of the one or more objects arranged inside the working area 114. In one embodiment, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on the three-dimensional position information or depth information. In another embodiment, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on the color information. In yet another embodiment, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on the variation information.

In yet another embodiment, the object detection unit 632 may detect at least one of the one or more objects disposed within the working area 114 based on (i) three-dimensional position information or depth information, and (ii) at least one of color information, variation information, and intensity information. For example, the object detection unit 632 first performs an object detection process based on the three-dimensional position information or depth information. If the detection accuracy of at least some objects is lower than a predetermined value, the object detection unit 632 may detect at least one of the one or more objects disposed within the working area 114 based on at least one of color information, variation information, and intensity information.

In yet another embodiment, the object detection unit 632 may detect at least one of the one or more objects disposed within the working area 114 based on (i) color information and (ii) at least one of the three-dimensional position information or depth information, the variation information, and the intensity information. For example, the object detection unit 632 first performs an object detection process based on the color information. If the detection accuracy of at least some objects is lower than a predetermined value, the object detection unit 632 may detect at least one of the one or more objects disposed within the working area 114 based on at least one of the three-dimensional position information or depth information, the variation information, and the intensity information.

In yet another embodiment, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on (i) the variation information and (ii) at least one of the three-dimensional position information or depth information, color information, and intensity information. For example, the object detection unit 632 first performs an object detection process based on the variation information. If the detection accuracy of at least some objects is lower than a predetermined value, the object detection unit 632 may detect at least one of the one or more objects arranged inside the working area 114 based on at least one of the three-dimensional position information or depth information, color information, and intensity information.

In this embodiment, when it is known that the surfaces of at least the work target 102 and the work table 112 among one or more objects arranged inside the work area 114 are flat, the object detection unit 632 may detect the work target 102 according to the following procedure. First, the object detection unit 632 uses the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622 before the work target 102 is placed on the work table 112 to approximate the point cloud around the work table 112 with an arbitrary plane formula (sometimes referred to as an approximate plane formula).

Next, after the work target 102 is placed on the work table 112, the object detection unit 632 reads the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622. Assuming that the work table 112 and the work target 102 are flat, the object detection unit 632 performs projection transformation of the three-dimensional point cloud data near the work table 112 among the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622 into the above-mentioned approximate plane formula. Projection transformation is a coordinate transformation of the point cloud data into a flat and stretched state while maintaining the intermolecular distance of the physically curved, uneven, or bent banknote, as if it has been ironed, even when the work target 102 is a repeatedly curled banknote, etc., thereby improving the accuracy of classification or estimation by the object detection unit 632 and/or type estimation unit 636.

In this embodiment, the object detection unit 632 detects, for example, the work object 102 based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622. The object detection unit 632 may detect the work object 102 by detecting at least a portion of the boundary of the work object 102 based on the above-mentioned color point cloud data 560.

The object detection unit 632 may detect at least a portion of the boundary of the work object 102 based on the three-dimensional position information and variation information included in the color point cloud data 560. The object detection unit 632 may detect at least a portion of the boundary of the work object 102 based on the three-dimensional position information, variation information, and color information included in the color point cloud data 560.

For example, the object detection unit 632 extracts points on the surface of the work object 102 from among the multiple points indicated by the variation information included in the color point cloud data 560, based on the value of the variation index for each of the multiple points. As described above, the value of the variation index differs depending on the material of the object. Therefore, by using the value of the variation index for each point, the object detection unit 632 can distinguish the work object 102 from other objects (e.g., the workbench 112) placed in the work area 114 based on the difference in material between the work object 102 and other objects placed in the work area 114. This allows the object detection unit 632 to detect the work object 102 from one or more objects placed in the work area 114.

More specifically, the object detection unit 632 first extracts at least one point from the above multiple points, the value of which for the variation index is within a predetermined first range. If the range of possible values for the variation index of the work object 102 is known, the object detection unit 632 may use the known range as the first range for the possible values of the variation index of the work object 102. If the range of possible values for the above variation index for a material whose diffuse reflectance characteristics are similar to those of the work object 102 is known, the object detection unit 632 may use the known range as the first range.

In particular, when the diffuse reflectance of the work object 102 to be detected is sufficiently large compared to the specular reflectance of the work object 102, the variation index is expected to be proportional to the diffuse reflectance. The above first range may be determined taking into account the diffuse reflectance of the work object 102 to be detected by the object detection unit 632.

Next, for each of the at least one extracted point, the object detection unit 632 compares the value of the variance index of each point with the value of the variance index of a point adjacent to each point. If the absolute value of the difference between the two is smaller than a predetermined value, the object detection unit 632 determines that the two are points on the surface of the same object. On the other hand, if the absolute value of the difference between the two is greater than a predetermined value, the object detection unit 632 determines that the two are points on the surface of different objects. This allows the object detection unit 632 to distinguish between at least two objects included in one or more objects arranged inside the working area 114.

If a database that stores the type of material in association with the range of values that the variation index can take is available, the object detection unit 632 may simplify the comparison process of the values of the variation index at the two adjacent points. For example, the object detection unit 632 first uses the material of the work object 102 as a key, refers to the database, and extracts the range of values that the variation index of the work object 102 can take. Next, the object detection unit 632 extracts one or more points whose variation index values are included in the range of the extracted range of values from among a plurality of points on the surface of one or more objects arranged in the work area 114. Next, the object detection unit 632 extracts a point on the surface of the work object 102 from the extracted one or more points. The object detection unit 632 may (i) perform a comparison process similar to the above for each of the extracted one or more points, or (ii) detect a closed figure formed by at least a part of the extracted one or more points as the work object 102.

The object detection unit 632 may detect the contour of one or more objects arranged inside the working area 114 and detect the one or more objects. The object detection unit 632 may assign identification information to each of the detected one or more objects. The object detection unit 632 may store the identification information of each object in association with the characteristic information of the appearance of each object. The characteristic information of the appearance of each object is determined, for example, based on an image of each object captured by the imaging unit 530. As a result, for example, when there are two transfer slips of the same type inside the working area 114, the object detection unit 632 can recognize the two transfer slips as two different instances. For example, when the worker 32 writes characters on one of the transfer slips, the characteristic information of the transfer slip with the characters written on it differs from the characteristic information of the transfer slip without the characters written on it. Therefore, even if the positions of the two transfer slips have changed between the first time, which is the most recent time that the object detection process was performed, and the second time, which is the last time that the object detection process was performed before the first time, the object detection unit 632 can identify the two transfer slips based on the differences in the above-mentioned characteristic information.

In this embodiment, the modeling unit 634 constructs a virtual space including a three-dimensional model of one or more objects arranged inside the work area 114 based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit 622. The modeling unit 634 may generate a three-dimensional model of each of the objects detected by the object detection unit 632. For example, the modeling unit 634 constructs a virtual space including a three-dimensional model of the work object 102. The modeling unit 634 may also construct a virtual space including a three-dimensional model of the work object 102 and a three-dimensional model of the workbench 112.

In this embodiment, the type estimation unit 636 classifies or estimates the type of the object detected by the object detection unit 632. The type estimation unit 636 may classify or estimate the type of the object detected by the object detection unit 632 using a known machine learning method. The type estimation unit 636 may classify or estimate the type of the object using a neural network classifier such as deep learning. The type estimation unit 636 may classify or estimate the type of the object based on an image of the object captured by the imaging unit 530. The type estimation unit 636 may classify or estimate the type of the object based on the boundary of the object detected by the object detection unit 632 and the image of the object captured by the imaging unit 530.

According to this embodiment, the object detection unit 632 detects the boundaries of one or more objects arranged in the working area 114 based on the value of the variation index. The type estimation unit 636 classifies or estimates the type of each object using an image of the inside of the boundary of each object, thereby improving the accuracy of the classification or estimation.

In this embodiment, the instruction receiving unit 642 receives various instructions from the supporter 34. The instruction receiving unit 642 may receive instructions that the supporter 34 inputs onto the operation screen 400 using the support terminal 140.

The instruction receiving unit 642 may receive an instruction (sometimes referred to as a first instruction) for identifying the indicated position described above. For example, the support person 34 can place the object 404 at any position on the three-dimensional model by performing a drag-and-drop operation or a similar operation on the object 404 on the operation screen 400.

When the supporter 34 has completed the placement operation of the object 404, the instruction receiving unit 642 acquires information from the support terminal 140 indicating that the object 404 has been placed at a specific position of the three-dimensional model displayed in the virtual space display area 440 of the operation screen 400. For example, the supporter 34 completes the placement operation of the object 404 by performing an operation of releasing the button of the pointing device 148 during a drag-and-drop operation of the object 404. As a result, the instruction receiving unit 642 receives an instruction to specify the position where the object 404 has been placed on the three-dimensional model as the above-mentioned designated position.

The instruction receiving unit 642 may receive an instruction (sometimes referred to as a second instruction) to identify the attribute of the support image 20. For example, the supporter 34 operates the pull-down list 462 for selecting the attribute of the object 404 on the operation screen 400 to select the attribute of the object 404. When the supporter 34 completes the selection operation of the attribute of the object 404, the instruction receiving unit 642 acquires information from the support terminal 140 indicating that the attribute of the object 404 has been determined. As a result, the instruction receiving unit 642 receives an instruction to identify the attribute selected on the operation screen 400 as the attribute of the support image 20.

The instruction receiving unit 642 receives, for example, (i) the relative position of the indicated position with respect to a reference point in the three-dimensional model of the work target 102, or (ii) an instruction to fix the relative position of the indicated position with respect to a reference point in the virtual space (sometimes referred to as a third instruction). The third instruction may be (i) a clear instruction indicating that the placement of the object 404 is complete, or (ii) that the operation for placing the object 404 is completed. The drag-and-drop operation of the object 404 may be an example of an operation for placing the object 404. In this case, the operation of the assistant 34 releasing the button of the pointing device 148 during the drag-and-drop operation of the object 404 may be an example of the third instruction. An example of a clear instruction indicating that the placement of the object 404 is completed is pressing the button 466.

As described above, depending on the attributes of the object 404, the support image 20 is not projected onto the surface of the work target 102 while the support person 34 is operating the support image 20 on the operation screen 400. In such a case, the third instruction may be an instruction for reflecting a change in the position or setting of the object 404 in the support image 20. Note that whether or not the support image 20 is projected onto the surface of the work target 102 while the support person 34 is operating the support image 20 on the operation screen 400 may be determined by the settings of the operation screen 400.

The instruction receiving unit 642 may receive an instruction (sometimes referred to as a fourth instruction) regarding starting or stopping the projection of the support image 20. An example of the fourth instruction is when the button 466 is pressed.

In this embodiment, the projection control unit 644 controls the operation of the projector 126. The projection control unit 644 controls the projector 126 to project the support image 20, for example, on the surface of the work object 102, onto a point or area (sometimes referred to as a projection position) that corresponds to a designated position on the three-dimensional model of the work object 102.

The projection control unit 644 may control at least one of the projection direction of the support image 20 and the shape and size of the support image 20. As described above, for example, when the work object 102 moves in real space, the projection control unit 644 controls the projector 126 so that the support image 20 projected onto the work object 102 follows the movement of the work object 102.

More specifically, when the attribute of the support image 20 indicated by the second instruction received by the instruction receiving unit 642 is the first attribute, the projection control unit 644 adjusts the projection direction of the support image 20 in the projector 126 and at least one of the shape and size of the support image 20 so that the support image 20 projected onto the work object 102 follows the movement of the work object 102 when the work object 102 moves in real space. The projection control unit 644 may control the projector 126 as described above when the instruction receiving unit 642 receives a third instruction and the attribute of the support image 20 indicated by the second instruction is the first attribute.

On the other hand, if the attribute of the support image 20 indicated by the second instruction received by the instruction receiving unit 642 is the first attribute, the projection control unit 644 does not adjust the projection direction of the support image 20 in the projector 126 or the shape and size of the support image 20, even if the work object 102 moves in real space. The projection control unit 644 may control the projector 126 as described above when the instruction receiving unit 642 receives a third instruction and the attribute of the support image 20 indicated by the second instruction is the second attribute.

The projection control unit 644 may control the start or stop of projection of the support image 20. For example, when the instruction receiving unit 642 receives the above-mentioned fourth instruction, the projection control unit 644 may decide whether or not to project the support image 20 onto the work target 102 based on the fourth instruction. The projection control unit 644 may control the operation of the projector 126 based on the above-mentioned decision.

The projection control unit 644 may control the brightness of the support image 20. In one embodiment, the projection control unit 644 adjusts the brightness of the support image 20 based on the brightness of the surroundings of the projection position in real space. For example, the projection control unit 644 determines whether the brightness of the surroundings of the projection position is brighter than a predetermined level based on the image data output by the imaging unit 530. If it is determined that the brightness of the surroundings of the projection position is brighter than a predetermined level, the projection control unit 644 reduces the intensity of the projection light of the support image 20. If it is determined that the brightness of the surroundings of the projection position is darker than a predetermined level, the projection control unit 644 increases the intensity of the projection light of the support image 20.

In another embodiment, the projection control unit 644 controls the brightness of at least a portion of the support image 20 in conjunction with the operation of the type estimation unit 636. The projection control unit 644 may control the brightness of a portion of the support image 20 in conjunction with the operation of the type estimation unit 636, or may control the brightness of the entire support image 20. For example, the projection control unit 644 controls the brightness of the portion of the support image 20 that is projected onto the object of the estimation process by the type estimation unit 636. This suppresses a decrease in estimation accuracy caused by the support image 20 being projected onto the object of the estimation process by the type estimation unit 636.

For example, when the type estimation unit 636 executes a process of estimating the type of the work object 102 based on the image data output by the imaging unit 530, the projection control unit 644 stops the projection of the support image 20 or reduces the intensity of the projection light of the support image 20. Note that the length of the period during which the projection of the support image 20 is stopped or the period during which the intensity of the projection light of the support image 20 is reduced may be as long as the imaging unit 530 can image the work object 102, and may be, for example, less than one second.

This allows the type estimation unit 636 to estimate the type of the work object 102 based on image data of the work object 102 captured without the support image 20 being projected. For example, if the work object 102 is a playing card and the shape of the marker 322 or pointer 324 is heart-shaped, the estimation accuracy may decrease if the type estimation unit 636 estimates the type of the work object 102 based on image data captured with the marker 322 or pointer 324 projected onto the work object 102. According to this embodiment, when the type estimation unit 636 executes the above estimation process, for example, the projection of the support image 20 is stopped, and therefore the estimation accuracy by the type estimation unit 636 is improved.

Note that the projection control unit 644 may stop the projection of the support image 20 or reduce the intensity of the projection light of at least a part of the support image 20 when the object of the estimation process by the type estimation unit 636 is the same as the object onto which the support image 20 is projected. When the object of the estimation process by the type estimation unit 636 is different from the object onto which the support image 20 is projected, the projection control unit 644 may not stop the projection of the support image 20 or reduce the intensity of the projection light of the support image 20.

For example, in a case where five objects including object A and object B are arranged inside the working area 114, and part of the support image 20 is projected onto object A but not onto object B, during a period in which the imaging unit 530 is imaging the inside of the working area 114 in order for the type estimation unit 636 to estimate the type of object A, the projection control unit 644 stops projecting the support image 20 or reduces the intensity of the projection light of at least a part of the support image 20. On the other hand, in the above case, during a period in which the imaging unit 530 is imaging the inside of the working area 114 in order for the type estimation unit 636 to estimate the type of object B, the projection control unit 644 does not need to stop projecting the support image 20 or reduce the intensity of the projection light of at least a part of the support image 20.

In this embodiment, the calibration unit 650 performs calibration to determine the positional relationship between the LIDAR 124 and the projector 126. For example, the calibration unit 650 first controls the projector 126 to project a calibration image including at least three line segments whose relative positions are known, onto any plane in real space. Next, the calibration unit 650 controls the imaging unit 530 of the LIDAR 124 to capture the above-mentioned calibration image. The calibration unit 650 determines the positional relationship between the LIDAR 124 and the projector 126 based on the calibration image captured by the imaging unit 530.

In this embodiment, the storage unit 660 stores various types of data. The storage unit 660 may store various types of data used for information processing in the support server 160. The storage unit 660 may store various types of data generated by information processing in the support server 160. The storage unit 660 may store three-dimensional model data of one or more objects arranged inside the working area 114. The storage unit 660 may store various types of data used for displaying the operation screen 400. The storage unit 660 may store data related to various settings determined using the operation screen 400.

[An example of another embodiment]
In the present embodiment, an example of the workroom control unit 162 has been described, taking as an example a case where the supporter 34 inputs an instruction to the support terminal 140 using the controller 144 or the pointing device 148, and the instruction receiving unit 642 receives information indicating the content of the instruction (sometimes referred to as a request) from the support terminal 140. However, the workroom control unit 162 is not limited to the present embodiment. In other embodiments, the instruction of the supporter 34 may be acquired by a device other than the controller 144 or the pointing device 148. For example, a single or multiple cameras arranged on the frame of glasses worn by the supporter 34 detect the shape or movement of the arm, hand, or finger of the supporter 34, and the instruction of the supporter 34 is input to the support terminal 140.

FIG. 7 shows a schematic diagram of an example of a calibration image 700. As shown in FIG. 7, the calibration image 700 includes a T-shaped figure. As shown in FIG. 7, the T-shaped figure includes three line segments, AB, BC, and BD, whose relative positions are known. This allows the calibration unit 650 to identify the position of the projector 126 based on (i) the size of the angle AOB between AB and the reference point O of the projector 126, (ii) the size of the angle BOC between BC and the reference point O of the projector 126, and (iii) the size of the angle BOD between BD and the reference point O of the projector 126.

Figure 8 shows an example of the internal configuration of the support terminal control unit 164. In this embodiment, when the supporter 34 operates a mark (which may be called a pointer, cursor, etc.) to point to a specific point on the screen of the support terminal 140, an example of the details of the support terminal control unit 164 is described using as an example a case in which the supporter 34 operates a mark (which may be called a pointer, cursor, etc.) to point to a specific point on the screen of the support terminal 140, and the support terminal control unit 164 supports the supporter 34 in operating the mark.

In this embodiment, the support terminal control unit 164 includes a setting storage unit 810, an indication mark control unit 820, an image data acquisition unit 832, an object estimation unit 834, and a setting extraction unit 836. In this embodiment, the setting storage unit 810 includes a first setting information storage unit 812 and a second setting information storage unit 814.

In this embodiment, the setting storage unit 810 stores information about various settings used by the information processing in the indication mark control unit 820. The operation of moving the cursor 402 in a three-dimensional virtual space is often more difficult than the operation of moving a mouse cursor on a two-dimensional screen. By applying the above settings, it can become easier to operate the cursor 402 in the three-dimensional virtual space.

In this embodiment, the first setting information storage unit 812 stores, for example, first setting information that specifies the movement of the cursor 402 near the boundary of the three-dimensional model of the work target 102. The first setting information storage unit 812 may store the above setting information for each type of work target 102, or may store the above setting information as an initial setting that is applied regardless of the type of the initial work target 102. Examples of the first setting information include (i) a setting regarding the correspondence between the distance from the boundary and the DPI of the controller 144 or the pointing device 148, (ii) a setting regarding whether or not the cursor 402 performs a bouncing action near the boundary, (iii) a setting regarding the bouncing mode when the cursor 402 is bounced, (iv) a setting regarding whether or not the cursor 402 is moved along the boundary for a predetermined period after the cursor 402 reaches the boundary near the boundary, and (v) a setting regarding the movement mode when the cursor 402 is moved along the boundary.

In this embodiment, the second setting information storage unit 814 stores information indicating the type of each of the one or more objects, and second setting information that specifies the movement of the cursor 402 at one or more positions of the three-dimensional model of each object corresponding to one or more positions on the surface of each object. Taking the work target 102 described in relation to FIG. 2 and FIG. 3 as an example, the one or more positions on the surface of each object are exemplified as the entry field 214, the signature field 216, the seal field 218, etc. The entry field 214, the signature field 216, and the seal field 218 are used relatively frequently. Therefore, if these positions can be easily specified as the above-mentioned specified positions, the burden on the supporter 34 will be greatly reduced. Examples of the second setting information include (i) a setting regarding the correspondence between the above-mentioned positions and the DPI of the controller 144 or the pointing device 148, and (ii) a setting regarding whether or not to fix the cursor 402 at the above-mentioned position or another predetermined position for a predetermined period when the cursor 402 reaches the above-mentioned position.

The indication mark control unit 820 provides a function for supporting the specification of a position by the cursor 402 in the virtual space display area 440. For example, the indication mark control unit 820 controls the movement of the cursor 402 based on a control signal output by the controller 144 or the pointing device 148.

In one embodiment, the indication mark control unit 820 supports the movement of the cursor 402 by the assistant 34 near the work target 102. The indication mark control unit 820 may switch the control method depending on whether the cursor 402 is located near the boundary of the three-dimensional model or not. For example, when the cursor 402 is located near the boundary of the three-dimensional model, the indication mark control unit 820 controls the movement of the cursor 402 based on the first setting information. For example, when the position of the object 404 approaches the boundary of the 3D model of the work target 102, the indication mark control unit 820 changes the setting related to the DPI of the controller 144 or the pointing device 148 to reduce the sensitivity of the cursor 402.

The indication mark control unit 820 may switch the control method depending on whether the cursor 402 is at a predetermined position on the three-dimensional model or at a position other than the predetermined position on the three-dimensional model. For example, when the cursor 402 is at a predetermined position on the three-dimensional model, the indication mark control unit 820 controls the movement of the cursor 402 based on the second setting information extracted by the setting extraction unit 836. For example, when the position of the object 404 approaches a position corresponding to the signature field 216 on the 3D model of the work target 102, the indication mark control unit 820 moves the cursor 402 to the upper left position of the signature field 216.

In another embodiment, the instruction mark control unit 820 supports the assistant 34 in operating the object 404 in the vicinity of the object 404. When the cursor 402 approaches the vicinity of an already placed object 404, the instruction mark control unit 820 may change the mode of the workroom control unit 162 to a mode for performing an operation according to the attributes of the object 404. When the cursor 402 moves away from the object 404, the instruction mark control unit 820 may change the mode of the workroom control unit 162 to a mode determined by initial settings.

For example, if the existing object 404 is a marker 322, when the cursor 402 approaches the object, the indication mark control unit 820 causes the object setting area 460 of the operation screen 400 to display a screen for inputting various settings related to the marker. On the other hand, if the existing object 404 is a pointer 324, when the cursor 402 approaches the object, the indication mark control unit 820 causes the object setting area 460 of the operation screen 400 to display a screen for inputting various settings related to the pointer.

In this embodiment, the image data acquisition unit 832 acquires image data of one or more objects arranged in the working area 114. The image data acquisition unit 832 may acquire color information of the color point cloud data 560 of one or more objects arranged in the working area 114 as image data of the objects. In this embodiment, the object estimation unit 834 estimates the type of each of the one or more objects arranged in the working area 114 based on the image data acquired by the image data acquisition unit 832.

In this embodiment, the setting extraction unit 836 refers to the second setting information storage unit 814 of the setting storage unit 810 to extract second setting information corresponding to the type of each object estimated by the object estimation unit 834. As described above, the second setting information is stored in association with information indicating the type of each of the one or more objects and second setting information that specifies the movement of the cursor 402 at each of one or more positions of the three-dimensional model of each object that corresponds to each of one or more positions on the surface of each object.

Figure 9 shows an example of the internal configuration of the point cloud data output unit 540. In this embodiment, the point cloud data output unit 540 includes a depth calculation unit 922, a three-dimensional position information calculation unit 924, a variation index calculation unit 926, and a variation correction unit 930. In this embodiment, the variation correction unit 930 includes a first correction unit 932, a second correction unit 934, a third correction unit 936, and a fourth correction unit 938.

In this embodiment, the depth calculation unit 922 calculates a depth indicating the distance between each point and the reference point of the distance measurement unit 520 for each of the multiple points based on information indicating the intensity of reflected light from each of the multiple points that scatter or reflect the light emitted by the irradiation unit 522. For example, the depth calculation unit 922 calculates the depth of each of the multiple points based on the time when the distance measurement unit 520 irradiates light and the time when the light receiving unit 524 receives the reflected light of the light. For example, when the distance measurement unit 520 scans the inside of the working area 114, the depth calculation unit 922 calculates the above-mentioned depth for each of the multiple points on the surface of one or more objects arranged inside the working area 114.

In this embodiment, the three-dimensional position information calculation unit 924 calculates the three-dimensional position of each of the multiple points based on the depth of each of the multiple points calculated by the depth calculation unit 922. As described above, the irradiation unit 522 outputs information indicating the time when the pulsed light is emitted and the scanning position at that time. The three-dimensional position information calculation unit 924 calculates the distance and angle from the reference point of the distance measurement unit 520 for each of the multiple points based on the information indicating the scanning position and the depth calculated by the depth calculation unit 922.

In one embodiment, the three-dimensional position information calculation unit 924 outputs a three-dimensional position of each of the above multiple points based on the reference point of the distance measurement unit 520, based on the distance and angle from the reference point of the distance measurement unit 520. In another embodiment, if the absolute coordinates of the reference point of the distance measurement unit 520 are known, the three-dimensional position information calculation unit 924 may output the absolute coordinates of each of the above multiple points based on the distance and angle from the reference point of the distance measurement unit 520 and the absolute coordinates of the reference point of the distance measurement unit 520.

In this embodiment, the variation index calculation unit 926 calculates a variation index value indicating the degree of variation in the intensity of reflected light from each of the multiple points based on the depth of each of the multiple points calculated by the depth calculation unit 922. The variation index calculation unit 926 may calculate the variation index value at each point based on the fluctuation in depth at each point. For example, the variation index calculation unit 926 calculates the variation index value for each of the multiple points based on the fluctuation in depth in a unit period having a predetermined length. Examples of the above index include variance, deviation, and standard deviation.

In this embodiment, the variance correction unit 930 corrects the value of the variance index calculated by the variance index calculation unit 926. As described above, the intensity of the reflected light emitted from the distance measurement unit 520 and scattered or reflected on the surface of the work object 102 and returned is significantly affected by the material of the work object 102. This is presumably due to differences in the diffuse reflectance, specular reflectance, and absorptance for light having a specific wavelength, depending on the type of material placed on the surface of the work object 102 and the structure of the surface.

In addition to the material of the work object 102, there are other factors (sometimes referred to as error factors) that can cause errors in the value of the variation index. The variation correction unit 930 corrects the value of the variation index taking into account the above error factors. In one embodiment, the variation correction unit 930 derives a correction coefficient k for correcting the value S of the variation index calculated by the variation index calculation unit 926. The variation correction unit 930 may calculate the above correction coefficient k using a function in which the correction coefficient k is the objective variable and one or more error factors are explanatory variables. In addition, the variation correction unit 930 calculates the corrected value Sd of the variation index by multiplying the above value S by the correction coefficient k. In another embodiment, the variation correction unit 930 derives a function in which the corrected value Sd of the variation index is the objective variable and the above value S and one or more error factors are explanatory variables. The variance correction unit 930 calculates the corrected variance index value Sd based on the function.

In this embodiment, the first correction unit 932 corrects the value of the variation index calculated by the variation index calculation unit 926 based on the distance between the reference point of the distance measurement unit 520 and the measurement target surface of the surface of the work target 102. The first correction unit 932 may perform a correction process such that the value obtained by subtracting the variation index value Sd after correction from the variation index value S before correction (sometimes referred to as the reduction amount due to correction) becomes larger as the above distance increases.

For example, the first correction unit 932 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the depth of each of the multiple points calculated by the depth calculation unit 922. Each of the multiple points may be an example of a measurement target surface.

As the distance between the reference point of the distance measuring unit 520 and the measurement target surface of the work target 102 increases, the area of the measurement target surface increases. Therefore, the degree of variation in the intensity of reflected light from the measurement target surface may also increase. It can be assumed that the intensity of variation in reflected light from the measurement target surface is proportional to the square of the distance from the surface of a sphere whose center is the point of interest on the reflecting surface and whose radius is the distance to the target. It has been experimentally confirmed that a function with the intensity of variation in reflected light from the measurement target surface as the objective variable and the above distance as the explanatory variable can be approximated by a quadratic equation. Therefore, the first correction unit 932, for example, calculates the corrected variation index value Sd by dividing the variation index value S calculated by the variation index calculation unit 926 by the square of the distance between the reference point of the distance measuring unit 520 and the measurement target surface of the work target 102.

This reduces the effect of the distance between the distance measuring unit 520 and the work object 102 on the value of the variation index. As a result, for example, when the object detection unit 632 uses the variation index to detect the work object 102 from one or more objects arranged in the work area 114, the difference in the variation index due to the difference between the material of the work object 102 and the material of other objects becomes clearer.

In this embodiment, the second correction unit 934 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the size of the light receiving element that constitutes the light receiving unit 524. The second correction unit 934 may perform the correction process such that the amount of reduction due to the correction increases as the size of the light receiving element increases.

For example, if the shape of the light receiving element can be considered to be a rectangle, the second correction unit 934 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the length of the long side of the light receiving element. For example, if the light receiving unit 524 is composed of multiple light receiving elements and at least two of the light receiving elements have different sizes, the second correction unit 934 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the size of the light receiving element that outputs the light receiving intensity of the reflected light from each of the multiple points.

As the size of the light receiving element increases, the area of the surface to be measured increases. In addition, as the size of the light receiving element increases, it becomes more susceptible to errors due to the distance between the distance measuring unit 520 and the work object 102. Similarly, as the size of the light receiving element increases, it becomes more susceptible to errors due to the inclination between the reference surface of the distance measuring unit 520 and the surface to be measured of the work object 102. Therefore, the variation in the intensity of the reflected light from the surface to be measured may also increase.

According to this embodiment, the correction process in the second correction unit 934 can reduce the effect of the size of the light receiving element on the value of the variation index. As a result, for example, when the object detection unit 632 uses the variation index to detect the work target 102 from one or more objects arranged in the work area 114, the difference in the variation index due to the difference between the material of the work target 102 and the material of the other objects becomes clearer.

In this embodiment, the third correction unit 936 corrects the value of the variation index calculated by the variation index calculation unit 926 based on the inclination between the reference surface of the distance measurement unit 520 and the measurement target surface of the work target 102. The third correction unit 936 may perform a correction process so that the value obtained by subtracting the variation index value Sd after correction from the variation index value S before correction (sometimes referred to as the reduction amount due to correction) becomes larger as the reference surface of the distance measurement unit 520 and the measurement target surface of the work target 102 become less parallel.

For example, the third correction unit 936 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the angle between the normal vector of the light receiving surface of the light receiving element that constitutes the light receiving unit 524 and the normal vector of the measurement target surface on the surface of one or more objects to be measured, where the light from the irradiation unit 522 is irradiated.

The third correction unit 936 may correct the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on (i) the angle between the normal vector of the light receiving surface of the light receiving element constituting the light receiving unit 524 and the normal vector of the measurement target surface irradiated with light from the irradiation unit 522 on the surface of one or more objects to be measured, and (ii) the angle between the normal vector of the light receiving surface of the light receiving element constituting the light receiving unit 524 and the vector indicating the traveling direction of the optical axis of the light irradiated onto the measurement target surface.

The third correction unit 936 may correct the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on (i) the angle between the normal vector of the light receiving surface of the light receiving element constituting the light receiving unit 524 and the normal vector of the measurement target surface irradiated with light from the irradiation unit 522 on the surface of one or more objects to be measured, (ii) the angle between the normal vector of the light receiving surface of the light receiving element constituting the light receiving unit 524 and the vector indicating the traveling direction of the optical axis of the light irradiated to the measurement target surface, and (iii) the size of the light receiving element constituting the light receiving unit 524.

As the reference surface of the distance measuring unit 520 and the measurement surface of the work object 102 become less parallel, the difference between the time when reflected light from one end of the measurement surface reaches the light receiving element and the time when reflected light from the other end of the measurement surface reaches the light receiving element increases. Also, as the reference surface of the distance measuring unit 520 and the measurement surface of the work object 102 become less parallel, the regular reflection component contained in the reflected light decreases, and the received light intensity of the reflected light decreases. Therefore, the degree of inclination of the two can be a factor in errors in the variation index.

According to this embodiment, the correction process in the third correction unit 936 can reduce the effect of the above-mentioned tilt on the value of the variation index. As a result, for example, when the object detection unit 632 uses the variation index to detect the work target 102 from one or more objects arranged in the work area 114, the difference in the variation index due to the difference between the material of the work target 102 and the material of the other objects becomes clearer.

In this embodiment, the fourth correction unit 938 corrects the value of the variation index calculated by the variation index calculation unit 926 based on the absorptance of the light emitted by the irradiation unit 522 on the surface to be measured. The fourth correction unit 938 may correct the value of the variation index calculated by the variation index calculation unit 926 based on the color of the surface to be measured or the intensity of the reflected light. The fourth correction unit 938 may perform a correction process such that the value obtained by subtracting the variation index value S before correction from the variation index value Sd after correction (sometimes referred to as the increase due to correction) becomes larger as the absorptance of light increases.

For example, the fourth correction unit 938 corrects the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on (i) the color of each of the multiple points acquired by the imaging unit 530, or (ii) the intensity of reflected light from each of the multiple points received by the light receiving unit 524. The fourth correction unit 938 may correct the value of the variation index for each of the multiple points calculated by the variation index calculation unit 926 based on the intensity of reflected light from each of the multiple points received by the light receiving unit 524 and the depth of each of the multiple points calculated by the depth calculation unit 922.

The fourth correction unit 938 may perform the above correction process when the material or color of one or more objects to be detected by the object detection unit 632 is known. The fourth correction unit 938 may perform the above correction process when (i) the size of the working area 114 is known, and (ii) the intensity of the reflected light from each of the multiple points received by the light receiving unit 524 is smaller than a threshold value determined in consideration of the size of the working area 114.

[An example of another embodiment]
In the present embodiment, an example of the work support system 100 has been described by taking as an example a case where the point cloud data output unit 540 arranged in the LIDAR 124 calculates the variation index. However, the work support system 100 is not limited to the present embodiment. In other embodiments, at least a part of the function of the point cloud data output unit 540 may be realized by the support terminal 140 or the support server 160. For example, at least one of the variation index calculation unit 926 and the variation correction unit 930 is arranged in the support terminal 140 or the support server 160.

[An example of another embodiment]
In the present embodiment, an example of the variation correction unit 930 has been described in which the first correction unit 932, the second correction unit 934, the third correction unit 936, and the fourth correction unit 938 each perform the correction process independently. However, the variation correction unit 930 is not limited to the present embodiment.

In another embodiment, the variance correction unit 930 may execute the functions of the first correction unit 932, the second correction unit 934, the third correction unit 936, and the fourth correction unit 938 in an integrated manner. For example, the variance correction unit 930 corrects the value of the variance index calculated by the variance index calculation unit 926 based on at least two combinations of the error factors to be corrected in the first correction unit 932, the error factors to be corrected in the second correction unit 934, the error factors to be corrected in the third correction unit 936, and the error factors to be corrected in the fourth correction unit 938.

The information processing in the variance index calculation unit 926 will be outlined with reference to FIG. 10. FIG. 10 shows an example of the depth variation 1020 calculated by the depth calculation unit 922 for a single measurement target surface (e.g., one of the above-mentioned multiple points) on the surface of the work target 102. In FIG. 10, the measurement period Ts indicates the period during which multiple pulsed lights are irradiated onto the measurement target surface. The evaluation period Δt indicates the extraction period of samples used by the variance index calculation unit 926 to calculate the variance index.

The variation index calculation unit 926 calculates a variation index of the depth D by statistically processing the depth D calculated based on the reflected light received by the light receiving unit 524 during the evaluation period Δt. As described above, examples of the variation index include variance, deviation, and standard deviation. The variation index calculation unit 926 may calculate the average value Da of the depth D during the evaluation period Δt.

The information processing in the first correction unit 932 will be outlined with reference to FIG. 11. FIG. 11 shows a schematic relationship between the distance (e.g., depth D) from a reference point 1124 set on a reference surface 1122 on the light receiving surface of the distance measuring unit 520 and the degree of variation in depth D during the measurement period Ts described in relation to FIG. 9. As shown in FIG. 11, a comparison of the variation 1142 of depth D at depth D1, the variation 1144 of depth D at depth D2, and the variation 1146 of depth D at depth D3 shows that the range of variation increases in this order.

The information processing in the second correction unit 934 and the third correction unit 936 will be outlined with reference to FIG. 12. In this embodiment, the distance measurement unit 520 includes a light receiving element 1222 and an optical system 1224. For the purpose of simplifying the explanation, in this embodiment, the distance measurement unit 520 includes a single light receiving element 1222, and the light receiving element 1222 has a substantially square shape with a side length of c.

12, the information processing in the second correction unit 934 and the third correction unit 936 is explained by taking as an example a case where the normal vector 1226 of the reference surface 1122 passing through the reference point 1124 and the vector 1242 indicating the traveling direction of the optical axis of the pulsed light 1240 are not parallel. Also, in FIG. 12, the information processing in the second correction unit 934 and the third correction unit 936 is explained by taking as an example a case where the normal vector 1226 of the reference surface 1122 of the distance measurement unit 520 and the normal vector 1272 of the measurement target surface 1270 arranged on the surface 1260 of the work target 102 are not parallel.

As the scanning of the surface 1260 progresses, the depth or three-dimensional position of multiple measurement target surfaces 1270 on the surface 1260 becomes clear. The normal vector 1272 of a specific measurement target surface 1270 can then be derived, for example, by calculating the inclination of the surface 1260 near the specific measurement target surface 1270 based on the depth of the specific measurement target surface 1270 and the depths of multiple measurement target surfaces 1270 arranged near the specific measurement target surface 1270. In this embodiment, the reference surface 1122 is set on the surface of the optical system 1224. However, the reference surface 1122 may be an example of a light receiving surface of a light receiving element.

12, the larger the length c of one side of the light receiving element 1222, the larger the difference in (i) the distance between one end 1274 of the single measurement target surface 1270 and the reference point 1124, and (i) the distance between the other end 1276 of the single measurement target surface 1270 and the reference point 1124. It can also be seen that the difference varies depending on the angle between the normal vector 1272 of the measurement target surface 1270 and the vector 1242 indicating the traveling direction of the optical axis of the pulsed light 1240. The angle between the normal vector 1272 of the measurement surface 1270 and the vector 1242 indicating the traveling direction of the optical axis of the pulsed light 1240 can be determined based on, for example, (i) the angle between the normal vector 1226 of the reference surface 1122 and the normal vector 1272 of the measurement surface 1270, and (ii) the angle between the normal vector 1226 of the reference surface 1122 and the vector 1242 indicating the traveling direction of the optical axis of the pulsed light 1240.

Figure 13 shows an example of an image 1300 generated based on the value of the variation index. Image 1300 is an example of an image that represents the standard deviation S of the depth D on the measurement target surface in 256 grayscale levels. The procedure for creating image 1300 is as follows.

First, using Microsoft's Azure Kinect DK, 1,000 yen, 5,000 yen, and 10,000 yen bills were captured in the dark on a table with a circular wooden top (Kvistbro, IKEA, blue color) to obtain color point cloud data. The table top was painted blue with acrylic paint.

When acquiring color point cloud data from Azure Kinect DK, the output of the light intensity of reflected light from each measurement target surface was extracted from Azure Kinect DK. Based on the fluctuation in the light intensity of the reflected light output from Azure Kinect DK, the standard deviation S of the depth D was calculated for each of the multiple measurement target surfaces. The calculated standard deviation value of each measurement target surface was converted into a 256-level grayscale to create image 1300.

As described above, the standard deviation S at a particular point indicates the degree of variation in the received light intensity of reflected light from that particular point. In addition, the standard deviation S at each point was calculated using data from a period in which the distance values for each point output by the Azure Kinect DK were stable. Specifically, the standard deviation S was calculated according to the following procedure.

First, the fluctuation of the depth D (i.e., the distance from the Azure Kinect DK) of the specific point or points surrounding the specific point was observed over a predetermined period of time. Next, data on the depth D during a period in which the above fluctuation range was smaller than a predetermined value was extracted. Next, the extracted data on the depth D was statistically processed to calculate the standard deviation S of the depth D. Note that for periods in which the above fluctuation range was larger than a predetermined value, a process was performed to return a fixed value in cases where a value could not be obtained normally. In this embodiment, 0 was adopted as the above fixed value.

As shown in FIG. 13, in image 1300, the boundaries between the table and each of the 1,000 yen, 5,000 yen, and 10,000 yen notes can be detected visually. However, because the image was taken in the dark, it was not possible to detect the boundaries between the table and each of the 1,000 yen, 5,000 yen, and 10,000 yen notes based on the RGB image created from the point cloud data. In addition, because the 1,000 yen, 5,000 yen, and 10,000 yen notes are very thin, it was not possible to detect the boundaries between the table and each of the 1,000 yen, 5,000 yen, and 10,000 yen notes based on the distance image created from the point cloud data.

Figure 14 shows an example of a computer 3000 in which aspects of the present invention may be embodied in whole or in part. At least a part of the work support system 100 may be realized by the computer 3000. For example, at least a part of the support server 160 is realized by the computer 3000.

A program installed on the computer 3000 may cause the computer 3000 to function as or perform operations associated with an apparatus according to an embodiment of the present invention or one or more "parts" of the apparatus, and/or to perform a process or steps of the process according to an embodiment of the present invention. Such a program may be executed by the CPU 3012 to cause the computer 3000 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 3000 according to this embodiment includes a CPU 3012, a RAM 3014, a GPU 3016, and a display device 3018, which are interconnected by a host controller 3010. The computer 3000 also includes input/output units such as a communication interface 3022, a hard disk drive 3024, a DVD-ROM drive 3026, and an IC card drive, which are connected to the host controller 3010 via an input/output controller 3020. The computer also includes legacy input/output units such as a ROM 3030 and a keyboard 3042, which are connected to the input/output controller 3020 via an input/output chip 3040.

The CPU 3012 operates according to the programs stored in the ROM 3030 and the RAM 3014, thereby controlling each unit. The GPU 3016 acquires image data generated by the CPU 3012 into a frame buffer or the like provided in the RAM 3014 or into itself, and causes the image data to be displayed on the display device 3018.

The communication interface 3022 communicates with other electronic devices via a network. The hard disk drive 3024 stores programs and data used by the CPU 3012 in the computer 3000. The DVD-ROM drive 3026 reads programs or data from the DVD-ROM 3001 and provides the programs or data to the hard disk drive 3024 via the RAM 3014. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

The ROM 3030 stores therein a boot program or the like that is executed by the computer 3000 upon activation, and/or a program that depends on the hardware of the computer 3000. The input/output chip 3040 may also connect various input/output units to the input/output controller 3020 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

The programs are provided by a computer-readable storage medium such as a DVD-ROM 3001 or an IC card. The programs are read from the computer-readable storage medium, installed in the hard disk drive 3024, RAM 3014, or ROM 3030, which are also examples of computer-readable storage media, and executed by the CPU 3012. The information processing described in these programs is read by the computer 3000, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing the operation or processing of information in accordance with the use of the computer 3000.

For example, when communication is performed between computer 3000 and an external device, CPU 3012 may execute a communication program loaded into RAM 3014 and instruct communication interface 3022 to perform communication processing based on the processing described in the communication program. Under the control of CPU 3012, communication interface 3022 reads transmission data stored in a transmission buffer area provided in RAM 3014, hard disk drive 3024, DVD-ROM 3001, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 3012 may also cause all or a necessary portion of a file or database stored on an external recording medium such as the hard disk drive 3024, the DVD-ROM drive 3026 (DVD-ROM 3001), an IC card, etc. to be read into the RAM 3014, and perform various types of processing on the data on the RAM 3014. The CPU 3012 may then write back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 3012 may perform various types of processing on the data read from the RAM 3014, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and writes back the results to the RAM 3014. The CPU 3012 may also search for information in a file, database, etc. in the recording medium. For example, when multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 3012 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The above-described program or software module may be stored in a computer-readable storage medium on the computer 3000 or in the vicinity of the computer 3000. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the above-described program to the computer 3000 via the network.

Although the present invention has been described above using an embodiment, the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear to those skilled in the art that various modifications or improvements can be made to the above embodiment. Furthermore, the details described for a specific embodiment can be applied to other embodiments to the extent that they are not technically inconsistent. It is clear from the claims that such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that the processes may be performed in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is necessary to perform the processes in this order.

For example, the present specification discloses the following:
[Item A-1]
a point cloud data acquisition unit that acquires three-dimensional point cloud data of a first object disposed in a three-dimensional space;
a modeling unit that constructs a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit;
an instruction receiving unit that receives a first instruction from a first user to specify a pointed position that is a point or an area arranged on a surface of the three-dimensional model of the first object arranged in the virtual space;
a projection control unit that controls a projection device that projects an image to project a first image onto a projection position that is a point or area corresponding to the designated position on a surface of the first object;
An information processing device comprising:
[Item A-2]
the projection control unit controls the projection device such that, when the first object moves in the three-dimensional space, the first image projected onto the first object follows the movement of the first object.
The information processing device according to item A-1.
[Item A-3]
the instruction receiving unit receives a second instruction from the first user to identify an attribute of the first image;
The projection control unit includes:
(i) when the attribute of the first image indicated by the second instruction is a first attribute, adjusting at least one of a projection direction of the first image and a shape and a size of the first image so that the first image projected onto the first object follows the movement of the first object when the first object moves in the three-dimensional space;
(ii) when the attribute of the first image indicated by the second instruction is a second attribute, even if the first object moves in the three-dimensional space, a projection direction of the first image and a shape and a size of the first image are not adjusted;
The information processing device according to item A-1 or A-2.
[Item A-4]
the instruction receiving unit receives, as an instruction from the first user, (i) a relative position of the designated position with respect to a reference point of the three-dimensional model of the first object, or (ii) a third instruction for fixing a relative position of the designated position with respect to a reference point of the virtual space;
The projection control unit includes:
(i) when the instruction receiving unit receives the third instruction and the attribute of the first image indicated by the second instruction is a first attribute, when the first object moves in the three-dimensional space, adjusting at least one of a projection direction of the first image and a shape and a size of the first image so that the first image projected onto the first object follows the movement of the first object;
(ii) when the instruction receiving unit receives the third instruction and the attribute of the first image indicated by the second instruction is the second attribute, even if the first object moves in the three-dimensional space, a projection direction of the first image and a shape and a size of the first image are not adjusted;
The information processing device according to item A-3.
[Item A-5]
the instruction receiving unit receives a fourth instruction from the first user regarding starting or stopping the projection of the first image;
The projection control unit determines whether or not to project the first image onto the first object based on the fourth instruction.
The information processing device according to any one of items A-1 to A-4.
[Item A-6]
Further, an indication mark control unit controls the movement of a mark indicating a specific point in the virtual space based on a control signal output by a pointing device or a controller used by the first user to specify the indication position in the virtual space.
The information processing device according to any one of items A-1 to A-5.
[Item A-7]
the indication mark control unit controls a movement of the mark based on first setting information that defines a manner of movement of the mark in the vicinity of a boundary of the three-dimensional model of the first object;
The information processing device according to item A-6.
[Item A-8]
an image data acquisition unit that acquires image data of the first object;
an object estimation unit that estimates a type of the first object based on the image data acquired by the image data acquisition unit;
a setting extraction unit that refers to a setting storage unit that stores information indicating the type of each of the one or more objects in association with second setting information that specifies how the mark moves at each of one or more positions of a three-dimensional model of each object corresponding to each of one or more positions on a surface of each object, and extracts the second setting information corresponding to the type of the first object estimated by the object estimation unit;
Further equipped with
The indication mark control unit controls the movement of the mark based on the second setting information extracted by the setting extraction unit.
The information processing device according to item A-6 or A-7.
[Item A-9]
the point cloud data acquisition unit acquires the three-dimensional point cloud data of the first object from a measurement device that outputs the three-dimensional point cloud data of the first object based on reflected light of the light obtained by irradiating the first object with light;
The measuring device has an imaging unit that images the first object,
the information processing device further includes a calibration unit that determines a positional relationship between the measurement device and the projection device;
The projection device projects a calibration image including at least three line segments whose relative positions to each other are known into the three-dimensional space;
the imaging unit captures the calibration image projected into the three-dimensional space by the projection device;
the calibration unit determines a positional relationship between the measurement device and the projection device based on the calibration image captured by the imaging unit.
The information processing device according to any one of items A-1 to A-9.
[Item A-10]
A program for causing a computer to function as the information processing device according to any one of items A-1 to A-10.
[Item A-11]
A point cloud data acquisition step of acquiring three-dimensional point cloud data of a first object disposed in a three-dimensional space;
a modeling step of constructing a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired in the point cloud data acquisition step;
an instruction receiving step of receiving a first instruction from a first user for specifying a pointed position which is a point or an area arranged on a surface of the three-dimensional model of the first object arranged in the virtual space;
a projection control step of controlling a projection device that projects an image to project a first image onto a projection position that is a point or area on a surface of the first object corresponding to the designated position;
An information processing method comprising the steps of:
[Item B-1]
a point cloud data acquisition unit that acquires three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged;
an object detection unit that detects at least a portion of a boundary of a first object included in the one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired by the point cloud data acquisition unit;
Equipped with
the point cloud data acquisition unit acquires the three-dimensional point cloud data of the three-dimensional space from a measurement device that outputs the three-dimensional point cloud data of the three-dimensional space based on reflected light of the light obtained by irradiating the one or more objects with light;
The three-dimensional point cloud data of the three-dimensional space is
Three-dimensional position information indicating a three-dimensional position of each of a plurality of points on a surface of the one or more objects;
Variation information indicating a degree of variation in the intensity of the reflected light from each of the plurality of points; and
Including,
the object detection unit detects at least the part of the boundary of the first object based on the three-dimensional position information and the variation information.
Detection device.
[Item B-2]
The object detection unit is
extracting points on the surface of the first object from the plurality of points based on the value of the index indicating the degree of variation at each of the plurality of points;
The detection device according to item B-1.
[Item B-3]
The object detection unit is
extracting at least one point, the value of which of the indexes indicating the degree of variation is within a predetermined first range, from the plurality of points;
For each of the at least one extracted points, a value of an index indicating the degree of variation of the point is compared with a value of an index indicating the degree of variation of a point adjacent to the point;
When the absolute value of the difference between the two is smaller than a predetermined value, it is determined that the two are points on the surface of the same object, and when the absolute value of the difference between the two is greater than the predetermined value, it is determined that the two are points on the surface of different objects, thereby distinguishing at least two objects included in the one or more objects.
The detection device according to item B-2.
[Item B-4]
The object detection unit extracts a point on the surface of the first object from one or more points, among the plurality of points, where a value of an index indicating the degree of variation is included within a numerical range determined based on a material of the first object.
The detection device according to item B-2 or B-3.
[Item B-5]
the three-dimensional point cloud data of the three-dimensional space further includes color information indicating the color of each of the plurality of points;
the object detection unit detects at least the part of the boundary of the first object based on the three-dimensional position information, the variation information, and the color information.
The detection device according to any one of items B-1 to B-4.
[Item B-6]
The measuring device is
an irradiation unit that irradiates the light onto the one or more objects arranged in the three-dimensional space;
a light receiving unit disposed near the irradiating unit and configured to receive reflected light of the light from the one or more objects;
Equipped with
The detection device or the measurement device is
a depth calculation unit that calculates a depth indicating a distance from the irradiation unit or the light receiving unit for each of the plurality of points on the surface of the one or more objects based on a time when the irradiation unit irradiates the light and a time when the light receiving unit receives the reflected light;
an index calculation unit that calculates an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on a change in the depth in a unit period having a predetermined length;
Further comprising:
The detection device according to any one of items B-1 to B-5.
[Item B-7]
The detection device or the measurement device is
A first correction unit corrects an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on the depth of each of the plurality of points calculated by the depth calculation unit.
The detection device according to item B-6.
[Item B-8]
The detection device or the measurement device is
a second correction unit that corrects an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on a size of a light receiving element that constitutes the light receiving unit;
The detection device according to item B-6 or B-7.
[Item B-9]
The detection device or the measurement device is
and a third correction unit that corrects an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on an angle between a normal vector of a light receiving surface of a light receiving element constituting the light receiving unit and a normal vector of a measurement target surface on a surface of the one or more objects onto which the light is irradiated.
The detection device according to any one of items B-6 to B-8.
[Item B-10]
The detection device or the measurement device is
and a fourth correction unit that corrects an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on the intensity of the reflected light from each of the plurality of points received by the light receiving unit.
The detection device according to any one of items B-6 to B-9.
[Item B-11]
A program for causing a computer to function as the detection device according to any one of items B-1 to B-10.
[Item B-12]
A point cloud data acquisition step of acquiring three-dimensional point cloud data of a three-dimensional space in which one or more objects are arranged;
an object detection step of detecting at least a portion of a boundary of a first object included in the one or more objects based on the three-dimensional point cloud data of the three-dimensional space acquired in the point cloud data acquisition step;
having
the point cloud data acquisition step includes a step of acquiring the three-dimensional point cloud data of the three-dimensional space from a measurement device that outputs the three-dimensional point cloud data of the three-dimensional space based on reflected light of the light obtained by irradiating the one or more objects with light,
The three-dimensional point cloud data of the three-dimensional space is
Three-dimensional position information indicating a three-dimensional position of each of a plurality of points on a surface of the one or more objects;
Variation information indicating a degree of variation in the intensity of the reflected light from each of the plurality of points; and
Including,
The object detection step includes detecting at least a portion of the boundary of the first object based on the three-dimensional position information and the variation information.
Detection methods.
[Item C]
An illumination unit that illuminates one or more objects arranged in a three-dimensional space with light;
a light receiving unit disposed near the irradiating unit and configured to receive reflected light of the light from the one or more objects;
a depth calculation unit that calculates a depth indicating a distance from the irradiation unit or the light receiving unit for each of the plurality of points on the surface of the one or more objects based on a time when the irradiation unit irradiates the light and a time when the light receiving unit receives the reflected light;
an index calculation unit that calculates an index indicating a degree of variation in intensity of the reflected light from each of the plurality of points based on a change in the depth in a unit period having a predetermined length;
A measuring device comprising:

10 Communication network, 20 Support image, 32 Worker, 34 Supporter, 100 Work support system, 102 Work object, 110 Work room, 112 Work table, 114 Work area, 122 Display, 124 Lidar, 126 Projector, 140 Support terminal, 142 Support terminal, 144 Controller, 146 Support terminal, 148 Pointing device, 160 Support server, 162 Work room control unit, 164 Support terminal control unit, 210 Title, 212 Encoded code, 214 Entry field, 216 Signature field, 218 Seal field, 322 Marker, 324 Pointer, 400 Operation screen, 402 Cursor, 404 Object, 420 Menu bar, 440 Virtual space display area, 442 Three-dimensional model, 444 Three-dimensional model, 460 Object setting area, 462 Pull-down list, 464 Pull-down list, 466 Button, 520 Distance measurement unit, 522 Irradiation unit, 524 Light receiving unit, 530 Imaging unit, 540 Point cloud data output unit, 560 Color point cloud data, 622 Point cloud data acquisition unit, 632 Object detection unit, 634 Modeling unit, 636 Type estimation unit, 642 Instruction reception unit, 644 Projection control unit, 650 Calibration unit, 660 Storage unit, 700 Calibration image, 810 Setting storage unit, 812 First setting information storage unit, 814 Second setting information storage unit, 820 Indication mark control unit, 832 Image data acquisition unit, 834 Object estimation unit, 836 Setting extraction unit, 922 Depth calculation unit, 924 Three-dimensional position information calculation unit, 926 Variation index calculation unit, 930 Variation correction unit, 932 First correction unit, 934 Second correction unit, 936 Third correction unit, 938 Fourth correction unit, 1020 Fluctuation, 1122 Reference surface, 1124 Reference point, 1142 Fluctuation, 1144 Fluctuation, 1146 Fluctuation, 1222 Light receiving element, 1224 Optical system, 1226 Normal vector, 1240 Pulsed light, 1242 Vector, 1260 Surface, 1270 Measurement target surface, 1272 Normal vector, 1274 End, 1276 End, 1300 Image, 3000 Computer, 3001 DVD-ROM, 3010 Host controller, 3012 CPU, 3014 RAM, 3016 GPU, 3018 Display device, 3020 Input/output controller, 3022 Communication interface, 3024 Hard disk drive, 3026 DVD-ROM drive, 3030 ROM, 3040 I/O chip, 3042 keyboard

Claims (10)

a point cloud data acquisition unit that acquires three-dimensional point cloud data of a first object disposed in a three-dimensional space;
a modeling unit that constructs a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit;
an instruction receiving unit that receives a first instruction from a first user for specifying a pointed position that is a point or an area arranged on a surface of the three-dimensional model of the first object arranged in the virtual space;
a projection control unit that controls a projection device that projects an image to project a first image onto a projection position that is a point or an area corresponding to the designated position on a surface of the first object;
an indication mark control unit that controls the movement of a mark indicating a specific point in the virtual space based on a control signal output by a pointing device or a controller used by the first user to specify the indication position in the virtual space;
Equipped with
the indication mark control unit controls the movement of the mark based on first setting information that defines a manner of movement of the mark in the vicinity of a boundary of the three-dimensional model of the first object.
Information processing device. the projection control unit controls the projection device such that, when the first object moves in the three-dimensional space, the first image projected onto the first object follows the movement of the first object.
The information processing device according to claim 1 . the instruction receiving unit receives a second instruction from the first user for identifying an attribute of the first image;
The projection control unit includes:
(i) when the attribute of the first image indicated by the second instruction is a first attribute, adjusting at least one of a projection direction of the first image and a shape and a size of the first image so that the first image projected onto the first object follows the movement of the first object when the first object moves in the three-dimensional space;
(ii) when the attribute of the first image indicated by the second instruction is a second attribute, even if the first object moves in the three-dimensional space, a projection direction of the first image and a shape and a size of the first image are not adjusted;
3. The information processing device according to claim 1 or 2. the instruction receiving unit receives, as an instruction from the first user, (i) a relative position of the designated position with respect to a reference point of the three-dimensional model of the first object, or (ii) a third instruction for fixing the relative position of the designated position with respect to a reference point of the virtual space;
The projection control unit includes:
(i) when the instruction receiving unit receives the third instruction and the attribute of the first image indicated by the second instruction is a first attribute, when the first object moves in the three-dimensional space, adjusting at least one of a projection direction of the first image and a shape and a size of the first image so that the first image projected onto the first object follows the movement of the first object;
(ii) when the instruction receiving unit receives the third instruction and the attribute of the first image indicated by the second instruction is a second attribute, even if the first object moves in the three-dimensional space, a projection direction of the first image and a shape and a size of the first image are not adjusted.
The information processing device according to claim 3 . the instruction receiving unit receives a fourth instruction from the first user regarding starting or stopping the projection of the first image;
The projection control unit determines whether or not to project the first image onto the first object based on the fourth instruction.
The information processing device according to any one of claims 1 to 4. a point cloud data acquisition unit that acquires three-dimensional point cloud data of a first object disposed in a three-dimensional space;
a modeling unit that constructs a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired by the point cloud data acquisition unit;
an instruction receiving unit that receives a first instruction from a first user for specifying a pointed position that is a point or an area arranged on a surface of the three-dimensional model of the first object arranged in the virtual space;
a projection control unit that controls a projection device that projects an image to project a first image onto a projection position that is a point or an area corresponding to the designated position on a surface of the first object;
an indication mark control unit that controls the movement of a mark indicating a specific point in the virtual space based on a control signal output by a pointing device or a controller used by the first user to specify the indication position in the virtual space;
an image data acquisition unit that acquires image data of the first object;
an object estimation unit that estimates a type of the first object based on the image data acquired by the image data acquisition unit;
a setting extraction unit that refers to a setting storage unit that stores information indicating the type of each of the one or more objects and second setting information that specifies how the mark moves at each of one or more positions of a three-dimensional model of each object corresponding to each of one or more positions on a surface of each object, in association with each other, and extracts the second setting information corresponding to the type of the first object estimated by the object estimation unit;
Equipped with
The indication mark control unit controls the movement of the mark based on the second setting information extracted by the setting extraction unit.
Information processing device. the point cloud data acquisition unit acquires the three-dimensional point cloud data of the first object from a measurement device that outputs the three-dimensional point cloud data of the first object based on reflected light of the light obtained by irradiating the first object with light;
the measuring device has an imaging unit that images the first object,
the information processing device further includes a calibration unit that determines a positional relationship between the measurement device and the projection device;
The projection device projects a calibration image including at least three line segments whose relative positions to each other are known into the three-dimensional space;
the imaging unit captures the calibration image projected into the three-dimensional space by the projection device;
the calibration unit determines a positional relationship between the measurement device and the projection device based on the calibration image captured by the imaging unit.
The information processing device according to any one of claims 1 to 6 . A program for causing a computer to function as the information processing device according to any one of claims 1 to 7 . A point cloud data acquisition step of acquiring three-dimensional point cloud data of a first object disposed in a three-dimensional space;
a modeling step of constructing a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired in the point cloud data acquisition step;
an instruction receiving step of receiving a first instruction from a first user for specifying a pointed position which is a point or an area arranged on a surface of the three-dimensional model of the first object arranged in the virtual space;
a projection control step of controlling a projection device that projects an image to project a first image onto a projection position that is a point or area corresponding to the designated position on the surface of the first object;
A step of controlling a movement of a mark indicating a specific point in the virtual space based on a control signal output by a pointing device or a controller used by the first user to specify the indicated position in the virtual space;
having
The step of controlling the movement of the mark includes controlling the movement of the mark based on first setting information that defines a manner in which the mark moves in the vicinity of a boundary of the three-dimensional model of the first object.
Information processing methods. A point cloud data acquisition step of acquiring three-dimensional point cloud data of a first object disposed in a three-dimensional space;
a modeling step of constructing a virtual space including a three-dimensional model of the first object based on the three-dimensional point cloud data acquired in the point cloud data acquisition step;
an instruction receiving step of receiving a first instruction from a first user for specifying a pointed position which is a point or an area arranged on a surface of the three-dimensional model of the first object arranged in the virtual space;
a projection control step of controlling a projection device that projects an image to project a first image onto a projection position that is a point or area corresponding to the designated position on the surface of the first object;
A step of controlling a movement of a mark indicating a specific point in the virtual space based on a control signal output by a pointing device or a controller used by the first user to specify the indicated position in the virtual space;
acquiring image data of the first object;
estimating a type of the first object based on the image data acquired in the step of acquiring the image data;
a step of extracting the second setting information corresponding to the type of the first object estimated in the step of estimating the type of the first object by referring to a setting storage unit that stores information indicating the type of each of the one or more objects in association with second setting information that specifies how the mark moves at each of one or more positions of a three-dimensional model of each object corresponding to each of one or more positions on a surface of each object;
having
The step of controlling the movement of the mark controls the movement of the mark based on the second setting information extracted in the step of extracting the second setting information.
Information processing methods.