JP2019174920A - Article management system and article management program - Google Patents

Abstract

An object in an object space can be easily and appropriately managed. In an article management system for managing an article to be managed disposed in a predetermined space, a processor unit is configured to: The three-dimensional position information that can specify the three-dimensional position of the group is specified, the article in the frame image is recognized based on one or more frame images in the moving image, and the three-dimensional position of the recognized article in a predetermined space is determined. Is configured to specify the item position information that can specify the item position. [Selection] Figure 2

Description

  The present invention relates to an article management system for managing articles to be managed arranged in a predetermined space.

  There is a demand for a technique for easily managing the position and state of an article. For example, in a general company that uses offices, the position and condition of equipment and fixtures (PCs, desks, chairs, shelves, etc.) are more accurate for asset management, office layout changes, and estimates before moving. And there is a need to grasp and manage easily. Also, in factories, etc., check the position and status of the device. For example, the device administrator can check whether the device at an arbitrary position is operating by checking the device's operating status or an external check lamp. However, there is a need to automate the task of manually entering the checklist.

  At present, such work is generally performed by human visual confirmation, except for PCs and expensive equipment, which is very expensive. Even in a PC or an expensive device, it is necessary to install management software in advance, which requires a lot of costs for management. Furthermore, it is difficult to automatically specify an accurate position of a PC or the like only by using software.

  In such article management, for example, a technique for managing the position of an article using an IC tag as described in Patent Document 1 is known.

JP 2006-338120 A

  However, in the article management by IC tag, it is necessary to attach an IC tag to each article, and it takes a great deal of cost to attach the tag. Furthermore, in the management by IC tag, when it is desired to immediately know the state of goods of other companies that are not managing the tag, for example, when any company moves the office, the logistics company moves the equipment of the office to be moved. It is not applicable when you want to know the status.

  For example, in a situation where it is necessary to grasp an individual of an individual article, article management using an IC tag is effective, but depending on the user, it is not necessary to grasp details of the individual, and “PC”, “shelf” ”,“ Desk ”,“ chair ”, etc., and their size may be grasped.

  In such a case, there is a need for a technique for grasping and managing the position and state of an article more immediately and at a low cost without incurring the advance cost of attaching an IC tag to the article. Has been.

  This invention is made | formed in view of the said situation, The objective is to provide the technique which can manage the article | item in the space made into object easily and appropriately.

  In order to achieve the above object, an article management system according to one aspect is an article management system for managing articles to be managed arranged in a predetermined space, and is based on a moving image obtained by photographing the predetermined space. The article in the frame image is recognized based on the three-dimensional position specifying unit that specifies the three-dimensional position information that can specify the three-dimensional position of the point group in the predetermined space and one or more frame images in the moving image. An article recognition unit; and an article position specifying unit that specifies article position information that can specify a three-dimensional position in a predetermined space of the recognized article.

  According to the present invention, articles in a target space can be managed easily and appropriately.

FIG. 1 is an overall configuration diagram of an article management system according to an embodiment. Figure 2 is a block diagram of an article management server according to an embodiment. FIG. 3 is a configuration diagram of a map database according to an embodiment. Figure 4 is a block diagram of an article database according to one embodiment. FIG. 5 is a configuration diagram of a smartphone according to an embodiment. Figure 6 is a block diagram of an article confirmation terminal according to an embodiment. Figure 7 is a flow chart of system utilization start processing according to one embodiment. FIG. 8 is a flowchart of map generation / article detection processing according to an embodiment. FIG. 9 is a diagram illustrating an example of a frame image according to an embodiment. FIG. 10 is a diagram illustrating article recognition and position detection in a frame image according to an embodiment. Figure 11 is a block diagram of an article detection table according to the embodiment. FIG. 12 is a diagram illustrating the generation of a 3D map and a 3D image based on the 3D map according to an embodiment. FIG. 13 is a diagram illustrating a three-dimensional image of another viewpoint based on a three-dimensional map according to an embodiment. Figure 14 is a flow chart of a three-dimensional image display processing according to the embodiment. FIG. 15 is a diagram illustrating an article management viewer screen according to an embodiment. FIG. 16 is a flowchart of a display change process according to an embodiment. Figure 17 is a diagram illustrating the area specifying search according to an embodiment. Figure 18 is a diagram showing an article status confirmation screen according to an embodiment. FIG. 19 is a diagram illustrating an article management viewer screen that displays a movement history according to an embodiment.

  Embodiments will be described with reference to the drawings. The embodiments described below do not limit the invention according to the claims, and all the elements and combinations described in the embodiments are essential for the solution of the invention. Is not limited.

  In the following description, information may be described using the expression “AAA table”, but the information may be expressed in any data structure. That is, the “AAA table” can be referred to as “AAA information” to indicate that the information does not depend on the data structure.

  In the following description, the “processor unit” includes one or more processors. The at least one processor is typically a CPU (Central Processing Unit). The processor may include hardware circuitry to perform some or all of the processing.

  In the following description, the process may be described with “program” (including modules, viewers, etc.) as the subject of operation, but the program is defined by being executed by a processor (for example, CPU). Since the processing is appropriately performed using a storage resource (for example, a memory) and / or a communication interface device (for example, a port), the main body of the processing may be a processor. The process described with the program as the subject of the operation may be performed by an apparatus including a processor. In addition, a hardware circuit that performs part or all of the processing performed by the processor may be included. The computer program may be installed on the device from a program source. The program source may be, for example, a program distribution server or a storage medium that can be read by a computer.

<System configuration>
FIG. 1 is an overall configuration diagram of an article management system according to an embodiment.

  The article management system 10 includes an article management server 101 as an example of a processing server, a smartphone 102 as an example of a photographing apparatus, and an article management terminal 103. The article management server 101, the smartphone 102, and the article management terminal 103 are connected via a network 105 so that they can communicate with each other. The smartphone 102 includes a camera unit 504 for capturing a moving image, which will be described later, and is used to capture a predetermined space (for example, the office 104) where an article to be managed exists. The network 105 may include, for example, a wired LAN (Local Area Network), a wireless LAN, and the Internet.

<Goods management server>
FIG. 2 is a configuration diagram of an article management server according to an embodiment.

  The article management server 101 is configured by, for example, a general server machine, and includes a processor unit 201, a storage device 202, and a network unit 211.

  The network unit 211 is an interface such as a wired LAN card or a wireless LAN card, and communicates with other devices (for example, the smartphone 102 and the article management terminal 103) via the network 105.

  The processor unit 201 executes various processes according to programs stored in the storage device 202.

  The storage device 202 is, for example, a hard disk drive, an SSD (Solid State Drive), a RAM (Random Access Memory), and the like, and stores a program executed by the processor unit 201 and data used by the processor unit 201. In the present embodiment, the storage device 202 stores an article management program 203, a map database 208, an article database 209, and a moving image file 210.

<Goods management program>
The article management program 203 includes a map generation module 204, a self-position estimation module 205, an article management module 206, and an article detection module 207.

  The article management program 203 is executed by the processor unit 201, and the position (viewpoint) of the camera that is shooting based on moving image data (moving image data: moving image data sent from the smartphone 102, for example). , A function (self-position estimation function) for estimating the posture (line of sight), a function for generating a three-dimensional map based on moving image data, recognition of an article (article object) shown in the moving picture, and the position of the article And a function of mapping an article on a three-dimensional map and making the mapped article searchable by a user.

  The article management program 203 can perform both streaming processing (real-time processing) and moving image file processing (non-real-time processing) as processing methods of moving image data transferred from the smartphone 102. When streaming video data, it is necessary that a network bandwidth of a predetermined level or more is required, and that the smartphone 102 and the article management program 203 are already connected to the network 105. When it is assumed to be used without choosing a place, the moving image data is stored in the local storage of the smartphone 102, and the moving image file is transferred from the smartphone 102 to the article management server 101 at the time when it can be connected to the stable network 105. Thus, non-streaming processing for moving image processing may be performed in the article management server 101. When performing non-streaming processing, the same configuration as streaming processing can be used except that the smartphone 102 and the article management server 203 have a function of transferring a moving image file.

<Map generation module and self-position estimation module>
The map generation module 204 and the self-position estimation module 205 are modules that operate in conjunction with each other, and estimate the self-position (the position of the camera being photographed) from the video data (two-dimensional video data) photographed by the smartphone 102. However, it is a module having a function of generating a three-dimensional map in space. These modules can be realized based on Visual SLAM (Visual Simultaneous Localization and Mapping) technology that performs self-localization and map generation in real time. As the video data to be used, video data from a monocular camera or stereo camera can be used, but if it is to be realized at a low cost, a monocular camera is prepared and the video data shot by the monocular camera is used. Use it. The map generation module 204 and the self-position estimation module 205 have a function of integrating a three-dimensional map of a plurality of moving image data into one three-dimensional map by matching feature points in the plurality of moving image data. Also good. The three-dimensional map data (three-dimensional position information) created by the map generation module 204 is stored in the map database 208. Here, when the processor unit 201 executes the map module 204 and the self-position estimation module 205, a three-dimensional position specifying unit is configured.

<Article management module>
The article management module 206 has a function of reading a frame image of a moving image and inputting it to the article detection module 207, a function of saving the result (article information) output by the article detection module 207 in the article database 209, and an article from the article database 209. A function of reading out the information of the article, and a function of managing article information such as a function of mapping the article position on the three-dimensional map.

<Article detection module>
The article detection module 207 has a function of recognizing an article and detecting the position of the article from frame image data. The article detection module 207 performs, for example, recognition of an article to be managed and position detection of an article based on a neural network. That is, the article detection module 207 internally includes a learning model that has been learned in advance by a large number of teacher data on articles to be managed (recognition targets by the model), and inputs the image data of the frame image to thereby obtain the frame image. The article in the frame is recognized, and position information (position coordinates, etc.) in the frame image of the article is calculated. Here, when the processor unit 201 executes the article detection module 207, an article recognition unit is configured. The processor unit 201 executes the article detection module 207 and the article management module 206 to configure an article position specifying unit.

<Map database>
FIG. 3 is a configuration diagram of a map database according to an embodiment.

  The map database 208 is a database for storing 3D map data generated by the map generation module 204 and the self-position estimation module 205, and includes a moving image management table 301 and a map data management table 306.

  The moving image management table 301 stores an entry corresponding to each moving image. Each entry includes columns of a moving image ID 302, a moving image storage destination file path 303, photographer information 304, and a time stamp 305.

  The moving image ID 302 stores an ID (moving image ID) for uniquely identifying a moving image file (moving image file storing moving image data transmitted from the smartphone 102) in the storage device 202 of the article management server 101. The moving image storage destination file path 303 stores a file path for storing moving image data in the local storage of the article management server 101. The photographer information 304 stores an ID for identifying the photographer of the moving image corresponding to the entry and the photographer's name. The time stamp 305 stores the date and time when the moving image corresponding to the entry was taken.

  The map data management table 306 stores an entry for each frame of video data. The entry includes columns of a moving image ID 307, a frame number 308, self-location information 309, and 3D point cloud data 310.

  The moving image ID 307 stores the moving image ID of the moving image to which the frame corresponding to the entry belongs. The frame number 308 stores the number (frame number) in the moving image of the frame corresponding to the entry. The frame number is incremented for each frame after the first frame (start frame) in the moving image is set to 1. The self-position information 309 stores information (self-position information) indicating the position and orientation of the camera when a frame image (frame image) corresponding to the entry is captured. As the self-position information, for example, coordinates (X, Y, Z) of the shooting position (viewpoint) in the space of the three-dimensional map, and vectors (X ′, Y ′, Z ′) indicating the shooting direction (line of sight) of the camera. It is good. The three-dimensional point group data 310 stores data (point group data) indicating the position of each point group (a plurality of points) on the three-dimensional map. There are various formats for the point cloud data. For example, each point may be represented in the (X, Y, Z) format.

<Article database>
FIG. 4 is a configuration diagram of an article database according to an embodiment.

  The article database 209 is a database for storing article information (article information) detected from a moving image (strictly speaking, a frame image in the moving image). The article database 209 includes a table that stores an entry corresponding to each article. An entry in the table of the article database 209 includes columns of an article ID 401, an article type 402, a three-dimensional coordinate 403, a moving image ID 404, and a frame number 405.

  The article ID 401 stores an ID (article ID) for uniquely identifying the article detected from the moving image. The item type 402 stores a value (article type) indicating the type of the item detected from the moving image. Examples of the item type include “PC”, “chair”, “desk”, and “shelf”. The three-dimensional coordinates 403 store coordinates (X, Y, Z) indicating the position of the article corresponding to the entry in the three-dimensional space. The moving image ID 404 stores the moving image ID of the moving image data in which the article corresponding to the entry is detected. The frame number 405 stores the frame number in the moving image data in which the article corresponding to the entry is detected. In the moving image, when the same article is detected in a plurality of frames, the frame number 405 stores the frame numbers of the plurality of frames in which the article is detected.

<Smartphone>
FIG. 5 is a configuration diagram of a smartphone according to an embodiment.
<Smartphone>
The smartphone 102 includes a processor unit 501, a storage device 502, a camera unit 504 as an example of a moving image shooting unit that performs moving image shooting, a display unit 505 as an example of a display unit, and a network unit 506.

  The network unit 506 is an interface such as a wired LAN card or a wireless LAN card, and communicates with other devices (for example, the article management server 101 and the article management terminal 103) via the network 105. The processor unit 501 executes various processes in accordance with programs stored in the storage device 502. The storage device 502 is, for example, an SSD, a flash memory, or a RAM, and stores a program executed by the processor unit 501 and data used by the processor unit 501. In the present embodiment, the storage device 502 stores the map application 503 as a program. The display unit 505 is, for example, a touch panel for displaying various images and receiving an operation input.

  The smartphone 102 is used by a photographer to take an image in a space (management target space) for managing articles, for example, in an office or factory. As a photographer's operation when shooting a moving image, the map application 503 is activated, and after logging in, the user only moves while shooting the management target space. In addition, when using the video taken with the camera of the smartphone 102 in this way, the range captured by the camera of the smartphone 102 is only a few meters ahead and the range of the viewing angle. It is necessary to walk so that all the articles to be managed in the management target space are reflected. Note that a 3D image based on the 3D map generated by the captured moving image is displayed on the display unit 505 of the smartphone 102 so that the photographer of the smartphone 102 appropriately captures the image while checking his / her position. You may make it display. For example, as a method of displaying a 3D image based on the 3D map on the smartphone 102, the map generation module 204 and the self-position estimation module 205 in the article management program 203 of the article management server 101 are incorporated in the map application 503. Alternatively, an image based on the three-dimensional map generated by the article management server 101 may be received via the network 105 and displayed. When the communication speed of the network 105 is low, the image quality based on the three-dimensional map may be roughened (that is, the data amount may be suppressed) and transmitted. Here, the processor unit 501 executes the map generation module 204 and the self-position estimation module 205 incorporated in the map application 503, or the processor unit 201 executes the map generation module 204 and the self-position estimation module 205. By doing so, a photographing support unit is configured.

<Map application>
The map application 503 is a program installed in the smartphone 102 in advance, and has a function of capturing moving images by the smartphone 102 and transferring the captured moving image data to the article management server 101.

<Goods management terminal>
FIG. 6 is a configuration diagram of an article confirmation terminal according to an embodiment.

  The article management terminal 103 is a terminal used by an article manager or a person who audits articles (hereinafter referred to as an article manager), and can browse and manage the article management position and state. The article management terminal 103 is configured by, for example, a general PC (Personal Computer), and includes a processor unit 601, a storage device 602, a display unit 603, a mouse / keyboard unit 605, and a network unit 606.

  The network unit 606 is an interface such as a wired LAN card or a wireless LAN card, and communicates with another device (for example, the article management server 101) via the network 105. The processor unit 601 executes various processes according to a program stored in the storage device 602. The storage device 602 is, for example, a hard disk drive, SSD, RAM, or the like, and stores programs executed by the processor unit 601 and data used by the processor unit 601. In the present embodiment, the storage device 602 stores an article management viewer 603 as a program. The display unit 604 displays various images, for example. The mouse / keyboard unit 605 receives various operation inputs from the user.

<Article Management Viewer>
The article management viewer 603 has a function capable of managing, browsing, and searching for articles on a three-dimensional image based on a three-dimensional map. The article management viewer 603 is, for example, a server / client type application that communicates with the article management program 203 of the article management server 101. The article management viewer 601 transmits a request based on a user operation to the article management server 101, receives a response from each module in the article management program 203 of the article management server 101, and displays the result. As an operation for the article management viewer 601, an operation for browsing a bird's-eye view three-dimensional image, an operation for confirming an arbitrary point or range on the three-dimensional image by enlarging / reducing / rotating, and the like are possible. Further, the article management viewer 601 has a function of searching for what kind of articles are present in the three-dimensional area when a three-dimensional area is designated on the three-dimensional image. Furthermore, the article management viewer 601 can also perform a text search for the article type. In addition, the article management viewer 601 can display the situation at the time of moving image shooting of the designated article. Thereby, for example, it is possible to confirm the state of the external lamp of the designated server machine. Furthermore, the article management viewer 601 can also display a movement history of the photographer walking with the smartphone 102. Here, when the processor unit 601 executes the article management viewer 603, an existing article specifying unit, a display designation receiving unit, an article designation receiving unit, an area designation receiving unit, and a display control unit are configured.

  Next, a description is given of processing operation in the article management system according to the present embodiment.

<System startup>
FIG. 7 is a flowchart of a system use start process according to an embodiment.

  When starting to use the system, it is necessary for the photographer to bring the smartphone 102 to the management target space where the management target article exists and start the map application 501 of the smartphone 102.

  When the map application 503 of the smartphone 102 is activated, the map application 503 is ready to accept a login from the photographer, accepts the login from the photographer (S701), and starts communication with the article management program 203 of the article management server 101. (S702). At this time, the map application 501 acquires photographer information in the smartphone 102 based on the login information, and transfers the photographer information to the article management program 203 (S703). The photographer information may be stored in association with the login ID by the article management server 101 and acquired by the article management server 101 based on the login ID. At the time of login, login authentication is performed, and subsequent processing is executed after the authentication is performed.

  The map application 503 activates the camera unit 504 of the smartphone 102, displays a moving image shot by the camera unit 504 on the display unit 505, and starts moving image shooting (S704). Next, the map application 503 starts transmission of moving image data to the article management program 203 (S705).

  Upon receiving the moving image data from the map application 503, the item management program 203 of the item management server 101 issues a moving image ID of the received moving image (S706), and starts map generation / article detection processing (see FIG. 8) (see FIG. 8). S707).

<Map generation / article detection process>
FIG. 8 is a flowchart of map generation / article detection processing according to an embodiment.

  Here, in FIG. 8, a case where the moving image data is stream-processed from the smartphone 102 to the article management server 101 is described as an example, but non-stream processing may be performed. In the non-stream processing, the moving image data may be saved as a file in the local storage (storage device 502) of the smartphone 102 and transferred to the article management server 101 at an arbitrary timing.

  First, the article management program 203 adds a new entry to the moving image management table 301 and registers information of the transferred moving image data (S801). Specifically, the article management program 203 stores the issued video ID in the video ID 302 of the added entry, and stores the video save destination file path (predetermined path, Or the path specified by the user), the photographer information 304 is stored in the photographer information 304, and the shooting start date and time is stored in the time stamp 305.

  Next, the article management program 203 repeatedly executes the loop A moving image processing (S802 to S810).

  In the moving image processing, the article management module 206 of the article management program 203 reads one frame image of the moving image (S802), and inputs the frame image to the article detection module 207 (S803). The article detection module 207 recognizes the article in the frame image and detects the position of the article in the frame (S804). Next, the article detection module 207 outputs an entry including the article type and coordinate data indicating the position in the frame to the article detection table 1101 (see FIG. 11) of the storage device 202 (S805).

  Here, the article detection processing from S803 to S805 will be specifically described.

  FIG. 9 is a diagram illustrating an example of a frame image according to an embodiment. FIG. 10 is a diagram illustrating article recognition and position detection in a frame image according to an embodiment. Figure 11 is a block diagram of an article detection table according to the embodiment.

  For example, when the frame image 901 shown in FIG. 9 is input to the article detection module 207 (S803), the article detection module 207 displays the reception sofa 1001, reception sofa 1002, and the like in the frame image 901 as shown in FIG. The types of articles to be managed such as the reception table 1003, the reception sofa 1004, the planting tree 1005, etc. are recognized, and the position of the rectangular area including the article is detected as the position of the article (S804). Thereafter, the article detection module 207 outputs an entry corresponding to each article to the article detection table 1101 (S806).

  The article detection table 1101 includes an entry corresponding to each article, and the entry includes an article type 1102 and in-frame coordinates 1103. The item type 1102 stores information indicating the type of item corresponding to the entry. The information indicating the type of the article may be a label corresponding to the type, but in the example of FIG. 11, the name of the article type is used. In the in-frame coordinates 1103, rectangular coordinate data corresponding to the article in the frame image is stored. In the example of FIG. 11, the x and y coordinates of the upper left point of the rectangle corresponding to the article when the upper left position of the frame image is the origin (0, 0), the horizontal width of the rectangle, and the vertical width of the rectangle It is said. The processing of S802 to S805 completes identification of the type of article existing in one frame and the position of the article.

  Next, the map generating module 204 reads the frame image (S806). Next, the self-position estimation module 205 and the map generation module 206 generate a three-dimensional map (three-dimensional position information) and estimate the self-position (S807). In generating the three-dimensional map, for example, VisualSLAM technology can be used. There are various realization methods in the VisualSLAM technology. For example, the feature points of a plurality of frame images are matched, and the shooting position (self-position) and the posture of the camera between the frame images are changed in three dimensions. The position of the feature point is calculated by optimization calculation, and for all pixels of the frame image, the depth estimation of the pixel and self-position / posture estimation are performed from the luminance change between the frame images of the part with high luminance gradient There is a method of generating 3D point cloud data by mapping pixels of a frame image to a 3D space. Here, as an example, a point cloud in a three-dimensional space is generated from a plurality of frame images, and the point cloud is used as a component of a three-dimensional map. Note that texture processing may be performed on the point cloud data to generate a three-dimensional map that is easier for humans to understand. Note that the generation of the three-dimensional map and the estimation of the self position in S807 and S808 require a plurality of frame images. For example, these processes are executed for the first frame image immediately after the processing in FIG. Instead, the subsequent frame images may be processed using one of the temporally previous frame images.

  Next, the article management module 206 executes processing for registering each article stored in the article detection table 1101 in the article database 207 (S808). Specifically, if it is a new article, the article management module 206 adds a new entry to the article database 207, determines an article ID for identifying the article, stores the article ID in the article ID 401, and is detected. The item type is stored in the item type 402, the three-dimensional coordinate of the item is stored in the three-dimensional coordinate 403, the moving image ID of the moving image data is stored in the moving image ID 404, and the frame number where the item is detected is added to the frame number 405. Here, the article management module 206 converts the coordinates in the frame stored in the position coordinates 1103 of the article detection table 1101 into the coordinate system of the three-dimensional map. As a method of converting the coordinates in the frame into the coordinate system of the three-dimensional map, for example, it can be specified from a plurality of frame images in which one corresponding article exists. If the same article has already been detected, that is, if it has been detected in a frame image processed in the past, the article management module 206 enters the frame number 405 of the entry in the article database 207 corresponding to the article. To the frame number of the frame to be processed. Here, whether or not the same article is detected may be determined based on whether or not the three-dimensional coordinates of the article are within a predetermined distance.

  Next, the map generation module 204 adds a new entry to the map data management table 306, stores the moving image ID of the processing target moving image in the moving image ID 307 of the entry, and stores the frame number of the processing target frame in the frame number 308. The self-position detected in S804 is stored in the self-position information 309, and the three-dimensional coordinates of each point generated in S807 are stored in the three-dimensional point group coordinates 310. (S809). Thereafter, the map generation module 204 stores the moving image data in a file of the path specified by the moving image storage destination file path 303 of the entry corresponding to the moving image in the map database 208 (S810). The moving image processing from S802 to S810 is repeatedly executed until transfer of moving image data from the map application 503 of the smartphone 102 is completed.

  Next, it is a figure explaining the three-dimensional image based on the production | generation of a three-dimensional map when a photographer walks, performing video imaging | photography with the smart phone 102, and a three-dimensional map.

  FIG. 12 is a diagram illustrating the generation of a 3D map and a 3D image based on the 3D map according to an embodiment. FIG. 13 is a diagram illustrating a three-dimensional image of another viewpoint based on a three-dimensional map according to an embodiment.

  When the photographer activates the map application 503 and walks on the smartphone while shooting a moving image with the camera unit 504 of the camera 102, the moving image data includes, for example, frame images 1201 to 1207. Note that the moving image data actually includes more continuous frame images than the frame images 1201 to 1207.

  Such moving image data is transferred to the article management program 203 of the article management server 101 by the map application 503. In the article management program 203 of the article management server 101, the map generation module 204 and the self-position estimation module 205 work together to generate a three-dimensional map of the point cloud in the photographed space. By using the generated three-dimensional map of the point group, a three-dimensional image 1208 showing a space observed from a certain viewpoint can be generated. Such a three-dimensional image can be created and displayed by the article management viewer 603 of the article confirmation terminal 103. In the three-dimensional image 1208, the movement history line 1209 can be displayed by connecting the movement history of the photographer (history of the self position) with a line. This three-dimensional image 1208 shows that the photographer has moved from the point 1210 to the point 1211. Note that the frame image captured at the point 1210 is a frame image 1201, and the frame image captured at the point 1211 is a frame image 1207.

  In addition, the article detection module 207 recognizes an article to be managed and detects the position of the article in the coordinate system of the three-dimensional map. FIG. 12 shows a display example when the article to be managed is a PC (including only a display) and the article detection module 207 causes the article detection module 207 to perform machine learning. In FIG. 12, the PC (PC 1212 or the like) that has been recognized and whose position has been detected is displayed as “▲” in the three-dimensional image 1208. In the article detection table 1101 shown in FIG. 11, the position of the article is acquired as rectangular position information. However, on the three-dimensional map, for example, the three-dimensional position of the article corresponds to the center of the rectangle. It may be mapped to a three-dimensional position. For example, the viewpoint at the time of generating a three-dimensional image based on a three-dimensional map may be changed. For example, a three-dimensional image as shown in FIG. 1301 may be generated and displayed. Thus, by mapping the position of the article in the three-dimensional image based on the three-dimensional map, the position of the article in the space can be grasped more efficiently.

<3D image display processing>
FIG. 14 is a flowchart of 3D image display processing according to an embodiment. FIG. 15 is a diagram illustrating an article management viewer screen according to an embodiment.

  When the article management viewer 603 is executed by the processor unit 601, the article management viewer 603 displays a three-dimensional image of the target space on the article management viewer screen 1501 (see FIG. 15) and a visible area (area displayed on the display). The item type and the number of items are displayed in the item information window 1502. In addition, when the photographing movement history button 1503 is pressed, the article management viewer 603 displays the movement history of the photographer who has photographed the moving image for generating the three-dimensional map that is the basis of the three-dimensional image. Are displayed on the three-dimensional image. The article management viewer 603 illustrating a process of displaying the screen below.

  The three-dimensional image display process is started when the article management apparatus 103 accepts an activation instruction for the article management viewer 603 by the article manager and activates the article management viewer 603. The article management viewer 606 communicates with the article management program 203 and reads the three-dimensional map data of a predetermined area (for example, a preset area or an area designated by the article manager) from the map database 208 ( S1401). Next, the article management viewer 606 reads article information (article ID, article type, three-dimensional coordinates, and frame number) of an entry corresponding to the same moving image ID as the three-dimensional map data read in S1401 from the article database 209 (S1402). ).

  Next, the article management viewer 603 displays a three-dimensional image area (specifically, a viewpoint in a three-dimensional space for generating a three-dimensional image and a viewpoint of a three-dimensional image displayed on the article management viewer screen 1501 of the display unit 604 of the article management terminal 103). (Gaze direction) is determined, and the article type and the number of articles in the three-dimensional space corresponding to the area displayed in the three-dimensional image are counted (S1403). For an article in a three-dimensional space corresponding to a region displayed in the three-dimensional image, the three-dimensional image of the article is displayed in the three-dimensional space displayed in the three-dimensional image specified by the viewpoint and the line-of-sight direction of the three-dimensional image. It can be specified by the coordinate belonging.

  Next, the article management viewer 603 includes a plurality of points as shown in FIG. 15 based on the acquired three-dimensional map data and the acquired article information, and includes an object indicating the position of the article 3 The three-dimensional image is displayed on the article management viewer screen 1501 (S1404). In the present embodiment, different objects are displayed depending on the type of article. Here, each point in the three-dimensional image is at a position obtained by converting the three-dimensional coordinates indicated by the three-dimensional point cloud data into the coordinate system of the three-dimensional image specified by the viewpoint and the line-of-sight direction of the three-dimensional image. The displayed object corresponding to the article is displayed at a position obtained by converting the three-dimensional coordinates of the article into the coordinate system of the three-dimensional image specified by the viewpoint and line-of-sight direction of the three-dimensional image.

  According to the article management viewer screen 1501, the article manager can view and confirm the position of the article on the three-dimensional image, and can confirm the type of article and the number of articles in the displayed area. it can.

<Display change processing by the article management viewer when operated by the article manager>
FIG. 16 is a flowchart of a display change process according to an embodiment. FIG. 17 is a diagram illustrating region designation search according to an embodiment. FIG. 18 is a diagram illustrating an article state confirmation screen according to an embodiment. FIG. 19 is a diagram illustrating an article management viewer screen that displays a movement history according to an embodiment.

  The display change process is a process that is executed when the article manager operates on the article management viewer screen 1501 using the mouse / keyboard unit 605.

  The article management viewer 603 receives an operation by the article manager using the mouse / keyboard unit 605 and determines the type of the operation (S1601).

  When the operation type is an operation for instructing an image change such as enlargement / reduction / rotation using the mouse / keyboard unit 605 with respect to the three-dimensional image on the article management viewer screen 1501 (S1601: image change instruction). In step S1602, the article management viewer 603 generates a three-dimensional image in accordance with the image change instruction, and acquires the type and number of articles in the area displayed in the newly generated three-dimensional image. . Next, the article management viewer 603 displays the generated three-dimensional image and also displays article information (type and number of articles) in the article information window 1502 (S1603). By performing this operation, the article manager can display a three-dimensional image corresponding to the three-dimensional space with various sizes and angles, and can appropriately display the position of the article in the three-dimensional image. it can. In addition, the position and number of articles in the display area can be easily confirmed.

  When the operation type is an operation for designating a three-dimensional area (search area) to be searched for the type and number of articles on the three-dimensional image on the article management viewer screen 1501 (S1601: area) In the designated search, the type and number of articles existing in the three-dimensional coordinates corresponding to the designated search area are acquired and calculated based on the article information (S1604). Here, as a method for designating a three-dimensional search area on the article management viewer screen 1501, for example, the article management viewer 603 uses a cuboid in a three-dimensional space for designating an area as shown in FIG. A search area 1701 to be displayed may be displayed, and an instruction to deform the search area 1701 (for example, horizontal adjustment, vertical adjustment, depth adjustment, etc.) may be received from the mouse / keyboard unit 605.

  Next, the article management viewer 603 displays the article information (article type and number) obtained in S1604 in the article information window 1502 (S1603).

  Also, the type of operation is an operation for displaying details of the state of any article in the three-dimensional image on the article management viewer screen 1501 (for example, a click on the article in the three-dimensional image by the mouse / keyboard unit). In this case, the article management viewer 603 transfers the article ID of the designated article to the article management program 203 of the article management server 101 (S1605). In response to this, the article management program 203 searches the article database 209 using the acquired article ID, acquires a moving picture ID from the moving picture ID 404 of the entry obtained by the search, and acquires a frame number from the frame number 405 (S1606). ). Next, the article management program 203 retrieves the entry corresponding to the movie ID from the movie management table 301 of the map database 208, and obtains the file path of the movie file from the movie storage destination file bus 303 of the entry obtained by the search. Then, the moving image file 210 indicated by the file path is acquired, the still image (frame image) corresponding to the frame number acquired in S1606 of the moving image file 210 is acquired (S1607), and the frame image is returned to the article management viewer 603. (S1608). Note that the article management program 203 may return the photographer information and the date and time of photographing together with the frame image to the article management viewer 603.

  Thereafter, the article management viewer 603 displays the article image and article information based on the frame image returned from the article management program 203 (S1609). The information on the article to be displayed may include the type of the article, the name of the photographer of the moving image, the shooting date and time, and the like. In addition, when displaying the details of an article in this way, it is preferable to take an image close to the article so that the state of the article can be clearly understood during movie shooting.

  For example, when a pentagonal object (an object corresponding to a large display) on a three-dimensional image on the article management viewer screen 1501 shown in FIG. 15 is clicked, an article image 1802 on the large display and article information are displayed as shown in FIG. An article state confirmation screen 1801 including the contained article information window 1803 is displayed.

  By viewing the article status confirmation screen 1801, the article manager can confirm the status of the article at the time of photographing in more detail. In addition, since the photographer and the shooting date and time are displayed in the article information window 1802, the article information window 1802 can be used as evidence for checking the article status at that date and time.

  In S1606, a plurality of frame numbers may be acquired from the frame number 405. In S1607, frame images of all these frame numbers may be transmitted, or some of them may be transmitted. Good. When a plurality of frame images are transmitted, the article management viewer 603 may select and display any one frame image from the plurality of frame images in S1609. As a method of selecting a frame image to be displayed, for example, a frame image in which the size of the target article in the image is large may be selected. A frame image in which the target article is located at or near the center of the frame image is selected. May be. In this way, the state of the article can be confirmed quickly and in detail. When a plurality of frame images are transmitted, the article management viewer 603 may display the plurality of frame images so as to be selectable in S1609. As a method of displaying in a selectable manner, any one of them may be displayed and a button for receiving an instruction to change the frame image to be displayed may be displayed.

  When the operation type is an operation in which the shooting movement history button 1503 for displaying the movement history in the three-dimensional image on the article management viewer screen 1501 is pressed, the article management viewer 603 displays the operation history. The moving image ID of the moving image that is the basis of the three-dimensional map data corresponding to the three-dimensional image is transferred to the item management program 203 of the item management server 101 (S1610).

  The article management program 203 includes self-position information (all time-series self-position information of movement history) of self-position information 309 of all entries corresponding to the moving picture IDs of the map management table 306 of the map database 208, and a moving picture management table. The photographer information 304 (photographer, photographing date and time) of the photographer information 304 of the entry corresponding to the moving image ID 301 is acquired (S1611), and the acquired information is returned to the article management viewer 603 (S1612). The article management viewer 603 specifies a position on the three-dimensional image corresponding to each received self-position information, displays a shooting history 1901 in which those positions are connected by a line in time series, and receives received photographer information. It is displayed on the photographing information window 1902 (S1613). For example, when the shooting movement history button 1503 is pressed on the article management viewer screen 1501 shown in FIG. 15, a shooting history 1901 and a shooting information window 1902 are displayed as shown in FIG. . According to the article management viewer screen 1501, the article manager can confirm how the photographer of the article has moved and photographed, and easily confirms when and how the photographer was created. can do.

  In addition, this invention is not limited to the above-mentioned embodiment, In the range which does not deviate from the meaning of this invention, it can change suitably and can implement.

  For example, in the above embodiment, the article management system has been illustrated as an example in which the smartphone 102, the article management server 101, and the article management terminal 103 are configured as separate devices, but the present invention is not limited thereto, Any one of a plurality of devices may be configured as one device. For example, all functions of the article management server 101 and the article management terminal 103 may be included in the smartphone, and the article management system may be realized by the smartphone, and the function of the article management terminal 103 may be included in the smartphone. The article management system may be realized by a smartphone and an article management server.

  In the above embodiment, the article management viewer 603 generates a three-dimensional image based on the three-dimensional map data. However, the present invention is not limited to this, and the three-dimensional image to be displayed is displayed on the article management server 101. It may be created on the article management program 203 side, and the article management viewer 603 may execute processing for displaying the created three-dimensional image.

  In the above embodiment, part or all of the processing performed by the processor unit may be performed by a hardware circuit. In addition, the program in the above embodiment may be installed from a program source. The program source may be a program distribution server or a storage medium (for example, a portable storage medium).

DESCRIPTION OF SYMBOLS 10 ... Goods management system 101 ... Goods management server 102 ... Smartphone 103 ... Goods management terminal 201,501,601 ... Processor part 202,502,602 ... Storage device 203 ... Goods management program 208 ... Map database 209: Article database 210: Video file 211, 506, 606 Network unit 503 Map application 504 Camera unit 603 Item management viewer

Claims (11)

An article management system for managing articles to be managed arranged in a predetermined space,
A three-dimensional position specifying unit that specifies three-dimensional position information capable of specifying a three-dimensional position of a point group in the predetermined space based on a moving image obtained by photographing the predetermined space;
An article recognition unit that recognizes the article in the frame image based on one or more frame images in the moving image;
An article position specifying unit for specifying article position information capable of specifying a three-dimensional position of the recognized article in the predetermined space;
An article management system. Based on the three-dimensional position information of the point group, a three-dimensional image from a predetermined viewpoint in the space is displayed, and the presence position of the article is displayed in the three-dimensional image based on the article position information. The article management system according to claim 1, further comprising a display control unit. A presence article specifying unit for specifying the type of article and the number of articles present in the region corresponding to the three-dimensional image;
The article management system according to claim 2, wherein the display control unit displays the specified kind of article and the number of articles. A display designation receiving unit for receiving a designation related to the display of the three-dimensional image;
The article management system according to claim 2, wherein the display control unit displays a three-dimensional image based on the designation. An article designation receiving unit for accepting designation of articles in the three-dimensional image;
The display control unit identifies one or more frame images including the designated article, and displays an image of the article based on at least one of the identified frame images. The article management system according to any one of the above. The article management system according to claim 5, wherein the display control unit displays a plurality of frame images so as to be selectable when there are a plurality of frame images including the identified article. When there are a plurality of frame images including the specified article, the display control unit has a large range in which the article exists or a frame in which the article exists at or near the center from among the plurality of frame images. The article management system according to claim 5, wherein an image is selected and displayed. An area designation accepting unit for accepting designation of a three-dimensional area in the space;
The existence article specifying unit specifies the type of article and the number of articles existing in the designated three-dimensional area,
The article management system according to any one of claims 2 to 7, wherein the display control unit displays a type of the article specified as being present in the designated three-dimensional region and a number of the articles. . Equipped with a photographic device,
The photographing apparatus includes a moving image photographing unit that photographs a moving image of the predetermined space,
A photographing support unit for displaying a three-dimensional image based on the three-dimensional position information of the point group in the predetermined space identified by the three-dimensional position identifying unit on the display unit of the photographing apparatus when photographing the moving image; The article management system according to any one of claims 1 to 8, further comprising: A photographing device and a processing server;
The imaging device
A moving image shooting unit for shooting a moving image of the predetermined space;
The processing server
The article management system according to any one of claims 1 to 9, comprising the three-dimensional position specifying unit, the article recognizing unit, and the article position specifying unit. An article management program for causing a computer constituting an article management system for managing articles to be managed arranged in a predetermined space to execute the program,
The computer,
A three-dimensional position specifying unit for specifying three-dimensional position information of a point group in the predetermined space based on a moving image obtained by photographing the predetermined space;
An article recognition unit that recognizes the article in the frame image based on one or more frame images in the moving image;
An article position specifying unit that specifies article position information that is a three-dimensional position of the recognized article in the predetermined space;
Goods management program to make it function.