WO2025186882A1 - Position specification system and recording medium - Google Patents

Abstract

Provided is a system in which a control unit 22 of a server 20 that has acquired an image captured by a terminal device 10 specifies the position of the terminal device. The server comprises a database 23 which stores a large number of 3D maps indicating respective areas and feature data related to the 3D maps. The feature data includes data obtained by vectorizing an image of a 3D map and text information related thereto, and context information of a 3D map, etc. A map narrowing unit 222 of the control unit vectorizes the acquired image, etc., performs vector retrieval with the feature data stored in the database, and provides the acquired image and the context information of the 3D map to an LLM to narrow down candidates of the 3D map (exclude 3D maps of similar shapes). A map matching unit 221 can accurately specify a position and a posture by matching the shape with the narrowed 3D map candidates.

Description

Location identification system and recording medium

The present invention relates to a location identification system and a recording medium.

Measuring the position and orientation of an imaging device based on image information is used for a variety of purposes, such as aligning real space with virtual objects in mixed reality/augmented reality, and estimating the position of users carrying the imaging device, robots, cars, drones, and other mobility devices. Position detection using a VPS (Viral Positioning System) can identify a location even in environments where GPS satellite signals cannot be received. Therefore, for example, if a user carrying an imaging device is inside a building or underground, it is possible to identify their location within that space.

This type of system for identifying a location using captured image information is disclosed, for example, in Patent Document 1.

Patent No. 7361075

Sales areas in supermarkets and convenience stores tend to have similar display shelves arranged in a similar layout. Furthermore, indoor stores in commercial complexes tend to use rooms of the same size along the corridors, and some have similar entrances. Exhibitions held at exhibition halls often have numerous booths with the same dimensions and shapes, arranged in the same layout.

In such cases, for example, if there are many similarly shaped sales areas within the same supermarket, the feature points of the images taken at each sales area will be similar. When similar feature points exist, the accuracy of location identification using VPS will decrease. This also occurs indoors in commercial complexes, exhibition halls, etc.

Furthermore, when performing image-based location identification within a wide indoor area, there are many images to compare. If there are many images with similar features, the system may not be able to correctly detect the location, and may freeze up without being able to identify one.

The above-mentioned problems are described as being independent of each other, and the present invention does not necessarily have to solve all of the problems described; it is sufficient if it can solve at least one of the problems. Furthermore, we intend to obtain rights to the configurations that solve these problems separately through divisional applications, amendments, etc.

(1) A location identification system that identifies the location where an acquired image was taken should preferably have a map filtering function that narrows down the 3D map candidates to be searched using feature data abstracted based on the image, and a map matching function that matches the shape of the image with the narrowed down 3D map candidates to identify the location where the image was taken.

By doing this, it is possible to narrow down the area in terms of content (narrow down the 3D maps) and then perform shape feature point matching with high accuracy. The process of narrowing down 3D map candidates includes, for example, selecting a predetermined number of 3D maps for shape matching from multiple 3D maps, selecting a predetermined number of 3D map portions of the area for shape matching within a single 3D map, or a combination of these.

(2) The map narrowing function may be configured to perform the abstraction using a large-scale language model with context information of the image and the 3D map as input, thereby narrowing down the candidates for the 3D map.

Searches using large-scale language models cannot pinpoint location or pinpoint posture or orientation, but they can identify similar shapes by taking their content into account. This allows for accurate selection of 3D map candidates. Then, by narrowing down the 3D map candidates, the map matching function in the subsequent stage can use shape matching to accurately pinpoint location, etc.

(3) The map refinement function may be configured to vectorize information based on the image to create the feature data.

(4) The map narrowing function may be configured to vectorize text information related to the image to create the feature data. Text information related to the image may be created by character recognition of characters contained in the image, by image recognition to indicate objects present in the image, or an explanatory document for the image. The explanatory document may be one created in advance by a person or one generated by providing the image to a generation AI.

(5) The abstraction performed by the map refinement function may be configured to be performed using the image and context information related to that image.

(6) The map narrowing function may be configured to use location information detected by the device that captured the image.

(7) It is preferable to configure the system to include a database that stores feature data to be compared and 3D maps and that can be accessed by the map narrowing function and the map matching function.

(8) The system may include a camera and a terminal having a communication function, and the terminal may be configured to use the communication function to send images captured by the camera to a server having the map filtering function. The terminal may also have a function to acquire the location identified by the map matching function and notify the user of the acquired location.

(9) The recording medium of the present invention may be a computer-readable recording medium on which a program for causing a computer to realize the functions of the location identification system described in any one of (1) to (8) is recorded.

The components of each invention described above in (1) to (8) can be combined in any combination of two or more, and the invention can be constituted by such combinations.

The present invention narrows down the candidate 3D maps to be searched using abstracted feature data, such as vector search or inference using large-scale language models, and then matches feature points with the narrowed-down 3D maps to accurately identify a location even within a space containing similar shapes.

FIG. 1 is a diagram showing a preferred embodiment of a location specifying system according to the present invention. FIG. 2 is a diagram showing an example of an actual system configuration. FIG. 3 is a flowchart illustrating the processing of the control unit of the server. FIG. 4 is a diagram showing an example of a place for identifying an indoor position. FIG. 5 is a diagram showing an example of a place for identifying an indoor position. FIG. 6 is a diagram showing an example of a place for identifying an indoor position. FIG. 7 is a diagram showing an example of a place for identifying an indoor position. FIG. 8 is a diagram showing an example of a place for identifying an indoor position. FIG. 9 is a diagram showing an example of a place for identifying an indoor position.

Below, a preferred embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention should not be construed as being limited to this embodiment, and various changes, modifications, and improvements may be made based on the knowledge of those skilled in the art, provided that they do not deviate from the scope of the present invention.

The positioning system of the present invention accurately identifies positions indoors where GPS signals cannot be received accurately, such as in exhibition halls, commercial facilities, various buildings, underground shopping malls, and underground parking lots. Based on information acquired in advance using cameras and LiDAR (Light Detection and Ranging) for the indoor area to be located, the system uses various computational processes, statistical models, and machine learning models to create and store a 3D map showing a specific indoor area, along with feature data related to the 3D map. The positioning system then acquires image data actually taken on-site, narrows down indoor locations based on the acquired image data and the stored feature data, and is further equipped with the function of accurately identifying the location by matching feature points with the stored 3D map.

FIG. 1 is a block diagram showing a preferred embodiment of a location identification system 1 according to the present invention, and FIG. 2 is a diagram showing an example of an actual configuration. As shown in FIG. 1, the location identification system 1 has a function in which a server 20 acquires image data captured by a terminal device 10 via a network 2 and identifies the location of the terminal device 10 based on the acquired image data. The network 2 may be, for example, a public line or the Internet, and is preferably accessible even when the devices are in distant locations.

The terminal device 10 includes a control unit 11, a camera 12, an input unit 13, a communication unit 14, a display unit 15, a GPS receiver unit 16, etc. When the terminal device 10 is carried by a person, it may be a smartphone, smart glasses, etc. When the terminal device 10 is a smartphone, the hardware that constitutes the terminal device 10 (control unit 11, camera 12, input unit 13, communication unit 14, display unit 15, GPS receiver unit 16) uses equipment that is standard equipment on smartphones.

The control unit 11 controls each part of the terminal device 10. The control unit 11 is, for example, a computer including a processor and memory. The memory is, for example, a main storage device having RAM (Random Access Memory) and ROM (Read Only Memory). The processor temporarily stores programs read from the ROM in the RAM. The RAM also provides a working area for the processor. The processor performs various controls by performing arithmetic processing while temporarily storing data generated during program execution in the RAM. If the terminal device 10 is a smartphone, the process for determining location may be performed, for example, by the control unit launching a smartphone app.

The communication unit 14 connects to the above-mentioned network 2 and has functions such as sending and receiving data and information with other devices connected to network 2, and connecting to Wi-Fi access points.

The control unit 11 displays images captured by the camera 12 on the display unit 15. The control unit 11 also transmits image data being captured to the server 20 using the network 2. The control unit 11 also transmits sensor information based on the current location to the server 20 using the network 2. The sensor information includes, for example, location information (longitude and latitude) received and acquired by the GPS receiving unit 16, and information identifying a Wi-Fi access point acquired by the communication unit 14. The timing for transmitting this information may be periodically at a predetermined interval, or triggered by a transmission instruction received by the input unit 13. The control unit 11 also has a function for displaying predetermined information on the display unit 15 based on the current location acquired from the server 20.

The server 20 comprises a communication unit 21 having an interface for connecting to and communicating with a network 2 such as an internet connection function, a control unit 22 that controls the operation of the server 20, and a database 23 that stores data for location identification. In addition, while Figure 1 shows an example in which the server 20 is composed of a single server computer, as shown in Figure 2, it may also be composed of multiple computers connected directly or via a network. The server that implements the database is not limited to a physical server, and various other forms may be used, such as being composed of a software program or implemented in the cloud.

The database 23 stores data for identifying a location based on images captured by the camera 12 of the terminal device 10. The database 23 includes a 3D map storage unit 231 and a feature data storage unit 232. The 3D map storage unit 231 stores a 3D map of the target area for which location is to be identified, created in advance. In other words, the 3D map is data based on a three-dimensional shape composed of a collection of feature points (x, y, z) or a polygon mesh created using a large number of images and the LiDAR depth data associated with each image. Therefore, a system administrator or the like moves around the indoor area for which location is to be identified in advance, taking images using a camera to acquire image data for each location, and acquiring depth data using LiDAR (Light Detection and Ranging) as needed. Various calculations and other processes are performed based on the multiple image data and depth data acquired at a certain indoor location to create a 3D map. This 3D map creation process can be performed using existing technology.

In the same indoor space, there can be multiple areas with similar shapes. As a result, there will be similar shapes and feature points on 3D maps in different locations. In particular, at exhibition halls, for the same exhibition, there will be many booths with similar dimensions and shapes, and the layout of multiple booths within the venue will often be the same. Furthermore, in large commercial facilities and shopping malls, there may be multiple stores with similar entrances. Furthermore, even within the same store, if many identical display shelves are prepared and arranged in the same layout, there will be many similar 3D maps.

The process of creating a 3D map for a given location involves, for example, dividing the area so that the characteristics of the 3D map data are uniquely determined. A 3D map of the divided area can be created by performing three-dimensional reconstruction from a sequence of slightly overlapping images acquired in the divided area and the posture information associated with each image. Furthermore, if seismic intensity data acquired using LiDAR is available, this can also be used to create a 3D map of the divided area. If LiDAR data is not available, the 3D map can be created based on image data alone.

In the case of an exhibition hall, for example, this division range may be a fixed area such as a corner of an event venue or a booth. In the case of a shopping mall, the division range may be, for example, an area including the entrance to each store. Furthermore, the division range within a store may be, for example, an area including one or more display shelves. The division range can be selected as appropriate, and in either case, one 3D map is created for each divided area. Each 3D map is assigned an ID, and location information identifying the actual area indoors is also associated and registered.

In this way, 3D maps are created for each of the multiple areas that a specific indoor area is divided into, so for example, multiple 3D maps are prepared for the same exhibition or event venue held at a single exhibition center. 3D maps are also created for each of the multiple divided areas inside a single commercial facility.

The server 20 stores 3D maps for multiple areas within different buildings and facilities. Therefore, the server 20 stores the 3D maps in a way that makes it clear which exhibition hall or commercial facility the 3D map pertains to. For example, it is possible to store 3D maps in separate storage areas for each building, or to register identification information that identifies the building, etc., as additional information for the 3D map. Furthermore, since the 3D maps will be different for different exhibitions held at the same exhibition hall, the 3D maps are stored separately for each exhibition in the 3D map storage unit 231.

The feature data storage unit 232 stores feature data abstracted into an intermediate representation from one or more image data images of the area for which the 3D map was created. Each 3D map has one or more pieces of this feature data. The abstracted feature data includes, for example, data vectorized using a multimodal embedding model from the images used to create the 3D map, explanatory information (context) present in the images of each divided area, and associated location information for identifying the approximate location. The context may be text data, but it is preferable to use a vectorized multidimensional vector. The vector representing the feature data may be created, for example, by detecting characters or objects from the image to generate explanatory text, and then vectorizing the text information using an embedding model. Furthermore, the vector representing the feature data is not limited to text information; for example, it may be generated directly by convolving the image. Generating vectors based on images allows for the generation of feature data based on information that cannot be generated by character detection, such as the appearance of walls, which is advantageous in terms of reducing information loss. Additionally, information combining explanatory information from multiple images taken from different angles within the same area may be used as context information for a single 3D map. The text used to create this explanatory information can be created, for example, by image recognition of image data and using OCR technology. It is also more preferable to use a multimodal large-scale language model to generate explanatory text using images as input. This creates text that describes the image information as if it were a human, and can also include non-textual visual information such as shape and color, generating feature data that can be more accurately distinguished from other locations. Furthermore, it is also preferable for a worker to view the created explanatory information (explanatory text), correct the text, re-take photos, and create new explanatory information.

Furthermore, the associated location information includes, for example, Wi-Fi access points and GPS information. For example, there are areas where GPS signals can be received even indoors, so the approximate location can be determined from GPS-based location information. Furthermore, Wi-Fi access points are installed in appropriate locations in underground shopping malls, etc. Wi-Fi access points have a small communication area. Therefore, information identifying the access point is used as feature data to associate the access point with a 3D map of the communication area.

The 3D map and feature data stored in database 23 can be created in any order, as long as they are based on the same area.

The control unit 22 is, for example, a computer including a processor and memory. The memory is, for example, a main storage device having RAM (Random Access Memory) and ROM (Read Only Memory). The processor temporarily stores programs read from the ROM in the RAM. The RAM also provides a working area for the processor. The processor performs various controls by performing arithmetic processing while temporarily storing data generated during program execution in RAM.

The control unit 22 has a map matching unit 221 and a map narrowing unit 222 as functions for performing position identification. The map matching unit 221 performs shape-based matching. That is, the map matching unit 221 matches feature points extracted based on image data captured by the terminal device 10 with the 3D maps stored in the 3D map storage unit 231, identifies one 3D map with a high degree of match, and derives the position and orientation at the time of capture by the camera 12 of the terminal device 10. Furthermore, the map matching unit 221 preferably has a function for calculating coordinates on the 3D map of the terminal device 10 or coordinates within the entire 3D map group based on the derived capture position and orientation. Furthermore, the shape-based matching process preferably utilizes an existing position identification algorithm based on comparing image data captured by a VPS or the like with a pre-prepared 3D map. The control unit 22 then sends position information regarding the identified position, orientation, etc. to the terminal device 10 via the network 2.

Prior to shape matching by the map matching unit 221, the map narrowing down unit 222 uses the acquired image data, etc., as a preliminary step to narrow down the area where the image was taken. In other words, the map narrowing down unit 222 abstracts the content represented by the image data and narrows down the map. The map narrowing down unit 222 extracts candidates for the 3D map based on the acquired image data, etc., and the feature data stored in the feature data storage unit 232. The detailed functions of this map narrowing down unit 222 will be described later.

Figure 3 is a flowchart explaining the processing of the control unit 22 when an image captured by the camera 12 of the terminal device 10 at the site is acquired by a user actually using the system. The user using the system launches a specific smartphone app installed on a smartphone, which is an example of the terminal device 10. The smartphone app accepts a specific input instruction, starts the camera 12, switches it to continuous shooting mode, and displays the image being captured on the display unit 15. The smartphone app also sends the image being captured to the server 20 at a specific timing. This specific timing may be a set time interval, or the acceptance of a transmission instruction given from the input unit 13 such as a tap on the screen. The control unit 11 also transmits the specific sensor information along with the image data.

The control unit 22 of the server 20 acquires the image captured by the terminal device 10 and sensor information (S1). The control unit 22 stores the acquired image data in a specified storage unit. The storage unit may be a temporary storage unit such as a volatile memory.

The map narrowing down unit 222 of the control unit 22 compares the acquired image with the context using a multimodal large-scale language model to determine the area, then processes it using a machine learning model to convert it into feature information, and narrows down the 3D map candidates where the image whose photographed position is to be detected is located (S2). The map narrowing down unit 222 sends the IDs of the narrowed down 3D map candidates to the map matching unit 221.

Next, the map matching unit 221 reads the image data stored in the temporary storage unit and extracts feature points using a machine learning model or SLAM method (S3). The map matching unit 221 then matches the obtained feature points with the candidate 3D maps narrowed down by the map narrowing down unit 222, identifies the 3D map that best matches, and derives the position and orientation within that 3D map (S4).

The control unit 22 calculates the coordinates on the terminal's 3D map and the coordinates within the entire group of 3D maps based on the derived shooting position and orientation (S5). The calculated coordinate data is then sent to the terminal device 10. The smartphone app then provides, for example, route guidance based on the position identified by the received coordinate data.

The map narrowing unit 222 that performs the above process S2 has (1) image recognition and vector search functions, and (2) a search function that uses multimodal LLMs (Large Language Models).

The image recognition and vector search function (1) converts various information into feature vectors using some method, and performs location identification based on the vector search; this is a similarity search. This vector search function vectorizes the acquired input image (sequence), calculates the degree of match with vectors associated with 3D maps stored in the feature data storage unit 232, and extracts feature data that meets specified criteria. The control unit 22 determines the 3D maps associated with the extracted feature data as map candidates. The specified criteria may be, for example, a match exceeding a threshold, or a predetermined number of highly matched maps. Vectorization processing of input images can involve vectorizing the image directly using a multimodal embedding model (generating a single abstract feature data set from multiple pieces of information), or abstracting and vectorizing the text information obtained by detecting characters in the image. The text information to be abstracted can be, for example, characters present in the image extracted through image recognition processing. Furthermore, the text information can be provided to the generation AI directly, or by image recognition, and the names of objects present in the image can be provided, and the description of the image generated by the generation AI can be used. This vector search can be processed quickly, so when there are many 3D maps to compare, it is possible to efficiently determine in a short amount of time whether a 3D map is eligible.

The search function using LLM (2) uses a large-scale language model to logically infer and identify a location using image and text information as input. This search function using LLM provides information about the candidate map (such as a description) stored in the feature data storage unit 232 as context to the multimodal LLM, and provides as input an image captured by the terminal device 10 and acquired via the network 2, thereby estimating a candidate 3D map. In other words, the context information for the 3D map and image information captured by the terminal device 10 are provided to the LLM, which then searches for 3D map candidates. The LLM model itself may be fine-tuned. Searches using LLM are characterized by higher accuracy compared to vector searches, but require longer processing times. Searches using LLM are limited in the number of areas they can cover.

Furthermore, when a user actually using the system takes a photograph on-site with the camera 12 of the terminal device 10, the captured image data will differ if the orientation of the camera 12 is different, even if the camera 12 is in the same position when the feature data, etc. was created in advance. Even in such cases, for example, in the case of a multimodal large-scale language model, image information from multiple angles is combined and edited to create feature data that effectively describes the range to which the 3D map applies, making it particularly tolerant to differences in angle. Therefore, even if the orientation of the camera 12 is different, if the range is within a certain range, the map can be narrowed down and extracted as a 3D map candidate.

On the other hand, with vector searches, even if the images are taken from the same position, if the orientation is different the degree of match will be low, and if the angle difference is large it will not be possible to detect them. Therefore, with vector searches, it is advisable to create feature vectors for images taken in multiple orientations within the range of a single 3D map, and associate each vector with the ID of the same map. In other words, feature data is created in advance, and multiple feature vectors are associated with a single 3D map when it is stored in database 23. In this way, the 3D map linked to the image taken during use can be appropriately identified by matching the feature vectors.

Either of the functions (1) and (2) can be used, or both can be combined. If one is used, it is recommended to use the more versatile (2). Also, since (2) has limitations in processing speed and the number of areas it can process, it is also possible to narrow down the results using only the image recognition and vector search of (1). If combining them, for example, if there are a large number of 3D map groups, it is recommended to narrow down the results using (1) and make the final decision using (2). Also, rather than dividing the order of use in this way, it is recommended to run the search functions of (1) and (2) in parallel, calculate the reliability for each, and find multiple 3D map candidates based on the reliability.

Furthermore, as mentioned above, search (1) can be processed quickly and is characterized by its ability to be applied to wide areas. On the other hand, search (2) can be sophisticated and allows for highly accurate detection, but it takes time, and if the target is wide, it may not be possible to narrow down the search in real time. Therefore, it is advisable to provide a function that allows for the use of either search depending on the location to be located. For example, if the area within building A is relatively small and there are few 3D maps and feature data to compare, search (2) is used to narrow down the search. On the other hand, if the area within building B is relatively large and there are many 3D maps and feature data to compare, search (1) is used to narrow down the search. To determine which search to perform, the control unit 22 can determine whether the user is in building A or building B based on the current location based on sensor information such as GPS signals and Wi-Fi access points, and then switch the search function to operate based on the current location.

The map narrowing unit 222 may also have a function to narrow down the target 3D map group in advance based on various sensor information such as GPS (GNSS) information, Wi-Fi information, and geomagnetic information. Furthermore, the control unit 22 has a function to, if it determines that the sent image data is outside the target area, send a message to that effect to the terminal device 10 and prompt the terminal device 10 to perform the necessary processing. The determination of whether or not the target area is outside may be made, for example, based on sensor information, by determining whether or not there are any facilities or the like to be located around the current position of the terminal device 10. The determination of whether or not the target area is outside may also be made if no 3D map candidates are detected as a result of a search by the map narrowing unit 222.

The map narrowing down unit 222 sends information about the extracted 3D map candidate or candidates, such as their IDs, to the map matching unit 221. The map matching unit 221 then performs matching processing only on the 3D map candidates. In this way, the matching targets are narrowed down, allowing the map matching unit 221 to identify accurate positions and attitudes (three-dimensional orientations).

In this way, the control unit 22 of this embodiment narrows down the area in terms of content (narrows down the 3D map) and then performs shape feature point matching. Furthermore, the map narrowing down unit 222, including vector search and LLM, can narrow down the area by abstracting comprehensive features including peripheral information rather than specific objects into vectors or intermediate representations in large-scale language models (also vectors), and then performing comprehensive comparative inference, etc. Then, the map matching unit 221 performs shape-based matching, thereby enabling the current location to be identified with high accuracy.

In other words, as mentioned above, 3D maps of areas with similar shapes have similar features, so they may not be able to be accurately identified by matching feature points on shape alone. On the other hand, even if the shapes are similar, when the images are abstracted to include colors, characters, etc., the features differ. For example, the vector directions are different, so the two can be distinguished.

The map narrowing unit 222 narrows down the search, eliminating 3D maps of different areas. On the other hand, abstract search processing can narrow down the search to similar 3D maps, etc., but cannot identify the exact location or orientation. Therefore, by eliminating similar 3D maps in advance, the map matching unit 221 can identify the location with high accuracy.

Therefore, the control unit 22 can distinguish between repetitive shapes and similar shapes. For example, the control unit 22 can distinguish between areas that cannot be distinguished by 3D map feature point matching alone. Furthermore, because the number of 3D maps to be processed for 3D map feature point matching can be reduced, the load on the point cloud matching process performed by the map matching unit 221 can be reduced and the speed can be increased, thereby expanding the coverage area.

In other words, even objects with similar shapes will have different characteristics when abstracted, making it possible to distinguish between similar 3D maps and extract the correct 3D map as a candidate. Therefore, it is possible to distinguish complex areas by making a comprehensive judgment using not only specific landmarks or objects, but also multimodal LLMs (large-scale language models) and image recognition processing and machine learning models that recognize overall features.

For example, in the case of an exhibition, similar booth packaging may be used, as shown in Figures 4 and 5. In these two figures, there is a similar counter 51 at the entrance, and the partition wall 52 from the surrounding area is also similar in shape. Therefore, discrimination based on shape alone may not be possible. On the other hand, the two booths differ in the size, number, and installation location of exhibits 53 such as posters pasted on the walls, and the text information 54 such as company names also differs. Therefore, the feature data that abstracts this information is different. Therefore, the two booths can be properly identified by narrowing down the search using the map narrowing unit 222.

Furthermore, in the case of a commercial facility, for example, even if two stores in a shopping mall have similar entrances, if they have different colors or store names, or if the products they carry are different, the map narrowing down unit 222 can make a distinction before the map matching unit 221 performs shape feature point matching, narrowing down the target 3D map candidates, allowing for appropriate and efficient shape feature point matching. For example, the stores shown in Figures 6 and 7 are located along a straight corridor, have wide, open entrances, and are similar in shape. However, because the store logos (text information) 55 and color schemes on the walls are different, the feature data that takes these factors into account is different. Therefore, the two stores can be appropriately identified by narrowing down the search results using the map narrowing down unit 222.

Furthermore, the stores shown in Figures 8 and 9 have similar configurations and shapes, such as curved shapes and ceiling lights 57, but carry different products 58 (for example, books and clothing). Even if the shelves are similar, if the products displayed are different, the shapes and colors of the products will be different, and the text will also be different. Therefore, the two stores can be properly identified by narrowing down the search using the map narrowing down unit 222.

According to this embodiment, by combining image recognition processing and vector search, which are suitable for capturing general characteristics, with 3D map feature matching, which is suitable for detecting positions with high accuracy, it becomes possible to identify positions that cannot be detected by GPS, such as inside complex buildings.

In this embodiment, indoor location can be determined using images and smartphone sensor information. The terminal device 10 may then acquire the location information determined by the server 20 and have the functionality to, for example, provide guidance to a specified item or destination, or present AR content.

The positioning system of this embodiment can accurately identify positions even indoors where similar shapes exist, something that existing VPSs struggle with. As a result, it is possible to identify the current location without special equipment in places such as large commercial facilities and event venues, and provide guidance and AR content tailored to the location. Furthermore, a system can be created that can update content and destination information in real time via a management app or API, making it possible to use the system in places with dynamic layouts such as retail stores and event venues.

[Modification]
The positioning system can be modified in various ways as described below. FIG. 2 is a diagram showing an example of an actual configuration. In this example, the server 20 is realized as a multiple client-server system. The map narrowing unit 222 is implemented in the first cloud system 31. The map matching unit 221 is implemented in the second cloud system 32. This second cloud system 32 performs processing to extract feature points from images and to estimate the current posture from the feature points. Furthermore, the entity data of the 3D map is stored and held in storage 33 associated with the second cloud system 32. Furthermore, the feature data of the 3D map and the 3D map ID are linked in a database 34.

The 3D map narrowing process and the 3D map matching process may be performed in part or in whole on the terminal device 10 containing the imaging equipment, or on computational resources such as a data center or external services (collectively referred to as the cloud).

Alternatively, a configuration may be adopted in which a terminal device containing the imaging equipment and a terminal device performing the calculation processing exist separately, and are connected via wired or wireless communication (collectively referred to as terminals). The map narrowing process may be performed in the cloud, and the terminal may obtain the results, after which the matching process with the 3D map may be performed again in the cloud, or both the map narrowing process and the matching process with the 3D map may be performed on the cloud, and the results obtained. Cloud processing may also be performed across multiple clouds.

The process of narrowing down the map can be performed on the cloud or on the device. The computational processes, machine learning models, feature data, and other information required for the map narrowing process can be downloaded in advance to the device. The range to be downloaded can be narrowed down using sensor information such as GPS and Wi-Fi. Narrowing down can also be done by user operation.

The process of extracting feature points from images or image sequences can be performed on the cloud or on the device.

The process of extracting feature points from an image or image sequence may be performed before, at the same time as, or after the map refinement process. Feature points may also be extracted using the same images as those used for map refinement.

The process of matching the 3D map with the image's feature points can be performed on the cloud or on the device. The 3D map can be downloaded in advance to perform the matching process on the device, or it can be obtained when the results of the map filtering process are returned.

The database that holds the feature data and 3D maps may not be a single database; the feature data for a specific 3D map may be associated with the 3D map and stored in separate databases or cloud systems.

It is also possible to use the final location information only for recording and utilization on the cloud without returning it to the device. It is also possible to use images that have already been taken to narrow down the map or match it with a 3D map.

In addition, in the above-described embodiments and variations, in this embodiment, 3D maps showing a predetermined indoor area are created for each of the areas into which the area is divided, but a single 3D map may be created for the entire predetermined indoor area. In this case, predetermined feature data is associated with each position within the single 3D map and stored in the feature data storage unit. The map narrowing unit narrows down the candidate points for shape matching based on the feature data generated based on the acquired image data, etc., and the feature data stored in the feature data storage unit. In other words, the 3D map portion corresponding to the narrowed-down point within the single 3D map is selected as a 3D map candidate. The map matching unit then performs shape matching on the 3D map portion within the single 3D map candidate.

By doing this, even in cases where shape matching would not be possible for the entire 3D map containing elements of a single, relatively broad, similar shape, shape matching can be performed within a narrowed-down range within a single 3D map, allowing for accurate location identification.

In this case, a single 3D map may be considered a collection of multiple 3D maps in the above-mentioned embodiments.

[Device equipped with a camera]
In the above-described embodiment, the terminal device 10 is mainly a smartphone. However, it may be a terminal equipped with a camera, such as smart glasses exemplified in the examples. Furthermore, the terminal is not limited to devices carried by people, such as smartphones and smart glasses, but may also be a mobile object that moves on its own. The mobile object may be a mobility such as an automobile or drone, or a robot. For example, application to a mobile object such as a drone or robot may be possible by recognizing the current floor and the location within a multi-story building, and using the information for autonomous driving. Furthermore, in the case of an automobile, it may be possible to recognize the current location in an underground parking lot, for example, and use the information for autonomous driving.

The terminal does not necessarily have to have a display; it may simply have a camera.

Furthermore, the location identification system is not limited to one that identifies the current location based on images captured in real time. For example, the location identification system may be applied to one that performs a series of processes using images that have already been captured, and estimates the location where the image being processed was captured.

Although various aspects of the present invention have been described above using embodiments and modifications, it should be noted that these embodiments and descriptions are not intended to limit the scope of the present invention, but are provided to aid in understanding the present invention. The scope of the present invention is not limited to the configurations and manufacturing methods explicitly described in the specification, but also includes combinations of the various aspects of the present invention disclosed herein. While the configurations of the present invention that are sought to be patented are specified in the appended claims, it is hereby emphasized that configurations not currently specified in the claims may be claimed in the future as disclosed herein.

1: Positioning system 2: Network 10: Terminal device 11: Control unit 12: Camera 13: Input unit 14: Communication unit 15: Display unit 16: GPS receiving unit 20: Server 21: Communication unit 22: Control unit 23: Database 31: First cloud system 32: Second cloud system 33: Storage 34: Database 221: Map matching unit 222: Map narrowing unit 231: 3D map storage unit 232: Feature data storage unit

Claims (9)

A location identification system for identifying a location where an acquired image was taken,
a map narrowing function that narrows down candidates of 3D maps to be searched using feature data abstracted based on the image;
a map matching function that performs shape matching of the image with the narrowed-down candidates of the 3D map and identifies the location where the image was taken;
A location system comprising: The location identification system according to claim 1 , wherein the map narrowing function performs the abstraction using a large-scale language model with context information of the image and the 3D map as input, thereby narrowing down the candidates for the 3D map. The location identification system described in claim 1, wherein the map refinement function creates the feature data by vectorizing information based on the image. The location identification system described in claim 3, wherein the map refinement function creates the feature data by vectorizing text information related to the image. The location identification system described in claim 1, wherein the abstraction performed by the map refinement function is performed using the image and context information related to the image. The location identification system described in claim 1, wherein the map narrowing function is performed using location information detected by the device that captured the image. The location identification system of claim 1 includes a database that stores feature data to be compared and 3D maps and that is accessible by the map narrowing function and the map matching function. equipped with a camera and a terminal with communication capabilities,
The position specifying system according to claim 1 , wherein the terminal has a function of transmitting, using the communication function, an image captured by the camera to a server having the map narrowing function. A computer-readable recording medium having recorded thereon a program for causing a computer to implement the functions of the location identification system described in any one of claims 1 to 8.