CN105975477B - A Method of Automatically Constructing Place Name Dataset Based on Network - Google Patents

Abstract

The invention discloses a method for automatically constructing a place name data set based on a network, which belongs to the field of computer application technology. The automatic construction of place name data set based on the network includes the following steps: 1, using the Google search engine API to extract geospatial data from the Google database; 2, filtering out irrelevant web pages from the extracted data; 3, importing the output of step 2 , to extract geographic information; 4. Select the geocoding tool to convert the extracted address information into geographic coordinates, and then mark them on the map. The invention makes full use of the advantages of the search engine of the data extraction module, and retrieves geographic information from web pages with appropriate search query keywords. In the web page filtering module, filtering algorithms are used to exclude useless interference data. Effectively and dynamically extract geographic information from unstructured data sources such as web pages, making the data both highly complete and real-time. This method has high practical value.

Description

A Method of Automatically Constructing Place Name Dataset Based on Network

technical field

The invention belongs to the field of computer application technology, in particular to a method for automatically constructing a place name data set based on a network.

Background technique

Place names play an important role in building various geographic applications, for example, location-based services on mobile devices. In response to these demands, a technical method that can automatically create a geographical name dataset is needed. Nowadays, the amount of valuable geographic information available on the Internet is huge and increasing day by day. How to obtain accurate and timely geographic information from the Internet to establish a geographical name dataset is a difficult problem at present.

Spatial data sources can be divided into two categories: structured data sources and unstructured data sources. Many researchers have retrieved data from structured data sources. They download geographic data files through links (such as DBpedia) or interactively (such as LinkedGeoData, Wikimapia and OpenStreetMap, LinkedGeoData provides a SPARQL interface, and Wikimapia and OpenStreetMap provide a RESTful API). These well-structured data sources provide static information but fail to present the latest changes. Also, the data in OpenStreetMap and Wikimapia, for example, is added by individuals and has not been verified by authoritative institutions. The data of Google Maps has been manually verified with high accuracy. As a result, data updates in Google Maps are slow, as validating new places usually takes a certain amount of time as a cost.

In contrast, the unstructured nature of web pages facilitates real-time changes in geographic information, and the latest geographic information is often provided on web pages.

Most of the place-name-based information retrieval based on web pages need to deal with ambiguous place names, because place names extracted from web pages often have ambiguity. For example, a place name also has non-geographical meaning, since places are often named after objects, people, physical features, or historical factors. On the other hand, many places will be named after more famous places, and two different places may have the same name. Some scholars use supervised learning techniques to solve the problem of geographical name ambiguity through the co-occurrence model from Wikipedia.

To sum up, at present, in the research of constructing place name datasets, there is a lack of methods that can extract effective and dynamically changing geographic information data from unstructured data sources on the premise of effectively solving the problem of place name ambiguity.

Contents of the invention

The object of the invention is to propose a method for automatically constructing a place-name dataset based on the network, characterized in that, automatically constructing a place-name dataset based on the network comprises the steps:

Step 1: Use Google Search Engine API to extract geospatial data from Google database;

Step 2: Filter out irrelevant web pages from the extracted data;

Step 3: Import the output of step 2 to extract geographic information;

Step 4: Select the geocoding tool to convert the extracted address information into geographic coordinates, and then mark them on the map.

The step 1 specifically includes the following steps:

Step A1: Extract street names from OSM (OpenSreetMap), that is, download OSM data into an XML file, which is composed of three primitive data types: node, road and correlation, and each primitive data type is designed with a series of labels, And each label basically consists of a (k, v) pair; OSM is a leading VGI (Volunteered Geographic Information) project, aiming to create a freely editable geographic platform for all volunteers worldwide , which currently has more than 1.6 million registered users, and nearly 30% of registered users have made actual contributions to the project;

Step A2: Determine the search keywords. The keywords queried by the search engine are composed of three parts, namely street name, city name and business type. The street name is obtained from the above step A1, and the business type is a popular business type provided manually first. , and then add the missing type by the result shown in the map behind;

Step A3: Select a search engine and extract geospatial data from the web search engine. The geospatial data depends on the working principle of the search engine. The working principle of the search engine is divided into collecting information, organizing information, accepting queries and visualization; Different methods are divided into full-text search, directory index and meta search. Said step 2 is specifically to select a specific filtering algorithm according to a specific target to filter the returned results: filter out many unwanted data from the results returned by the search engine, extract the desired data, and include a large number of The real estate listings in , which mainly contain residential addresses; here, use the search engine to search for the recently sold houses contained in any real estate company website, and the search results obtained include all real estate company URLs. Then, extract the URLs of these websites; next, automatically filter out the URLs of these real estate companies in the results of step 1, avoiding the waste of time and resources caused by parsing the useless geographic information from the real estate company websites in step 1 .

Described step 3 comprises the steps:

Step C1: In the process of address extraction, there are two cases. The first case is that the entire address information is in one line, and the second case is that the address information is in multiple lines;

Step C2: In the first case of step C1, determine whether a line in the web page starts with a number, contains a city name, and the length of the line should be less than a given critical value, if the length of the line exceeds the given critical value value, there is very little possibility of other address information in this line;

Step C3: In the second case of step C1, use the same method in step C2 to judge whether the two lines together represent an address: if the first line starts with a number, and the second line contains a city name, in the two lines On the premise that the length of the line is less than a given threshold, the two lines are extracted together as an address;

Step C4: Determine whether there is more than one extracted address. If there is only one address corresponding to the corresponding location in the webpage, the website title is the name of the location. The probability of this situation is quite high; if it contains multiple addresses, return the address list, that is That is, when more than one address is extracted from a web page, all the addresses in the page are extracted and returned to the address list;

Step C5: On the premise that the address list is returned in step C4, search for each address in the list. Among all the returned web pages, if the returned web page contains only one address and is the same as the index address, the corresponding URL title is recognized as a place name;

Step C6: Finally, from each address in the address list, the corresponding place name is obtained.

Described step 4 comprises the steps:

Step D1: Upload the data set to a geocoding tool so that the data appears on it;

Step D2: The geocoding tool automatically detects the location data and displays it in the form of labels;

Step D3: Click on the label, and the corresponding information will be displayed;

Step D4: According to step D2, the geocoding tool automatically detects the location data, and displays it in the form of labels to select the data that can be displayed or the data that cannot be displayed, or choose what form to display.

Beneficial Effects of the Invention The method for automatically constructing place name data sets based on the network proposed by the present invention fully utilizes the advantages of the search engine of the data extraction module, and retrieves geographic information from web pages with appropriate search query keywords. In the web page filtering module, filtering algorithms are used to exclude useless interference data. Effectively and dynamically extract geographic information from unstructured data sources such as web pages, making the data both highly complete and real-time. Therefore, it overcomes the shortcomings that most geographic information data sets come from structured data sources, the data is not complete, and the real-time performance is poor; this method has high practical value.

Description of drawings

Figure 1 is a schematic diagram of the working principle of the search engine.

Figure 2 is a schematic diagram of retrieving geographic information from web pages.

Figure 3 is a schematic diagram of the Google search engine's geographic information marking on the map.

Fig. 4 is a schematic diagram of converting the extracted address information into geographic coordinate marks on a map.

Detailed ways

The present invention proposes a method for automatically constructing a place name data set based on a network, which will be described below in conjunction with the accompanying drawings and embodiments.

Figure 1 is a schematic diagram of the working principle of the search engine. , there are four modules in total, including data acquisition (collecting information), Web page filtering (organizing information), information extraction (accepting queries) and visualization. Data acquisition is to obtain relevant web pages from web pages according to keywords; web page filtering module is to remove irrelevant pages from the acquired pages, such as real estate homepages; information extraction is to extract addresses, place names, etc. from acquired web pages Geographical information; visualization is to display the acquired geographic information on a map for easy comparison and search.

In this method, the problem of how to select query words is solved, and based on the given query words, useless feedback results are deleted and useful URLs are filtered out. After the URL is extracted, different parsing is performed in different situations, that is, when the complete geographical information exists in one or more lines, different parsing methods are performed. On this basis, regardless of whether one or more addresses are parsed out, useful place names and corresponding geographic information can eventually be extracted and visualized through geocoding. In the following example, 10 regions are selected as tests, and through the calculation of precision rate, recall rate and F value (harmonic mean of precision rate and recall rate), it is finally concluded that: the geographical information in the Google panorama map Under the premise of ground truth, compared with OSM and Google Maps, this method is significantly better than the former, on the whole equal to the latter, and better than the latter in some areas, indicating that this method has high practical value.

Example,

In this method, automatically constructing a place name data set based on the network includes the following steps:

Step 1: Use Google Search Engine API to extract geospatial data from Google database;

Step 2: Filter out irrelevant web pages from the extracted data;

Step 3: Import the output of step 2 to extract geographic information;

Step 4: Select the geocoding tool to convert the extracted address information into geographic coordinates, and then mark them on the map. The specific operation is as follows:

Step 1: This step includes four specific operation steps A1-A4. First, summarize the results of steps A1 and A2 into the following table:

Step A3, select a search engine, and extract geospatial data from a web search engine, because it largely depends on the working principle of the search engine, here select Google search engine;

Step A4 is to use two search keywords <City Name><Place Types> and <Street Names><CityName><Place Types> to search in the Google search engine, you can find that the latter search results are more complete, so after taking who (as shown in Figure 3).

Step 2: Filter out irrelevant web pages from the extracted data;

Here, use a search engine to search for recently sold homes contained in any real estate company website, for example: search for "2117 Tondolea LN" in Google, and the first page returned contains real estate company websites, such as Zillow, Redfin , Movoto, Trulia, Realtor, RE/MAX and more. Then, extract the URLs of these websites, such as www.zillow.com and www.redfin.com, etc. Next, automatically filter out the URLs of these real estate companies in the results of step 1.

Step 3: Import the output of step 2 to extract geographic information (as shown in Figure 2);

Said step 3 includes six concrete steps of C1-C6, which are described as follows:

Step C1: In the address extraction process, two situations will be encountered. The first situation is that the entire address information is in one line, and the second situation is that the address information is in multiple lines;

Step C2: In the first case of step C1, determine whether a line in the webpage starts with a number and contains a city name, and set the threshold value to 100;

Step C3: In the second case of step C1, use the same method in step C2 to judge whether the two lines together represent an address;

Step C4: Determine whether the extracted address is more than one. If there is only one address corresponding to the corresponding place in the web page, we assume that the website title is the name of the place. If it contains multiple addresses, return the address list;

Step C5: On the premise that the address list is returned in step C4, search for each address in the list. Among all the returned web pages, if the returned web page contains only one address and is the same as the index address, the corresponding URL title is recognized as a place name;

The pseudo codes of steps C4 and C5 are as follows:

Step C6: Finally, get the corresponding place name from each address in the address list.

Step 4: Select the geocoding tool, convert the extracted address information into geographic coordinates, and then mark it on the map (as shown in Figure 4); the specific operation steps are as follows:

Step D1: Upload the dataset to Google Fusion Tables (a geocoding tool), and the uploaded data will appear on it;

Step D2: Google Fusion Tables automatically detects location data and displays it in the form of a label called Map of<locationcolumn name>;

Step D3: Click on the label, and the corresponding information will be displayed.

Step D4: You can choose which data will be displayed, which will not be displayed, and in which form you can choose to display it. For example, two pins are defined in the work, and the blue pins with Y labels show the information obtained from the information extraction step. The correct place name, while the red pins with the N label present those that were filtered out, such as homes for rent or for sale.

In summary, the present invention uses the advantages of the data extraction module search engine to retrieve geographical information from web pages with appropriate search query keywords; uses the web page filtering module to propose a filtering algorithm to eliminate those useless interference data. Its Extract Location Information module uses existing algorithms to extract useful information such as place names and addresses. Visualize the extracted place names on the map through the visualization module, the purpose is to evaluate the effect of the place name data set that has been generated.

Claims (1)

1. a kind of method for constructing geographical name data collection automatically based on network, which is characterized in that construct ground concrete number automatically based on network Include the following steps: according to collection

Step 1: extracting geographical spatial data from Google's database using google search engine API；Determine that search is crucial first The keyword of word, search engine inquiry consists of three parts, i.e. street name, city name and business type, wherein street name Title is that street name is extracted from disclosed street map OSM, and business type is first to provide welcome business type manually, so Increase the type of missing by the result of map denotation below afterwards；Then search engine is selected, is taken out from network search engines Geographical spatial data is taken, which depends on the working principle of search engine, and the working method of search engine is divided into Information is collected, information is arranged, receives inquiry；According to the difference of way of search, and it is divided into full-text search, directory index and member and searches Rope；

Step 2: incoherent webpage is filtered out from the data extracted；

Step 3: the output of steps for importing 2, extracting geographic information；It specifically includes:

Step C1:, can be there are two types of situation in the extraction process of address, the first situation is entire address information all in a line, Second situation is address information in multirow；

Step C2: in the first of step C1, judge whether a line in webpage is started with number, comprising city name Claim, and the length of row is less than given critical value；

Step C3: under the second situation of step C1, being differentiated two rows to be linked to be with the same method of step C2 is afterwards together It is no to represent an address: if the first row, with number beginning, the second row contains city name, to be less than given face in the length of two rows Under the premise of dividing value, this two row is extracted together as address；

Step C4: the address extracted is determined whether more than one, if comprising multiple addresses, return address list is It says, when the address extracted in a webpage is more than one, addresses all in the webpage is extracted, return to address In list；

Step C5: under the premise of step C4 return address list, each of list address is scanned for, is returned all It is and identical as index address if the webpage returned only includes an address in the webpage returned, then corresponding web site title It is recognized as place name；

Step C6: finally, from each of address list address, corresponding place name is obtained:

Step 4: selecting geographical encoding tool, the address information of extraction is converted into geographical coordinate, is then marked on map；Packet Include following steps:

Step D1: it uploads on a kind of data set to geocoding tool, appears in data above；

Step D2: the automatic detection position data of geocoding tool, and show in tag form；

Step D3: clicking label, and corresponding information will show；

Step D4: according to the automatic detection position data of step D2 geocoding tool, and show in tag form to select energy exhibition The data shown or the data that cannot be shown, or selection are shown in what manner；Two kinds of stitch, band Y are defined at work The blue stitch of label illustrates the correct place name obtained from information extraction step, and the red stitch with N label What is presented is the information that those are filtered off.