JPWO2010131425A1 - Sequentially generated data analysis apparatus, system, method, and program - Google Patents

Abstract

The sequential generation data analyzing apparatus includes a prior analysis control unit that performs control to execute a process of performing preliminary analysis on sequential generation data that is sequentially generated before an analysis request. The sequential generation data analysis system includes a data generation device that generates sequential generation data that is sequentially generated data, a sequential generation data analysis device that performs an analysis process of the sequential generation data generated by the data generation unit, and the sequential generation data A service providing device that executes processing for providing a data analysis service based on the analysis result of the generated data analyzing device, and the sequential generated data analyzing device analyzes the sequentially generated data in advance prior to the analysis request. It includes a pre-analysis control means for performing control to execute the processing to be performed.

Description

  The present invention relates to a sequential generation data analysis apparatus, a sequential generation data analysis system, a sequential generation data analysis method, and a sequential generation data analysis program that collect and analyze sequentially generated data.

  As a related technique for analyzing data at high speed, for example, Patent Document 1 describes a cache control device that performs data analysis using a cache server. FIG. 31 is a block diagram showing the configuration of the cache control device described in Patent Document 1. As shown in FIG. 31, the cache control device described in Patent Document 1 includes an analysis server that analyzes data, and stores an analysis result between a service providing terminal that uses the analysis result and stores the analysis result. A cache server to be used next is provided. By providing such a configuration, when the service providing terminal issues the same query, the analysis server uses the previous analysis result stored in the cache server for the second and subsequent queries. However, it is possible to eliminate the need to perform analysis every time, and to expect a high speed data analysis.

JP 2008-210094 A

  However, the related technique described in Patent Document 1 has a problem that the first query is not accelerated. Therefore, the cache server can speed up data analysis only with an application that issues the same query many times. For example, when browsing a web page using a cache is taken as an example, it is not possible to speed up access to a web page that no one has seen. In other words, the data analysis can be speeded up only in a usage environment where many users issue the same query many times.

  On the other hand, for example, when sequentially generated data such as log data accumulated by a server or data detected by a sensor is to be analyzed, the latest data is often analyzed only once. Therefore, even if the related technique described in Patent Document 1 is used, the effect of increasing the speed of data analysis is low.

  Further, in the related technique described in Patent Document 1, it is assumed that there is no change in the data to be analyzed. Therefore, when new data is added, the analysis result may change even when data analysis is performed on the same query, and the cache server data cannot be used secondarily.

  On the other hand, for example, when data generated sequentially such as log data accumulated by a server or data detected by a sensor is targeted, the data to be analyzed is constantly changing. Therefore, even if the related technique described in Patent Document 1 is used, there is no opportunity for secondary use of the analysis result stored in the cache server, and the analysis server needs to execute the analysis process every time. Therefore, it is impossible to increase the speed of data analysis.

  As described above, when data analysis is performed in a situation where data generated sequentially, such as log data accumulated by a server or data detected by a sensor, is analyzed, the related technique described in Patent Document 1 Even if is used, the data to be analyzed cannot be analyzed at high speed.

  Accordingly, the present invention provides a sequential data analysis apparatus and a sequential data analysis that can perform analysis according to a request for response time for each service even when collecting and analyzing data that occurs sequentially. An object is to provide a system, a sequential generation data analysis method, and a sequential generation data analysis program.

  The sequential generation data analysis apparatus according to the present invention is characterized by including a prior analysis control unit that performs control to execute a process of performing prior analysis on sequential generation data that is sequentially generated before an analysis request.

  A sequential generation data analysis system according to the present invention includes a data generation device that generates sequential generation data that is data that is generated sequentially, a sequential generation data analysis device that performs analysis processing of sequential generation data generated by the data generation means, And a service providing device that executes a process for providing a data analysis service based on the analysis result of the generated data analysis device. The sequential generated data analysis device performs a process of pre-analyzing sequentially generated data before an analysis request. It includes a pre-analysis control means for performing control to execute the above.

  The sequential generation data analysis method according to the present invention includes a pre-analysis control step for performing control for executing a process of pre-analyzing sequential generation data that is sequentially generated before an analysis request.

  The sequential generation data analysis program according to the present invention is for causing a computer to execute a pre-analysis control process for performing a control for executing a process of pre-analyzing sequential generation data that is sequentially generated before an analysis request. It is.

  According to the present invention, even in the case of collecting and analyzing sequentially generated data, it is possible to perform analysis in response to a request for response time for each service.

It is a block diagram which shows an example of a structure of the sequential generation data analysis system by this invention. It is a block diagram which shows the example of the physical server structure of a sequential generation data analysis system. It is explanatory drawing which shows the example of an input screen of the flow line analysis service of Web access. It is explanatory drawing which shows the example of the service requirement which a service requirement setting means sets. It is explanatory drawing which shows the example of analysis data. It is explanatory drawing which shows the example of the table which an analysis data storage means memorize | stores. It is explanatory drawing which shows the example of the table which an analysis result memory | storage means memorize | stores. It is a block diagram which shows the more detailed structural example of a prior analysis control means. It is a flowchart which shows an example of the process in which a sequential generation data analysis system analyzes sequential generation data. It is a flowchart which shows an example of a prior analysis process. It is a flowchart which shows an example of the reception process of the analysis command (analysis request | requirement) which an analysis command reception means performs. It is a block diagram which shows an example of a structure of the sequential generation data analysis system in 2nd Embodiment. It is a block diagram which shows the more detailed structural example of the prior analysis control means in 2nd Embodiment. It is a block diagram which shows the example of the physical server structure of the sequential generation data analysis system in 2nd Embodiment. It is explanatory drawing which shows the example of the analysis command history information which an analysis command history means memorize | stores. It is a flowchart which shows an example of the process in which the sequential generation data analysis system in 2nd Embodiment analyzes sequential generation data. It is a flowchart which shows an example of the prior analysis process in 2nd Embodiment. It is a block diagram which shows an example of a structure of the sequential generation data analysis system in 3rd Embodiment. It is a block diagram which shows the more detailed structural example of the prior analysis control means in 3rd Embodiment. It is a block diagram which shows the more detailed structural example of the analysis cost monitoring means in 3rd Embodiment. It is a block diagram which shows the example of the physical server structure of the sequential generation data analysis system in 3rd Embodiment. It is explanatory drawing which shows the example of the table which an analysis cost calculation means memorize | stores. It is explanatory drawing which shows the example of the table which an analysis cost monitoring means memorize | stores. It is a flowchart which shows an example of the process in which the sequential generation data analysis system in 3rd Embodiment analyzes sequential generation data. It is a flowchart which shows an example of the prior analysis process in 3rd Embodiment. It is a block diagram which shows the more detailed structural example of the prior analysis control means in 4th Embodiment. It is explanatory drawing which shows the example of the table which a free capacity storage monitoring means sets beforehand. It is a flowchart which shows an example of the process in which the sequential generation data analysis system in 4th Embodiment analyzes sequential generation data. It is a flowchart which shows an example of the prior analysis process in 4th Embodiment. It is a block diagram which shows the minimum structural example of a sequential generation data analyzer. 2 is a block diagram illustrating a configuration of a cache control device described in Patent Literature 1. FIG.

Embodiment 1. FIG.
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of a sequential generation data analysis system according to the present invention. As shown in FIG. 1, the sequential generation data analysis system includes a service providing unit 1, a service requirement setting unit 2, an analysis data generation unit 3, an analysis data storage unit 4, an analysis unit 5, an analysis instruction receiving unit 6, and an analysis result storage. Means 7 and pre-analysis control means 8 are included. In the present embodiment, the sequential data analysis system is realized using a plurality of information processing apparatuses such as personal computers that operate according to a program.

  In the present embodiment, the sequential data analysis system can be applied, for example, to a use for performing a flow analysis of web access based on log data generated from web data. The sequential generation data analysis system can be applied to, for example, a traffic information providing system that collects traffic information (for example, position information of automobiles on a road) and detects and provides a traffic jam location on the road. In addition, for example, the sequential data analysis system can be applied to the use of algorithmic trading in which stock price fluctuations are monitored, stock price fluctuations are matched with pre-entered trading rules, and automatic stock trading is performed. That is, the sequential generation data analysis system can be applied to any system that generates a large amount of data sequentially and performs analysis while reflecting the latest data in real time.

  Specifically, the service providing unit 1 is realized by a CPU and a network interface unit of an information processing apparatus that operates according to a program. The service providing unit 1 has a function of collecting and analyzing sequentially generated data such as log data output from a server and data output from a sensor (for example, a temperature sensor or a humidity sensor). In addition, the service providing means 1 uses data analysis results to provide data to business users (for example, website owners) that operate devices (for example, web servers) that provide various services to general users. A function of providing an analysis service (for example, an analysis result distribution service) is provided.

  Specifically, the service requirement setting unit 2 is realized by a CPU of an information processing apparatus that operates according to a program. The service requirement setting unit 2 sets a service response time requirement (hereinafter also referred to as a service requirement) for each service provided by the service providing unit 1, and stores the setting contents in a storage device (for example, a magnetic field included in the information processing device). A function of storing in a disk device or an optical disk device). Note that the service requirement is a requirement indicating whether or not data to be analyzed in advance, which will be described later, and a requirement indicating a priority for performing a prior analysis. For example, the service requirement setting unit 2 sets a service requirement in accordance with a setting request from a business user who operates a Web server, and stores the setting content in a storage device.

  Specifically, the analysis data generating unit 3 is realized by a CPU of an information processing apparatus that operates according to a program. The analysis data generating unit 3 has a function of sequentially generating data to be analyzed such as log data output from the server and data output from the sensor. In the present embodiment, the case of analyzing data that occurs sequentially in time series is shown. However, if the data has some order, for example, the positional order such as the order in which the distance is short or the order in which the distance is long. The present invention can also be applied to a case where analysis is performed by sequentially inputting data having order. Further, the data is not limited to data generated continuously at short time intervals (for example, every few seconds), but may be data generated at long time intervals such as several hours or days as long as the data is generated sequentially. It can also be applied to the case where the analysis is performed by inputting data such that the generation interval is not constant.

  Specifically, the analysis data storage unit 4 is realized by a temporary storage device such as a memory provided in the information processing device. The analysis data storage unit 4 stores data generated by the analysis data generation unit 3. As described above, in the present embodiment, the sequential generation data analysis system does not analyze the data generated by the analysis data generation means 3 after accumulating it in a database device such as a magnetic disk device or an optical disk device. Perform high-speed analysis in real time only by temporarily storing it in a temporary storage device.

  Specifically, the analysis unit 5 is realized by a CPU of an information processing apparatus that operates according to a program. The analysis unit 5 has a function of analyzing the data stored in the analysis data storage unit 4.

  Specifically, the pre-analysis control unit 8 is realized by a CPU of an information processing apparatus that operates according to a program. The pre-analysis control unit 8 selects data to be pre-analyzed from the data stored in the analysis data storage unit 4 in accordance with the service requirements set by the service requirement setting unit 2, and the analysis request query from the service providing unit 1 Is provided with a function of executing a pre-analysis (inputting the analysis means 5 to execute an analysis process) before inputting. In the present embodiment, “preliminary analysis” means that data is analyzed in advance before an analysis request is made.

  Specifically, the analysis result storage unit 7 is realized by a temporary storage device such as a memory provided in the information processing device. The analysis result storage unit 7 stores the pre-analysis result obtained by the pre-analysis control unit 8.

  Specifically, the analysis command receiving means 6 is realized by a CPU and a network interface unit of an information processing apparatus that operates according to a program. The analysis command receiving unit 6 has a function of receiving (inputting or receiving) an analysis request query from the service providing unit 1. Also, the analysis command receiving means 6 uses the pre-analysis result stored in the analysis result storage means 7 in order to create response information (analysis result) for the query of the analysis request, or executes analysis processing on the analysis means 5 It has a function to determine whether to make it happen. The analysis command accepting unit 6 outputs either the pre-analysis result stored in the analysis result storage unit 7 or the analysis result executed by the analysis unit 5 to the service providing unit 1 as response information to the query of the analysis request. (Or send) function.

  Next, the physical server configuration of the sequential generation data analysis system will be described. FIG. 2 is a block diagram illustrating an example of a physical server configuration of the sequential data analysis system. As shown in FIG. 2, the sequential data analysis system includes, for example, an application server 101, an analysis server 102, a data analysis service server 103, a data analysis reception server 104, and a data high-speed analysis server 105.

  The application server 101 is, for example, a server (for example, a web server) operated by a business user who provides various web services to an unspecified number of general users. The analysis data generation unit 3 and the analysis data storage unit 4 Including. The application server 101 operates using the analysis data generation unit 3 and the analysis data storage unit 4 to generate log data to be analyzed and temporarily store the log data in the analysis data storage unit 4.

  The analysis server 102 is, for example, a server that is operated by a provider that provides an analysis service for sequentially generated data, and includes the analysis unit 5. The analysis server 102 operates using the analysis unit 5 to analyze data (for example, log data) generated by the application server 11.

  The data analysis service server 103 is, for example, a server operated by a provider that provides an analysis service for sequentially generated data, and includes a service providing unit 1 and a service requirement setting unit 2. The data analysis service server 103 operates using the service providing unit 1 and the service requirement setting unit 2 to perform a process of providing a data analysis result to a business user who operates the application server 101.

  The data analysis reception server 104 is, for example, a server that is operated by a business provider that provides an analysis service for sequentially generated data, and includes analysis command reception means 6. The data analysis reception server 104 operates using the analysis command reception means 6 to perform processing for receiving (receiving) an analysis request query from the data analysis service server 103.

  The high-speed data analysis server 105 is, for example, a server operated by a provider that provides a service for analyzing sequentially generated data, and includes an analysis result storage unit 7 and a prior analysis control unit 8. The high-speed data analysis server 105 receives (inputs) service requirements from the data analysis service server 103 by operating using the analysis result storage unit 7 and the prior analysis control unit 8. Further, the high-speed data analysis server 105 controls the analysis server 102 in order to analyze in advance the data generated by the application server 101 before the analysis request query is issued according to the input service requirements.

  Note that the servers 101 to 105 shown in FIG. 2 do not necessarily have to be realized by using different information processing apparatuses. Depending on the capabilities of the servers, the servers 101 to 105 shown in FIG. Any one of the above may be realized by using one information processing apparatus. For example, in many cases, the analysis server 102, the data analysis reception server 104, and the data high-speed analysis server 105 can be realized by the same single information processing apparatus.

  In addition, it is assumed that the servers 101 to 105 and the units 1 to 8 illustrated in FIG. 2 share settings in advance such as a network and a port number that require communication, and can communicate with each other.

  In the example illustrated in FIG. 2, the sequential data analysis device is realized by, for example, the data high-speed analysis server 105. Further, for example, the sequential data analysis device is realized by an information processing device that realizes the analysis server 102, the data analysis reception server 104, and the data high-speed analysis server 105.

  Hereinafter, a method for realizing each means shown in FIGS. 1 and 2 will be described.

  The service providing means 1 is a means for providing data analysis results to a business user who operates the application server 101. For example, the service providing unit 1 distributes an analysis result such as a flow analysis of web access to a business user's terminal via a network. Hereinafter, a case of performing a flow analysis of Web access will be described as an example.

  FIG. 3 is an explanatory diagram showing an example of an input screen of a web access flow line analysis service. Specifically, for example, the service providing unit 1 transmits screen information (for example, Web page information) of the input screen illustrated in FIG. 3 in response to a request from the terminal of the business user. Next, the operator user's terminal displays the input screen shown in FIG. 3 based on the screen information received from the service providing means 1. Then, the operator user performs a request operation of the web access flow line analysis service by performing an operation while viewing the input screen displayed on the terminal. The screen example shown in FIG. 3 indicates “from which page the access is frequent” for each Web page. However, the screen example shown in FIG. 3 is an example, and the screen example used for providing the analysis service is not limited to that shown in the present embodiment.

  In FIG. 3, a URL input field (U01) is a URL input part of a Web page to be analyzed. Specifically, when a business user who uses the analysis service inputs a URL to the URL input unit U01, the business user's terminal inputs the URL of the Web page to be analyzed in accordance with the business user's operation.

  The analysis button (U02) is a button (input unit) for performing a request operation for the flow line analysis of the Web page specified by the input URL. Specifically, when a business user who uses the analysis service clicks the analysis button U02 with a mouse or the like, the business user's terminal provides a service for requesting the analysis of the Web page indicated by the input URL via the network. Send to means 1.

  The ranking (U03) is a display area for displaying the ranking of Web pages in the order of frequent access. Further, the URL (U04) of the access source is a display area for displaying which Web page has moved from. The access count (U05) is a display area for displaying the access count.

  The service providing means 1 receives (receives) input information of the URL of the Web page to be analyzed from the terminal of the business user. When the analysis button U02 is pressed, the service providing unit 1 receives the analysis request and issues (outputs) an analysis request query to the analysis command receiving unit 6. The following is an example of the query format of the analysis request. However, an example of the format of this analysis request query is not limited to that shown in the present embodiment.

Analysis request query format: (GetReferrerRanking, URL)
Example: (GetReferrerRanking, “http://bar.co.jp/~foo/Page-001”)

“GetReferrerRanking” is a command name for specifying flow line analysis, and is a fixed value. The command name is not limited to the example shown above.
The URL is a URL representing data to be analyzed (data of a Web page), and is an input value from the URL input field (U01).

  In response to the analysis request query described above, the analysis command accepting means 6 returns (outputs or transmits) the analysis result to the service providing means 1 in the following format.

Analysis result format: ((ranking, access source URL, number of accesses), (ranking, access source URL, number of accesses))
Example: ((1, http://bar.co.jp/~foo/Page-999, 1230), (2, http://aaa.co.jp/~bbb/Page-123, 881). .)

  However, the format of the response result shown above is an example, and is not limited to that shown in the present embodiment.

  When the service providing means 1 receives (inputs or receives) an answer (analysis result) to the query of the analysis request, as shown in FIG. 3, the ranking (U03), the URL (U04) of the access source, the number of accesses ( The information of the display screen displaying the analysis result is generated in the field of (U05). Further, the service providing means 1 transmits information on the generated display screen to the operator user's terminal via the network, and the analysis result is displayed on the operator user's terminal.

  The service requirement setting means 2 is a means for setting and storing a service response time requirement (service requirement) for each service provided by the business user (for example, for each Web page).

  FIG. 4 is an explanatory diagram showing an example when response time requirements (service requirements) are set for each Web page. In FIG. 4, URL (T02-01) is the URL of the Web page for which service requirements are set. The response time requirement (T02-02) is a set value of the response time requirement (service requirement). In the example shown in FIG. 4, an early response is required, and High is set as a service requirement for data that is to be preliminarily analyzed. On the other hand, Low may be set as a service requirement for data that may be slow in response and does not need to be preliminarily analyzed.

  In order to receive the analysis service, the business operator inputs the service requirement setting in advance. Further, the service requirement setting means 2 sets a service requirement for each data based on the setting instruction information received from the operator user's terminal, and stores it in the storage device in advance. Then, in response to a request from the prior analysis control unit 8, the service requirement setting unit 2 uses a URL input (or received) from the prior analysis control unit 8 as a key, and a response time requirement (service requirement) corresponding to this URL. Are retrieved and extracted and output (or transmitted) to the prior analysis control means 8.

  Further, in the present embodiment, the case where the service requirement is set according to the setting operation of the operator user is shown, but the service requirement setting unit 2 may automatically set the service requirement according to a predetermined rule. For example, the service requirement setting means 2 may detect a site introducing a promotion of a new service and automatically set it as a target of prior analysis when performing a web access flow analysis service. Good.

  In addition, the service requirement setting unit 2 may automatically detect data whose contents change, and automatically set the data as a prior analysis target. In this case, for example, paying attention to the fact that the contents of the top page of the website generally fluctuate frequently, so that the service requirement setting means 2 is subject to the pre-analysis for the information on the top cage of the website. Service requirements may be automatically set. The service requirement setting unit 2 can determine whether or not the page is the top page of the Web site, for example, by determining whether or not a file name such as “index.html” is included in the URL. .

  Further, for example, when applied to a traffic information providing system, the service requirement setting means 2 automatically sets traffic information related to a national road as a prior analysis target or automatically sets traffic information related to a station front as a prior analysis target. It may be set.

  The analysis data generation means 3 is means for generating data to be analyzed. In the present embodiment, a case where the analysis data generation unit 3 is realized by a Web server and generates access log data will be described as an example. As shown in FIG. 5, the analysis data generation means 3 is data having fields of date and time (T03-01), URL (T03-02), and Referrer (T03-03) as Web server access log data. Shall be generated. The date (T03-01) is the date when the Web page was accessed. URL (T03-02) is the URL of the Web page to be accessed. Also, Referrer (T03-03) is the URL of the Web page that is the access source.

  In the example shown in FIG. 5, for example, at the date “2008/12/16 10:00:00”, the web page “http://bar.co.jp/~foo/Page-001” It can be seen that there was an access that moved from “: //bar.co.jp/~foo/Page-002”.

  In the example shown in FIG. 5, the case where the access log data of the Web page is described in the CSV format and the text data is shown. However, the access log data format is not limited to that shown in this embodiment. I can't. Further, the analysis data generating means 3 uses data having fields other than the date and time (T03-01), URL (T03-02), and Referrer (T03-03) shown in FIG. 5 as access log data of the Web page. It may occur. In this embodiment, in order to simplify the description, access log data having only the fields shown in FIG. 5 is used for the description.

  The analysis data generation means 3 transfers (outputs) the generated data to the analysis data storage means 4 for temporary storage. For example, the analysis data generation unit 3 may transfer (output) data to the analysis data storage unit 4 in the following format.

Data format: (Date, URL, Referrer)
Example: (2008/12/16 10:00:00, http://bar.co.jp/~foo/Page-001, http://bar.co.jp/~foo/Page-002)

  However, the format of the data shown above is an example, and is not limited to that shown in the present embodiment.

  The analysis data storage unit 4 is a unit for temporarily storing data generated by the analysis data generation unit 3. The data stored in the analysis data storage unit 4 is searched and extracted by the analysis unit 5 and the prior analysis control unit 8. The analysis data storage unit 4 may be realized using a relational database.

  The analysis data storage unit 4 stores at least data having at least the same field as the data generated by the analysis data generation unit 3. For example, as shown in FIG. 6, the analysis data storage unit 4 stores data including date and time (T04-01), URL (T04-02), and Referrer (T04-03). The date (T04-01) is the date when the Web page was accessed. URL (T04-02) is the URL of the Web page to be accessed. Also, Referrer (T04-03) is the URL of the Web page that is the access source.

  When actually operating the system, it is necessary to delete old data from the analysis data storage means 4 as needed. However, since it is not directly related to the present invention, the following explanation will be given. In order to make it easy to understand, it is basically assumed that data is continuously added to the analysis data storage unit 4.

  The analysis unit 5 is a unit for analyzing data. In the present embodiment, the analysis unit 5 performs a flow line analysis of the data of the Web access log. When the analysis unit 5 receives (inputs) an analysis request query from the analysis command receiving unit 6 or the pre-analysis control unit 8, the analysis unit 5 stores the analysis target data (in this example, Web access log data). Read from. Then, the analysis means 5 executes analysis processing based on the read data, and replies (outputs) the analysis result to the means (analysis command reception means 6 or pre-analysis control means 8) that has received the request.

  The query of the analysis request output from the analysis command receiving unit 6 or the prior analysis control unit 8 may be a query having the same format as the query output from the service providing unit 1 to the analysis command receiving unit 6. Moreover, the analysis means 5 should just output the data of the same format as the reply which the analysis command reception means 6 outputs to the service provision means 1 as a reply of an analysis result.

  The analysis means 5 performs an analysis process according to the following flow line analysis method. However, the analysis processing algorithm shown in the present embodiment is merely an example, and the analysis unit 5 may execute the analysis processing according to another analysis method.

Step 1:
First, the analysis unit 5 searches the URL data specified by the analysis request query based on each data URL field (T04-02) stored in the analysis data storage unit 4, and extracts data that matches the URL. To do.
Step 2:
Next, the analysis means 5 counts the number of appearances for each value of Referrer for the extracted data set.
Step 3:
Then, the analysis unit 5 rearranges the data in the order of appearance, and ranks the data.

  The analysis result storage means 7 is a means for storing the result of prior analysis. The analysis result storage means 7 receives (writes) the analysis result of the analysis process executed in advance from the pre-analysis controller 08 and stores the input pre-analysis result as needed. Moreover, the analysis command reception means 6 searches and extracts the prior analysis results stored in the analysis result storage means 7 as necessary. The analysis result storage means 7 may store two types of tables, that is, an analysis result management table and an analysis result detail table, as shown in FIG.

  The analysis result management table is a table including a correspondence relationship between the URL of the analyzed data and the pre-analysis result of the data. As shown in FIG. 7, the analysis result storage means 7 stores one analysis result management table. The analysis result storage means 7 stores an analysis result management table including URL (T07-01) and analysis result (T07-02) fields. Note that URL (T07-01) is a field including the URL of data to be analyzed. The analysis result (T07-02) is a field including information indicating a link to the pre-analysis result.

  The analysis result detail table is a table showing detailed analysis contents of the analysis results linked in the analysis result management table. Information of each analysis content included in the analysis result detail table is linked from the analysis result of the analysis result management table. Further, as shown in FIG. 7, the analysis result storage means 7 has an analysis result detailed table including fields of ranking (T07-03), URL of access source (T07-04), and number of accesses (T07-05). Remember. The ranking (T07-03) is a field including information indicating the ranking in the order of the most accesses. The access source URL (T07-04) is a field including information (specifically, the URL of the access source web page) indicating from which web page it has moved. Access (T07-05) is a field including information indicating the count value of the number of accesses.

  The pre-analysis control unit 8 is a main control unit that controls pre-analysis processing. The pre-analysis control means 8 determines which URL data should be pre-analyzed according to the requirements shown below. For example, the pre-analysis control unit 8 determines whether or not the data of the URL should be pre-analyzed according to the service requirement set for each URL by the service requirement setting unit 2. For example, the prior analysis control means 8 detects a change in the analysis data and determines whether the data should be analyzed in advance. Then, the pre-analysis control unit 8 causes the analysis unit 5 to pre-analyze data determined to be a pre-analysis target.

  FIG. 8 is a block diagram showing a more detailed configuration example of the prior analysis control means 8. As shown in FIG. 8, the pre-analysis control unit 8 includes an analysis target monitoring unit 801, a pre-analysis target generation unit 802, a pre-analysis request unit 803, and a pre-analysis result registration unit 804.

  The analysis target monitoring unit 801 has a function of monitoring the addition status of data stored in the analysis data storage unit 4. Further, the pre-analysis target generation unit 802 is a list information of data to be pre-analyzed preferentially based on the monitoring result of the analysis target monitoring unit 801 and the service requirement information stored in the service requirement setting unit 2. The function to create. The pre-analysis request unit 803 has a function of requesting the analysis unit 5 to perform pre-analysis of data indicated in the list information of the pre-analysis target generated by the pre-analysis target generation unit 802. The pre-analysis result registration unit 804 has a function of causing the analysis result storage unit 7 to store the analysis result obtained by performing the pre-analysis processing by the analysis unit 5.

  The analysis target monitoring unit 801 repeatedly executes processing for monitoring information stored in the analysis data storage unit 4 periodically (for example, in units of 5 minutes). The analysis target monitoring unit 801 detects whether there is data added after the previous monitoring process is performed among the data stored in the analysis data storage unit 4. When detecting the additional data, the analysis target monitoring unit 801 stores the value of the date and time with the newest date and time, for example, every time the monitoring process is executed. Then, the analysis target monitoring unit 801 may determine that data having a date and time newer than the stored date and time is newly added data when executing the next monitoring process. However, this detection method is an example, and a detection field is added to the information stored in the analysis data storage unit 4, and the analysis target monitoring unit 801 writes the processed information in the already processed field. By performing processing, it may be possible to detect addition of data.

  The pre-analysis target generation unit 802 extracts a set of URLs included in the additional data detected by the analysis target monitoring unit 801. Specifically, the pre-analysis target generation unit 802 may remove duplicate URLs from the extracted URLs of the additional data, and sequentially register the URLs of all the added data in the hash table. The list of keys in the hash table is a set of URLs of data that are candidates for pre-analysis. Hereinafter, a set of URLs included in the hash table is referred to as an additional URL.

  Also, the pre-analysis target generation unit 802 obtains (inputs) the service requirements corresponding to each additional URL from the service requirement setting unit 2. Then, the pre-analysis target generation unit 802 identifies, from among the additional URLs, the corresponding service requirement value “High” as a target for performing pre-analysis. On the other hand, if there is no service requirement value of High, the prior analysis target generation unit 802 identifies the service requirement value of Low as the target of the prior analysis. By doing so, URL data for which high-speed processing is a requirement can be preferentially analyzed. In the present embodiment, in order to simplify the description, the case where the service requirement is set in two stages of High and Low has been described. However, the service requirement is divided into finer stages (for example, set to three or more stages). In other words, the determination process for determining whether or not to be subject to prior analysis may be performed in a plurality of stages.

  The pre-analysis request unit 803 outputs an analysis request to the analysis unit 5 with respect to the data to be subjected to the pre-analysis specified by the pre-analysis target generation unit 802. For example, the prior analysis request unit 803 may output an analysis request in the following format, for example.

Format: (Analyze <{set of URLs})
Example: (Analyze, {"http://bar.co.jp/~foo/Page-999","http://hml.co.jp/~xyz/Page-777"})

  In the example shown above, Analyze is a command name. Note that the format example of the analysis request output by the prior analysis request unit 803 is not limited to that shown in the present embodiment.

  When the analysis unit 5 receives (inputs) an analysis request including the above command, the analysis unit 5 sequentially analyzes the URL data included as a parameter in the analysis request, and returns (outputs) the analysis result to the prior analysis request unit 803. In this case, the analysis means 5 outputs the analysis result of the answer format shown below, for example.

Analysis result format: {Analysis result set}
Analysis result: (URL to be analyzed, {collection of flow line information})

  Here, the flow line information is (ranking, URL of access source, number of accesses). An example of the analysis result output in the above format is shown below.

Example: {
(Http://bar.co.jp/~foo/Page-001, {(1, http: //bar.co.jp/~foo/Page-999, 1230),
(2, http://aaa.co.jp/~bbb/Page-123,881), (3, ...)}),
(Http://bar.co.jp/~foo/Page-002, {(1, http: //bar.co.jp/~foo/Page-111, 1357),
(2, http://aaa.co.jp/~bbb/Page-654,881), (3, ...)}),. . .
}

  The pre-analysis result registration unit 804 registers the pre-analysis result in the analysis result storage unit 7 by the following method. First, the pre-analysis result registration unit 804 searches and determines whether or not a new pre-analysis result is already registered in the analysis result storage unit 7. Specifically, the pre-analysis result registering unit 804 stores the URL field (T06-01) included in the analysis result management table stored in the analysis result storage unit 7 using the URL of the pre-analysis result data to be searched as a key. Search for. The pre-analysis result registration unit 804 determines that the URL has already been registered if it finds a matching URL, and determines that it has not been registered if the URL is not found.

  If the pre-analysis result registering unit 804 determines that the record has been registered, the pre-analysis result registration unit 804 overwrites the registered record with a new pre-analysis result. That is, the pre-analysis result registration unit 804 overwrites the record of the analysis result detail table linked to the analysis result field with the new flow line information.

  If it is determined that there is no registration, the pre-analysis result registration unit 804 adds a new record to the analysis result management table and registers the pre-analysis result in the added record. That is, the pre-analysis result registration unit 804 adds a new record to the analysis result management table, and registers the URL of the data to be analyzed in the URL field of the new record in the analysis result management table. The prior analysis result registration unit 804 adds a new analysis result detail table to the analysis result field of the new record, and registers link information to the added analysis result detail table. Furthermore, the pre-analysis result registration unit 804 registers the flow line information that is the new pre-analysis result in the newly added analysis result detail table.

  The analysis command receiving means 6 is a means for branching the processing of the analysis request query in order to reply the analysis result at high speed to the analysis request query received (input) from the service providing means 1. When the analysis command receiving unit 6 determines that the pre-analysis result can be used for the analysis request query from the service providing unit 1, the analysis instruction receiving unit 6 loads the pre-analysis result from the pre-analysis result storage unit 7. The pre-analysis result is output (or transmitted) to the service providing means 1. By doing so, it is possible to answer an analysis request query at high speed. If the analysis command accepting unit 6 determines that the pre-analysis result cannot be used, the analysis command receiving unit 6 forwards the analysis request query to the analyzing unit 5 and performs an analysis process taking a normal time to analyze the analysis request. The result is answered (output or transmitted) to the service providing means 1. The branching method of the analysis process will be described later.

  In the present embodiment, the storage device of the information processing apparatus that implements the sequential generation data analysis system stores various programs for analyzing sequential generation data at high speed. For example, a storage device of an information processing apparatus that realizes a sequential generation data analysis system performs a control in advance that causes a computer to execute a process of analyzing sequential generation data that is sequentially generated before an analysis request. A sequential generation data analysis program for executing analysis control processing is stored.

  Next, the operation will be described. FIG. 9 is a flowchart illustrating an example of processing in which the sequential generation data analysis system analyzes sequential generation data. As shown in FIG. 9, in the present embodiment, an apparatus that provides various services to a user by collecting and analyzing sequentially generated data such as log data output by a server and data output by a sensor. The following processing is executed for (system).

  As shown in FIG. 9, the operation step of the sequential generation data analysis system includes an analysis target data generation step (step S1) in which the analysis target data is sequentially generated. Also included is an analysis step (step S2) for analyzing the analysis target data accumulated in the analysis target data generation step S1. Also included is an analysis service utilization step (step S3) for starting the analysis and providing the analysis result to the business user in accordance with the request for the data analysis request input according to the business user's operation. In addition, a pre-analysis step (step S4) for executing an analysis in advance before receiving a specific analysis request in the analysis service utilization step S3 is included. If the data is already analyzed in the pre-analysis step S4 when the analysis request is received in the analysis service utilization step S3, the pre-analysis result is output as an answer, and the pre-analysis is still performed. If there is no data, it includes an analysis service control step S5 that requests the analysis to execute the process of the analysis step S2 and outputs the analysis result as an answer.

  Actually, the analysis data generating means 3 generates and outputs data sequentially at any time. Therefore, in practice, the sequential generation data analysis system executes the processes of steps S1 to S5 shown in FIG. 9 in parallel, and proceeds with the processes while communicating appropriately. Hereinafter, first, processing executed in each step will be described.

  In the analysis target data generation step (step S1), the analysis data generation means 3 performs processing of an application use step (step S101) for generating data to be analyzed when using an application, a server, etc., and an application use step S101. The analysis data storage step (step S102) for accumulating the analysis target data generated in the process in the analysis data storage means 4 is executed.

  In the analysis service utilization step (step S3), the service requirement setting means 2 performs analysis requirements such as response time of analysis (service requirements) for each analysis service before receiving the analysis service according to the setting operation of the operator user. The processing of the analysis service requirement setting step (step S301) is set. Further, the service providing means 1 accepts (inputs) an analysis service request in accordance with the request operation of the business user and performs an analysis service request step (step S302) for requesting the analysis command accepting means 6 and analysis. The analysis result is received (input) from the command receiving means 6 and the analysis service providing step (step S303) for providing the analysis result to the business user is executed.

  In the analysis service control step (step S5), the analysis command receiving means 6 executes the processing of the analysis command receiving step (step S501) that receives (inputs) an analysis request from the service providing means 1. Further, the analysis command receiving means 6 performs a pre-analysis availability check step (step S502) for investigating whether or not the data requested to be analyzed by this command (analysis request) has already been pre-analyzed and analyzed. Execute the process. In addition, when it is determined that the pre-analysis result can be used, the analysis instruction receiving unit 6 obtains (extracts) the pre-analysis result from the analysis result storage unit 7 and performs the process of the pre-analysis result using step (step S503). Execute. In addition, when it is determined that the pre-analysis result is unusable, the analysis command receiving unit 6 executes a process of an analysis request step (step S0504) for requesting the analysis unit 5 to execute an analysis process. Further, the analysis command receiving means 6 is an analysis command answering step for answering (outputting) the analysis result obtained in the prior analysis result using step (step S503) or the analysis result requesting step (step S504) to the service providing means 1. The process of (Step S505) is executed.

  FIG. 10 is a flowchart showing an example of the pre-analysis process in the pre-analysis step (step S4). As shown in FIG. 10, in the pre-analysis step (step S4), the pre-analysis control means 8 sets the analysis requirements (service requirements) set according to the operator user input operation in the analysis service utilization step (step S3). The analysis service requirement reading step (step S401) to be read is executed. Further, the prior analysis control means 8 performs an analysis target change detection step (step S402) for detecting data added after the previous monitoring process is performed on the analysis data accumulated in the analysis target data generation step (step S1). Execute the process.

  Further, the prior analysis control means 8 determines whether or not the service requirement read in the analysis service requirement reading step (step S401) is satisfied from the additional data detected in the analysis target change detection step (step S402). Thus, the pre-analysis target determining step (step S403) for selecting data to be pre-analyzed is executed. Further, the pre-analysis control unit 8 executes the process of the pre-analysis request step (step S404) for requesting the analysis unit 5 to execute the analysis process on the pre-analysis data selected in the pre-analysis determination step (step S403). To do. Further, the pre-analysis control unit 8 performs processing in a pre-analysis result registration step (step S405) in which the response obtained from the analysis unit 5 in the pre-analysis request step (step S404) is stored in the analysis result storage unit 7 as a pre-analysis result. Execute.

  The pre-analysis control unit 8 extracts a pre-analysis result stored in the analysis result storage unit 7 and answers (outputs) the pre-analysis data extraction request from the analysis command receiving unit 6 The result answer step (step S406) is executed.

  Since the data is sequentially generated at any time and the analysis request is also generated at any time, the pre-analysis control means 8 performs the pre-analysis result registration step (step S401) to the pre-analysis result registration step (step S401). The processing up to step S405) is repeatedly executed at regular intervals independently of the processing in the preliminary analysis result answering step (step S406).

  The pre-analysis control unit 8 starts and executes the process of the pre-analysis result answering step (step S406) when requested by the analysis command receiving unit 6, and returns to the request waiting state when the process ends. To control.

  Further, the pre-analysis control means 8 performs the process from the analysis service requirement reading step (step S401) to the pre-analysis result registration step (step S405) and the process of the pre-analysis result answering step (step S406). By sharing the data of the pre-analysis result registered in the process of the registration step (step S405), it executes so as to communicate asynchronously.

  Hereinafter, a method for realizing the processing of each step described above will be described in correspondence with each means.

  In the present embodiment, for example, the processing of the analysis target data generation step (step S1) is executed by the application server 101 operating. Specifically, the process of the application using step (step S101) is executed by the analysis data generating unit 3. Further, the analysis data storing step (step S102) is executed by causing the analysis data generating means 3 to sequentially store the generated data in the analysis data storing means 4.

  In this embodiment, for example, the processing of the analysis service utilization step (step S3) is executed by the operation of the data analysis service server 103. Specifically, the processing of the analysis service requirement setting step (step S301) is executed by the service requirement setting means 2. The processing of the analysis service requesting step (step S302) and the analysis service providing step (step S303) is executed by the service providing unit 1.

  In the present embodiment, for example, the analysis step (step S2) is executed by the analysis unit 5 of the analysis server 102.

  In the present embodiment, for example, the processing of the analysis service control step (step S5) is executed by the analysis command receiving means 6 of the data analysis receiving server 104.

  FIG. 11 is a flowchart showing an example of an analysis command (analysis request) reception process executed by the analysis command reception unit 6.

  First, the analysis command receiving unit 6 performs a process of receiving (inputting) an analysis request query from the service providing unit 1 (reception step S501). It is assumed that the analysis command receiving means 6 inputs a query including the command GetReferrerRanking as described above, and that the query includes the URL of data to be analyzed as a parameter. Hereinafter, the URL included in this query is referred to as an analysis target URL.

  Next, the analysis command receiving unit 6 searches the data stored in the analysis result storage unit 7 to determine whether or not the pre-analysis result can be used for the data specified by the URL to be analyzed (utilization of pre-analysis). Possibility check step S502). In this case, for example, the analysis command receiving unit 6 determines whether there is a URL that matches the URL to be analyzed among the URLs in the URL field (T07-01) of the analysis result management table stored in the analysis result storage unit 7. Search for.

  If the search is successful (when there is a matching URL), the analysis command receiving means 6 extracts one record that matches the URL from the analysis result management table stored in the analysis result storage means 7 as the search result. To do. When the search fails (when there is no matching URL), the analysis command accepting unit 6 sets a value (for example, NULL) indicating that the search fails (no available pre-analysis result exists). Return (as search result).

  Next, if there is a search result (that is, if a record is extracted as the search result in step S502), the analysis command receiving means 6 determines that the pre-analysis result can be used, and uses the pre-analysis result. The process proceeds to step (step S504). If there is no search result (that is, if a NULL value is obtained in step S502), the analysis command receiving means 6 determines that the pre-analysis result cannot be used, and proceeds to the processing of step (step S505). (Branch step S503).

  In the pre-analysis result utilization step S504, the analysis instruction receiving means 6 extracts from the analysis result storage means 7 the analysis result detailed table of the link destination indicated in the analysis result field of the search result record. Then, the analysis command receiving means 6 obtains the extracted analysis result detail table as an answer (preliminary analysis result) to the analysis request query.

  In the analysis request step step S505, the analysis instruction receiving means 6 transfers the analysis request query to the analysis means 5 in order to execute the analysis processing of the data designated by the analysis request query. In this case, for example, the analysis command receiving means 6 may transfer the analysis request query from the service providing means 1 to the analysis means 5 as it is. Then, the analysis command receiving unit 6 stands by for processing until the analysis result is received (input) from the analysis unit 5.

  Next, the analysis command receiving means 6 obtains (inputs) information in the same format as the analysis result detailed table stored in the analysis result storage means 7 from the analysis means 5 as an analysis result. Then, the analysis command accepting unit 6 obtains the analysis result input from the analysis unit 5 as a response (analysis result) to the analysis request query.

  Next, the analysis command receiving unit 6 answers (outputs) the pre-analysis result obtained in step S504 or the analysis result obtained in step S505 to the service providing unit 1 (analysis command answering step S506).

  In the present embodiment, for example, the processing of the preliminary analysis step (step S4) is executed by the data high-speed analysis server 105. Specifically, the analysis service requirement reading step (step S401) is executed by the pre-analysis target generation unit 802, and the pre-analysis target generation unit 802 reads data (service requirements) from the service requirement setting unit 2. The analysis target change detection step (step S402) is executed by the analysis target monitoring unit 801. Further, the process of the pre-analysis target determination step (step S403) is executed by the pre-analysis target generation unit 802. The process of the prior analysis request step (step S404) is executed by the prior analysis request means 803. Further, the process of the pre-analysis result registration step (step S405) is executed by the pre-analysis result registration unit 804. Further, the process of the preliminary analysis result answering step (step S406) is executed by the analysis command receiving means 6.

  As described above, according to the present embodiment, the prior analysis control unit 8 performs an analysis in advance before a business user requests a data analysis service. When a data analysis request is actually made, an analysis service is provided to the business user using this preliminary analysis result. By doing so, it is possible to greatly shorten the time from the operator user's operation of the analysis request until the analysis result is obtained, and it is possible to perform processing for providing a high-speed analysis service. . Therefore, even in the case of collecting and analyzing sequentially generated data, it is possible to perform analysis in response to a request for response time for each service.

  In the present embodiment, a case is shown in which service requirements are set for each data (for example, for each URL of a Web page) and whether or not to be subject to prior analysis is set. The manner is not limited to that shown in the present embodiment. For example, when a plurality of types of analysis processing can be executed, a service requirement may be set for each type of analysis processing, and whether or not to make a pre-analysis target may be set. Hereinafter, a modified example in the case where service requirements are set for each type of analysis processing will be described.

  When setting service requirements for each type of analysis processing (analysis method), the sequential data analysis system functions as shown in FIG. 1 so that it operates even when the analysis method differs for each data to be analyzed. It includes the following components that are expanded.

  The service providing unit 1 has a function of inputting data to be analyzed and an analysis method thereof. The service requirement setting means 2 has a function of setting a response time requirement (service requirement) for a set of data to be analyzed and an analysis method.

  The analysis unit 5 has a function of receiving a plurality of types of analysis and executing a plurality of types of analysis processing. The prior analysis control unit 8 has a function of determining target data to be analyzed in advance and an analysis method to be analyzed in advance based on the setting information set by the service requirement setting unit 2. The prior analysis control unit 8 has a function of designating an analysis target and an analysis method to the analysis unit 5 and instructing a prior analysis process. Moreover, the analysis result storage means 7 stores a pre-analysis result obtained by performing a pre-analysis process on a set of an analysis object and an analysis method as a pre-analysis result.

  Further, the analysis command receiving unit 6 uses the pre-analysis result stored in the analysis result storage unit 7 for the set of the analysis target and the analysis method, or causes the analysis unit 5 to newly execute an analysis process. It has a function to determine whether.

  Specifically, for example, the service providing unit 1 configures the analysis button U02 on the input screen shown in FIG. 3 so that the type of analysis processing can be selected, and the type of analysis processing (analysis method) according to the operator's user operation. ) Can be input. For example, the service providing unit 1 may display a list of analysis methods using a pull-down menu.

  Further, the service requirement setting means 2 sets and stores the service requirement using a table to which a field T02-03 including setting contents of the analysis method is added in the service requirement setting table shown in FIG. 4, for example. do it. For example, the service providing unit 1 and the service requirement setting unit 2 may perform processing by specifying an analysis method by an analysis name, or specify an ID (analysis ID) that can specify the analysis method internally. Then, management and processing may be executed.

  Moreover, the analysis means 5 can execute an analysis process by a plurality of types of analysis methods. In this case, for example, the analysis command receiving means 6 and the prior analysis control means 8 output an analysis request including an analysis name or an analysis ID as a parameter to the analysis means 5 in addition to the URL to be analyzed. do it. Then, the analysis means 5 may execute the specified type of analysis processing according to the analysis name or analysis ID included in the input analysis request.

  Further, the pre-analysis control unit 8 does not determine the data to be pre-analyzed based only on the URL to be analyzed, but based on the combination of the URL to be analyzed and the analysis method, the pre-analysis target data and analysis What is necessary is just to judge a method and to control a prior analysis process. For example, when searching for the service requirement set by the service requirement setting unit 2, the pre-analysis control unit 8 uses the set of the URL field T02-01 and the analysis method field T02-03 as a search key. The data and the analysis method may be searched. The prior analysis control means 8 outputs an analysis request including the analysis method as a parameter in addition to the URL of the data to be analyzed when the analysis means 5 is requested to execute the preliminary analysis processing. That's fine.

  Further, the analysis result storage means 7 may store a table in which a field T07-03 including an analysis method is added to the analysis result management table shown in FIG.

  Further, the analysis command receiving unit 6 receives (inputs) information specifying the URL to be analyzed and the analysis method from the service providing unit 1. Then, the analysis command accepting unit 6 checks (determines) whether or not the pre-analysis result can be used based on the input combination of the URL to be analyzed and the analysis method. In this case, for example, the analysis command receiving unit 6 uses the combination of the input URL to be analyzed and the analysis method as a search key, and the URL field T07-01 of the analysis result management table stored in the analysis result storage unit 7 and the analysis method The field T07-03 may be searched. Further, when the analysis command accepting unit 6 requests the analysis unit 5 to execute the analysis process, in addition to the URL of the data to be analyzed, the analysis method also outputs an analysis request in which the analysis method is designated as a parameter. Good.

  With the configuration as described above, even when the analysis method differs for each data to be analyzed, it is possible to perform a prior analysis and to enable a process for providing a high-speed analysis service.

Embodiment 2. FIG.
Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 12 is a block diagram illustrating an example of the configuration of the sequential generation data analysis system according to the second embodiment. FIG. 13 is a block diagram showing a more detailed configuration example of the preliminary analysis control unit 8 in the second embodiment.

  As shown in FIG. 12, this embodiment is different from the first embodiment in that the sequential data analysis system includes an analysis command history means 9 in addition to the components shown in FIG.

  Specifically, the analysis command history unit 9 is realized by a storage device such as a memory, a magnetic disk device, or an optical disk device provided in the information processing apparatus. The analysis command history unit 9 stores analysis command history information indicating the history of the query of the analysis request received by the analysis command reception unit 6.

  As shown in FIG. 13, the present embodiment is different from the first embodiment in that the prior analysis control means 8 includes a priority preliminary analysis determination means 805 in addition to the components shown in FIG. .

  The priority pre-analysis determination unit 805 has a function of counting the number of analysis histories and determining the priority so that instructions that appear more frequently in the analysis command history information stored in the analysis command history unit 9 are pre-analyzed. Prepare.

  FIG. 14 is a block diagram illustrating an example of a physical server configuration of the sequential data analysis system according to the second embodiment. As shown in FIG. 14, when configuring the sequential generation data analysis system according to the present embodiment, the analysis command history unit 9 may be configured to be included in, for example, the data analysis reception server 104. However, the example shown in FIG. 14 is an example. For example, the data high-speed analysis server 105 may include the analysis command history unit 9, and the analysis command history information is obtained by communicating with the analysis command receiving unit 6. You may make it receive (input).

  The analysis command history unit 9 stores the history information of the query of the analysis request requested by the service providing unit 1. In this case, the analysis command history unit 9 stores the analysis command history information in accordance with the analysis request query input via the analysis command receiving unit 6. That is, the analysis command receiving means 6 registers history information in the analysis command history means 9 every time an analysis request query is received (input).

  FIG. 15 is an explanatory diagram showing an example of analysis command history information stored in the analysis command history means 9. As shown in FIG. 15, the analysis command history unit 9 stores the date (T09-01) and the URL (T09-02) in association with each other. The date and time (T09-01) is the date and time when the analysis command receiving means 6 receives a query for an analysis request. URL (T09-02) is the value of the URL parameter specified in the query including the command GetReferrerRanking.

  The priority pre-analysis determination unit 805 receives (inputs) additional URLs of data to be analyzed from the pre-analysis target generation unit 802, and URLs (T09−) in which the analysis command history unit 9 stores the appearance frequency of each additional URL. 02). Further, the priority pre-analysis determination unit 805 sorts the additional URLs in descending order according to the counted number, and returns (outputs) the sorting result to the pre-analysis target generation unit 802.

  In addition, when counting the appearance frequency, the priority pre-analysis determination unit 805, for example, matches the URL in the URL field (T09-02) stored in the analysis command history unit 9 for each additional URL and records Count the number.

  In addition, when the pre-analysis target generation unit 802 receives (inputs) the sorted additional URLs, the pre-analysis target generation unit 802 performs control to request the analysis unit 5 to perform analysis processing one by one from the top of the sorted additional URLs.

  Generally, the analysis time required for the analysis process executed by the analysis unit 5 is a heavy process that takes several minutes to several hours. Therefore, if the additional URL data obtained as an analysis target is analyzed all at once, as in the sequential generation data analysis system shown in the first embodiment, a huge amount of analysis time may be required. I will not return. In this case, there is room for improvement because the preliminary analysis stored in the analysis result storage unit 7 cannot be updated while the analysis process is being executed.

  For example, in consideration of the actual usage mode of the analysis service, some of the information that is subjected to the pre-analysis process and stored in the analysis result storage means 7 may be requested by the operator user. Some may not be required at all. In such a case, if the pre-analysis processing is performed in a batch, the pre-analysis results cannot be used until all the analysis processing is completed, and the analysis results that are frequently used are provided instead. May not be received at high speed. In addition, there is a possibility that a situation in which a wasteful pre-analysis process is performed for a long time even though it is not actually used by a business user.

  Therefore, in the present embodiment, the pre-analysis processing is performed according to the priority order by pre-analyzing the data for the query of the analysis request having a higher appearance frequency instead of collectively processing the pre-analysis target data. By performing the processing in a time-distributed manner, it is possible to prevent a situation in which analysis results of high priority data cannot be obtained at high speed.

  In the present embodiment, the case where the priority of the pre-analysis processing is determined according to the appearance frequency is shown, but the method of determining the priority of the pre-analysis processing is not limited to that shown in the present embodiment. Prior analysis processing may be executed by determining priorities in order of increasing time.

  Next, the operation will be described. FIG. 16 is a flowchart illustrating an example of processing in which the sequentially generated data analysis system according to the second embodiment analyzes sequentially generated data. As shown in FIG. 16, in the present embodiment, the analysis command receiving means 6 executes the processing of the analysis command receiving step (step S501) and then executes the query of the analysis request received in the analysis command receiving step (step S501). An analysis command history step (step S506) for storing analysis command history information indicating a history is executed. Then, the analysis command receiving means 6 executes the processing of the pre-analysis availability check step (step S502) after executing the processing of the analysis command history step (step S506).

  FIG. 17 is a flowchart illustrating an example of a pre-analysis process in the second embodiment. As shown in FIG. 17, in this embodiment, the pre-analysis control unit 8 executes the process of the analysis target change detection step (step S402) of the pre-analysis step (step S4), and then in the analysis command history information. The number of analysis histories is counted so that more frequently appearing commands are pre-analyzed, and the process of the priority all analysis determination step (step S402) for determining the priority is executed. And the prior analysis control means 8 performs the process of a prior analysis object determination step (step S403), after performing the process of a priority all analysis determination step (step S402).

  In the present embodiment, for example, the processing of the analysis command history step (step S506) is executed by causing the analysis command receiving unit 6 to store history information in the analysis command history unit 9. Further, the process of the priority pre-analysis determination step (step S407) is executed by the priority pre-analysis determination unit 805.

  As described above, according to the present embodiment, the priority level for executing the pre-analysis in consideration of the frequency of the analysis request received from the operator user by the analysis command history unit 9 and the priority pre-analysis determination unit 805. And pre-analysis is performed with time dispersion according to the determined priority. For this reason, it is possible to execute the analysis processing at high speed more effectively.

Embodiment 3. FIG.
Next, a third embodiment of the present invention will be described with reference to the drawings. FIG. 18 is a block diagram illustrating an example of a configuration of a sequential generation data analysis system according to the third embodiment. FIG. 19 is a block diagram showing a more detailed configuration example of the preliminary analysis control means 8 in the third embodiment.

  As shown in FIG. 18, this embodiment is different from the first embodiment in that the sequential data analysis system includes an analysis cost monitoring unit 10 in addition to the components shown in FIG. 1. Note that the sequential data analysis system may be configured to include the analysis cost monitoring means 10 in addition to the components shown in the second embodiment.

  Specifically, the analysis cost monitoring unit 10 is realized by a CPU of an information processing device that operates according to a program and a storage device such as a memory. The analysis cost monitoring unit 10 has a function of monitoring the time taken for each analysis process executed by the analysis unit 5.

  Further, as shown in FIG. 19, the present embodiment is different from the first embodiment in that the prior analysis control means 8 includes priority prior analysis determination means 805A in addition to the components shown in FIG. .

  Based on the monitoring result of the analysis cost monitoring unit 10, the priority pre-analysis determination unit 805A determines the priority so that the pre-analysis control unit 8 performs control to pre-analyze a command having a high analysis cost. It has a function. Note that the priority pre-analysis determination unit 805A may also have a priority determination function similar to the priority pre-analysis determination unit 805 shown in the second embodiment.

  FIG. 20 is a block diagram illustrating a more detailed configuration example of the analysis cost monitoring unit 10 in the third embodiment. As shown in FIG. 20, the analysis cost monitoring unit 10 includes an analysis cost calculation unit 1001 and an analysis cost storage unit 1002.

  The analysis cost calculation unit 1001 temporarily stores the analysis start time in a storage device such as a memory, and at the timing when the analysis end time is received (input) from the analysis unit 5, for example, it takes an analysis. A function for calculating the cost of analysis by calculating time is provided. The analysis cost storage unit 1002 stores the calculation result of the analysis cost calculation unit 1001 (analysis cost for each analysis process).

  FIG. 21 is a block diagram illustrating an example of a physical server configuration of a sequential data analysis system according to the third embodiment. As shown in FIG. 21, when configuring the sequential generation data analysis system in the present embodiment, the analysis cost monitoring means 10 may be configured to be included in the analysis server 102, for example. However, the example shown in FIG. 21 is merely an example. For example, the high-speed data analysis server 105 may include the analysis cost monitoring unit 10, and the analysis status is received (input) by communicating with the analysis unit 5. You may do it.

  When executing the analysis processing, the analysis unit 5 outputs a start command and an end command to the analysis cost monitoring unit 10 at the timing of analysis start and analysis end, respectively. For example, the analysis unit 5 outputs a start command and an end command in the format shown below.

Start analysis:
Start command format: (StartAnalysis, URL of data to be analyzed)
Example: (StartAnalysis, “http://bar.co.jp/~foo/Page-001”)

End of analysis:
End instruction format: (EndAnalysis, URL of data to be analyzed)
Example: (EndAnalysis, “http://bar.co.jp/~foo/Page-001”)

  Here, StartAnalysis and EndAnalysis are command names. Note that the command names included in the start instruction and the end instruction are not limited to the expressions shown above. Further, the format of the start instruction and the end instruction itself is also an example, and is not limited to the expression shown above.

  The analysis cost calculation means 1001 receives (inputs) from the analysis means a start instruction including a StartAnalysis command and an end instruction including an EndAnalysis command. Then, the analysis cost calculation unit 1001 calculates the cost (time) required for each analysis process by obtaining the time difference between the input of the start command and the end command.

  FIG. 22 is an explanatory diagram illustrating an example of information stored in the analysis cost calculation unit 1001. As shown in FIG. 22, when the analysis cost calculation unit 1001 receives a start command including a StartAnalysis command, the analysis cost calculation unit 1001 stores the start time in association with each URL as a table. In FIG. 22, URL (T10-01) is the URL of data for which analysis has started. The analysis start time (T10-02) is the time when the analysis is started.

  When the analysis cost calculation unit 1001 receives (inputs) an end command including an EndAnalysis command, the analysis cost calculation unit 1001 matches the URL value included in the end command with the stored URL (T10-01). ) Next, the analysis cost calculation unit 1001 extracts the analysis start time (T10-02) corresponding to the matched URL. Then, the analysis cost calculation unit 1001 calculates the cost required for this analysis process by obtaining the difference between the value of the extracted analysis start time (T10-02) and the current time.

  The analysis cost storage unit 1002 receives (inputs) and stores the calculation result of the cost of the analysis process from the analysis cost calculation unit 1001. In this case, the analysis cost calculation unit 1001 outputs the calculation result of the analysis cost in the following format, for example.

Analysis cost calculation result format: (SetCost, URL of analysis, analysis cost)
Example: (SetCost, “http://bar.co.jp/~foo/Page-001, 10)

  The analysis cost storage unit 1002 stores the calculation result of the analysis cost as a table having a format shown in FIG. In FIG. 23, URL (T10-03) is the URL of the analysis target data. The analysis cost (T10-04) is the total time required for the analysis obtained by the analysis cost calculation unit 1001.

  The analysis cost calculation unit 1001 outputs a SetCost command, and searches the URL field (T10-03) of the table stored in the analysis cost storage unit 1002 using the URL value of this command as a key. Then, the analysis cost calculation unit 1001 analyzes the total of the value of the analysis cost field (T10-04) of the record specified by the search and the analysis cost included in the SetCost instruction. The value of the cost field (T10-04) is overwritten.

  Note that the cost calculation method of the analysis processing shown in the present embodiment is an example, and the analysis cost calculation unit 1001 may obtain an average value or the like as the analysis cost instead of the total value, for example. Further, the format of an instruction output by the analysis cost calculation unit 1001 is an example, and is not limited to that shown in the present embodiment.

  In the second embodiment, the pre-analysis control unit 8 executes the process of sorting the additional URLs of the data to be pre-analyzed according to the number of data of the analysis command history information. In this embodiment, the pre-analysis control unit 8 What is necessary is just to sort according to the value of the analysis cost (T10-04) which 1002 memorize | stores. The pre-analysis control unit 8 does not process the data to be pre-analyzed at once, but may perform time-distributed processing by, for example, preferential processing in ascending order of analysis cost.

  Note that there is no entry in the analysis cost storage unit 1002 if the analysis unit 5 has not yet executed an analysis process. In this case, a fixed value (default value) may be stored as a calculation cost in advance in the analysis cost storage unit 1002, and the operation may be covered.

  Further, the priority determination process shown in the second embodiment and the priority determination process shown in the third embodiment may be used in combination. In this case, the priority pre-analysis determination unit 805 obtains a score obtained by adding the count value and the analysis cost value with an appropriate weight, and sorts the additional URLs of the data to be analyzed according to the obtained score. Good. Note that this sorting method is an example, and the method in which the priority pre-analysis determination unit 805 sorts the additional URLs of the data to be analyzed based on the count value and the analysis cost value is described in this embodiment. It is not limited to the method.

  Next, the operation will be described. FIG. 24 is a flowchart illustrating an example of processing in which the sequentially generated data analysis system according to the third embodiment analyzes sequentially generated data. As shown in FIG. 24, in this embodiment, in addition to the processing shown in the first embodiment and the second embodiment, the analysis cost monitoring means 10 executes the processing of the analysis cost monitoring step (step S6). To do.

  FIG. 25 is a flowchart illustrating an example of the pre-analysis process in the third embodiment. As shown in FIG. 25, in the present embodiment, the prior analysis control unit 8 executes the process of the analysis target change detection step (step S402) and then performs the analysis process calculated in the analysis cost monitoring step (step S6). The priority pre-analysis determination step (step S407) for determining the prioritization of data to be pre-analyzed using the cost is executed. In addition, the pre-analysis control unit 8 executes the process of the pre-analysis target determination step (step S403) after executing the process of the priority pre-analysis determination step (step S407).

  In the present embodiment, for example, the analysis cost monitoring step (step S6) is executed by the analysis cost monitoring means 10. Further, the process of the priority pre-analysis determination step (step S407) is executed by the priority pre-analysis determination unit 805.

  As described above, according to the present embodiment, the analysis cost monitoring unit 10 determines the priority for executing the pre-analysis in consideration of the time required for the analysis process, and the pre-analysis is time-distributed according to the determined priority. And run. For this reason, it is possible to execute the analysis processing at high speed more effectively.

  In the present embodiment, as in the second embodiment, the analysis command history means 9 may also be provided. With such a configuration, in addition to the time required for the analysis process, the frequency of receiving the analysis request from the operator user is also considered, and the priority for executing the pre-analysis is determined, and the priority is determined in advance according to the determined priority. Analysis can be performed with time dispersion. Therefore, it is possible to more efficiently execute high-speed analysis processing.

Embodiment 4 FIG.
Next, a fourth embodiment of the present invention will be described with reference to the drawings. FIG. 26 is a block diagram showing a more detailed configuration example of the preliminary analysis control unit 8 in the fourth embodiment. As shown in FIG. 26, the present embodiment is different from the first embodiment in that the prior analysis control unit 8 includes a free storage capacity monitoring unit 806 in addition to the components shown in FIG. The prior analysis control unit 8 may be configured to include a free storage capacity monitoring unit 806 in addition to the components shown in the second embodiment or the third embodiment.

  The free storage capacity monitoring unit 806 has a function of monitoring the free capacity of a resource such as a memory of an information processing apparatus (PC) in which the pre-analysis control unit 8 operates in order to narrow down data to be analyzed in advance.

  The free storage capacity monitoring means 806 is set in advance in a table in which the relationship between the free capacity of resources and the number of data that can be analyzed in advance is set. The free storage capacity monitoring unit 806 obtains the number of data that can be analyzed in advance according to a preset table.

  FIG. 27 is an explanatory diagram showing an example of a table preset by the free storage capacity monitoring unit 806. As shown in FIG. 27, the free storage capacity monitoring unit 806 sets a table in which the free storage capacity (T0806-01) and the preanalysable number (T0806-02) are associated in advance. The free storage capacity (T0806-01) is an available free storage capacity out of the total capacity of resources such as a memory provided in the information processing apparatus in which the prior analysis control unit 8 operates. The preanalysable number (T0806-02) is the maximum number of data that can be preanalyzed with respect to the free storage capacity.

  When the free storage capacity monitoring unit 806 receives an inquiry about the number of data that can be preanalyzed from the preanalysis target generation unit 802 (when a calculation request is input), the system of the information processing apparatus (PC) in which the preanalysis control unit 8 operates. Obtain (extract) the free storage capacity of a resource such as a memory from a call. Then, the free storage capacity monitoring unit 806 extracts the corresponding preanalysable number from the table shown in FIG. 27 based on the extracted free storage capacity. By performing such processing, the free storage capacity monitoring unit 806 calculates the number that can be pre-analyzed, and returns (outputs) the calculation result to the pre-analysis target generation unit 802. For example, in the case of the example shown in FIG. 27, when the free storage capacity monitoring unit 806 extracts the free capacity as 700 Kbytes, it calculates and returns (outputs) the number of data that can be analyzed in advance as 50.

  When a plurality of applications are operating in the information processing apparatus in which the prior analysis control unit 8 operates, the free storage capacity monitoring unit 806 is a memory used by other applications operating on the same information processing apparatus. The capacity may be taken into account and monitored.

  The pre-analysis target generation unit 802 obtains the number of data that can be preanalyzed (inputs the calculation result) by inquiring the free storage capacity monitoring unit 806 about the preanalysable number when determining the preanalysis target. The pre-analysis target generation unit 802 performs processing to narrow down the data to be analyzed when the number of data to be pre-analyzed is larger than the number of data that can be pre-analyzed.

  Note that the pre-analysis target generation unit 802 narrows down the data to be pre-analyzed, for example, as in the second embodiment or the third embodiment, the additional URL of the pre-analysis target data depends on the priority. In the case of being sorted, the prior analysis candidates may be deleted in descending order of priority. Further, when the additional URLs are not sorted as in the first embodiment, the pre-analysis target generation unit 802 may delete the pre-analysis target data at random.

  Next, the operation will be described. FIG. 28 is a flowchart illustrating an example of processing in which the sequentially generated data analysis system according to the fourth embodiment analyzes sequentially generated data. As shown in FIG. 28, in this embodiment, in addition to the processing shown in the first embodiment, the second embodiment, and the third embodiment, the prior analysis control means 8 is an information processing apparatus (PC). Based on the free capacity of the resource such as the memory, the processing of the free storage capacity monitoring step (step S7) for estimating the number of data that can be analyzed in advance is executed.

  FIG. 29 is a flowchart illustrating an example of a pre-analysis process according to the fourth embodiment. As shown in FIG. 29, in this embodiment, the pre-analysis control means 8 performs the analysis calculated in the process of the free storage capacity monitoring step (step S7) after executing the process of the analysis object change detection step (step S402). Based on the number of possible data, the process of the pre-analysis target narrowing step (step S408) for narrowing down the pre-analysis target data is executed. In addition, the pre-analysis control unit 8 executes the process of the pre-analysis target determination step (step S403) after executing the process of the pre-analysis target narrowing step (step S408).

  In this embodiment, for example, the process of the free storage capacity monitoring step (step S7) is executed by the free storage capacity monitoring unit 806. Further, the process of the pre-analysis target narrowing step (step S408) is executed by the priority pre-analysis determination unit 802.

  As described above, according to the present embodiment, the free storage capacity monitoring unit 806 determines the priority for executing the prior analysis in consideration of the free resources of the information processing apparatus (PC) that executes the analysis processing. , Narrow down the data to be analyzed in advance. Therefore, it is possible to prevent a situation in which the preliminary analysis cannot be performed due to a shortage of resources of the information processing apparatus that executes the analysis process, and it is possible to execute the analysis process more effectively and more efficiently.

  Next, the minimum configuration of the sequential data analysis apparatus according to the present invention will be described. FIG. 30 is a block diagram illustrating a minimum configuration example of the sequential generation data analysis apparatus. As shown in FIG. 30, the sequential generation data analysis apparatus includes a pre-analysis control unit 8 as a minimum component. In FIG. 30, the prior analysis control means 8 has a function of performing control to execute a process of performing prior analysis on sequentially generated data, which is data that is sequentially generated, before an analysis request.

  According to the minimally generated sequential data analysis apparatus shown in FIG. 30, even if data generated sequentially is collected and analyzed, it is possible to perform analysis in response to a request for response time for each service. .

  In each of the embodiments described above, the characteristic configuration of the sequential generation data analysis apparatus as shown in the following (1) to (18) is shown.

(1) Sequentially generated data analysis devices (for example, the data high-speed analysis server 105. For example, the analysis server 102, the data analysis reception server 104, and the information processing device for realizing the data high-speed analysis server 105) are preceded by an analysis request. In addition, a pre-analysis control means (for example, a control for executing a process of pre-analyzing sequentially generated data (for example, log data output from a web server, data output from a temperature sensor or a humidity sensor) that is data generated sequentially) (Provided by the pre-analysis control means 8).

(2) Further, the sequential generation data analysis device is configured to analyze whether or not the sequential generation data to be analyzed has already been pre-analyzed by control by the pre-analysis control unit when an analysis request is generated ( For example, when the analysis method determination unit determines that the analysis is not performed in advance, the analysis method determination unit performs control to execute processing for analyzing the sequentially generated data to be analyzed, and the analysis result May be configured to output an analysis result that has already been pre-analyzed.

(3) Further, the sequential generation data analysis apparatus includes a pre-analysis result storage unit (for example, realized by the analysis result storage unit 7) that stores a pre-analysis result pre-analyzed by control by the pre-analysis control unit, The analysis method determining means determines whether the analysis target is generated sequentially by determining whether or not the pre-analysis result corresponding to the sequential generation data to be analyzed is stored in the pre-analysis result storage means when the analysis request is generated. It may be configured to determine whether the data has already been pre-analyzed.

(4) Further, in the sequential generation data analysis apparatus, the pre-analysis control means sets a pre-analysis target based on a pre-analysis condition (for example, service requirement) indicating a target condition to be pre-set. It may be configured to perform control for determining the sequentially generated data and executing a process of analyzing the sequentially generated data in advance.

(5) Further, in the sequential generation data analyzing apparatus, the prior analysis control unit includes a sequential generation data monitoring unit (for example, realized by the analysis target monitoring unit 801) for monitoring the generation status of the sequential generation data, and the sequential generation data. Based on the monitoring result of the monitoring means and the pre-analysis condition set by the pre-analysis condition setting means, a pre-analysis target specifying means (for example, pre-analysis) that specifies sequential generation data to be pre-analyzed preferentially And a pre-analysis execution control unit (for example, a pre-analysis requesting unit 803) that performs control to execute analysis of sequentially generated data of the pre-analysis target specified by the pre-analysis target specifying unit. Pre-analysis that stores the pre-analysis results of the analysis results of the pre-analysis executed by the control of the pre-analysis execution control means Pre-analysis result registration means for storing the result storage means (for example, implemented by prior analysis result registration means 804) and may be configured to include.

(6) Further, the sequential generation data analysis apparatus includes an analysis request history storage unit (for example, realized by the analysis command history unit 9) for storing analysis request history information, and the prior analysis control unit includes an analysis request history Appearance for determining the priority of sequential data to be pre-analyzed based on historical information stored in the storage means so that sequential data corresponding to an analysis request with a high frequency of occurrence is preferentially analyzed. It may be configured to include a frequency priority pre-analysis determination unit (for example, realized by the priority pre-analysis determination unit 805).

(7) Further, the sequential generation data analyzing apparatus includes an analysis cost monitoring unit (for example, realized by the analysis cost monitoring unit 10) for monitoring an analysis cost for analyzing the sequentially generated data. And an analysis cost priority pre-analysis determination unit (for example, realized by the priority pre-analysis determination unit 805A) that determines the priority of sequentially generated data to be pre-analyzed based on the monitoring result of the analysis cost monitoring unit. It may be configured.

(8) Further, in the sequential generation data analysis device, the analysis cost monitoring means temporarily stores a start time which is a time when the analysis of the sequential generation data is started, and performs the analysis when the analysis of the sequential generation data is finished. Based on the end time that is the end time and the temporarily stored start time, an analysis cost calculation unit that calculates the analysis cost (for example, realized by the analysis cost calculation unit 1001), and a calculation result of the analysis cost calculation unit Analysis cost storage means (for example, realized by the analysis cost storage means 1002).

(9) Further, the sequential data analysis apparatus includes resource monitoring means (for example, realized by the free storage capacity monitoring means 806) for monitoring the free capacity of the resource of the information processing apparatus used for the analysis processing of the sequentially generated data. The pre-analysis control unit may be configured to narrow down sequentially generated data to be pre-analyzed based on the monitoring result of the resource monitoring unit.

(10) The sequential generation data analysis device may be configured to include a prior analysis control unit that performs control to execute a process of performing preliminary analysis on sequential generation data that is sequentially generated before an analysis request. Good.

(11) Further, the sequential generation data analysis device includes an analysis method determination unit that determines whether or not the sequential generation data to be analyzed has already been pre-analyzed by control by the pre-analysis control unit when an analysis request is generated. If the analysis method determination unit determines that the analysis is not performed in advance, the analysis method determination unit performs a process of analyzing the sequentially generated data to be analyzed, outputs the analysis result, and determines that the analysis is already performed in advance. The analysis result that has already been analyzed in advance may be output.

(12) In addition, the sequential generation data analysis device includes a pre-analysis result storage unit that stores a pre-analysis result pre-analyzed by control by the pre-analysis control unit, and the analysis method determination unit is configured to generate an analysis request when the analysis request is generated. By determining whether or not the pre-analysis result corresponding to the sequentially generated data to be analyzed is stored in the pre-analysis result storage unit, it is determined whether or not the sequentially generated data to be analyzed has already been pre-analyzed. It may be configured as follows.

(13) In the sequential data analysis device, the pre-analysis control unit determines sequential data to be pre-analyzed based on a pre-analysis condition that is set in advance and indicates a condition to be pre-analyzed. Further, it may be configured to perform control for executing processing for pre-analyzing the sequentially generated data.

(14) Further, in the sequential data analysis device, the pre-analysis control unit includes a sequential data monitoring unit that monitors a generation state of the sequential data, a monitoring result of the sequential data monitoring unit, and a pre-analysis condition setting unit. Based on the pre-analysis conditions that have been set, the pre-analysis target specifying unit that specifies sequential generation data to be pre-analyzed, and the pre-analysis target specifying unit specified by the pre-analysis target specifying unit Pre-analysis execution control unit that performs control to execute analysis, and pre-analysis result registration that stores the analysis result of the pre-analysis executed by the control of the pre-analysis execution control unit in the pre-analysis result storage unit that stores the pre-analysis result May be included.

(15) The sequential generation data analysis apparatus includes an analysis request history storage unit that stores analysis request history information, and the prior analysis control unit generates an appearance frequency based on the history information stored in the analysis request history storage unit. It is configured to include an appearance frequency priority pre-analysis determination unit that determines the priority of sequential data to be pre-analyzed so that sequential data corresponding to a high analysis request is given priority. May be.

(16) Further, the sequential generation data analysis apparatus includes an analysis cost monitoring unit that monitors an analysis cost for analysis of the sequential generation data, and the prior analysis control unit is configured in advance based on the monitoring result of the analysis cost monitoring unit. An analysis cost priority prior analysis determination unit that determines the priority of the sequentially generated data to be analyzed may be included.

(17) Further, in the sequential generation data analysis apparatus, the analysis cost monitoring unit temporarily stores a start time that is a time when the analysis of the sequential generation data is started, and performs the analysis when the analysis of the sequential generation data is finished. An analysis cost calculation unit that calculates an analysis cost based on an end time that is an end time and a temporarily stored start time, and an analysis cost storage unit that stores a calculation result of the analysis cost calculation unit May be.

(18) Further, the sequential generation data analysis apparatus includes a resource monitoring unit that monitors the free capacity of the resources of the information processing apparatus used for the analysis processing of the sequential generation data, and the prior analysis control unit displays the monitoring result of the resource monitoring unit. Based on this, it may be configured to narrow down sequentially generated data to be analyzed in advance.

  Moreover, in each embodiment shown above, the characteristic structure of the sequential generation data analysis system as shown to the following (19)-(36) is shown.

(19) The sequential generation data analysis system sequentially executes a data generation apparatus (for example, the application server 101) that generates sequential generation data that is sequential generation data, and analysis processing of the sequential generation data generated by the data generation unit. The analysis result of the generated data analysis device (for example, the data high-speed analysis server 105. For example, the analysis server 102, the data analysis reception server 104, and the information processing device that implements the data high-speed analysis server 105) And a service providing device (for example, the data analysis service server 103) that executes a process for providing a data analysis service, and the sequentially generated data analyzing device pre-analyzes the sequentially generated data before the analysis request. It includes a pre-analysis control means for performing control to execute processing. .

(20) In the sequential generation data analysis system, the sequential generation data analysis apparatus determines whether or not the sequential generation data to be analyzed has already been pre-analyzed by control by the pre-analysis control means when an analysis request is generated. If the analysis method determination means determines that it has not been pre-analyzed, the analysis method determination means performs control to execute processing for analyzing the sequentially generated data to be analyzed, outputs the analysis result, and has already been analyzed in advance. If it is determined that the result is already analyzed, the analysis result that has already been analyzed in advance may be output.

(21) In the sequential generation data analysis system, the sequential generation data analysis apparatus includes a pre-analysis result storage unit that stores a pre-analysis result pre-analyzed by control by the pre-analysis control unit, and the analysis method determination unit includes: When an analysis request is generated, the analysis target sequential generation data is already pre-analyzed by determining whether the preliminary analysis result corresponding to the analysis target sequential generation data is stored in the preliminary analysis result storage means. It may be configured to determine whether or not.

(22) Further, in the sequential generation data analysis system, the service providing apparatus sets a pre-analysis condition setting unit (for example, realized by the service requirement setting unit 2) that sets a pre-analysis condition indicating a condition to be pre-analyzed. And the pre-analysis control means determines the sequential generation data to be pre-analyzed based on the pre-analysis conditions set by the pre-analysis condition setting means, and executes a process of pre-analyzing the sequential generation data It may be configured to perform.

(23) In the sequential generation data analysis system, the prior analysis control means includes a sequential generation data monitoring means for monitoring a generation status of the sequential generation data, a monitoring result of the sequential generation data monitoring means, and a preliminary analysis condition setting means. Based on the set pre-analysis conditions, the pre-analysis target specifying means for specifying the pre-analysis target data to be pre-analyzed and the pre-analysis target specifying means specifying the pre-analysis target specifying means Pre-analysis execution control means for performing control to execute analysis, and pre-analysis result registration for storing the analysis result of the pre-analysis executed by the control of the pre-analysis execution control means in the pre-analysis result storage means for storing the pre-analysis result Means may be included.

(24) In the sequential generation data analysis system, the sequential generation data analysis apparatus includes an analysis request history storage unit that stores analysis request history information, and the prior analysis control unit stores the history stored in the analysis request history storage unit. Appearance frequency priority pre-analysis determination means for determining the priority of sequentially generated data to be pre-analyzed based on information so that sequential data corresponding to an analysis request having a high appearance frequency is preferentially analyzed May be included.

(25) In the sequential generation data analysis system, the sequential generation data analysis device includes an analysis cost monitoring unit that monitors an analysis cost for analyzing the sequential generation data, and the prior analysis control unit includes the analysis cost monitoring unit. An analysis cost priority pre-analysis determination unit that determines the priority of sequentially generated data to be pre-analyzed based on the monitoring result may be included.

(26) In the sequential generation data analysis system, the analysis cost monitoring means temporarily stores a start time that is a time when the analysis of the sequential generation data is started, and performs the analysis when the analysis of the sequential generation data is finished. An analysis cost calculation unit that calculates an analysis cost based on an end time that is an end time and a temporarily stored start time, and an analysis cost storage unit that stores a calculation result of the analysis cost calculation unit May be.

(27) In the sequential generation data analysis system, the sequential generation data analysis apparatus includes a resource monitoring unit that monitors the free capacity of the resource of the information processing apparatus used for the analysis process of the sequential generation data. Based on the monitoring result of the resource monitoring means, it may be configured to narrow down the sequentially generated data to be analyzed in advance.

(28) A sequential generation data analysis system includes a data generation device that generates sequential generation data that is sequentially generated data, a sequential generation data analysis device that executes analysis processing of sequential generation data generated by a data generation unit, And a service providing device that executes a process for providing a data analysis service based on the analysis result of the generated data analysis device. The sequential generated data analysis device performs a process of pre-analyzing sequentially generated data before an analysis request. It may be configured to include a pre-analysis control unit that performs control to execute the above.

(29) In the sequential generation data analysis system, the sequential generation data analysis apparatus determines whether or not the sequential generation data to be analyzed has already been pre-analyzed by control by the pre-analysis control unit when an analysis request is generated. If the analysis method determination unit determines that it has not been pre-analyzed, the analysis method determination unit performs control to execute a process of analyzing sequentially generated data to be analyzed, outputs the analysis result, and has already been analyzed in advance. If it is determined that the result is already analyzed, the analysis result that has already been analyzed in advance may be output.

(30) In the sequential generation data analysis system, the sequential generation data analysis device includes a pre-analysis result storage unit that stores a pre-analysis result pre-analyzed by control by the pre-analysis control unit, and the analysis method determination unit includes: When an analysis request is generated, the analysis target sequential generation data is already pre-analyzed by determining whether the preliminary analysis result corresponding to the analysis target sequential generation data is stored in the preliminary analysis result storage unit. It may be configured to determine whether or not.

(31) In the sequential generation data analysis system, the service providing apparatus includes a pre-analysis condition setting unit that sets a pre-analysis condition indicating a condition to be pre-analyzed, and the pre-analysis control unit is configured to set the pre-analysis condition. Based on the pre-analysis conditions set by the unit, it may be configured to determine the sequentially generated data to be pre-analyzed, and to perform control to execute a process of pre-analyzing the sequentially generated data.

(32) In the sequential generation data analysis system, the prior analysis control unit includes a sequential generation data monitoring unit that monitors a generation state of the sequential generation data, a monitoring result of the sequential generation data monitoring unit, and a preliminary analysis condition setting unit. Based on the pre-analysis conditions that have been set, the pre-analysis target specifying unit that specifies sequential generation data to be pre-analyzed, and the pre-analysis target specifying unit specified by the pre-analysis target specifying unit Pre-analysis execution control unit that performs control to execute analysis, and pre-analysis result registration that stores the analysis result of the pre-analysis executed by the control of the pre-analysis execution control unit in the pre-analysis result storage unit that stores the pre-analysis result May be included.

(33) In the sequential generation data analysis system, the sequential generation data analysis apparatus includes an analysis request history storage unit that stores analysis request history information, and the prior analysis control unit stores the history stored in the analysis request history storage unit. Based on the information, the appearance frequency priority pre-analysis determination unit that determines the priority of the sequentially generated data to be analyzed in advance so that the sequentially generated data corresponding to the analysis request having a high appearance frequency is preferentially analyzed. May be included.

(34) In the sequential generation data analysis system, the sequential generation data analysis apparatus includes an analysis cost monitoring unit that monitors an analysis cost for analyzing the sequential generation data, and the prior analysis control unit is an analysis cost monitoring unit. An analysis cost priority pre-analysis determination unit that determines the priority of sequentially generated data to be pre-analyzed based on the monitoring result may be included.

(35) Further, in the sequential generation data analysis system, the analysis cost monitoring unit temporarily stores a start time which is a time when the analysis of the sequential generation data is started, and performs the analysis when the analysis of the sequential generation data is finished. An analysis cost calculation unit that calculates an analysis cost based on an end time that is an end time and a temporarily stored start time, and an analysis cost storage unit that stores a calculation result of the analysis cost calculation unit May be.

(36) In the sequential generation data analysis system, the sequential generation data analysis apparatus includes a resource monitoring unit that monitors the free capacity of the resource of the information processing apparatus used for the analysis processing of the sequential generation data. Based on the monitoring result of the resource monitoring unit, it may be configured to narrow down sequentially generated data to be analyzed in advance.

  Moreover, in each embodiment shown above, the characteristic structure of the sequential generation data analysis method as shown to the following (37)-(45) is shown.

(37) The sequential generation data analysis method includes a pre-analysis control step for performing control for executing a process of performing pre-analysis on sequential generation data that is sequentially generated before an analysis request.

(38) Further, the sequential generation data analysis method includes an analysis method determination step for determining whether or not the sequential generation data to be analyzed has already been analyzed in advance when an analysis request is generated. If it is determined that it has not been pre-analyzed, it performs control to analyze the sequentially generated data to be analyzed, outputs the analysis result, and if it has already been pre-analyzed, it has already been pre-analyzed The analysis result may be output.

(39) Further, the sequential generation data analysis method includes a pre-analysis result storage step of storing the pre-analysis result pre-analyzed in the pre-analysis result storage means, and when an analysis request is generated in the analysis method determination step, By determining whether or not the pre-analysis result corresponding to the sequentially generated data to be analyzed is stored in the pre-analysis result storage unit, it is determined whether or not the sequentially generated data to be analyzed has already been pre-analyzed. It may be configured.

(40) Further, the sequential generation data analysis method includes a pre-analysis condition setting step for setting a pre-analysis condition indicating a condition to be pre-analyzed, and based on the set pre-analysis condition in the pre-analysis control step Further, it may be configured to determine the sequentially generated data to be analyzed in advance and perform control to execute a process of analyzing the sequentially generated data in advance.

(41) Further, the sequential generation data analysis method monitors the generation status of the sequential generation data in the preliminary analysis control step, and based on the monitoring result of the generation status of the sequential generation data and the set preliminary analysis conditions, Specify the sequential data to be pre-analyzed with priority, perform control to execute the sequential data of the specified pre-analysis target, and store the pre-analysis results and pre-analysis results You may comprise so that it may memorize | store in a result memory | storage means.

(42) The sequential generation data analysis method includes an analysis request history storage step of storing analysis request history information in the analysis request history storage means, and history information stored in the analysis request history storage means in the prior analysis control step. The priority of the sequentially generated data to be analyzed in advance may be determined so that the sequentially generated data corresponding to the analysis request having a high appearance frequency is preferentially analyzed based on .

(43) The sequential generation data analysis method includes an analysis cost monitoring step for monitoring an analysis cost for analysis of the sequential generation data, and the preliminary analysis target is based on the analysis cost monitoring result in the preliminary analysis control step. It may be configured to determine the priority of the sequentially generated data.

(44) In the sequential generation data analysis method, in the analysis cost monitoring step, the start time, which is the time at which the analysis of the sequential generation data is started, is temporarily stored. The analysis cost may be calculated based on the end time that is the end time and the temporarily stored start time, and the calculation result of the analysis cost may be stored in the analysis cost storage unit.

(45) Further, the sequential generation data analysis method includes a resource monitoring step of monitoring the free capacity of the resource of the information processing apparatus used for the analysis processing of the sequential generation data, and based on the monitoring result of the resource in the prior analysis control step. The sequential generation data to be analyzed in advance may be narrowed down.

  Furthermore, in each embodiment shown above, the characteristic structure of the sequential generation data analysis program as shown to the following (46)-(54) is shown.

(46) The sequential generation data analysis program causes a computer to execute a pre-analysis control process for performing a control to execute a pre-analysis process of sequential generation data that is sequentially generated before an analysis request. It is.

(47) The sequential generation data analysis program causes the computer to execute an analysis method determination process for determining whether or not the sequential generation data to be analyzed has already been pre-analyzed when an analysis request is generated. In the method determination process, if it is determined that it has not been pre-analyzed, the control for executing the process of analyzing the sequentially generated data to be analyzed is performed. A process for outputting an analysis result that has already been analyzed in advance may be executed.

(48) The sequential generation data analysis program causes the computer to execute a pre-analysis result storage process for storing the pre-analysis result pre-analyzed in the pre-analysis result storage means, and an analysis request is generated in the analysis method determination process. Whether or not the sequential analysis data to be analyzed has already been pre-analyzed by determining whether or not the preliminary analysis result corresponding to the sequential generation data to be analyzed is stored in the preliminary analysis result storage means It is also possible to execute a process for determining the above.

(49) Further, the sequential generation data analysis program is configured to sequentially generate data to be pre-analyzed based on a pre-analysis condition that is set in advance in the pre-analysis control process and indicates a condition to be pre-analyzed. May be executed, and a process for performing a control for executing a process of pre-analyzing the sequentially generated data may be executed.

(50) Further, the sequential generation data analysis program causes the computer to monitor the generation status of the sequential generation data, the monitoring result of the generation status of the sequential generation data, and the set pre-analysis condition in the preliminary analysis control processing. Based on the above, the process that specifies the sequential data that should be subject to prior analysis, the process that performs the control to execute the analysis of the sequential data that has been identified, and the analysis result of the prior analysis A process of storing in a pre-analysis result storage unit that stores the pre-analysis result may be executed.

(51) Further, the sequential generation data analysis program causes the computer to execute analysis request history storage processing for storing analysis request history information in the analysis request history storage means. In the prior analysis control processing, the analysis request history storage means Based on the history information to be stored, the processing for determining the priority of the sequentially generated data to be pre-analyzed is performed so that the sequentially generated data corresponding to the analysis request having a high appearance frequency is preferentially analyzed. It may be a thing.

(52) Further, the sequential generation data analysis program causes the computer to execute an analysis cost monitoring process for monitoring an analysis cost for analyzing the sequentially generated data, and based on the analysis cost monitoring result in the prior analysis control process. A process for determining the priority of sequentially generated data to be analyzed in advance may be executed.

(53) Further, the sequential generation data analysis program temporarily stores in the computer the start time that is the time when the analysis of the sequential generation data is started in the analysis cost monitoring process, and when the analysis of the sequential generation data is finished, Based on the end time, which is the time when the analysis is finished, and the temporarily stored start time, the processing for calculating the analysis cost and the processing for storing the calculation result of the analysis cost in the analysis cost storage means are executed. There may be.

(54) Further, the sequential generation data analysis program causes the computer to execute a resource monitoring process for monitoring the free capacity of the resource of the information processing apparatus used for the analysis process of the sequentially generated data. Based on the result, a process for narrowing down the sequentially generated data to be analyzed in advance may be executed.

  While the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  This application claims the priority on the basis of the JP Patent application 2009-118031 for which it applied on May 14, 2009, and takes in those the indications of all here.

  The present invention can be applied to an apparatus that provides users with various services by collecting and analyzing sequentially generated data such as log data output from a server and data output from a sensor. For example, the present invention can be applied to a sequential data high-speed analysis apparatus, method, and program that perform analysis processing in response to a request for response time for each service. The present invention is particularly applicable to an apparatus, a method, and a program that provide a service that requires high speed in which generated data is reflected in an analysis result without time lag.

DESCRIPTION OF SYMBOLS 1 Service provision means 2 Service requirement setting means 3 Analysis data generation means 4 Analysis data storage means 5 Analysis means 6 Analysis command reception means 7 Analysis result storage means 8 Pre-analysis control means 801 Analysis object monitoring means 802 Pre-analysis object generation means 803 Advance Analysis request means 804 Pre-analysis result registration means 805 Priority pre-analysis determination means 806 Free storage capacity monitoring means 9 Analysis command history means 10 Analysis cost monitoring means 1001 Analysis cost calculation means 1002 Analysis cost storage means 101 Application server 102 Analysis server 103 Data analysis Service server 104 Data analysis reception server 105 Data high-speed analysis server

Claims (15)

A sequential data analysis apparatus comprising: a prior analysis control unit that performs control to execute a process of performing prior analysis on sequential generation data that is sequentially generated before an analysis request. When an analysis request occurs, the analysis method determination means for determining whether or not the sequential generation data to be analyzed has already been pre-analyzed by control by the pre-analysis control means,
The analysis method determination means includes
If it is determined that it has not been analyzed in advance, it performs control to analyze the sequentially generated data to be analyzed, and outputs the analysis result,
The sequential generation data analysis device according to claim 1, wherein when it is determined that the analysis has already been performed, the analysis result that has already been analyzed in advance is output. Pre-analysis result storage means for storing a pre-analysis result pre-analyzed by control by the pre-analysis control means,
The analysis method determination means determines whether the analysis target is generated by determining whether or not a preliminary analysis result corresponding to the sequentially generated data to be analyzed is stored in the preliminary analysis result storage means when an analysis request is generated. The sequential generation data analysis apparatus according to claim 2, wherein it is determined whether or not the sequential generation data has been previously analyzed. The pre-analysis control unit is configured to determine a sequential generation data to be pre-analyzed based on the preset pre-analysis condition indicating a condition to be pre-analyzed, and to pre-analyze the sequential generation data The sequential generation data analysis device according to any one of claims 1 to 3, wherein control is performed to execute the control. The pre-analysis control means is
Sequentially generated data monitoring means for monitoring the occurrence status of sequentially generated data;
Based on the monitoring result of the sequential generation data monitoring unit and the preanalysis condition set by the preanalysis condition setting unit, a preanalysis target specifying unit that specifies sequential generation data to be preferentially analyzed, ,
Pre-analysis execution control means for performing control to execute analysis of sequentially generated data of the pre-analysis target specified by the pre-analysis target specifying means;
The pre-analysis result registration means for storing the analysis result of the pre-analysis executed by the control of the pre-analysis execution control means in the pre-analysis result storage means for storing the pre-analysis result. The sequential generation data analysis apparatus according to any one of the above. An analysis request history storage means for storing analysis request history information is provided,
Based on the history information stored in the analysis request history storage unit, the pre-analysis control unit sets the pre-analysis target so that sequential generation data corresponding to an analysis request with a high appearance frequency is preferentially analyzed. The sequential generation data analysis device according to any one of claims 1 to 5, further comprising an appearance frequency priority preliminary analysis determination unit that determines the priority of the sequential generation data. With analysis cost monitoring means to monitor the analysis cost for analyzing the sequentially generated data,
The pre-analysis control unit includes an analysis cost priority pre-analysis determination unit that determines the priority of sequentially generated data to be pre-analyzed based on a monitoring result of the analysis cost monitoring unit. The sequential generation data analysis device according to any one of the above. Analysis cost monitoring means
Temporarily storing a start time, which is a time when analysis of sequentially generated data is started, and when analysis of the sequentially generated data is finished, an end time, which is a time when the analysis is ended, and the temporarily stored start time Analysis cost calculation means for calculating the analysis cost based on
The sequential generation data analysis device according to claim 7, further comprising an analysis cost storage unit that stores a calculation result of the analysis cost calculation unit. Resource monitoring means for monitoring the free capacity of the resources of the information processing device used for analysis processing of sequentially generated data,
The sequential analysis data analysis device according to any one of claims 1 to 8, wherein the preliminary analysis control unit narrows down the sequential generation data to be analyzed in advance based on a monitoring result of the resource monitoring unit. . A data generator for generating sequentially generated data which is sequentially generated data;
A sequential data analysis device for performing an analysis process of the sequential data generated by the data generation means;
A service providing device that executes a process of providing a data analysis service based on the analysis result of the sequential data analysis device;
The sequential generation data analysis system includes a prior analysis control unit that performs control to execute a process of performing preliminary analysis of the sequential generation data before an analysis request. The sequential generation data analysis device includes an analysis method determination unit that determines whether or not the sequential generation data to be analyzed has already been pre-analyzed by control by the pre-analysis control unit when an analysis request is generated,
The analysis method determination means includes
If it is determined that it has not been analyzed in advance, it performs control to analyze the sequentially generated data to be analyzed, and outputs the analysis result,
The sequential generation data analysis system according to claim 10, wherein when it is determined that the analysis is already performed in advance, the analysis result that has been analyzed in advance is output. A sequential data analysis method comprising a pre-analysis control step for performing a control to execute a process of pre-analyzing sequentially generated data that is sequentially generated before an analysis request. An analysis method determination step for determining whether or not the sequential generation data to be analyzed has already been pre-analyzed when an analysis request occurs,
In the analysis method determination step,
If it is determined that it has not been analyzed in advance, it performs control to analyze the sequentially generated data to be analyzed, and outputs the analysis result,
The sequential generation data analysis method according to claim 12, wherein when it is determined that the analysis has been performed in advance, the analysis result that has been analyzed in advance is output. On the computer,
A sequential generation data analysis program for executing a pre-analysis control process for performing a control for executing a process for performing a pre-analysis of sequential generation data that is sequentially generated before an analysis request. On the computer,
When an analysis request occurs, the analysis method determination process is executed to determine whether the sequential generation data to be analyzed has already been analyzed in advance.
In the analysis method determination process,
When it is determined that the analysis is not performed in advance, the process of analyzing the sequential generation data of the analysis target is performed, and the process of outputting the analysis result;
The sequential generation data analysis program according to claim 14, wherein when it is determined that the analysis has been performed in advance, the processing for outputting the analysis result that has already been analyzed in advance is executed.

Images