JPWO2018061070A1 - Computer system and analysis source data management method - Google Patents

Abstract

  The processor of the management computer reads the analysis source data stored in the storage device and stores it in the data storage area, outputs the analysis result of the analysis source data as analysis result data, and reads the analysis source data read. The location is stored in the data location information, the analysis result data and the analysis source data are associated and stored in the analysis result generation source information, and when the analysis source data is deleted from the data storage area, the data location information is updated, and the analysis result data Receiving the data acquisition request including the data, referring to the analysis result generation source information from the analysis result data, specifying analysis source data corresponding to the analysis result data, searching data location information with the specified analysis source data, and analyzing source Identify the location of the data.

Description

  The present invention relates to a data search method in an analysis system.

  With the spread of cloud computing (hereinafter referred to as "cloud"), various analysis services have appeared by aggregating business system data in customer data centers on the cloud. The customer uses the analysis service to create an analysis report, and makes decisions on improvement of the business system. In recent years, analysis services may include analysis from various places such as geographically dispersed data centers, edge computers that collect information of IoT (Internet of Things) devices, and so forth.

  In the analysis service, there are cases where it is desirable to add additional information to the analysis result once generated and to analyze it again. As a specific example, for example, from the customer information (age, gender, occupation, etc.) possessed by the power company, the forecasting analysis of the energy usage was performed, and the rate plan (analysis report X) was presented. Three months after the analysis, the analyst, together with information such as power usage status, customer behavior information, perform forecasting analysis of usage amount, and wants to present a rate plan, and customer information from three months ago In this case, data analysis is performed using (age, gender, occupation, etc.) again.

  In the data warehouse, the relationship between source data (data before analysis) and data after data conversion (analysis result) is held, and source data can be searched from data after data conversion Patent Document 1 discloses a technique for making it.

International Publication No. 2014/062277

  In the analysis service on the cloud, data used for analysis (customer system information and the like) is often deleted from the cloud after analysis is completed due to a contract reason (for example, storage capacity limitation). As described above, in the conventional example, since the information of the already analyzed source data does not exist on the analysis cloud, the source data of the analysis result can not be used, and the same source data as the existing analysis data is used. There was a problem that it was not possible to generate additional reports quickly.

  The present invention is a computer system having a management computer including a processor and a memory, and one or more storage devices connected to the management computer, the processor including analysis source data stored in the storage device. The processor reads and stores it in a predetermined data storage area, and the processor outputs the result of performing a predetermined analysis on the analysis source data of the data storage area as analysis result data, and the processor reads the analysis source read in The location of data is stored in the data location information, and the processor associates the analysis source data used when generating the analysis result data with the analysis result data, and stores the analysis source data in analysis result generation source information, the processor When the location of the used analysis source data is changed or deleted, the data location information is updated. The processor receives a data acquisition request including information of analysis result data, and refers to the analysis result generation source information from the information of the analysis result data, and uses the analysis used when generating the analysis result data. The source data is identified, and the processor searches the data location information in the identified analysis source data to identify the location of the analysis source data.

  According to the present invention, tracing analysis source data from analysis result data enables quick generation of additional analysis reports.

FIG. 5 is a block diagram showing a first embodiment of the present invention and showing an example of processing in a data center. FIG. 2 is a block diagram showing a first embodiment of the present invention and showing an example of a computer system of a data center. FIG. 2 is a block diagram showing an example of an analysis cloud according to the first embodiment of the present invention. FIG. 2 is a block diagram showing an example of a gateway computer according to the first embodiment of this invention. FIG. 7 shows the first embodiment of the present invention, and shows an example of an analysis result generation source table. FIG. 7 shows the first embodiment of the present invention, and shows an example of a data location information table. FIG. 7 shows the first embodiment of the present invention, and is a diagram showing an example of a data table. FIG. 7 shows the first embodiment of the present invention, and is a diagram showing an example of a data acquisition information table. FIG. 7 shows the first embodiment of the present invention, and shows an example of a data replication management table. FIG. 7 shows the first embodiment of the present invention, and shows an example of a data backup management table. FIG. 7 shows the first embodiment of the present invention, and shows an example of an ETL processing information table. It is a flow chart which shows a 1st example of the present invention, and shows an example of analysis source data trace processing. It is a flow chart which shows a 1st example of the present invention, and shows an example of metadata generation and storage processing of analysis source data. It is a flowchart which shows a 1st Example of this invention, and shows an example of a metadata production | generation and storage process of analysis result data. It is a flowchart which shows a 1st Example of this invention, and shows an example of the metadata update process of analysis source data. FIG. 8 is a block diagram showing a second embodiment of the present invention and showing an example of a computer system of a data center.

  Hereinafter, an embodiment of the present invention will be described with reference to the attached drawings.

  Several embodiments will be described with reference to the drawings. The embodiments described below do not limit the invention according to the claims, and all of the elements described in the embodiments and their combinations are essential for the solution means of the invention. There is no limit. In these drawings, the same reference numerals indicate the same components throughout the drawings. In the following description, the information of the present invention will be described by the expression such as "aaa table", but such information may be expressed by other than the data structure such as the table. Therefore, the "aaa table" or the like may be called "aaa information" to indicate that it does not depend on the data structure. Furthermore, when describing the content of each information, although the expressions "identification information", "identifier", "name", and "ID" are used, they can be mutually substituted.

  In the following description, although the description may be made with "program" as the subject, the program may execute processing defined by the processor as a memory and communication port (communication device, management I / F, data I / F) In the description, the processor may be used as the subject. Further, the processing disclosed with the program as the subject may be processing performed by a computer such as a management server or an information processing apparatus. Also, part or all of the program may be realized by dedicated hardware. Also, various programs may be installed on each computer by a program distribution server or a storage medium readable by a computer.

  Hereinafter, a set of one or more computers that manage computer systems and display the display information of the present invention may be referred to as a management system. When the management server displays the display information, the management server is the management system, and the combination of the management server and the display computer is also the management system. In addition, in order to speed up the management processing and increase reliability, processing equivalent to that of the management server may be realized by a plurality of computers, and in this case, the plurality of computers (for display when the display computer performs display) A computer is also a management system.

  A computer system according to the present embodiment will be described.

  FIG. 1 is a schematic view of the present embodiment. Description of each component will be described later. Assume that an environment in which data used in a business system operating on the business server 3000 of the data center 5 is transferred to an analysis cloud 6 after execution of ETL (Extract / Transform / Load) processing, and analysis processing is performed.

  After performing analysis processing, when you want to refer to the data used for analysis in the past, the analysis source data trace program 9150 refers to (1) analysis result generation source table (analysis result generation source information) 9110 for analysis results ( Data (analysis source data) used for the analysis is specified from analysis result data), (2) location of analysis data is specified with reference to data location information table 9120, and (3) data acquisition information table 9140 is referenced. By specifying the data acquisition method, it is possible to acquire data used in analysis in the past. In order to carry out these processes, metadata related to ETL processes and data transfer held on the data center 5 are shared with the analysis cloud 6.

  In addition, data such as information on the IoT device 10 in field 7, for example, information on manufacturing robots in a factory, and information on a camera for photographing the inside of a factory are collected in the gateway computer 20 in field 7 and A configuration may be used in which the data is stored, transferred to the analysis cloud 6, and stored in the data table (data storage area) 9130 by the message broker 30 on the analysis cloud 6. Also in this case, the above-described processes (1) to (3) are the same.

Here, the field 7 indicates, for example, one or more factories that produce machine parts and the like, and includes one or more IoT devices 10, one or more management computers 1000, and a gateway computer 20.

  The computer system according to this embodiment includes one or more data centers 5 including one or more management computer (management apparatus) 1000 shown in FIG. 2 and one including one or more management computer 9000 shown in FIG. It consists of the analysis cloud 6 or one or more fields 7 as the above data center.

  FIG. 2 is a block diagram showing an example of the data center 5 including one or more management computers 1000 in the computer system according to the first embodiment.

  The first embodiment shows an example including one or more business servers 3000, one or more storage apparatuses 2000, and one or more ETL servers 4000 in addition to one or more management computers 1000. All functions described in the first embodiment may be held by the management computer 1000, for example, and the present invention is not limited to the illustrated configuration.

  The storage apparatus 2000, the business server 3000, and the ETL server 4000 are connected to each other via a network 6000 (specifically, a fiber channel or the like) for data communication such as a SAN (Storage Area Network).

  The management computer 1000, the storage system 2000, the work server 3000, and the ETL server 4000 are connected to one another via a management network 5000. In the example shown in FIG. 2, the storage system 2000, the business server 3000, and the ETL server 4000 are mutually connected via the network 6000 for data communication.

  The connection is not limited to direct connection via fiber channel, but may be via network equipment such as one or more fiber channel switches. The connection may be any network for data communication, and may be an IP (Internet Protocol) network. Also, the same network may be used as the management network 5000 as the data communication network.

  The management computer 1000 includes a memory 1100, a communication device 1200, a processor 1300, an output device 1400, an input device 1500, a storage device 1600, and a data I / F 1800, which are mutually connected via the internal bus 1700. Connected

  The memory 1100 stores a configuration information collection program 1110, a metadata sharing program 1120, a data acquisition program 1130, a data replication management table 1140, and a data backup management table 1150.

  The configuration information collection program 1110 includes configuration information from the storage device 2000, the business server 3000, and the ETL server 4000, for example, the related information of the volume 2210 used by the business server 3000 and the business server, and the data replication shown in FIG. It is a program for collecting information of the management table 1140, information of the data backup management table 1150 shown in FIG.

  The metadata sharing program 1120 stores metadata (specifically, information stored in FIG. 9 or 10) corresponding to the data transmitted to the analysis cloud 6 held on the data center 5 with the analysis cloud 6 It is a program to share.

  The data acquisition program 1130 is a program that receives an acquisition request for data required in the analysis cloud 6 from the analysis source data trace program 9150 in the analysis cloud 6 and executes acquisition of the requested data.

  The data replication management table 1140 stores information on data replication from the data center 5 to the analysis cloud 6. The data backup management table 1150 stores information on data backup in the data center 5.

  The communication device 1200 is a device for connecting to the management network 5000. The processor 1300 executes a program deployed on the memory 1100. The output device 1400 is a device that outputs the processing result executed by the management computer 1000, and is, for example, a display or the like. The input device 1500 is a device, such as a keyboard, for the administrator to input an instruction to the management computer 1000. The storage device 1600 is a hard disk drive (HDD) that stores information, a solid state drive (SSD), or the like. The data I / F 1800 is an interface device for connecting to the network 6000 for data communication.

  Although various programs and tables are stored in the memory 1100 in the example shown in FIG. 2, they may be stored in the storage device 1600 or another storage medium (not shown). In this case, the processor 1300 reads the target program on the memory 1100 at the time of program execution, and executes the read program.

  Further, the above-described program and table may be stored in the memory 2100 of the storage device 2000, and the storage device 2000, the business server 3000 and the ETL server 4000 may execute the stored programs. In addition, another business server 3000 or another device such as a switch (not shown) may store the above program and table and execute the stored program.

  The management computer 1000 can communicate with programs operating on the storage system 2000, the business server 3000, and the ETL server 4000 via the management network 5000.

  The storage device 2000 includes a memory 2100, a logical volume providing unit 2200, a disk I / F controller 2300, a management I / F 2400, a processor 2500 and a data I / F 2600, which communicate with the communication path 2700 such as an internal bus. Connected through.

  The memory 2100 has a disk cache 2110. The memory 2100 also stores a replication program 2120 and a backup program 2130. The disk cache 2110 is a storage area for temporarily storing information. The replication program 2120 is a program for performing replication of data stored in the volume 2210 of the storage system 2000 to another storage system 2000.

  The backup program 2130 is a program for backing up data stored in the volume 2210 of the storage system 2000 to another location, for example, another volume 2210. These programs may be periodically executed or may be executed at any timing such as a user request.

  Here, the various programs and tables are stored in the memory 2100, but may be stored in a storage device (2200) or another storage medium (not shown). In this case, the processor 2500 reads the target program on the memory 2100 at the time of program execution, and executes the read program.

  The logical volume provision unit 2200 includes a disk pool 2220 configured by a physical area 2230, logically divides the storage area of the disk pool 2220, and provides the logically divided storage area as a volume 2210. Here, the physical area 2230 is, for example, a physical disk or a parity group configured of a plurality of physical disks.

  A device outside the storage device 2000 can access the physical area 2230 via the volume 2210. A physical area number is assigned to the physical area 2230, a disk pool number is assigned to the disk pool 2220, and a volume number is assigned to the volume 2210.

  Thus, the storage system 2000 can uniquely identify the physical area 2230, the disk pool 2220, and the logical volume 2210. In the example shown in FIG. 2, the disk pool 2220 (POOL1) configured from one physical area (parity group PG1) is logically divided, and one volume 2210 (Vol1) is a device outside the storage apparatus 2000 (for example, , Business server 3000).

  The disk I / F controller 2300 is an interface device for connecting to the volume providing unit 2200. The management I / F 2400 is an interface device for connecting to the management network 5000. The processor 2500 executes a program expanded on the memory 2100. The data I / F 2600 is an interface device for connecting to the network 6000 for data communication.

  Also, the logical volume provision unit 2200 may create the entire storage area of one disk pool 2220 as one logical volume 2210. Further, the logical volume providing unit 2200 may be a physical disk 2230 other than a parity group, for example, a physical disk itself or a storage medium such as a flash memory.

  The work server 3000 includes a memory 3100, a data I / F 3200, a processor 3300, and a management I / F 3400, which are connected to one another via a communication path 3500 such as an internal bus. The memory 3100 stores a business program 3110.

  The business program 3110 is a program for realizing the business provided by the business server 3000, and is, for example, a DBMS (Data Base Management System) or a file system. The business server 3000 stores business data in the logical volume 2210 provided from the storage system 2000 and provides various business.

  Although various programs are stored in the memory 3100 in the example shown in FIG. 2, they may be stored in another storage device (not shown). In this case, the processor 3300 reads a target program on the memory 3100 at the time of processing execution, and executes the read program.

  The data I / F 3200 is an interface device for connecting to the network 6000 for data communication. The processor 3300 executes a program developed on the memory 3100. The management I / F 3400 is an interface device for connecting to the management network 5000.

  The ETL server 4000 includes a memory 4100, a data I / F 4200, a processor 4300, and a management I / F 4400, which are connected to one another via a communication path 4500 such as an internal bus. The memory 4100 stores an ETL program 4110 and an ETL processing information table 4120.

  The ETL program 4110 is a program for realizing the ETL process provided by the ETL server 4000. For example, the business data stored in the storage device 2000 or the like is selected, processed into a format easy to analyze, and processed. A series of processing is performed to write data in a storage device for transferring data to the analysis cloud 6. Specifically, processing such as data matching, duplicate deletion, aggregation, sorting, code conversion, cleansing, table creation, and insertion into a table is performed.

  For example, the ETL program 4110 mutually converts a format based on well-known or publicly known formats such as CSV, XML, JSON, ZIP, HTML, and defined table schema information such as Oracle, DB2, Microsoft SQL, Postgre SQL, etc. Information necessary for data selection, processing, analysis, and writing, etc., and ETL processing is performed using these.

  The ETL process information table 4120 holds execution history information of the ETL process executed by the ETL program 4110. Although various programs are stored in the memory 4100 in the example shown in FIG. 2, they may be stored in another storage device (not shown). In this case, the processor 4300 reads a target program on the memory 4100 at the time of processing execution, and executes the read program.

  The data I / F 4200 is an interface device for connecting to the network 6000 for data communication. The processor 4300 executes a program expanded on the memory 4100. The management I / F 4400 is an interface device for connecting to the management network 5000.

  FIG. 3 is a block diagram showing an example of an analysis cloud 6 including one or more management computers 9000 in the computer system according to the first embodiment.

  In the first embodiment, an example including one or more storage devices (storage devices) 2000 and a message broker 30 in addition to one or more management computers is shown. The management computer 9000 may hold all the functions described in the first embodiment, and is not limited to the configuration of FIG.

  The management computer 9000 and the storage system 2000 are connected to each other via a network 6000 for data communication. Any network may be used as the network 6000 for data communication, and for example, an IP (Internet Protocol) network is used.

  The management computer 9000 includes a memory 9100, a communication device 9200, a processor 9300, an output device 9400, an input device 9500, a storage device 9600, and a data I / F 9800, which are mutually connected via an internal bus 9700. Be done.

  The memory 9100 includes an analysis result generation source table 9110, a data location information table 9120, a data table 9130, a data acquisition information table 9140, an analysis source data tracing program 9150, a metadata management program 9160, and an analysis program 9170. Including.

  The analysis result generation source table 9110 stores data of the analysis result in the analysis cloud 6 and related information of analysis source data used for analysis. The data location information table 9120 stores information indicating the location of data at each time. The data table 9130 stores information representing data values. The data acquisition information table 9140 stores information on processing required to acquire data. The analysis source data trace program 9150 is a program for specifying the location of analysis source data used for analysis from analysis results and acquiring the data. The metadata management program 9160 stores meta information on data, such as location information on data. The analysis program 9170 is a program that performs analysis based on analysis source data.

  The communication device 9200 is a device for connecting to the management network 5000. The processor 9300 executes a program loaded on the memory 9100. The output device 9400 is a device that outputs the processing result executed by the management computer 9000, such as a display. The input device 9500 is a device, such as a keyboard, for the administrator to input an instruction to the management computer 9000.

  The storage device 9600 is a hard disk drive (HDD) that stores information, a solid state drive (SSD), or the like. The data I / F 9800 is an interface device for connecting to the network 6000 for data communication.

  Although various programs and tables are stored in the memory 9100 in the example shown in FIG. 3, they may be stored in the storage device 9600 or another storage medium (not shown). In this case, the processor 9300 reads a target program on the memory 9100 at the time of program execution, and executes the read program.

  Further, the above-described program and table may be stored in the memory 2100 of the storage device 2000, and the storage device 2000 may execute the stored program. Also, another device such as a server or a switch (not shown) may store the aforementioned program and table and execute the stored program. The management computer 9000 can communicate with a program operating on the storage system 2000 via the management network 5000.

  Here, the data center 5 and the analysis cloud 6 are connected to each other via a gateway (not shown) or the like for communication via the management network 5000. The data center 5 is a facility for storing customer's business data and executing the customer's business program. The data center 5 and the analytic cloud 6 are often geographically dispersed with each other. The management network 5000 connecting between the data center 5 and the analysis cloud 6 may be any network type such as a wide area network (WAN) or a local area network (LAN).

  Gateways arranged in the data center 5 and the analysis cloud 6 are used for communication between the data center 5 and the analysis cloud 6 with a network protocol used in the data center 5 and in the analysis cloud 6. Conversion processing with the network protocol.

  In addition, the data center 5 and the analysis cloud 6 are connected to each other for data communication via the network 6000 for data communication, and the network 6000 for data communication is any network type such as SAN or IP. The same network as the management network 5000 may be used.

  Further, in the present embodiment, the storage apparatus 2000 is prepared, and remote replication between the data center 5 and the analysis cloud 6 is performed using the replication program 2120 which is a function held by the storage. For example, the data stored in the storage device on the business server 3000 is transmitted to the analysis cloud 6 using a remote copy program (not shown) operating on the business server 3000, for example. It is good.

  The message broker 30 receives data from the field 7, performs format conversion of the data in accordance with the table definition, and stores the data in the data table 9130 of the management computer 9000. Although an example in which the message broker 30 is prepared separately from the management computer 9000 has been described here, the function of the message broker 30 may be provided by the management computer 9000 and is not limited to the illustrated example.

  FIG. 4 is a block diagram showing an example of the field 7 in the computer system according to the first embodiment. The first embodiment shows an example including one or more management computers 9000, one or more IoT devices 10, and one or more gateway computers 20. However, the management computer 9000 and the gateway computer described in the first embodiment will be described. The twenty functions may be provided by either a management computer or a gateway computer, and are not limited to the illustrated example.

  The management computer 9000, the IoT device 10, and the gateway computer 20 are connected to one another via a management network 5000. As the management network 5000, for example, an IP (Internet Protocol) network can be used.

  The management computer 9000 is the same as that shown in FIG. The gateway computer 20 collects data of the IoT device 10, such as sensor data, and provides means for transmitting the data to the message broker 30 of the analysis cloud 6.

  The gateway computer 20 includes a memory 21, a communication device 22, a processor 23, an output device 24, an input device 25 and a storage device 26, which are connected to one another via an internal bus 27. The memory 21 stores a collection definition table 211 and a gateway program 212.

  The collection definition table 211 is a table which is referred to by the gateway program 212, from which sensors of the IoT device 10 collect sensor data, and to which analysis cloud 6 to transmit the collected sensor data (not shown) ).

  The gateway program 212 collects data from the IoT device 10 based on the collection definition information stored in the collection definition table 211, and transmits the data to the message broker 30 of the analysis cloud 6.

  The communication device 22, the processor 23, the output device 24, the input device 25, the storage device 26, and the internal bus 27 have the same configuration as the management computer 9000, and thus the description thereof is omitted.

  FIG. 5 is a diagram showing an example of the analysis result generation source table 9110 according to the first embodiment. The analysis result generation source table 9110 manages information indicating correspondence between data of analysis results executed on the analysis cloud 6 and analysis source data used for the analysis.

  The analysis result generation source table 9110 includes the fields of analysis result data 9111, analysis source data 9112, and source data date 9113 in one entry.

  The analysis result data 9111 stores an identifier indicating data of an analysis result executed on the analysis cloud 6. The analysis source data 9112 stores an identifier indicating data used when generating the analysis result data 9111.

  The source data date 9113 stores a value (for example, a generated date) indicating time information of the analysis source data 9112 used when generating the analysis result data 9111. In the example shown in FIG. 5, analysis result data using the information of data A of the period 2016/06 / 01-06 / 14 and the information of data B of the period 2016/06 / 01-06 / 14 Indicates that X was generated.

  FIG. 6 is a diagram showing an example of the data location information table 9120 according to the first embodiment. The data location information table 9120 manages information indicating the location of data at the current time together with date and time information.

  The data location information table 9120 includes fields of a data ID 9121, an analysis result flag 9122, a date and time 9123, and a data location 9124 in one entry.

  The data ID 9121 stores an identifier for specifying data. The analysis result flag 9122 stores an identifier indicating whether the data specified by the data ID 9121 is analysis result data or analysis source data used when generating an analysis result. Specifically, in the case of an analysis result, "True" is stored as an analysis result flag, and in the case of analysis source data, "False" is stored as an analysis result flag.

  The date and time 9123 stores information indicating the acquisition (or generation) date and time of the data indicated by the data ID 9121. The data location 9124 stores an identifier indicating the location of the data indicated by the data ID 9121 at the date and time indicated by the date and time 9123.

  In the first line of the example shown in FIG. 6, the value of the period of date and time 2016/06 / 15-06 / 30 of the data A, which is analysis source data, is stored in “Local”, ie, the analysis cloud 6, 2016 / The value of the period 01 / 01-06 / 30 is stored in the data center 5 indicated by "Datacenter1".

  That is, the data in the first row represents that the data of the period 2016/01 / 01-06 / 14 has been deleted from the analysis cloud 6. In the third line of the example shown in FIG. 6, the value of the period of date and time 2016/01 / 01-06 / 30 of the data C which is the analysis source data is "Local", that is, the analysis cloud 6, " It indicates that it is stored in both of the fields 7 indicated by Field 1 ", for example, the factory.

  FIG. 7 is a diagram showing an example of the data table 9130 according to the embodiment. The data table 9130 manages data transmitted from the data center 5 to the analysis cloud 6.

  The data table 9130 includes fields of data ID 9131, date and time 9132, and value 9133 in one entry.

  The data ID 9131 stores an identifier for specifying data. The date and time 9132 stores information indicating the date of acquisition (or generation) of data. The Value 9133 stores the value at the date 9132 of the data represented by the data ID 9131. In the example shown in FIG. 7, the value of the date and time "2016/06/15 10:00" of data A is 1800, the value of date and time "2016/06/15 11:00" is 2000, and date and time "2016/06/15 The value of 12:00 "represents 3000.

  In the first embodiment, an example in which Value is a simple numerical value is shown to simplify the description, but the present invention is not limited to this. For example, the value is represented by a combination of numerical values and character strings It may be any form of value, such as a value or a value composed of a set of Key-Value.

  FIG. 8 is a diagram showing an example of the data acquisition information table 9140 according to the first embodiment. The data acquisition information table 9140 manages an acquisition method of data existing in the data center 5. The data acquisition information table 9140 includes fields of a data ID 9141, a data location 9142, and a data acquisition command 9143 in one entry.

  The data ID 9141 stores an identifier for specifying data. The data location 9142 stores information indicating the location of data. The data acquisition command 9143 stores commands necessary for acquiring the data specified by the data ID 9141.

  The first line of the example shown in FIG. 8 indicates that it is necessary to execute a command “Get contract (A)” in order to obtain data A from “DataCenter 1”.

  Also, in the example of the second line, in order to obtain data B from "DataCenter 2", it is represented by the identifier 120 at the designated time point by executing the "Restore backup volume 120" command with designating the time information. Restore the data from the volume backup, get the data restored to the volume represented by the identifier 120 'with the command "Get volume (120')", and use the command "ETL from Oracle to Postgres" with the volume 120 ' Data from Oracle format to Postgres format by ETL program, and execute data transfer from "DataCenter 2" to Cloud for analysis 6 using "Set remote replication" command. It represents the fact that there is a necessity.

  In the first embodiment, the data specified by the data ID 9141 is an example existing at only the maximum two places of the originally existing data center 5 and the analysis cloud 6, and one data is specified for each data specified by the data ID 9141. An example is shown in which only one data location is shown, but if the location of the data indicated by the data ID 9141 changes depending on the time, a time information column is added so that the data location for each time and data acquisition command can be indicated. Also good.

  FIG. 9 is a diagram showing an example of the data replication management table 1140 according to the first embodiment.

  The data replication management table 1140 includes fields of copy source location ID 1141, copy source storage ID 1142, copy source volume ID 1143, copy destination location ID 1144, copy destination storage ID 1145, copy destination volume ID 1146, and last update time 1147. Include in one entry.

  The copy source location ID 1141 stores an identifier of the data center 5 when transmitting data from the data center 5 to the analysis cloud 6. The copy source storage ID 1142 stores an identifier of a storage that is a data storage destination in the data center 5 when data is transmitted from the data center 5 to the analysis cloud 6.

  The copy source volume ID 1143 stores an identifier of a volume of a storage that is a data storage destination in the data center 5 when data is transmitted from the data center 5 to the analysis cloud 6. The copy destination location ID 1144 stores an identifier of the analysis cloud 6 when data is transmitted from the data center 5 to the analysis cloud 6.

  The copy destination storage ID 1145 stores an identifier of a storage that is a data storage destination in the data center 5 when data is transmitted from the data center 5 to the analysis cloud 6. The copy destination volume ID 1146 stores an identifier of a storage volume which is a data storage destination in the analysis cloud 6 when data is transmitted from the data center 5 to the analysis cloud 6. The last update time 1147 stores the last time data was transferred from the data center 5 to the analysis cloud 6.

  In the first embodiment, an example of remote replication in units of Volumes has been described. However, the present invention is not limited to this, and remote replication in units of files or tables may be used.

  FIG. 10 is a configuration diagram showing an example of the data backup management table 1150 according to the first embodiment. The data backup management table 1150 includes fields of storage ID 1151, volume ID 1152, backup destination storage ID 1153, backup destination volume ID 1154 and backup date 1155 in one entry.

  The storage ID 1151 stores an identifier that uniquely identifies the storage. The volume ID 1152 stores an identifier for uniquely identifying the volume in the storage ID 1151. The backup destination storage ID 1153 stores an identifier that uniquely identifies the storage of the backup destination. The backup destination volume ID 1154 stores an identifier for uniquely identifying the backup destination volume. The backup date 1155 stores information on the date and time when the backup was performed.

  FIG. 11 is a block diagram of an example of the ETL processing information table 1160 according to the first embodiment.

  The ETL processing information table 1160 is used by the business program 3110 on the business server 3000 executed by the ETL program 4110 of the ETL server 4000 to select data stored in the storage device 2000 etc. and process it into a format easy to analyze Manage information on a series of processes of writing data in a storage device for transferring data to the analysis cloud 6.

  The ETL processing information table 1160 includes the fields of execution time 1801, operation 1802, source 1803, and target 1804 in one entry.

  The execution time 1801 stores information indicating the time when the ETL process was performed. Operation 1802 stores information indicating processing executed by the ETL program.

  The Source 1803 stores the volume identifier of the storage in which the input data to the process shown in Operation 1802 is stored. The Target 1804 stores the volume identifier of the storage that stores the output data of the process shown in Operation 1802.

  In FIG. 11, the ETL server 4000 starts ETL processing at 10:00 on the 1st of June, reads data of Volume 10 of Storage 1, generates SQL request from the read data, and schema of data format for transfer. An example is shown in which a table is generated on the volume 3 of Storage 1 based on the information, and data is inserted into the generated table using the generated SQL request.

  Next, each process executed by the management computer 9000 will be described.

  FIG. 12 is a flowchart 100 of processing of tracing data (analysis source data) of a generation source of an analysis result according to the first embodiment.

  The process of tracing data is a process of specifying and recording the location of data. This process is performed by the processor 9300 of the management computer 9000 executing the analysis source data trace program 9150 developed on the memory 9100. Hereafter, the specific example of this process is shown.

  First, the analysis source data trace program 9150 receives a request for acquiring data used as a source of existing analysis results (step 101). The data acquisition request may be received from the input device 9500 in addition to being received from an external computer via the management network 5000. For example, when the user inputs a request to create a report in which another data is added to the existing analysis result via the input device 9500 of the management computer 9000, the request is received. As a specific example, when the user adds data C to the analysis result X to create a report, the management computer 9000 requests acquisition of data (analysis source data) used as a source of the analysis result X To receive. The acquisition request includes the identifier of the data of the analysis result.

  Next, the analysis source data trace program 9150 refers to the analysis result generation source table 9110 shown in FIG. 5, and specifies the analysis source data used as the source of the existing analysis result received in step 101 (step 102). . For example, the data used as a source of analysis result X is the information of 2016/6/1 to 2016/6/14 of data A and the information of 2016/6/1 to 2016/6/14 of data B. Identify that.

  Next, the analysis source data trace program 9150 refers to the data location information table 9120 shown in FIG. 6, and identifies the storage location of the data identified in step 102 (step 103). For example, in order to specify the location of 2016/6/1 to 2016/6/14 of data A and the location of 2016/6/1 to 2016/6/14 of data B, the analysis source data trace program 9150 It refers to the data location information table 9120, and specifies that the information of the corresponding time of the data A exists in "Datacenter1" and the information of the corresponding time of the data B exists in the analysis cloud 6 and Datacenter2.

  In the first embodiment, an example has been described in which data having the same time as the source data for which the acquisition request has been received in step 101 exists, but it is not necessary to completely match depending on the content of analysis. Alternatively, information on the closest time at which the source data exists may be substituted.

  Then, the analysis source data trace program 9150 determines whether or not the corresponding data exists in the analysis cloud 6 (step 104), and if the corresponding data exists in the analysis cloud 6, the data of the management computer 9000 Information is acquired from the table 9130, and the process is ended (step 105).

  When there is no corresponding data in the analysis cloud 6, the analysis source data trace program 9150 refers to the data acquisition information table 9140 shown in FIG. 8 and compares the data acquisition program 1130 of the management computer 1000 with the corresponding time. It requests acquisition of information (step 106).

  Here, as in the example shown in FIG. 8, the analysis source data trace program 9150 converts the specific operation content, for example, the format of the database for restoring the backup volume at the corresponding time from Oracle to Postgres, or Designate a remote copy to the analysis cloud 6, and the like, and make an information acquisition request at the corresponding time.

  Alternatively, the analysis source data trace program 9150 issues an information acquisition request for the corresponding time by issuing only the data acquisition command 9143 to the data acquisition program 1130 of the management computer 1000 of the data storage location specified in step 103, Specific operation content may be determined on the management computer 1000 side.

  In the first embodiment, an example of a case where data backup is obtained and ETL processing is performed is described. However, the present invention is not limited thereto, and a case where data being processed instead of backup is acquired or ETL processing is performed. There is also a case where information is sent to the analysis cloud 6 without being processed.

  Here, if the corresponding data is old information, aggregation processing is performed in the data center 5, and there is no case where information of granularity in units of one day remains, and there is also a case where there is only aggregated information such as one week. In this case, the management computer 1000 of the data center 5 may return aggregation information for a predetermined period including the designated time. In addition, data transmission from the data center 5 to the analysis cloud 6 is performed using the network 6000 for data communication.

  By the above process, the management computer 9000 of the analysis cloud 6 can specify the location of the requested data, and acquire the corresponding data from the data center 5 or the analysis cloud 6.

  FIG. 13 is a flowchart of processing of generating and storing metadata of analysis source data at the time of data collection to the analysis cloud 6 according to the embodiment. Here, metadata corresponds to location information of data, information on a method of acquiring data, etc., which are necessary when tracing data. Step 203 and subsequent steps of this process are performed by the processor 9300 of the management computer 9000 executing the metadata management program 9160 developed on the memory 9100. Hereinafter, a specific example of this flowchart will be shown.

  First, data transmission from the data center 5 to the analysis cloud 6 is performed. This may be performed by a known or known method. For example, in the first embodiment, the remote copy function of the storage system 2000 is used to copy data from the storage system 2000 of the data center 5 to the storage system 2000 of the analysis cloud 6 An example of using the function to In addition, it is not limited to this, For example, you may utilize the backup software etc. which operate | move with the management computer 1000. FIG.

  The metadata sharing program 1120 of the management computer 1000 of the data center 5 detects that the data has been transmitted (step 201), and transmits information about the data to the metadata management program 9160 of the management computer 9000 of the analysis cloud 6 (Step 202).

  Specifically, the data ID of the data transmitted to the analysis cloud 6, the transmission date and time, the ID of the transmission source data center 5, and the processing content of the data in the transmission source data center 5 are transmitted. The metadata sharing program 1120 of the management computer 1000 includes information about transmission of data from the storage apparatus 2000 in the data center 5, the business server 3000, the ETL server 4000, etc. to the analysis cloud 6, and the ETL server 4000. It is assumed that information such as data processing content is periodically collected and held.

  Next, the metadata management program 9160 of the management computer 9000 of the analysis cloud 6 receives the data (step 203), and among the received information, the data ID, the transmission date and time, and the ID of the transmission source data center 5 Based on this, the data location information table 9120 shown in FIG. 6 is updated (step 204).

  Subsequently, the metadata management program 9160 updates the data acquisition information table 9140 shown in FIG. 8 based on the processing content of the transmission data in the transmission source data center 5 among the received information (step 205). That is, the metadata management program 9160 acquires a command or the like for acquiring the data based on the processing content of the received data, and stores it in the data acquisition command 9143 of the data acquisition information table 9140.

  The data acquisition command 9143 stores a command according to the data storage mode. For example, when the data is backed up as backup data, a restore command is stored, and when converting the data format, the conversion destination of the data format is specified. In other words, the data acquisition command 9143 stores an acquisition method corresponding to the storage format of the current data.

  In the first embodiment, in step 201, the metadata sharing program 1120 of the management computer 1000 of the data center 5 starts processing by detecting that data has been transmitted to the analysis cloud 6. For example, the metadata management program 9160 of the analysis cloud 6 detects that data has been received, and requests the metadata sharing program 1120 to transmit information on received data. Or the like, or meta information may be periodically shared. In addition, the metadata sharing program 1120 may store metadata in the volume 2210 of the storage device 2000, and transmit information via the network 6000 for data communication using the function of the replication program 2120.

  When data is transmitted from the data center 5 to the analysis cloud 6 by the above processing, the information of the data is transmitted to the analysis cloud 6, and the management computer 9000 stores the data location information table 9120 and the data acquisition information table 9140. It can be updated.

  FIG. 14 is a flowchart of processing of updating metadata after execution of analysis processing in the analysis cloud 6 according to the first embodiment. This process is performed by the processor 9300 of the management computer 9000 executing the metadata management program 9160 developed on the memory 9100. Hereinafter, a specific example of this flowchart will be shown.

  First, analysis processing is performed using the information collected from the data center 5. This may be performed by a known or known method. For example, the user refers to the data displayed on the output device 9400 and considers which data to use during which period to analyze, and then the input device By performing the analysis execution operation via 9500, the analysis program 9170 executes the analysis processing, and the analysis result can be referenced in the form of a report or the like via the output device 9400.

  The metadata management program 9160 detects that the analysis process is executed by the analysis program 9170 and the analysis result is stored in the data table 9130 (step 301). The metadata management program 9160 stores the correspondence between the data of the analysis result and the analysis source data used for the analysis in the analysis result generation source table 9110 (step 302). Next, the metadata management program 9160 stores generation time information and location information of analysis result data in the data location information table 9120 (step 303).

  By the above processing, every time analysis is completed, the relationship between the data of the analysis result and the analysis source data used for analysis is stored in the analysis result generation source table 9110, and the generation time information and location information of the data of the analysis result are data locations It is stored in the information table 9120 to generate information including the location of each data.

  FIG. 15 is a flowchart of processing of updating metadata after deleting analysis source data from the analysis cloud 6 according to the first embodiment. This process can be executed at a predetermined timing (for example, a predetermined cycle) other than after deletion of analysis source data.

  On the analysis cloud 6, analysis source data (such as customer system information) used for analysis is often deleted from the cloud after analysis is completed due to contract reasons, capacity reasons, etc. Implement the process assuming the following cases.

  First, the analysis program 9170 deletes, from the data table 9130, data whose use has been completed with analysis source data used for analysis. The process is periodically executed by the analysis program 9170, automatically executed after the execution of the analysis process, or deleted by any method such as the user executing a deletion request via the input device 9500. You may.

  When data is deleted from the data table 9130 (step 401), the metadata corresponding to the data deleted by the analysis program 9170 is deleted. Specifically, the analysis program 9170 determines whether or not the information at all times of the data has been deleted (step 402). If the information is deleted, the corresponding data is local from the data location information table 9120. The entry indicating that it is (on the analysis cloud 6) is deleted (step 403).

  When only information of a part of time (or period) of the data is deleted, the analysis program 9170 indicates that the corresponding data in the data location information table 9120 is local (on the analysis cloud 6). The information of the entry date 9123 is updated (step 404).

  Even after the execution of this process, even if the data does not exist on the analysis cloud 6, its metadata information, specifically the data location information table 9120, includes the information of the data center 5 of the data collection source, etc. Is one of the features of the present invention.

  As described above, according to the first embodiment, even if analysis source data is deleted from the analysis cloud 6, data before analysis can be traced from the analysis result. In the present embodiment, for example, the data is a block. From the viewpoint of a customer using a computer system, data can be accessed without being aware of the position of the data. This makes it possible to create additional analysis reports easily and quickly.

  The computer system according to the second embodiment includes one or more management computers 1000 shown in FIG. 16, one or more business servers 3000, one or more edge servers 7000, and one or more file servers 8000. It comprises an analysis cloud 6 as one or more data centers including one or more data centers 5, one or more management computers 9000, and one or more edge servers 7000.

  Here, the management computer 1000 in the data center 5, the business server 3000, and the management computer 9000 in the analysis cloud 6 have the same configurations as in the first embodiment, and thus the description thereof is omitted.

  The respective components are connected to one another via a data communication network 6000 (specifically, IP etc.) and are also connected to one another via a management network 5000. The connection is not limited to direct connection, and may be connected via a network device such as one or more switches. Also, the same network may be used as a data communication network and a management network.

  The file server program 7110 of the edge server 7000 is a program (for example, an NFS server program) for providing a file sharing service to the business server 3000 in response to an input / output request (I / O request) from the business server 3000.

  The file sharing program 7120 of the analysis cloud 6 configures a pseudo file system (for example, a virtual file system) by the file system (not shown) of the file server 8000 and the file system of the edge server 7000 and transparently transmits the file. It is a movable program.

  The file server program 8110 of the file server 8000 of the data center 5 is a program (for example, an NFS server program) that provides a file sharing service. The file sharing program 8120 is similar to the file sharing program 7120 of the edge server 7000, and a pseudo file system (for example, a virtual file system) is configured by the file system (not shown) of the file server 8000 and the file system of the edge server 7000. It is a program that configures and allows files to be moved transparently. The update information to each edge server 7000 is reflected to the file server 8000, and the update information to the file server 8000 is reflected to each edge server 7000.

  In the second embodiment, the writing process to the edge server 7000 by the business server 3000 of the data center 5 is reflected on the file server 8000, and the data reflected on the file server 8000 is transferred to the edge server 7000 of the analysis cloud 6. Ru.

  If there is no information on the acquisition request data in the edge server 7000 of the analysis cloud 6, as in the first embodiment, the management computer 9000 generates an analysis result generation source table 9110, an analysis data location information table 9120, and data. Change the settings to synchronize the file server 8000 with data (for example, log) of backup at predetermined time or its equivalent of data in the business server 3000 and edge server 7000 of the data center 5 with reference to the acquired information table 9140 Do.

  Then, the information of the backup is reflected in the file server 8000, and the data reflected in the file server 8000 is transferred to the edge server 7000 of the analysis cloud 6, so that the data can be referred to in the analysis cloud 6 Become. In the second embodiment, an example in which there is no processing or the like of the ETL program 4110 in the first embodiment is shown.

  According to the above-described second embodiment, even if analysis source data is deleted from the analysis cloud 6, data before analysis can be traced from the data of the analysis result. In the second embodiment, for example, the data is a file. From the viewpoint of the customer using the computer system of the second embodiment, data can be accessed without being aware of the position of the data. This allows for the rapid creation of additional analysis reports.

  The present invention is not limited to the embodiments described above, but includes various modifications. For example, the embodiments described above are described in detail in order to illustrate the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. In addition, addition, deletion, or replacement of other configurations may be applied singly or in combination with some of the configurations of the respective embodiments.

  Further, each of the configurations, functions, processing units, processing means, and the like described above may be realized by hardware, for example, by designing part or all of them with an integrated circuit. In addition, each configuration, function, and the like described above may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as a program, a table, and a file for realizing each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

  Further, control lines and information lines indicate what is considered to be necessary for the description, and not all control lines and information lines in the product are necessarily shown. In practice, almost all configurations may be considered to be mutually connected.

Claims (14)

Management computer including processor and memory,
A computer system having one or more storage devices connected to the management computer;
The processor reads analysis source data stored in the storage device and stores the analysis source data in a predetermined data storage area.
The processor outputs a result of performing a predetermined analysis on analysis source data of the data storage area as analysis result data,
The processor stores the location of the read analysis source data in data location information;
The processor associates the analysis source data used when generating the analysis result data with the analysis result data, and stores the analysis source data in analysis result generation source information.
The processor updates the data location information when the location of the used analysis source data is changed or deleted.
The processor receives a data acquisition request including information of analysis result data, refers to the analysis result generation source information from the information of the analysis result data, and generates analysis source data used when generating the analysis result data To identify
The computer system, wherein the processor searches the data location information with the identified analysis source data, and identifies a location of the analysis source data. The computer system according to claim 1, wherein
The computer system, wherein the processor reads the analysis source data from the storage device or the data storage area based on the identified location. The computer system according to claim 1, wherein
When the analysis result data is held in the management computer, the processor holds the location of the analysis source data in the data location information even if the analysis source data is deleted from the data storage area. A computer system characterized by The computer system according to claim 1, wherein
The data location information includes information on the location, identifier and date and time of the analysis source data,
The analysis result generation source information includes analysis source data identifier and date and time information,
Identification of analysis source data used when generating the analysis result data is
The identifier and date of analysis source data used when generating the analysis result data are specified by referring to the analysis result generation source information from the information of the analysis result data, and the identifier and date of the specified analysis source data Searching the data location information to identify the location of the analysis source data. The computer system according to claim 2, wherein
The processor stores information on acquisition when reading the analysis source data in data acquisition information.
When the specified location is other than the data storage area, the data acquisition information is acquired, applied to the storage device, and the analysis source data is read. The computer system according to claim 2, wherein
The storage device is connected to a management device that manages backup of the analysis source data;
When the location of the analysis source data is a backup, the processor requests the analysis source data from the management device, and the management device restores the analysis source data by restoration and sends it to the management computer. Characteristic computer system. The computer system according to claim 2, wherein
The storage device is connected to a management device that manages consolidated data of the analysis source data;
When the location of the analysis source data is aggregated data, the processor requests the management device for the analysis source data, and the management device transmits the aggregated data as the analysis source data to the management computer. Characteristic computer system. An analysis source data management method for managing analysis source data stored in one or more storage devices connected to a management computer including a processor and a memory, comprising:
A first step of the management computer reading analysis source data stored in the storage device and storing the analysis source data in a predetermined data storage area;
A second step of the management computer outputting, as analysis result data, a result of performing a predetermined analysis on analysis source data of the data storage area;
A third step of the management computer storing the location of the read analysis source data in data location information;
A fourth step of the management computer storing the analysis source data used when generating the analysis result data in association with the analysis result data in analysis result generation source information;
A fifth step of updating the data location information when the location of the used analysis source data is changed or deleted by the management computer;
The management computer receives a data acquisition request including information on analysis result data, refers to the analysis result generation source information from the information on the analysis result data, and generates an analysis source used when generating the analysis result data A sixth step of identifying data;
A seventh step of the management computer searching the data location information in the identified analysis source data to specify the location of the analysis source data;
An analysis source data management method characterized in that it comprises: The analysis source data management method according to claim 8, wherein
The analysis source data management method according to claim 8, further comprising an eighth step of reading the analysis source data from the storage device or the data storage area based on the identified location. The analysis source data management method according to claim 8, wherein
The fifth step is
When the analysis result data is held in the management computer, the location of the analysis source data is held in the data location information even if the analysis source data is deleted from the data storage area. Analysis source data management method. The analysis source data management method according to claim 8, wherein
The data location information includes information on the location, identifier and date and time of the analysis source data,
The analysis result generation source information includes analysis source data identifier and date and time information,
The sixth step is
The identifier and date of analysis source data used when generating the analysis result data are specified by referring to the analysis result generation source information from the information of the analysis result data, and the identifier and date of the specified analysis source data A method of managing the analysis source data, wherein the data location information is searched in step (b) to identify the location of the analysis source data. The analysis source data management method according to claim 9, wherein
The third step is
Storing information on acquisition when reading the analysis source data in data acquisition information;
The eighth step is
The analysis source data management method characterized in that, when the specified location is other than the data storage area, the data acquisition information is acquired and applied to the storage device to read the analysis source data. The analysis source data management method according to claim 9, wherein
The eighth step is
When the location of the analysis source data is a backup, the storage device is connected to request a management device that manages backup of the analysis source data to request the analysis source data, and the management device restores the analysis source data by restoration. A method of managing an analysis source data, comprising: restoring and transmitting to the management computer. The analysis source data management method according to claim 9, wherein
The eighth step is
When the location of the analysis source data is aggregated data, the storage device is connected to request the management source data to manage the aggregation data of the analysis source data, and the management device transmits the aggregated data. And transmitting the analysis source data to the management computer.

Images