CN115268931A

CN115268931A - Data acquisition system, method, device and medium based on cloud native scene

Info

Publication number: CN115268931A
Application number: CN202210912096.3A
Authority: CN
Inventors: 赖玮; 杨杰
Original assignee: Shanghai Hard Link Network Technology Co ltd
Current assignee: Shanghai Hard Link Network Technology Co ltd
Priority date: 2022-07-29
Filing date: 2022-07-29
Publication date: 2022-11-01

Abstract

The application is applicable to the technical field of computers, and discloses a data acquisition system, a method, equipment and a medium based on a cloud native scene, wherein the system comprises a management background, a plurality of acquisition software and a communication configuration service platform; setting and managing visual acquisition software by using a management background, and automatically releasing acquisition task configuration information according to the acquired acquisition condition information; monitoring the management background by using a communication configuration service platform to acquire acquisition task configuration information; and monitoring the communication configuration service platform by using acquisition software to acquire acquisition task configuration information so as to execute a data acquisition task according to the acquisition task configuration information. The visual configuration of the acquisition conditions is realized through the management background, and the simple and efficient acquisition software development and data acquisition are realized through the information interaction among the management background, the plurality of acquisition software and the communication configuration service platform.

Description

Data acquisition system, method, device and medium based on cloud native scene

Technical Field

The application relates to the technical field of computers, in particular to a data acquisition system, method, equipment and medium based on a cloud native scene.

Background

At present, most Internet companies select to switch their servers from IDCs (local machine rooms maintained by themselves) to cloud hosts provided by cloud manufacturers, so that monitoring of service data is crucial in a cloud-based native scene, abnormalities of services can be found in advance through monitoring, the abnormalities are repaired, and economic loss caused by service faults is avoided. The business data monitoring is a very important link in a cloud native scene, and the monitoring is carried out on the premise of acquiring required business data, so that the business data acquisition is naturally a precondition for the business data monitoring and is also a very important link. The monitoring of the business data under the cloud native scene is constructed based on Prometheus and Grafana, and the data acquisition is to acquire the needed business data by developing acquisition software as a data acquisition unit.

Because the data of each service are different and the calculation mode of the data is also different, each service needs to develop own acquisition software. The scheme can lead to the fact that developers need to spend a lot of time on development of the acquisition software, each acquisition software is different, maintenance cost is high, once new business data are added, a new acquisition software needs to be developed, and development efficiency is low.

Disclosure of Invention

The embodiment of the application discloses a data acquisition system, a data acquisition method, data acquisition equipment and a data acquisition medium based on a cloud native scene, wherein visual acquisition software setting and management are realized by utilizing a management background based on information interaction among the management background, acquisition software and a communication configuration service platform, and the acquisition software development efficiency is improved.

The embodiment of the application discloses a data acquisition system based on a cloud native scene, which comprises a management background, a plurality of acquisition software and a communication configuration service platform;

the management background is used for acquiring acquisition condition information through an acquisition software configuration window and issuing acquisition task configuration information according to the acquisition condition information;

the communication configuration service platform is used for monitoring the management background and acquiring the acquisition task configuration information;

the acquisition software is used for monitoring the communication configuration service platform so as to acquire the acquisition task configuration information and execute the data acquisition task according to the acquisition task configuration information.

Optionally, the service authority of each acquisition software is different, and a plurality of acquisition software are deployed on different service servers according to the service authority;

executing a data acquisition task according to the acquisition task configuration information, wherein the data acquisition task comprises the following steps: and acquiring the service authority according to the acquisition task configuration information, and determining corresponding acquisition software according to the service authority.

Optionally, the executing the data collection task according to the collection task configuration information includes: and acquiring a data source query statement according to the acquisition task configuration information, and executing a data acquisition task by using the data source query statement.

Optionally, the management background is further configured to manage a data source through a data source management window, where the data source is a collection object of the collection software; the managing the data source through the data source management window comprises:

responding to a data source adding instruction, and adding data sources according to data source configuration information input by the data source management window;

in response to an editing instruction aiming at a single data source, modifying the single data source according to the data source configuration information input by the data source management window;

the data source configuration information comprises a data source identification code, a type and an address, and the data source identification code is used for uniquely identifying the data source.

Optionally, the management background is further configured to monitor the communication configuration service platform, obtain the acquisition software information of all running acquisition software, and generate an acquisition software list; and when the data source is changed, judging whether the acquisition software can be normally executed.

The embodiment of the application discloses a data acquisition method based on a cloud native scene, which comprises the following steps:

acquiring acquisition condition information through an acquisition software configuration window of a management background, and issuing acquisition task configuration information according to the acquisition condition information;

monitoring the management background through a communication configuration service platform to acquire the configuration information of the acquisition task;

and monitoring the communication service platform through a plurality of acquisition software, acquiring the acquisition task configuration information, and executing a data acquisition task according to the acquisition task configuration information.

Optionally, the method further comprises:

managing a data source through a data source management window of the management background, wherein the data source is a collection object of the collection software; the managing the data source through the data source management window comprises the following steps:

Optionally, the method further comprises:

registering acquisition software information to the communication configuration service platform through the acquisition software;

monitoring the communication configuration service platform through the management background, acquiring the acquisition software information of all running acquisition software, and generating an acquisition software list; and when the data source is changed, judging whether the acquisition software can be normally executed.

The embodiment of the application discloses electronic equipment, includes: the device comprises a memory and a processor, wherein the memory stores a computer program, and the computer program causes the processor to realize any method disclosed by the embodiment of the application when being executed by the processor.

Embodiments of the present application disclose a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements any of the methods disclosed in embodiments of the present application.

Compared with the prior art, the embodiment of the application has the following beneficial effects that:

the method comprises the steps of a management background, a plurality of acquisition software and a communication configuration service platform; setting and managing visual acquisition software by using a management background, and automatically releasing acquisition task configuration information according to the acquired acquisition condition information; monitoring the management background by using a communication configuration service platform to acquire acquisition task configuration information; and monitoring the communication configuration service platform by using acquisition software to acquire acquisition task configuration information, so as to execute the data acquisition task according to the acquisition task configuration information. The visual configuration of the acquisition conditions is realized through the management background, and the simple and efficient acquisition software development and data acquisition are realized through the information interaction among the management background, the plurality of acquisition software and the communication configuration service platform.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic diagram of a prior art data acquisition system of the present application;

FIG. 2 is a schematic view of a data acquisition system provided in a first embodiment of the present application;

FIG. 3 is a schematic diagram of a configuration window of acquisition software for a management background in the first embodiment of the present application;

FIG. 4 is another schematic diagram of the acquisition software configuration window of the management background in the first embodiment of the present application;

FIG. 5 is a diagram illustrating a data source management window for managing a background according to a first embodiment of the present application;

FIG. 6 is a flow chart of a data collection method provided in a second embodiment of the present application;

FIG. 7 is a further schematic diagram of a data acquisition system and method provided by embodiments of the present application;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".

Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.

Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically stated.

The meaning of the relevant features in the cloud-native scenario referred to in the embodiments of the present application is as follows.

Cloud-native: the cloud native computing foundation provides official definitions: cloud-native technology enables organizations to build and run scalable applications in new dynamic environments (such as public, private, and hybrid clouds). Containers, service grids, microservices, immutable infrastructure, and declarative APIs are examples of this approach.

Cloud service: is an increasing, usage and interaction model of internet-based related services, typically involving the provision of dynamically scalable and often virtualized resources over the internet. Cloud is a metaphor of network and internet. In the past, telecommunications networks were often represented by clouds and later also by the abstraction of the internet and the underlying infrastructure. The cloud service means that a required service is obtained through a network in an on-demand and easily-extensible manner. Such services may be IT and software, internet related, or other services. It means that computing power can also be circulated as a commodity over the internet.

Monitoring service data: monitoring related business data of enterprises, finding out the abnormity of business in advance through monitoring, and repairing the abnormity, thereby avoiding economic loss caused by business faults. At present, more enterprises have service functions in cloud, and the monitoring of service data is very important based on a cloud native scene.

Grafana: and the data visualization panel tool is used for displaying the acquired data.

Prometheus: software that stores business monitoring data.

Collecting software: under the cloud native scene, the general name of the data software is acquired and needs to be developed according to the service scene.

Redis: the method is an open-source memory data structure storage, is used as a database, a cache and a message middleware, and supports various data structures, such as: strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hypertrogs, geospatial Indexes with radius queries, and stream.

MySQL: a relational database.

Elastic search: is a Lucene-based search server. It provides a distributed multi-user capable full-text search engine based on RESTful web interface. The Elasticsearch is developed by adopting Java language and is released as open source code under Apache licensing terms, and is a popular enterprise-level search engine.

Data access right: at present, most of business data are stored in Redis, mySQL and elastic search, so that the software needs to be accessed when the business data is collected.

At present, most Internet companies select to switch their servers from local machine rooms maintained by an IDC (Internet Data Center) into cloud hosts provided by cloud manufacturers, and the monitoring of service Data is crucial based on a cloud native scene, so that the abnormality of a service can be found in advance through monitoring, the abnormality can be repaired, and economic loss caused by service failure is avoided. The service data monitoring is a very important link in a cloud native scene, and the monitoring is based on the premise of acquiring required service data, so that the service data acquisition is naturally a precondition for the service data monitoring and is also a very important link. The monitoring of the business data under the cloud native scene is constructed based on Prometheus and Grafana, and the data acquisition is to acquire the needed business data by developing acquisition software as a data acquisition unit. The current way of collecting data by software is shown in fig. 1: and corresponding acquisition software needs to be developed aiming at each kind of service data acquisition. Because the data of each service is different and the calculation mode of the data is also different, each service needs to develop own acquisition software. The scheme can lead to the fact that developers need to spend a lot of time on development of the acquisition software, each acquisition software is different, maintenance cost is high, once new business data are added, a new acquisition software needs to be developed in a targeted mode, and efficiency is low. Meanwhile, the acquisition server for deploying the acquisition software has too large data access authority to access data of all services, and the security cannot be guaranteed.

Fig. 2 shows a schematic diagram of a data acquisition system disclosed in the first embodiment of the present application, which includes a management background, a plurality of acquisition software, and a communication configuration service platform. The management background is used for acquiring acquisition condition information through an acquisition software configuration window and issuing acquisition task configuration information according to the acquisition condition information; the communication configuration service platform is used for monitoring the management background and acquiring the acquisition task configuration information; the acquisition software is used for monitoring the communication configuration service platform so as to acquire the acquisition task configuration information and execute the data acquisition task according to the acquisition task configuration information.

The acquisition software configuration window of the management background, as shown in fig. 3, can be used to implement the visual configuration of the acquisition task. Items that the configuration window shown in fig. 3 must fill include a collection task, a collection condition, a time field, a collection frequency, a tag of software storing business monitoring data, and a monitoring status. The specific item to be filled and the specific content filling or option setting of the item are not limited in this embodiment, as long as sufficient information required for performing data collection can be given. For example, the collection condition may be a SELECT statement shown in fig. 3, or may be another statement, as shown in fig. 4, the collection condition is "log.file.path: xxxxxx AND log.file.path: XXXX". Different items are set according to the type and the target of the collection task, and collection condition items are filled in.

The management background modifies or newly builds acquisition condition information (related information required by the software to execute an acquisition task) through the acquisition software configuration window, and releases acquisition task configuration information according to the acquisition condition information. The communication configuration service platform is used for monitoring the management background and acquiring the configuration information of the acquisition task; and the acquisition software is used for monitoring the communication configuration service platform so as to acquire the acquisition task configuration information and execute the data acquisition task according to the acquisition task configuration information. Therefore, the data acquisition system realizes visual acquisition condition configuration. When the business scale is very large, for example, more than 1000 collection tasks, 1000 collection scripts are needed if a script writing mode is adopted, how to place the scripts and how to search the scripts cause a great deal of complicated work on how to quickly take effect after the scripts are changed. The management background in this embodiment presets the type of the collection task, and configures different filling items in different types, where the filling items need to be filled by a user or selected according to selectable items. Therefore, the complex operation process of the traditional editing script file can be changed, and the setting of the acquisition conditions is simple, efficient and convenient to manage. Optionally, in some embodiments, the types of the collection tasks may be added or modified on the management background, and corresponding filling items may be configured for the new types after addition or modification.

In the embodiment, as the communication configuration service platform interacts with the management background and the acquisition software, the simple and efficient thermal update of the acquisition task is realized, namely, the acquisition software can monitor the relevant information to execute the data acquisition task as soon as the management background publishes the acquisition task.

In some embodiments, the service authority of each acquisition software is different, and a plurality of acquisition software are deployed on different service servers according to the service authority; executing a data acquisition task according to the acquisition task configuration information, wherein the data acquisition task comprises the following steps: and acquiring service authority according to the acquisition task configuration information, and determining corresponding acquisition software according to the service authority. As shown in fig. 2, each acquisition software is deployed on a different server, so as to prevent a server that can access all data rights from appearing, and ensure data security.

In some embodiments, the plurality of collection software is unified collection software, that is, the same program, and the unified collection software supports collection tasks of a plurality of data sources and supports loading of different configuration files to execute different collection tasks. Due to the adoption of the uniform acquisition software, the operation of automatically generating the data acquisition instruction can be conveniently realized. Compared with the prior art in which different acquisition software is used for completing different acquisition tasks, such as Node acquisition software and Redis acquisition software, the embodiment does not define recharge acquisition software and user acquisition software for completing service acquisition, but integrates the recharge acquisition software and the user acquisition software into one acquisition software. The multiple collection software in fig. 2 is to illustrate that collection software for different tasks is deployed in different service servers, and in fact, the multiple collection software is unified collection software.

More specifically, the unified acquisition software is set to have the following standard functions: the system comprises a data acquisition function, a data pushing function, an identification code acquisition function, an execution interval acquisition function, a circular execution function and a task storage pool acquisition function. The data acquisition function: the required data is collected from data sources (Redis, mySQL and elastic search) into the internal memory of the collection software. The data pushing function: and pushing the data in the acquired software memory to software (promemeus) for storing service monitoring data for unified storage. The identification code acquiring function comprises: the identification code (ID) is used as the unique identification of each acquisition task, and is the premise for distinguishing and managing various acquisition tasks by uniform acquisition software. The acquisition execution interval function: each acquisition task has its own execution time interval, for example every 5 seconds, or every 10 seconds, and therefore requires the function of acquiring the execution interval. The loop execution function: the collection tasks are processes that need to be executed continuously and circularly, so a scheduler is needed to execute each collection task continuously according to a set time interval. Collection task storage pool function: the unified acquisition software undertakes the execution of a large number of acquisition tasks consisting of multiple data sources, so a pool is required to manage and store all tasks that need to be executed.

In some embodiments, the executing a data collection task according to the collection task configuration information includes: and acquiring a data source query statement according to the acquisition task configuration information, and executing a data acquisition task by using the data source query statement.

The data sources Redis, mySQL and Elasticissearch support query statements, and the collection task is a process of querying the data source and then writing data into a storage space. In this embodiment, the developer only needs to perform simple configuration according to the configuration items on the visual interface, for example, configuration items such as configuration acquisition conditions and acquisition frequency. And the management background simplifies the filling of the configuration items, for example, the query sentence required to be filled in the acquisition condition in fig. 2 is the simplified query sentence, and the developer only needs to fill in the main logic, thereby greatly reducing the learning cost and the operation cost of the developer. The management background can carry out grammar analysis on the configuration information to obtain the configuration information of the acquisition task, and the acquisition software carries out grammar analysis again after obtaining the configuration information of the acquisition task to obtain the query statement corresponding to the data source. The query statement obtained by the acquisition software through syntax analysis is the query statement which is attached with additional information and can be used for executing the acquisition task. Through the steps, the query statement is updated in a hot mode, and the data acquisition task is executed automatically.

In some embodiments, the management background is further configured to manage a data source through a data source management window, where the data source is a collection object of the collection software; the managing the data source through the data source management window comprises:

Each service line has its own data source, for example, a user service line has its own MySQL and Redis, and a recharge service line also has its own MySQL and Redis. In the process of service development, the change of the data source means that a database and a database table are added, or the structure of the database table is changed, and at the moment, the collected rule also needs to be changed. For the management of the newly added database table, only a data source needs to be newly added in a management background, for the change of the acquisition rule caused by adding or deleting fields of the database table, only corresponding SQL statements need to be changed in the management background, and hard coding in program codes is not needed. And each data source can be managed independently, so that the flexibility is better. In a hard coding mode in a traditional program, any change of a data source needs to be redeveloped, tested and redeployed, updating is slow, and development cost is high. In the embodiment of the application, the unified acquisition software supports acquisition tasks of various data sources, and the management and configuration of the data sources are supported by using the management background, so that the change of each data source can be managed conveniently, the configuration of newly-added data sources is supported, and the problem that the configuration is hard-coded in a program code or in a separate configuration file and is difficult to change and manage is solved. Optionally, the single data source may also be deleted in response to a delete instruction for the single data source.

In some embodiments, in order to prevent the collection service from being affected by the abnormal downtime of the communication configuration service platform, the local file is used as a backup cache of the collection task. Specifically, each acquisition software stores acquisition task configuration information acquired from the communication configuration service platform in a memory, and writes the acquisition task configuration information into a local file, so that the problem that the acquisition software cannot acquire the acquisition task configuration information after the communication configuration service platform goes down abnormally is solved.

In some embodiments, the collection software is further configured to register collection software information with the communication configuration service platform, and the management background is further configured to monitor the communication configuration service platform, obtain the collection software information of all running collection software, and generate a collection software list; and when the data source is changed, judging whether the acquisition software can be normally executed. The collected software information comprises information such as a server IP, a monitoring port, a affiliated service and the like. If the data source changes and the normal execution of the acquisition software is influenced, the situation can be fed back to the user through the management background, so that the user can modify the situation in time and smoothly execute the data acquisition service.

In the embodiment, the logic of the acquisition task is changed from coding to configuration, so most functions are configured and completed in the management background. And issuing the acquisition task configuration information to a plurality of server nodes of the unified acquisition software through a communication configuration service platform. Meanwhile, in this embodiment, the acquisition software registers acquisition software information to the communication configuration service platform, and the management background monitors the communication configuration service platform, acquires the acquisition software information of all the running acquisition software, generates an acquisition software list, and facilitates the unified management of the management background.

Optionally, the communication configuration service platform in the embodiment of the present application may be an Etcd, distributed key-value pair data storage system.

Fig. 6 illustrates a data acquisition method based on a cloud-native scenario according to a second embodiment of the present application. Referring to fig. 6, the data acquisition method disclosed in this embodiment includes:

s601: acquiring acquisition condition information through an acquisition software configuration window of a management background, and issuing acquisition task configuration information according to the acquisition condition information;

s602: monitoring the management background through a communication configuration service platform to acquire the configuration information of the acquisition task;

s603: monitoring the communication service platform through a plurality of acquisition software, acquiring the acquisition task configuration information, and executing a data acquisition task according to the acquisition task configuration information.

The acquisition software configuration window of the management background, as shown in fig. 3, can be used to implement the visual configuration of the acquisition task. Items that the configuration window shown in fig. 3 must fill include a collection task, a collection condition, a time field, a collection frequency, a tag of software storing business monitoring data, and a monitoring state. The specific item to be filled and the specific content filling or selection setting of the item are not limited in this embodiment, as long as sufficient information required for performing data acquisition can be given. For example, the collection condition may be a SELECT statement shown in fig. 3, or may be another statement, as shown in fig. 4, the collection condition is "log.file.path: xxxxxx AND log.file.path: XXXX". Filling in collection condition items according to the type and the target of the collection task, and setting different items.

The management background modifies or newly builds acquisition condition information (related information required by the software to execute an acquisition task) through the acquisition software configuration window, and releases acquisition task configuration information according to the acquisition condition information. The communication configuration service platform is used for monitoring the management background and acquiring the configuration information of the acquisition task; and the acquisition software is used for monitoring the communication configuration service platform so as to acquire the acquisition task configuration information and execute the data acquisition task according to the acquisition task configuration information. Therefore, the data acquisition system realizes visual acquisition condition configuration. When the business scale is very large, for example, more than 1000 collection tasks, 1000 collection scripts are needed if a script writing mode is adopted, how to place the scripts and how to search the scripts cause a great deal of complicated work on how to quickly take effect after the scripts are changed. The management background in this embodiment sets the type of the collection task in advance, and configures different filling items in different types, where the filling items need to be filled by a user or selected according to selectable items to set the content of the items. Therefore, the complex operation process of the traditional editing script file can be changed, and the setting of the acquisition conditions is simple, efficient and convenient to manage. Optionally, in some embodiments, the types of the collection tasks may be added or modified on the management background, and corresponding filling items may be configured for the new types after addition or modification.

In the embodiment, because the communication configuration service platform interacts with the management background and the acquisition software, the simple and efficient thermal update of the acquisition task is realized, namely, the acquisition software can monitor the relevant information to execute the data acquisition task once the management background publishes the acquisition task.

In some embodiments, the service authority of each acquisition software is different, and a plurality of acquisition software are deployed on different service servers according to the service authority; executing a data acquisition task according to the acquisition task configuration information, wherein the data acquisition task comprises the following steps: and acquiring the service authority according to the acquisition task configuration information, and determining corresponding acquisition software according to the service authority.

In some embodiments, the multiple collection software are unified collection software, that is, the same program, and the unified collection software supports collection tasks of multiple data sources and supports loading of different configuration files to execute different collection tasks.

More specifically, the unified collection software is set to have the following standard functions: the system comprises a data acquisition function, a data pushing function, an identification code acquisition function, an execution interval acquisition function, a circular execution function and a task storage pool acquisition function. The data acquisition function: the required data is collected from data sources (Redis, mySQL and elastic search) into the internal memory of the collection software. The data pushing function: and pushing the data in the acquired software memory to software (Prometheus) for storing service monitoring data for unified storage. The identification code acquiring function comprises: the identification code (ID) is used as the unique identification of each acquisition task, and is the premise for distinguishing and managing various acquisition tasks by uniform acquisition software. The acquisition execution interval function: each acquisition task has its own execution time interval, for example every 5 seconds, or every 10 seconds, and therefore requires the function of acquiring the execution interval. The loop execution function: the collection tasks are processes that need to be executed continuously and circularly, so a scheduler is needed to execute each collection task continuously according to a set time interval. Collection task storage pool function: the unified collection software undertakes the execution of a large collection of tasks made up of multiple data sources, so a pool of storage is required to manage and store all the tasks that need to be executed.

In some embodiments, the method further comprises: managing a data source through a data source management window of the management background, wherein the data source is a collection object of the collection software; the managing the data source through the data source management window comprises:

Optionally, the single data source may also be deleted in response to a delete instruction for the single data source.

In some embodiments, in order to prevent the collection service from being affected by the abnormal downtime of the communication configuration service platform, the local file is used as a backup cache of the collection task. Specifically, each acquisition software stores acquisition task configuration information acquired from the communication configuration service platform in a memory, and writes the acquisition task configuration information into a local file, so that the problem that the acquisition software cannot acquire the acquisition task configuration information after the communication configuration service platform is abnormally shut down is solved.

In some embodiments, the method further comprises:

It should be noted that, for the information interaction, execution process, and other contents included in the foregoing method embodiments, specific functions and technical effects brought by the system embodiments of the present application based on the same concept may be specifically referred to a part of the system embodiments, and are not described herein again.

Fig. 7 shows a schematic diagram of a data acquisition system based on a cloud-native scene provided in a first embodiment of the present application, and a signal interaction process in the system further shows an execution process of a data acquisition method based on a cloud-native scene provided in a second embodiment of the present application. The system and method embodiments of the present application are further elaborated with reference to the illustration in fig. 7.

The acquisition software in fig. 7 may be a unified acquisition software, the unified acquisition software is started to register its own original data information (information such as IP of the server, monitoring port, and affiliated service) in the communication configuration service platform (Etcd), and the management background obtains all running unified acquisition software lists by monitoring the communication configuration service platform (Etcd), thereby facilitating unified management.

The management background issues the acquisition task configuration information to a communication configuration service platform (Etcd), and the unified acquisition software acquires the configuration information of the acquisition task to be executed by monitoring the communication configuration service platform (Etcd).

Each unified collection software stores the collection task configuration information acquired from the communication configuration service platform (Etcd) in a memory and writes the collection task configuration information into a local file, so that the problem that the unified collection software cannot acquire the collection task data after the communication configuration service platform (Etcd) is abnormally shut down is solved.

And each uniform acquisition software acquires a data source query statement according to the acquisition task configuration information and executes a data acquisition task by using the data source query statement.

After each uniform acquisition software finishes the acquisition task and acquires the service data, the data is pushed to software (Prometheus) for storing the service monitoring data for uniform storage.

As shown in fig. 7, the data acquisition scheme based on the cloud-based native scene frees developers from knowledge for learning Prometheus and developing acquisition software, and can acquire desired business data to Prometheus only by configuring in a management background without learning related knowledge, so that a mode that each developer writes acquisition software by himself in a conventional mode is changed, and labor cost and development time are greatly saved; and the hot update of the collection task can be realized, and the time and the complicated flow for releasing the collection task are saved.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. For the specific working processes of the units and modules in the system, reference may be made to the corresponding processes in the foregoing method embodiments, which are not described herein again.

Fig. 8 shows a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. As shown in fig. 8, the electronic apparatus 8 of this embodiment includes: at least one processor 80 (only one processor is shown in fig. 8), a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80, the processor 80 implementing the steps of any of the various method embodiments described above when executing the computer program 82.

The electronic device 8 may be a computing device such as an 8-computer, an 8-cloud server, and the like. The electronic device may include, but is not limited to, a processor 80, a memory 81. Those skilled in the art will appreciate that fig. 8 is merely an example of the electronic device 8, and does not constitute a limitation of the electronic device 8, and may include more or less components than those shown, or some of the components may be combined, or different components may be included, such as an input output device, a network access device, and the like.

The Processor 80 may be a Central Processing Unit (CPU), and the Processor 80 may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 81 may in some embodiments be an internal storage unit of the electronic device 8, such as a hard disk or a memory of the electronic device 8. The memory 81 may also be an external storage device of the electronic device 8 in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the electronic device 8. Further, the memory 81 may also include both an internal storage unit and an external storage device of the electronic device 8. The memory 81 is used for storing an operating system, an application program, a BootLoader (BootLoader), data, and other programs, such as program codes of the computer program. The memory 81 may also be used to temporarily store data that has been output or is to be output.

The embodiment of the present application further discloses a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps that can be implemented in the above method embodiments.

The embodiment of the application discloses a computer program product, which when running on an electronic device, enables the electronic device to implement the steps in the above method embodiments when executed.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include at least: any entity or device capable of carrying computer program code to a photographing apparatus/terminal apparatus, a recording medium, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In some jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and proprietary practices.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments disclosed in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims

1. A data acquisition system based on a cloud native scene is characterized by comprising a management background, a plurality of acquisition software and a communication configuration service platform;

the communication configuration service platform is used for monitoring the management background and acquiring the configuration information of the acquisition task;

2. The data acquisition system according to claim 1, wherein each of the acquisition software acquires a service authority according to the acquisition task configuration information, the service authority of each acquisition software deployed on different service servers according to the service authority is different, and a plurality of acquisition software are deployed on different service servers according to the service authority;

3. The data collection system of any one of claims 1-2, wherein the performing a data collection task according to the collection task configuration information comprises: and acquiring a data source query statement according to the acquisition task configuration information, and executing a data acquisition task by using the data source query statement.

4. The data acquisition system of claim 3, wherein the management back-end is further configured to manage a data source through a data source management window, wherein the data source is an acquisition object of the acquisition software; the managing the data source through the data source management window comprises:

5. The data acquisition system of claim 4, wherein the acquisition software is further configured to register acquisition software information with the communication configuration service platform, and the management backend is further configured to monitor the communication configuration service platform, obtain the acquisition software information of all running acquisition software, and generate an acquisition software list; and when the data source is changed, judging whether the acquisition software can be normally executed.

6. A data acquisition method based on a cloud native scene is characterized in that,

monitoring the management background through a communication configuration service platform to acquire the acquisition task configuration information;

monitoring the communication service platform through a plurality of acquisition software, acquiring the acquisition task configuration information, and executing a data acquisition task according to the acquisition task configuration information.

7. The data acquisition method according to claim 6, wherein each acquisition software has different service authority, and a plurality of acquisition software are deployed on different service servers according to the service authority;

8. The data collection method of any one of claims 6 to 7, wherein the performing a data collection task according to the collection task configuration information comprises: and acquiring a data source query statement according to the acquisition task configuration information, and executing a data acquisition task by using the data source query statement.

9. The data acquisition method of claim 8, further comprising:

10. The data acquisition method of claim 9, further comprising:

11. An electronic device, comprising: a memory having stored therein a computer program which, when executed by the processor, causes the processor to carry out the method of any one of claims 6 to 10.

12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 6 to 10.