CN111078980A - Management method, device, equipment and storage medium based on credit investigation big data - Google Patents
Management method, device, equipment and storage medium based on credit investigation big data Download PDFInfo
- Publication number
- CN111078980A CN111078980A CN201911350958.2A CN201911350958A CN111078980A CN 111078980 A CN111078980 A CN 111078980A CN 201911350958 A CN201911350958 A CN 201911350958A CN 111078980 A CN111078980 A CN 111078980A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- big data
- credit
- loading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/03—Credit; Loans; Processing thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Development Economics (AREA)
- General Business, Economics & Management (AREA)
- Technology Law (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Quality & Reliability (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the invention discloses a credit investigation big data-based management method, a credit investigation big data-based management device, credit investigation big data-based management equipment and a storage medium. The method is applied to a data center system and comprises the following steps: loading third-party data to a preset big data platform through an ETL (extract transform and load) process according to an information tag of the third-party data, wherein the information tag comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data purpose, an input parameter, a return parameter, abnormal calling and a concurrency condition; and loading the self business data and the network crawling data to a preset big data platform through an ETL (extract transform load) process. By applying the technical scheme of the embodiment of the invention, various credit investigation big data can be uniformly managed, so that the data can be well utilized, and the decision level of an enterprise is improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a credit investigation big data-based management method, device, equipment and storage medium.
Background
With the gradual consensus of various industries, the credit industry puts higher requirements on the traditional IT in the process of digital transformation. For example, the strategy planning of "small foreground, large foreground and medium background" is also fundamentally promoted by the scenarization, the servitization, the engineization, the medium background, the unified technology, the intellectualization and the like.
At present, the big credit data is disorderly managed, particularly, third-party data cannot be shared, data management is difficult, the big credit data cannot be well managed, and enterprises cannot make decisions by comprehensively, safely and efficiently utilizing data assets.
Disclosure of Invention
The embodiment of the invention provides a management method, a device, equipment and a storage medium based on credit investigation big data, and aims to solve the problem that in the prior art, various credit investigation big data cannot be uniformly managed, so that the data cannot be well utilized.
In a first aspect, an embodiment of the present invention provides a management method based on credit investigation big data,
the management method based on the credit investigation big data is applied to a data center station system, and comprises the following steps:
loading third-party data to a preset big data platform through an ETL (extract transform and load) process according to an information tag of the third-party data, wherein the information tag comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data purpose, an input parameter, a return parameter, abnormal calling and a concurrency condition;
and loading the self business data and the network crawling data to a preset big data platform through an ETL (extract transform load) process.
The further technical proposal is that,
in the process of loading data to a preset big data platform, data cleaning, data fusion, data dimension reduction and data standardization are required to be carried out on the data.
The further technical scheme is that the management method based on credit investigation big data further comprises the following steps:
and performing off-line calculation, real-time calculation, graph calculation and artificial intelligence model training on the data in a preset big data platform.
The further technical scheme is that the management method based on credit investigation big data further comprises the following steps:
and if a service request of a terminal is received, sending service content of the service request corresponding to the service request to the terminal, wherein the service request comprises at least one of query service, portrait label service, API service, message push service, data visualization service and text search service.
The further technical scheme is that the management method based on credit investigation big data further comprises the following steps:
and monitoring and managing the data quality, data operation, task flow and data safety of the whole process of the data center station system.
In a second aspect, an embodiment of the present invention further provides a management device based on credit investigation big data, including:
the system comprises a first loading unit, a second loading unit and a third-party data processing unit, wherein the first loading unit is used for loading the third-party data to a preset big data platform through an ETL (extract-transform-load) process according to an information label of the third-party data, and the information label comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data purpose, an input parameter, a return parameter, abnormal calling and concurrency conditions;
and the second loading unit is used for loading the self service data and the network crawling data to a preset big data platform through an ETL (extract transform load) process.
The further technical scheme is that in the process of loading data to a preset big data platform, data cleaning processing, data fusion processing, data dimension reduction processing and data standardization processing need to be carried out on the data.
The further technical scheme is that the management device based on credit investigation big data further comprises:
the credit investigation big data-based management device further comprises:
the processing unit is used for performing off-line calculation, real-time calculation, graph calculation and artificial intelligence model training on data in a preset big data platform;
the management unit is used for monitoring and managing the data quality, data operation, task flow and data safety of the whole process of the data center system;
the terminal comprises a sending unit, a receiving unit and a display unit, wherein the sending unit is used for sending service content of a service request corresponding to the service request to the terminal if the service request of the terminal is received, and the service request comprises at least one of query service, portrait label service, API service, message push service, data visualization service and text search service.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the above method when executing the computer program.
In a fourth aspect, the present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program can implement the above method when being executed by a processor.
The embodiment of the invention provides a credit investigation big data-based management method, a credit investigation big data-based management device, credit investigation big data-based management equipment and a storage medium. The method is applied to a data center system and comprises the following steps: loading third-party data to a preset big data platform through an ETL (extract transform and load) process according to an information tag of the third-party data, wherein the information tag comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data purpose, an input parameter, a return parameter, abnormal calling and a concurrency condition; and loading the self business data and the network crawling data to a preset big data platform through an ETL (extract transform load) process. By applying the technical scheme of the embodiment of the invention, various credit investigation big data can be uniformly managed, so that the data can be well utilized, and the decision level of an enterprise is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a management method based on credit investigation big data according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a middle station system according to an embodiment of the present invention; and
FIG. 3 is a schematic block diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Referring to fig. 1-2, fig. 1 is a schematic flow chart illustrating a management method based on big credit data according to an embodiment of the present invention. Fig. 2 is a schematic structural diagram of a middle station system according to an embodiment of the present invention. The credit investigation big data-based management method is applied to a data center system, and as shown in the figure, the method comprises the following steps S1-S2.
S1, loading the third-party data to a preset big data platform through an ETL process according to an information label of the third-party data, wherein the information label comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data use, an input parameter, a return parameter, calling abnormity and concurrency conditions;
in specific implementation, third-party data is called for credit investigation enterprises, the public data module classifies the third-party data according to tags of the third-party data, and a plurality of subject domain data are obtained through dimensional modeling. The tags include data collaborations (name, code), collaboration channels, calling data methods (name, code), developed data products (name, code, type), data usage, input parameters, return parameters, calling exceptions, and concurrency.
In a specific implementation, the data center system comprises a common data module. The public data module is used for loading credit investigation data to a big data platform of an enterprise through an ETL process, and providing data support for further research and development of the data.
In specific implementation, since the third-party data is mostly provided in a data service manner and cannot be directly loaded to the big data platform, the information tag of the third-party cooperative data needs to be extracted first for storage, and then the third-party cooperative data needs to be loaded to the big data platform. The information tag comprises a data cooperation mechanism (name, code), a cooperation channel, a calling data mode (name, code), a developed data product (name, code, type), a data purpose, an input parameter, a return parameter, a calling exception and a concurrency condition.
And S2, loading the self service data and the network crawling data to a preset big data platform through an ETL process.
In an embodiment, in the process of loading data to a preset big data platform, data cleaning, data fusion, data dimension reduction and data standardization are required to be performed on the data.
In specific implementation, a visualized data model is designed and constructed based on data elements defined by specifications. The unified index caliber is realized through a data index structured standardization mode, and the theme domain data is constructed by utilizing dimension modeling. A business core object-centric connectivity and tagging architecture is formed. And the metadata of the big data platform is collected, analyzed and managed, and a metadata center is constructed. The above steps S1 and S2 accompany the data schema design and data specification definition design of the standard specification in the implementation process.
Further, the management method based on credit investigation big data further comprises the following steps:
and performing off-line calculation, real-time calculation, graph calculation and artificial intelligence model training on the data in a preset big data platform.
In specific implementation, analysis calculation and deep mining are carried out on the data of the big data platform by utilizing the off-line calculation, real-time calculation, graph calculation and artificial intelligence model training capabilities of the big data platform. By further processing the business direction of the big data platform data, access to each business data source (middle component) is provided, and a model and a solution for exploring and finding data value are provided for business personnel. Constructing a business model; through statistics, big data analysis technology, artificial intelligence technology and other methods, the analysis, calculation and value mining are carried out on the business data, and the value and the function of the business data are exerted and mined.
Further, the management method based on credit investigation big data further comprises the following steps:
and if a service request of a terminal is received, sending service content of the service request corresponding to the service request to the terminal, wherein the service request comprises at least one of query service, portrait label service, API service, message push service, data visualization service and text search service.
By constructing a data service query engine, providing a channel of an intermediate component, providing a uniform data outlet and a data query logic for business, and realizing uniform theme data service. The data service includes at least one of a query service, a portrait tagging service, an API service, a message push service, a data visualization service, an intelligent recommendation, and a text search service.
By constructing a data service query engine, providing a channel of an intermediate component, providing a uniform data outlet and a data query logic for business, and realizing uniform theme data service.
In a specific implementation, the data center system further comprises a data service module. The data service module is used for sending service content of the service request corresponding to the service request to the terminal if the service request of the terminal is received.
The data center is closer to the service, and provides data service for the service in a more efficient mode, so that the system develops a multifunctional data service module. The data service module may provide the following services.
1. The query service supports offline query and real-time query of data across heterogeneous databases, and comprises common SQL query and OLAP multidimensional query.
2. A portrait label service provides a system for presenting a factual portrait and an abstract portrait of an information subject such as an enterprise and a organization. Such as the exhibition of the image labels of the corporate public information, bidding information, judicial information, annual newspaper information, credit evaluation information, corporate prospect prediction and the like.
3. The API service is oriented to the business unified data outlet and the data query logic, and generates a normalized business data API interface on the basis, and provides the normalized business data API interface to a foreground system in a diversified mode.
4. The message pushing service supports message pushing functions of WeChat, mailbox, nail and the like.
5. And the data visualization service provides data visualization display, and comprises modes of a BI report, an information map, a data billboard, a data large screen and the like.
6. The text search service provides a visual text search function by utilizing the elastic search and text analysis technology, and realizes the quick response of a large amount of text information (bidding, judicial punishment and the like) of main bodies such as enterprises and institutions.
7. The intelligent recommendation is a set of recommended intelligent service framework, and aims to enable business personnel and developers to quickly acquire business data meeting business requirements of the business personnel and the developers on the set of framework. The function is mainly provided by considering the directions of generating respective k-s values and data quality and the like from third-party data and enterprise owned data in a specific application scene (such as an address verification model) through mass data mining and artificial intelligence technology.
In an embodiment, the credit investigation big data-based management method provided by the present invention further includes the following steps:
and monitoring and managing the data quality, data operation, task flow and data safety of the whole process of the data center station system.
In specific implementation, data is corrected in time through the two functions of task scheduling and data quality detection, so that the quality of the data is guaranteed.
The data center system also comprises a data management module.
The data management module is used for displaying and managing the metadata of the enterprise data assets. The method helps business members to arrange which data and the source and the purpose of the data exist in the enterprise under the current condition, and realizes the monitoring of data flow and task flow by technicians and the monitoring of data flow direction and use by a management layer.
1. Data blood-source tracking, which is the way the graphical presentation table data comes, is generated through which transformation and processing.
2. The data map is a graphical centralized metadata management system, which visually shows the data assets of enterprises, and can search the structural definition of corresponding data.
3. The task flow management, the analysis and the selection realize the target route, sequence and process, can effectively plan, design, realize and optimize the task flow, find the fault and diagnose, position, analyze and debug in time, and simplify the operation and maintenance work.
4. Data operation management mainly manages the relationship between data assets and business products. The service product calls which API interfaces, uses which third-party data, is the content of the enterprise owned data and the like, and which client uses the product, the version of the product currently maintained and used and the like are maintained.
5. And the function of data security management manages the aspects of data security, user authority and the like. The department or service line and each member thereof have respective authority levels, and the corresponding levels have corresponding data authorities, which comprise access, use, detail data derivation, data definition, data structure and the like.
6. And the monitoring center is mainly used for monitoring all data assets, a big data platform and a data center system.
The above 4 modules are not independent individuals, but are related to each other and interact with each other to form a unified whole. Data service is not independent of data research and development; new data generated by data research and development, data service and data management is a new data sediment of a public data module; the public data module after precipitation starts from the business level and can generate new data research and development and data service; and the processes can not be monitored and managed by the data management module.
By applying the technical scheme of the embodiment of the invention, various credit investigation big data can be uniformly managed, so that the data can be well utilized, and the decision level of an enterprise is improved. The concrete aspects are as follows:
data standardization, data asset management of enterprises is unified, data standards or data indexes among different databases of different service lines of different departments are prevented from being non-unified, and unified specifications of domains, themes, models, fields, index naming and the like of the data assets are achieved.
The data sharing method has the advantages that the data sharing capability is high in efficiency, data can be found in all departments, related data components are not repeatedly developed, the data, the API and the corresponding data quality are found in the system, whether the product is supported by high-quality data or not can be immediately known through analysis and test, the data use threshold is lowered, and the data service innovation capability is stimulated.
The unified and diversified cross-source data service is characterized in that a unified and standardized data interface service engine is developed depending on heterogeneous databases of all departments and business lines, and comprises SQL (structured query language) query, ad hoc query, OLAP (on-line analytical processing) multidimensional analysis, multi-mode data push, text search, online analysis, customized data analysis models and the like.
A sound data security management mechanism not only uses desensitization processing, encryption processing, data leakage defense and other technologies aiming at sensitive data, but also realizes data difference access aiming at different users, clients and organizational structures, and monitors and tracks the source and the destination of the data.
The method has strong real-time performance, can respond to the requirements of business and application development more quickly, and can trace back and be more accurate.
Corresponding to the management method based on the credit investigation big data, the invention also provides a management device based on the credit investigation big data. The credit investigation big data-based management device comprises a unit for executing the credit investigation big data-based management method, and the device can be configured in a data center system. Specifically, the credit investigation big data-based management device comprises the following units:
the system comprises a first loading unit, a second loading unit and a third-party data processing unit, wherein the first loading unit is used for loading the third-party data to a preset big data platform through an ETL (extract-transform-load) process according to an information label of the third-party data, and the information label comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data purpose, an input parameter, a return parameter, abnormal calling and concurrency conditions;
and the second loading unit is used for loading the self service data and the network crawling data to a preset big data platform through an ETL (extract transform load) process.
The classification unit is used for classifying the third-party data according to tags of the third-party data to obtain a plurality of subject domain data, and the tags comprise data cooperation mechanisms (names and codes), cooperation channels, calling data modes (names and codes), developed data products (names, codes and types), data purposes, input parameters, return parameters, calling exceptions and concurrence conditions.
In an embodiment, in the process of loading data to a preset big data platform, data cleaning, data fusion, data dimension reduction and data standardization are required to be performed on the data.
And designing and constructing a visualized data model based on the data elements defined by the specification. The unified index caliber is realized through a data index structured standardization mode, and the theme domain data is constructed by utilizing dimension modeling. And the metadata of the big data platform is collected, analyzed and managed, and a metadata center is constructed.
In an embodiment, the credit investigation big data-based management device further includes:
the processing unit is used for performing off-line calculation, real-time calculation, graph calculation and artificial intelligence model training on data in a preset big data platform;
the management unit is used for monitoring and managing the data quality, data operation, task flow and data safety of the whole process of the data center system;
the terminal comprises a sending unit, a receiving unit and a display unit, wherein the sending unit is used for sending service content of a service request corresponding to the service request to the terminal if the service request of the terminal is received, and the service request comprises at least one of query service, portrait label service, API service, message push service, data visualization service and text search service.
It should be noted that, as can be clearly understood by those skilled in the art, the detailed implementation process of the management device and each unit based on credit investigation big data may refer to the corresponding description in the foregoing method embodiment, and for convenience and brevity of description, no further description is provided herein.
The above described credit investigation big data based management apparatus may be implemented in the form of a computer program, which may be run on a computer device as shown in fig. 3.
Referring to fig. 3, fig. 3 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a terminal or a server, where the terminal may be an electronic device with a communication function, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, and a wearable device. The server may be an independent server or a server cluster composed of a plurality of servers.
Referring to fig. 3, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, causes the processor 502 to perform a credit investigation big data based management method.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 can be caused to execute a credit investigation big data-based management method.
The network interface 505 is used for network communication with other devices. Those skilled in the art will appreciate that the configuration shown in fig. 3 is a block diagram of only a portion of the configuration associated with the present application and does not constitute a limitation of the computer device 500 to which the present application may be applied, and that a particular computer device 500 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps:
loading third-party data to a preset big data platform through an ETL (extract transform and load) process according to an information tag of the third-party data, wherein the information tag comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data purpose, an input parameter, a return parameter, abnormal calling and a concurrency condition;
and loading the self business data and the network crawling data to a preset big data platform through an ETL (extract transform load) process.
In an embodiment, the processor 502 needs to perform data cleaning, data fusion, data dimension reduction, and data standardization on the data during the process of loading the data onto a preset big data platform.
In one embodiment, processor 502 further implements the steps of:
and performing off-line calculation, real-time calculation, graph calculation and artificial intelligence model training on the data in a preset big data platform.
In one embodiment, processor 502 further implements the steps of:
and if a service request of a terminal is received, sending service content of the service request corresponding to the service request to the terminal, wherein the service request comprises at least one of query service, portrait label service, API service, message push service, data visualization service and text search service.
In one embodiment, processor 502 further implements the steps of:
and monitoring and managing the data quality, data operation, task flow and data safety of the whole process of the data center station system.
It should be understood that, in the embodiment of the present Application, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It will be understood by those skilled in the art that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program instructing associated hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program. The computer program, when executed by a processor, causes the processor to perform the steps of:
loading third-party data to a preset big data platform through an ETL (extract transform and load) process according to an information tag of the third-party data, wherein the information tag comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data purpose, an input parameter, a return parameter, abnormal calling and a concurrency condition;
and loading the self business data and the network crawling data to a preset big data platform through an ETL (extract transform load) process.
In an embodiment, the processor needs to perform data cleaning, data fusion, data dimension reduction, and data standardization on the data during the process of executing the computer program to load the data onto a preset big data platform.
In an embodiment, the processor, in executing the computer program, further implements the steps of:
and performing off-line calculation, real-time calculation, graph calculation and artificial intelligence model training on the data in a preset big data platform.
In an embodiment, the processor, in executing the computer program, further implements the steps of:
and if a service request of a terminal is received, sending service content of the service request corresponding to the service request to the terminal, wherein the service request comprises at least one of query service, portrait label service, API service, message push service, data visualization service and text search service.
By constructing a data service query engine, providing a channel of an intermediate component, providing a uniform data outlet and a data query logic for business, and realizing uniform theme data service.
In an embodiment, the processor, in executing the computer program, further implements the steps of:
and monitoring and managing the data quality, data operation, task flow and data safety of the whole process of the data center station system.
The storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, which can store various computer readable storage media.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be merged, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, while the invention has been described with respect to the above-described embodiments, it will be understood that the invention is not limited thereto but may be embodied with various modifications and changes.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A management method based on credit investigation big data is applied to a data center station system, and is characterized in that the management method based on credit investigation big data comprises the following steps:
loading third-party data to a preset big data platform through an ETL (extract transform and load) process according to an information tag of the third-party data, wherein the information tag comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data purpose, an input parameter, a return parameter, abnormal calling and a concurrency condition;
and loading the self business data and the network crawling data to a preset big data platform through an ETL (extract transform load) process.
2. The credit investigation big data-based management method as claimed in claim 1, wherein the data is required to be subjected to data cleaning, data fusion, data dimension reduction and data standardization in the process of being loaded on a preset big data platform.
3. The credit big data-based management method according to claim 1, wherein the credit big data-based management method further comprises:
and performing off-line calculation, real-time calculation, graph calculation and artificial intelligence model training on the data in a preset big data platform.
4. The credit big data-based management method according to claim 1, wherein the credit big data-based management method further comprises:
and if a service request of a terminal is received, sending service content of the service request corresponding to the service request to the terminal, wherein the service request comprises at least one of query service, portrait label service, API service, message push service, data visualization service and text search service.
5. The credit big data-based management method according to claim 1, wherein the credit big data-based management method further comprises:
and monitoring and managing the data quality, data operation, task flow and data safety of the whole process of the data center station system.
6. A management device based on credit investigation big data is characterized by comprising:
the system comprises a first loading unit, a second loading unit and a third-party data processing unit, wherein the first loading unit is used for loading the third-party data to a preset big data platform through an ETL (extract-transform-load) process according to an information label of the third-party data, and the information label comprises a data cooperation mechanism, a cooperation channel, a data calling mode, a developed data product, a data purpose, an input parameter, a return parameter, abnormal calling and concurrency conditions;
and the second loading unit is used for loading the self service data and the network crawling data to a preset big data platform through an ETL (extract transform load) process.
7. The credit big data-based management device according to claim 6,
in the process of loading data to a preset big data platform, data cleaning, data fusion, data dimension reduction and data standardization are required to be carried out on the data.
8. The credit big data-based management device according to claim 6, wherein the credit big data-based management device further comprises:
the processing unit is used for performing off-line calculation, real-time calculation, graph calculation and artificial intelligence model training on data in a preset big data platform;
the management unit is used for monitoring and managing the data quality, data operation, task flow and data safety of the whole process of the data center system;
the terminal comprises a sending unit, a receiving unit and a display unit, wherein the sending unit is used for sending service content of a service request corresponding to the service request to the terminal if the service request of the terminal is received, and the service request comprises at least one of query service, portrait label service, API service, message push service, data visualization service and text search service.
9. A computer arrangement, characterized in that the computer arrangement comprises a memory having stored thereon a computer program and a processor implementing the method according to any of claims 1-5 when executing the computer program.
10. A computer-readable storage medium, characterized in that the storage medium stores a computer program which, when being executed by a processor, is adapted to carry out the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911350958.2A CN111078980A (en) | 2019-12-24 | 2019-12-24 | Management method, device, equipment and storage medium based on credit investigation big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911350958.2A CN111078980A (en) | 2019-12-24 | 2019-12-24 | Management method, device, equipment and storage medium based on credit investigation big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111078980A true CN111078980A (en) | 2020-04-28 |
Family
ID=70317408
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911350958.2A Pending CN111078980A (en) | 2019-12-24 | 2019-12-24 | Management method, device, equipment and storage medium based on credit investigation big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111078980A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111797297A (en) * | 2020-09-09 | 2020-10-20 | 平安国际智慧城市科技股份有限公司 | Page data processing method and device, computer equipment and storage medium |
| CN112231119A (en) * | 2020-10-16 | 2021-01-15 | 广西科技大学 | Distributed cloud management system data center platform service design method |
| CN112328585A (en) * | 2020-11-17 | 2021-02-05 | 珠海大横琴科技发展有限公司 | Data processing method and device |
| CN113626415A (en) * | 2021-08-27 | 2021-11-09 | 天元大数据信用管理有限公司 | Credit data output method, device and medium |
| CN114547173A (en) * | 2022-02-23 | 2022-05-27 | 平安国际智慧城市科技股份有限公司 | Data warehouse construction method, device and equipment and computer storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105809606A (en) * | 2016-03-08 | 2016-07-27 | 贵州省邮电规划设计院有限公司 | Big data application platform system |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
-
2019
- 2019-12-24 CN CN201911350958.2A patent/CN111078980A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105809606A (en) * | 2016-03-08 | 2016-07-27 | 贵州省邮电规划设计院有限公司 | Big data application platform system |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111797297A (en) * | 2020-09-09 | 2020-10-20 | 平安国际智慧城市科技股份有限公司 | Page data processing method and device, computer equipment and storage medium |
| CN112231119A (en) * | 2020-10-16 | 2021-01-15 | 广西科技大学 | Distributed cloud management system data center platform service design method |
| CN112231119B (en) * | 2020-10-16 | 2024-01-30 | 广西科技大学 | Distributed cloud management system data center platform service design method |
| CN112328585A (en) * | 2020-11-17 | 2021-02-05 | 珠海大横琴科技发展有限公司 | Data processing method and device |
| CN112328585B (en) * | 2020-11-17 | 2024-07-09 | 珠海大横琴科技发展有限公司 | Data processing method and device |
| CN113626415A (en) * | 2021-08-27 | 2021-11-09 | 天元大数据信用管理有限公司 | Credit data output method, device and medium |
| CN113626415B (en) * | 2021-08-27 | 2024-02-23 | 天元大数据信用管理有限公司 | Credit data output method, equipment and medium |
| CN114547173A (en) * | 2022-02-23 | 2022-05-27 | 平安国际智慧城市科技股份有限公司 | Data warehouse construction method, device and equipment and computer storage medium |
| CN114547173B (en) * | 2022-02-23 | 2024-11-08 | 深圳平安智慧医健科技有限公司 | A data warehouse construction method, device, equipment and computer storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110140126B (en) | Method, server and memory device for modeling management attributes | |
| Elgendy et al. | Big data analytics: a literature review paper | |
| US9323736B2 (en) | Natural language metric condition alerts generation | |
| CN111078980A (en) | Management method, device, equipment and storage medium based on credit investigation big data | |
| US10725981B1 (en) | Analyzing big data | |
| US9361320B1 (en) | Modeling big data | |
| US20140100923A1 (en) | Natural language metric condition alerts orchestration | |
| US20140100901A1 (en) | Natural language metric condition alerts user interfaces | |
| US20210406981A1 (en) | Method and apparatus of determining display page, electronic device, and medium | |
| US20150286693A1 (en) | Analyzing a query and provisioning data to analytics | |
| CN111046237B (en) | User behavior data processing method and device, electronic equipment and readable medium | |
| CN110109905A (en) | Risk list data generation method, device, equipment and computer storage medium | |
| US20130054285A1 (en) | Computing and managing conflicting functional data requirements using ontologies | |
| US10977236B2 (en) | Dynamic recommendation methods and systems for database changes | |
| US20110313932A1 (en) | Model-based project network | |
| US10623520B1 (en) | System and method for tagging in identity management artificial intelligence systems and uses for same, including context based governance | |
| US12412143B2 (en) | Systems and methods for creating, training, and evaluating models, scenarios, lexicons, and policies | |
| CN110675238A (en) | Client label configuration method, system, readable storage medium and electronic equipment | |
| CN114239511A (en) | Method and apparatus for filling data | |
| US11763014B2 (en) | Production protection correlation engine | |
| US8428989B2 (en) | Cross functional area service identification | |
| US20100030596A1 (en) | Business Process Intelligence | |
| Daza et al. | Impact of business intelligence on incident management in the control center of a security company | |
| CN116703125A (en) | Task processing method and device for product research and development and computer equipment | |
| CN116521660A (en) | Enterprise value assessment method and related device based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200428 |