CN118093552A - Visual information data processing system based on Internet of things - Google Patents

Abstract

The invention provides a visual information data processing system based on the Internet of things, which comprises a processor for executing database operation and a memory for storing database script language codes; an extended control structure for implementing complex logic decisions and flow control; data processing functions for performing advanced data aggregation, transformation, and analysis operations. The database system is optimized for high concurrency and large data volume environments, a multi-core processor architecture, a high-speed memory and an efficient data transmission mechanism are used, high performance and low delay in the process of processing large-scale data operation are ensured, the instantaneity and the flexibility of processing the data of the Internet of things are enhanced, and the system can better adapt to the rapidly-changing data processing requirements.

Description

Visual information data processing system based on Internet of things

Technical Field

The invention relates to the technical field of data processing and information, in particular to a visual information data processing system based on the Internet of things, and belongs to the technical field of data visualization and Internet of things data management.

Background

With the development and popularization of the internet of things technology, more and more devices are connected to the internet, and a large amount of data is generated; currently, database management mainly relies on Structured Query Language (SQL) as its main scripting language, which is excellent in terms of simple query, data manipulation and basic transaction management, and can efficiently perform data retrieval and simple data update tasks;

when processing a large amount of, diversified and real-time data generated in modern application scenes such as the internet of things, the existing database scripting language such as SQL faces obvious defects including processing highly complex data operations, realizing complex data analysis and processing logic and efficiently integrating data from a plurality of data sources, in such cases, the SQL often needs to rely on external application programs or middleware to supplement functions of the SQL, so that the complexity of a system is increased, the efficiency and the instantaneity of data processing are also influenced, and therefore, the visual information data processing system based on the internet of things is provided.

Disclosure of Invention

In view of the above, the present invention provides a visual information data processing system based on the internet of things, so as to solve or alleviate the technical problems existing in the prior art, and at least provide a beneficial choice.

The technical scheme of the invention is realized as follows: a visual information data processing system based on the Internet of things comprises

A processor for performing database operations and a memory storing database scripting language code;

An extended control structure for implementing complex logic decisions and flow control;

A data processing function for performing advanced data aggregation, conversion, and analysis operations;

an interpreter for interpreting or compiling the extended scripting language for processing directly within the database;

the interpreter is tightly integrated with the database engine, so that the data access and operation performance are optimized;

The security auditing and authority control module is used for executing security auditing and authority control module to ensure the security of database operation;

Wherein the expansion of the scripting language operates on more complex data within the database without reliance on external applications.

Further preferably, the extended control structure includes a conditional statement, a loop statement, and an exception handling statement; wherein the conditional statement supports multi-condition judgment and nested conditional logic; the loop statement supports iterative processing on a database table or a query result set; exception handling statements allow recognition and handling of incidents during data processing.

Further preferably, the data processing functions include data aggregation, statistical analysis, and data conversion capabilities; the data aggregation function is capable of performing complex aggregation operations on data from different tables or queries.

Further preferably, the script interpreter comprises an optimization engine, and the execution plan of the script is automatically optimized according to the context of script execution and the current state of the database; the optimization engine adjusts the execution policy according to the data access mode, the data size and the script complexity.

Further preferably, the scripting language supports user-defined stored procedures and functions; custom memory procedures are written to perform repetitive and complex data operations.

Further preferably, the security auditing and authority control module for managing and executing the scripting language codes is further included, and the role-based access control is supported, so that fine-grained authority management is performed on the scripting language codes executed by different users.

Further preferably, the method further comprises a query optimizer, wherein the query optimizer automatically optimizes the execution plan of the query statement according to the storage structure of the data, the index configuration and the query history.

Further preferably, the script directly calls an external API, and the script language is seamlessly integrated with an external Application Program Interface (API) to realize the import and export of data and the interaction with other databases and network services.

By adopting the technical scheme, the embodiment of the invention has the following advantages:

1. The database system is optimized for high concurrency and large data volume environments, a multi-core processor architecture, a high-speed memory and an efficient data transmission mechanism are used, high performance and low delay in the process of processing large-scale data operation are ensured, the instantaneity and the flexibility of processing the data of the Internet of things are enhanced, and the system can better adapt to the rapidly-changing data processing requirements.

2. The invention reduces the dependence on external tools by embedding complex data operation and analysis functions into the database system, thereby simplifying the overall architecture, supporting the storage process and functions defined by users and the seamless integration with external APIs, and improving the flexibility and the user friendliness of the system.

The foregoing summary is for the purpose of the specification only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the present invention will become apparent by reference to the drawings and the following detailed description.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of the system of the present invention.

Detailed Description

Hereinafter, only certain exemplary embodiments are briefly described. As will be recognized by those of skill in the pertinent art, the described embodiments may be modified in various different ways without departing from the spirit or scope of the present invention. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

As shown in FIG. 1, an embodiment of the present invention provides a visual information data processing system based on the Internet of things, including

A processor for performing database operations and a memory storing database scripting language code;

The processor is a multi-core architecture, each core can process different database operation requests in parallel, data processing efficiency in a high concurrency environment is ensured, data query operation is adjusted according to response time requirements, in addition, the processor supports advanced vector processing instructions for accelerating complex mathematical operations, and particularly, performance is remarkably improved when data statistics and aggregation operations are executed;

The memory adopts a solid state drive technology to provide high-speed data reading and writing capability, the memory is divided into a plurality of areas and is respectively used for storing script codes, temporary processing data and database operation logs, the script code storage area adopts an efficient file system organization structure to support quick retrieval and loading of the script codes, the temporary data area is designed into a cache area and can store intermediate data required by the database operation being executed, I/O operation times in the data processing process are reduced, and the operation log area is used for recording operation types, execution time and operation results of all database operations.

The processor and the memory are connected through a high-speed data bus, so that high efficiency and low delay of data transmission are ensured, the data bus supports high-bandwidth data transmission, and data flows rapidly between the processor and the memory during large-scale data operation such as data import/export and large-scale query processing are ensured;

database operations provide a high performance, high reliability database operating environment through the close coordination of their high performance processors and high capacity memory, suitable for handling large numbers of complex database scripting language code and operations.

An extended control structure for implementing complex logic decisions and flow control;

The expansion control structure comprises a multistage nested logic judgment and condition execution mechanism, allows a highly complex decision path to be realized in the database script, comprises conditional statements based on data values, data types, database states or any predefined condition, such as I F-THEN-ELSE and SWI TCH-CASE structures, can be used in a nested manner, realizes multi-level logic judgment, and can execute different operations according to a plurality of attribute values of data lines by nested I F statements;

Multiple levels of nested logical judgment and conditional execution mechanisms are used in database scripts, especially nested I F-THEN-ELSE and SWI TCH-CASE structural examples:

Multistage nested logic judgment: the script first checks the amount (each. Amounts) of each order and applies different discount policies according to the difference of the amounts, which is the first-level I F statement;

Nested IF-THEN-ELSE structure: the script checks the status of the order (each. Status) and performs different operations on different status, which is the I F structure of the second level;

SWITCH-CASE structure: the script uses the SWITCH-CASE structure to perform specific operations according to the customer ID (each_id), providing a clear way to handle the situation of multiple different conditions;

By expanding the control structure, the database script can flexibly process complex business logic to realize highly customized data operation, thereby improving the processing capacity of the database and the flexibility of application programs;

Further, the control structure supports loop-based data processing, including FOR loops and WH I LE loops, repeating operations on each row in the database table or on a set of rows meeting a particular condition;

FOR example, FOR loops are used to traverse a set of query results, applying complex data transformation or aggregation operations to each result, and WH I LE loops are used to repeatedly perform a series of operations, such as repeatedly updating data until a desired state is reached, before certain conditions are met.

In addition, the control structure introduces a TRY-CATCH-F I NALLY block exception handling mechanism that allows the script to handle runtime errors and exceptions, e.g., CATCH blocks can capture these exceptions and allow the script to take appropriate recovery measures when data operations fail due to incorrect formats or foreign key constraints.

Data processing functions for performing advanced data aggregation, transformation and analysis operations, integrated into a database management system, providing the ability to perform in-depth analysis and efficient processing of stored data, including data aggregation operations such as SUM, AVG, mbi, MAX, COUNT, etc., allowing statistical analysis of data sets in a database to derive SUM, average, minimum, maximum, COUNT, etc., information, e.g., to quickly calculate the total or average value of sales data, or to COUNT the number of orders in a particular time frame;

In addition to the basic aggregate function, advanced statistics enable users to more in-depth analyze data sets that evaluate data distribution, volatility, or central trends, for example, using variance and standard deviation to measure sales fluctuations, or using percentiles to learn about customer satisfaction distribution.

Data conversion includes data formatting, type conversion, and data cleansing operations, the data formatting allowing a user to convert data from one format to another, such as converting a date time from one format to another, or formatting a number to a monetary representation, and the type conversion functionality supports converting data between different data types, such as converting a string to a number or date type.

Further, the data processing also includes data fusion and merge operations that allow a user to combine data from different data sources together for analysis, including data join, merge, and join operations, allowing the user to integrate different data into a unified view based on common fields or conditions.

An interpreter for interpreting or compiling the extended scripting language for processing directly within the database;

The interpreter is a common computer program and is used for reading, analyzing and executing codes written by a high-level programming language, is a prior art in computer science and is widely applied to various programming languages and environments, the interpreter analyzes the codes of a script language through the syntax analyzer, processes complex syntax structures including nested control sentences, complex expressions and multi-layer function calls through the syntax analyzer, can accurately identify various syntax elements and construct an internal representation for subsequent processing, and performs semantic analysis on the basis of syntax analysis to ensure the logic consistency and correctness of the codes;

The interpreter supports the execution of advanced data processing operations including data aggregation, conversion and complex analysis, executes various data operations defined in the script, whether basic SQL queries or complex data analysis functions, allows the user to define own functions and stored procedures, and invokes custom code in the script, providing great flexibility, enabling the user to extend the language functions according to the own needs.

The interpreter is tightly integrated with the database engine, so that the data access and the operation performance are optimized, the interpreter is tightly integrated with the database engine, the data structure and the function inside the database can be directly accessed, and the script execution efficiency is higher through integration, so that the data can be directly processed inside the database without frequent data exchange;

The system comprises a security auditing and authority control module for executing security, ensuring the security of database operation, and being responsible for verifying the identity (authentication) of users and determining the database resources (authorization) which can be accessed by the users, wherein generally, the users need to provide a user name and a password when accessing the database, and once the authentication is successful, the access authority to specific data is granted according to the role and authority setting of the users;

Wherein the expansion of the scripting language operates on more complex data within the database without reliance on external applications.

In this embodiment, the extended control structure includes a conditional statement, a loop statement, and an exception handling statement; wherein the conditional statement supports multi-condition judgment and nested conditional logic; the loop statement supports iterative processing on a database table or a query result set; the exception handling statements allow recognition and handling of accidents during data processing, the extended control structure adds functionality similar to high-level programming language to the database scripting language, greatly enhances its flexibility and processing power, the conditional statements such as I F-THEN-ELSE enable the script to execute different logic paths according to different conditions, the cyclic statements such as FOR and WHILE allow repeated handling of the data set, effectively handling large amounts of data, and the exception handling statements such as TRY-CATCH are used to gracefully handle errors and accidents during data processing, enhancing the robustness and reliability of the script.

In this embodiment, the data processing functions include data aggregation, statistical analysis, and data conversion capability; the ability of the data aggregation function to perform complex aggregation operations on data from different tables or queries significantly expands the ability of the database to process and analyze the data, the data aggregation function allows data to be aggregated from multiple tables or queries, performing operations such as summing, averaging, maximum and minimum, the statistical analysis ability providing insight into the data set, and the data conversion function supporting adjustments in data format and structure, enhancing the applicability and flexibility of the data.

In this embodiment, the script interpreter includes an optimization engine that automatically optimizes the execution plan of the script according to the context of script execution and the current state of the database; the optimization engine adjusts the execution strategy according to the data access mode, the data size and the script complexity, and the optimization engine can select the most effective execution strategy through the data access mode, the data size and the script complexity, for example, when complex queries of a large amount of data are processed, the execution mode of the queries is adjusted, so that the resource consumption is reduced and the processing speed is increased.

In this embodiment, the scripting language supports user-defined stored procedures and functions; the writing of custom memory procedures to perform repeated and complex data operations is an important extension of the functionality of the database scripting language, and may write its own blocks of program code (i.e., memory procedures and functions) and store these codes in the database for reuse, and custom memory procedures and functions may be used to encapsulate complex or repeated data operation logic, improving code reusability, while making database operations more modular and efficient.

In this embodiment, the system further includes a security auditing and authority control module for managing and executing the scripting language codes, and supports role-based access control, so as to perform fine-grained authority management on the scripting language codes executed by different users.

In this embodiment, the method further includes a query optimizer, where the query optimizer automatically optimizes an execution plan of the query statement according to a storage structure of the data, an index configuration and a query history.

In this embodiment, the script directly invokes an external API, the scripting language performs seamless integration with an external Application Program Interface (API) to implement importing and exporting of data, and interacts with other databases and network services, through the seamless integration, the database script can directly interact with external systems and services to implement importing and exporting of data, and interoperate with other databases and network services, for example, the database script can directly invoke an API of a Web service to obtain real-time data, and integrate the data into a local database for further analysis and processing, and similarly, the script can export the data in the database to an external system, such as CRM or ERP systems, to support a cross-system business process.

Embodiment one: automated inventory management and report generation

Setting an electronic commerce company, wherein the inventory management process needs to be automated, and periodically generating inventory reports; implementation steps

1. Inventory data update storage process: defining a storage process to automatically update inventory data, wherein the storage process receives new inventory data (such as commodity IDs, newly added quantity and sales quantity) as input and updates corresponding inventory records;

2. Inventory shortage early warning: defining a storage process for checking the stock quantity of all the commodities and identifying the commodities with insufficient stock;

3. Automatic report generation: automatically running a script every week to generate an inventory report, wherein the report comprises a commodity list with insufficient inventory and commodities with highest sales;

The present embodiment implements complex inventory management logic through an extended database scripting language, with stored procedures for handling common inventory update and inspection tasks, while automated report generation functions provide useful business insight on a regular basis.

Embodiment two: customer data analysis and marketing

Analyzing the customer data to better locate its marketing campaign;

Implementation steps

1. Data import and cleaning: importing client data from an external AP I directly by using a script, and cleaning;

2. Client subdivision: subdividing the clients according to purchase history and behavior data;

3. targeted marketing campaigns: executing targeted marketing campaigns based on the client segment data;

4. Adjusting the execution policy using an optimization engine to process the large-scale customer dataset;

According to the embodiment, the client data are processed and analyzed by using the expanded script language, an effective marketing activity is performed, the data are directly imported from an external API, seamless integration of data flow is realized, the client subdivision and the directional marketing script enable a company to perform targeted marketing activity according to the behaviors and the value of the client, and in addition, the optimization engine ensures that high-efficiency performance can be maintained even when a large amount of data are processed.

The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that various changes and substitutions are possible within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (8)

1. A visual information data processing system based on the Internet of things is characterized in that: comprising

A processor for performing database operations and a memory storing database scripting language code;

An extended control structure for implementing complex logic decisions and flow control;

A data processing function for performing advanced data aggregation, conversion, and analysis operations;

an interpreter for interpreting or compiling the extended scripting language for processing directly within the database;

the interpreter is tightly integrated with the database engine, so that the data access and operation performance are optimized;

The security auditing and authority control module is used for executing security auditing and authority control module to ensure the security of database operation;

Wherein the expansion of the scripting language operates on more complex data within the database without reliance on external applications.

2. The visual information data processing system based on the internet of things according to claim 1, wherein: the extended control structure comprises a conditional statement, a cyclic statement and an exception handling statement; wherein the conditional statement supports multi-condition judgment and nested conditional logic; the loop statement supports iterative processing on a database table or a query result set; exception handling statements allow recognition and handling of incidents during data processing.

3. The visual information data processing system based on the internet of things according to claim 1, wherein: the data processing functions comprise data aggregation, statistical analysis and data conversion capability; the data aggregation function is capable of performing complex aggregation operations on data from different tables or queries.

4. The visual information data processing system based on the internet of things according to claim 1, wherein: the script interpreter comprises an optimizing engine, and the executing plan of the script is automatically optimized according to the context of script execution and the current state of the database; the optimization engine adjusts the execution policy according to the data access mode, the data size and the script complexity.

5. The visual information data processing system based on the internet of things according to claim 1, wherein: the scripting language supports user-defined stored procedures and functions; custom memory procedures are written to perform repetitive and complex data operations.

6. The visual information data processing system based on the internet of things according to claim 1, wherein: the security auditing and authority control module is used for managing and executing the script language codes, supporting access control based on roles and carrying out fine-granularity authority management on the script language codes executed by different users.

7. The visual information data processing system based on the internet of things according to claim 1, wherein: the system also comprises a query optimizer, wherein the query optimizer automatically optimizes the execution plan of the query statement according to the storage structure, index configuration and query history of the data.

8. The visual information data processing system based on the internet of things according to claim 1, wherein: the script directly calls an external API, and the script language is seamlessly integrated with an external Application Program Interface (API) to realize the importing and exporting of data and the interaction with other databases and network services.