CN110007967B - A data processing method, device and device based on streaming framework - Google Patents

Abstract

The embodiment of the invention provides a data processing method, a device and equipment based on a streaming framework, wherein the method comprises the following steps: acquiring task parameters corresponding to data to be processed; determining an analysis unit for processing the data to be processed and an analysis sequence of the analysis unit according to the task parameters; loading the determined analysis unit; connecting each loaded analysis unit according to the analysis sequence to obtain an analysis unit combination; inputting the data to be processed into the analysis unit combination, and analyzing the data to be processed by utilizing the thread contained in each analysis unit; and acquiring a processing result according to the output content of the analysis unit combination. Therefore, in the scheme, after the task parameters are obtained, the analysis unit is loaded, the analysis unit does not continuously run in a process mode, and after the analysis unit is loaded, the data is analyzed by using the threads contained in the analysis unit.

Description

A data processing method, device and device based on streaming framework

technical field

The present invention relates to the technical field of data processing, and in particular, to a data processing method, device and device based on a streaming framework.

Background technique

In some scenarios with a large amount of data processing, the streaming framework is usually used for data processing. Streaming frameworks such as Storm can be used for data processing in various scenarios such as real-time analysis, online machine learning, continuous computing, and distributed remote calls.

However, in the process of data processing using Storm, each analysis unit runs continuously as a process, occupying a lot of resources.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a data processing method, apparatus and device based on a streaming framework, so as to reduce the resource occupancy rate.

To achieve the above purpose, an embodiment of the present invention provides a data processing method based on a streaming framework, including:

Get the task parameters corresponding to the data to be processed;

According to the task parameter, determine the analysis unit that processes the data to be processed, and the analysis sequence of the analysis unit; wherein, each analysis unit includes an independent thread;

Loading the determined analysis units; and according to the analysis sequence, connecting each of the loaded analysis units to obtain an analysis unit combination;

Acquire the data to be processed, input the data to be processed into the combination of analysis units, and analyze the data to be processed by using a thread included in each analysis unit in the combination of analysis units;

According to the output content of the combination of the analysis units, the processing result is acquired.

Optionally, the obtaining task parameters corresponding to the data to be processed may include:

obtaining the first configuration file corresponding to the data to be processed;

The determining, according to the task parameters, an analysis unit that processes the data to be processed and an analysis sequence of the analysis unit may include:

According to the identification and identification sequence of the analysis units contained in the first configuration file, the analysis units that process the data to be processed and the analysis sequence of the analysis units are determined.

Optionally, after acquiring the data to be processed, it may further include: generating a structure object with a preset structure for the data to be processed;

The inputting the data to be processed into the combination of analysis units, and using the thread included in each analysis unit in the combination of analysis units to analyze the data to be processed, including:

Input the structure object into the first analysis unit in the combination of analysis units, use the thread included in the first analysis unit to analyze the structure object, obtain the analysis result, and add the analysis result to the structure object to obtain the output object of the first analysis unit;

According to the analysis sequence, successively determine each analysis unit after the first analysis unit as the current analysis unit; input the output object of the previous analysis unit of the current analysis unit into the current analysis unit, and use the The thread, analyzes the output object of the last analysis unit, obtains the analysis result, adds the analysis result to the output object of the last analysis unit, and obtains the output object of the current analysis unit;

The obtaining a processing result according to the output content combined by the analysis unit may include:

The processing result is acquired according to the output object of the last analysis unit in the combination of analysis units.

Optionally, after obtaining the output object of the first analysis unit, it may further include:

adding the output object of the first analysis unit to the output queue of the first analysis unit;

The inputting the output object of the previous analysis unit of the current analysis unit into the current analysis unit includes:

From the input queue of the current analysis unit, according to the preset queue sequence, sequentially read the output objects of the previous analysis unit;

After obtaining the output object of the current analysis unit, the method further includes:

Adds the output object of the current unit to the output queue of the current analysis unit.

Optionally, obtaining the processing result according to the output object of the last analysis unit in the combination of analysis units may include:

Judging whether the output object of the last analysis unit in the combination of analysis units satisfies the output condition;

If satisfied, obtain the processing result from the output object of the last analysis unit;

If not satisfied, return to the step of inputting the structure object into the first analysis unit in the combination of analysis units.

Optionally, before acquiring the data to be processed, it may further include: acquiring a second configuration file corresponding to the data to be processed;

reading the identifier of the interactive accessory library contained in the second configuration file, and loading the interactive accessory inventory corresponding to the read identifier, where the interactive accessory library includes an input interface, an output interface, an assembly accessory and a detection accessory;

After acquiring the data to be processed, it may further include:

calling the input interface, and inputting the data to be processed into the interactive accessory library;

The generating a structure object with a preset structure for the data to be processed may include:

Using the assembly accessories, the data to be processed is assembled to generate a structure object with a preset structure;

The judging whether the output object of the last analysis unit in the analysis unit combination satisfies the output condition may include:

Invoke the detection accessory to detect the output object of the last analysis unit in the analysis unit combination, and determine whether the detection result satisfies the output condition;

After obtaining the processing result from the output object of the last analysis unit, it may further include:

The output interface is called to output the processing result.

Optionally, the interactive accessory library further includes deleting accessories, and after calling the output interface and outputting the processing result, it may further include:

The delete accessory is called to delete the output object of the last analysis unit.

Optionally, after obtaining the processing result according to the output content of the combination of analysis units, the method may further include: uninstalling the analysis units in the combination of analysis units.

In order to achieve the above purpose, an embodiment of the present invention also provides a data processing device based on a streaming framework, including:

a first obtaining module, used for obtaining task parameters corresponding to the data to be processed;

a determination module, configured to determine an analysis unit for processing the data to be processed and an analysis sequence of the analysis unit according to the task parameter; wherein each analysis unit includes an independent thread;

a first loading module for loading the determined analysis unit;

a connection module, configured to connect each of the loaded analysis units according to the analysis sequence to obtain a combination of analysis units;

a second acquisition module, configured to acquire the data to be processed;

an analysis module, configured to input the data to be processed into the combination of analysis units, and analyze the data to be processed by using a thread included in each analysis unit in the combination of analysis units;

The third obtaining module is configured to obtain the processing result according to the output content of the combination of the analysis units.

Optionally, the first obtaining module may be specifically configured to: obtain a first configuration file corresponding to the data to be processed;

The determining module can be specifically used for:

According to the identification and identification sequence of the analysis units contained in the first configuration file, the analysis units that process the data to be processed and the analysis sequence of the analysis units are determined.

Optionally, the device may also include:

a generating module, for generating a structure object with a preset structure for the data to be processed;

The analysis module can be specifically used for:

Input the structure object into the first analysis unit in the combination of analysis units, use the thread included in the first analysis unit to analyze the structure object, obtain the analysis result, and add the analysis result to the structure object to obtain the output object of the first analysis unit;

According to the analysis sequence, successively determine each analysis unit after the first analysis unit as the current analysis unit; input the output object of the previous analysis unit of the current analysis unit into the current analysis unit, and use the The thread, analyzes the output object of the last analysis unit, obtains the analysis result, adds the analysis result to the output object of the last analysis unit, and obtains the output object of the current analysis unit;

The third acquisition module can be specifically used for:

The processing result is acquired according to the output object of the last analysis unit in the combination of analysis units.

Optionally, the device may further include: a first adding module and a second adding module, wherein,

a first adding module, configured to add the output object of the first analysis unit to the output queue of the first analysis unit;

The analysis module is further configured to sequentially read the output objects of the previous analysis unit from the input queue of the current analysis unit according to a preset queue sequence;

The second adding module is configured to add the output object of the current unit to the output queue of the current analysis unit after the output object of the current analysis unit is obtained.

Optionally, the device may also include:

a judgment module for judging whether the output object of the last analysis unit in the combination of analysis units satisfies the output condition; if so, triggers the third acquisition module; if not, triggers the analysis module;

The third obtaining module may be specifically configured to: obtain the processing result from the output object of the last analysis unit.

Optionally, the device further includes: a fourth acquisition module, a second loading module, an input module and an output module, wherein,

a fourth obtaining module, configured to obtain a second configuration file corresponding to the data to be processed;

The second loading module is configured to read the identifier of the interactive accessory library contained in the second configuration file, and load the interactive accessory inventory corresponding to the read identifier, and the interactive accessory library includes an input interface and an output interface. , assembly accessories and testing accessories;

an input module, configured to call the input interface and input the data to be processed into the interactive accessory library;

The generating module may be specifically configured to: assemble the data to be processed by using the assembly accessories to generate a structure object with a preset structure;

The judging module can be specifically used for:

Invoke the detection accessory to detect the output object of the last analysis unit in the analysis unit combination, and determine whether the detection result satisfies the output condition;

The output module is configured to call the output interface to output the processing result after obtaining the processing result from the output object of the last analysis unit.

Optionally, the interactive accessory library further includes deleting accessories, and the device may further include:

The deletion module is used for calling the deletion accessory to delete the output object of the last analysis unit.

Optionally, the device may also include:

The unloading module is configured to unload the analysis units in the analysis unit combination after acquiring the processing result according to the output content of the analysis unit combination.

To achieve the above object, an embodiment of the present invention also provides an electronic device, including a processor and a memory;

memory for storing computer programs;

The processor is configured to implement any one of the above data processing methods based on the streaming framework when executing the program stored in the memory.

Applying the embodiment of the present invention, after the task parameters are acquired, the analysis unit is loaded again. The analysis unit does not run continuously as a process, and after the analysis unit is loaded, the thread included in the analysis unit is used to analyze the data. Compared with the analysis unit The solution that runs continuously in a process mode reduces the resource occupancy rate.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a first schematic flowchart of a data processing method based on a streaming framework provided by an embodiment of the present invention;

2 is a second schematic flowchart of a data processing method based on a streaming framework provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a streaming processing framework provided by an embodiment of the present invention;

4 is a schematic diagram of the connection between an analysis unit and a pipeline according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of another streaming processing framework provided by an embodiment of the present invention;

6 is a schematic structural diagram of a data processing apparatus based on a streaming framework provided by an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In order to solve the above technical problems, embodiments of the present invention provide a data processing method, apparatus and electronic device based on a streaming framework. The method and apparatus can be applied to various electronic devices, which are not specifically limited.

The following first describes in detail a data processing method based on a streaming framework provided by an embodiment of the present invention.

1 is a first schematic flowchart of a data processing method based on a streaming framework provided by an embodiment of the present invention, including:

S101: Acquire task parameters corresponding to the data to be processed.

For example, the data to be processed may be a video, an image, etc., which is not specifically limited. It should be noted that, in S101, the data to be processed is not acquired, but the task parameters corresponding to the data to be processed are acquired.

As an implementation manner, S101 may include: acquiring a first configuration file corresponding to the data to be processed. In order to distinguish it from the configuration file in the following content, the configuration file obtained in S101 is referred to as the first configuration file. The first configuration file may include the identification and identification sequence of the analysis unit, and the identification and identification sequence of the analysis unit belong to the task parameters. Alternatively, the task parameter may also include a numerical value, and the corresponding relationship between the numerical value and the identification of the analysis unit and the identification sequence is preset. The specific content of the task parameters is not limited.

S102: Determine an analysis unit that processes the data to be processed and an analysis sequence of the analysis unit according to the task parameter. Among them, each analysis unit contains an independent thread.

As mentioned above, the analysis unit identification and the identification sequence can be obtained, and the identification is the identification of the analysis unit that processes the data to be processed. The identification of the analysis unit can be a serial number, an ID, etc. That is, the processing sequence of each analysis unit that processes the data to be processed.

Taking the data to be processed as a video as an example, multiple processing steps such as decoding the video, detecting human objects, extracting face features, and extracting features of people's clothing can be performed, and each processing step can correspond to an analysis unit. There is a sequence between these processing steps, and correspondingly, an analysis sequence also exists between the analysis units that process the data to be processed. For different data to be processed, the analysis units that process the data may be different, and the analysis sequences of the analysis units that process the data may also be different.

For example, if a video contains face data, multiple processing steps such as decoding, face target detection, and face feature extraction need to be performed on the video, and each processing step may correspond to an analysis unit. It is assumed that there are three analysis units for processing the data to be processed: analysis unit 1, analysis unit 2 and analysis unit 3, wherein the analysis unit 1 is used to decode the data to be processed, and the analysis unit 2 is used to detect the person in the data to be processed. Face target, the analysis unit 3 is used to extract the face features in the data to be processed.

For another example, another video contains license plate data, which requires multiple processing steps such as decoding, vehicle target detection, and license plate feature extraction, and each processing step may correspond to an analysis unit. It is assumed that there are three analysis units for processing the data to be processed: analysis unit 1, analysis unit 4 and analysis unit 5, wherein the analysis unit 1 is used to decode the data to be processed, and the analysis unit 4 is used to detect vehicles in the data to be processed The target, the analysis unit 5 is used to extract the license plate features in the data to be processed.

Each analysis unit includes an independent thread, and each analysis unit uses its own thread to process the data to be processed.

S103: Load the determined analysis units; and connect each of the loaded analysis units according to the analysis sequence to obtain a combination of analysis units.

In the device (execution body) or other devices communicatively connected to the device, multiple analysis units are pre-stored in the form of plug-ins, and the determined analysis units that process the data to be processed are loaded into the memory.

There may be no connection relationship between the above-mentioned pre-stored multiple analysis units in advance. After the determined analysis units are loaded into the memory and the analysis order of each analysis unit for processing the data to be processed is determined, the loaded analysis units are connected according to the analysis order to obtain an analysis unit combination. In this way, the connection relationship between the analysis units is adjustable, and different connection relationships can be established for different data to be processed to obtain different combinations of analysis units.

Continuing the above example, the data to be processed is a video containing face data, and it is determined that the data to be processed is the analysis unit 1-3, and the analysis sequence from front to back is: analysis unit 1, analysis unit 2, analysis unit 3 . The three analysis units are connected according to the analysis sequence. Specifically, the output interface of the analysis unit 1 is connected with the input interface of the analysis unit 2, and the output interface of the analysis unit 2 is connected with the input interface of the analysis unit 3 to obtain the analysis unit combination.

S104: Acquire the to-be-processed data, input the to-be-processed data into the analysis unit combination, and analyze the to-be-processed data by using a thread included in each analysis unit in the analysis unit combination.

For example, the task parameter acquired in S101 may further include address information of the data to be processed, and the address information may be a URL (Uniform Resource Locator, Uniform Resource Locator), or may be other, which is not specifically limited. According to the address information, the data to be processed can be obtained.

Still take the data to be processed as the video as an example to illustrate, first this video is input into the analysis unit 1, and the thread in the analysis unit 1 is used to decode the video to obtain the decoded data; The decoded data is input into the analysis unit 2, Utilize the thread in the analysis unit 2 to detect the face target in the decoded data to obtain the face target data; input the face target data into the analysis unit 3, and use the thread in the analysis unit 3 to extract the face feature in the data to be processed , to get the facial feature data.

S105: Acquire a processing result according to the output content of the analysis unit combination.

As an implementation manner, the output content can be directly used as the processing result; or, the output content can be format-converted, and the converted data can be used as the processing result; or, valid data can be extracted from the output content as The processing results, etc., are not specifically limited.

For example, in the above example, the analysis unit 3 is the last unit in the combination of analysis units, the output content of the analysis unit 3 is the output content of the combination of analysis units, and the output content is face feature data, which can be directly used as the face feature data. process result.

Applying the embodiment shown in FIG. 1 of the present invention, after the task parameters are obtained, the analysis unit is loaded again. The analysis unit does not run continuously in a process mode, and after the analysis unit is loaded, the thread included in the analysis unit is used to analyze the data. Compared with the solution that the analysis unit runs continuously in a process mode, the resource occupancy rate is reduced, and the thread is easier to control than the process.

As an implementation manner, after the data to be processed is acquired, a structure object with a preset structure may be generated for the data to be processed. For example, the preset structure may include three parts, the first part may be called a header structure, and the header structure may store processing progress information for processing the data to be processed, error code information during processing, and the like. The second part may store the data to be processed, and the third part may store the analysis result of each analysis unit in the analysis combination.

In this way, the data to be processed is input into the combination of analysis units, and the data to be processed is analyzed by using a thread included in each analysis unit in the combination of analysis units, which may include:

1. Input the structure object into the first analysis unit in the combination of analysis units, use the thread included in the first analysis unit to analyze the structure object, and obtain the analysis result. added to the structure object to obtain the output object of the first analysis unit;

2. According to the analysis sequence, sequentially determine each analysis unit after the first analysis unit as the current analysis unit; input the output object of the previous analysis unit of the current analysis unit into the current analysis unit, and use the current analysis unit The included thread analyzes the output object of the previous analysis unit to obtain an analysis result, and adds the analysis result to the output object of the previous analysis unit to obtain the output object of the current analysis unit.

S105 may include: acquiring a processing result according to the output object of the last analysis unit in the combination of analysis units.

Continuing the above example, each analysis unit adds its own analysis results to the third part of the above preset structure, that is, the output object of the last analysis unit contains all the contents of the structure object, and also includes Analysis results of all analysis units in this analysis unit combination. For example, the analysis result of the last analysis unit may be extracted from the output object of the last analysis unit as the processing result; or, the analysis result of each analysis unit may be extracted from the output object of the last analysis unit The result is taken as the processing result; alternatively, the output object of the last analysis unit can also be taken as the processing result.

As an implementation manner, each analysis unit may include an input queue and an output queue, the analysis unit obtains the output object of the previous analysis unit from the input queue, or obtains the structure object, and the analysis unit adds its own output object to the output queue.

Specifically, after the output object of the first analysis unit is obtained, the output object of the first analysis unit is added to the output queue of the first analysis unit;

The inputting the output object of the previous analysis unit of the current analysis unit into the current analysis unit includes: from the input queue of the current analysis unit, sequentially reading the output object of the previous analysis unit according to a preset queue sequence;

After the output object of the current analysis unit is obtained, the output object of the current unit is added to the output queue of the current analysis unit.

For example, if the data to be processed is a video, each analysis unit can process each image in the video separately. Assuming that the analysis unit 1-2 is used to process the video, the structure object corresponding to the first image is first input into the analysis unit 1, and the analysis unit 1 analyzes the structure object by using its own thread to obtain an output object, which is The output object is added to the output queue of the analysis unit 1; similarly, the analysis unit 1 performs the above processing on each image in the video.

The input queue of the analysis unit 2 and the output queue of the analysis unit 1 can be connected through a pipeline, so that the analysis unit 2 can obtain the output object of the analysis unit 1 from its own input queue. The analysis unit 2 sequentially reads the output objects of the analysis unit 1 from its own input queue, and can read one output object at a time. Analysis unit 2 analyzes an output object read by using the thread included in itself, obtains an output object, and adds the output object to the output queue of analysis unit 2; The above processing is performed for each output object.

In this embodiment, an input/output queue is set for each analysis unit. For each analysis unit, there may be multiple output objects in the input queue of the analysis unit. The processing rate is low, and the analysis unit does not need to wait, which improves the data processing efficiency compared to the scheme without input/output queues.

As an implementation manner, after obtaining the processing result according to the output object of the last analysis unit in the analysis unit combination, it can also be judged whether the output object of the last analysis unit in the analysis unit combination satisfies the output condition; , obtain the processing result from the output object of the last analysis unit; if not satisfied, return to execute the step of inputting the structure object into the first analysis unit in the combination of analysis units.

Taking the above video containing license plate data as an example, the output object of the last analysis unit contains license plate feature data, such as license plate number; output conditions can be set for the license plate number, such as whether there are unrecognized characters, and if so , the first analysis unit and subsequent steps of inputting the structure object into the analysis unit combination are repeated. Until there are no unrecognized characters in the license plate number contained in the output object of the last analysis unit, or until the number of repetitions reaches a threshold. In other words, the output condition may also be for whether the number of repetitions reaches a threshold.

In this embodiment, a cycle mechanism is implemented, that is, when the output object of the last analysis unit does not meet the output condition, each analysis unit is used for analysis in a cycle until the output object meets the output condition, or the number of cycles reaches the threshold, In this way, the accuracy of the processing result can be improved. Such a loop mechanism cannot be implemented in the existing Storm framework. Therefore, compared with the Storm framework, the present embodiment improves the accuracy of the processing result.

As an embodiment, before acquiring the data to be processed, a second configuration file corresponding to the data to be processed may also be acquired; the identifier of the interactive accessory library contained in the second configuration file is read, and the identifier corresponding to the read identifier is read. The interactive accessories inventory is loaded, and the interactive accessories library includes input interfaces, output interfaces, assembly accessories and detection accessories. The "input interface" mentioned here refers to the input interface of the interactive accessory library, and the "output interface" mentioned here refers to the output interface of the interactive accessory library. The input/output interface of the interactive accessory library can be an interface connected to an upper-layer application or user equipment, which can be understood as an external interface, while the input/output interface of the analysis unit in the above content and the input of the pipeline in the following content / The output interface is not connected to the upper-layer application, user equipment, etc., and can be understood as an internal interface.

After acquiring the data to be processed, the input interface of the interactive accessory library can also be called, and the data to be processed can be input into the interactive accessory library.

In this embodiment, one or more interactive accessory libraries exist in the device (execution body) or other devices communicatively connected to the device, and each interactive accessory library includes an input interface, an output interface, an assembly accessory, and a detection accessory. Each piece of data to be processed may correspond to a second configuration file. In order to distinguish the description from the configuration file in the above content, the configuration file containing the identification of the interactive accessory library is referred to as the second configuration file.

Alternatively, the first configuration file and the second configuration file may also belong to the same configuration file, and the configuration file includes not only the identification of the interactive accessory library, but also the identification and identification sequence of the analysis unit.

Each piece of data to be processed may correspond to a second configuration file, and the second configuration file includes the identifier of the interactive accessory library that processes the piece of data to be processed. The identifier of the interactive accessory library may be a serial number, an ID, etc., which is not specifically limited, and the interactive accessory library corresponding to the identifier is loaded into the memory.

The generating a structure object with a preset structure for the data to be processed may include: using the assembly accessories to assemble the data to be processed to generate a structure object with a preset structure.

In this embodiment, the input interface in the interactive accessories library is called, the data to be processed is input into the interactive accessories library, and the assembly accessories are used to assemble the to-be-processed data to generate a structure object with a preset structure.

In addition, determine the analysis unit for processing the data to be processed and the sequence of each analysis unit, and according to the sequence, connect the determined analysis units into an analysis unit combination; input the structure object into the analysis unit combination, and the analysis unit combination outputs an output object.

The judging whether the output object of the last analysis unit in the analysis unit combination satisfies the output condition may include: calling the detection accessory, detecting the output object of the last analysis unit in the analysis unit combination, and judging the detection result Whether the output conditions are met.

If not satisfied, re-input the structure object into the analysis unit combination; if satisfied, obtain the processing result from the output object, and call the output interface in the interactive accessory library to output the processing result, where the "output" can be The processing result is sent to the user equipment or the upper-layer application.

The interactive accessory library may also include a delete accessory, after calling the output interface and outputting the processing result, the delete accessory may also be called to delete the output object of the last analysis unit, so that memory resources can be saved, Guaranteed not to leak memory.

As an implementation manner, after the data to be processed is processed, the analysis units in the analysis unit combination can be unloaded, so that memory occupation can be reduced.

FIG. 2 is a second schematic flowchart of a data processing method based on a streaming framework provided by an embodiment of the present invention,

S201: Acquire a second configuration file corresponding to the data to be processed.

S202: Read the identifier of the interactive accessory library contained in the second configuration file, and load the interactive accessory inventory corresponding to the read identifier, where the interactive accessory library includes input interfaces, output interfaces, assembly accessories, and detection accessories.

In this embodiment, there may be two levels of loading. In order to distinguish the description, the configuration file used for the first-level loading is called the second configuration file, and the configuration file used for the second-level loading is called the first configuration file. The first-level loading is to load the interactive accessory library, and the second-level loading is to load the analysis unit.

A specific processing framework can be shown in Figure 3, including an interactive accessory library, a plug-in manager and an analysis unit. The plug-in manager loads the interactive accessory library by reading the second configuration file, and loads the analysis unit by reading the first configuration file. There can be multiple interactive accessory libraries and analysis units, and different interactive accessory libraries and analysis units are usually loaded for different data to be processed.

In the first-level loading, the plug-in manager obtains the identifier of the interactive accessory library for processing the data to be processed by reading the second configuration file; the plug-in manager loads the interactive accessory library corresponding to the identifier into the memory, and maps the interactive accessory The interface address of the library, the interface address includes the input interface address and the output interface address.

S203: Obtain a first configuration file corresponding to the data to be processed; determine the analysis unit for processing the data to be processed and the analysis sequence of the analysis unit according to the identification and identification sequence of the analysis unit included in the first configuration file; Among them, each analysis unit contains an independent thread.

S204: Load the determined analysis units; and connect each of the loaded analysis units according to the analysis sequence to obtain an analysis unit combination.

S204 is the second-level loading, the plug-in manager in FIG. 3 obtains the identification of the analysis unit and the identification sequence by reading the first configuration file, the plug-in manager loads the analysis unit corresponding to the identification into the memory, and according to the identification sequence, the loaded analysis units are connected in series to obtain a combination of analysis units.

In this embodiment, as shown in FIG. 3 , the analysis units may be connected through pipes, and the data transmitted between the analysis units flows from one analysis unit to another analysis unit through the pipes. The first analysis unit and the last analysis unit are connected by pipes to the interactive accessories library. As shown in Figure 4, the internal interface of the pipeline includes an input interface pin and an output interface pout, wherein pin is connected to the output interface of the analysis unit 1, and pout is connected to the input interface of the analysis unit 2, so that the data can be sent from Analysis unit 1 flows to analysis unit 2.

For example, data buffering may not be performed in the pipeline; in addition, the number of repeated transmissions may be set in the pipeline. For example, in Figure 4, if the data transmission between the analysis unit 1 and the analysis unit 2 fails, the transmission is repeated until The transmission is repeated as many times as set by the pipeline.

In this embodiment, an encapsulation layer may be set between the interactive accessory library and the analysis unit, and the encapsulation layer is used to shield the logical relationship inside each analysis unit. For example, as shown in FIG. 5, an encapsulation layer is set between the plug-in manager, each analysis unit and the interactive accessory library, and the interactive accessory library only needs to call the interface of the encapsulation layer, and can respond to the interface of each analysis unit in a streaming manner. In this way, for the caller, there is no need to traverse and call all the analysis units, nor to care about the calling logic between the analysis units. The operation is simple and the experience is better.

Moreover, in this embodiment, each analysis unit includes an independent thread, that is to say, the operations in each analysis unit are performed by different threads, and the threads do not affect each other, even if the operations in one analysis unit are performed by different threads It does not affect the operation in other analysis units, improves the operation efficiency, and reduces the resource occupancy rate compared with the solution in which the analysis unit runs continuously in a process mode.

S205: Acquire data to be processed.

S206: Invoke the input interface, and input the data to be processed into the interactive accessory library.

S207: Assemble the to-be-processed data by using the assembly accessory to generate a structure object with a preset structure.

In S202, the interactive accessory library is loaded into the memory, and the interactive accessory library includes assembly accessories, and the assembly accessories can assemble the data to be processed into a structure object of a preset structure. For example, the preset structure may include three parts, the first part may be called a header structure, and the header structure may store processing progress information for processing the data to be processed, error code information during processing, and the like. The second part may store the data to be processed, and the third part may store the analysis result of each analysis unit in the analysis combination.

S208: Input the structure object into the first analysis unit in the combination of analysis units, use the thread included in the first analysis unit to analyze the structure object, obtain an analysis result, and add the analysis result to the structure body object to get the output object of the first analysis unit.

S209: According to the analysis sequence, successively determine each analysis unit after the first analysis unit as the current analysis unit; input the output object of the previous analysis unit of the current analysis unit into the current analysis unit, and use the The thread analyzes the output object of the previous analysis unit to obtain the analysis result, and adds the analysis result to the output object of the previous analysis unit to obtain the output object of the current analysis unit.

As an implementation manner, each analysis unit may include an input queue and an output queue, the analysis unit obtains the output object of the previous analysis unit from the input queue, or obtains the structure object, and the analysis unit adds its own output object to the output queue.

Specifically, after the output object of the first analysis unit is obtained, the output object of the first analysis unit is added to the output queue of the first analysis unit; from the input queue of the current analysis unit, According to the preset queue sequence, the output objects of the previous analysis unit are sequentially read; after the output objects of the current analysis unit are obtained, the output objects of the current unit are added to the output queue of the current analysis unit.

In this embodiment, an input/output queue is set for each analysis unit. For each analysis unit, there may be multiple output objects in the input queue of the analysis unit. The processing rate is low, and the analysis unit does not need to wait, which improves the data processing efficiency compared to the scheme without input/output queues.

S210: Invoke the detection accessory to detect the output object of the last analysis unit in the analysis unit combination, and determine whether the detection result satisfies the output condition, if yes, execute S211, and if not, return to execute S208.

As mentioned above, for different data to be processed, different interactive accessory libraries can be loaded, that is, each accessory in the interactive accessory library is suitable for the data to be processed. Therefore, the detection accessories in the interactive accessory library can be used for The output object of the data to be processed is detected.

For example, if the data to be processed is a video containing license plate data, and the output object of the last analysis unit contains the recognition result of the license plate number, the detection accessory can detect whether there are unrecognized characters in the recognition result; if so, the detection result If the output condition is not satisfied, execute S208-S210 repeatedly; if not, the detection result satisfies the output condition, and execute S211.

In addition, the detection accessory may also set a threshold for the number of times of repeated execution. If the detection result does not meet the output condition, and the number of repeated executions of S208-S210 reaches the threshold, the detection result is considered to meet the output condition.

When returning to perform S208, the analysis results of each analysis unit stored in the third part of the output object may be deleted for the output object of the last analysis unit, and the information stored in the first part of the output object may be updated, such as The processing progress information, the error code information in the processing process, etc. are updated to the initial values, etc., which are not specifically limited. Input the output object of the last analysis unit processed above as a structure object into the first analysis unit.

Alternatively, when returning to perform S208, the assembly components may be recalled to regenerate the structure object of the preset structure, and the newly generated structure object may be input into the first analysis unit.

In this embodiment, a cycle mechanism is implemented, that is, when the output object of the last analysis unit does not meet the output condition, each analysis unit is used for analysis in a cycle until the output object meets the output condition, or the number of cycles reaches the threshold, In this way, the accuracy of the processing result can be improved. Such a loop mechanism cannot be implemented in the existing Storm framework. Therefore, compared with the Storm framework, the present embodiment improves the accuracy of the processing result.

S211: Acquire the processing result from the output object of the last analysis unit, and call the output interface to output the processing result.

Continuing the above example, each analysis unit adds the analysis result obtained by itself to the third part of the above preset structure, so that the output object of the last analysis unit contains all the contents of the structure object, and also includes the analysis Analysis results for all analysis units in the unit combination.

For example, the analysis result of the last analysis unit may be extracted from the output object of the last analysis unit as the processing result; or, the analysis result of each analysis unit may also be extracted from the output object of the last analysis unit The result is taken as the processing result; alternatively, the output object of the last analysis unit can also be taken as the processing result.

S212: Call the delete accessory to delete the output object of the last analysis unit.

S213: Uninstall the interactive accessory library and the analysis units in the analysis unit combination.

As an embodiment, after the data to be processed is processed, the interactive accessory library and the analysis unit can be uninstalled through the plug-in manager. Corresponding to loading the interactive accessory library and the analysis unit, the interactive accessory library can be unloaded through the second configuration file, and the analysis unit can be unloaded through the first configuration file.

Specifically, the memory space applied for when loading the interactive accessory library in S202 can be released according to the address handle of the interactive accessory library contained in the second configuration file; and S204 can be released according to the address handle of the analysis unit contained in the first configuration file The memory space allocated when the analysis unit is loaded.

In this implementation manner, after the data to be processed is processed, the memory occupied by the processing process is released, which can reduce the memory occupancy rate.

A specific implementation is described below in conjunction with Figures 3-5:

1. The device receives a processing task: identify the license plate number in a video. The device obtains two configuration files related to the processing task: configuration file A and configuration file B. The data to be processed is the video, the configuration file A is the second configuration file in the above content, and the configuration file B is the first configuration file in the above content.

Read the identifier of the interactive accessory library contained in the configuration file A, assuming the serial number 3, then load the interactive accessory library with serial number 3 into the memory through the plug-in manager. The interactive accessory library includes input interface, output interface, assembly Accessories and inspection accessories. Acquire the video, call the input interface in the interactive accessories library, and input the video into the interactive accessories library. Using the assembly parts in the interactive parts library, assemble the video into a structure object with a preset structure.

For example, the preset structure may include three parts, the first part may be called a header structure, and the header structure may store processing progress information for processing the data to be processed, error code information during processing, and the like. The second part may store the data to be processed, and the third part may store the analysis result of each analysis unit in the analysis combination.

In addition, read the analysis unit identification and identification sequence contained in the configuration file B, assuming that the identification and identification sequence are: analysis unit 1, analysis unit 4 and analysis unit 5, then through the plug-in manager, load these three analysis units into memory, and in this order, connect the three analysis units to obtain a combination of analysis units. Specifically, as shown in FIG. 3 , the three analysis units can be connected through pipes, wherein the analysis unit 1 and the analysis unit 5 are connected with the interactive accessory library through pipes. Each analysis unit contains a separate thread.

As shown in Figure 5, an encapsulation layer is set between the plug-in manager and each analysis unit and the interactive accessory library, and the interactive accessory library only needs to call the interface of the encapsulation layer, and can respond to the interface of each analysis unit in a streaming manner. In this way, for the caller, there is no need to traverse and call all the analysis units, nor to care about the calling logic between the analysis units. The operation is simple and the experience is better.

Input the structure object assembled by the assembly parts into the input queue of the first analysis unit in the analysis unit combination, that is, the input queue of analysis unit 1 . The analysis unit 1 can periodically obtain a structure object from its own input queue, and read the data to be processed stored in the second part of the structure object, that is, the video; The segment video is decoded to obtain decoded data, and the decoded data is added to the third part of the structure object as the analysis result, and the structure object after adding the decoded data is the output object of the analysis unit 1; the analysis unit 1 will The output object is output to its own output queue.

The output queue of the analysis unit 1 is connected with the input queue of the analysis unit 4 through a pipeline, so that there are one or more output objects of the analysis unit 1 in the input queue of the analysis unit 4 . The analysis unit 4 may periodically acquire an output object of the analysis unit 1 from its own input queue.

The analysis unit 4 reads the decoded data stored in the third part of the output object and the processing progress information of the first part in the output object; Vehicle target, add the detected vehicle target data as the analysis result to the third part of the output object, and the output object after adding the vehicle target data is the output object of the analysis unit 4; the analysis unit 4 outputs the output object to its own output queue.

The output queue of the analysis unit 4 is connected with the input queue of the analysis unit 5 through a pipeline, so that there are one or more output objects of the analysis unit 4 in the input queue of the analysis unit 5 . The analysis unit 5 may periodically acquire an output object of the analysis unit 4 from its own input queue.

The analysis unit 5 reads the vehicle target data stored in the third part of the output object and the processing progress information of the first part of the output object; The license plate number, the license plate number is added to the third part of the output object as the analysis result, and the output object after adding the license plate number is the output object of the analysis unit 5; the analysis unit 5 outputs the output object to its own in the output queue.

The detection accessories in the interactive accessories library can detect the output object of the analysis unit 5, and determine that there are no unrecognized characters in the license plate number; Obtain the license plate number as the processing result, and call the output interface to output the processing result.

If there is an unrecognized character in the license plate number, the detection result does not meet the output condition, and the step of inputting the structure object into the analysis unit 1 is returned to execute until the detection result meets the output condition, or until the number of times of return execution reaches the threshold.

When returning to the step of inputting the structure object into the analysis unit 1, for the output object of the analysis unit 5, the analysis results of each analysis unit stored in the third part of the output object may be deleted, and the first part of the output object may be deleted. The information stored in the server is updated, for example, the processing progress information, the error code information in the processing process, etc. are updated to the initial values, etc., which are not specifically limited. The output object of the analysis unit 5 that has undergone the above processing is input to the analysis unit 1 as a structure object.

Alternatively, when returning to the step of inputting the structure object into the analysis unit 1 , the assembly components may be recalled to regenerate the structure object of the preset structure, and the newly generated structure object may be input into the analysis unit 1 .

After the processing result is output, the delete accessory in the interactive accessory library is called to delete the output object of the analysis unit 5 . In addition, the interactive accessory library and analysis unit can also be uninstalled through the plugin manager. Specifically, the memory space applied for loading the interactive accessory library can be released according to the address handle of the interactive accessory library contained in the configuration file A; the application for loading the analysis unit can be released according to the address handle of the analysis unit contained in the configuration file B memory space.

Applying this embodiment, in the first aspect, the analysis unit is loaded after the task parameters are acquired. The analysis unit does not run continuously as a process, and after the analysis unit is loaded, the thread included in the analysis unit is used to analyze the data. For the solution that the analysis unit runs continuously in a process mode, the resource occupancy rate is reduced, and the thread is easier to control than the process; secondly, the loop mechanism is implemented, that is, when the output object of the last analysis unit does not meet the output conditions In the case of cyclic use of each analysis unit for analysis, until the output object meets the output conditions, or the number of cycles reaches the threshold, the accuracy of the processing results can be improved; thirdly, through two-level loading, the interactive accessories library is loaded separately And analysis unit, load different interactive accessory libraries and analysis units for different data to be processed, and the loading is flexible and adjustable.

Corresponding to the foregoing method embodiments, an embodiment of the present invention further provides a data processing apparatus based on a streaming framework, as shown in FIG. 6 , including:

The first obtaining module 601 is used to obtain task parameters corresponding to the data to be processed;

A determination module 602, configured to determine an analysis unit for processing the data to be processed and an analysis sequence of the analysis unit according to the task parameter; wherein each analysis unit includes an independent thread;

a first loading module 603, configured to load the determined analysis unit;

a connection module 604, configured to connect each of the loaded analysis units according to the analysis sequence to obtain a combination of analysis units;

a second obtaining module 605, configured to obtain the data to be processed;

An analysis module 606, configured to input the data to be processed into the combination of analysis units, and analyze the data to be processed by using a thread included in each analysis unit in the combination of analysis units;

The third obtaining module 607 is configured to obtain the processing result according to the output content of the combination of the analysis units.

As an implementation manner, the first obtaining module 601 may be specifically configured to: obtain a first configuration file corresponding to the data to be processed;

The determining module 602 can be specifically used for:

According to the identification and identification sequence of the analysis units contained in the first configuration file, the analysis units that process the data to be processed and the analysis sequence of the analysis units are determined.

As an embodiment, the device may further include:

a generating module (not shown in the figure), for generating a structure object with a preset structure for the data to be processed;

The analysis module 606 can be specifically used for:

Input the structure object into the first analysis unit in the combination of analysis units, use the thread included in the first analysis unit to analyze the structure object, obtain the analysis result, and add the analysis result to the structure object to obtain the output object of the first analysis unit;

According to the analysis sequence, successively determine each analysis unit after the first analysis unit as the current analysis unit; input the output object of the previous analysis unit of the current analysis unit into the current analysis unit, and use the The thread, analyzes the output object of the last analysis unit, obtains the analysis result, adds the analysis result to the output object of the last analysis unit, and obtains the output object of the current analysis unit;

The third obtaining module 607 can be specifically used for:

The processing result is acquired according to the output object of the last analysis unit in the combination of analysis units.

As an implementation manner, the apparatus may further include: a first adding module and a second adding module (not shown in the figure), wherein,

a first adding module, configured to add the output object of the first analysis unit to the output queue of the first analysis unit;

The analysis module 606 can also be used to sequentially read the output objects of the previous analysis unit from the input queue of the current analysis unit according to the preset queue sequence;

The second adding module is configured to add the output object of the current unit to the output queue of the current analysis unit after the output object of the current analysis unit is obtained.

As an embodiment, the device may further include:

Judgment module (not shown in the figure), for judging whether the output object of the last analysis unit in the analysis unit combination satisfies the output condition; if satisfied, triggers the third acquisition module 607; if not, triggers the analysis module 606;

The third obtaining module 607 may be specifically configured to: obtain the processing result from the output object of the last analysis unit.

As an implementation manner, the apparatus may further include: a fourth obtaining module, a second loading module, an input module and an output module (not shown in the figure), wherein,

a fourth obtaining module, configured to obtain a second configuration file corresponding to the data to be processed;

The second loading module is configured to read the identifier of the interactive accessory library contained in the second configuration file, and load the interactive accessory inventory corresponding to the read identifier, and the interactive accessory library includes an input interface and an output interface. , assembly accessories and testing accessories;

an input module, configured to call the input interface and input the data to be processed into the interactive accessory library;

The generating module may be specifically configured to: assemble the data to be processed by using the assembly accessories to generate a structure object with a preset structure;

The judging module can be specifically used for:

Invoke the detection accessory to detect the output object of the last analysis unit in the analysis unit combination, and determine whether the detection result satisfies the output condition;

The output module is configured to call the output interface to output the processing result after obtaining the processing result from the output object of the last analysis unit.

As an implementation manner, the interactive accessory library further includes deletion accessories, and the apparatus further includes:

A delete module (not shown in the figure) is used to call the delete accessory to delete the output object of the last analysis unit.

As an embodiment, the device may further include:

An unloading module (not shown in the figure) is configured to unload the analysis units in the analysis unit combination after acquiring the processing result according to the output content of the analysis unit combination.

Using the embodiment shown in FIG. 6 of the present invention, after the task parameters are acquired, the analysis unit is loaded again. The analysis unit does not run continuously in a process mode, and after the analysis unit is loaded, the thread included in the analysis unit is used to analyze the data. Compared with the solution in which the analysis unit runs continuously in a process mode, the resource occupancy rate is reduced.

An embodiment of the present invention further provides an electronic device, as shown in FIG. 7 , including a processor 701 and a memory 702,

a memory 702 for storing computer programs;

The processor 701 is configured to implement any of the above data processing methods based on the streaming framework when executing the program stored in the memory 702 .

The memory mentioned in the above electronic device may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; may also be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiment of the data processing apparatus based on the streaming framework shown in FIG. 6 and the embodiment of the electronic device shown in FIG. 7, since they are basically similar to the data processing based on the streaming framework shown in FIGS. 1-5 Therefore, the description is relatively simple, and for related details, please refer to the partial description of the embodiment of the data processing method based on the streaming framework shown in FIG. 1-5 .

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (15)

1. A data processing method based on a streaming framework, comprising:

acquiring task parameters corresponding to data to be processed;

determining an analysis unit for processing the data to be processed and an analysis sequence of the analysis unit according to the task parameters; wherein each analysis unit comprises an independent thread;

loading the determined analysis unit; connecting each loaded analysis unit according to the analysis sequence to obtain an analysis unit combination;

acquiring the data to be processed, inputting the data to be processed into the analysis unit combination, and analyzing the data to be processed by using threads contained in each analysis unit in the analysis unit combination;

acquiring a processing result according to the output content of the analysis unit combination;

after the data to be processed is obtained, the method further comprises the following steps: generating a structural body object with a preset structure aiming at the data to be processed;

the inputting the data to be processed into the analysis unit combination, and analyzing the data to be processed by using the thread included in each analysis unit in the analysis unit combination, includes:

inputting the structural body object into a first analysis unit in the analysis unit combination, analyzing the structural body object by using threads contained in the first analysis unit to obtain an analysis result, and adding the analysis result to the structural body object to obtain an output object of the first analysis unit;

according to the analysis sequence, sequentially determining each analysis unit behind the first analysis unit as a current analysis unit; inputting an output object of a last analysis unit of a current analysis unit into the current analysis unit, analyzing the output object of the last analysis unit by using a thread contained in the current analysis unit to obtain an analysis result, and adding the analysis result to the output object of the last analysis unit to obtain the output object of the current analysis unit;

the acquiring a processing result according to the output content of the analysis unit combination comprises:

and acquiring a processing result according to the output object of the last analysis unit in the analysis unit combination.

2. The method according to claim 1, wherein the acquiring of the task parameter corresponding to the data to be processed includes:

acquiring a first configuration file corresponding to data to be processed;

the determining, according to the task parameter, an analysis unit that processes the data to be processed and an analysis order of the analysis unit includes:

and determining an analysis unit for processing the data to be processed and an analysis sequence of the analysis unit according to the identification and the identification sequence of the analysis unit contained in the first configuration file.

3. The method of claim 1, further comprising, after said obtaining the output object of the first analysis unit:

adding the output object of the first analysis unit to the output queue of the first analysis unit;

the inputting an output object of a last analysis unit of the current analysis unit into the current analysis unit includes:

sequentially reading the output object of the last analysis unit from the input queue of the current analysis unit according to a preset queue sequence;

after the obtaining of the output object of the current analysis unit, the method further includes:

and adding the output object of the current unit to the output queue of the current analysis unit.

4. The method according to claim 1, wherein the obtaining a processing result according to an output object of a last analysis unit in the analysis unit combination comprises:

judging whether the output object of the last analysis unit in the analysis unit combination meets the output condition;

if yes, acquiring a processing result from an output object of the last analysis unit;

and if not, returning to execute the step of inputting the structural body object into the first analysis unit in the analysis unit combination.

5. The method of claim 4, further comprising, prior to obtaining the data to be processed: acquiring a second configuration file corresponding to the data to be processed;

reading an identifier of an interactive accessory library contained in the second configuration file, and loading an interactive accessory inventory corresponding to the read identifier, wherein the interactive accessory library comprises an input interface, an output interface, an assembly accessory and a detection accessory;

after the data to be processed is obtained, the method further comprises the following steps:

calling the input interface, and inputting the data to be processed into the interactive accessory library;

the generating a structural body object with a preset structure aiming at the data to be processed comprises the following steps:

assembling the data to be processed by using the assembly accessories to generate a structural body object with a preset structure;

the judging whether the output object of the last analysis unit in the analysis unit combination meets the output condition includes:

calling the detection accessory, detecting an output object of the last analysis unit in the analysis unit combination, and judging whether a detection result meets an output condition;

after obtaining the processing result from the output object of the last analysis unit, the method further includes:

and calling the output interface and outputting the processing result.

6. The method of claim 5, wherein the interactive accessory library further comprises a delete accessory, and after the processing result is output by calling the output interface, further comprises:

and calling the deletion accessory to delete the output object of the last analysis unit.

7. The method of claim 1, further comprising, after obtaining a processing result based on the output content of the analysis unit combination:

unloading the analysis units in the analysis unit combination.

8. A streaming framework-based data processing apparatus, comprising:

the first acquisition module is used for acquiring task parameters corresponding to data to be processed;

the determining module is used for determining an analysis unit for processing the data to be processed and an analysis sequence of the analysis unit according to the task parameters; wherein each analysis unit comprises an independent thread;

the first loading module is used for loading the determined analysis unit;

the connection module is used for connecting each loaded analysis unit according to the analysis sequence to obtain an analysis unit combination;

the second acquisition module is used for acquiring the data to be processed;

the analysis module is used for inputting the data to be processed into the analysis unit combination and analyzing the data to be processed by utilizing threads contained in each analysis unit in the analysis unit combination;

the third acquisition module is used for acquiring a processing result according to the output content of the analysis unit combination;

the generating module is used for generating a structural body object with a preset structure aiming at the data to be processed;

the analysis module is specifically configured to:

inputting the structural body object into a first analysis unit in the analysis unit combination, analyzing the structural body object by using threads contained in the first analysis unit to obtain an analysis result, and adding the analysis result to the structural body object to obtain an output object of the first analysis unit;

according to the analysis sequence, sequentially determining each analysis unit behind the first analysis unit as a current analysis unit; inputting an output object of a last analysis unit of a current analysis unit into the current analysis unit, analyzing the output object of the last analysis unit by using a thread contained in the current analysis unit to obtain an analysis result, and adding the analysis result to the output object of the last analysis unit to obtain the output object of the current analysis unit;

the third obtaining module is specifically configured to:

and acquiring a processing result according to the output object of the last analysis unit in the analysis unit combination.

9. The apparatus of claim 8, wherein the first obtaining module is specifically configured to: acquiring a first configuration file corresponding to data to be processed;

the determining module is specifically configured to:

and determining an analysis unit for processing the data to be processed and an analysis sequence of the analysis unit according to the identification and the identification sequence of the analysis unit contained in the first configuration file.

10. The apparatus of claim 8, further comprising: a first adding module and a second adding module, wherein,

a first adding module, configured to add the output object of the first analysis unit to the output queue of the first analysis unit;

the analysis module is further configured to sequentially read an output object of the previous analysis unit from an input queue of the current analysis unit according to a preset queue order;

and the second adding module is used for adding the output object of the current unit into the output queue of the current analysis unit after the output object of the current analysis unit is obtained.

11. The apparatus of claim 8, further comprising:

the judging module is used for judging whether the output object of the last analysis unit in the analysis unit combination meets the output condition; if yes, triggering the third acquisition module; if not, triggering the analysis module;

the third obtaining module is specifically configured to: and acquiring a processing result from an output object of the last analysis unit.

12. The apparatus of claim 11, further comprising: a fourth obtaining module, a second loading module, an input module and an output module, wherein,

the fourth acquisition module is used for acquiring a second configuration file corresponding to the data to be processed;

the second loading module is used for reading the identification of the interactive accessory library contained in the second configuration file and loading the interactive accessory inventory corresponding to the read identification, wherein the interactive accessory library comprises an input interface, an output interface, an assembly accessory and a detection accessory;

the input module is used for calling the input interface and inputting the data to be processed into the interactive accessory library;

the generation module is specifically configured to: assembling the data to be processed by using the assembly accessories to generate a structural body object with a preset structure;

the judgment module is specifically configured to:

calling the detection accessory, detecting an output object of the last analysis unit in the analysis unit combination, and judging whether a detection result meets an output condition;

and the output module is used for calling the output interface and outputting the processing result after the processing result is obtained from the output object of the last analysis unit.

13. The apparatus of claim 12, wherein the library of interaction accessories further comprises a delete accessory, the apparatus further comprising:

and the deleting module is used for calling the deleting accessory and deleting the output object of the last analysis unit.

14. The apparatus of claim 8, further comprising:

and the unloading module is used for unloading the analysis units in the analysis unit combination after acquiring the processing result according to the output content of the analysis unit combination.

15. An electronic device comprising a processor and a memory;

a memory for storing a computer program;

a processor for implementing the method steps of any of claims 1 to 7 when executing a program stored in the memory.

Images