CN115604282A - Resource data processing method and device, electronic equipment, storage medium - Google Patents

Abstract

The present application discloses a resource data processing method and device, electronic equipment, and a storage medium. The method includes: if the amount of data requested by the received resource request exceeds a preset quantity threshold, obtaining the resource usage of the processing task in the background Data; according to the resource usage data, detect the relationship between the resources of the background processing task and the resources required by the resource request; if the relationship indicates that the resources of the background processing task are less than the resources required by the resource request, the tasks corresponding to the resource request are processed in batches, Get multiple processing files and send them to the initiator corresponding to the resource request. For resource requests whose requested data volume exceeds the preset quantity threshold, this application processes the tasks corresponding to the resource requests in batches by monitoring the resource usage data in the background in real time, ensuring the integrity of the requested data and avoiding background problems. Loading a large amount of data leads to resource overflow, which affects system stability.

Description

Method and device for processing resource data, electronic equipment, and storage medium

technical field

The present application relates to the technical field of data processing, and in particular to a resource data processing method and device, electronic equipment, and a storage medium.

Background technique

In the big data management platform, it is often necessary to export different data based on resource requests to achieve two-way communication. However, when the resource request requests to export a large amount of data, the background needs to consume a long time to calculate and occupy a large amount of memory resources, causing the user to wait for a long time or even time out, and loading a large amount of data into the memory will also cause memory overflow, resulting in unstable system performance .

Contents of the invention

In order to solve the above technical problems, the embodiments of the present application provide a method and device for processing resource data, electronic equipment, a computer-readable storage medium, and a computer program product.

According to an aspect of an embodiment of the present application, a method for processing resource data is provided, including:

If the amount of data requested by the received resource request exceeds the preset quantity threshold, obtain the background resource usage data for the processing task; according to the resource usage data, detect the resources required by the background processing task and the resource request The relationship between resources; if the relationship indicates that the resources of the background processing task are less than the resources required by the resource request, then the tasks corresponding to the resource request are processed in batches to obtain multiple processing files; the multiple processing files are obtained A processing file is sent to the originator corresponding to the resource request.

According to an aspect of the embodiment of the present application, a device for processing resource data includes: an acquisition unit, configured to acquire background resource usage for processing tasks if the amount of data requested by the received resource request exceeds a preset quantity threshold data; a detection unit, configured to detect the relationship between the resources of the background processing task and the resources required by the resource request according to the resource usage data; a batching unit, configured to if the relationship represents the background processing task If the resource is less than the resources required by the resource request, the tasks corresponding to the resource request are processed in batches to obtain multiple processing files; the feedback unit is used to send the multiple processing files to the resource corresponding to the resource request the originator.

In another exemplary embodiment, the resource usage data includes the total resources of the background and the resources occupied by the current processing task; and according to the resource usage data, the resources of the background processing task and the resource request are detected The relationship of required resources includes: obtaining currently available resources based on the total resources and the occupied resources; detecting the relationship between the available resources and the resources required by the resource request.

In another exemplary embodiment, the detecting the relationship between the available resources and the resources required by the resource request includes: adjusting the resource amount of the available resources to obtain a target resource amount threshold; wherein, the The target resource threshold is less than the resource amount of the available resources; obtaining the resource amount of the resource required by the resource request; detecting the relationship between the target resource amount threshold and the resource amount of the resource required by the resource request.

In another exemplary embodiment, the available resources are available memory, and the resource request carries query parameters; the detecting the relationship between the available resources and the resources required by the resource request includes: Query the data to be exported that matches the query parameters, and determine the space occupied by a single piece of data from the database; based on the amount of data corresponding to the data to be exported and the space occupied by the single piece of data, obtain The space requirement of the resource request; detecting the size relationship between the available memory and the space requirement.

In another exemplary embodiment, the available resources are available memory, and the resources required by the resource request are space requirements; the tasks corresponding to the resource requests are processed in batches to obtain multiple processing files , comprising: dividing the space requirement based on the available memory to obtain a plurality of subspace requirements; processing the data to be exported corresponding to the resource request in batches according to the plurality of subspace requirements, Obtain the plurality of processing files.

In another exemplary embodiment, the resource request carries an identifier; before acquiring background resource usage data for processing tasks if the amount of data requested by the received resource request exceeds a preset quantity threshold, The method includes: receiving the resource request sent by the initiator; detecting the type of the identifier contained in the resource request; if the first type of identifier is detected, determining the data corresponding to the resource request The amount exceeds the preset number threshold; if the second type of identification is detected, it is determined that the data amount corresponding to the resource request does not exceed the preset number threshold.

In another exemplary embodiment, the sending the plurality of processing files to the initiator corresponding to the resource request includes: respectively compressing the plurality of processing files using a specified compression algorithm to obtain a plurality of compressed file; sending multiple compressed files to the originator corresponding to the resource request.

According to an aspect of the embodiment of the present application, an electronic device includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more When the processor executes, the electronic device realizes the resource data processing method as described above.

According to an aspect of the embodiments of the present application, a computer-readable storage medium stores computer-readable instructions thereon, and when the computer-readable instructions are executed by a computer processor, the computer executes the above-mentioned resource data processing method.

According to an aspect of the embodiments of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the resource data processing method provided in the above-mentioned various optional embodiments.

In the technical solution provided by the embodiments of the present application, when the amount of data requested by the received resource request exceeds the preset quantity threshold, the resource usage data of the background for the processing task is obtained by monitoring the background resources in real time; according to Resource usage data, which detects the relationship between the resources of the background processing task and the resources required by the resource request; if the relationship indicates that the resources of the background processing task are less than the resources required by the resource request, it means that the amount of data requested by the resource request not only exceeds the preset quantity threshold, but also The size of background resources makes it impossible for the background to send all the requested data concurrently, resulting in incomplete return data. Therefore, the tasks corresponding to resource requests are processed in batches, and multiple processing files are obtained and sent to the originator corresponding to resource requests. In this way, this application monitors the resource usage data in the background in real time for resource requests whose requested data volume exceeds the preset quantity threshold, and when it is detected that the resources of the background processing task are less than the resources required by the resource request, the task corresponding to the resource request is classified. Batch processing ensures the integrity of the data requested by resource requests, and at the same time avoids resource overflow due to loading a large amount of data in the background, which in turn affects system stability.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application. Apparently, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative efforts. In the attached picture:

FIG. 1 is a schematic diagram of an implementation environment involved in the present application;

Fig. 2 is a schematic diagram between the background, the terminal and the user of the application of the resource data processing method in an exemplary embodiment under the implementation environment shown in Fig. 1;

Fig. 3 is a flowchart of a resource data processing method shown in an exemplary embodiment of the present application;

FIG. 4 is a flowchart of step S302 in the embodiment shown in FIG. 3 in an exemplary embodiment;

FIG. 5 is a flowchart of step S302 in the embodiment shown in FIG. 3 in an exemplary embodiment;

Fig. 6 is a flowchart in an exemplary embodiment of the steps of judging the execution conditions before step S302 in the embodiment shown in Fig. 3;

FIG. 7 is a flowchart of step S302 in the embodiment shown in FIG. 3 in an exemplary embodiment;

FIG. 8 is a flowchart of step S303 in the embodiment shown in FIG. 3 in an exemplary embodiment;

Fig. 9 is a flowchart in an exemplary embodiment of the steps of receiving and identifying resource requests before step S301 in the embodiment shown in Fig. 3;

Fig. 10 is a flowchart of a resource data processing method shown in another exemplary embodiment of the present application;

Fig. 11 is a block diagram of a device for processing resource data shown in an exemplary embodiment of the present application;

Fig. 12 is a schematic structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application.

detailed description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present application as recited in the appended claims.

The block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices entity.

The flow charts shown in the drawings are only exemplary illustrations, and do not necessarily include all contents and operations/steps, nor must they be performed in the order described. For example, some operations/steps can be decomposed, and some operations/steps can be combined or partly combined, so the actual order of execution may be changed according to the actual situation.

The "plurality" mentioned in this application refers to two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships. For example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. The character "/" generally indicates that the contextual objects are an "or" relationship.

In related technologies, a user can initiate a resource request by displaying a web page, and the background website and its system will query, process and export corresponding data based on the resource request. Usually the resource request requests to export a large amount of data. At this time, the background needs to take a long time to calculate and occupy a large amount of memory resources to respond to the resource request and return the corresponding large amount of data, which will cause the user to wait for a long time or even time out. , and loading a large amount of data into the memory will also cause memory overflow, making the performance of the background system unstable.

At present, the problem of memory overflow in the background is mostly solved by limiting the parameter range of resource requests or limiting the amount of data returned in a single time. However, the method of limiting the parameter range of resource requests requires artificially performing multiple batch operations, which is operable. And the problem of poor experience; limiting the amount of data returned at a time may have problems such as incomplete returned data.

In order to solve the above problems, the embodiments of the present application propose a method and device for processing resource data, electronic equipment, and a computer-readable storage medium, which mainly involve data processing technology, and these embodiments will be described in detail below.

First please refer to FIG. 1 , which is a schematic diagram of an implementation environment involved in the present application. The implementation environment includes a terminal 10 and a background 20, and the terminal 10 and the background 20 communicate through a wired or wireless network.

The background 20 is used to perform background processing on the resource request sent by the user through the display web page of the terminal 10 as the originator, by monitoring the background resource in real time when the amount of data requested by the received resource request exceeds the preset quantity threshold , to obtain background resource usage data for processing tasks; according to the resource usage data, detect the relationship between the resources required by the background processing task and the resources required by the resource request; The amount of requested data not only exceeds the preset quantity threshold, but also the size of background resources makes it impossible for the background to concurrently send all the requested data, resulting in incomplete data returned to the initiator, so the tasks corresponding to resource requests are processed in batches , obtain multiple processing files and send them to the originator corresponding to the resource request, that is, return the multiple processing files to the terminal 10 for display.

Please refer to FIG. 2. FIG. 2 is a schematic diagram between the background 20, the terminal 10 and the user of the resource data processing method application in an exemplary embodiment under the implementation environment shown in FIG. 1, wherein the background 20 includes interception Detector 21, listener 22, evaluation module 23 and business processing module 24. There is interaction between the user and the display interface of the terminal 10, and there is interaction between the terminal 10 and the background 20. After the user logs in on the page of the terminal 10, the resource request is sent through the terminal 10, and the interceptor 21 of the background 20 intercepts and analyzes the resource request. If the amount of data requested by the received resource request exceeds the preset quantity threshold, the resource request is sent to the estimation module 23, and at the same time, the resource usage data of the current background for processing tasks is obtained through the listener 22, and the resource request is obtained through the estimation module 23 After the required resources are obtained, the relationship between the resources of the background processing task and the resources required by the resource request is obtained; when the resources of the background processing task are less than the resources required by the resource request, the business processing module 24 processes the tasks corresponding to the resource request in batches, A plurality of processed files are obtained and pushed to the terminal 10 via the interceptor 21 for display.

Compared with the prior art solution for sorting out resource requests, the resource data processing method provided by this implementation environment can ensure the integrity of the requested data and avoid background problems caused by loading a large amount of data. lead to resource overflow.

It should be noted that the terminal 10 in the implementation environment shown in FIG. 1 can be any electronic device such as a smart phone, a tablet, a notebook computer, or a computer; the background 20 can be an independent server, or can provide cloud services, cloud databases, Cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (Content Delivery Network, CDN), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms, There are no restrictions here.

Fig. 3 is a flowchart of a resource data processing method shown in an exemplary embodiment of the present application. The method can be applied to the implementation environment shown in FIG. 1 , and is specifically executed by the background 20 in the embodiment environment shown in FIG. 1 . However, in other implementation environments, the method may be executed by devices in other implementation environments, which is not limited in this embodiment.

As shown in FIG. 3, in an exemplary embodiment, the resource data processing method may include steps S301 to S304, which are described in detail as follows:

Step S301, if the amount of data requested by the received resource request exceeds a preset amount threshold, acquire background resource usage data for processing tasks.

Among them, the preset quantity threshold is used to monitor the size of the data requested by the resource request. The interceptor configuration monitors and analyzes the preset quantity value, and intercepts resource requests whose requested data volume exceeds the preset quantity threshold, so that The resource request will not directly enter the business processing module, avoiding resource overflow due to loading a large amount of data in the background, resulting in incomplete returned data. Wherein, the specific value of the preset quantity threshold includes, but is not limited to, it can be set according to requirements.

And the background will monitor the resource usage data of the processing tasks in the background in real time. In this way, when intercepting resource requests whose requested data volume exceeds the preset quantity threshold, the current background resource usage data for processing tasks can be obtained. The resource usage data can be Including information such as the total amount of resources, the amount of resources occupied, and the amount of available resources.

Step S302 , according to the resource usage data, the relationship between the resource of the background processing task and the resource required by the resource request is detected.

After it is determined that the amount of data requested by the received resource request exceeds the preset amount threshold, the data amount of the data requested by the resource request is acquired, and then the resources required to process the data of the amount of data are obtained. After obtaining the resources required by the resource request, by detecting the relationship between the resources required by the background processing task and the resources required by the resource request, the detected relationship can be characterized as the resources required for the background processing task being less than the required resources for the resource request, or as the background The resource of the processing task is greater than or equal to the required resource of the resource request. Through the content represented by the relationship between the resource of the background processing task and the required resource of the resource request, it can be determined whether the background can concurrently request all the data requested by the resource.

Step S303, if the relationship represents that the resource of the background processing task is less than the resource required by the resource request, the tasks corresponding to the resource request are processed in batches to obtain multiple processing files.

If the above relationship is characterized by the fact that the resources of the background processing task are greater than or equal to the resources required by the resource request, it means that the resources of the background processing task can meet the conditions of all the data requested by the concurrent resource request, and the background directly concurrently processes the task corresponding to the resource request. It will lead to resource overflow, and the feedback data will not be incomplete, so there is no need to process the tasks corresponding to the resource request in batches, and the resource request can be directly transmitted to the business processing module to complete the business processing.

If the resource of the background processing task is less than the resource required by the resource request, it means that the resource of the background processing task cannot meet the conditions of all the data requested by the concurrent resource request. Therefore, in order to avoid resource overflow due to loading a large amount of data in the background, return In the case of incomplete data, the tasks corresponding to resource requests are processed in batches, and when the tasks in each batch are processed in the background, all tasks in the batch can be implemented concurrently without causing resource overflow. After processing, each batch corresponds to a processing file, so that after the tasks corresponding to the resource request are processed in batches, multiple processing files are obtained.

Step S304, sending multiple processing files to the originator corresponding to the resource request.

After processing each batch of tasks corresponding to the resource request, feed back the obtained multiple processing files, that is, send them to the originator corresponding to the resource request, so as to complete the processing of the resource data requested by a resource request.

It can be seen from the above that in the method provided by this embodiment, the resource usage data of the background for the processing task is obtained in real time by monitoring the background resources in real time; when it is detected that the amount of data requested by the received resource request exceeds the preset quantity threshold , according to the current resource usage data, detect the relationship between the resources of the background processing task and the resources required by the resource request; if the relationship indicates that the resources of the background processing task are less than the resources required by the resource request, it means that the amount of data requested by the resource request not only exceeds the preset The number threshold, and the size of the background resources make it impossible for the background to concurrently send all the requested data. If it is processed, it is easy to only want to send out and the feedback data is incomplete. Therefore, the tasks corresponding to the resource requests are processed in batches, so that the background can be processed smoothly. Complete data is fed back to each batch of tasks, and then multiple processing files obtained by batch processing are sent to the originator corresponding to the resource request. In this way, while ensuring the integrity of the data requested by the resource request, it avoids resource overflow due to loading a large amount of data in the background, thereby affecting system stability.

Please refer to FIG. 4. FIG. 4 is a flow chart of step S302 in the embodiment shown in FIG. 3 in an exemplary embodiment, wherein the acquired background resource usage data for processing tasks includes total background resources and current processing tasks resources that are occupied. As shown in FIG. 4 , step S302 may specifically include steps S401 to S402. Through the above steps, the relationship between the resources of the background processing task and the resources required by the resource request is detected. The details are as follows:

Step S401, obtain currently available resources based on total resources and occupied resources.

The currently available resources are calculated based on the total resources of the background included in the acquired resource usage data for processing tasks in the background and the resources occupied by the current processing task. The total resources in the background and the resources occupied by the current processing task can be obtained by intuitive detection. Therefore, in order to achieve efficient processing, after obtaining the heavy resources and occupied resources, the current available resources can be obtained through simple calculations, and the application into the processing of resource requests.

Step S402, detecting the relationship between the available resources and the resources required by the resource request.

By acquiring the data volume of the data requested by the resource request, the resources required to process the data of the data volume are obtained. When the resource request is actually processed in the background, the resource request is processed through the idle resources in the background. Therefore, in order to determine whether the background can concurrently request all the data requested by the resource, it is necessary to detect the background processing task resources and resource requests. Resource relationship, which specifically detects the relationship between available resources and resources required by resource requests, directly reflects whether the background can concurrently request all the data requested by resource requests.

The detected relationship can be characterized as the background available resources are less than the required resources of the resource request, and can also be characterized as the background available resources are greater than or equal to the required resources of the resource request, through the relationship between the background available resources and the required resources of the resource request The content of the representation can determine whether the idle resources in the background can concurrently request all the data requested by the resource.

In this way, the technical solution of the above embodiment of the present application obtains the currently available resources according to the total background resources and the occupied resources, and then according to the relationship between the currently available resources idle in the background and the resources required by the resource request, it is determined whether it is necessary to match the source request The tasks are processed in batches to avoid resource overflow caused by loading a large amount of data in the background.

In another embodiment of the present application, as shown in Figure 5, in order to prevent data saturation of background resources, the target resource threshold can be obtained based on the available resources in the background, and the relationship between the target resource amount threshold and the resource amount required by the resource request can be detected . As shown in FIG. 5, step S302 in the embodiment shown in FIG. 3 may specifically include steps S501 to S502 in an exemplary embodiment, and the details are as follows:

Step S501, adjusting the resource amount of available resources to obtain a target resource amount threshold.

Since the available resources in the background are obtained through simple calculations of the total resources and the occupied resources, the available resources represent the idle resources in the background. If the available resources are exactly equal to the resources required by the resource request, only the available resources will be used. Detecting the relationship between the available resources and the resources required by the resource request will cause all the remaining idle resources in the background to process the resources requested by the resource request, resulting in low fault tolerance of the system and prone to data processing failures.

Therefore, in this embodiment of the present application, the resource amount of the available resources is adjusted to obtain the target resource amount threshold, wherein the target resource threshold is smaller than the resource amount of the available resources, and the method of adjusting the resource amount of the available resources may be based on the implementation setting The adjustment coefficient of less than one can also be adjusted by using a fixed difference, as long as the adjusted target resource threshold is less than the resource amount of available resources, there is no limitation on the adjustment method here.

Step S502, acquiring the resource amount of the resource required by the resource request, and detecting the relationship between the target resource amount threshold and the resource amount of the resource required by the resource request.

By acquiring the data amount of the data requested by the resource request, the resources required to process the data of the data amount are obtained, and then the relationship between the target resource amount threshold and the resource amount of the resource required by the resource request is detected. The detected relationship can be characterized as the target resource amount threshold is less than the resource required by the resource request, and can also be characterized as the target resource amount threshold is greater than or equal to the required resource of the resource request, and the relationship between the target resource amount threshold and the required resource of the resource request The content represented by the relationship.

In this way, in this embodiment, since the target resource threshold is less than the resource amount of available resources, when the detected relationship indicates that the target resource amount threshold is greater than or equal to the resource required by the resource request, all available resources remaining in the background will not appear. Resource requests the requested resource for processing. In this way, based on the target resource threshold, it is determined whether the background idle resources can concurrently request all the data requested by the resource, and it avoids resource overflow caused by loading a large amount of data in the background, avoids background data saturation, and improves the fault tolerance of data processing.

In another exemplary embodiment of the present application, the background resources available for obtaining the relationship with the resources required by the resource request are memory resources. As shown in FIG. 6, the background resources of the present application also include the number of CPU cores, so The resource usage data for processing tasks in the background also includes the total number of CPU cores and the number of occupied CPU cores. And before step S302 in the embodiment shown in FIG. 3 detects the relationship between the resource of the background processing task and the resource required by the resource request based on the resource usage data of the memory resource in step S302, a step of judging the execution condition is also included. The execution condition is related to the CPU Correspondence to the number of cores, as shown in Figure 6, specifically includes steps S601 to S602:

In step S601, the number of currently available CPU cores is obtained based on the total number of CPU cores and the number of occupied CPU cores, and an execution condition is determined based on the number of currently available CPU cores.

After it is determined that the amount of data requested by the received resource request exceeds the preset quantity threshold, the background will add the task corresponding to the resource request to the task queue in the background. In addition to the task corresponding to the newly added resource request in the new queue, It also includes tasks being executed, and the number of CPU cores occupied by the tasks being executed is the number of CPU cores occupied by the current processing task included in the obtained resource usage data.

The background resource usage data for processing tasks obtained through the listener includes the total number of CPU cores in the background and the number of CPU cores occupied by the current processing task, and the current available CPU is calculated based on the total number of CPU cores and the number of occupied CPU cores The number of cores. In order to ensure the normal operation of the background, the execution condition is determined based on the number of currently available CPU cores. The execution condition is characterized by the task corresponding to the resource request being less than or equal to the current available concurrent CPU core number, that is, the total number of all tasks in the task queue must be less than or equal to The total number of CPU cores.

Step S602, if the resource request satisfies the execution condition, then detect the relationship between the available resources and the resources required by the resource request.

If the resource request does not meet the execution conditions, it means that the task corresponding to the resource request is greater than the current available concurrent CPU cores, and the background CPU resources cannot fully execute the task corresponding to the resource request. Therefore, it is determined that the execution condition is not met and the resource is stopped. Identification and processing of requests, and error feedback to the originator of the resource request; if the resource request meets the execution conditions, then detect the relationship between the available resources and the resources required by the resource request, and determine whether the tasks corresponding to the resource request need to be batched processing.

In the solution provided by this embodiment, before detecting the relationship between the available resources and the resources required by the resource request, it is determined whether the resource request satisfies the execution conditions corresponding to the number of background CPU cores to ensure the normal operation of the background.

As shown in FIG. 7 , in another exemplary embodiment of the present application, the available background resource for obtaining the relationship with the resource required by the resource request is a memory resource. Please refer to FIG. 7. FIG. 7 is a flowchart of step S302 in an exemplary embodiment in the embodiment shown in FIG. Previously, the step of obtaining the resources required by the resource request through the query parameters carried in the resource request is also included. As shown in FIG. 7, step S302 may specifically include steps S701 to S703, through the above steps to obtain the resources required for the resource request, the details are as follows:

Step S701, querying the database for the data to be exported that matches the query parameters, and determining the space occupied by a single piece of data from the database.

The query parameters carried in the resource request correspond to the time range of the resource request, specific data types and other information. Based on the query parameters, the estimation module queries from the database and obtains the data to be exported that matches the query attempt. The data to be exported is represented as for the background Data that is fed back to the originator of the resource request after processing.

And determine the space size occupied by a single piece of data from the database. In this embodiment, the space size occupied by a single piece of data can be obtained based on the size of all data in the database. Specifically, the size of each data in the database is obtained to obtain the data in the database. Size range, take the maximum value in the size range as the space occupied by a single piece of data that needs to be determined in this application.

Step S702, based on the amount of data corresponding to the data to be exported and the space occupied by a single piece of data, the space requirement of the resource request is obtained.

After obtaining the amount of data corresponding to the data to be exported and the space occupied by a single piece of data, the space requirement of the resource request can be calculated. Specifically, the resource request can be obtained from the space occupied by a single piece of data * the amount of data space requirements.

Step S703, detecting the size relationship between the available memory and the space requirement.

Since the available background resources used to obtain the relationship with the resources required by the resource request are memory resources, the detection object is the available memory. Therefore, in order to determine whether the background can concurrently request all the data requested by the resource, the resource of the background processing task is detected and The relationship between resources required by resource requests, which specifically detects the relationship between available memory and space requirements, directly reflects whether the background can concurrently request the requested data to be exported. The detected relationship can be represented as the background available memory is less than the space requirement of the resource request, and can also be represented as the background available memory is greater than or equal to the resource request space requirement, through the relationship between the background available memory and the resource request space requirement The content of the representation can determine whether the background idle resources can concurrently request all the requested data to be exported.

In this embodiment, according to the space occupied by a single piece of data in the database and the data to be exported corresponding to the resource request, the space requirement corresponding to the data to be exported corresponding to the resource request is obtained, and the size relationship between the space requirement and the possible memory in the background Perform detection and analysis to visually determine whether the idle resources in the background can concurrently request all the requested data to be exported.

In an exemplary embodiment of the present application, in the resource data processing method provided in the present application, available resources are available memory, and resources required for resource requests are space requirements. Based on this, in another embodiment of the present application, please refer to FIG. 8 , which is a flowchart of step S303 in the embodiment shown in FIG. 3 in an exemplary embodiment. As shown in Figure 8, step S303 may include steps S801 to step S802, and a plurality of processing files are obtained through the above steps, which are described in detail as follows:

In step S801, the space requirement is divided based on available memory to obtain a plurality of subspace requirements.

When the size relationship between the available memory and the space requirement is characterized by the fact that the background available memory is less than the space requirement of the resource request, it means that the memory resources of the background processing task cannot meet the conditions of all the data to be exported requested by the concurrent resource request, so In order to avoid incomplete returned data due to resource overflow caused by loading a large amount of data in the background, it is necessary to process tasks corresponding to resource requests in batches.

In this embodiment, the manner of dividing the task batch corresponding to the resource request is to divide the space requirement based on the specific size of the available memory to obtain multiple subspace requirements.

Step S802, process the data to be exported corresponding to the resource request in batches according to the requirements of multiple subspaces, and obtain multiple processing files.

Among them, the specific size of each subspace requirement is less than or equal to the size of the available memory. Based on this, the data to be exported corresponding to the resource request is processed in batches according to the requirements of multiple subspaces, and multiple processing files are obtained. It can make the data to be exported in each batch processed in the background concurrently without causing memory resource overflow.

In the above embodiments, the space requirements are divided by the available memory to obtain multiple subspace requirements, so as to specifically realize the task of processing the tasks corresponding to the resource requests in batches and obtain multiple processing files. In each batch of resource requests When the corresponding data to be exported is processed, the available memory in the background can be completed without resource overflow, and after a batch of processing, the available memory in the background will be free for the next batch of resource requests The corresponding data to be exported is processed, that is, the available memory is reused until all batches are processed and multiple processed files are obtained.

In an exemplary embodiment of the present application, the resource request carries an identifier, which can be used to identify the type of the resource request. Please refer to FIG. 9. FIG. 9 is a flowchart in an exemplary embodiment of the steps of receiving and identifying a resource request before step S301 in the embodiment shown in FIG. 3, which includes steps S901 to S904:

Step S901, receiving a resource request sent by an initiator.

The background receives the resource request sent by the user through the display webpage of the terminal as the initiator. The resource request indicates that the initiator requests background feedback and processes the resource data corresponding to the resource request. Different resource requests request different resource data. , including different data content and different data volumes.

In step S902, the type of the identifier contained in the resource request is detected.

Different resource requests carry different identifiers, and the identifiers represent the types of the resource requests. In this embodiment, the resource request types may include the first type and the second type. Therefore, after obtaining the resource request sent by the originator, the background recognizes the identifier contained in the resource request and can detect the type of the resource request.

In step S903, if the first type of identifier is detected, it is determined that the amount of data corresponding to the resource request exceeds a preset amount threshold.

Among them, the preset quantity threshold is a preset value, which is used to indicate that the data volume corresponding to the resource request does not exceed the preset quantity threshold, and the resource request is determined to be a request for small data volume data, and if it exceeds the preset quantity threshold, it is determined that the The resource request is a request for a large amount of data. It should be noted that the above-mentioned large and small are relative to a preset quantity threshold.

In this embodiment of the present application, the first type is defined as a resource request corresponding to a data volume of requested feedback exceeding a preset quantity threshold, that is, the resource request is a request for a large amount of data. In this way, after the first type is detected and identified, it is determined that the amount of data corresponding to the resource request exceeds the preset quantity threshold, and the interceptor will intercept the resource request and send it to the estimation module.

In step S904, if the second type of identifier is detected, it is determined that the amount of data corresponding to the resource request does not exceed a preset amount threshold.

In this embodiment of the present application, the second type is defined as the data volume of the requested feedback data corresponding to the resource request does not exceed the preset quantity threshold, that is, the resource request is a request for data with a small data volume. In this way, after detecting and identifying the identification as the second type, it is determined that the amount of data corresponding to the resource request does not exceed the preset quantity threshold, and the interceptor will directly transmit the resource request to the business processing module to complete the corresponding business processing.

Thus, through the method of this embodiment, this application determines whether the amount of data corresponding to the resource request exceeds the preset quantity threshold according to the identification carried by the resource request, and if it exceeds the preset quantity threshold, then apply the resource request provided by this application. The processing method of the resource data, when it is detected that the relationship between the resource of the background processing task and the resource required by the resource request is less than that, the task corresponding to the resource request is processed in batches, and the integrity of the data requested by the resource request is guaranteed. At the same time, it avoids the overflow of resources in the background due to loading a large amount of data, which in turn affects the stability of the system.

Please refer to FIG. 10. FIG. 10 is a flowchart of a resource data processing method shown in another exemplary embodiment of the present application. The difference between the solution of this embodiment and the embodiment shown in FIG. Before reaching the initiator, there is also a step of compressing the processing file. Therefore, in addition to steps S301 to S303, this embodiment also includes steps S1001 to S1002. The detailed description of steps S1001 to S1002 is as follows:

Step S1001, using a specified compression algorithm to compress a plurality of processed files respectively to obtain a plurality of compressed files.

For the obtained multiple processing files, in order to ensure the integrity of the feedback data, the multiple processing files are compressed before being sent to the resource request initiator to further reduce the size of the returned data, and sending after compression can avoid Loss or corruption of files. Wherein, the specified compression algorithm can be set according to requirements, or a common compression method can be adopted, which is not limited here.

Step S1002, sending multiple compressed files to the originator corresponding to the resource request.

Multiple compressed files obtained by using the specified compression algorithm can be directly sent to the originator corresponding to the resource request in the form of a file, or can be returned to the originator after generating a download link. The specific method of return is not limited here. The party's display page performs a download action to view the request result.

In this embodiment, multiple processed files are compressed to form multiple compressed files and sent to the originator corresponding to the resource request, reducing the overall data volume returned, further ensuring the integrity of the returned data volume, and sending data after compression. Avoid file loss or damage.

Fig. 11 is a block diagram of an apparatus 1100 for processing resource data according to an exemplary embodiment of the present application. As shown in Figure 11, the device includes:

An acquisition unit 1101, configured to acquire background resource usage data for processing tasks if the amount of data requested by the received resource request exceeds a preset quantity threshold;

The detection unit 1102 is configured to detect the relationship between the resources of the background processing task and the resources required by the resource request according to the resource usage data;

The batching unit 1103 is configured to process the tasks corresponding to the resource request in batches to obtain multiple processing files if the resource representing the background processing task is less than the resource required by the resource request;

The feedback unit 1104 is configured to send multiple processing files to the originator corresponding to the resource request.

The device applies the resource data processing method provided by this application, and obtains background resources for processing tasks through the real-time monitoring of background resources by the acquisition unit 1101 when the amount of data requested by the received resource request exceeds the preset quantity threshold use data; the detection unit 1102 detects the relationship between the resources of the background processing task and the resources required by the resource request according to the resource use data; Exceeding the preset quantity threshold, and the size of the background resource makes it impossible for the background to concurrently send all the requested data, resulting in incomplete returned data. Therefore, the batching unit 1103 processes the tasks corresponding to the resource request in batches to obtain multiple processing The file is sent to the initiator corresponding to the resource request through the feedback unit 1104. In this way, the integrity of the data requested by the resource request is guaranteed, and at the same time, the resource overflow due to loading a large amount of data in the background is avoided, thereby affecting the system stability.

In another exemplary embodiment, the device also includes:

The type identification unit is configured to receive the resource request sent by the initiator; detect the type of the identifier contained in the resource request; if the first type of identifier is detected, determine that the amount of data corresponding to the resource request exceeds the preset quantity threshold; If an identifier of the second type is detected, it is determined that the amount of data corresponding to the resource request does not exceed the preset amount threshold.

In another exemplary embodiment, the detection unit 1102 is further configured to obtain the currently available resources based on the total resources and the occupied resources when the resource usage data includes the total resources of the background and the resources occupied by the current processing task; The relationship of resources available to resources required by a resource request.

In another exemplary embodiment, the detection unit 1102 is further configured to adjust the resource amount of the available resources to obtain the target resource amount threshold; wherein, the target resource threshold is smaller than the resource amount of the available resources; obtain the resource required by the resource request The amount of resources; detect the relationship between the target resource amount threshold and the resource amount required by the resource request.

In another exemplary embodiment, the detecting unit 1102 is also configured to query the database for the data to be exported that matches the query parameters, and determine the space occupied by a single piece of data from the database; The amount of data and the space occupied by a single piece of data are used to obtain the space requirement of the resource request; the relationship between the available resources detected as available memory and the space requirement is obtained.

In another exemplary embodiment, the batching unit 1103 is further configured to divide the resources required for the resource request of the space requirement based on the available resources of the available memory to obtain a plurality of subspace requirements; The corresponding data to be exported is processed in batches according to the requirements of multiple subspaces, and multiple processed files are obtained.

In another exemplary embodiment, the feedback unit 1104 is further configured to use a specified compression algorithm to respectively compress multiple processed files to obtain multiple compressed files; and send the multiple compressed files to the originator corresponding to the resource request.

It should be noted that the device for processing resource data provided by the above embodiments and the method for processing resource data provided by the above embodiments belong to the same concept, and the specific ways of performing operations by each module and unit have been carried out in the method embodiments Detailed description will not be repeated here. In practical applications, the device for processing resource data provided by the above-mentioned embodiments can allocate the above-mentioned functions by different functional modules according to needs, that is, divide the internal structure of the device into different functional modules, so as to complete all or all of the above-described Some functions are not limited here.

The embodiment of the present application also provides an electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, The electronic device is made to implement the method for processing resource data provided in the foregoing embodiments.

Fig. 12 shows a schematic structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application. It should be noted that the computer system 1200 of the electronic device shown in FIG. 12 is only an example, and should not limit the functions and scope of use of this embodiment of the present application.

As shown in FIG. 12 , a computer system 1200 includes a central processing unit (Central Processing Unit, CPU) 1201, which can be stored in a program in a read-only memory (Read-Only Memory, ROM) 1202 or loaded into a random access memory ( RandomAccessMemory, RAM) 1203 to perform various appropriate actions and processing, such as performing the methods in the above-mentioned embodiments. In RAM1203, various programs and data necessary for system operation are also stored. The CPU 1201 , ROM 1202 , and RAM 1203 are connected to each other via a bus 1204 . An input/output (Input/Output, I/O) interface 1205 is also connected to the bus 1204 .

The following components are connected to the I/O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CathodeRayTube, CRT), a liquid crystal display (LiquidCrystalDisplay, LCD) etc., and a speaker; including a hard disk, etc. and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I/O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1210 as necessary so that a computer program read therefrom is installed into the storage section 1208 as necessary.

In particular, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, where the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication portion 1209 and/or installed from removable media 1211 . When the computer program is executed by a central processing unit (CPU) 1201, various functions defined in the system of the present application are performed.

It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM), flash memory, optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program thereon. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. . A computer program embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Wherein, each block in the flowchart or block diagram may represent a module, a program segment or a part of code, and the above-mentioned module, program segment or part of code includes one or more executable instructions for realizing the specified logic function. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block in the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a A combination of dedicated hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software or by hardware, and the described units may also be set in a processor. Wherein, the names of these units do not constitute a limitation of the unit itself under certain circumstances.

Another aspect of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above resource data processing method is realized. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently without being assembled into the electronic device.

Another aspect of the present application also provides a computer program product or computer program, the computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the resource data processing methods provided in the above-mentioned embodiments.

The above are only preferred embodiments of the application, and are not intended to limit the application. Any modifications, equivalent replacements or improvements made within the spirit and principles of the application shall be included in the protection scope of the application. within.

Claims (10)

1. A method for processing resource data, the method comprising:

if the data quantity requested by the received resource request exceeds a preset quantity threshold value, acquiring resource use data of a background aiming at the processing task;

detecting the relation between the resource of the background processing task and the resource required by the resource request according to the resource usage data;

if the resource of the background processing task represented by the relationship is smaller than the resource required by the resource request, performing batch processing on the task corresponding to the resource request to obtain a plurality of processing files;

and sending the plurality of processing files to an initiator corresponding to the resource request.

2. The method of claim 1, wherein the resource usage data includes total resources in the background and resources occupied by current processing tasks; the detecting the relationship between the resource of the background processing task and the resource required by the resource request according to the resource usage data includes:

obtaining a current available resource based on the total resource and the occupied resource;

detecting a relationship of the available resource to a resource required by the resource request.

3. The method of claim 2, wherein the detecting a relationship between the available resource and the resource required by the resource request comprises:

adjusting the resource quantity of the available resources to obtain a target resource quantity threshold value; wherein the target resource threshold is less than the resource amount of the available resource;

acquiring the resource quantity of the resource required by the resource request;

and detecting the relation between the target resource quantity threshold and the resource quantity of the resource required by the resource request.

4. The method according to claim 2, wherein the available resource is an available memory, and the resource request carries a query parameter; the detecting the relation between the available resource and the resource required by the resource request comprises:

inquiring data to be derived matched with the inquiry parameters from a database, and determining the space size occupied by the single data from the database;

obtaining the space demand of the resource request based on the data volume corresponding to the data to be exported and the space size occupied by the single data;

detecting a magnitude relationship between the available memory and the space requirement.

5. The method of claim 2, wherein the available resource is available memory, and the resource required by the resource request is a space requirement; the processing the task corresponding to the resource request in batches to obtain a plurality of processing files comprises:

dividing the space demand based on the available memory to obtain a plurality of subspace demands;

and processing the data to be exported corresponding to the resource request in batches according to the demand quantity of the plurality of subspaces to obtain a plurality of processing files.

6. The method according to any one of claims 1 to 5, wherein the resource request carries an identifier; before the obtaining resource usage data of the background for the processing task if the data amount requested by the received resource request exceeds a preset number threshold, the method includes:

receiving the resource request sent by the initiator;

detecting the type of the identifier contained in the resource request;

if the identifier is detected to be the first type, determining that the data volume corresponding to the resource request exceeds a preset number threshold;

and if the identifier is detected to be the second type identifier, determining that the data volume corresponding to the resource request does not exceed the preset number threshold.

7. The method according to any one of claims 1 to 5, wherein the sending the plurality of processing files to the initiator corresponding to the resource request includes:

compressing the plurality of processing files respectively by adopting a specified compression algorithm to obtain a plurality of compressed files;

and sending the plurality of compressed files to an initiator corresponding to the resource request.

8. An apparatus for processing resource data, comprising:

the acquisition unit is used for acquiring resource use data of a background aiming at the processing task if the data quantity requested by the received resource request exceeds a preset quantity threshold;

the detection unit is used for detecting the relation between the resources of the background processing task and the resources required by the resource request according to the resource use data;

the batching unit is used for batching the task corresponding to the resource request to obtain a plurality of processing files if the resource of the background processing task represented by the relationship is smaller than the resource required by the resource request;

and the feedback unit is used for sending the processing files to the initiator corresponding to the resource request.

9. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs which, when executed by the one or more processors, cause the electronic device to implement the method of processing resource data according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer-readable instructions which, when executed by a processor of a computer, cause the computer to perform the method of processing resource data of any one of claims 1 to 7.

Images