CN111131074A - Data processing method, device, system, server and readable storage medium - Google Patents

Abstract

The invention discloses a data processing method, device, system, server and readable storage medium. The method includes: receiving Internet traffic data packets, obtaining identification information of a gateway router sending the Internet traffic data packets; For each memory queue and the identification information, determine the target memory queue corresponding to the identification information; store the Internet traffic data packets in the target memory queue, and use the single thread corresponding to the target memory queue to process the Internet traffic data packets. In the present invention, the server stores the received Internet traffic data packets in the corresponding target memory queue, and uses the single thread corresponding to the target memory queue to process the Internet traffic data, thereby avoiding the inability to guarantee the timeliness of multi-threaded processing data, and ensuring The orderliness of the data packets sent to the downstream business system ensures that the downstream business system can accurately analyze the user's online behavior.

Description

A data processing method, device, system, server and readable storage medium

technical field

The present invention relates to the field of network technologies, and in particular, to a data processing method, device, system, server and readable storage medium.

Background technique

The user terminal is connected to the gateway router through WiFi, and the data packets of Internet traffic data generated during this period will be sent to the UDP SERVER of the remote server in the form of UDP through the plug-in on the gateway, that is, the gateway router will report the Internet access of the user terminal in the form of UDP in real time. The data packet of the traffic data, the content of the data packet is the header of the layer 2 data packet plus the pcap packet format, which is received and processed by the UDP SERVER, and the UDP SERVER processes the data packet and sends it to the downstream business system. Orderly Internet traffic data can accurately analyze a user's complete Internet behavior.

UDP SERVER usually processes data packets in an asynchronous form, that is, it usually uses multi-threading in UDP SERVER for processing. There is only one UDP SERVER in the data transmission architecture shown in Figure 1, client 1 and client 2. And client M can be understood as gateway router 1, gateway router 2 and gateway router M respectively, client 1, client 2 and client M respectively send the Internet traffic data of the user terminal, that is, data packets to the corresponding UDP SERVER, UDP SERVER The data is stored in the local memory queue, the memory queue is decoupled, and then a thread pool is started. Through the background multi-threads including thread 1, thread 2 and thread N, the data is taken out of the memory queue for business processing in an asynchronous and non-blocking manner, and then Send data packets to downstream business systems.

Using the data transmission architecture shown in Figure 1, the Internet traffic data of the same gateway router will be processed concurrently by multiple threads in the same UDPSERVER. The Internet traffic data sent to the downstream business system is out of order. And add multiple UDP SERVERs on the basis of Figure 1, that is, if the UDP SERVER is deployed in a cluster, the Internet traffic data of the same gateway router may be forwarded to multiple UDP SERVERs, and multiple threads in multiple UDP SERVERs The concurrent processing of the data cannot guarantee the timing of the data.

To sum up, in the prior art, multiple senders send data concurrently, which can ensure that multiple threads in the UDP SERVER concurrently fetch data from the same memory queue, and can ensure the timing of multiple threads fetching data, but due to Under high concurrency, multiple threads will concurrently preempt the data in the queue, and each data packet can only be processed once. Since the timeliness of each thread's processing cannot be guaranteed, it is possible that the thread that gets the data first will process it later. After completion, the data sent to the downstream service system is out of order, and the downstream service system cannot analyze the correct user online behavior through the out-of-order data packets, and may even cause analysis errors.

SUMMARY OF THE INVENTION

The present invention provides a data processing method, device, system, server and readable storage medium, which are used to solve the problem that the timeliness of multi-threaded data processing in the prior art cannot be guaranteed, resulting in that the downstream business system cannot correctly analyze the user's online behavior. question.

The present invention provides a data processing method, which is applied to a server, and the method includes:

Receive Internet traffic data packets, and obtain identification information of the gateway router that sends the Internet traffic data packets;

According to each pre-initialized memory queue and the identification information, determine the target memory queue corresponding to the identification information;

The Internet traffic data packets are stored in the target memory queue, and a single thread corresponding to the target memory queue is used to process the Internet traffic data packets.

Further, the identification information of the gateway router includes a media access control MAC address and/or an internet protocol IP address.

Further, if the identification information of the gateway router includes a MAC address, determining the target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information includes:

Determine the number of pre-initialized memory queues, and the sequence number of each memory queue;

performing hash processing on the MAC address to obtain a hash value of the MAC address;

The sequence number of the target memory queue corresponding to the MAC address is determined according to the number of the memory queues and the hash value of the MAC address.

Further, determining the sequence number of the target memory queue corresponding to the MAC address according to the number of the memory queues and the hash value of the MAC address includes:

The hash value of the MAC address is modulo the number of the memory queues, and the sequence number of the target memory queue corresponding to the MAC address is determined according to the result of the modulo.

Further, the receiving Internet traffic data packets includes:

Receive the Internet traffic data packet sent by the load balancing server, wherein the Internet traffic data packet is determined by the load balancing server according to the obtained egress IP address of the gateway router that sends the Internet traffic data packet. The egress IP address corresponds to sent to the server after the server.

Further, after the single thread corresponding to the target memory queue is used to process the Internet traffic data packets, the method further includes:

Using the single thread corresponding to the target memory queue, the processed Internet traffic data packets are sent to the corresponding downstream devices.

The present invention provides a data processing device applied to a server, the device comprising:

an acquisition module, used for receiving Internet traffic data packets, and acquiring the identification information of the gateway router that sends the Internet traffic data packets;

a determining module, configured to determine the target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information;

A processing module, configured to store the Internet traffic data packets in the target memory queue, and use a single thread corresponding to the target memory queue to process the Internet traffic data packets.

The present invention provides a server, including a memory, a processor and a transceiver;

The processor is configured to read the program in the memory, and execute the following processes: controlling the transceiver to receive Internet traffic data packets, and acquiring the identification information of the gateway router that sends the Internet traffic data packets; For each memory queue and the identification information, determine the target memory queue corresponding to the identification information; store the Internet traffic data packets in the target memory queue, and use the single thread corresponding to the target memory queue to process the Internet traffic data packets.

Further, the processor is specifically configured to determine the number of pre-initialized memory queues and the sequence number of each memory queue if the identification information of the gateway router includes a MAC address; perform hash processing on the MAC address, Obtain the hash value of the MAC address; determine the sequence number of the target memory queue corresponding to the MAC address according to the number of the memory queues and the hash value of the MAC address.

Further, the processor is specifically configured to take the hash value of the MAC address modulo the number of the memory queues, and determine the sequence number of the target memory queue corresponding to the MAC address according to the result of the modulo.

Further, the processor is specifically configured to control the transceiver to receive the Internet traffic data packet sent by the load balancing server, wherein the Internet traffic data packet is the Internet traffic data sent by the load balancing server according to the obtained data. The egress IP address of the gateway router of the packet is sent to the server after determining the server corresponding to the egress IP address.

Further, the processor is further configured to use the single thread corresponding to the target memory queue to process the Internet traffic data packets by using the single thread corresponding to the target memory queue, and use the single thread corresponding to the target memory queue to process the Internet traffic data after processing. The packet is sent to the corresponding downstream device.

The invention provides a server, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

A computer program is stored in the memory, and when the program is executed by the processor, the processor causes the processor to perform any one of the above steps of the method.

The present invention provides a computer-readable storage medium, which stores a computer program executable by a server, and when the program runs on the server, causes the server to perform the steps of any one of the above-mentioned methods.

The present invention provides a data processing system, the data processing system includes a gateway router and a server; wherein,

The gateway router is used to send Internet traffic data packets to the server;

The server is used to receive the Internet traffic data packets, and obtain the identification information of the gateway router that sends the Internet traffic data packets; according to each pre-initialized memory queue and the identification information, determine the target memory corresponding to the identification information queue; store the Internet traffic data packets in the target memory queue, and use the single thread corresponding to the target memory queue to process the Internet traffic data packets.

Further, the server is specifically configured to determine the number of pre-initialized memory queues and the serial number of each memory queue if the identification information of the gateway router includes a MAC address; perform hash processing on the MAC address to obtain The hash value of the MAC address; the serial number of the target memory queue corresponding to the MAC address is determined according to the number of the memory queues and the hash value of the MAC address.

Further, the server is specifically configured to take the hash value of the MAC address modulo the number of the memory queues, and determine the sequence number of the target memory queue corresponding to the MAC address according to the result of the modulo.

Further, the system also includes:

a load balancing server, configured to receive the Internet traffic data packets sent by the gateway router; according to the obtained egress IP address of the gateway router that sends the Internet traffic data packets, determine the server corresponding to the egress IP address, and sending the Internet traffic data packet to the server;

The server is specifically configured to receive Internet traffic data packets sent by the load balancing server.

Further, the system further includes: downstream equipment; wherein,

The server is further configured to use the single thread corresponding to the target memory queue to process the Internet traffic data packets, and use the single thread corresponding to the target memory queue to send the processed Internet traffic data packets to the corresponding downstream equipment.

The present invention provides a data processing method, device, system, server and readable storage medium. The method includes: receiving Internet traffic data packets, obtaining identification information of a gateway router sending the Internet traffic data packets; For each memory queue and the identification information, determine the target memory queue corresponding to the identification information; store the Internet traffic data packets in the target memory queue, and use the single thread corresponding to the target memory queue to process the Internet traffic data packets. In the present invention, the server stores the received Internet traffic data packets in the corresponding target memory queue, and uses the single thread corresponding to the target memory queue to process the Internet traffic data, thereby avoiding the inability to guarantee the timeliness of multi-threaded data processing and ensuring The orderliness of the data packets sent to the downstream business system ensures that the downstream business system can accurately analyze the user's online behavior.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a schematic diagram of a data processing architecture provided by the prior art;

2 is a schematic diagram of a data processing process according to Embodiment 1 of the present invention;

3 is a schematic diagram of a data processing architecture provided in Embodiment 1 of the present invention;

4 is a schematic diagram of a data processing architecture provided in Embodiment 5 of the present invention;

5 is a schematic structural diagram of a server according to Embodiment 6 of the present invention;

6 is a schematic structural diagram of a server according to Embodiment 6 of the present invention;

7 is a schematic structural diagram of a server according to Embodiment 6 of the present invention;

8 is a schematic structural diagram of a server according to Embodiment 7 of the present invention;

9 is a schematic structural diagram of a server according to Embodiment 8 of the present invention;

FIG. 10 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention.

Detailed ways

In order to ensure the timeliness of data processing, so that the downstream business system can accurately analyze the user's online behavior, the embodiments of the present invention provide a data processing method, device, system, server and readable storage medium.

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1:

2 is a schematic diagram of a data processing process provided by an embodiment of the present invention, and the process includes the following steps:

S201: Receive an Internet traffic data packet, and obtain identification information of a gateway router that sends the Internet traffic data packet.

The data processing method provided by the embodiment of the present invention is applied to a server. Specifically, the server is a UDP SERVER, and the UDP SERVER may be a physical server or a virtual server.

The server can receive Internet traffic data packets. The Internet traffic data packets received by the server are the Internet traffic data packets of the terminal reported by the gateway router. Usually, the same gateway router reports the Internet traffic data packets in the order sent by the terminal, that is, the same gateway router reports the Internet traffic data packets. Internet traffic data packets can ensure orderliness.

Since the gateway router reports the terminal's Internet traffic data packets to the server, when the server obtains the identification information of the gateway router that sends the Internet traffic data packets, it can find the identification of the Internet traffic data packets sent in the received Internet traffic data packets. information, the identification information of the Internet traffic data packet may be one or some characteristic fields in the Internet traffic data packet.

The identification information of the gateway router includes a MAC (Media Access Control, media access control) address and/or an IP (Internet Protocol, Internet Protocol) address. The identification information of the gateway router can uniquely identify a gateway router.

S202: Determine a target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information.

After the server obtains the identification information of the gateway router that sends the Internet traffic data packets, it can determine the target memory queue corresponding to the identification information according to each pre-initialized memory queue and the obtained identification information.

Multiple memory queues are pre-initialized in the server. For example, the number of pre-initialized memory queues is N. When the server pre-initializes the memory queues, it may be initialized within the range allowed by the server hardware.

The server determines the target memory queue corresponding to the identification information according to the pre-initialized identification information of each memory queue and the gateway router. To determine the target memory queue corresponding to the identification information, it can be based on a pre-saved algorithm fun(feature field, N), where fun represents a certain algorithm, and the input parameters are the feature field and the number of queues N, and the feature field can be understood as the gateway router. identification information, so as to determine the target memory queue corresponding to the identification information, etc.

S203: Store the Internet traffic data packets in the target memory queue, and use a single thread corresponding to the target memory queue to process the Internet traffic data packets.

When the server pre-initializes the memory queue, it can also initialize a thread pool within the range allowed by the hardware, in which each thread in the thread pool corresponds to the memory queue one-to-one, that is, each memory queue has its corresponding single thread, so The size of the thread pool is the same as the number of memory queues. For example, when the number of pre-initialized memory queues is N, the size of the thread pool is also N. Since the initialization of the thread pool is relatively dependent on the hardware resources of the server, it can be done in the hardware first. Initialize a thread pool of size N within the allowed range, and then initialize N memory queues of the same size as the thread pool.

In order to ensure that the order in which the data packets are processed is consistent with the order in which the data packets are received, you can find one or some characteristic fields in the data packets, and use this field to distribute the data evenly into multiple memory queues through an algorithm. , allowing a single thread to synchronously block the processing, preventing multiple threads from processing data packets with the same characteristic fields, and solving the problem of out-of-order data packets in the original single memory queue multi-thread preemptive processing method, satisfying the same characteristics. The data packets are strictly ordered, which finally solves the problem of abnormal traffic analysis caused by out-of-order data packets. Therefore, the Internet traffic data packets are stored in the determined target memory queue, and the single thread corresponding to the target memory queue is used to process the Internet traffic data packets.

3 is a schematic diagram of a data processing architecture provided by an embodiment of the present invention. There is a server UDP SERVER in the data processing architecture, and client 1, client 2 and client M can be respectively understood as gateway router 1, gateway router 2 and gateway Router M, client 1, client 2 and client M respectively send the Internet traffic data of the user terminal, that is, data packets, to the UDP SERVER. After the UDP SERVER receives the Internet traffic data packets, according to the pre-saved fun(feature field, N) rule, calculate the queue number of the Internet traffic data packet, and store the Internet traffic data packet into the corresponding target memory queue, if stored in the memory queue, use the single thread 1 corresponding to the memory queue to process the Internet traffic data packet, If it is stored in the memory queue 2, the single thread 2 corresponding to the memory queue 2 is used to process the Internet traffic data packet; if it is stored in the memory queue 3, the single thread N corresponding to the memory queue N is used to process the Internet traffic data packet. At this time, the Internet traffic data of the same gateway router will be processed by the single thread corresponding to the memory queue in the one UDP SERVER. The sequence of processing the data packets by the single thread is the same as the sequence of receiving the data packets, so the data sent to the downstream business system can be guaranteed. Internet traffic data packets are ordered.

In the embodiment of the present invention, the server stores the received Internet traffic data packets in the corresponding target memory queue, and uses the single thread corresponding to the target memory queue to process the Internet traffic data, which avoids that the timeliness of multi-threaded processing data cannot be guaranteed. The order of the data packets sent to the downstream business system is ensured, thereby ensuring that the downstream business system can accurately analyze the user's online behavior.

Example 2:

On the basis of the above embodiment, in the embodiment of the present invention, after the single thread corresponding to the target memory queue is used to process the Internet traffic data packet, the method further includes:

Using the single thread corresponding to the target memory queue, the processed Internet traffic data packets are sent to the corresponding downstream devices.

In order to ensure that the downstream business system can accurately analyze the complete Internet behavior of users, the server uses the single thread corresponding to the target memory queue to process the Internet traffic data packets, and uses the single thread corresponding to the target memory queue to send the processed Internet traffic data packets to The downstream device, because the sequence of sending the processed Internet traffic data packets by a single thread is consistent with the sequence in which the single thread receives and processes Internet traffic data packets, so it can ensure that the time sequence of the data received by the downstream device is in order, so it can be Accurately analyze the complete online behavior of users.

The downstream device is a downstream device installed with a downstream service system, and specifically, the downstream device may be a message middleware.

Example 3:

In order to ensure that the Internet traffic data packets sent by the same gateway router can be stored in the corresponding memory queue in a strict and orderly manner, on the basis of the above embodiments, in this embodiment of the present invention, if the identification information of the gateway router includes a MAC address, The determining of the target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information includes:

Determine the number of pre-initialized memory queues, and the sequence number of each memory queue;

performing hash processing on the MAC address to obtain a hash value of the MAC address;

The sequence number of the target memory queue corresponding to the MAC address is determined according to the number of the memory queues and the hash value of the MAC address.

If the server determines the target memory queue corresponding to the identification information according to the pre-stored algorithm fun (feature field, N), in order to ensure that the Internet traffic data packets can be stored in the corresponding memory queue in an orderly manner, the server can use the gateway router's MAC address and The sequence number of each memory queue to achieve the determination of the target memory queue.

Because the server has pre-initialized memory queues, the server can determine the number of pre-initialized memory queues and determine the sequence number of each memory queue. The serial number of each memory queue may be determined in advance, and the serial number of each memory queue may be a serial number in a sequence such as 1-N, or other serial numbers set manually.

In order to ensure that the Internet traffic data packets of the same gateway router are stored in the same memory queue, the server can hash the MAC address of the gateway router to obtain the hash value of the MAC address. The hash value of the MAC address is unique .

The process of hashing the information including the MAC address can be implemented by the prior art, and details are not described in this embodiment of the present invention.

The server determines the hash value of the MAC address, and can determine the sequence number of the target memory queue corresponding to the MAC address according to the number of memory queues and the hash value of the MAC address, which can be specifically based on the hash value of the MAC address and the memory queue. The result of taking the modulo of the number to determine the serial number of the target memory queue corresponding to the MAC address may be based on the result of taking the remainder of the hash value of the MAC address and the number of memory queues to determine the serial number of the target memory queue corresponding to the MAC address, etc.

After determining the sequence number corresponding to the target memory corresponding to the MAC address, the server stores the received Internet traffic data packets in the target memory queue corresponding to the determined sequence number.

Example 4:

On the basis of the foregoing embodiments, in this embodiment of the present invention, determining the sequence number of the target memory queue corresponding to the MAC address according to the number of the memory queues and the hash value of the MAC address includes:

The hash value of the MAC address is modulo the number of the memory queues, and the sequence number of the target memory queue corresponding to the MAC address is determined according to the result of the modulo.

In order to further ensure that the order of single-threaded processing of data and the order of receiving packets are consistent and orderly, the server can use the hash value of the MAC address to modulo the number of memory queues so that the Internet traffic data packets of the same gateway router are stored in the same memory queue. Each memory queue is processed by only one single thread, thus ensuring that for the same gateway router, the data is processed in a synchronous blocking manner, so that the order in which the data is processed is consistent with the order in which it is received.

Therefore, when the server determines the serial number of the target memory queue corresponding to the MAC address according to the number of memory queues and the hash value of the MAC address, the server modulizes the hash value of the MAC address to the number of memory queues, and determines the MAC address according to the result of the modulo. The serial number of the target memory queue corresponding to the address. Specifically, if the serial numbers of the memory queues are arranged in order, the server can directly use the hash value of the MAC address modulo the number of memory queues as the target memory corresponding to the MAC address. The serial number of the queue. If the serial number of the memory queue is artificially set without regularity, the server modulo the number of the memory queue according to the hash value of the MAC address, and the difference between the pre-saved modulo results and the serial number of the memory queue. Correspondence, determine the sequence number of the target memory queue corresponding to the result of the modulo, etc.

Because in the embodiment of the present invention, the sequence number of the target memory queue corresponding to the MAC address is determined according to the result of taking the modulo of the number of memory queues based on the hash value of the MAC address, so that the order of the number of single-threaded processing can be guaranteed to be consistent with the order of received data. This ensures that the downstream business system can accurately analyze the user's online behavior.

Example 5:

On the basis of the foregoing embodiments, in this embodiment of the present invention, the receiving Internet traffic data packets includes:

Receive the Internet traffic data packet sent by the load balancing server, wherein the Internet traffic data packet is determined by the load balancing server according to the obtained egress IP address of the gateway router that sends the Internet traffic data packet. The egress IP address corresponds to sent to the server after the server.

If the UDP SERVER is deployed in a cluster, the load balancer forwards the data packets of the gateway router to the UDP SERVER according to the round-robin algorithm, so there will still be a small number of out-of-order data packets when the UDP SERVER is deployed in the cluster. In order to fundamentally solve this problem, the load balancing server needs to be transformed.

The load balancing server can be deployed between the gateway router and the server. The Internet traffic data packets sent by the gateway router to the server are first received by the load balancing server. The load balancing server can perform load balancing according to the egress IP address of each gateway router. The Internet traffic data packets corresponding to the IP addresses will only be forwarded to the corresponding fixed server for processing, which avoids the situation that the Internet traffic data packets of the same gateway router are processed by multiple servers.

After receiving the Internet traffic data packets sent by the gateway router to the server, the load balancing server obtains the egress IP address of the gateway router, and determines the server corresponding to the egress IP address of the gateway router according to the pre-stored correspondence between the egress IP address and the server. , and then forward the received Internet traffic data packets to the server corresponding to the egress IP address of the gateway router.

FIG. 4 is a schematic diagram of a data processing architecture provided by an embodiment of the present invention, which is an improvement for the data processing architecture improved on the basis of FIG. 3. In the data processing architecture, there is a server cluster composed of multiple servers UDP SERVERs, and a load balancer is The load balancing server, client 1, client 2 and client M concurrently send the Internet traffic data of the user terminal, that is, data packets, to the load balancing server, and the load balancing server receives the Internet traffic data sent by client 1, that is, gateway router 1. Packet as an example, the load balancing server parses the egress IP address of client 1, and forwards the Internet traffic data packets to the server corresponding to the egress IP address of client 1 according to the pre-stored correspondence between the egress IP address and the server, ensuring that The Internet traffic data packets of the same egress IP address are forwarded to the same server UDP SERVER, and the data packets sent by the same client will not appear on two or more UDP SERVERs. The data processing process of the server is similar to Figure 3. The server parses the Internet traffic data packets and obtains the corresponding characteristic field fdata. The characteristic field fdata is the identification information of the client such as the MAC address, and the memory queue is calculated according to queueNum=fun(fdata, N). The sequence number, N is the number of threads or queues, the Internet traffic data packets are stored in the target memory queue corresponding to the sequence number, the single thread corresponding to the target memory queue pops the data from the target memory queue for post-processing, and the single thread will process the processed Data is transmitted to downstream business systems.

For the current problem of out-of-order data packets, in the case of TCP, the sequence number sequence number in the TCP protocol is used to reorganize the data packets to achieve order; in the case of UDP, most of them are added in the UDP data packet body. The definition field is used to indicate the data sequence, and the server sorts it according to the sequence field; this method is aimed at the premise that multiple clients send data packets concurrently, the first layer improves the load algorithm of the load balancer, and the same client The data packet is only forwarded to the same server, and then through some characteristic fields in the data packet, according to a certain algorithm, the packet of the same client is processed through the unique single-threaded synchronous blocking process on the server side, which will not cause many problems. The problem of out-of-order data packets caused by thread preemption does not change the original data packet format, and the overall architecture is still an asynchronous non-blocking high-performance processing method.

Since the load balancing server in the embodiment of the present invention determines the corresponding server according to the egress IP address of the gateway router, and then sends the Internet traffic data packet to the corresponding server, it avoids the Internet traffic data packet of the same gateway router being processed by multiple servers. situation happens.

Example 6:

On the basis of the foregoing embodiments, FIG. 5 is a schematic structural diagram of a data processing system provided by an embodiment of the present invention, where the data processing system includes: a gateway router 501 and a server 502; wherein,

The gateway router 501 is used to send Internet traffic data packets to the server 502;

The server 502 is configured to receive the Internet traffic data packets, and obtain the identification information of the gateway router 501 that sends the Internet traffic data packets; according to each pre-initialized memory queue and the identification information, determine the corresponding identification information. A target memory queue; the Internet traffic data packets are stored in the target memory queue, and a single thread corresponding to the target memory queue is used to process the Internet traffic data packets.

The server 502 is specifically configured to determine the number of pre-initialized memory queues and the serial number of each memory queue if the identification information of the gateway router 501 includes a MAC address; perform hash processing on the MAC address to obtain the The hash value of the MAC address; according to the number of the memory queues and the hash value of the MAC address, the sequence number of the target memory queue corresponding to the MAC address is determined.

The server 502 is specifically configured to take the hash value of the MAC address modulo the number of the memory queues, and determine the sequence number of the target memory queue corresponding to the MAC address according to the result of the modulo.

As shown in Figure 6, the system further includes:

The load balancing server 603 is configured to receive the Internet traffic data packet sent by the gateway router 501; server 502, and send the Internet traffic data packet to the server 502;

The server 502 is specifically configured to receive the Internet traffic data packets sent by the load balancing server 603 .

As shown in FIG. 7, the system further includes: downstream equipment 704; wherein,

The server 502 is further configured to use the single thread corresponding to the target memory queue to process the Internet traffic data packets, and then use the single thread corresponding to the target memory queue to send the processed Internet traffic data packets to Corresponding downstream device 704 .

Example 7:

On the basis of the foregoing embodiments, an embodiment of the present invention further provides a server 800, as shown in FIG. 8, including: a processor 801, a memory 802, and a transceiver 803;

The processor 801 is configured to execute the program in the read memory 802, and execute the following processes:

Control the transceiver 803 to receive Internet traffic data packets, and obtain the identification information of the gateway router that sends the Internet traffic data packets; according to each pre-initialized memory queue and the identification information, determine the target memory corresponding to the identification information queue; store the Internet traffic data packets in the target memory queue, and use the single thread corresponding to the target memory queue to process the Internet traffic data packets.

Based on the same inventive concept, an embodiment of the present invention also provides a server. Since the principle of the above server for solving problems is similar to that of the data processing method, the implementation of the above server can refer to the implementation of the method, and the repetition will not be repeated.

In FIG. 8 , the bus architecture may include any number of interconnected buses and bridges, in particular one or more processors represented by processor 801 and various circuits of memory represented by memory 802 linked together. The bus architecture may also link together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and, therefore, will not be described further herein. Transceiver 803 may be a number of elements, including a transmitter and a receiver, providing a means for communicating with various other devices over a transmission medium. The processor 801 is responsible for managing the bus architecture and general processing, and the memory 802 may store data used by the processor 801 in performing operations.

Optionally, the processor 801 may be a CPU (central processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field-Programmable Gate Array, field programmable gate array) or a CPLD (Complex Programmable Logic Device, complex programmable logic devices).

The processor 801 is specifically configured to determine the number of pre-initialized memory queues and the serial number of each memory queue if the identification information of the gateway router includes a MAC address; perform hash processing on the MAC address to obtain the The hash value of the MAC address; according to the number of the memory queues and the hash value of the MAC address, the sequence number of the target memory queue corresponding to the MAC address is determined.

The processor 801 is specifically configured to take the hash value of the MAC address modulo the number of the memory queues, and determine the sequence number of the target memory queue corresponding to the MAC address according to the result of the modulo.

The processor 801 is specifically configured to control the transceiver 803 to receive the Internet traffic data packet sent by the load balancing server, wherein the Internet traffic data packet is the Internet traffic data packet sent by the load balancing server according to the obtained data packet. The egress IP address of the gateway router is sent to the server after determining the server corresponding to the egress IP address.

The processor 801 is further configured to use the single thread corresponding to the target memory queue to process the Internet traffic data packets, and use the single thread corresponding to the target memory queue to send the processed Internet traffic data packets. to the corresponding downstream device.

In the embodiment of the present invention, the server stores the received Internet traffic data packets in the corresponding target memory queue, and uses the single thread corresponding to the target memory queue to process the Internet traffic data, avoiding the timeliness of multi-threaded processing of data. The guarantee ensures the orderliness of the data packets sent to the downstream business system, thereby ensuring that the downstream business system can accurately analyze the user's online behavior.

Example 8:

On the basis of the foregoing embodiments, an embodiment of the present invention further provides a server 900, as shown in FIG. 9, including: a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, The communication interface 902, the memory 903 completes the mutual communication through the communication bus 904;

A computer program is stored in the memory 903, and when the program is executed by the processor 901, the processor 901 is caused to perform the following steps:

Receive Internet traffic data packets, and obtain the identification information of the gateway router that sends the Internet traffic data packets;

According to each pre-initialized memory queue and the identification information, determine the target memory queue corresponding to the identification information;

The Internet traffic data packets are stored in the target memory queue, and a single thread corresponding to the target memory queue is used to process the Internet traffic data packets.

The communication bus mentioned by the above server may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface 902 is used for communication between the above-mentioned server and other devices.

The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it may also be a digital instruction processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array, or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

In the embodiment of the present invention, when the processor executes the program stored in the memory, the server in the present invention stores the received Internet traffic data packets in the corresponding target memory queue, and adopts the single-threaded processing corresponding to the target memory queue. The Internet traffic data avoids that the timeliness of multi-threaded processing data cannot be guaranteed, and ensures the orderliness of data packets sent to the downstream business system, thereby ensuring that the downstream business system can accurately analyze the user's Internet behavior.

Example 9:

On the basis of the foregoing embodiments, embodiments of the present invention further provide a computer storage-readable storage medium, where a computer program executable by a server is stored in the computer-readable storage medium, and when the program is stored in the server When running on the server, the following steps are implemented when the server is executed:

Receive Internet traffic data packets, and obtain identification information of the gateway router that sends the Internet traffic data packets;

According to each pre-initialized memory queue and the identification information, determine the target memory queue corresponding to the identification information;

The Internet traffic data packets are stored in the target memory queue, and a single thread corresponding to the target memory queue is used to process the Internet traffic data packets.

The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in the server, including but not limited to magnetic storage such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc., optical storage such as CD, DVD, BD, HVD, etc., as well as semiconductor memories such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid state disk (SSD), and the like.

A computer program is stored in the computer-readable storage medium provided in the embodiment of the present invention, and when the computer program is executed by the processor, the server in the present invention stores the received Internet traffic data packets in the corresponding target memory queue, and implements The single thread corresponding to the target memory queue is used to process the Internet traffic data, which avoids the inability to guarantee the timeliness of multi-threaded data processing, ensures the orderliness of data packets sent to the downstream business system, and ensures that the downstream business system can accurately analyze the data. User online behavior.

10 is a schematic diagram of a data processing apparatus provided by an embodiment of the present invention, applied to a server, and the apparatus includes:

Obtaining module 1001, configured to receive Internet traffic data packets, and obtain identification information of the gateway router that sends the Internet traffic data packets;

A determination module 1002, configured to determine a target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information;

The processing module 1003 is configured to store the Internet traffic data packets in the target memory queue, and use a single thread corresponding to the target memory queue to process the Internet traffic data packets.

The identification information of the gateway router includes a media access control MAC address and/or an internet protocol IP address.

If the identification information of the gateway router includes a MAC address, the determining module is specifically configured to determine the number of pre-initialized memory queues and the sequence number of each memory queue; perform hash processing on the MAC address to obtain the Hash value of the MAC address; according to the number of the memory queues and the hash value of the MAC address, determine the sequence number of the target memory queue corresponding to the MAC address.

The determining module is specifically configured to take the hash value of the MAC address modulo the number of the memory queues, and determine the sequence number of the target memory queue corresponding to the MAC address according to the result of the modulo.

The obtaining module is specifically configured to receive the Internet traffic data packet sent by the load balancing server, wherein the Internet traffic data packet is the egress IP address of the gateway router that sends the Internet traffic data packet obtained by the load balancing server , which is sent to the server after determining the server corresponding to the exit IP address.

The device also includes:

The sending module is configured to use the single thread corresponding to the target memory queue to send the processed Internet traffic data packet to the corresponding downstream device.

In the embodiment of the present invention, the server stores the received Internet traffic data packets in the corresponding target memory queue, and uses the single thread corresponding to the target memory queue to process the Internet traffic data, which avoids that the timeliness of multi-threaded processing data cannot be guaranteed. The order of the data packets sent to the downstream business system is ensured, thereby ensuring that the downstream business system can accurately analyze the user's online behavior.

For the system/apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

It should be noted that, in this document, relational terms such as first and second, etc. are only used to distinguish one entity or one operation from another entity or another operation, and do not necessarily require or imply these entities Or there is any such actual relationship or order between operations.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

While the preferred embodiments of the present application have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of this application.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (19)

1. A data processing method is applied to a server, and the method comprises the following steps:

receiving an internet traffic data packet, and acquiring identification information of a gateway router which sends the internet traffic data packet;

determining a target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information;

and storing the internet traffic data packet into the target memory queue, and processing the internet traffic data packet by adopting a single thread corresponding to the target memory queue.

2. The method of claim 1, wherein the identification information of the gateway router comprises a Media Access Control (MAC) address and/or an Internet Protocol (IP) address.

3. The method according to claim 1 or 2, wherein if the identification information of the gateway router includes a MAC address, the determining the target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information comprises:

determining the number of pre-initialized memory queues and the serial number of each memory queue;

performing hash processing on the MAC address to obtain a hash value of the MAC address;

and determining the sequence number of the target memory queue corresponding to the MAC address according to the number of the memory queues and the Hash value of the MAC address.

4. The method of claim 3, wherein the determining the sequence number of the target memory queue corresponding to the MAC address according to the number of the memory queues and the hash value of the MAC address comprises:

and performing modulus operation on the number of the memory queues by the hash value of the MAC address, and determining the sequence number of a target memory queue corresponding to the MAC address according to the modulus operation result.

5. The method of claim 1, wherein the receiving the internet traffic packet comprises:

the method comprises the steps of receiving an internet traffic data packet sent by a load balancing server, wherein the internet traffic data packet is sent to the load balancing server after the load balancing server determines a server corresponding to an outlet IP address according to the outlet IP address of a gateway router which sends the internet traffic data packet.

6. The method according to claim 1, wherein after the single thread corresponding to the target memory queue is used to process the internet traffic packet, the method further comprises:

and sending the processed internet traffic data packet to corresponding downstream equipment by adopting a single thread corresponding to the target memory queue.

7. A data processing apparatus, applied to a server, the apparatus comprising:

the gateway router comprises an acquisition module, a transmission module and a processing module, wherein the acquisition module is used for receiving an internet traffic data packet and acquiring identification information of a gateway router which transmits the internet traffic data packet;

the determining module is used for determining a target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information;

and the processing module is used for storing the internet traffic data packet into the target memory queue and processing the internet traffic data packet by adopting a single thread corresponding to the target memory queue.

8. A server comprising a memory, a processor, and a transceiver;

the processor is used for reading the program in the memory and executing the following processes: controlling the transceiver to receive an internet traffic data packet and acquiring identification information of a gateway router which sends the internet traffic data packet; determining a target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information; and storing the internet traffic data packet into the target memory queue, and processing the internet traffic data packet by adopting a single thread corresponding to the target memory queue.

9. The server according to claim 8, wherein the processor is specifically configured to determine a number of pre-initialized memory queues and a sequence number of each memory queue if the identification information of the gateway router includes a MAC address; performing hash processing on the MAC address to obtain a hash value of the MAC address; and determining the sequence number of the target memory queue corresponding to the MAC address according to the number of the memory queues and the Hash value of the MAC address.

10. The server according to claim 9, wherein the processor is specifically configured to modulo the number of the memory queues by the hash value of the MAC address, and determine a sequence number of a target memory queue corresponding to the MAC address according to a result of the modulo.

11. The server according to claim 8, wherein the processor is specifically configured to control the transceiver to receive an internet traffic data packet sent by a load balancing server, where the internet traffic data packet is sent to the server after the load balancing server determines, according to an obtained exit IP address of a gateway router that sends the internet traffic data packet, a server corresponding to the exit IP address.

12. The server according to claim 8, wherein the processor is further configured to, after processing the internet traffic data packet by using the single thread corresponding to the target memory queue, send the processed internet traffic data packet to a corresponding downstream device by using the single thread corresponding to the target memory queue.

13. A server, comprising: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

the memory has stored therein a computer program which, when executed by the processor, causes the processor to carry out the steps of the method of any of claims 1 to 6.

14. A computer-readable storage medium, in which a computer program executable by a server is stored, which program, when run on the server, causes the server to carry out the steps of the method according to any one of claims 1 to 6.

15. A data processing system, comprising a gateway router and a server; wherein,

the gateway router is used for sending an internet traffic data packet to the server;

the server is used for receiving the internet traffic data packet and acquiring the identification information of the gateway router which sends the internet traffic data packet; determining a target memory queue corresponding to the identification information according to each pre-initialized memory queue and the identification information; and storing the internet traffic data packet into the target memory queue, and processing the internet traffic data packet by adopting a single thread corresponding to the target memory queue.

16. The system of claim 15, wherein the server is specifically configured to determine a number of pre-initialized memory queues and a sequence number of each memory queue if the identification information of the gateway router includes a MAC address; performing hash processing on the MAC address to obtain a hash value of the MAC address; and determining the sequence number of the target memory queue corresponding to the MAC address according to the number of the memory queues and the Hash value of the MAC address.

17. The system according to claim 16, wherein the server is specifically configured to modulo the number of the memory queues by the hash value of the MAC address, and determine a sequence number of a target memory queue corresponding to the MAC address according to a result of the modulo.

18. The system of claim 15, wherein the system further comprises:

the load balancing server is used for receiving the internet traffic data packet sent by the gateway router; determining the server corresponding to the outlet IP address according to the acquired outlet IP address of the gateway router which sends the internet traffic data packet, and sending the internet traffic data packet to the server;

the server is specifically configured to receive an internet traffic data packet sent by the load balancing server.

19. The system of claim 15, wherein the system further comprises: downstream equipment; wherein,

the server is further configured to send the processed internet traffic data packet to a corresponding downstream device by using the single thread corresponding to the target memory queue after processing the internet traffic data packet by using the single thread corresponding to the target memory queue.

Images