CN107707516A - A kind of IP address analysis method and system - Google Patents

Abstract

The invention provides a kind of IP address analysis method and system.This method includes：Collect the historical data of IP address；The historical data of IP address is analyzed, generates the credit data of IP address.It can be applied in the scenes such as IP address legitimate verification, IP address interception, solve the problems, such as IP address Safety perception mistake, the generation for effectively preventing the erroneous judgement of IP address legitimacy, IP address from blocking by mistake.

Description

Method and system for analyzing IP address

technical field

The invention relates to the field of Internet user portraits, in particular to an IP address analysis method.

Background technique

The network layer firewall can be regarded as a kind of IP packet filter, which operates on the underlying TCP/IP protocol stack. In the way of enumeration, only packets that meet specific rules are allowed to pass through, and the rest are prohibited from passing through the firewall (except viruses, which cannot prevent virus intrusion). These rules can usually be defined or modified by the administrator, but some firewall devices may only apply built-in rules.

It is also possible to formulate firewall rules from another looser angle, as long as the packet does not meet any of the "negative rules", it will be released. Most operating systems and network devices have built-in firewall functions.

Newer firewalls can use various attributes of packets to filter, such as: source IP address, source port number, destination IP address or port number, service type (such as HTTP or FTP). It can also be filtered by attributes such as communication protocol, TTL value, source domain name or network segment... and so on.

The existing interception scheme intercepts through a single fixed rule, which does not have enough granularity and dimension for the analysis of access requests, and lacks awareness of the identity of the visitor, which is likely to cause false blocking.

Contents of the invention

The present invention aims to solve the problems described above. It is an object of the present invention to provide a solution to any of the above problems. Specifically, the present invention provides the ability to.

According to a first aspect of the present invention, a kind of IP address analysis method is provided, comprising:

collect historical data on IP addresses;

Analyze the historical data of the IP address to generate the credit data of the IP address.

Wherein, the step of collecting the historical data of the IP address includes:

Collect and parse raw logs during the first cycle;

Formatting the information in the original log to obtain a predefined indicator, the predefined indicator at least includes any one or more of the following information:

Time, IP, number of requests during working hours, number of requests during rest periods, number of requests during sleep periods, file size requested during working hours, file size requested during rest periods, file size requested during sleep periods, number of UserAgents during working hours, number of UserAgents during rest periods, sleep The number of UserAgents in the time period, the number of UserAgents on the mobile terminal, the number of UserAgents on the PC terminal, the number of access sources, the number of domain names accessed, and the number of hours;

The predefined indicators corresponding to each IP address are stored, and each IP address corresponds to one or more predefined indicators.

Wherein, the step of collecting the historical data of the IP address also includes:

In the second cycle, obtain the third-party IP library and/or third-party IP blacklist from the third-party platform one or more times.

Among them, the historical data of the IP address is analyzed, and the credit data of the generated IP address includes:

Preprocess and normalize the predefined indicators of the first period corresponding to working days in the second period to obtain the median value of each working day, and the predefined indicators of the first period corresponding to rest days in the second period After the indicators are preprocessed and normalized, the median value of each rest day is obtained, and the second cycle includes multiple first cycles corresponding to working days and multiple first cycles corresponding to rest days;

Performing weighted average processing on the median value of each working day respectively to obtain the weighted mean value of working days;

Perform weighted average or maximum value processing on the median value of each rest day to obtain the weighted average or maximum value of rest days;

Based on the weighted average value of one or more working days and the weighted average value of one or more rest days, the current temporary specific indicators for the second period in the second period are calculated, and the temporary specific indicators for the current second period include:

This cycle is the probability of office exit IP, this cycle is the probability of home exit IP, this cycle is the probability of real people, the activity score of this cycle, and the number of people in this cycle are grouped;

According to the final specific indicators of the previous second cycle and the third-party IP library and/or third-party IP blacklist, the temporary specific indicators of the current second cycle are adjusted to obtain the final specific indicators of the current second cycle, and the current The final specific index of the second period is used as the credit data of the IP address.

Wherein, the median value of each working day is obtained after preprocessing and normalizing the predefined indicators of the first period corresponding to working days in the second period, and the first period corresponding to rest days in the second period The steps to obtain the median value of each rest day after preprocessing and normalizing the predefined indicators of the period include:

Calculate the hours of the first cycle corresponding to the working day, the score of the user agent on the mobile terminal, the score of the user agent on the PC terminal, the score of the number of requests during the rest period vs. the work period, the score of the number of requests during the rest period vs. the sleep period, and the score of the number of domain names visited. For the above The weighted average of the scores is used to obtain the median value of the probability of exporting IP for households on weekdays;

Calculate the number of requests in the first cycle corresponding to the working day, the number of requests in the working period VS the rest period, the number of requests in the rest period VS the sleep period, the number of PC-side UserAgents, and the number of UserAgents in the working period VS rest period, for the above The weighted average of the score values is used to obtain the median value of the IP probability of an office exit on a working day;

Calculate the distribution score of the number of requests in the first period corresponding to the working day, the distribution score of the number of UserAgents, the score of the number of hours, the score of the number of domain names VS the number of sources, the score of the number of UserAgents on the mobile side VS PC side, and take the weighted average of the above scores to get the working day is the median value of the real probability;

Calculate the scores of the number of domain names accessed in the first period corresponding to the working day, the number of requests during working hours, the number of requests during rest periods, the number of requests during sleep periods, the number of hours, and the number of request sources, and take the weighted average of the above scores to get Median value of workday activity;

Calculation of the hourly occurrence score, mobile UserAgent score, PC UserAgent score, rest period VS work period request score, rest period VS sleep period request score, and access domain name score for the first cycle corresponding to the rest day. The weighted average of the scores is used to obtain the median value of the probability that the rest day is the family's export IP;

Calculate the number of requests for the first period of the rest period corresponding to the rest day, the distribution of UserAgents, the number of hours, the number of domain names vs. the number of sources, and the number of users on mobile vs. PCs. Take the weighted average of the above scores to get the break The day is the median value of the probability of a real person;

Calculate the scores of the number of domain names accessed in the first period, the scores of requests during working hours, the scores of requests during rest periods, the scores of requests during sleep periods, the scores of hour occurrences, and the scores of request sources in the first cycle corresponding to the rest day, and take the weighted average of the above scores to get The median activity level on rest days.

Wherein, the step of calculating the temporary specific indicators for the current second period in the second period according to one or more weighted averages of working days and one or more weighted averages of rest days includes:

After preprocessing and normalization, the scores of PC-side UserAgents on weekdays vs. holidays, mobile-side UserAgents on weekdays vs. holidays, and the number of requests on weekdays vs. holidays are obtained. The weighted average of the above three scores is taken to compare with the work The weighted average of the median value of the IP probability of office exits on a daily basis is the IP probability of office exits in the current cycle;

Taking working days as the median value of household export IP probability and rest days as the weighted average of household export IP probability as the household export IP probability in this cycle;

The weighted average of the median probability of being a real person on weekdays and the median probability of being a real person on rest days is taken as the probability of being a real person in this cycle;

The weighted average of the median activity level on weekdays and the median activity level on rest days is used as the activity score for this cycle.

The maximum number of UserAgents on mobile terminals on weekdays, the number of UserAgents on mobile terminals on rest days, the number of UserAgents on PCs on weekdays, and the number of UserAgents on PCs on rest days are used as the grouping of the number of people in this cycle.

Wherein, the final specific indicators include at least any one or multiple items of the following information,

IP, IPInt, update ID, the number of times the IP is updated, the final number of people grouped, the final sum of office exit IP probabilities, the final sum of home exit IP probabilities, the final sum of real-person probabilities, and the final sum of activity scores,

Among them, "IPInt" is a long integer corresponding to the IP address, "update ID" is the number of times to update the final specific index in the second cycle, and "this IP update number" is the number of the final specific index in the second cycle of an IP address update frequency,

The step of adjusting the temporary specific indicators of the current second cycle according to the final specific indicators of the previous second cycle and the third-party IP library and/or third-party IP blacklist to obtain the final specific indicators of the current second cycle include:

For the IP address involved in the final specific index of the last second cycle and the temporary specific index of the current second cycle, the final specific index of the current second cycle is obtained through the following calculation:

When the population grouping of the temporary specific indicators in the current second cycle and the final population grouping of the final specific indicators in the previous second cycle are adjacent groups, select the group with the largest number of people as the final population grouping of the current second cycle, otherwise select the current The grouping of the number of people in the temporary specific indicators for the second cycle,

The sum of the IP probability of the office exit in the temporary specific index of the current second cycle plus the final IP probability of the office exit in the final specific index of the second cycle is used as the final sum of the IP probability of the office exit in the current second cycle,

In the temporary specific index of the current second cycle, the sum of the final IP probability of household export plus the final IP probability of household export in the final specific index of the second cycle is used as the final sum of IP probability of household export in the current second cycle,

The sum of the probability of being a real person in the temporary specific indicators of the current second cycle plus the final probability of being a real person in the final specific indicators of the second cycle is the sum of the final probability of being a real person in the current second cycle,

After adding the sum of the activity score in the temporary specific index of the current second period to the final activity score of the second period, divide by the final update times of the update ID record, and multiply by the IP update times, as the current second period The sum of the final liveness scores;

For the IP address involved in the temporary specific index of the current second cycle that is not involved in the final specific index of the last second cycle, the temporary specific index of the current second cycle is obtained by the following calculation:

The number of people grouped in the temporary specific indicators of the current second cycle is used as the final number of people grouped in the current second cycle,

Taking the IP probability of being an office exit in the temporary specific indicators of the current second cycle as the final sum of the probability of being an office exit IP in the current second cycle,

Taking the export IP probability of households in the temporary specific indicators of the current second cycle as the final IP probability of household exports in the current second cycle,

Taking the probability of being a real person in the temporary specific indicators of the current second cycle as the sum of the final probability of real people in the current second cycle,

Divide the activity score in the temporary specific indicators of the current second period by the update ID, multiply the number of IP updates, and use it as the sum of the final activity scores of the current second period;

Using the final specific index of the current second cycle to cover the final specific index of the last second cycle, recording the number of times of updating the final specific index of the second cycle and the number of updates to the corresponding IP address.

Wherein, according to the final specific index of the last second cycle and the third-party IP library and/or third-party IP blacklist, the temporary specific index of the current second cycle is adjusted to obtain the final specific index of the current second cycle The steps also include:

Filtering out the data corresponding to the IP in the current second-period temporary specific indicators that is not grammatical or the corresponding IP is the LAN IP;

According to the additional information of IP addresses contained in the third-party IP database, adjust the temporary specific indicators in the current second cycle to be office exit IP probability, home exit IP probability and real person probability;

Generate IP credit taint data according to the third-party IP blacklist, and add the IP credit taint data into the final specific indicators of the current second cycle.

Wherein, the method also includes:

provide an interface to a third party allowing access to credit data at said IP address via said interface; or,

Receiving an IP verification request for an IP address from a third party, searching for credit data corresponding to the IP address, evaluating the credit level of the IP address according to the credit data, and returning the evaluation result to the third party.

According to another aspect of the present invention, an IP address analysis system is also provided, including a big data platform and an offline computing platform;

The big data platform is used to store original logs, calculate original logs, collect and store historical data of IP addresses;

The offline computing platform is used to analyze the historical data of the IP address collected by the big data platform, and generate the credit data of the IP address.

The method and system of the present invention collect historical data of IP addresses, analyze the historical data of IP addresses, generate credit data of IP addresses, and realize detailed and accurate analysis of IP addresses, and determine the attributes of IP addresses with big data. With a comprehensive and accurate understanding of the IP address credit situation, it can be applied to IP address legality verification, IP address interception and other scenarios, which solves the problem of wrong perception of IP address security and effectively prevents misjudgment of IP address legality, IP address False blocking occurs.

Other characteristic features and advantages of the present invention will become apparent from the following description of exemplary embodiments read with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate the embodiments of the invention and together with the description serve to explain the principles of the invention. In the drawings, like reference numerals are used to denote like elements. The drawings in the following description are some, but not all, embodiments of the present invention. Those skilled in the art can obtain other drawings based on these drawings without creative efforts.

FIG. 1 exemplarily shows the flow of an IP address analysis method provided by Embodiment 1 of the present invention;

Fig. 2 exemplarily shows the application principle of the technical solution provided by the embodiment of the present invention;

FIG. 3 exemplarily shows the architecture of an IP address analysis system provided by Embodiment 2 of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other.

The existing interception scheme intercepts through a single fixed rule, which does not have enough granularity and dimension for the analysis of access requests, and lacks awareness of the identity of the visitor, which is likely to cause false blocking.

In order to solve the above problems, an embodiment of the present invention provides an IP address analysis method. Embodiment 1 of the present invention will be described below with reference to the accompanying drawings.

The embodiment of the present invention provides an IP address analysis method, which analyzes the IP address in detail based on past access history data of the IP address, related additional information and other data, obtains the credit data of the IP address, and uses the credit data of the IP address to The evaluation of IP addresses realizes detailed and accurate analysis and judgment of IP addresses, evaluates the security degree of IP addresses with credit data, and effectively prevents misjudgments and blocking. The specific process is shown in Figure 1, including:

Step 101, collecting historical data of IP addresses;

In this step, the original logs, such as CDN server logs, are collected according to the first cycle, and analyzed, and the information in the CDN server logs is formatted to obtain predefined indicators. The CDN server log is specifically a CDN nginx log, and may also be calculated through other network access logs.

The predefined indicators include at least one or more of the following information:

Time, IP, number of requests during working hours, number of requests during rest periods, number of requests during sleep periods, file size requested during working hours, file size requested during rest periods, file size requested during sleep periods, number of UserAgents during working hours, number of UserAgents during rest periods, and UserAgent during sleep periods number, the number of UserAgents at the mobile end, the number of UserAgents at the PC end, the number of access sources, the number of access domain names, and the number of hours of appearance. The time when the CDN server log is generated (ie user access time), the "IP" refers to the IP address involved, and the "number of requests during working hours" refers to the number of requests sent by the IP address during the working hours, The "number of requests during the rest period" refers to the number of requests sent by the IP address during the rest period, and the "number of requests during the sleep period" refers to the number of requests sent by the IP address during the sleep period, The "requested file size during working hours" refers to the sum of the file sizes requested by the IP address during the working hours, and the "requested file size during rest periods" refers to the sum of the file sizes requested by the IP address during the rest periods. The total file size, the "sleep period request file size" refers to the total file size requested by the IP address during the sleep period, and the "Working period UserAgent number" refers to the IP address during the working period The number of UserAgents that appear under the address, the "number of UserAgents during the rest period" refers to the number of UserAgents that appear under the IP address during the rest period, and the "number of UserAgents during the sleep period" refers to the number of UserAgents that appear under the IP address during the rest period. The number of UserAgents appearing under the IP address, the "number of UserAgents at the mobile terminal" refers to the number of UserAgents accessed by the IP address through the mobile terminal, and the "number of UserAgents at the PC end" refers to the number of UserAgents accessed by the IP address through the PC terminal The number of UserAgents, the "number of access sources" refers to the number of access sources of the IP address in the first period, and the "number of domain names visited" refers to the number of visits from the IP address in the first period The number of domain names, the "number of occurrence hours" refers to the number of hours that the IP address access occurs within the first period (that is, the hour of IP address access is counted as one occurrence hour, such as 2 o'clock and 4 o'clock If all IP accesses are available, the hour value will be set to 2).

The predefined indicators corresponding to each IP address are stored, and each IP address corresponds to one or more predefined indicators. Specifically, the predefined index can be stored in a hive table.

The first period involved in this step is preferably 1 natural day (24 hours).

Preferably, the historical data of the IP address can also be obtained from the third-party platform, such as obtaining the third-party IP library and/or the third-party IP blacklist from the third-party platform once or more in the second cycle, and the third-party IP library is often Additional information including IP addresses (such as the distribution of IP addresses or IP address segments, that is, the country, province, city, and operator corresponding to the IP segment, and may also indicate a company name or a data center). The third-party platform data is updated irregularly, so it can be obtained after the third-party platform data is updated, or before the second cycle is ready for calculation.

Step 102, analyzing the historical data of the IP address to generate the credit data of the IP address;

In this step, pre-process and normalize the predefined indicators of the first period corresponding to the working day in the second period to obtain the median value of each working day, and the first period corresponding to the rest day in the second period After preprocessing and normalizing the predefined indicators of the cycle, the median value of each rest day is obtained. The second cycle includes multiple first cycles corresponding to working days and multiple first cycles corresponding to rest days; The weighted average (including arithmetic weighted average or geometric weighted average) of the above-mentioned median values of each working day is processed to obtain the weighted average value of the working day; value; based on the weighted average value of one or more working days and the weighted average value of one or more rest days, the current temporary specific indicators for the second cycle in the second cycle are obtained. The temporary specific indicators for the current second cycle include: Probability of export IP, this cycle is the probability of family export IP, this cycle is the probability of real people, the activity score of this cycle, and the number of people in this cycle are grouped; according to the final specific indicators of the previous and second cycles and the third-party IP library and/or third-party IP The blacklist is to adjust the temporary specific index of the current second cycle to obtain the final specific index of the current second cycle, and use the final specific index of the current second cycle as the credit data of the IP address.

The second cycle is an integer multiple of the first cycle; preferably, when the first cycle is days, the second cycle is months or weeks.

The specific algorithm of this step is illustrated below with an example. The first period involved is 1 day, and the second period is 1 month; the normalization algorithm involved in the embodiment of the present invention uses a deformed sigmoid function, 1.0/(1.0+math.exp(-numerator/denominator+4.0 )), since the input values are all greater than or equal to 0, add 4.0.

1. Summary of workday data:

a) Find the median value of the daily data on weekdays

In this step, the IP is used as the dimension to calculate the number of Mobile UserAgents, the number of PC UserAgents, and the median number of requests in the first cycle corresponding to the working day.

The number of requests: the number of requests during the working period plus the number of requests during the rest period plus the number of requests during the sleep period.

(1) Probability that the working day IP address is the home export IP:

Fraction of hourly occurrences: the numerator of the normalization algorithm is 6, and the denominator is the number of hours;

Mobile UserAgent score: Normalization algorithm, the numerator is the average number of mobile UserAgents under each IP within a period of time (such as 10, updated irregularly), and the denominator is the number of mobile UserAgents;

Score of PC-side UserAgents: normalized algorithm, the numerator is the average value of PC-side UserAgents under each IP within a period of time (such as 5, updated irregularly), and the denominator is the number of PC-side UserAgents;

Rest period VS fraction of requests during working hours: Normalization algorithm, the numerator is the number of requests during the rest period divided by 4, and the denominator is the number of requests during the working period divided by 12;

Rest period VS sleep period request score: normalization algorithm, the numerator is the number of requests during the rest period divided by 4, and the denominator is the number of requests during the sleep period divided by 8;

Score of the number of domain names visited: normalized algorithm, the numerator is the average number of domain names visited by each IP every day for a period of time (updated from time to time), and the denominator is the number of domain names;

The weighted mean of the number of occurrences in the above hours, the number of mobile UserAgent scores, the number of PC-side UserAgent scores, the score of requests during rest periods vs. working hours, the scores of requests during rest periods vs. sleep periods, and the scores of access domain names is the family export IP probability on weekdays Median.

(2) The working day is the median value of the IP probability of the office exit:

Score of requests during working hours: Normalization algorithm, the numerator is the number of requests during working hours divided by 12, and the denominator is the average number of hours of requests for each IP working hours during a period of time (updated irregularly);

Score of requests during working hours VS break hours: preprocessing and normalization algorithm, the numerator is the number of requests during working hours divided by 12, and the denominator is the number of requests during rest periods divided by 4;

Rest period VS score of sleep period requests: preprocessing and normalization algorithm, the numerator is the number of requests during the rest period divided by 4, and the denominator is the number of requests during the sleep period divided by 8;

PC-side UserAgent number score: preprocessing and normalization algorithm, the numerator is the average value of PC-side UserAgent numbers under each IP within a period of time (such as 10, updated irregularly), and the denominator is the PC-side UserAgent number;

Number of UserAgents during working hours vs. rest periods: preprocessing and normalization algorithm, the numerator is the number of UserAgents during working hours, and the denominator is the number of UserAgents during rest periods;

The above scores for the number of requests during working hours, the number of requests for working hours vs. rest periods, the scores for requests for rest periods vs. sleep periods, the scores for the number of UserAgents on the PC side, and the scores for the number of UserAgents during working hours vs. break periods. The above scores are weighted and averaged to obtain the working day is the median value of the office exit IP probability;

(3) The working day is the middle value of the probability of a real person:

Request distribution score: preprocessing and normalization algorithm, the numerator is the standard deviation of the number of requests divided by 12 during the working period, the number of requests divided by 4 during the rest period, and the number of requests divided by 8 during the sleep period, and the denominator is 1;

UserAgent number distribution score: Normalization algorithm, the numerator is the number of UserAgents during the working period, the number of UserAgents during the rest period, and the standard deviation of the number of UserAgents during the sleep period, and the denominator is 1;

Hour occurrence fraction: Normalization algorithm, the numerator is 6, and the denominator is the hour occurrence number;

The number of domain names VS the number of sources: normalization algorithm, the numerator is the number of sources, and the denominator is the number of domain names;

Fraction of the number of UserAgents on the mobile VSPC: a normalization algorithm, the numerator is the number of UserAgents on the mobile side, and the denominator is the number of UserAgents on the PC side.

The weighted average of the above scores is the median value of the probability of being a real person on a weekday.

(4) Median value of workday activity:

Score of the number of domain names visited: normalization algorithm, the numerator is the number of domain names visited, and the denominator is 10;

Score of requests during working hours: Normalization algorithm, the numerator is the number of requests during working hours divided by 12, and the denominator is the average number of hours of requests for each IP working hours during a period of time (updated irregularly);

Rest period request count score: Normalization algorithm, the numerator is the number of rest period requests divided by 4, and the denominator is the average number of hours of requests for each IP rest period within a period of time (updated irregularly);

Score of requests during sleep period: Normalization algorithm, the numerator is the number of requests during sleep period divided by 8, and the denominator is the average number of hours of requests for each IP sleep period within a period of time (updated irregularly);

Hour occurrence fraction: Normalization algorithm, the numerator is the hour occurrence number, and the denominator is 6;

Request source number score: normalization algorithm, the numerator is the number of request sources, and the denominator is the average daily number of request sources per IP (updated irregularly).

A weighted average of all the above scores yields the median weekday activity.

b) Find the weighted average of the median value of the daily data on weekdays:

Taking IP as the dimension, calculate the weighted average of the number of Mobile UserAgents, the weighted average of the number of PC-side UserAgents, and the weighted average of the number of requests corresponding to the working day in the first period. The working day is the weighted mean of the IP probability of the home exit, and the weighted average of the IP probability of the office exit on the working day The average value is the probability-weighted average of real people on weekdays, and the weighted average of activity on weekdays.

2. Summary of rest day data:

a) Find the median value of the daily data on rest days

Taking IP as the dimension, calculate the median value of the number of Mobile UserAgents, PC UserAgents, and requests in the first period corresponding to the rest day.

The number of requests: the number of requests during the working period plus the number of requests during the rest period plus the number of requests during the sleep period.

(1) The rest day is the median value of the family's export IP probability: similar to the working day algorithm;

(2) The rest day is the middle value of the probability of a real person: similar to the working day algorithm;

(3) Median value of activity on rest days: similar to the algorithm on working days.

b) Find the weighted mean or maximum value of the median value of the daily data on rest days:

Taking IP as the dimension, calculate the maximum number of mobile-side UserAgents, the maximum number of PC-side UserAgents, and the weighted average of requests in the first cycle corresponding to the rest day. The rest day is the weighted average of the probability of family exit IP, and the rest day is the weighted average of the probability of real people. , the weighted mean of activity on rest days.

3. Summarize the data of working days and rest days to obtain temporary specific indicators for the current second cycle:

Calculate the working days and rest days according to the IP connection, and get:

IPInt: Convert the IP to the corresponding long integer.

Grouping the number of people in this cycle: calculate the number of PC-side UserAgents on weekdays, and calculate the weighted average of the number of PC-side UserAgents corresponding to the weekdays in the first cycle. The number of mobile UserAgents on working days, and calculate the weighted average of the number of mobile UserAgents corresponding to working days in the first period. For the number of PC-side UserAgents on rest days, calculate the weighted average of the number of PC-side UserAgents corresponding to the rest days in the first period. The number of UserAgents on the mobile terminal on rest days, and calculate the weighted average of the number of UserAgents on the mobile end corresponding to the rest days in the first period. After finding the maximum number of UserAgents on the PC side on weekdays, the number of UserAgents on the mobile side on weekdays, the number of UserAgents on the PC side on rest days, and the number of UserAgents on the mobile end on rest days, group them according to the following: 1:0-1, 2:2-5, 3:6-10, 4:11-30, 5:31-50, 6:51-100, 7:101-500, 8:501-2000, 9:>2000.

This cycle is the probability of office export IP:

Number of PC-side UserAgents on weekdays vs. holidays: normalization algorithm, the numerator is the number of PC-side UserAgents on weekdays, and the denominator is the number of PC-side UserAgents on holidays;

Number of mobile UserAgents on weekdays vs. holidays: normalization algorithm, the numerator is the number of mobile UserAgents on weekdays, and the denominator is the number of mobile UserAgents on holidays;

Working day VS fraction of holiday requests: normalization algorithm, the numerator is the number of workday requests, and the denominator is the number of holiday requests;

The weighted average of the above three scores, and the weighted average of the median value of the IP probability of office exits on working days is the IP probability of office exits in this cycle.

This cycle is the household export IP probability: the weighted average value of the middle value of the household export IP probability on weekdays and the rest day is the household export IP probability.

This cycle is the probability of a real person: the weighted average of the median probability of being a real person on weekdays and the median probability of being a real person on rest days.

Activity score for this period: the weighted average of the median activity on weekdays and the median activity on rest days.

Grouping of the number of people in this cycle: group by the maximum of the number of mobile UserAgents on weekdays, the number of mobile UserAgents on rest days, the number of PC UserAgents on weekdays, and the number of PC UserAgents on rest days.

Store the temporary specific indicators of the current second cycle into the MySQL temporary data table and enter the adjustment stage.

adjustment stage

The current second cycle temporary specific indicators stored in MySQL and the final specific indicators of the previous second cycle stored in MySQL in the previous second cycle are used as the input of this stage.

Traversing all IPs in the MySQL temporary data table, each IP corresponds to the current second period temporary specific indicators.

1. Filter out data with ungrammatical IP and filter out LAN IP.

2. Obtain the third-party IP library information, judge the additional information contained in the third-party IP library, judge whether the string contains the following sensitive strings, and return the corresponding adjustment index: "Company", "Data Center", "GSM/TD-SCDMA /LTE". The adjustment index includes the adjustment index for the three probabilities of "probability of real person", "probability of exporting IP for office", "probability of exporting IP for family". If sensitive strings are not included, the adjustment index of the three probabilities is 1. Multiply the three probabilities by the adjustment indices of the three probabilities, and agree that the probabilities must be in the range of [0.05,0.95]. If it is less than 0.05, it will return 0.05, and if it is greater than 0.95, it will return 0.95.

3. Obtain the final specific indicators stored in MySQL in the last second cycle.

For an IP address, depending on whether the IP address has the final specific index of the previous second cycle, the method of generating the final specific index of the current second cycle for the IP address is also different, as follows:

a) If there are final specific indicators for the last and second cycle of this IP, perform corresponding operations on the following indicators:

Update ID: the number of times the second cycle data is updated.

IPInt: Convert the IP to the corresponding long integer.

The number of IP updates: add 1 to the number of IP updates in the previous second cycle.

Final number of people grouping: If the number of people grouping of the temporary specific indicators of the current second cycle and the final number of people grouping of the final specific indicators of the previous second cycle are adjacent groups, the group with the largest number of people will be the final group; otherwise, it will be the temporary data grouping.

Finally, it is the sum of the IP probability of office exits: the IP probability of office exits in the temporary specific indicators of the current second cycle plus the final IP probability of office exits in the final specific indicators of the second cycle.

Finally, it is the sum of household export IP probabilities: the household export IP probability in the temporary specific indicators of the current second cycle plus the final final specific indicators of the second cycle is the final sum of household export IP probabilities.

The sum of the final probability of being a real person: the IP probability of a real person in the temporary specific indicators of the current second cycle plus the sum of the final probability of being a real person in the final specific indicators of the second cycle.

The sum of the final activity score: the activity score in the temporary specific index of the current second period plus the sum of the final activity score of the second period, divided by the final update times of the update ID record, multiplied by the IP update times .

If there is no such IP, perform corresponding operations on the following indicators:

IPInt: Convert the IP to the corresponding long integer.

Update ID: the number of times the second cycle data is updated.

The IP update times: 1.

Final number of people grouping: the number of people grouping for the temporary specific indicators in the current second cycle.

Finally, it is the sum of the IP probability of office exits: the temporary specific index for the current second cycle is the IP probability of office exits.

Finally, it is the sum of household export IP probabilities: the temporary specific indicator for the current second cycle is the household export IP probability.

Finally, it is the sum of real-person probabilities: the real-person probability of temporary specific indicators in the current second cycle.

The sum of the final activity scores: the activity score of the temporary specific indicators in the current second period divided by the update ID, multiplied by the number of updates of the IP.

In addition, IP credit stain data can also be generated according to the third-party IP blacklist, and the IP credit stain data can be added to the final specific index of the current second cycle.

Specifically, the information in the blacklist generally includes the IP addresses or IP address segments included in the blacklist. Credit taint data can preferably be expressed in the form of credit taint scores, for example, the more third-party blacklists exist, the higher the credit taint score, and the credit taint score is 0 if it does not exist in the blacklist.

Using the final specific index of the current second cycle to cover the final specific index of the previous second cycle, recording the number of times of updating the final specific index of the second cycle and the number of updates to the corresponding IP address. Update all the above data to MySQL.

When the current second-period temporary specific indicators and final specific indicators are stored in MySQL, the IP can be used as the index; the IP can also be converted into the corresponding long integer and then the corresponding long integer can be used as the index, divided into 256 parts according to the IPInt and then stored in separate tables. Convenient query and improve query speed.

After obtaining the final specific index updated according to the current second cycle, the final specific index corresponding to the IP address is used as the credit data of the IP address, and the IP address is evaluated according to the credit data. An interface can be provided to a third party to allow access to the credit data of the IP address through the interface; an IP verification request for the IP address can also be received from a third party, and the credit data corresponding to the IP address can be searched for, and according to the credit data Perform a credit rating evaluation on the IP address, and return the evaluation result to the third party.

It can be applied to the firewall's IP interception operation. The firewall judges the legality of the IP address based on the credit data of the IP address. It can also independently form an IP credit rating platform to provide the IP verification result to the firewall. It can also provide a secondary verification mechanism without affecting the existing firewall functions, that is, when the firewall determines that the IP address is suspicious, the IP credit rating platform will conduct secondary verification based on the credit data to further improve the accuracy of firewall interception To prevent accidental blocking.

The IP address analysis method provided by the embodiment of the present invention can be combined with the existing Internet architecture. As shown in FIG. The IP address analysis method provided by the embodiment obtains the IP user attribute data mainly composed of final specific indicators, the IP stain data mainly generated according to the third-party IP blacklist, and the IP address database data obtained according to the third-party IP address database, and IP user attribute data, IP taint data and IP address database data are integrated into the IP credit rating platform to perform credit rating on IP addresses and obtain credit data of IP addresses. The credit data of IP addresses comprehensively describes the characteristics of IP addresses, which can be used to confirm the security of IP addresses in the field of information security, or in the field of IP-based user portraits, and realize the accurate description of IP addresses based on big data analysis. The application results can also be fed back to the IP credit rating platform to perform algorithm iteration and parameter adjustment on the existing results, endowing the system with the ability of self-learning and self-adjustment, and further improving the accuracy of IP analysis by the IP credit rating platform.

Embodiment 2 of the present invention will be described below with reference to the accompanying drawings.

The embodiment of the present invention provides a kind of IP address analyzing system, its structure is shown in Figure 3, comprises:

Big data computing platform and offline computing platform.

Among them, the big data platform includes: Hadoop computing platform, spark computing platform;

Offline computing platforms include: servers or server clusters.

The big data platform is used to store original logs, calculate original logs, collect and store historical data of IP addresses;

The offline computing platform is used to analyze the historical data of the IP address collected by the big data platform, and generate the credit data of the IP address.

The offline computing platform can also communicate with a third party through the big data platform, provide the credit data of the IP address to the third party, or receive a query request from a third party and return the auxiliary IP address verification based on the credit data. information.

Preferably, the IP address analysis system also includes a storage platform, the storage platform supports the MySQL system, and can be used to store the original log, the credit data of the IP address, the third-party IP library obtained from the third party and the third-party IP blacklist , the final specific indicators of this cycle, the final specific indicators of the previous second cycle, the temporary specific indicators of the current second cycle, and the intermediate data generated during the calculation process, etc.

The embodiment of the present invention provides an IP address analysis system, which can be combined with an IP address analysis method provided by the embodiment of the present invention, by collecting historical data of IP addresses, analyzing the historical data of IP addresses, and generating The credit data of IP addresses realizes the detailed and accurate analysis of IP addresses, determines the attributes of IP addresses with big data, and has a comprehensive and accurate understanding of the credit status of IP addresses, which can be applied to IP address legality verification, IP address interception, etc. In the scenario, the problem of erroneous perception of IP address security is solved, effectively preventing misjudgment of the legality of IP addresses and false blocking of IP addresses.

The content described above can be implemented alone or combined in various ways, and these variants are all within the protection scope of the present invention.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those of ordinary skill in the art should understand that: it can still modify the technical solutions described in the aforementioned embodiments, or perform equivalent replacements for some of the technical features; and these The modification or replacement does not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (10)

1. A kind of IP address analysis method, it is characterized in that, comprising: collect historical data on IP addresses; Analyze the historical data of the IP address to generate the credit data of the IP address. 2. IP address analysis method according to claim 1, is characterized in that, the step of the historical data of described collection IP address comprises: Collect and parse raw logs during the first cycle; Formatting the information in the original log to obtain a predefined indicator, the predefined indicator at least includes any one or more of the following information: Time, IP, number of requests during working hours, number of requests during rest periods, number of requests during sleep periods, file size requested during working hours, file size requested during rest periods, file size requested during sleep periods, number of UserAgents during working hours, number of UserAgents during rest periods, sleep The number of UserAgents in the time period, the number of UserAgents on the mobile terminal, the number of UserAgents on the PC terminal, the number of access sources, the number of domain names accessed, and the number of hours; The predefined indicators corresponding to each IP address are stored, and each IP address corresponds to one or more predefined indicators. 3. IP address analysis method according to claim 2, is characterized in that, the step of the historical data of described collection IP address also comprises: In the second cycle, obtain the third-party IP library and/or third-party IP blacklist from the third-party platform one or more times. 4. the IP address analysis method according to claim 3, is characterized in that, analyzing the historical data of IP address, generating the credit data of IP address comprises: Preprocess and normalize the predefined indicators of the first period corresponding to working days in the second period to obtain the median value of each working day, and the predefined indicators of the first period corresponding to rest days in the second period After the indicators are preprocessed and normalized, the median value of each rest day is obtained, and the second cycle includes multiple first cycles corresponding to working days and multiple first cycles corresponding to rest days; Performing weighted average processing on the median value of each working day respectively to obtain the weighted mean value of working days; Perform weighted average or maximum value processing on the median value of each rest day to obtain the weighted average or maximum value of rest days; Based on the weighted average value of one or more working days and the weighted average value of one or more rest days, the current temporary specific indicators for the second period in the second period are calculated, and the temporary specific indicators for the current second period include: This cycle is the probability of office exit IP, this cycle is the probability of home exit IP, this cycle is the probability of real people, the activity score of this cycle, and the number of people in this cycle are grouped; According to the final specific indicators of the previous second cycle and the third-party IP library and/or third-party IP blacklist, the temporary specific indicators of the current second cycle are adjusted to obtain the final specific indicators of the current second cycle, and the current The final specific index of the second period is used as the credit data of the IP address. 5. The IP address analysis method according to claim 4, characterized in that, after preprocessing and normalizing the predefined indicators of the first cycle corresponding to the working day in the second cycle, the middle value of each working day is obtained. value, the step of obtaining the median value of each rest day after preprocessing and normalizing the predefined indicators corresponding to the rest days in the second cycle of the first cycle includes: Calculate the hours of the first cycle corresponding to the working day, the score of the user agent on the mobile terminal, the score of the user agent on the PC terminal, the score of the number of requests during the rest period vs. the work period, the score of the number of requests during the rest period vs. the sleep period, and the score of the number of domain names visited. For the above The weighted average of the scores is used to obtain the median value of the probability of exporting IP for households on weekdays; Calculate the number of requests in the first cycle corresponding to the working day, the number of requests in the working period VS the rest period, the number of requests in the rest period VS the sleep period, the number of PC-side UserAgents, and the number of UserAgents in the working period VS rest period, for the above The weighted average of the score values is used to obtain the median value of the IP probability of an office exit on a working day; Calculate the distribution score of the number of requests in the first period corresponding to the working day, the distribution score of the number of UserAgents, the score of the number of hours, the score of the number of domain names VS the number of sources, the score of the number of UserAgents on the mobile side VS PC side, and take the weighted average of the above scores to get the working day is the median value of the real probability; Calculate the scores of the number of domain names accessed in the first period corresponding to the working day, the number of requests during working hours, the number of requests during rest periods, the number of requests during sleep periods, the number of hours, and the number of request sources, and take the weighted average of the above scores to get Median value of workday activity; Calculation of the hourly occurrence score, mobile UserAgent score, PC UserAgent score, rest period VS work period request score, rest period VS sleep period request score, and access domain name score for the first cycle corresponding to the rest day. The weighted average of the scores is used to obtain the median value of the probability that the rest day is the family's export IP; Calculate the number of requests for the first period of the rest period corresponding to the rest day, the distribution of UserAgents, the number of hours, the number of domain names vs. the number of sources, and the number of users on mobile vs. PCs. Take the weighted average of the above scores to get the break The day is the median value of the probability of a real person; Calculate the scores of the number of domain names accessed in the first period, the scores of requests during working hours, the scores of requests during rest periods, the scores of requests during sleep periods, the scores of hour occurrences, and the scores of request sources in the first cycle corresponding to the rest day, and take the weighted average of the above scores to get The median activity level on rest days. 6. The IP address analysis method according to claim 4, characterized in that, the current second cycle temporary in the second cycle is calculated based on one or more weighted average values of working days and one or more weighted average values of rest days. The steps for specific indicators include: After preprocessing and normalization, the scores of PC-side UserAgents on weekdays vs. holidays, mobile-side UserAgents on weekdays vs. holidays, and the number of requests on weekdays vs. holidays are obtained. The weighted average of the above three scores is taken to compare with the work The weighted average of the median value of the IP probability of office exits on a daily basis is the IP probability of office exits in the current cycle; Taking working days as the median value of household export IP probability and rest days as the weighted average of household export IP probability as the household export IP probability in this cycle; The weighted average of the median probability of being a real person on weekdays and the median probability of being a real person on rest days is taken as the probability of being a real person in this cycle; The weighted average of the median activity level on weekdays and the median activity level on rest days is used as the activity score for this cycle. The maximum number of UserAgents on mobile terminals on weekdays, the number of UserAgents on mobile terminals on rest days, the number of UserAgents on PCs on weekdays, and the number of UserAgents on PCs on rest days are used as the grouping of the number of people in this cycle. 7. IP address analysis method according to claim 4, is characterized in that, described final concrete index comprises any one or any multiple of following information at least, IP, IPInt, update ID, the number of times the IP is updated, the final number of people grouped, the final sum of office exit IP probabilities, the final sum of home exit IP probabilities, the final sum of real-person probabilities, and the final sum of activity scores, Among them, "IPInt" is a long integer corresponding to the IP address, "update ID" is the number of times to update the final specific index in the second cycle, and "this IP update number" is the number of the final specific index in the second cycle of an IP address update frequency, The step of adjusting the temporary specific indicators of the current second cycle according to the final specific indicators of the previous second cycle and the third-party IP library and/or third-party IP blacklist to obtain the final specific indicators of the current second cycle include: For the IP address involved in the final specific index of the last second cycle and the temporary specific index of the current second cycle, the final specific index of the current second cycle is obtained through the following calculation: When the population grouping of the temporary specific indicators in the current second cycle and the final population grouping of the final specific indicators in the previous second cycle are adjacent groups, select the group with the largest number of people as the final population grouping of the current second cycle, otherwise select the current The grouping of the number of people in the temporary specific indicators for the second cycle, The sum of the IP probability of the office exit in the temporary specific index of the current second cycle plus the final IP probability of the office exit in the final specific index of the second cycle is used as the final sum of the IP probability of the office exit in the current second cycle, In the temporary specific index of the current second cycle, the sum of the final IP probability of household export plus the final IP probability of household export in the final specific index of the second cycle is used as the final sum of IP probability of household export in the current second cycle, The sum of the probability of being a real person in the temporary specific indicators of the current second cycle plus the final probability of being a real person in the final specific indicators of the second cycle is the sum of the final probability of being a real person in the current second cycle, After adding the sum of the activity score in the temporary specific index of the current second period to the final activity score of the second period, divide by the final update times of the update ID record, and multiply by the IP update times, as the current second period The sum of the final liveness scores; For the IP address involved in the temporary specific index of the current second cycle that is not involved in the final specific index of the last second cycle, the temporary specific index of the current second cycle is obtained by the following calculation: The number of people grouped in the temporary specific indicators of the current second cycle is used as the final number of people grouped in the current second cycle, Taking the IP probability of being an office exit in the temporary specific indicators of the current second cycle as the final sum of the probability of being an office exit IP in the current second cycle, Taking the export IP probability of households in the temporary specific indicators of the current second cycle as the final IP probability of household exports in the current second cycle, Taking the probability of being a real person in the temporary specific indicators of the current second cycle as the sum of the final probability of real people in the current second cycle, Divide the activity score in the temporary specific indicators of the current second period by the update ID, multiply the number of IP updates, and use it as the sum of the final activity scores of the current second period; Using the final specific index of the current second cycle to cover the final specific index of the last second cycle, recording the number of times of updating the final specific index of the second cycle and the number of updates to the corresponding IP address. 8. IP address analysis method according to claim 7, is characterized in that, described according to the final specific index of last second period and third-party IP storehouse and/or third-party IP blacklist, to described current second The steps to adjust the interim specific indicators for the current cycle to obtain the final specific indicators for the current second cycle also include: Filtering out the data corresponding to the IP in the current second-period temporary specific indicators that is not grammatical or the corresponding IP is the LAN IP; According to the additional information of IP addresses contained in the third-party IP database, adjust the temporary specific indicators in the current second cycle to be office exit IP probability, home exit IP probability and real person probability; Generate IP credit taint data according to the third-party IP blacklist, and add the IP credit taint data into the final specific indicators of the current second cycle. 9. IP address analysis method according to claim 1, is characterized in that, this method also comprises: provide an interface to a third party allowing access to credit data at said IP address via said interface; or, Receiving an IP verification request for an IP address from a third party, searching for credit data corresponding to the IP address, evaluating the credit level of the IP address according to the credit data, and returning the evaluation result to the third party. 10. An IP address analysis system, characterized in that it includes a big data platform and an offline computing platform; The big data platform is used to store original logs, calculate original logs, collect and store historical data of IP addresses; The offline computing platform is used to analyze the historical data of the IP address collected by the big data platform, and generate the credit data of the IP address.