CN111931066B - Real-time recommendation system design method - Google Patents
Real-time recommendation system design method Download PDFInfo
- Publication number
- CN111931066B CN111931066B CN202010950935.1A CN202010950935A CN111931066B CN 111931066 B CN111931066 B CN 111931066B CN 202010950935 A CN202010950935 A CN 202010950935A CN 111931066 B CN111931066 B CN 111931066B
- Authority
- CN
- China
- Prior art keywords
- data
- data flow
- real
- flow
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a design method of a real-time recommendation system, which comprises the following steps: A. acquiring real-time data of a service system database, and sending the data to different message queues; B. according to the data streams in different message queues, carrying out association between the data streams on different data streams, and forming a result stream after association; C. and dynamically associating the result flow with the rule configuration table, and outputting data meeting the rules. The method can immediately sense the rule change according to different set recommendation rules, and output data conforming to the rules in real time by applying the method to the data stream, so that accurate recommendation is realized.
Description
Technical Field
The invention relates to the technical field of data real-time calculation, in particular to a design method of a real-time recommendation system.
Background
The appearance and popularization of the internet bring a great deal of information to users, and the requirement of the users on the information in the information age is met, but the quantity of the information on the internet is greatly increased along with the rapid development of the network, so that the users cannot obtain the part of information which is really useful for the users when facing a great amount of information, and the use efficiency of the information is reduced on the contrary, which is the so-called information overload problem. One very potential solution to the information overload problem is a recommendation system, which is a personalized information recommendation system that recommends information, products, etc. of interest to a user according to the information needs, interests, etc. of the user. A good recommendation system not only can provide personalized services for users, but also can establish close relations with the users, and the users can generate dependence on the recommendation.
The improvement of the timeliness of information recommendation is one of the keys for improving the good experience of the user, so that the real-time recommendation is an improvement and optimization of a recommendation system. At present, under a service scene that an enterprise recommends in real time, an offline data warehouse concept is generally utilized to store relevant data into a database, users meeting rules are screened out through various association relations, then logic is packaged, and the users meeting the rules who run out the previous day every other day are pushed to a service department or run batch at an hourly scheduling interval through timing scheduling of a scheduler.
The scheme cannot dynamically sense and adjust the rule in real time, and cannot screen out the users meeting the rule in real time, so that the user experience is poor.
Disclosure of Invention
The invention provides a design method of a real-time recommendation system, aiming at the problems that the rules cannot be dynamically adjusted in real time and the user experience is poor in the existing scheme. The method aims to dynamically sense the rule and timely adjust the rule so as to enable the recommended activities to have the best effect to the user to have good experience to the maximum extent.
The invention discloses a design method of a real-time recommendation system, which comprises the following steps:
A. acquiring real-time data of a service system database, and sending the data to different message queues;
B. according to the data streams in different message queues, carrying out association between the data streams on different data streams, and forming a result stream after association;
C. and dynamically associating the result flow with the rule configuration table, and outputting data meeting the rules.
The invention collects the latest data by collecting the data in real time, and carries out matching association between data streams; and collecting the data of the rule configuration table in real time, and performing matching association again to screen out the data which accords with the rule. The rule change can be immediately sensed according to different set recommendation rules, the rule change is applied to data flow, data meeting the rules are output in real time, and accurate recommendation is achieved.
Further, step A includes collecting data logs in a service system database to obtain data modified by a user and corresponding fields; and sending the data to different message queues according to different tables to which the data belongs.
And step A, acquiring real-time data, acquiring a data log, and obtaining corresponding fields so as to perform matching correlation between data streams.
Further, step B includes:
b1, establishing buffer for synchronously buffering the data sent to different message queues and setting a timer;
b2: storing data flowing into the message queue a, and storing a data flow A flowing into the message queue a into a database;
b3: the data flow B which flows into the other message queue B later queries the data flow A stored in the database, and the data flow A is matched and associated with the data flow B through corresponding fields: if the data flow A and the data flow B can be matched, associating the data flow A and the data flow B, and outputting the associated result flow to a message queue c; if the data flow A and the data flow B can not be matched, entering a buffer to inquire the data flow A, associating the data flow A and the data flow B after matching, emptying the buffer, and outputting the associated result flow to a message queue c;
B4. and configuring data failure time aiming at each piece of data in the buffer, and triggering a clearing mechanism to clear the piece of data when the time is up.
The steps B1 and B4 create registers and set timers to time clear data to prevent system memory overflow and program crash. The steps B2 and B3 perform matching association between data streams to obtain a data stream containing a plurality of data stream fields, and the data stream is used as a screening object in the following steps.
Further, step C includes:
C1. collecting a data log of a rule configuration table, and sending data to a message queue d;
C2. establishing a rule configuration data buffer, and synchronously buffering the data sent to the message queue c;
C3. and inquiring and matching the rule configuration data stored in the buffer by the result stream in the message queue c, associating the matched result stream with the rule configuration data stream, and outputting the data meeting the rule.
And C, matching and associating the data stream obtained in the step B with a rule configuration table so as to screen the data conforming to the rule, and obtaining and outputting the data conforming to the rule.
The design method of the real-time recommendation system can immediately sense the rule change according to different set recommendation rules, and output data meeting the rules in real time by applying the data to data streams, so that accurate recommendation is realized.
Drawings
FIG. 1 is a flow chart of a design method of a real-time recommendation system according to the present invention.
FIG. 2 is a diagram of a scenario of data stream association in a real-time recommendation system design method according to the present invention.
Detailed Description
The present invention will be described in further detail with reference to the following examples. This should not be understood as limiting the scope of the above-described subject matter of the present invention to the following examples. Various substitutions and alterations according to the general knowledge and conventional practice in the art are intended to be included within the scope of the present invention without departing from the technical spirit of the present invention as described above.
As shown in fig. 1 and fig. 2, the method for designing a real-time recommendation system of the present invention includes:
A. and acquiring a Binlog log of MySQL data of the service system, outputting the Binlog to different Kafka message queues according to different tables, and receiving data in the different Kafka message queues by a receiving end.
B. Establishing a buffer, synchronously buffering the data sent to different message queues respectively, setting a timer, configuring data failure time aiming at each piece of data in the buffer, and triggering a clearing mechanism when the time is reached to clear the data.
C. And storing the data stream flowing into the message queue a, and storing the data stream A flowing into the message queue a in a database.
D. The data flow B which flows into the other message queue B later queries the data flow A stored in the database, and the data flow A is matched and associated with the data flow B through corresponding fields: if the data flow A and the data flow B can be matched, associating the data flow A and the data flow B, and outputting the associated result flow to a message queue c; if the data flow A and the data flow B can not be matched, entering a buffer to inquire the data flow A, associating the data flow A and the data flow B after matching, emptying the buffer, and outputting the associated result flow to a message queue c;
E. and collecting binlog logs of the rule configuration table in real time, synchronizing the stock rules to a message queue when a system is deployed for the first time, and storing the stock rules into a buffer.
F. And inquiring and matching the rule configuration data stored in the buffer by the result stream in the message queue c, associating the matched result stream with the rule configuration data stream, and outputting the data meeting the rule.
The design method of the real-time recommendation system can immediately sense the rule change according to different set recommendation rules, and output data meeting the rules in real time by applying the data to data streams, so that accurate recommendation is realized.
Claims (3)
1. A real-time recommendation system design method is characterized by comprising the following steps:
A. acquiring real-time data of a service system database, and sending the data to different message queues;
B. according to the data streams in different message queues, carrying out association between the data streams on different data streams, and forming a result stream after association;
C. dynamically associating the result stream with a rule configuration table, and outputting data meeting the rule in real time;
wherein the step C comprises:
C1. collecting a data log of a rule configuration table in real time, and sending data to a message queue d;
C2. establishing a rule configuration data buffer, and synchronously buffering the data sent to the message queue c;
C3. and inquiring and matching the rule configuration data stored in the buffer by the result stream in the message queue c, associating the matched result stream with the rule configuration data stream, and outputting the data meeting the rule in real time.
2. The real-time recommendation system design method of claim 1, wherein: the step A comprises the following steps: collecting data logs in a service system database to obtain data modified by a user and corresponding fields; and sending the data to different message queues according to different tables to which the data belongs.
3. The real-time recommendation system design method of claim 2, wherein: the step B comprises the following steps:
b1, establishing buffer for synchronously buffering the data sent to different message queues and setting a timer;
b2: storing data flowing into the message queue a, and storing a data flow A flowing into the message queue a into a database;
b3: the data flow B which flows into the other message queue B later queries the data flow A stored in the database, and the data flow A is matched and associated with the data flow B through corresponding fields: if the data flow A and the data flow B can be matched, associating the data flow A and the data flow B, and outputting the associated result flow to a message queue c; if the data flow A and the data flow B can not be matched, entering a buffer to inquire the data flow A, associating the data flow A and the data flow B after matching, emptying the buffer, and outputting the associated result flow to a message queue c;
B4. and configuring data failure time aiming at each piece of data in the buffer, and triggering a clearing mechanism to clear the piece of data when the time is up.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010950935.1A CN111931066B (en) | 2020-09-11 | 2020-09-11 | Real-time recommendation system design method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010950935.1A CN111931066B (en) | 2020-09-11 | 2020-09-11 | Real-time recommendation system design method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111931066A CN111931066A (en) | 2020-11-13 |
| CN111931066B true CN111931066B (en) | 2021-09-07 |
Family
ID=73310108
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010950935.1A Active CN111931066B (en) | 2020-09-11 | 2020-09-11 | Real-time recommendation system design method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111931066B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114595250B (en) * | 2021-02-03 | 2025-01-14 | 亚信科技（中国）有限公司 | Multi-stream association method, device, electronic device and computer-readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105512297A (en) * | 2015-12-10 | 2016-04-20 | 中国测绘科学研究院 | Distributed stream-oriented computation based spatial data processing method and system |
| CN106156307A (en) * | 2016-06-30 | 2016-11-23 | 北京奇虎科技有限公司 | The data handling system of a kind of real-time calculating platform and method |
| CN106293960A (en) * | 2016-07-27 | 2017-01-04 | 福建富士通信息软件有限公司 | A kind of method and system realizing data conversion based on strom and internal memory grid |
| CN107743077A (en) * | 2017-11-30 | 2018-02-27 | 西北工业大学 | A method and device for evaluating network performance of information-physical fusion system |
| CN108156141A (en) * | 2017-12-14 | 2018-06-12 | 北京奇艺世纪科技有限公司 | A kind of real time data recognition methods, device and electronic equipment |
| CN110209700A (en) * | 2019-05-24 | 2019-09-06 | 北京奇艺世纪科技有限公司 | A kind of data stream association method, apparatus, electronic equipment and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7139844B2 (en) * | 2000-08-04 | 2006-11-21 | Goldman Sachs & Co. | Method and system for processing financial data objects carried on broadcast data streams and delivering information to subscribing clients |
| CN101364893A (en) * | 2007-08-08 | 2009-02-11 | 华为技术有限公司 | Control device, execution device, method and system for generating filtering rules |
| CN101286895B (en) * | 2008-05-22 | 2010-08-18 | 上海交通大学 | Dynamic configurable data monitoring system and method for distributed network |
| CN102263773B (en) * | 2010-05-25 | 2014-06-11 | 腾讯科技（深圳）有限公司 | Real-time protection method and apparatus thereof |
| CN102035685B (en) * | 2010-12-20 | 2014-08-13 | 中兴通讯股份有限公司 | Alarm treating method and DPI (Deep Packet Inspection) device |
| CN104333485A (en) * | 2014-10-31 | 2015-02-04 | 北京思特奇信息技术股份有限公司 | Business data acquisition and analysis method and system based on interchanger total quantity |
| US10679137B2 (en) * | 2016-01-04 | 2020-06-09 | Adobe Inc. | Systems and methods for determining real-time visitor segments |
-
2020
- 2020-09-11 CN CN202010950935.1A patent/CN111931066B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105512297A (en) * | 2015-12-10 | 2016-04-20 | 中国测绘科学研究院 | Distributed stream-oriented computation based spatial data processing method and system |
| CN106156307A (en) * | 2016-06-30 | 2016-11-23 | 北京奇虎科技有限公司 | The data handling system of a kind of real-time calculating platform and method |
| CN106293960A (en) * | 2016-07-27 | 2017-01-04 | 福建富士通信息软件有限公司 | A kind of method and system realizing data conversion based on strom and internal memory grid |
| CN107743077A (en) * | 2017-11-30 | 2018-02-27 | 西北工业大学 | A method and device for evaluating network performance of information-physical fusion system |
| CN108156141A (en) * | 2017-12-14 | 2018-06-12 | 北京奇艺世纪科技有限公司 | A kind of real time data recognition methods, device and electronic equipment |
| CN110209700A (en) * | 2019-05-24 | 2019-09-06 | 北京奇艺世纪科技有限公司 | A kind of data stream association method, apparatus, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111931066A (en) | 2020-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11831590B1 (en) | Apparatus and method for context-driven determination of optimal cross- protocol communication delivery | |
| US11843651B2 (en) | Personalized recommendation method and system, and terminal device | |
| US10798040B2 (en) | Publish/subscribe mashups for social networks | |
| US20130204999A1 (en) | System and Method for Automatic Sub-Panel Creation and Management | |
| KR20070026595A (en) | Broadcast / multicast service method based on user location information | |
| CN102184257A (en) | Unified searching method, device and system | |
| CN111144938B (en) | Method and system for rating sales lead applicable to automobile industry | |
| CN106203989A (en) | A kind of information processing method and device | |
| CN111931066B (en) | Real-time recommendation system design method | |
| WO2016045367A1 (en) | Multi-data-source data fusion method and device | |
| CN108268529A (en) | It is a kind of that the data summarization method and system dispatched with multi engine are abstracted based on business | |
| CN104811810A (en) | Real-time regional audience rating and audience share statistical system based on intelligent television and method thereof | |
| CN103997662A (en) | Program pushing method and system | |
| CN109829098A (en) | Search result optimization method, device and server | |
| Jiang | Relationship between guaranteed rate server and latency rate server | |
| CN110335148A (en) | Securities data parallel processing system (PPS) and method | |
| CN105786941A (en) | Information mining method and device | |
| WO2010151194A1 (en) | Method and arrangement in a communication network | |
| CN107343111A (en) | A kind of cloud call center data management system | |
| CN116911525A (en) | Employee scheduling methods, devices, equipment, storage media and computer products | |
| Boon et al. | Heavy traffic analysis of roving server networks | |
| CN112818183B (en) | Data synthesis method, device, computer equipment and storage medium | |
| CN116955732A (en) | A multi-dimensional reasoning system function block matching recommendation method and system | |
| CN111368207B (en) | Data processing method and system, candidate data pool, electronic device and computer readable storage medium | |
| CN111506818B (en) | Flight data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |