CN112346951B - Service testing method and device - Google Patents
Service testing method and device Download PDFInfo
- Publication number
- CN112346951B CN112346951B CN201910721492.6A CN201910721492A CN112346951B CN 112346951 B CN112346951 B CN 112346951B CN 201910721492 A CN201910721492 A CN 201910721492A CN 112346951 B CN112346951 B CN 112346951B
- Authority
- CN
- China
- Prior art keywords
- target
- user groups
- service
- user
- click
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Prevention of errors by analysis, debugging or testing of software
- G06F11/3668—Testing of software
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a service testing method and device; the method comprises the following steps: determining a plurality of user groups of a target service layer in a plurality of service layers included in the corresponding service based on a preset number of homogeneous indexes; when the target service layer adopts parameter configuration of a preset version, respectively acquiring first click data corresponding to the target service layer of different user groups; performing similarity matching on first click data corresponding to the target business layer of different user groups; selecting at least two service index homogeneous target user groups from the plurality of user groups based on the similarity matching result; respectively obtaining the flow of the target business layer corresponding to each target user group; and executing different service tests on the target service layer based on the acquired flow, wherein the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested. The invention can improve the efficiency of service test.
Description
Technical Field
The present invention relates to the internet technology, and in particular, to a method and apparatus for testing services.
Background
With the development of internet technology, the experience of users is more focused in service development, multiple configuration parameters of the same service are usually tested, and the parameters which are most suitable for the user experience are determined from multiple alternatives.
In the related art, each time a service is tested, a target user group with similar attributes needs to be determined to ensure the accuracy of the service test, but the problems of long test time and low test efficiency are caused.
Disclosure of Invention
The embodiment of the invention provides a service testing method and device, which can improve service testing efficiency.
The technical scheme of the embodiment of the invention is realized as follows:
the embodiment of the invention provides a service testing method, which comprises the following steps:
Determining a plurality of user groups of a target service layer in a plurality of service layers included in the corresponding service based on a preset number of homogeneous indexes;
when the target service layer adopts parameter configuration of a preset version, respectively acquiring first click data corresponding to the target service layer of different user groups;
performing similarity matching on first click data corresponding to the target business layer of different user groups;
selecting at least two service index homogeneous target user groups from the plurality of user groups based on the similarity matching result;
respectively obtaining the flow of the target business layer corresponding to each target user group;
And executing different service tests on the target service layer based on the acquired flow, wherein the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested.
The embodiment of the invention provides a service testing device, which comprises:
a group determining unit, configured to determine, based on a preset number of homogeneous indexes, a plurality of user groups of a target service layer in a plurality of service layers included in a corresponding service;
the data acquisition unit is used for respectively acquiring first click data of different user groups corresponding to the target service layer when the target service layer adopts parameter configuration of a preset version;
The similarity matching unit is used for performing similarity matching on the first click data corresponding to the target business layer by different user groups;
A group selection unit, configured to select, from the plurality of user groups, a target user group with at least two homogeneous business indexes based on the result of the similarity matching;
The flow obtaining unit is used for obtaining the flow of the target business layer corresponding to each target user group respectively;
And the service test unit is used for executing different service tests on the target service layer based on the acquired flow, wherein the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested.
In the above scheme, the group determining unit is further configured to obtain a user portrait of a user corresponding to the target service layer;
and determining a plurality of user groups of which the user information similarity corresponding to the target service layer meets a similarity condition based on the acquired user portraits.
In the above scheme, the data obtaining unit is further configured to obtain click rates and/or click volumes of the services corresponding to different user groups respectively when the target service layer adopts a parameter configuration of a preset version.
In the above scheme, the group selection unit is further configured to determine at least two user groups whose similarities reach a similarity threshold value in the plurality of user groups;
and selecting at least two user groups with the similarity reaching a similarity threshold value from the plurality of user groups as the at least two target user groups.
In the above scheme, the similarity matching unit is further configured to determine a plurality of click rate intervals;
And acquiring click rate intervals in which the click rates of the target business data layers corresponding to different user groups are located.
In the above scheme, the group selection unit is further configured to determine at least two user groups with click rates in the same interval as a target user group with a homogeneous service index.
In the above scheme, the similarity matching unit is further configured to obtain differences between click rates of the services corresponding to different user groups;
and determining two user groups with the smallest difference value in different user groups.
In the above scheme, the service testing unit is further configured to allocate the traffic to the service configured by the configuration parameters of the different versions to be tested respectively when the target service layer is configured by the configuration parameters of the at least two different versions to be tested;
respectively acquiring second click data corresponding to the service of different target user groups according to the corresponding relation between the flow and the configuration parameters of the version to be tested;
and determining configuration parameters meeting test conditions in the configuration parameters of at least two different versions to be tested based on the second click data.
In the above scheme, the service testing unit is further configured to obtain click rates corresponding to the services of different target user groups respectively;
Comparing the click rate corresponding to different target user groups to determine the minimum click rate;
determining target user groups, of which the difference value between the click rates corresponding to different target user groups and the minimum value of the click rates reaches a preset threshold value;
And determining the configuration parameters corresponding to the target user groups of which the difference values reach the preset threshold value as the configuration parameters meeting the test conditions.
An embodiment of the present invention provides an electronic device, including:
a memory for storing executable instructions;
And the processor is used for realizing the test method for providing the service according to the embodiment of the invention when executing the executable instructions stored in the memory.
The embodiment of the invention provides a storage medium which stores executable instructions for realizing the service testing method provided by the embodiment of the invention when a processor is caused to execute.
The embodiment of the invention has the following beneficial effects:
When the target service layer adopts the parameter configuration of a preset version, respectively acquiring first click data of the target service layer corresponding to different user groups; performing similarity matching on first click data corresponding to the target business layer of different user groups; based on the result of similarity matching, selecting at least two target user groups with same service indexes from a plurality of user groups, so that different service tests can be executed on a target service layer according to the acquired target user groups, one target user group is not required to be determined for each service test, and the service test efficiency is improved.
Drawings
FIG. 1 is a test system for services provided by an embodiment of the present invention;
Fig. 2 is a schematic hardware structure of a server according to an embodiment of the present invention;
Fig. 3 is a flow chart of a method for testing services according to an embodiment of the present invention;
fig. 4 is a flow chart of a method for testing services provided by an embodiment of the present invention;
fig. 5 is a flow chart of a method for testing services provided by an embodiment of the present invention;
FIG. 6 is a schematic diagram of flow distribution provided by an embodiment of the present invention;
Fig. 7 is a schematic diagram of a composition structure of a service testing device according to an embodiment of the present invention.
Detailed Description
The present invention will be further described in detail with reference to the accompanying drawings, for the purpose of making the objects, technical solutions and advantages of the present invention more apparent, and the described embodiments should not be construed as limiting the present invention, and all other embodiments obtained by those skilled in the art without making any inventive effort are within the scope of the present invention.
It should be noted that, in the embodiment of the present application, related data (such as user portrait, etc.) of user information and the like is related, when the embodiment of the present application is applied to a specific product or technology, user permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with related laws and regulations and standards of related countries and regions.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.
In the following description, the terms "first", "second", etc. are used merely to distinguish between similar objects and do not represent a particular ordering of the objects, it being understood that the "first", "second", etc. may be interchanged with a particular order or sequence, as permitted, to enable embodiments of the invention described herein to be practiced otherwise than as illustrated or described herein.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to be limiting of the invention.
Before describing embodiments of the present invention in further detail, the terms and terminology involved in the embodiments of the present invention will be described, and the terms and terminology involved in the embodiments of the present invention will be used in the following explanation.
1) The user portraits comprise user interest portraits and user base portraits; wherein,
User interest portraits, which refer to virtual representations of real users, are target user models built on a series of attribute data for indicating interest categories of users;
the user basic portrait is a tagged user information overall view abstracted by user basic information such as the true gender, age, resident login place and the like of the user.
2) Click rate, which is the ratio of the number of times a certain content on a website page is clicked to the number of times the content is displayed;
3) Click quantity refers to the number of times that a certain content on a website page is clicked within a certain period of time on the website page;
4) The buckets are realized by carrying out hash calculation on the appointed columns, dividing the data under one column name into a group of buckets through hash values, and enabling each bucket to correspond to one storage file under the column name.
Referring to fig. 1, fig. 1 is a schematic diagram of an alternative architecture of a service testing system 100 according to an embodiment of the present invention, in order to support an exemplary application, terminals (terminals 400-1, 400-2, 400-3, and 400-4 are shown as examples) are connected to a server 200 through a network 300, where the network 300 may be a wide area network or a local area network, or a combination of both.
The server 200 determines a plurality of user groups of a target service layer among a plurality of service layers included in the corresponding service based on a preset number of homogeneous indexes;
the terminal is used for receiving a first operation of a user for the service when the service adopts target parameter configuration, and sending first click data corresponding to the first operation to the server;
The server 200 is configured to obtain first click data of target service layers corresponding to different user groups respectively; performing similarity matching on first click data of target business layers corresponding to different user groups; selecting at least two target user groups with same service indexes from a plurality of user groups based on a similarity matching result; respectively obtaining the flow of the target business layer corresponding to each target user group; based on the acquired flow, different service tests are executed on the target service layer, wherein the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested.
The following describes an electronic device implementing a test method for implementing a service according to an embodiment of the present invention. In some embodiments, the electronic device may be implemented as various types of terminals such as a smart phone, a tablet computer, a notebook computer, and the like, and may also be a server. In the embodiment of the invention, the electronic equipment is taken as a server as an example, and the hardware structure of the server is described in detail.
Fig. 2 is a schematic hardware structure of a server according to an embodiment of the present invention, and it may be understood that fig. 2 only illustrates an exemplary structure of the server, but not all the structures, and some or all of the structures illustrated in fig. 2 may be implemented as required. Referring to fig. 2, a server provided in an embodiment of the present invention includes: at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The various components in the server are coupled together by a bus system 205. It is understood that the bus system 205 is used to enable connected communications between these components. The bus system 205 includes a power bus, a control bus, and a status signal bus in addition to the data bus. But for clarity of illustration, the various buses are labeled in figure 2 as bus system 205.
The user interface 203 may include, among other things, a display, keyboard, mouse, trackball, click wheel, keys, buttons, touch pad, or touch screen, etc.
It will be appreciated that the memory 202 may be either volatile memory or nonvolatile memory, and may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (RO M), a programmable Read Only Memory (PROM, programmable Read-Only Memory), an erasable programmable Read Only Memory (EPROM, erasable Programmable Read-Only Memory), a Flash Memory (Flash Memory), or the like. The volatile memory may be random access memory (RAM, random Access Memory) which acts as external cache memory. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRA M, static Random Access Memory), synchronous static random access memory (SSRAM, sy nchronous Static Random Access Memory). The memory 402 described in embodiments of the present invention is intended to comprise these and any other suitable types of memory.
The memory 202 in embodiments of the present invention is used to store various types of data to support the operation of the server 200. Examples of such data include: any executable instructions, such as executable instructions, for operation on server 200, a program implementing the testing method of the business of embodiments of the present invention may be included in the executable instructions.
As an example of implementation of the service testing apparatus provided by the embodiment of the present invention by combining software and hardware, the service testing apparatus provided by the embodiment of the present invention may be directly embodied as a combination of software modules executed by the processor 201, the software modules may be located in a storage medium, the storage medium is located in the memory 202, and the processor 201 reads executable instructions included in the software modules in the memory 202, and performs the service testing method provided by the embodiment of the present invention in combination with necessary hardware (including, for example, the processor 201 and other components connected to the bus 205).
By way of example, the Processor 201 may be an integrated circuit chip having signal processing capabilities such as a general purpose Processor, such as a microprocessor or any conventional Processor, a digital signal Processor (DSP, digital Signal Processor), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like.
As an example of a hardware implementation of the service testing device provided by the embodiment of the present invention, the device provided by the embodiment of the present invention may be directly implemented by the processor 201 in the form of a hardware decoding processor, for example, one or more Application specific integrated circuits (ASICs, applications SPECIFIC INTEGRATE D circuits), DSPs, programmable logic devices (PLDs, programmable Logic Device), complex Programmable logic devices (CPLDs, complex Programmable Logic Device), field-Programmable gate arrays (FPGAs), field-Programmable GATE ARRAY) or other electronic components may be used to implement the test method for implementing the service provided by the embodiment of the present invention.
The method for testing services disclosed in the embodiment of the present invention may be implemented by the processor 201. The processor 201 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the method for testing a service may be performed by instructions in the form of integrated logic circuits of hardware or software in the processor 201. The Processor 201 may be a general purpose Processor, a digital signal Processor (DSP, digit al Signal Processor), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor 201 may implement or perform the methods, steps and logic blocks disclosed in embodiments of the present invention. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the service testing method disclosed by the embodiment of the invention can be directly embodied as the execution completion of the hardware decoding processor or the execution completion of the combination execution of the hardware and software modules in the decoding processor. The software module may be located in a storage medium, where the storage medium is located in the memory 202, and the processor 201 reads information in the memory 202, and in combination with its hardware, performs the steps of the method for testing services provided by the embodiment of the present invention.
Based on the above description of the service testing system and the electronic device according to the embodiments of the present invention, the following description will explain the service testing method provided by the embodiments of the present invention, and fig. 3 is a flow chart of the service testing method provided by the embodiments of the present invention, in some embodiments, the service testing method may be implemented by a server, for example, by the server 200 in fig. 1, and in combination with fig. 1 and 3, the service testing method provided by the embodiments of the present invention includes:
Step 301: the server determines a plurality of user groups of a target service layer in a plurality of service layers included in the corresponding service based on a preset number of homogeneous indexes.
Here, the service includes an algorithm class, an operation class, and a user interface class. The algorithm is the place with the most application of business test, and whether a new algorithm or small optimization can improve the business index of the algorithm can be verified through business test, such as recommendation, searching, accurate advertisement and products of the algorithm related to accurate operation. Various types of operation means, such as user operation (user pull, member operation, etc.), content operation (programming of video industry, etc.), activity operation (lottery, etc.), etc., can use the service test to verify which operation policy is more effective. The user interface is any part of the internet product which can be directly perceived by a user, the user interacts with the internet product through the user interface, and the perception of the user on a product is firstly established through the user; a compact and attractive user interface, and smooth user interaction can often give a good first impression to the user. For the optimization of the visual and interactive parts of the user, the experience of the designer is often insufficient, and technical means are needed to verify which user interface display style and which interactive mode is preferred by the user and can bring the maximum benefit.
It should be noted that the homogeneity index may be user information such as age, gender, and academic of the user, the user may configure one or more homogeneity indexes according to the need, and the server determines, according to the homogeneity index preset by the user, a plurality of user groups corresponding to a target service layer in a plurality of service layers included in the service. Here, each user group includes a plurality of users, wherein the plurality represents two or more.
In practical implementation, a user may configure the service layer according to different modules of the service, for example, the search service includes three modules, namely, a user display module, a result ordering module and an advertisement insertion module, so that the three service layers may be configured to respectively correspond to the three modules, and the server may simultaneously obtain the first click data of the different service layers, where the different service layers do not interfere with each other. When the preset homogeneity indexes of the user are age and gender and the target service layer is the result ordering layer, the server can group the users according to the age and gender corresponding to the user corresponding to the result ordering layer, so as to obtain a plurality of user groups corresponding to the result ordering layer.
In some embodiments, the server may determine the plurality of user groups corresponding to a target service layer of the plurality of service layers included in the service by: acquiring user portraits of users corresponding to the service; based on the obtained user portraits, a plurality of user groups for which the similarity of the user information of the corresponding service meets a similarity condition are determined.
Here, the user representation may include at least one of: user interest portraits and user base portraits. In some embodiments, the server may obtain user portraits of all users corresponding to the service, screen out a part of users from all users according to the obtained user portraits, and divide the screened users into a plurality of user groups, so that the similarity of user information corresponding to any two user groups satisfies a similarity condition; in other embodiments, the server may divide the plurality of users corresponding to the service into a plurality of user groups, and then obtain a user representation of the user corresponding to the service, and determine, from the plurality of user groups, a plurality of user groups whose user information similarity of the service satisfies a similarity condition according to the obtained user representation.
Here, the fact that the similarity of the user information of the service meets the similarity condition indicates that the components of the users in the user groups are identical, for example, the server obtains the ages and the sexes of the users corresponding to the service, determines a plurality of user groups according to the ages and the sexes of the users, wherein the number ratio of male users to female users in each user group is 1:1, and the ages are between 20 and 40.
In some embodiments, the server may determine the plurality of user groups for the corresponding traffic by: distributing the flow of the corresponding service to a plurality of sub-buckets; determining users corresponding to the flows in each sub-bucket according to the information carried in the access requests corresponding to the flows in each sub-bucket; and determining a plurality of user groups corresponding to the service according to the user portraits of the users corresponding to the flows in each sub-bucket, wherein the user groups comprise at least one user corresponding to the flows in the sub-bucket.
It should be noted that the traffic may be the number of individual visitors (UV, unique Visitor) or the web browsing volume (PV, page View). The flow rate corresponds to the number of access requests, for example, the flow rate is 10000, and the corresponding number of access requests is 10000.
In practical implementation, the flow is distributed to different sub-barrels in a random flow mode, and the access flow in each sub-barrel is the same, and because the information carried by the access request comprises the user identification, the server can determine the users corresponding to the flow in each sub-barrel based on the user identification, acquire the user images of the users, and perform similarity matching on the user information corresponding to the flow in different sub-barrels to determine at least two user groups.
According to the embodiment of the invention, the user information corresponding to the flows in different sub-barrels is subjected to similarity matching, and the matching is the average value of the user information corresponding to each sub-barrel, so that the similarity matching of each user information is not needed, and the matching efficiency is improved.
In some embodiments, the number of users in each user group may be configured, and the barrel number corresponding to the user group is selected according to the configured number. For example, the flow in each sub-bucket is 1000, that is, the number of users corresponding to the flow in each sub-bucket is 1000, and the number of users in a user group is configured to be 2000, and then the user group includes users corresponding to the flows in two sub-buckets. Here, the number of users in each user group may be different, for example, the first user group may include users corresponding to the traffic in the sub-bucket a and the sub-bucket B, and the second user group may include users corresponding to the traffic in the sub-bucket C.
Step 302: and when the target service layer adopts the parameter configuration of a preset version, respectively acquiring first click data of the target service layer corresponding to different user groups.
Here, the parameter may be a flow algorithm for the target service layer, or may be an interface display parameter for the service. When the target service layer is a result ordering layer of the search service, a user presets an ordering algorithm of result ordering, after different user groups send search requests, the server orders the search results corresponding to the search requests according to the ordering algorithm of the preset result ordering, a search result page is generated, and click data of the different user groups on the search result page for the search results is obtained.
In practical implementation, the first click data may include click data of a service corresponding to each user in the user group, or may be an average value of click data of a plurality of services corresponding to users in the user group.
In some embodiments, the server may allocate the users in the user group to a plurality of sub-buckets, and obtain an average value of the first click data of all the users in each sub-bucket, so that accuracy of subsequent determination of the target user group can be ensured, and the calculation amount is reduced.
In some embodiments, the first click data may be a click rate and/or a click volume, and when the service adopts the target parameter configuration, the click rates and/or the click volumes of the services corresponding to different user groups are respectively obtained.
Step 303: and performing similarity matching on the first click data of the target business layers corresponding to different user groups.
In some embodiments, the server may perform similarity matching by: determining a plurality of click rate (click volume) intervals; and acquiring click rate intervals in which the click rates of the target business data layers corresponding to different user groups are located.
For example, five click rate intervals are preset, which are respectively less than 3%, 3% -8%, 8% -13%, 13% -18%, and greater than 18%, and the user groups are divided according to the click rate intervals.
In some embodiments, when the number of target user groups is two, the server may perform similarity matching by: acquiring the difference value between click rates of the corresponding services of different user groups; two user groups with the smallest difference value among different user groups are determined.
For example, the click rates corresponding to the user groups A, B, C, 7, and 17, respectively, where the difference between the click rates corresponding to the user group a and the user group B is at least 2%, and the user group a and the user group B are two user groups with the smallest difference.
Step 304: and selecting at least two target user groups with same service indexes from the plurality of user groups based on the similarity matching result.
In some embodiments, when the server obtains a click rate interval in which click rates of different user groups corresponding to the target service data layer are located, at least two user groups with click rates in the same interval may be determined as target user groups with homogeneous service indexes.
In some embodiments, when the server acquires two user groups with the smallest difference value from the different user groups, the two acquired user groups with the smallest difference value are used as the target user groups with homogeneous service indexes.
In some embodiments, the server may further determine at least two user groups of the plurality of user groups for which the similarity reaches a similarity threshold; and selecting at least two user groups with the similarity reaching a similarity threshold value from the plurality of user groups as the at least two target user groups.
In some embodiments, the server may determine that the similarity of at least two user groups reaches a similarity threshold by hypothesis testing.
In practical implementation, the assumption that the similarity of at least two user groups reaches the similarity threshold is set to zero, and the assumption that the similarity of at least two user groups does not reach the similarity threshold is set to alternative assumption is not satisfied if p-value < alpha/n, and the assumption that zero is characterized is not satisfied otherwise. Here, p-value is the probability that the original hypothesis is established calculated through a statistical method, the smaller the p-value is, the greater the probability that the original hypothesis is refused is, the alpha characterizes the detection level, n is the number of parameters included in the first click data, and if the first click data is the click rate and the click quantity, n is 2. In actual implementation, alpha may be set to 0.05.
In some embodiments, when the parameters of the first click data are multiple, the p-value may be corrected by an error rate judgment Family (FWER) or false positive rate (FDR, false discovery rate) algorithm.
Step 305: and respectively acquiring the flow of each target user group corresponding to the target service layer.
It should be noted that one user may trigger one flow, or may trigger multiple flows. For example, user a may have swiped a microblog only once a day, and may be noted as user a triggered a flow in the microblog during the day. For another example, if user a swipes n microblogs in one day, it can be noted that user a triggers n flows.
Step 306: and executing different service tests on the target service layer based on the acquired flow, wherein the service tests are used for selecting the configuration parameters of the target version from the configuration parameters of at least two different versions to be tested.
Here, the server can execute different service tests based on the acquired traffic, and does not need to determine a target user group for each service test, so that the time of the service test can be saved, and the efficiency of the service test can be improved.
In some embodiments, the server may perform traffic testing on the target traffic layer by: when the target service layer adopts at least two different versions to be tested to configure the configuration parameters, respectively distributing the flow to the services configured by the configuration parameters of the different versions to be tested; respectively acquiring second click data of the corresponding services of different target user groups according to the corresponding relation between the flow and the configuration parameters of the version to be tested; and determining configuration parameters meeting the test conditions from the configuration parameters of at least two different versions to be tested based on the second click data.
Taking a user interface test of a page as an example, the page A adopts configuration parameter configuration of a first version to be tested, a navigation bar is red, the page B adopts configuration parameter configuration in a second version to be tested, the navigation bar is black, and flow is respectively distributed to the pages configured by the configuration parameters of different versions to be tested, so that when a target user group A accesses the page, the page A is displayed, and when a target user group B accesses the page, the page B is displayed, and click data corresponding to different pages of the target user group A, B are respectively acquired. And when the click data of the target user group A is significantly larger than the click data of the target user group B, determining that the page A is a page preferred by the user.
In some embodiments, the server obtains click rates (click volumes) of the services corresponding to different target user groups respectively; comparing the click rate (click volume) corresponding to different target user groups to determine the minimum value of the click rate (click volume); determining target user groups, of which the difference value between the click rate (click quantity) corresponding to different target user groups and the minimum value of the click rate (click quantity) reaches a preset threshold; and determining the configuration parameters corresponding to the target user groups of which the difference values reach the preset threshold values as the configuration parameters meeting the test conditions.
For example, the click rates corresponding to the target user group A, B, C are 2%, 3%, and 10%, respectively, the minimum value of the click rates corresponding to the target user group A, B, C is 2%, and if the preset threshold is 5%, then the configuration parameters corresponding to the target user group are determined to be the configuration parameters that meet the test conditions.
It should be noted that, if the difference between the click rate corresponding to a certain target user group and the minimum value of the click rates corresponding to the target user group is greater than a preset threshold, it is indicated that the click rate corresponding to the target user group is far greater than the click rates corresponding to other user groups, and it is indicated that the configuration parameters adopted by the service corresponding to the user group are better than other configuration parameters, and it is determined that the configuration parameters are configuration parameters meeting the test conditions.
In some embodiments, configuration parameters meeting test conditions can be determined by a hypothesis test mode, in actual implementation, two groups of configuration parameters are parameters corresponding to an original scheme and parameters corresponding to a new scheme respectively, the click rate and/or click rate of the original hypothesis as the new scheme are set to be higher than those of the original scheme, p-valu e is calculated by a statistical method, and if p-value < alpha, the characteristic zero hypothesis is not met, the configuration parameters meeting the test conditions are parameters corresponding to the original scheme; otherwise, the zero assumption is established, and the configuration parameters meeting the test conditions are parameters corresponding to the new scheme. In actual implementation, alpha may be set to 0.05.
According to the embodiment of the invention, when the target service layer adopts the parameter configuration of the preset version, the first click data of the target service layer corresponding to different user groups are respectively obtained; performing similarity matching on first click data corresponding to the target business layer of different user groups; based on the result of similarity matching, selecting at least two target user groups with same service indexes from a plurality of user groups, so that different service tests can be executed on a target service layer according to the acquired target user groups, one target user group is not required to be determined for each service test, and the service test efficiency is improved.
Next, taking a search ordering layer in a search service as an example, a test method of the service provided by the embodiment of the invention is described. Fig. 4 is a flow chart of a method for testing services provided by an embodiment of the present invention, as shown in fig. 4,
Step 401: the server distributes the traffic corresponding to the search ranking layer to 1000 sub-buckets.
Step 402: and determining the users corresponding to the flows in each sub-bucket according to the information carried in the access requests corresponding to the flows in each sub-bucket.
Step 403: and determining a plurality of user groups of which the similarity of the user's homogeneity indexes reaches a similarity threshold according to the homogeneity indexes of the users corresponding to the flows in each sub-bucket.
Here, each user group includes at least one user corresponding to the traffic in the sub-bucket.
Step 404: and acquiring a first click rate of each user group aiming at the target page.
Here, the target page displays the search results in a target ranking manner.
Step 405: and performing similarity matching on click rates of the target pages corresponding to different user groups to determine two user groups with similarity reaching a similarity threshold value in the plurality of user groups.
Step 406: and taking the two user groups with the similarity reaching the similarity threshold as target user groups.
Step 407: and acquiring a second click rate of the first target user group for the page corresponding to the original ordering mode and a third click rate of the second target user group for the page corresponding to the new ordering mode.
Step 408: and when the second click rate is increased by more than thirty percent relative to the third click rate, if so, determining that the new sorting mode is a mode meeting the target sorting mode.
Here, if the second click rate is raised by more than thirty percent with respect to the third click rate, it means that the new ranking mode is significantly better than the original ranking mode.
The method for testing the service provided by the embodiment of the invention is continuously described. Fig. 5 is a flow chart of a method for testing services provided by an embodiment of the present invention, referring to fig. 5, in actual implementation, the method for testing services provided by the embodiment of the present invention includes the following steps.
First, the traffic of the corresponding traffic layer is divided into a plurality of buckets, for example, the traffic of the traffic layer is divided into 10000 buckets according to a random traffic mode, and the traffic of each bucket is one ten thousandth.
Here, the traffic may be UV or PV, and the user creates corresponding service layers according to service requirements and allocates traffic to each layer. For example, fig. 6 is a schematic flow distribution diagram provided in the embodiment of the present invention, as shown in fig. 6, a search service includes three modules, a user shows, a result orders, and an advertisement inserts, and three service layers can be created to respectively correspond to the three modules.
It should be noted that, traffic in the same layer can only hit one service test, and cannot hit two or more service tests at the same time. Traffic in different layers may hit different traffic tests, but traffic is orthogonal. That is, the traffic hitting the service test in the user presentation layer-B is uniformly distributed in the result sorting layer-C, so that the service tests in the two layers can be ensured not to interfere with each other.
In some embodiments, the user may create a cross-layer test from the service, for example, as shown in fig. 6, the user has created a user presentation layer-B and a result ordering layer-C, the user wants to test configuration parameters of the user presentation layer-B and the result ordering layer-C at the same time, and since the traffic of each layer is orthogonal, the user needs to create a merging layer-E, and test configuration parameters of the user presentation layer-B and the result ordering layer-C at the same time by the merging layer-E. Here, different configuration parameters correspond to different interfaces or flows.
And then, carrying out pre-AA stage experiments through the idle module, and acquiring basic information of users corresponding to the flows in each sub-bucket, and when the service layer adopts parameter configuration of a preset version, acquiring first operation information of the users corresponding to the flows in each sub-bucket.
It should be noted that the idle module records the usage of the flow, including the allocated flow and the unallocated flow. Here, the idle module obtains user information corresponding to the flow in the idle sub-bucket and first operation information, where the user information refers to age, gender, academic of the user, and the first operation information may be click rate, and the like. And storing the acquired user information and the first operation information into an idling database. It should be noted that, the users corresponding to each sub-bucket are allocated to 50 sub-buckets, so the storage format of the idle database is 51 columns of data: the sub-bucket ID and the user information and the first operation information of the users corresponding to the 50 sub-buckets.
And then, determining two user groups, wherein each user group comprises at least one user corresponding to the barrel flow, acquiring user information and first operation information corresponding to each user group from a database, and judging whether the user components of the two user groups are identical or not through hypothesis testing according to the acquired user information and first operation information. In actual implementation, the number of users corresponding to the user group may be configured by the user.
For example, the first user group includes users corresponding to {12, 1019} two sub-buckets, the second user group includes users corresponding to {3330, 8912} two sub-buckets, the sub-bucket 12 is combined with the user information and the first operation information of the sub-bucket 1019, the sub-bucket 3330 is combined with the user information and the first operation information of the sub-bucket 8912, and then the hypothesis test is performed.
Here, assuming that the check level alpha=0.05 of the check, if the number of p-value < alpha/index indicates that the user constituent components of the two user groups are not identical, it is necessary to redefine the two user groups; otherwise, the user components of the two user groups are the same, so that the two user groups respectively correspond to the two different configuration parameters. Here, the index includes first operation information and basic information of the user. If there are multiple indicators, then either FWER or FDR can be used to correct the p-value.
And finally, in the AB stage of the operation experiment, second operation information of the two user groups relative to different configuration parameters is respectively acquired, and whether the second operation information corresponding to the two user groups is obviously different is judged, so that which of the two groups of configuration parameters is better is obtained. And updating the unallocated traffic of the service layer and returning to the idle module.
Here, the second operation information may be a click rate, a click quantity, or the like. In practical implementation, it may also be assumed that the second operation information corresponding to the two user groups is significantly different.
It should be noted that, in the same layer, testing of multiple configuration parameters may be performed simultaneously; in a test system, testing of configuration parameters may be performed in multiple layers simultaneously.
Next, description will be given of a service testing device provided in an embodiment of the present invention, in some embodiments, the service testing device may be implemented by using a software module, fig. 7 is a schematic structural diagram of the service testing device provided in the embodiment of the present invention, and referring to fig. 7, the service testing device includes:
a group determining unit 701, configured to determine, based on a preset number of homogeneous indexes, a plurality of user groups of a target service layer in a plurality of service layers included in a corresponding service;
a data obtaining unit 702, configured to obtain first click data corresponding to the target service layer by different user groups when the target service layer adopts a parameter configuration of a preset version;
A similarity matching unit 703, configured to perform similarity matching on first click data corresponding to the target service layer for different user groups;
a group selection unit 704, configured to select, from the plurality of user groups, a target user group with at least two homogeneous business indexes based on the result of the similarity matching;
A flow obtaining unit 705, configured to obtain, respectively, a flow of the target service layer corresponding to each of the target user groups;
And a service testing unit 706, configured to execute different service tests on the target service layer based on the acquired traffic, where the service tests are used to select configuration parameters of a target version from configuration parameters of at least two different versions to be tested.
The embodiment of the invention also provides electronic equipment, which comprises:
A memory for storing an executable program;
And the processor is used for realizing the service testing method provided by the embodiment of the invention when executing the executable program stored in the memory.
The embodiment of the invention also provides a storage medium storing executable instructions, wherein the executable instructions are stored, and when the executable instructions are executed by a processor, the processor is caused to execute the test method of the service provided by the embodiment of the invention.
All or part of the steps of the embodiments may be performed by hardware associated with program instructions, and the foregoing program may be stored in a computer readable storage medium, which when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: a mobile storage device, a random access Memory (RAM, random Access Memory), a Read-Only Memory (ROM), a magnetic disk or an optical disk, or the like, which can store program codes.
Or the above-described integrated units of the invention may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solution of the embodiments of the present invention may be embodied essentially or in a part contributing to the related art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program code, such as a removable storage device, RAM, ROM, magnetic or optical disk.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for testing a service, the method comprising:
distributing the flow of the corresponding service to a plurality of sub-barrels, and determining the users corresponding to the flow in each sub-barrel according to the information carried in the access request corresponding to the flow in each sub-barrel;
Acquiring user portraits of users corresponding to a target service layer in the users corresponding to the flow in each sub-bucket, wherein the target service layer belongs to a plurality of service layers included in the service; based on the obtained user portraits, determining a plurality of user groups of which the user information similarity corresponding to the target service layer in each sub-bucket meets a similarity condition, wherein the user groups comprise users corresponding to the flow in at least one sub-bucket;
when the target service layer adopts parameter configuration of a preset version, respectively acquiring first click data corresponding to the target service layer of different user groups;
performing similarity matching on first click data corresponding to the target business layer of different user groups;
selecting at least two service index homogeneous target user groups from the plurality of user groups based on the similarity matching result;
respectively obtaining the flow of the target business layer corresponding to each target user group;
when the target service layer adopts at least two different versions to be tested to configure the configuration parameters, respectively distributing the flow to the services configured by the configuration parameters of the different versions to be tested;
respectively acquiring second click data corresponding to the service of different target user groups according to the corresponding relation between the flow and the configuration parameters of the version to be tested;
And determining the configuration parameters of the target version meeting the test conditions in the configuration parameters of at least two different versions to be tested based on the second click data.
2. The method of claim 1, wherein when the target service layer adopts the parameter configuration of the preset version, the step of obtaining the first click data corresponding to the target service layer from different user groups respectively includes:
and when the target service layer adopts parameter configuration of a preset version, respectively acquiring click rates and/or click volumes of the services corresponding to different user groups.
3. The method of claim 1, wherein the selecting at least two target user groups of homogeneous traffic metrics from the plurality of user groups based on the results of the similarity matching comprises:
determining at least two user groups of which the similarity reaches a similarity threshold value in the plurality of user groups;
And selecting at least two user groups, of which the similarity reaches a similarity threshold, from the plurality of user groups as the at least two target user groups.
4. The method of claim 1, wherein the similarity matching of the first click data corresponding to the target business layer for different user groups comprises:
determining a plurality of click rate intervals;
and acquiring click rate intervals in which the click rates of the target business layers corresponding to different user groups are located.
5. The method of claim 4, wherein the selecting at least two target user groups of homogeneous traffic metrics from the plurality of user groups based on the results of the similarity matching comprises:
And determining at least two user groups with click rate in the same interval as a target user group with homogeneous service index.
6. The method of claim 1, wherein the similarity matching of the first click data corresponding to the target business layer for different user groups comprises:
Obtaining the difference value between click rates of the business corresponding to different user groups;
and determining two user groups with the smallest difference value in different user groups.
7. The method of claim 1, wherein determining, based on the second click data, a configuration parameter that satisfies a test condition among configuration parameters of at least two different versions to be tested, comprises:
respectively obtaining click rates of the businesses corresponding to different target user groups;
Comparing the click rate corresponding to different target user groups to determine the minimum click rate;
determining target user groups, of which the difference value between the click rates corresponding to different target user groups and the minimum value of the click rates reaches a preset threshold value;
And determining the configuration parameters corresponding to the target user groups of which the difference values reach the preset threshold value as the configuration parameters meeting the test conditions.
8. A device for testing a service, the device comprising:
the group determining unit is used for distributing the flow of the corresponding service to a plurality of sub-barrels and determining the users corresponding to the flow in each sub-barrel according to the information carried in the access request corresponding to the flow in each sub-barrel; acquiring user portraits of users corresponding to a target service layer in the users corresponding to the flow in each sub-bucket, wherein the target service layer belongs to a plurality of service layers included in the service; based on the obtained user portraits, determining a plurality of user groups of which the user information similarity corresponding to the target service layer in each sub-bucket meets a similarity condition, wherein the user groups comprise users corresponding to the flow in at least one sub-bucket;
the data acquisition unit is used for respectively acquiring first click data of different user groups corresponding to the target service layer when the target service layer adopts parameter configuration of a preset version;
The similarity matching unit is used for performing similarity matching on the first click data corresponding to the target business layer by different user groups;
A group selection unit, configured to select, from the plurality of user groups, a target user group with at least two homogeneous business indexes based on the result of the similarity matching;
The flow obtaining unit is used for obtaining the flow of the target business layer corresponding to each target user group respectively;
the service testing unit is used for respectively distributing the flow to the service configured by the configuration parameters of different versions to be tested when the target service layer is configured by the configuration parameters of at least two versions to be tested; respectively acquiring second click data corresponding to the service of different target user groups according to the corresponding relation between the flow and the configuration parameters of the version to be tested; and determining the configuration parameters of the target version meeting the test conditions in the configuration parameters of at least two different versions to be tested based on the second click data.
9. An electronic device, comprising:
a memory for storing executable instructions;
A processor for implementing the method for testing a service according to any one of claims 1 to 7 when executing executable instructions stored in said memory.
10. A storage medium storing executable instructions for causing a processor to perform the method of testing a service of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910721492.6A CN112346951B (en) | 2019-08-06 | 2019-08-06 | Service testing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910721492.6A CN112346951B (en) | 2019-08-06 | 2019-08-06 | Service testing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112346951A CN112346951A (en) | 2021-02-09 |
| CN112346951B true CN112346951B (en) | 2024-08-27 |
Family
ID=74366489
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910721492.6A Active CN112346951B (en) | 2019-08-06 | 2019-08-06 | Service testing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112346951B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685377B (en) * | 2021-03-11 | 2021-06-25 | 腾讯科技（深圳）有限公司 | Detection processing method, device, server and medium based on historical object data |
| CN112905476B (en) * | 2021-03-12 | 2023-08-11 | 网易（杭州）网络有限公司 | Test execution method and device, electronic equipment and storage medium |
| CN115222435B (en) * | 2021-04-19 | 2025-10-03 | 腾讯科技（深圳）有限公司 | A business scenario comparison method, device, terminal and storage medium |
| CN113657930B (en) * | 2021-08-12 | 2024-05-28 | 广州虎牙科技有限公司 | Method and device for testing policy effectiveness, electronic equipment and readable storage medium |
| CN114490379B (en) * | 2022-01-25 | 2025-04-11 | 中国工商银行股份有限公司 | Testing method and device for approval workflow |
| CN115391184A (en) * | 2022-08-09 | 2022-11-25 | 北京百度网讯科技有限公司 | A method, device and equipment for AB testing |
| CN119359372A (en) * | 2024-09-20 | 2025-01-24 | 科大讯飞股份有限公司 | An advertising experiment method, device, equipment, storage medium, product and system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9134961B1 (en) * | 2011-05-08 | 2015-09-15 | Panaya Ltd. | Selecting a test based on connections between clusters of configuration changes and clusters of test scenario runs |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8214346B2 (en) * | 2008-06-27 | 2012-07-03 | Cbs Interactive Inc. | Personalization engine for classifying unstructured documents |
| US9348735B1 (en) * | 2011-05-08 | 2016-05-24 | Panaya Ltd. | Selecting transactions based on similarity of profiles of users belonging to different organizations |
| CN107545444B (en) * | 2016-06-23 | 2021-03-16 | 腾讯科技（深圳）有限公司 | Business advertisement data recommendation method and device |
| US10579509B2 (en) * | 2018-01-21 | 2020-03-03 | Microsoft Technology Licensing, Llc. | Machine learning comparison tools |
| CN109151576A (en) * | 2018-06-20 | 2019-01-04 | 新华网股份有限公司 | Multimedia information clipping method and system |
-
2019
- 2019-08-06 CN CN201910721492.6A patent/CN112346951B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9134961B1 (en) * | 2011-05-08 | 2015-09-15 | Panaya Ltd. | Selecting a test based on connections between clusters of configuration changes and clusters of test scenario runs |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112346951A (en) | 2021-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112346951B (en) | Service testing method and device | |
| CA3109481A1 (en) | Identification and application of hyperparameters for machine learning | |
| CN111242709B (en) | A message push method and its device, equipment and storage medium | |
| CN111159563B (en) | Method, device, equipment and storage medium for determining user interest point information | |
| CN109558384B (en) | Log classification method, device, electronic equipment and storage medium | |
| CN110704677B (en) | Program recommendation method and device, readable storage medium and terminal equipment | |
| US20180329926A1 (en) | Image-based semantic accommodation search | |
| CN105989066A (en) | Information processing method and device | |
| CN113065071B (en) | Product information recommendation method and computer equipment | |
| CN109828902B (en) | Interface parameter determining method and device, electronic equipment and storage medium | |
| CN112182107B (en) | List data acquisition method, device, computer equipment and storage medium | |
| CN111414410A (en) | Data processing method, device, equipment and storage medium | |
| CN112559866B (en) | University book reading recommendation method, device, equipment and storage medium | |
| JP6204923B2 (en) | Assessment device, assessment system, assessment method, and program | |
| CN110909267B (en) | Method and device for displaying entity object side, electronic equipment and storage medium | |
| CN109101630B (en) | Method, device and equipment for generating search result of application program | |
| CN110674386B (en) | Resource recommendation method, device and storage medium | |
| CN116932891A (en) | Resource object display method, device, equipment, storage medium and product | |
| CN110223108B (en) | Click through rate prediction method, device and equipment | |
| CN107688948A (en) | Claims Resolution data processing method, device, computer equipment and storage medium | |
| CN109885504B (en) | Recommendation system test method, device, medium and electronic equipment | |
| CN114547116A (en) | Data pushing method, device, equipment and medium | |
| CN114168599A (en) | Data summarizing method and device and terminal equipment | |
| CN114663200B (en) | Product recommendation method and device, electronic equipment and storage medium | |
| CN114519114A (en) | Multimedia resource classification model construction method and device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |