HK1098595B - System and method for grading individuals' performances - Google Patents

Description

System and method for rating an individual's performance

Technical Field

The present invention relates to a system and method for rating the performance of an individual.

Background

Generally, quantizers are well known. The quantizer receives an input signal and generates an output signal indicating which of a set of amplitude ranges the input signal falls within. One typical application of a simple quantizer is an analog-to-digital converter, in which: the size of the range, or "quantization step", is set by the accuracy of the output digital signal and the acceptable input signal range. However, the quantization steps need not all be the same size. For example, an analog-to-digital converter may be used as a speech compressor by making the quantization step for large input signals smaller than for small input signals.

The inventors of the present invention have appreciated a need for a quantizer that can adapt the size of its quantization steps based on a statistical model of the desired output signal. This can compensate for unwanted system gain variations and non-linearities.

Disclosure of Invention

According to the present invention, there is provided an adaptive quantizer comprising:

inputting;

means for storing a distribution representation of the quantizer output with respect to a desired input signal;

means for recording the actual input signal over a statistically significant period of time;

processing means for setting a quantization step size in dependence on said recorded input signal such that said quantizer output distribution tends to match said distribution of representations.

According to the present invention there is also provided a system for rating the performance of an individual, such as a test score or an employee review score, the system comprising:

an input for receiving information representing a performance of an individual;

a memory for storing a representation of a desired output distribution of the individual grades;

means for recording an actual performance represented by information received by the input means over a statistically significant period of time;

processing means for setting a grade boundary in dependence on the recorded actual performance such that the output grade distribution tends to match the desired distribution of the representation.

One embodiment of the invention is a system for rating a student's academic performance, the system comprising:

an input for receiving information representative of a student's test score;

a memory for storing a representation of a desired distribution of student ratings in relation to the test output;

means for recording an actual score over a statistically significant period of time, the actual score being represented by information received by the input means;

processing means for setting a grade boundary in dependence on the recorded actual scores such that the output grade distribution tends to match the desired distribution of the representation.

The means for recording the input signal may comprise analogue to digital converter means and memory means arranged to periodically store the output of said analogue to digital converter means.

The representation of the desired output distribution may comprise an array of probabilities for the quantizer or level or a percentile value for each quantization level or level.

The processing means may be configured to change the quantisation step or level such that applying the recorded signal/performance/score to the changed step produces a distribution of the representations. Optionally, the processing means may be configured to assign a quantization step or level to the input signal value in dependence on a percentage of the input signal value/representation/score and a percentage value of the quantization level or level in the recorded input signal/representation/score.

The input may be configured to receive input from a plurality of different sources. An input from different sources is considered as one input signal. The input may comprise means for extracting HTML form data from an http request. It will be appreciated that with the GET or POST method, the meter may construct an http request and encode the measurement in the request as form data.

According to an embodiment of the invention, there is provided a method for rating an individual's performance, the method comprising: receiving information indicative of a performance of an individual; the performance is ranked according to a rank boundary to provide an output rank distribution. The method further comprises the following steps: storing in a memory a representation of a desired output distribution of the levels of the individuals; recording an actual performance represented by the information representative of the performance of the individual received over a statistically significant period of time; setting, with processing means, a level boundary in dependence on the recorded actual performance such that the output level distribution tends to match the desired distribution of the representation.

Drawings

Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

fig. 1 is a block diagram of a speech processor in accordance with the present invention;

FIG. 2 is a flow diagram of a quantization stride determination process executed in the processor of FIG. 1;

FIG. 3 is a graph illustrating a probability density function for a desired input signal;

FIG. 4 is a graph showing a probability density function of an actual input signal;

FIG. 5 is a flowchart of a new quantization stride calculation step of the process shown in FIG. 2;

FIG. 6 illustrates the main physical components of an online testing system for students according to the present invention;

FIG. 7 illustrates a database of the system shown in FIG. 6;

FIG. 8 is a signaling diagram illustrating signaling between a client and a server in the system shown in FIG. 6;

FIG. 9 illustrates a question web page;

FIG. 10 is a flow chart showing tagging of questions solved by a student;

FIG. 11 is a flow chart illustrating a quantizer update process;

FIG. 12 shows a database of a more complex system of the type shown in FIG. 6;

FIG. 13 is a flow chart showing tagging of questions solved by a student; and

fig. 14 is a flowchart showing a quantizer update process.

Detailed Description

Referring to fig. 1, a speech processor according to the present invention includes: a linear analog-to-digital converter (ADC)1, a memory 2, a processor 3 and a controllable analog-to-digital converter 4.

The input analog signal is digitized by the linear ADC 1 and periodically written into the memory 2. The input analog signal is also fed to the controllable ADC 4. Initially, the quantization step of the controllable ADC 4 is set to a default range. For example, the heights of the quantization steps of the controllable ADCs 4 may all be the same.

The processor 3 may send a control signal to the controllable ADC 4 to set the quantization step of the controllable ADC. The quantization step control signal is generated by the processor 3 by analyzing the input signal in a moving window. The width of the moving window corresponds to a period in which the output data is periodically written from the linear ADC 1 to the memory 2.

Referring to fig. 2, at intervals the processor 3 reads blocks of input signal samples from the memory 2 (step s 1).

Referring to fig. 3, the desired probability density function for an input signal may be divided into regions in vertical lines from the amplitude axis. These lines correspond to the boundaries between quantization steps of the linear quantizer. The probability that a desired input signal sample will fall within a particular quantization level is given by the area given by the probability curve and the vertical lines at the upper and lower limits of that level.

The processor 3 is programmed with an array ("model array") containing the quantizer scale probabilities for a predetermined speech signal probability density function. In other words, the nth element of the model array defines an ideal probability of being the controllable quantizer output for the nth value when the input signal is a desired input signal that is an input signal without undesirable gain and non-linear effects.

Referring again to fig. 2, the processor calculates an amplitude probability density function for the input signal based on the samples read from the memory 2 (step s 2). This can be done by constructing a representation of the histogram, ignoring silence (silence), and fitting a curve to the histogram.

For purposes of illustration only, the amplitude probability density function of the input signal may be viewed as shown in FIG. 4. The distribution shown in fig. 4 is skewed to a smaller magnitude than the distribution shown in fig. 3. This is consistent with attenuation. Also, the shape of the curve is not a linear transformation of the curve shown in fig. 3, which indicates some non-linearity in the signal path to the speech processor input.

Once the amplitude probability density function of the input signal is calculated in step s2, a new quantization step for the controllable ADC 4 is determined (step s 3).

Referring to fig. 5, the first step of determining a new quantization step includes: the next element of the model array, the first element, is obtained (step s 11). The quantization step upper bound is then determined by finding the magnitude value that makes the area under the magnitude probability density function of the input signal between the previous upper bound (0 in the case of the first step) and the upper bound of the current step equal or approximate to the currently selected element of the model array (step s 12). Expressed mathematically, it is necessary to give b of the formula_nThe value:

or

Here, f (p) is an amplitude probability density function of the input signal, a is the input signal amplitude, m is the model array, b is the quantization step boundary, and n is the index of the current quantization step.

The new boundary is stored and the model array index is added (step s 13). If the model array index has passed the end of the model array, the process terminates, otherwise the process returns to step s11 to process the next quantization step.

Referring again to fig. 2, the new quantization step is sent by the processor 3 to the controllable ADC 4 (step s4), and the controllable ADC 4 sets its quantizer according to the new step.

In one variant, the model array stores percentages for the quantization levels, and the quantization steps are set by moving the thresholds of the controllable quantizer to obtain the percentages in the model with the actual input signals.

Problems of unwanted system gain variations and non-linearities may also occur in the ranking of educational test results. If the pool of testees is large enough, the characteristics of the referents as a group may be considered constant over a period of time. However, the difficulty of the test questions varies with time. Thus, the unwanted system gain variations and non-linearities may be attributed to differences in the difficulty of the test questions.

In a large-scale public examination, examination grades such as A, B, C are retroactively assigned to different score ranges on the basis of the distribution of scores obtained by the whole group of examiners. Thus, there is a problem of providing rating results to students taking "mock" tests alone for extended periods of time. This problem is particularly evident when the "simulated" test is automatically set and rated.

The adaptive quantizer according to the invention provides a solution to this problem.

First, a simple embodiment will be described. This simple embodiment enables students to answer questions in a subject at a academic level.

Referring to fig. 6, the online test system includes a Web server machine 11 and a database machine 12 connected through a local area network. The client machine 13 can communicate with the Web server machine 11 through the internet 14.

The Web server machine 11 runs a conventional Web server program, such as Apache, using a server-side description language, such as PHP or ASP, which provides dynamic content. A conventional SQL database system, such as MySQL or microsoft SQL Server, runs on the database machine 12.

Referring to fig. 7, the database system includes a test system database 21. The test system database includes a plurality of tables including a Model Table (Model Table)22 and a quantization Table (quantization Table) 23.

The model table 22 stores a representation of the class boundaries in terms of percentages for each class upper boundary and corresponds to the model array in the first embodiment described above. The model table 22 contains columns for the Grade (Grade) and the Boundary (Boundary). The level boundary is stored as a percentage of students falling in or below the level. For example, for level U (lowest level), the boundary is 5%; for level F, the boundary is 10%; for level E, the boundary is 20%, and so on until for level a, the boundary is 100%.

The quantization table 23 provides a mapping between the scores of the students for a particular question on the one hand and the ratings of the students on the other hand. The quantization table 23 includes a question ID column, a possible percentage column, a score count column, a score cumulative count column, a rank column, and a possible score percentage column.

The question ID column only identifies the questions to which the data in the other columns relates.

In general, a single question will not have 100 points assigned to it. It may be the case that a problem has a score of 4 to 20 assigned to it. If a question has 4 points assigned, the student may get a score of 0%, 25%, 50%, 75%, or 100%. Likewise, if a question has 20 points assigned to it, the student may get a score of 0%, 5%, 10%, 15%, etc. For a given problem, there is a row in the quantization table 23 for each possible score. These possible scores are stored in the possible percentage column.

The score count column contains a count of the number of times each question/percentage score combination has occurred.

The score cumulative count field contains the sum of the score count field values for the questions identified in the question ID field.

The possible percentage scores in the possible percentage column are mapped to a rank. The rating is defined by the contents of the rating column.

The possible score percentage column contains: try the identified question, and obtain the percentage of students with respect to the line's score.

The database further comprises: a question table 26 containing columns for question IDs and main question text; and an auxiliary question table 27 containing a column for question id, a text column, and a solution column in the question table 26 to which the auxiliary question belongs.

Now, a process in which the student accepts the test and receives the grade will be described.

Referring to fig. 8, to initiate a question answering session, a student has his Web browser sending an http request for a question page from the client machine 13 to the server machine 11. The server machine 11 responds by sending the requested page in an http response.

Referring to fig. 9, a question page is displayed by the student's browser and includes, in the present example, an introduction question text 31, six auxiliary question sections 32-1,. and 32-6, and a submit button 34. Each auxiliary question portion includes an auxiliary question text and four answer options associated with radio buttons.

The student reads the introductory question text 31 and the auxiliary question text and selects the radio button next to the answer they consider correct. The student can change the mind and select another radio button at this time. After the student is finally satisfied with their selection, the student clicks the submit button 34.

Clicking on the submit button 34 causes the student's Web browser to send the identity of the selected radio button to the Web server machine 11 in a request for a resource, which is defined in the action parameters of the HTML form in the question page.

Referring to fig. 10, when the Web server machine 11 receives a request for a resource defined in the action parameters of the HTML form in the question page, the request is processed to obtain form data, in particular the question id and the value of the selected radio button (step s 21).

The primary and secondary question tables 26, 27 are consulted to obtain the correct solution value for the identified question (step s 22). The values from the form data are compared to the correct values (step s23) and an approximate percentage score, i.e. 0%, 17%, 33%, 50%, 66%, 83% or 100%, is calculated based on the number of correct answers to the secondary question (step s 24).

The rating for the calculated percentage score and question is then retrieved from the quantization table (step s 25). The student's percentage score is stored (s26) and a new page is sent to the student as a response to the request to carry form data (step s 27). In this example, the student's rank is included in the response page.

It will be appreciated that conventional session management techniques may be used to present the problem over multiple pages.

Now, a process of generating the map will be described.

The quantization stride update process is performed at intervals, such as at night.

Referring to fig. 11, the quantified stride update process begins with a possible percentage score for the problem being retrieved from the database 21 (step s 31). Then, percentages are calculated for each of these possible scores within the moving window based on the student score records recorded in the quantization table 23 of the database 21 (s 32).

For each possible score percentage, the closest score percentage above the current score percentage is obtained from the rank boundary table 23 (step s 33). These values are then used to update the rank values in the quantization table 22 (step s 34).

Thus, the grade assigned to a particular percentage score depends on the history of that score obtained and the desired grade percentage.

In order to make the adaptive quantization process easier to understand, the foregoing embodiments are intentionally made simple. Now, a more sophisticated system, which is more desirable in practice, will be described.

In this more complex system, the hardware is substantially as described with reference to fig. 6. However, the processing of the server machine 11 is more complicated. The Web server running on the server machine 11 is programmed to use the data from the database machine 12 to provide a log-in page, a problem page and a report page. The successful person logging in, session processing, and generation of Web pages from data sources are well known to those skilled in the art of Web application design. The member may be a student or a teacher, but only the student will perform the test exercise.

Referring to fig. 12, the database 121 of this embodiment contains an extended version of the table of the previous embodiment, as well as some additional tables. The table is: model table 122, quantization table 123, points reference table 124, questions table 125, auxiliary questions section 126, members table 127, class details table 128, class table 129, school table 131, category table 132, and exercise table 133.

The member table 127 includes columns for member id, member personal details, user id and password for accessing the system, school reference to school table, member type (e.g., student or teacher).

The class table 129 includes a school reference column containing a reference to the school table 131 and a code column. The class detail table 128 only provides a link between the member id in the member table 127 and the class code in the class table 129.

The exercise table 133 contains columns for member id, exercise id, and references to records in the quantization table 123.

The category table 132 stores records of the member's performance in the exercises they have attended. The fields of category table 132 include an ID field, a year field, subject, sub-subject and type fields for information relating to the exercise, a field for reference to exercise table 133, a field for time and duration of the exercise, and fields for rank and point.

Quantization table 123 includes columns for: issue ID, academic level, possible percentage score, score count, count of attempts made to issue, lower and upper levels, possible points score, percentage for possible score and issue subject.

The model table 122 has columns for academic levels, lessons, classes, and class boundary percentages.

Finally, the points reference table 124 has academic-level, rank, and points columns.

Those skilled in the art will readily understand how to use the information stored in the tables to generate a web page for reporting the performance of a member, class, or school.

When the member completes the exercise, the member submits the question form.

Referring to fig. 12, when the Web server receives form data, the relevant program or script extracts the member ID, the time taken, which may be determined by the Javascript counter, the question ID, and the selection made in the solution to the question (step s 41).

A percentage score is calculated (s 42). Increasing a count of attempts made to record problem fields with the extracted problem ID to the quantization table (step s43), wherein the problem fields have the extracted problem ID; and retrieves a record from the quantization table 124 for the combination of the question ID and the calculated percentage score (step s 44).

Next, a record is created in the category table 132 using the information about the question from the question table 125 (step s 45). The level field is set according to the level field of the quantization table record, and the point field is set according to the possible point score field of the quantization table record.

Finally, the program or script sends the appropriate response page to the member's client machine 13.

At intervals, the quantization table 133 is updated. Specifically, the upper and lower levels and possible point scores are updated.

Referring to fig. 13, a quantization table update process is performed for each question (steps s51 and s 57). A count of the number of times each possible score has been achieved in the moving window of the first three months is obtained from the category table (step s 52). From these counts, the percentage for each possible score is calculated (step s 53). These percentages are then compared to the rating boundary percentages from the model table 122 to obtain the upper and lower ratings for the quantization table 123 (s 54). Points for the new upper and lower levels of the problem academic level are taken from the point reference table 124 and averaged to generate a new possible point score (step s 55). The new upper and lower rankings and the new possible point scores are used to update the quantization table record for the current question/percentage score combination (step s 56).

Thus, the rating and allocation of points is determined based on the history of inputs and the desired output distribution.

In a variation of this embodiment, the member is presented with an open question in a first web page and provided with a scoring schedule in a second page. The second page includes a check box that enables a member to identify the items in the breakdown list that are contained by the member in their answers. When the Web server receives form data from the second page, the percentage score is determined by the number of boxes checked by the member. Once the percentage score is determined, the ranking is performed as shown in FIG. 13.

It will be appreciated that the adaptive quantizer is applicable in a variety of situations where a signal is subject to system amplitude noise and the signal may be characterized in a statistical manner. For example, an adaptive quantifier may be used to rate the performance of employees on a relative basis within a company.

Claims (12)

1. A system for rating an individual's performance, comprising:

an input for receiving information representing a performance of an individual;

a memory and a processing device, wherein the memory is connected with the processing device,

characterized in that the system further comprises:

means for recording an actual performance represented by information received by the input over a statistically significant period of time, wherein:

the memory stores a representation of a desired output distribution of the levels of the individuals;

the processing means sets the level boundaries in dependence on the recorded actual performance such that the output level distribution tends to match the desired distribution of the representation.

2. The system of claim 1, wherein the representation comprises an array of probabilities for each possible level output.

3. The system of claim 1, wherein the representation includes a percentile value for each of the levels.

4. A system according to claim 2 or 3, wherein the processing means is configured to change the grade boundaries such that applying the recorded performance to the changed grade boundaries results in the represented grade distribution.

5. A system according to claim 2 or 3, wherein the processing means is configured to assign a rating to an input individual performance in dependence on a percentage of the input individual performance and a percentage value of the rating in the recorded individual performances.

6. A system according to any of claims 1-3, wherein the input is configured to receive input from a plurality of different sources.

7. A system according to any of claims 1-3, wherein said input comprises means for extracting HTML form data from an http request.

8. A system according to any one of claims 1 to 3 for ranking the quiz scores of students.

9. A system according to any of claims 1-3, comprising at least one source for providing information to said input for representing the performance of an individual.

10. The system of claim 9, wherein the processing device comprises a server and the source comprises a client for the server.

11. The system of claim 10, wherein:

the server is operable to provide the client with a question page for completion by an individual,

the client is operable to provide data corresponding to the solution to the problem page completed by the individual to the server, and

the processing means is operable to score and rank the solutions according to set rank boundaries.

12. A method for rating an individual's performance, comprising:

receiving information indicative of a performance of an individual;

ranking the performance according to a rank boundary to provide an output rank distribution;

characterized in that the method further comprises:

storing in a memory a representation of a desired output distribution of the levels of the individuals;

recording an actual performance represented by the information representative of the performance of the individual received over a statistically significant period of time;

setting, with processing means, a level boundary in dependence on the recorded actual performance such that the output level distribution tends to match the desired distribution of the representation.