WO2025083691A1

WO2025083691A1 - Systems and methods for evaluating traits of a subject

Info

Publication number: WO2025083691A1
Application number: PCT/IL2024/051016
Authority: WO
Inventors: Oded Maimon; Zohar MAIMON
Original assignee: I Verse Human Avatar Ltd
Current assignee: I Verse Human Avatar Ltd
Priority date: 2023-10-16
Filing date: 2024-10-15
Publication date: 2025-04-24
Anticipated expiration: 2026-04-16

Abstract

The present disclosure provides a solution for evaluating the traits of a subject where the systems and methods extract knowledge of an industrial psychologist, use it to interview a subject for a specific job, and evaluate the traits of the subject. The method comprises receiving transcribed data indicative of one or more responses of the subject to respective one or more questions. The method comprises analyzing transcribed data using a first machine learning model to determine for each response of the subject from said respective one or more questions, a sub-question score of each of a plurality of sub¬ questions. Each score of sub-question or criterion contributes to the final score of one or more traits of the subject. The method further comprises determining based on the sub-question scores a trait score of each of at least one trait of the subject and generating trait data based thereon.

Description

SYSTEMS AND METHODS FOR EVALUATING TRAITS OF A SUBJECT

TECHNOLOGICAL FIELD

The present disclosure is related to the field of artificial intelligence task-oriented engaged conversation (TOEC) and more particularly to the field of automated scoring of interviews based on audiovisual recordings of interview responses.

BACKGROUND ART

Artificial Intelligence (Al) task-oriented engaged conversation (TOEC) systems utilize high-level human expert knowledge to interact with humans to accomplish a wide variety of tasks. Examples of expert knowledge used by TOEC systems include knowledge from medical doctors, lawyers, salespersons, call center service representatives, and industrial psychologists.

Interviews are often integral to any placement process, such as for employment, college admission, army recruitment, etc. In all these cases, an industrial psychologist or an expert professional interviews candidates to evaluate the relevant traits and qualities of the candidate that match the role that the candidate is being interviewed for. Unfortunately, interviews are typically labor-intensive and time-consuming as they are usually conducted and evaluated by humans (who are specialists in their field).

GENERAL DESCRIPTION

The present disclosure provides a solution for evaluating the traits of a subject. By way of example, the systems and methods may extract knowledge of an industrial psychologist, use it to interview a subject for a specific job, and evaluate the traits of the subject.

Therefore, an aspect of the present disclosure provides a method for evaluating traits (e.g. qualities) of a subject. The method comprises receiving transcribed data indicative of one or more responses of the subject to respective one or more questions. The transcribed data comprises text of the response of the subject to one or more questions. It is to be noted that the transcribed data may be formed of a plurality of data pieces, e.g. files, each file comprises data related to a response to a different question. The transcription should be in a suitable language that can be analyzed by the machine learning algorithm or model. If the original transcription of the response is in a different language, a translation of the transcription is performed prior to the analysis. The method further comprises analyzing said transcribed data using a first machine learning model to determine for each response of the subject from said respective one or more questions, a sub-question score of each of a plurality of non-presented sub-questions, wherein each sub-question is related to a respective question, and therefore to the response of this question, and the actual evaluation of the subject is based on the responses on these subquestions. It is to be noted that the sub-questions are not presented to the subject, and they are only used for the analysis in the background. The sub-questions can also be regarded as the criteria for scoring the response of the subject to the questions he/she was asked. Each score of sub-question or criterion contributes, in some extent, to the final score of one or more traits of the subject. The method further comprises determining based on the sub-question scores a trait score of each of at least one trait of the subject and generating trait data based thereon; and outputting said trait data. The output can be a transmission to a device or to a cloud-based server, it can be a graphical representation of the traits of the subject or any other presentation.

The first machine learning model is typically based on large language models, and it is specifically trained to evaluate whether a response of the subject answered one or more sub-questions and optionally grading the quality of the answer in a certain scale. The sub-questions are used as evaluators for traits of the subject and by analyzing the responses of all the sub-questions, the traits profile of the subject can be evaluated.

It is to be noted that any combination of the described embodiments with respect to any aspect of this present disclosure is applicable. In other words, any aspect of the present disclosure can be defined by any combination of the described embodiments.

In some embodiments of the method, the transcribed data comprises transcription of the one or more responses and the respective one or more questions, whereas the method comprises differentiating between the one or more questions and the one or more responses and identifying each response of the one or more responses as related to a respective question. In some embodiments, the method further comprises receiving recorded data that comprises a recorded video, including audio, of the subject and applying a transcription algorithm on said recorded video to generate said transcribed data.

In some embodiments, the method further comprises presenting a selected question from said one or more questions to the subject and recording the response of the subject to said specific question to generate said recorded data. It is to be noted that the recording can include also the question and not just the response.

In some embodiments, the method further comprises determining a follow-up question from said one or more questions based on the response of the subject to said selected question and presenting said follow-up question to the subject.

In some embodiments of the method, determining said follow-up question from said one or more questions is based on one or more sub-question scores of said selected question. The follow-up question is selected based on a contextual relation to the response of the subject to the last question. The follow-up question can be aimed to make the subject elaborate more on a certain aspect of his/her response for the last question or aimed to continue to a different topic for evaluation of the subject.

In some embodiments, the method further comprises determining from the recorded video visual responses, namely language body behavior that is manifested by visual cues, of the subject to said selected question and generate visual response data based thereon. Said analyzing further comprises affecting one or more sub-questions scores of said selected question based on the visual response data. Namely, the behavior of the subject in the video affects the sub-questions scores. The behavior that can affect the questions may be the blinking profde of the subject during the response to the selected question, the gaze direction of the subject during the response, etc. It is to be noted that the time window in which the subject's response to the selected question is analyzed may include also the time that comprises the asking of the question.

In some embodiments of the method, determining for each response of the subject a sub-question score of each of a plurality of sub-questions, comprises determining a level of correlation between a transcription of said response and text indicative of an answer to said sub-question, wherein said sub-question score is based on said level of correlation.

In some embodiments of the method, determining said trait score of a trait of the subject comprises applying a function on one or more trait-related sub-question scores of sub-questions of one or more questions, wherein each of said one or more trait-related sub-questions scores are related to a common trait.

In some embodiments of the method, the first machine learning algorithm or model is trained using text indicative of at least one of said one or more questions, said one or more sub-questions and answers of said one or more sub-questions.

In some embodiments of the method, said outputting said trait data comprises graphically presenting a plurality of trait scores of each of said at least one trait.

In some embodiments of the method, said outputting said trait data comprises generating and presenting a written summary indicative of the traits of subject, wherein said generating and presenting composing said written summary based on the subquestion scores.

In some embodiments of the method, said generating a written summary comprises applying a second trained machine learning algorithm or model on each subquestion score to compose a written explanation for the respective trait score. In other words, the method includes taking all the sub-question scores and use them to generate a written explanation for supporting the reason a certain score was given. The written explanation may include reference to the sub-questions and how the response of the subject is evaluated for responding to these sub-questions,

In some embodiments of the method, the written summary comprises explanations for each determined trait score based on scores of sub-questions.

In some embodiments of the method, the written summary comprises a suitability score for a selected job position based on the plurality of trait scores. The suitability score provides an indication of whether the subject has all the required skills for the job or if the subject fits to the specific job position he/she is interviewed for.

In some embodiments of the method, the written summary comprises an explanation for the determined suitability score based on the plurality of trait scores. This can be made by a machine learning algorithm/model or any other known method.

In some embodiments of the method, at least some of the sub-questions, or in some embodiments all of the sub-questions, are deterministic questions and the respective sub-questions score of each deterministic question is binary, namely, whether the subject responded to the sub-question or not. In some embodiments of the method, the subject is a candidate for a job position, and said evaluating traits comprises evaluating the suitability of the subject to the job position.

In some embodiments, the method further comprises receiving a job position data, indicative of the job position the subject is intended to position.

In some embodiments of the method, said one or more questions are selected based on the job position data.

Yet another aspect of the present disclosure provides a system for evaluating traits of a subject. The system comprises a processing unit, which can be a centralized unit or distributed unit, namely a unit that is composed of a plurality of sub-processing unit, each is configured for performing a part of the algorithm. The processing unit is configured for: (i) receiving transcribed data indicative of one or more responses of the subject to respective one or more questions; (ii) analyzing said transcribed data using a first machine learning algorithm or model to determine for each response of the subject from said respective one or more questions, a sub-question score of each of a plurality of nonpresented sub-questions, wherein each sub-question is related to a respective question; (iii) determining based on the sub-question scores a trait score of each of at least one trait of the subject and generating trait data based thereon; and (iv) outputting said trait data. The sub-questions can also be regarded as scoring criteria for a response of the subject to a specific question. Base on the scores of the sub-questions or criteria, the traits of the subject is evaluated.

In some embodiments of the system, the transcribed data comprises transcription of the one or more responses and the respective one or more questions, wherein the method comprises differentiating between the one or more questions and the one or more responses and identifying each response of the one or more responses as related to a respective question.

In some embodiments of the system, the processing unit further configured for receiving recorded data that comprises a recorded video of the subject and applying a transcription algorithm or model on said recorded video to generate said transcribed data.

In some embodiments of the system, the processing unit further configured for presenting a selected question from said one or more questions to the subject and recording the response of the subject to said specific question to generate said recorded data. In some embodiments of the system, the processing unit further configured for determining a follow-up question from said one or more questions based on the response of the subject to said selected question and presenting said follow-up question to the subject.

In some embodiments of the system, the processing unit further configured for determining said follow-up question from said one or more questions is based on one or more sub-question scores of said selected question.

In some embodiments of the system, the processing unit further configured for determining from the recorded video visual responses of the subject to said selected question and generate visual response data based thereon. Said analyzing further comprises affecting one or more sub-questions scores of said selected question based on the visual response data.

In some embodiments of the system, said determining for each response of the subject a sub-question score of each of a plurality of sub-question comprises determining a level of correlation between a transcription of said response and text indicative of an answer to said sub-question, wherein said sub-question score is based on said level of correlation.

In some embodiments of the system, said determining said trait score of a trait of the subject comprises applying a function on one or more trait-related sub-question scores, wherein each of said one or more trait-related sub-questions scores are related to a common trait.

In some embodiments of the system, the first machine learning algorithm/model is trained using text indicative of at least one of said one or more questions, said one or more sub-questions and answers of said one or more sub-questions.

In some embodiments of the system, said outputting said trait data comprises graphically presenting a plurality of trait scores of each of said at least one trait.

In some embodiments of the system, said outputting said trait data comprises generating and presenting a written summary indicative of the traits of subject, wherein said generating and presenting composing said written summary based on the subquestion scores.

In some embodiments of the system, said generating a written summary comprises applying a second trained machine learning algorithm/model on each sub-question score to compose a written explanation for the respective trait score. In some embodiments of the system, the written summary comprises explanations for each determined trait score based on scores of sub-questions.

In some embodiments of the system, the written summary comprises a suitability score for a selected job position based on the plurality of trait scores.

In some embodiments of the system, the written summary comprises explanation for the determined suitability score based on the plurality of trait.

In some embodiments of the system, the at least some of the sub-questions are deterministic questions and the respective sub-questions score of each deterministic question is binary.

In some embodiments of the system, the subject is a candidate for a job position and said evaluating traits comprises evaluating the suitability of the subject to the job position.

In some embodiments of the system, the processing unit further configured for receiving a job position data, indicative of the job position the subject is intended to position.

In some embodiments of the system, said one or more questions are selected based on the job position data.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to better understand the subject matter that is disclosed herein and to exemplify how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:

Fig. 1A schematically illustrates a system for evaluating traits of a subject, according to some embodiments of the invention;

Fig. IB schematically illustrates an exemplary implementation of a system for evaluating traits of a subject, according to some embodiments of the invention;

Fig. 2 schematically illustrates a machine-learning based interviewer of a system for evaluating traits of a subject, according to some embodiments of the invention;

Fig. 3A is an example of a graphical output of a system for evaluating traits of a subject, according to some embodiments of the invention; Fig. 3B is an example of a textual output of a system for evaluating traits of a subject, according to some embodiments of the invention;

Fig. 4 schematically illustrates a machine-learning process of a system for evaluating traits of a subject, according to some embodiments of the invention; and

Figs. 5A-5B are flowcharts exemplifying different embodiments of a method for evaluating traits of a subject according to an aspect of the present disclosure.

Fig. 6 schematically illustrates an exemplary realization of the components of the trait evaluator in four ways: process step, model, method, and knowledge source.

Fig. 7 is a block diagram exemplifying an embodiment of a system for evaluating traits of a subject according to an aspect of the present disclosure.

DETAILED DESCRIPTION

The following figures are provided to exemplify embodiments and realization of the invention of the present disclosure.

In the figures throughout the application, like elements of different figures were given similar reference numerals shifted by the number of hundreds corresponding to the number of the respective figure.

Reference is now made to Fig. 1A, which schematically illustrates a system 100 for evaluating traits of a subject, according to some embodiments of the invention, and to Fig. IB, which schematically illustrates an exemplary implementation of the system for evaluating traits of a subject, according to some embodiments of the invention.

System 100 may include a processor 110, a database 120, a network 130 and a plurality of user devices 140a,b. Processor 110 may communicate with plurality of user devices 140a, b and database 120 via network 130. For example, system 100 may be implemented using cloud -based services.

A user device 140a may be used by the subject (e.g., subject 141a as shown in Fig. IB) to access system 100. For example, the subject may access system 100 via a link in an email message sent to user device 140a. The email and link may be an invitation to a job interview. A user device 140b may be used by a job recruitment professional (e.g., human resources manager 141b as shown in Fig. IB) to access system 100. For example, the job recruitment professional may send a link via an email to user device 140a via network 130. The email and link may be an invitation to a job interview. The subject may use a software-as-a-service (SaaS) application to access system 100 with user device 140 (e.g., 240 in Fig. 2).

Processor 110 may determine one or more questions to ask the subject based on a strategic goal for an interview (e.g., a particular job description). Processor 110 may present a specific question from said one or more questions to the subject. A recording of the response of the subject to said specific question is obtained and stored in a database (e.g., database 120 and a MongoDB 121 as shown in Figs. 1A and IB, respectively) as recorded data. The recorded data may include the question and the response to the question. The recorded data may include a recorded video (including audio) of the subject (e.g., recorded interview videos from a dedicated recording system). Processor 110 may retrieve said recorded data from the database. Processor 110 may apply a transcription algorithm on said recorded video to generate transcribed data. Processor 110 may store in database 120 said transcribed data. The transcription may be in a suitable language that can be analyzed by a machine learning algorithm. If the original transcription of the response is in a different language, a translation of the transcription may be performed by processor 110 prior to the analysis.

Processor 110 may receive transcribed data (e.g., from database 120 and MongoDB 121 as shown in Figs. 1A and IB, respectively) indicative of one or more responses of the subject (e.g., subject 14 la as shown in Fig. IB) to respective one or more questions. The transcribed data may include transcription of the one or more responses and the respective one or more questions. Processor 110 may differentiate between the one or more questions and the one or more responses in said transcribed data. Processor 110 may identify each response of the one or more responses as related to a respective question.

Processor 110 may analyze said transcribed data using a first machine learning algorithm (e.g., using an evaluator 111 as shown in Figs. 1A and/or IB) to determine for each response of the subject to said respective one or more questions, a sub-question score of each of a plurality of sub-questions, wherein each sub-question may be related to a respective sub-question and therefore to the response of said respective sub-question. Processor 110 may determine for each response of the subject a sub-question score of each of a plurality of sub-questions, based on determining a level of correlation between a transcription of said response and text indicative of an answer to said sub-question. Said sub-question score may be based on said level of correlation. Processor 110 may analyze each response of the subject from said respective one or more questions over a selected time window that may include or exclude a time that comprises asking said respective one or more questions.

Text indicative of at least one of said one or more questions may be stored in a database (e.g., from database 120 and MongoDB 121 as shown in Figs. 1A and IB, respectively). Said text may be an input to the first machine learning algorithm (e.g., proprietary datasets 210 as shown in Fig. 2).

It is to be noted that the sub-questions are not presented to the subject and they only serve to decompose the response of the subject and facilitate to simplify the evaluation of the response of the subject to the question that is related to the subquestions. At least some of the sub-questions may be deterministic questions and the respective sub-questions score of each deterministic question is binary, namely whether the subject responded to the sub-question or not.

Processor 110 may determine a follow-up question from said one or more questions based on the response of the subject to said selected question. Processor 110 may present said follow-up question to the subject. Processor 110 may determine said follow-up question from said one or more questions based on one or more sub-question scores of said selected question. Processor 110 may determine said follow-up question from said one or more questions based on a dependency of the follow-up question on the said one or more questions. The dependency may be based on a contextual relationship between the follow-up question and the said one or more questions.

Processor 110 may present said specific question and said follow-up question to the subject using a third machine-learning algorithm. The third machine learning algorithm may engage in a real-time conversation with the subject (e.g., 230 in Fig. 2).

Processor 110 may determine from the recorded video visual responses of the subject to said selected question and generate visual response data based thereon, namely the physical behavior of the subject during the time the question asked and during the response of the subject. Visual responses may include a blinking pattern of the subject, a direction of gaze of the subject, hand movements of the subject, head movements of the subject, and any relevant facial expressions or body-language. Processor 110 may update one or more sub-questions scores of said selected question based on the visual response. The first, second and third machine learning algorithms (e.g., 430 in Fig. 4) may be trained using text indicative of at least one of said one or more questions, said one or more sub-questions and answers of said one or more sub-questions. Said text may be derived from occupational psychology and expert knowledge (e.g., 410 in Fig. 4) and/or data from real interviews (e.g., 420 in Fig. 4). Each one of said first, second and third machine learning algorithms may be implemented using one or more large language models (LLMs). The expert knowledge may include feedback by an occupational psychologist on interviews conducted by system 100.

Processor 110 may determine based on the sub-question scores a trait score of each of at least one trait of the subject and generating trait data based thereon. Processor 110 may determine based on the sub-question scores a trait score of each of at least one quality trait of the subject and generating trait data based thereon using a second machine learning algorithm (e.g., using a report generator 120 as shown in Figs. 1A and/or IB). Processor 110 may determine said trait score of a trait of the subject based on applying a function on one or more trait-related sub-question scores of sub-questions of one or more questions, wherein each of said one or more trait-related sub-questions scores are related to a common trait.

Processor 110 may output said trait data. The output may be a transmission to a device or to a cloud-based server. Processor 110 may output said trait data by graphically presenting a plurality of trait scores of each of said at least one trait (e.g., as shown in Fig. 3A). Processor 110 may output said trait data by generating and presenting a written summary indicative of the traits of the subject, based on the sub-question scores (e.g., as shown in Fig. 3B). Said written summary may include a professional opinion and an expert recommendation for the next steps in a hiring process based on the results of the interview.

Reference is now made to Figs. 5A-5B, which are flowcharts of a method for evaluating traits of a subject according to some embodiments of the invention.

The operations of Figs. 5A-5B may be performed using the equipment of Figs. 1A and/or Fig. IB and/or using any other suitable equipment.

Figs. 5A-5B are flow diagrams exemplifying non-limiting embodiments of the method according to an aspect of the present disclosure. Figs. 5A-5B exemplify a method for evaluating traits (e.g. qualities) of a subject. In other words, the purpose of the method of the present disclosure is to evaluate the traits profile of the subject and therefore its suitability for a certain purpose. The subject may be a candidate for a job position and said evaluating traits comprises evaluating the suitability of the subjectto the job position. The evaluation is performed based on communication with the subject, typically through a video interview. The method can be realized in a recorded interview or on a live interview. During the interview, the subject is asked some questions, either by a virtual interviewer or a real-person interviewer. The responses to these questions are transcribed and transcribed data is generated based thereon. The method, as exemplified in Fig. 5A comprises receiving the transcribed data 530 indicative of one or more responses of the subjectto respective one or more questions. The transcribed data comprises transcription of the one or more responses and the respective one or more questions. The transcription is performed by any state-of-the-art automatic speech recognition algorithms, e.g. Whisper model. Any question that is asked is composed of a plurality of sub-questions, which can be scale-based questions (for example, the responses can be ranked on a scale of between 1-5 or any other normalized scale between X-Y) or can be Yes/No questions, namely questions that the responses of the candidate on them are ranked binarily. In any realization, the scale used to rank the response assists in defining the quality of the answer to the question by the candidate. For example, a question that subject can be asked is: “Could you describe a situation where you had to learn a new subject or skill in a short period? How did you approach this task, and what challenges did you face”. This question is composed of at least the following sub-questions: “Did the candidate learn the required content relatively quick?”, “Was the information needed to learn complex?”, “Was the information new and unfamiliar?”, “Was the learning based on multiple sources?, “Did the candidate learn alone?”, “Does the candidate display excitement, curiosity and positive attitude?”, “Is the learning described as easy?”, “How much of the learning did you handle independently?” and “How effectively do you feel you managed to learn the required material in the time you had?”. The sub-questions define the building blocks of the question that is output to the subject and they constitute the criteria for assessment of the traits of the subject. In other words, the sub-questions span the space of the question. Therefore, the complexity of evaluating a response of a subject to a question and determining one or more traits based on the response is reduced by using this subquestions / criteria approach. While the subject is asked a general and relatively complex question, the method seeks whether the subject actually responded to more basic questions composing this question. The transcribed data is then analyzed 532 to determine whether the subject answered any of the sub-questions by applying a trained machine learning model, e.g. a large language model such as a model based on GPT4. The machine learning was trained by ground truth data that was created by professionals, such as psychologists, Human Resource professionals, behavioral analysts, career counselors, personality assessors, etc. The trait score of each of the evaluated traits by the method is determined 534 based on the responses scores of the sub-questions. By processing all the scores of the sub-questions, a trait profde of the subject is determined, and the method comprises generating trait data 536 that comprises the scores of all the evaluated traits. The method further comprises outputting the trait data 538. For example, the trait data may be output in the form of a polygon presenting the trait scores, as seen in Fig. 3A. The output of the trait profile of the subject may be presented with respect to certain requirements of set of traits scores to allow evaluation of the subject to the purpose demanding the specific set of traits scores, e.g. for a job position. The output may also include a report with explanations for each of the evaluated scores. Namely, the report includes explanations of the reasons for giving a specific trait score. The wording of the explanations is created based on a trained machine learning model that uses as an input the sub-questions and the scores of the subject to the sub-questions. In some embodiments, the input may also include the transcribed data or parts thereof. The model may be based on a large language model, such as GPT4 or any other large language model trained for this purpose. Therefore, the model generates the explanation text that includes basis for the trait score.

Reference is now being made to Fig. 5B, which differs from the example of Fig. 5 A by including several additional steps. In this example, the method further comprises presenting a selected question 524 to the subject. The question is selected from a plurality of questions stored in a data base. The selection is made based on the type of the interview and the purpose of the interview, e.g. a specific job position with specific traits requirements. Therefore, by inputting parameters defining the interview and its purposes, e.g. in the form of job position data, a set of questions may be selected to be asked during the interview, which the responses to the sub-questions composing them will lead to the evaluation of the traits needed to be evaluated for the specific purpose. The response of the subject is recorded and the method further comprises generating recorded data 526 that comprises the recorded video of the response of the subject including audio, and applying a transcription algorithm 528 on said recorded video to generate the transcribed data 530. The recorded video is typically a digital interview conducted with the subject that comprise a plurality of responses to questions by the subject. In case that the recorded video includes both the questions and the responses, the method further comprises differentiating between the one or more questions and the one or more responses 531 and identifying each response of the one or more responses as related to a respective question. This allows to perform the analysis of the response with respect to the relevant subquestions.

In some embodiments, the method further comprises analyzing in real-time the response of the subject to the question that was output. The analysis may indicate that the subject did not answer any of the sub-questions or responded on an insufficient number of sub-questions, e.g. below a certain selected threshold. In these cases, the method further comprises determining a follow-up question from said one or more questions based on the real-time analysis of the response of the subject to said selected question and presenting said follow-up question to the subject.

While responding to the questions, the subject exhibits a certain body language, either through facial expressions orthrough other non-verbal cues. In some embodiments, the recorded data of the video is analyzed to determine body language data indicative of the body language of the person associated with each response. This may assist to understand whether the response is genuine and how confident the subject is in the response. This data may be supportive for the evaluation of the score of the subject.

The determination of the score of each trait is performed based on a calculation of the scores of all the sub-questions that are considered as contributing to the trait. It may be that some sub-questions will affect more on the eventual score than others, therefore the calculation may include applying weight factors on each of the scores of the subquestions. The scale of the final scores of the trait may be determined based on some benchmark dictated by professionals or by literature. The same is also relevant for the contribution of each of the sub-questions to a specific trait.

Reference is now being made to Fig. 6, which presents an exemplary realization of the components of the trait evaluator in four ways: process step, model, method, and knowledge source. The figure shows: what (the steps of the process), with what (which model), how, and what is the knowledge (the data). The figure shows the four LLMs: one for the GP -Grading principles, one for the overall score (GPT3 that is tailored to the requirements of the model), one to explain the score of each trait, and one for the overall explanation, namely the written report.

Fig. 7 is a block diagram of exemplary embodiments of the system of the present disclosure. Fig. 7 exemplifies a system 700 for evaluating traits) of a subject according to an aspect of the present disclosure. The subject may be a candidate for a job position and said evaluating traits comprises evaluating the suitability of the subject to the job position. It is to be noted that the above-described method can be realized by such system.

The system 700 comprises at least one processing circuitry 704, coupled with memories (not in drawings) and stores programming instructions to be executed by the processing circuitry 704.

The system comprises a recorder 702 which is configured to record at least a part of an interview of the subject that includes responses of the subject to specific questions and to generate recorded data RD. An input and pre-processing module 706 of the system is configured to receive the recorded data RD from the recorder 702 that comprises a recorded video including audio, of the subject, including a video recording of visual responses of the subject to a presented question Q being output to the subject by an output module 710 of the system. The input and pre-processing module 706 is also configured to generate visual response data VRD based on the visual response in the recorded data RD. The visual response data VRD comprises data indicative of the body language of the subject, while responding to each question. Parameters extracted from the body language analysis can further assist in the process of evaluating the subject, affecting their scores and their evaluated traits. The input and pre-processing module 706 is configured to apply a transcription algorithm on the recorded data RD to generate transcribed data TD. The transcribed data TD comprises transcription of one or more responses of the subject to the presented questions. This transcription is a decomposition of the questions and responses into multiple pairs of sub-questions and their associated response. The process of transforming the recorded data into the transcribed data is previously described. This decomposition provides a reduction of the complexity level of the evaluation procedure.

The process of analyzing the transcribed data TD is being done by the evaluation module 708, using trained machine learning model, which ultimately determines for each pair of sub-question and response, a sub-question score. The process of training the machine-learning model is previously described. Based on the multiple sub-questions scores, the evaluation module generates scores data SD. The evaluation module 708 is further configured to determine based on the subquestion scores SD trait scores TS, comprises scores of each of at least one trait of the subject. Each sub-question score may affect at least one trait score, indicative of a trait of the subject. For example, a specific sub-question score, based on the response of the subject associated with a specific sub-question, is indicative of the subject's problemsolving ability and the subject's ingenuity. Thus, this sub-question score will affect the subject's traits scores associated with the traits of problem-solving ability and ingenuity. The determination of the traits scores can be done using a set of predetermined functions, which receives at least one sub-question score as an input and outputs at least one trait score. These functions may be adjusted over time. The output module 710 is configured to receive trait score TS and to generate and output trait data TraitD based thereon. Outputting the trait data TraitD comprises graphically presenting a plurality of trait scores indicative of the subjects analyzed traits. Furthermore, the outputted trait data TraitD may also comprise a generated written summary regarding the subject's evaluated traits. This process of generating the written summary may be done using another machine learning algorithm. The written summary also comprises a suitability score for a selected purpose, such as a job position, based on the plurality of trait scores, as well as an explanation regarding the determined suitability score.

The output module 710 is also configured to receive the scores data SD and to select and present a question Q to the subject, based on the scores data SD. This question is selected from an existing repository of questions in the data base. This repository may be updated over time. The presented question Q may also be a follow-up question which is determined based on the response of the subject to a previous selected question. This is being done for example in cases in which the subject's response to the presented question was insufficient. The determination of the follow-up question from is based on the sub-question scores, in the scores data SD, of the selected question.

The input and pre-processing module 706 may also be configured to receive a job position data JPD indicative of the job position that the subject is intended to position. The job position data JPD may affect the traits to be evaluated of the subject and therefore the questions output to the subject.

Claims

CLAIMS:

1. A method for evaluating traits of a subject, comprising: receiving transcribed data indicative of one or more responses of the subject to respective one or more questions; analyzing said transcribed data using a first machine learning algorithm to determine for each response of the subject from said respective one or more questions, a sub-question score of each of a plurality of sub-questions, wherein each sub-question is related to a respective question; determining based on the sub-question scores a trait score of each of at least one trait of the subject and generating trait data based thereon; and outputting said trait data.

2. The method of claim 1, wherein the transcribed data comprises transcription of the one or more responses and the respective one or more questions, wherein the method comprises differentiating between the one or more questions and the one or more responses and identifying each response of the one or more responses as related to a respective question.

3. The method of claims 1 or 2, further comprising receiving recorded data that comprises a recorded video of the subject and applying a transcription algorithm on said recorded video to generate said transcribed data.

4. The method of claim 3, comprising presenting a selected question from said one or more questions to the subject and recording the response of the subject to said specific question to generate said recorded data.

5. The method of claim 4, comprising determining a follow-up question from said one or more questions based on the response of the subject to said selected question and presenting said follow-up question to the subject.

6. The method of claim 5, wherein determining said follow-up question from said one or more questions is based on one or more sub-question scores of said selected question.

7. The method of claim 4, further comprising determining from the recorded video visual responses of the subject to said selected question and generate visual response data based thereon; wherein said analyzing further comprises affecting one or more sub-questions scores of said selected question based on the visual response data.

8. The method of claim 1, wherein determining for each response of the subject a sub-question score of each of a plurality of sub-questions, comprises determining a level of correlation between a transcription of said response and text indicative of an answer to said sub-question, wherein said sub-question score is based on said level of correlation.

9. The method of claim 1, wherein determining said trait score of atrait ofthe subject comprises applying a function on one or more trait-related sub-question scores, wherein each of said one or more trait-related sub-questions scores are related to a common trait.

10. The method of claim 1, wherein the first machine learning algorithm is trained using text indicative of at least one of said one or more questions, said one or more subquestions and answers of said one or more sub-questions.

11. The method of claim 1, wherein said outputting said trait data comprises graphically presenting a plurality of trait scores of each of said at least one trait.

12. The method of claim 1, wherein said outputting said trait data comprises generating and presenting a written summary indicative of the traits of subject, wherein said generating and presenting composing said written summary based on the subquestion scores.

13. The method of claim 12, wherein said generating a written summary comprises applying a second trained machine learning algorithm on each sub-question score to compose a written explanation for the respective trait score.

14. The method of claim 12 or 13, wherein the written summary comprises explanations for each determined trait score based on scores of sub-questions.

15. The method of claim 12, wherein the written summary comprises a suitability score for a selected job position based on the plurality of trait scores.

16. The method of claim 15, wherein the written summary comprises explanation for the determined suitability score based on the plurality of trait.

17. The method of claim 1, wherein at least some of the sub-questions are deterministic questions and the respective sub-questions score of each deterministic question is binary.

18. The method of claim 1, wherein the subject is a candidate for a job position and said evaluating traits comprises evaluating the suitability ofthe subjectto the job position.

19. The method of claim 18, comprising receiving a job position data, indicative of the job position the subject is intended to position.

20. The method of claim 19, wherein said one or more questions are selected based on the job position data.

21. A system for evaluating traits of a subject, comprising: a processing unit configured for: receiving transcribed data indicative of one or more responses of the subject to respective one or more questions; analyzing said transcribed data using a first machine learning algorithm to determine for each response of the subject from said respective one or more questions, a sub-question score of each of a plurality of sub-questions, wherein each sub-question is related to a respective question; determining based on the sub-question scores a trait score of each of at least one trait of the subject and generating trait data based thereon; and outputting said trait data.

22. The system of claim 21, wherein the transcribed data comprises transcription of the one or more responses and the respective one or more questions, wherein the method comprises differentiating between the one or more questions and the one or more responses and identifying each response of the one or more responses as related to a respective question.

23. The system of claims 21 or 22, wherein the processing unit further configured for receiving recorded data that comprises a recorded video of the subject and applying a transcription algorithm on said recorded video to generate said transcribed data.

24. The system of claim 23, wherein the processing unit further configured for presenting a selected question from said one or more questions to the subject and recording the response of the subject to said specific question to generate said recorded data.

25. The system of claim 24, wherein the processing unit further configured for determining a follow-up question from said one or more questions based on the response of the subject to said selected question and presenting said follow-up question to the subject.

26. The system of claim 25, wherein the processing unit further configured for determining said follow-up question from said one or more questions is based on one or more sub-question scores of said selected question.

27. The system of claim 24, wherein the processing unit further configured for determining from the recorded video visual responses of the subject to said selected question and generate visual response data based thereon; wherein said analyzing further comprises affecting one or more sub-questions scores of said selected question based on the visual response data.

28. The system of claim 21, wherein determining for each response of the subject a sub-question score of each of a plurality of sub-question comprises determining a level of correlation between a transcription of said response and text indicative of an answer to said sub-question, wherein said sub-question score is based on said level of correlation.

29. The system of claim 21, wherein determining said trait score of a trait of the subject comprises applying a function on one or more trait-related sub-question scores, wherein each of said one or more trait-related sub-questions scores are related to a common trait.

30. The system of claim 21, wherein the first machine learning algorithm is trained using text indicative of at least one of said one or more questions, said one or more subquestions and answers of said one or more sub-questions.

31. The system of claim 21, wherein said outputting said trait data comprises graphically presenting a plurality of trait scores of each of said at least one trait.

32. The system of claim 21, wherein said outputting said trait data comprises generating and presenting a written summary indicative of the traits of subject, wherein said generating and presenting composing said written summary based on the subquestion scores.

33. The system of claim 32, wherein said generating a written summary comprises applying a second trained machine learning algorithm on each sub-question score to compose a written explanation for the respective trait score.

34. The system of claim 32 or 33, wherein the written summary comprises explanations for each determined trait score based on scores of sub-questions.

35. The system of claim 32, wherein the written summary comprises a suitability score for a selected job position based on the plurality of trait scores.

36. The system of claim 35, wherein the written summary comprises explanation for the determined suitability score based on the plurality of trait.

37. The system of claim 21, wherein at least some of the sub-questions are deterministic questions and the respective sub-questions score of each deterministic question is binary.

38. The system of claim 21, wherein the subject is a candidate for a job position and said evaluating traits comprises evaluating the suitability of the subjectto the job position.

39. The system of claim 38, wherein the processing unit further configured for receiving a job position data, indicative of the job position the subject is intended to position.

40. The system of claim 39, wherein said one or more questions are selected based on the job position data.