WO2026009328A1 - Learning device, learning method, and learning program - Google Patents

Abstract

This learning device receives input of information indicating the type of a unit of analysis in a machine learning model, and data extracted for each unit of analysis of said type. The learning device subsequently adds, to data extracted in a unit of analysis for each type of unit of analysis, embedded information indicating the type of the unit of analysis. The learning device trains the machine learning model by using data obtained by adding, to the data extracted for each unit of analysis, embedded information indicating the type of the unit of analysis. The learning device subsequently outputs a parameter of the trained machine learning model.

Description

Learning device, learning method, and learning program

The present invention relates to a machine learning model learning device, learning method, and learning program.

Traditionally, when modeling conversations using machine learning, a modeling method that is robust to the type of unit of analysis required has been required.

Here, conversation modeling refers to efforts to estimate the attributes of the conversation itself and the attributes of the conversation participants from data such as the audio, video, and text of the conversation. Furthermore, the conversation data may be multimodal data including the audio, video, and text of the conversation. Note that conversation modeling also includes efforts to restore the multimodal data of the conversation itself, for example, using an autoencoder or transformer.

Furthermore, a unit of analysis is a category that defines the time interval that separates the data, which is necessary for instantiating conversational data. Typical units of analysis include utterances and turns, and instances extracted based on these units of analysis are used to train the model.

Technology has been proposed to estimate the range of paragraphs in sequential data of dialogues containing multiple topics (see Patent Document 1).

Patent No. 7425368

However, because the above technology limits the unit of analysis to speech, it is not possible to input data with various types of data segments into the model. Furthermore, the model is not robust to the type of data segmentation.

The present invention aims to solve the above-mentioned problems by constructing a model that can accept input data with various types of data delimiters and is robust to the type of data delimiter (i.e., the performance of the model does not deteriorate regardless of the type of data delimiter).

In order to solve the above-mentioned problems, the present invention is characterized by comprising a data input unit that accepts input of information indicating the type of unit of analysis in a machine learning model and data extracted for each unit of analysis of that type; an addition unit that adds embedded information indicating the type of unit of analysis to the data extracted for that unit of analysis; and a model learning unit that learns the machine learning model using data in which embedded information indicating the type of unit of analysis has been added to the data extracted for that unit of analysis.

According to the present invention, it is possible to input data with various types of data delimiters and to construct a model that is robust to the type of data delimiter.

FIG. 1 is a diagram for explaining the underlying technology of the learning device. FIG. 2 is a diagram illustrating an example of processing executed by the learning device. FIG. 3 is a diagram illustrating an example of the configuration of a learning device. FIG. 4 is a flowchart illustrating an example of a processing procedure executed by the learning device. FIG. 5 is a diagram illustrating an example of a computer that executes a learning program.

Below, a description will be given of a form (embodiment) for carrying out the present invention with reference to the drawings. The present invention is not limited to this embodiment.

[Prerequisite technology]
First, the underlying technology of the learning device of this embodiment will be described using Figure 1. Data d shown in Figure 1 is time-series data showing the state of a conversation between members a, b, ..., z. In Figure 1, rectangles with letters written on them represent the utterances of the members. In the following, an example will be described in which the type of unit (u) of analysis of the machine learning model to be learned by the learning device is utterance (division unit number = k).

FIG. 1 shows an extraction interval for extracted data d _i from data d. The extraction interval for extracted data d _i is defined by the unit of analysis of the machine learning model to be learned by the learning device. In this example, the unit of analysis is an utterance. Therefore, the learning device sets the extraction interval for extracted data d i based on the start time (interval start time s _i ^k ) and end time (interval end time e _i ^k ) of each utterance ₍ k) in the user's conversation.

The unit of analysis (u) can be anything other than an utterance, such as a minute period of time equivalent to one frame of video, a turn of speech, five seconds after an utterance, or a randomly sampled three-second interval. All of these must be used appropriately depending on the target of analysis for the machine learning model.

For example, if the task of a machine learning model is to estimate the intention of a user's speech, it is common to use the speech as the unit of analysis. Also, if the task of a machine learning model is to recognize a user's facial expression, it is common to use a small amount of time equivalent to one frame of video as the unit of analysis.

Different units of analysis require different focus points in the data, as well as different quantities and qualities of information, when performing a task. For this reason, in the past, different units of analysis have not been trained together in conversation modeling.

[overview]
On the other hand, the learning device of this embodiment learns different units of analysis (analysis units) all at once in learning a machine learning model.

For example, the learning device extracts data from the data used for training the machine learning model in a data extraction section for an arbitrary analysis unit, and adds embedding information indicating the type of analysis unit (AU (Analyze Unit) embedding) to the extracted data. This allows the learning device to embed information indicating the type of analysis unit in the data used for training the machine learning model.

As a result, even if a learning device trains a machine learning model using data extracted from multiple types of analytical units, AU embedding absorbs the differences between data from different analytical units, making it possible to build a machine learning model that is robust to the type of analytical unit. Furthermore, because the learning device can use data extracted from a variety of analytical units to train a machine learning model, it is possible to improve the amount of data available for training and build a machine learning model with higher performance.

The input data to the above machine learning model can be any data. For example, the input data can be any or a combination of audio signals, facial images, or transcripts of speech from each or all of the participants in the conversation. Furthermore, deep learning, etc., can be used to train the machine learning model.

Next, an example of the processing performed by the learning device will be described using Figure 2. For example, the input to the learning device is conversation data, the type of analysis unit, and the start and end times of the data for that analysis unit.

Here, it is assumed that there are k∈N (N: natural number) types of analytical units u. The learning device extracts extracted data d _i from data d using the start time s _i ^k ∈R (R: real number, ≧0) and end time e _i ^k ∈R of the i ∈N-th data extraction interval of the k-th analytical unit.

Then, the learning device vectorizes the extracted data d _i . For example, the learning device embeds the extracted data d i into an n∈N-dimensional data vector by vectorizing the extracted data d _i .

The learning device also embeds (emb) the number k of the analysis unit u into an n-dimensional vector (AU embedding, analysis unit vector) that indicates the type of analysis unit. Embedding (emb) is, for example, a process of replacing an arbitrary integer with an n-dimensional vector.

The learning device inputs the sum of the above data vector and analytical unit vector into a model (machine learning model) and trains the model. The model can be any type, such as a neural network or SVM (Support Vector Machine).

Furthermore, when the learning device inputs multimodal data into the model, it executes the series of processes shown by reference numeral 201 in Figure 2 in parallel for each piece of multimodal data. The learning device then fuses, for example, the sum of the data vector and the analysis unit vector for each piece of multimodal data using a multimodal fusion method, and then inputs the result into the model. Examples of multimodal fusion methods include early fusion and vector concatenation.

By performing the above process, the learning device can build a model that can perform tasks based on various analytical units for multimodal data.

In conventional methods, model training targets only one analysis unit, and therefore does not take into account, for example, the path (see reference numeral 202) that creates the analysis unit vector shown in Figure 2. In contrast, the training device of this embodiment also includes a path (see reference numeral 202) that creates the analysis unit vector during model training. This allows the training device to perform weighting using the information (analysis unit vector) obtained via the path (see reference numeral 202) when training a model using data extracted via the path (see reference numeral 201).

For example, consider the case where a learning device uses data from two types of analysis units (analysis units A and B) to learn a single model. Here, assume that the data from analysis unit A always contains significant data, and the data from analysis unit B always contains faint data.

In this case, if the learning device assigns the same weight to the data from the two units of analysis when training the model, there is a possibility that the data from unit of analysis B will not have much of an impact on the model. However, if the weights for the data from unit of analysis A and the data from unit of analysis B can be appropriately assigned when training the model, for example by heavily weighting the data from unit of analysis B and lightly weighting the data from unit of analysis A, then even when training a model using data from two different units of analysis, it is believed that the data from both units of analysis will work well on the model.

Therefore, in order to create weights that match the type of analysis unit, the learning device performs an embedding process of information indicating the type of analysis unit, as shown by reference numeral 202 in Figure 2. As a result, if the model to be learned is a neural network-based model, for example, parameters are adjusted to match the type of analysis unit during the model learning process, i.e., appropriate weights are created. This allows the learning device to input data for various types of analysis units and to build a model that is robust to the type of analysis unit.

[Configuration example]
Next, an example of the configuration of the learning device 10 will be described with reference to Fig. 3. The learning device 10 includes, for example, an input/output unit 11, a storage unit 12, and a control unit 13.

The input/output unit 11 is an interface that handles the input and output of various data. For example, the input/output unit 11 accepts input of data (data d) for model training.

The memory unit 12 stores data, programs, etc. referenced when the control unit 13 executes various processes. The memory unit 12 is realized by semiconductor memory elements such as RAM (Random Access Memory) or flash memory, or storage devices such as hard disks or optical disks. For example, the memory unit 12 stores model learning data (data d) received by the input/output unit 11, and parameters of the model obtained by model learning by the control unit 13.

The control unit 13 is responsible for overall control of the learning device 10. The functions of the control unit 13 are realized, for example, by the CPU (Central Processing Unit) executing a program stored in the memory unit 12.

The control unit 13 includes, for example, a learning unit 131 that learns a model. The learning unit 131 includes a data input unit 132, a data extraction unit 133, a vectorization unit 134, an embedding unit 135, an addition unit 136, and a model learning unit 137. The estimation unit 138, shown with a dashed line, may or may not be included; cases where it is included will be described later.

The data input unit 132 accepts input of data for model training. The data input unit 132 also accepts input of information indicating the type of analysis unit in the model and information (start time and end time) indicating the extraction interval for data of that type of analysis unit.

The data extraction unit 133 extracts data for each analysis unit from the model training data based on information indicating the data extraction interval for each type of analysis unit. For example, if the types of analysis units are analysis unit A and analysis unit B, the data extraction unit 133 extracts data for analysis unit A and data for analysis unit B from the model training data.

The vectorization unit 134 vectorizes the data extracted by the data extraction unit 133. For example, the vectorization unit 134 converts the data extracted by the data extraction unit 133 into an n-dimensional vector (data vector). For example, if the data is audio data, the vectorization unit 134 converts it into a vector using a pre-training model such as wav2vec or HuBERT, or a toolkit such as OpenSMILE.

The embedding unit 135 performs an embedding process for information indicating the type of analysis unit received by the data input unit 132. For example, the embedding unit 135 embeds the information indicating the type of analysis unit into an n-dimensional vector (analysis unit vector) using, for example, an embedding layer of a neural network.

The addition unit 136 calculates the sum of the above data vector and analysis unit vector for each type of analysis unit. The model learning unit 137 inputs the sum of the data vector and analysis unit vector for each type of analysis unit into the model and learns the model. The model learning unit 137 then stores the model parameters obtained through learning in the memory unit 12.

This allows the learning device 10 to build a robust model for a variety of analysis units.

[Example of processing procedure]
Next, an example of a processing procedure executed by the learning device 10 will be described with reference to Fig. 4. First, the data input unit 132 of the learning device 10 receives input of data for model training, information indicating the type of analysis unit, and information indicating the extraction interval of the data for the analysis unit (S1).

After S1, the learning device 10 executes the processes shown in S2 to S5 below for each type of analysis unit accepted in S1.

First, the data extraction unit 133 extracts data for each analysis unit from the data used to train the model, based on information indicating the extraction interval of data for each analysis unit received in S1 (S2). Then, the vectorization unit 134 vectorizes the data extracted in S2 (S3). Furthermore, the embedding unit 135 performs an embedding process for information indicating the type of analysis unit received in S1 (S4). Then, the addition unit 136 adds the data vectorized in S3 and the result of the embedding process in S4 (analysis unit vector) (S5).

Once the learning device 10 has performed steps S2 to S5 for all types of analysis units received in S1, the model learning unit 137 inputs the result of adding the vectorized data and the analysis unit vector into the model and performs model learning (S6).

By performing the above process, the learning device 10 can input data for various types of analysis units and construct a model that is robust to the type of analysis unit.

[Other embodiments]
The learning device 10 may perform an estimation process on input data using the model learned by the above process. In this case, the learning device 10 further includes an estimation unit 138 shown in Fig. 3. The estimation unit 138 includes a data input unit 139 and an estimation processing unit 140.

The data input unit 139 accepts input data for the trained model (data to be estimated and information indicating the type of analysis unit).

The type of analysis unit to be input is at least one of the types of analysis unit used to train the model. For example, if the types of analysis unit used to train the model are A, B, and C, then A, B, C, or any combination of these, can be input.

The estimation processing unit 140 inputs information indicating the data to be estimated and the type of analysis unit into the trained model, and uses the model to estimate the data using that analysis unit. The estimation processing unit 140 then outputs the estimation results.

For example, consider a case where the learning unit 131 learns a model using analysis units A, B, and C, and the data input unit 139 receives input of information indicating the data to be estimated and the analysis units A and C. In this case, the analysis processing unit 140 inputs the data to be estimated and the information indicating the analysis units A and C into the model, and estimates the data using the analysis units A and C using the model. The estimation processing unit 140 then outputs the estimation results.

[System configuration, etc.]
Furthermore, the components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[program]
The learning device 10 can be implemented by installing a program (learning program) as package software or online software on a desired computer. For example, by running the program on an information processing device, the information processing device can function as the learning device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

FIG. 5 shows an example of a computer that executes a learning program. The computer 1000 has, for example, memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores a boot program such as BIOS (Basic Input Output System). Hard disk drive interface 1030 is connected to hard disk drive 1090. Disk drive interface 1040 is connected to disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into disk drive 1100. Serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. Video adapter 1060 is connected to, for example, a display 1130.

The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. In other words, the programs that define the processes executed by the learning device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, on the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration of the learning device 10 is stored on the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

Furthermore, data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, the CPU 1020 reads the program modules 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

The program module 1093 and program data 1094 do not necessarily have to be stored on the hard disk drive 1090; they may instead be stored on a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN (Local Area Network) or WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

REFERENCE SIGNS LIST 10 Learning device 11 Input/output unit 12 Memory unit 13 Control unit 131 Learning unit 132, 139 Data input unit 133 Data extraction unit 134 Vectorization unit 135 Embedding unit 136 Addition unit 137 Model learning unit 138 Estimation unit 140 Analysis processing unit

Claims (6)

a data input unit that receives input of information indicating the type of unit of analysis in the machine learning model and data extracted for each unit of analysis of the type;
an adding unit that adds embedded information indicating the type of the unit of analysis to the data extracted in the unit of analysis;
and a model learning unit that learns the machine learning model using data in which embedded information indicating the type of unit of analysis is added to the data extracted for each unit of analysis. The data input unit
The learning device according to claim 1 , wherein the learning device receives input of data extracted from multimodal data for each unit of analysis of the type. a vectorization unit that vectorizes the data extracted in units of analysis;
The adding unit
The learning device according to claim 1 , wherein the embedding information having the same number of dimensions as the vectorized data is added to the vectorized data. The learning device according to claim 1, further comprising an estimation unit that outputs an estimation result for the data using the unit of analysis of the type by inputting data to be estimated and information indicating the type of unit of analysis of the data into the machine learning model after learning. A learning method executed by a learning device, comprising:
receiving input of information indicating the type of unit of analysis in the machine learning model and data extracted for each unit of analysis of the type;
adding embedded information indicating the type of the unit of analysis to the data extracted in the unit of analysis;
and training the machine learning model using data in which embedded information indicating the type of unit of analysis is added to the data extracted for each unit of analysis. receiving input of information indicating the type of unit of analysis in the machine learning model and data extracted for each unit of analysis of the type;
adding embedded information indicating the type of the unit of analysis to the data extracted in the unit of analysis;
and a step of training the machine learning model using data in which embedded information indicating the type of unit of analysis is added to the data extracted for each unit of analysis.