JP6614545B1 - Estimation system and estimation apparatus - Google Patents

Abstract

An estimation system and an estimation apparatus capable of improving accuracy of estimating a key input are provided. An estimation system for estimating a character string input using a virtual keyboard, wherein angle information relating to an angle of the user's finger is obtained based on shooting information obtained by shooting an operation of a user pressing the virtual keyboard. A reference storing at least three levels of association between acquisition means for acquiring the target information, past target information acquired in advance, and reference information including a character string input by the past target information A database, the reference database, and evaluation means for obtaining evaluation information including first relations of three or more stages between the target information and the reference information, and generating an estimation result based on the evaluation information And generating means. [Selection] Figure 1

Description

  The present invention relates to an estimation system and an estimation device for estimating a character string input using a virtual keyboard.

  Conventionally, as a technique for controlling key input without using a physical keyboard, for example, an information processing apparatus disclosed in Patent Document 1 has been proposed.

  In Patent Document 1, an extraction unit that extracts a hand region from an image input by an imaging unit that captures a hand, a reference line determination unit that determines a predetermined line in the image as a pressed reference line based on the hand region, A determination unit that determines that the pressing is performed when the lowermost part of the region exceeds the pressing reference line, and a first position determination unit that determines a position in the depth direction based on the aspect ratio of the hand region when the determination unit determines that the pressing is performed And a second position determination unit that determines a horizontal position based on the position of the lowermost part of the hand area when the determination unit determines that the button is pressed, and an input based on the determined depth position and horizontal position. An information processing apparatus including an input key determination unit that determines a key is disclosed.

International Publication No. 2012/176315

  Here, for example, in an information processing apparatus such as Patent Document 1, an input command is estimated based on a threshold based on a captured image from one direction by one camera. For this reason, it is necessary to perform key input according to a threshold value, and there is a concern that variations in pressing depth cannot be taken into account, and it is difficult to improve the accuracy of key input estimation. Therefore, the improvement of the precision which estimates key input is mentioned as a subject.

  Therefore, the present invention has been devised in view of the above-described problems, and an object of the present invention is to provide an estimation system and an estimation device that can improve the accuracy of estimating a key input. It is in.

An estimation system according to a first aspect of the present invention is an estimation system that estimates a character string input using a virtual keyboard, and is based on shooting information obtained by shooting an operation of the user pressing the virtual keyboard. Three or more levels of association between acquisition means for acquiring target information including angle information regarding angles, past target information acquired in advance, and reference information including a character string input by the past target information Is obtained by referring to the reference database in which the information is stored, and evaluation information including three or more levels of first association between the target information acquired by the acquisition unit and the reference information. means, and a generating means for generating an estimation result based on the evaluation information, the angle of the finger, the coordinates in the joint portion of the finger coordinates of the user Characterized in that it presents a angle calculated from 3 points or more coordinates of the heart.

  An estimation system according to a second invention is the estimation system according to the first invention, wherein the acquisition means includes shooting information acquisition means for acquiring the shooting information, and coordinate data acquisition means for acquiring coordinate data of the user's finger from the shooting information. And angle information acquisition means for acquiring the angle information from the coordinate data.

  The estimation system according to a third invention is the estimation system according to the second invention, wherein the photographing information includes three-dimensional image data generated using a three-dimensional camera, and the coordinate data acquisition means is configured to obtain the coordinates from the three-dimensional image data. It is characterized by acquiring data.

  The estimation system according to a fourth aspect of the present invention is the estimation system according to any one of the first to third aspects, wherein the target information and the past target information include optical flow information indicating movement of the user's finger. .

The estimation system according to a fifth aspect of the present invention is the estimation system according to any one of the first to fourth aspects, wherein the target information and the past target information include keyboard information indicating a type of the virtual keyboard.

The estimation system according to a sixth aspect of the present invention is characterized in that, in any one of the first to fifth aspects, the reference database is constructed by machine learning.

An estimation device according to a seventh aspect of the present invention is an estimation device that estimates a character string input using a virtual keyboard, and is based on shooting information obtained by shooting a user's operation of pressing the virtual keyboard. Three or more levels of association between an acquisition unit that acquires target information including angle information regarding an angle, past target information acquired in advance, and reference information including a character string input by the past target information Is obtained by referring to the reference database in which the information is stored, and evaluation information including three or more levels of first association between the target information acquired by the acquisition unit and the reference information. comprising a part, and a generator for generating an estimation result based on the evaluation information, the angle of the finger, 3 around the coordinates in the joint portion of the finger coordinates of the user Ten以 Characterized in that it presents a angle calculated from the coordinates.

According to 1st invention- 6th invention, an evaluation means refers to a reference database, and acquires the evaluation information containing the 1st relevance of 3 steps | paragraphs or more between object information and reference information. For this reason, it is possible to generate an estimation result based on the past result, and it is possible to estimate a key input in consideration of variations in the pressing depth. Thereby, it is possible to improve the accuracy of estimating the key input.

Further, according to the first to sixth inventions, the target information includes angle information. For this reason, compared with the case where the position of the hand is calculated from one image, the positional relationship of the hand with respect to the virtual keyboard can be easily acquired regardless of the hand characteristics that are different for each user. This makes it possible to acquire information necessary for estimating key input with high accuracy.

  According to the second invention, the coordinate data acquisition means acquires coordinate data from the photographing information. Further, the angle information acquisition means acquires angle information from the coordinate data. For this reason, the capacity of data to be evaluated can be reduced. This makes it possible to suppress an increase in storage capacity when storing target information. In addition, parameters necessary as evaluation targets can be suppressed, and further improvement in estimation accuracy can be achieved.

  According to the third invention, the photographing information includes three-dimensional image data generated using a three-dimensional camera. For this reason, when acquiring angle information, a conventional three-dimensional camera can be used without providing a new camera or the like. Thereby, it is possible to easily prepare an environment for estimating key input.

  According to the fourth invention, the target information and the past target information include optical flow information. For this reason, by combining the optical flow information and the angle information, it is possible to consider the movement at the time of key input for each user. Thereby, it is possible to further improve the accuracy of estimating the key input.

According to the fifth aspect , the target information and the past target information include keyboard information. For this reason, even when a virtual keyboard having a different key arrangement or the like is used, key input can be estimated with high accuracy.

According to the sixth invention, the reference database is constructed by machine learning. For this reason, it is possible to realize key input estimation excluding subjectivity as compared with the case of setting a threshold for the pressing depth calculated from the image.

According to the seventh aspect , the evaluation unit refers to the reference database, and acquires evaluation information including the first association degree of three or more stages between the target information and the reference information. For this reason, it is possible to generate an estimation result based on the past result, and it is possible to estimate a key input in consideration of variations in the pressing depth. Thereby, it is possible to improve the accuracy of estimating the key input.

According to the seventh aspect , the target information includes angle information. For this reason, compared with the case where the position of the hand is calculated from one image, the positional relationship of the hand with respect to the virtual keyboard can be easily acquired regardless of the hand characteristics that are different for each user. This makes it possible to acquire information necessary for estimating key input with high accuracy.

Fig.1 (a) is a schematic diagram which shows an example of the estimation system in this embodiment, and FIG.1 (b) is a schematic diagram which shows the other example of the estimation system in this embodiment. FIG. 2 is a schematic diagram illustrating an example of the operation of the estimation system. FIG. 3 is a schematic diagram illustrating an example of an acquisition unit that acquires angle information and optical flow information. FIG. 4A is a schematic diagram illustrating an example of the configuration of the estimation device in the present embodiment, and FIG. 4B is a schematic diagram illustrating an example of the function of the estimation device in the present embodiment. FIG. 5 is a schematic diagram illustrating an example of a reference database in the present embodiment. FIG. 6 is a schematic diagram showing a first modification of the reference database in the present embodiment. FIG. 7 is a schematic diagram showing a second modification of the reference database in the present embodiment. FIG. 8 is a flowchart showing an example of the operation of the estimation system in the present embodiment.

  Hereinafter, an example of an estimation system and an estimation apparatus according to an embodiment to which the present invention is applied will be described with reference to the drawings.

  With reference to FIG. 1, an example of the estimation system 100 and the estimation apparatus 1 in this embodiment will be described.

  The estimation system 100 in the present embodiment includes an estimation device 1 as shown in FIG. The estimation device 1 is connected to the user terminal 3 via, for example, the communication network 4. The estimation system 100 estimates a character string input using the virtual keyboard 8. In the estimation system 100, the virtual keyboard 8 is projected by the user terminal 3, or the virtual keyboard 8 may be projected by the estimation device 1 as shown in FIG. 1B, for example, and the virtual keyboard 8 may be projected by another terminal or the like. 8 may be projected.

  In the estimation system 100, for example, as shown in FIG. 2, the estimation apparatus 1 acquires target information and generates an estimation result for the target information. The target information includes angle information related to the angle of the user's finger, and may include shooting information, for example. The shooting information includes image data obtained by shooting the operation of the user pressing the virtual keyboard 8 and position information of the user's hand in the image data. The target information may include, for example, optical flow information indicating the movement of the user's finger. In addition to being generated by the user terminal 3, the target information may be generated by the estimation device 1, for example.

  The image data may be generated using a known camera provided in the user terminal 3 or the estimation device 1, or may be generated using a known camera independent of the user terminal 3 or the estimation device 1. The image data includes an image obtained by imaging a user's hand pressing one key, and may include, for example, a plurality of images (for example, moving images) obtained by imaging the movement of the user's hand before and after pressing at least one key. The image data may be, for example, a moving image obtained by imaging the movement of the user's finger in addition to an image obtained by imaging the user's finger. The image data includes information about the keyboard (equivalent to keyboard information described later) such as the shape, projection size, key height, and key layout of the virtual keyboard 8.

  The image data includes 3D image data, for example, RGB image data and infrared image data. The three-dimensional image data is generated using a known three-dimensional camera (3D camera). A three-dimensional camera may be substituted by using a plurality of RGB cameras, for example. The RGB image data is generated using an RGB camera corresponding to a visible light region (for example, 400 nm to 800 nm). The infrared image data is generated using an infrared camera corresponding to an infrared light region (for example, 800 nm to 1,000 μm). The RGB image data and the infrared image data may be generated using, for example, one multispectral camera corresponding to the visible light region to the infrared light region.

  The position information is generated using, for example, a known distance measuring camera that measures the separation distance from the object. The position information may be generated using a distance measurement camera provided in the user terminal 3 or the estimation device 1 or may be generated using a distance measurement camera independent of the user terminal 3 or the estimation device 1.

  The position information includes distance data indicating the distance between the distance measurement camera and the user's hand (object) captured by the image data. The distance data indicates a distance between the distance measurement camera and at least one place of a user's hand set in advance. The distance data may indicate a distance between the distance measurement camera and each user's fingertip, for example. The user's fingertip indicates a portion assumed to contact the virtual keyboard 8, and for example, a total of 10 locations corresponding to the fingertip of the right hand and the fingertip of the left hand are targets of the distance data.

  The distance data is generated, for example, in the same period as when image data is captured. The distance data may be generated only during a period before and after the user presses the virtual keyboard 8, for example. The distance data is generated as numerical data and may be indicated by a vector or a matrix.

  The position information may include coordinate data, for example. The coordinate data is indicated (generated) by a coordinate space using, for example, the x-axis, y-axis, and z-axis based on the position of the distance measurement camera, and may be acquired from, for example, three-dimensional image data. Similarly to the distance data, the coordinate data indicates the coordinates of at least one location of the user's hand, for example, the coordinates for each fingertip of the user. The coordinate data is generated, for example, in the same period as when image data is captured. The coordinate data may be generated only during a period before and after the user presses the virtual keyboard 8, for example. The coordinate data is generated as numerical data and may be indicated by a vector or a matrix.

  The angle information is acquired based on three-dimensional image data, for example, as shown in FIG. The angle information is acquired from the coordinate data acquired from the three-dimensional image data, or may be acquired from the coordinate data generated by the distance measurement camera, for example. The angle information may be obtained, for example, by calculating an angle from three or more coordinates centered on the coordinates of the joint portion among the coordinates of each finger. A known technique can be used as a method for obtaining the angle information.

  The optical flow information is acquired based on RGB image data, for example, as shown in FIG. The optical flow information is acquired from a plurality of RGB images or moving images, or may be acquired from a plurality of coordinate data having different acquisition times, for example. The optical flow information indicates, for example, a change with time of the position at each fingertip. The optical flow information is acquired in an image format, or may be acquired in a matrix (vector) format, for example. A known technique can be used as a method for acquiring optical flow information.

  Note that the projection unit, the three-dimensional camera, the RGB camera, the infrared camera, and the distance measurement camera that project the virtual keyboard 8 described above may be provided in the same terminal (for example, the estimation device 1 or the user terminal 3). In addition, the above-described image data and position information may be generated using a known camera such as Kinect (registered trademark) or a multispectral camera. In this case, for example, a camera with two or more eyes may be used.

  The target information may include keyboard information, for example. The keyboard information indicates the type of the virtual keyboard 8 used by the user. The keyboard information includes, for example, the key layout, the size of each key, the key height, and the like in addition to the projection size of the virtual keyboard 8.

  The target information may include language information, for example. The language information indicates the type of language that the user inputs using the virtual keyboard 8. In the estimation system 100, an input character string may be estimated based on language information. In this case, it is possible to estimate the key input with high accuracy without depending on the language used by the user. In addition, it is possible to generate an estimation result in consideration of a typo or the like.

  The estimation result indicates a result of estimating the content corresponding to the input key when the user presses the virtual keyboard 8. As an estimation result, a character string is generated, and for example, a control command for the estimation device 1 or the user terminal 3 is generated. In the estimation system 100, the accuracy of the estimation result can be improved by increasing the types of information included in the target information. In particular, the combination of angle information and optical flow information makes it possible to capture the movement of the user's finger with high accuracy.

(Estimation device 1)
Next, an example of the estimation apparatus 1 in the present embodiment will be described with reference to FIG. FIG. 4A is a schematic diagram illustrating an example of a configuration of the estimation device 1 in the present embodiment, and FIG. 4B is a schematic diagram illustrating an example of a function of the estimation device 1 in the present embodiment.

  As the estimation apparatus 1, for example, an electronic device such as a personal computer (PC) is used, and for example, an electronic device such as a smartphone, a tablet terminal, a wearable terminal, an IoT (Internet of Things) device, Raspberry Pi (registered trademark), or the like. A single board computer may be used. As shown in FIG. 4A, for example, the estimation device 1 includes a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and a storage unit. 104 and I / Fs 105 to 107. Each component 101 to 107 is connected by an internal bus 110.

  The CPU 101 controls the estimation device 1 as a whole. The ROM 102 stores the operation code of the CPU 101. The RAM 103 is a work area used when the CPU 101 operates. The storage unit 104 stores various types of information such as shooting information and target information. As the storage unit 104, for example, an HDD (Hard Disk Drive), a data storage device such as an SSD (solid state drive) or a floppy disk is used. For example, the estimation device 1 may have a GPU (Graphics Processing Unit) (not shown). By having a GPU, it is possible to perform higher-speed arithmetic processing than usual.

  The I / F 105 is an interface for transmitting and receiving various types of information to and from the user terminal 3 and the server 2 via the communication network 4 such as the Internet. The I / F 106 is an interface for transmitting / receiving information to / from the input unit 108. For example, a keyboard is used as the input part 108, and the administrator of the estimation apparatus 1 inputs various information or a control command of the estimation apparatus 1 through the input part 108. The I / F 107 is an interface for performing transmission / reception of various information with the output unit 109. The output part 109 outputs various information stored in the storage unit 104 or the processing status of the estimation apparatus 1. A display is used as the output portion 109, and for example, a touch panel type may be used.

  For example, when the output portion 109 includes a projection unit that projects the virtual keyboard 8, the input portion 108 includes various cameras for generating target information (for example, the above-described three-dimensional camera, RGB camera, infrared camera, and distance). At least one of the measurement cameras). When a terminal such as a smartphone or a tablet is used as the estimation device 1, a known projection device (projection unit) provided is used as a part of the output portion 109, and a known camera (imaging unit) provided. May be used as part of the input portion 108.

<Reference database>
The storage unit 104 stores, for example, a reference database in which past target information, reference information, and association degrees acquired in advance are stored. The past target information indicates the same type of information as the target information described above and includes past angle information. For example, at least one of past optical flow information, past image data, past position information, and past keyboard information is included. May be included. The reference information includes a character string input according to past target information, and includes, for example, a control command of the estimation device 1 or the user terminal 3. The degree of association indicates the degree of relationship between past target information and reference information.

  In the reference database, for example, as shown in FIG. 5, three or more levels of association between past target information and reference information are stored. The degree of association is indicated by three or more levels such as percentage, 10 levels, or 5 levels, for example, by a line characteristic (for example, thickness). The reference database is formed by an algorithm that can calculate the association degree using, for example, past target information and reference information. The past target information and the reference information have a plurality of data, and the relationship between each past target information and each reference information is associated with the degree of association.

  For example, “Information A” included in the past target information indicates an association degree “80%” with “Reference A” included in the reference information, and between “Reference B” included in the reference information The degree of association is “15%”. That is, the “association degree” indicates the degree of connection between the data, and the higher the association degree, the stronger the connection between the data.

  The past target information is stored in the reference database in the form of an image (may be a plurality of images showing a change over time or a moving image) or a character string, or in the form of a numerical value, a matrix (vector), or a histogram, for example. May be stored. The reference information is stored in the reference database in the form of a character string or the like, and may be stored in the form of a numerical value, a matrix (vector), or a histogram, for example. That is, “Information A”, “Information B”, and “Information C” shown in FIG. 5 are object information acquired in the past (for example, in addition to angle information, optical flow information, RGB image data, infrared image data, distance, Data, coordinate data, and / or keyboard information) indicates data stored in an image format, a matrix, or the like. “Reference A”, “Reference B”, and “Reference C” are acquired in the past. The data stored in the form of a character string or a matrix corresponding to the estimation result is shown.

  The reference database is constructed using machine learning, for example. In this case, the association degree is calculated using machine learning. For example, deep learning is used for machine learning. The reference database is constructed by, for example, a neural network, in which case the association degree may be indicated by a hidden layer and a weight variable. That is, in the estimation system 100, a reference database constructed by machine learning can be used. In this case, a complicated relationship between past target information each having a plurality of data and reference information is represented as an association degree. Can be shown with high accuracy. As a result, the accuracy of the estimation result can be improved.

  The past target information may be stored in the reference database by dividing past angle information and past optical flow information, for example, as shown in FIG. In this case, the association degree is calculated based on the relationship between the combination of the past angle information and the past optical flow information and the reference information. The past target information may be stored in the reference database by dividing at least one of past image data and past position information in addition to the above, for example.

  For example, the combination of “angle A” included in the past angle information and “flow A” included in the past optical flow information indicates an association degree “90%” with “reference A”. "20%" is shown in the relationship with "B". In this case, the past angle information and the past optical flow information can be stored independently of each other. For this reason, when generating an estimation result, it becomes possible to improve accuracy and expand the range of options.

  The past image data may be stored in the reference database by dividing past RGB image data and past infrared image data, for example. The past position information may be stored in the reference database by dividing past distance data and past coordinate data, for example.

  The past target information may include composite data and similarity as shown in FIG. 7, for example. The composite data is indicated by three or more degrees of similarity with the past angle information or the past optical flow information. The composite data is stored in the reference database in the form of an image, a character string, or the like, or may be stored in the form of a numerical value, a matrix, or a histogram, for example.

  FIG. 4B is a schematic diagram illustrating an example of the function of the estimation device 1. The estimation apparatus 1 includes an acquisition unit 11, a storage unit 12, an evaluation unit 13, a generation unit 14, and an output unit 15, and may include an update unit 16, for example. Note that each function shown in FIG. 4B is realized by the CPU 101 executing a program stored in the storage unit 104 or the like using the RAM 103 as a work area, and may be controlled by, for example, artificial intelligence.

<Acquisition unit 11>
The acquisition unit 11 acquires target information. The acquisition unit 11 acquires target information from an external terminal such as the user terminal 3 via the communication network 4 and the I / F 105, and for example, an imaging unit (for example, the above-described three-dimensional camera, RGB camera, Target information is acquired from an infrared camera and / or a distance measurement camera. The acquisition unit 11 may acquire shooting information and acquire angle information based on the shooting information as target information, or may acquire angle information or the like as target information in a lump. The frequency and period at which the acquisition unit 11 acquires the target information is arbitrary.

  For example, the acquisition unit 11 acquires coordinate data from the acquired imaging information, and acquires angle information from the coordinate data. Thereby, the acquisition unit 11 acquires target information including angle information. For example, the acquisition unit 11 acquires optical flow information from a plurality of acquired RGB images. Thereby, the acquisition unit 11 acquires target information including angle information and optical flow information. In addition, the acquisition part 11 can acquire angle information and optical flow information using the well-known technique mentioned above.

  The acquisition unit 11 receives various types of information transmitted to the estimation device 1. The acquisition unit 11 receives various types of information such as target information transmitted from an external terminal such as the user terminal 3 via the communication network 4 and the I / F 105.

<Storage unit 12>
The storage unit 12 retrieves various information such as a reference database stored in the storage unit 104 as necessary. The storage unit 12 stores various information acquired or generated by the configurations 11 and 13 to 16 in the storage unit 104.

<Evaluation unit 13>
The evaluation unit 13 refers to the reference database, and acquires evaluation information including the first association degree between the target information and the reference information. For example, when the evaluation unit 13 refers to the reference database illustrated in FIG. 4, the past target information (for example, “information A”: the past first target information) that is the same as or similar to the information included in the target information is used. select. As past target information, information that partially matches or completely matches target information is selected, and information that is similar (including the same concept or the like) is selected, for example. When the target information is represented by a numerical value such as a matrix, a numerical range included in the past target information to be selected may be set in advance.

  The evaluation unit 13 selects reference information associated with the selected past first target information, and the degree of association (first degree of association) between the selected past target information and the reference information, and acquires it as evaluation information. . The first association degree may be calculated by the evaluation unit 13 in addition to a part selected from the association degree.

  For example, the evaluation unit 13 selects the reference information “reference A” associated with the first target information “information A”, and the first association degree “80%” between “information A” and “reference A”, Obtained as evaluation information. The reference information and the first association degree may include a plurality of data. In this case, in addition to the above-mentioned “reference A” and “80%”, the reference information “reference B” associated with the first target information “information A”, and between “information A” and “reference B” The first association degree “15%” may be selected, and “reference A” and “80%”, and “reference B” and “15%” may be acquired as evaluation information.

  The evaluation information may include target information. In addition, the first relevance is indicated in three or more levels such as a percentage. For example, when the reference database is configured by a neural network, the first relevance indicates a weight variable associated with the selected past evaluation target information.

<Generator 14>
The generation unit 14 generates an estimation result based on the evaluation information. The generation unit 14 uses format data such as an output format stored in advance in the storage unit 104 or the like, for example, and converts it into a format that can be understood by the user (for example, a character string) based on information acquired as an evaluation result. The converted information is generated as an evaluation result.

  Generator 14, for example, as shown in FIG. 2, to generate estimation result containing the string to be estimated such as "○○'s Hello.". The generation unit 14 may generate an estimation result including control information for controlling the estimation device 1 or the user terminal 3, for example. For example, a known technique (for example, artificial intelligence technique) may be used for setting the format when generating the estimation result and for converting the evaluation information into the estimation result.

  The generation unit 14 determines the content of the estimation result based on, for example, the first association degree of the evaluation information. For example, the generation unit 14 generates an estimation result based on the reference information associated with the first association degree of “50%” or more, and uses the reference information associated with the first association degree of less than “50%” as the estimation result. You may set so that it may not reflect. For example, the administrator or the like may set a threshold value or the like in advance, and the threshold range or the like can be arbitrarily set. Moreover, the production | generation part 14 may determine the content of an estimation result, for example based on the result of computing 2 or more 1st association degrees, or the comparison of 2 or more 1st association degrees.

  The estimation result may include, for example, evaluation information. The estimation result may include target information.

<Output unit 15>
The output unit 15 outputs an estimation result. The output unit 15 transmits the estimation result to the output part 109 via the I / F 107, and also transmits the estimation result to the user terminal 3 or the like via the I / F 105, for example.

  The output unit 15 may determine whether or not to transmit the estimation result to the user terminal 3 based on the estimation result, for example. For example, the administrator may set a threshold value or the like in advance.

<Update unit 16>
For example, the update unit 16 updates the reference database. When the update unit 16 newly acquires a relationship between past target information and reference information, the update unit 16 reflects the relationship in the association degree. For example, when the user evaluates the estimation accuracy of keyboard input based on the estimation result generated by the generation unit 14 and the estimation device 1 acquires the evaluation result, the update unit 16 is included in the reference database based on the evaluation result. Update relevance.

<Server 2>
The server 2 stores, for example, the various information described above. In the server 2, for example, various types of information transmitted via the communication network 4 are accumulated. The server 2 may store information similar to that of the storage unit 104, for example, and may transmit and receive various types of information to and from the estimation device 1 via the communication network 4. That is, the estimation apparatus 1 may use the server 2 instead of the storage unit 104.

<User terminal 3>
The user terminal 3 includes a projection unit that projects the above-described virtual keyboard 8 and a camera that generates target information. A mobile phone (mobile terminal) is mainly used as the user terminal 3, and other than that, it is embodied in all electronic devices other than electronic devices such as smartphones, tablet terminals, wearable terminals, personal computers, and IoT devices. May be used. The user terminal 3 may be directly connected to the estimation device 1, for example, in addition to being connected to the estimation device 1 via the communication network 4, for example. A user acquires an estimation result from the estimation apparatus 1 using the user terminal 3, for example. Note that the user terminal 3 may include a projection unit and a plurality of cameras provided independently of each other.

<Communication network 4>
The communication network 4 is, for example, an Internet network to which the estimation device 1 or the like is connected via a communication circuit. The communication network 4 may be a so-called optical fiber communication network. The communication network 4 may be realized by a known communication network such as a wireless communication network in addition to a wired communication network.

(Example of operation of estimation system 100)
Next, an example of operation | movement of the estimation system 100 in this embodiment is demonstrated. FIG. 8 is a flowchart showing an example of the operation of the estimation system 100 in the present embodiment.

<Acquisition means S110>
As shown in FIG. 8, target information including angle information is acquired (acquisition means S110). The acquisition unit 11 acquires target information including angle information based on shooting information. The virtual keyboard 8 is projected by, for example, the projection unit of the user terminal 3 or the estimation device 1. The shooting information is generated by various cameras included in the user terminal 3 or the estimation device 1, for example.

  The acquisition unit 11 acquires shooting information and acquires angle information from the shooting information. Thereby, object information including angle information is acquired. For example, the acquisition unit 11 stores the target information in the storage unit 104 via the storage unit 12. The acquisition unit 11 may acquire image data or the like after acquiring in advance at least one of keyboard information and language information.

  The acquisition unit 11 may acquire target information generated in an arbitrary period, for example, in addition to acquiring target information every time shooting information is generated.

  For example, as illustrated in FIG. 3, the acquisition unit S110 includes a photographing information acquisition unit S111, a coordinate data acquisition unit S112, and an angle information acquisition unit S113. In the shooting information acquisition unit S111, the acquisition unit 11 acquires shooting information and acquires, for example, 3D image data generated using a 3D camera. In the coordinate data acquisition unit S112, the acquisition unit 11 may acquire coordinate data generated by a distance measurement camera, for example, in addition to acquiring coordinate data from the photographing information. In the angle information acquisition unit S113, the acquisition unit 11 acquires angle information from the coordinate data. Regarding the target for obtaining the angle information, an angle with the first joint of the finger as the apex, an angle with the second joint as the apex, or the like is used, and can be arbitrarily set according to the situation.

  The acquisition unit S110 includes, for example, an optical flow information acquisition unit S114. In the optical flow information acquisition unit S114, the acquisition unit 11 acquires an RGB image generated using an RGB camera, and acquires optical flow information from a plurality of RGB images.

<Evaluation means S120>
Next, with reference to the reference database, evaluation information including the first association degree between the target information and the reference information is acquired (evaluation means S120). The evaluation unit 13 acquires the target information acquired by the acquisition unit 11, and acquires, for example, a reference database stored in the storage unit 104. In addition to acquiring one piece of evaluation information for one piece of target information, the evaluation unit 13 may obtain one piece of evaluation information for a plurality of pieces of target information, for example. For example, the evaluation unit 13 stores the evaluation information in the storage unit 104 via the storage unit 12.

<Generating means S130>
Next, an estimation result is generated based on the evaluation information (generation unit S130). The generation unit 14 acquires the evaluation information acquired by the evaluation unit 13 and acquires format data such as an output format stored in the storage unit 104, for example. The generation unit 14 may acquire one estimation result for a plurality of evaluation information, for example, in addition to acquiring one estimation result for one evaluation information. The generation unit 14 stores the estimation result in the storage unit 104 via the storage unit 12, for example.

  Thereby, for example, the output unit 15 transmits the estimation result to the user terminal 3 or the like, or outputs the estimation result to the output part 109, and the operation of the estimation system 100 in the present embodiment ends.

<Update means S140>
For example, when a relationship between past target information and reference information is newly acquired, the relationship may be reflected in the association degree (update unit S140). For example, when the user evaluates the estimation accuracy of keyboard input based on the estimation result generated by the generation unit 14 and the estimation device 1 acquires the evaluation result, the update unit 16 is included in the reference database based on the evaluation result. Update relevance.

  Thereby, operation | movement of the estimation system 100 in this embodiment may be complete | finished. In addition, the timing in the case of implementing update means S140 is arbitrary.

  According to the present embodiment, the evaluation unit S120 refers to the reference database, and acquires evaluation information including the first association degree of three or more stages between the target information and the reference information. For this reason, it is possible to generate an estimation result based on the past result, and it is possible to estimate a key input in consideration of variations in the pressing depth. Thereby, it is possible to improve the accuracy of estimating the key input.

  Further, according to the present embodiment, the target information includes angle information. For this reason, compared with the case where the position of the hand is calculated from one image, the positional relationship of the hand with respect to the virtual keyboard 8 can be easily obtained regardless of the characteristics of the hand that are different for each user. This makes it possible to acquire information necessary for estimating key input with high accuracy.

  According to the present embodiment, the coordinate data acquisition unit S112 acquires coordinate data from the shooting information. Moreover, angle information acquisition means S113 acquires angle information from coordinate data. For this reason, the capacity of data to be evaluated can be reduced. This makes it possible to suppress an increase in storage capacity when storing target information.

  Further, according to the present embodiment, the shooting information includes 3D image data generated using a 3D camera. For this reason, when acquiring angle information, a conventional three-dimensional camera can be used without providing a new camera or the like. Thereby, it is possible to easily prepare an environment for estimating key input.

  Further, according to the present embodiment, the target information and the past target information include optical flow information. For this reason, by combining the optical flow information and the angle information, it is possible to consider the movement at the time of key input for each user. Thereby, it is possible to further improve the accuracy of estimating the key input.

  Further, according to the present embodiment, the acquisition unit S110 acquires optical flow information from a plurality of RGB images. For this reason, when acquiring optical flow information, a conventional RGB camera can be used without providing a new camera or the like. This makes it possible to easily prepare an environment for improving the accuracy of key input estimation.

  Further, according to the present embodiment, the target information and past target information include keyboard information. For this reason, even when the virtual keyboard 8 having a different key arrangement or the like is used, the key input can be estimated with high accuracy.

  Further, according to the present embodiment, the reference database is constructed by machine learning. For this reason, it is possible to realize key input estimation excluding subjectivity as compared with the case of setting a threshold for the pressing depth calculated from the image.

  According to the present embodiment, the evaluation unit 13 refers to the reference database, and acquires evaluation information including the first association degree of three or more stages between the target information and the reference information. For this reason, it is possible to generate an estimation result based on the past result, and it is possible to estimate a key input in consideration of variations in the pressing depth. Thereby, it is possible to improve the accuracy of estimating the key input.

  Further, according to the present embodiment, the target information includes angle information. For this reason, compared with the case where the position of the hand is calculated from one image, the positional relationship of the hand with respect to the virtual keyboard 8 can be easily obtained regardless of the characteristics of the hand that are different for each user. This makes it possible to acquire information necessary for estimating key input with high accuracy.

  Note that according to the present embodiment, the target information and the past target information may include language information. In this case, it is possible to estimate the key input with high accuracy without depending on the language used by the user.

  Note that, according to the present embodiment, when the relationship between the past target information and the reference information is newly acquired, the updating unit S140 may reflect the relationship in the association degree. In this case, the association degree can be easily updated, and it is possible to continuously improve the estimation accuracy.

  Note that according to the present embodiment, the image data may include RGB image data and infrared image data. In this case, even when the imaging environment caused by illumination or the like is different, the shape of the hand or the like can be acquired with high accuracy. Thereby, it is possible to further improve the accuracy of estimating the key input.

  Note that according to the present embodiment, the position information may include distance data. In this case, the position of the hand with respect to the depth direction can be easily acquired. Thereby, it is possible to further improve the accuracy of estimating the key input.

  Note that according to the present embodiment, the projection unit, the RGB camera, the infrared camera, and the distance measurement camera may be provided in the same terminal. In this case, the virtual keyboard 8 can be used without selecting a place.

  Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

1: estimation device 2: server 3: user terminal 4: communication network 8: virtual keyboard 10: housing 11: acquisition unit 12: storage unit 13: evaluation unit 14: generation unit 15: output unit 16: update unit 100: estimation System 101: CPU
102: ROM
103: RAM
104: Storage unit 105: I / F
106: I / F
107: I / F
108: input part 109: output part 110: internal bus S110: acquisition means S120: evaluation means S130: generation means S140: update means

Claims (7)

An estimation system for estimating a character string input using a virtual keyboard,
Acquisition means for acquiring target information including angle information related to an angle of the user's finger based on shooting information obtained by shooting an operation of the user pressing the virtual keyboard;
A reference database in which past degrees of target information acquired in advance and reference information including character strings input by the past target information are stored in three or more levels of association;
An evaluation unit that refers to the reference database and acquires evaluation information including the first association degree in three or more stages between the target information acquired by the acquisition unit and the reference information;
Generating means for generating an estimation result based on the evaluation information;
Equipped with a,
The angle of the finger indicates an angle calculated from coordinates of three or more points centered on coordinates in a joint portion among the coordinates of the finger of the user . The acquisition means includes
Shooting information acquisition means for acquiring the shooting information;
Coordinate data acquisition means for acquiring coordinate data of the user's finger from the imaging information;
Angle information acquisition means for acquiring the angle information from the coordinate data;
The estimation system according to claim 1, further comprising: The imaging information includes 3D image data generated using a 3D camera,
The estimation system according to claim 2, wherein the coordinate data acquisition unit acquires the coordinate data from the three-dimensional image data. The estimation system according to claim 1, wherein the target information and the past target information include optical flow information indicating movement of the user's finger. The target information and the historical target information, the estimation system of claim 1 or one of claims 4, characterized in that it comprises a keyboard information indicating the type of the virtual keyboard. The reference database may estimation system of any one of claims 1-5, characterized in that those constructed by machine learning. An estimation device for estimating a character string input using a virtual keyboard,
An acquisition unit that acquires target information including angle information related to an angle of the user's finger based on shooting information obtained by shooting an operation of the user pressing the virtual keyboard;
A reference database in which past degrees of target information acquired in advance and reference information including character strings input by the past target information are stored in three or more levels of association;
An evaluation unit that refers to the reference database and acquires evaluation information including the first association degree in three or more stages between the target information acquired by the acquisition unit and the reference information;
A generating unit that generates an estimation result based on the evaluation information;
Equipped with a,
The angle of the finger indicates an angle calculated from three or more coordinates centered on a coordinate in a joint portion of the user's finger coordinates .

Images