JP6367031B2 - Electronic device remote control system and program - Google Patents

Description

  The present invention relates to a natural user interface, and more particularly to a system for remotely operating an electronic device based on a user's handwriting action.

  In recent years, various studies have been made on the natural user interface (NUI). NUI is a method for manipulating an electronic device by an action naturally performed by a human. Until now, an operation by a gesture (for example, Patent Document 1) or an operation by speech (voice) (for example, Patent Document 2). Consideration has been made.

  However, these NUIs that use gestures and utterances are somewhat questionable from the viewpoint of usability. This is because many people feel reluctant to shake hands or talk to electrical appliances, and learning specific gestures and utterances that are unfamiliar from the beginning is a heavy burden on users. Because it becomes.

JP 2009-230460 A International Publication No. 2003/052737

  The present invention has been made in view of the above-described problems in the prior art, and an object of the present invention is to provide a new electronic device remote operation system with high usability.

  As a result of intensive studies on the configuration of the electronic device remote control system with high usability, the present inventor has conceived the following configuration and arrived at the present invention.

  That is, according to the present invention, the handwriting input accepting means for accepting a handwritten locus as an input, the handwritten drawing analyzing means for analyzing the content drawn by the user from the handwritten locus, and the content drawn by the user in a predetermined notation An operation request specifying means for specifying the operation content of the electronic device requested by the user, and an operation code generating means for generating an operation code of the electronic device corresponding to the specified operation content. An operating system is provided.

  As described above, according to the present invention, a novel electronic device remote operation system with high usability is provided.

The functional block diagram of the electronic device remote control system of this embodiment. The figure which shows the electronic device remotely operated by writing. The figure which shows a mode that the electronic device is operated remotely by writing in time series. The figure which shows a mode that the electronic device is operated remotely by writing in time series. The figure which shows a mode that the electronic device is operated remotely by writing in time series. The figure which shows a mode that the electronic device is operated remotely by writing in time series. The figure which shows a mode that the electronic device is operated remotely by writing in time series. The figure which shows a mode that the electronic device is operated remotely by writing in time series.

  Hereinafter, the present invention will be described with reference to embodiments shown in the drawings, but the present invention is not limited to the embodiments shown in the drawings. In the drawings referred to below, the same reference numerals are used for common elements, and the description thereof is omitted as appropriate.

  An object of the present invention is to realize a natural user interface (NUI) for remotely operating an electronic device based on a user's handwriting action. Currently, touchpads, touch panels, pen tablets, digital pens, and the like have been commercialized as means for acquiring a user's handwritten trajectory as digital data, but digital pens are suitable for use in the present invention. This is because the digital pen is first recognized as a writing instrument by the user, so it is easy to embrace a remote control-less sensation, and it is easy for the user to accept the action of writing on paper as the most familiar thing. . Therefore, in the following, an embodiment using a digital pen will be described.

  FIG. 1 is a functional block diagram of an electronic device remote control system 100 according to an embodiment of the present invention. The electronic device remote operation system 100 of the present embodiment includes a handwriting trajectory acquisition unit including the digital pen 12 (hereinafter collectively referred to as the digital pen 12), an information processing device 50, and an operation signal output device 60. It is configured to include.

  The digital pen 12 has a function of acquiring a user's handwritten locus as digital data. Currently, as a method for acquiring a user's handwritten trajectory as digital data, a pen tip trajectory coordinate is acquired by sensing a weak radio wave emitted from the pen tip with a tablet-type sensor laid in a paper pad ( Electromagnetic induction method), ultrasonic wave emitted from the pen nib is received by two sensors and the trajectory coordinates of the nib are calculated based on the triangulation principle (ultrasonic wave method), printed on paper A method (Anoto pattern method) of acquiring a trace coordinate of the pen tip from the pattern by reading a fine pattern (Anoto pattern) with a CCD camera of the pen tip has been put into practical use. Any method may be adopted.

  The information processing apparatus 50 is referred to as a computer, and a user draws a handwriting input accepting unit 10 that accepts a handwritten locus acquired by the digital pen 12 as an input by installing a predetermined program. The handwritten drawing analysis unit 20 that analyzes the content that has been performed, the operation request specification unit 30 that specifies the operation content of the electronic device requested by the user, and the operation code generation unit 40 that generates the operation code of the electronic device corresponding to the specified operation content Function as.

  The operation signal output device 60 generates an operation signal based on the operation code input from the operation code generation unit 40 of the information processing device 50, and the operation signal is transmitted to a communication medium (electronic, (Radio wave, sound wave, light, etc.) Here, the operation signal output device 60 is provided with a plurality of signal output means corresponding to the type in the case where each of the plurality of electronic devices to be operated uses different communication media.

  As described above, the configuration of the electronic device remote operation system 100 according to the present embodiment has been outlined. However, when using this system, the user digitally displays the content of the electronic device operation requested by the user based on a predetermined notation. Draw on paper 14 with pen 12. In response to this, in the information processing device 50, the handwritten drawing analysis unit 20 analyzes the user's drawing content, and the operation request specifying unit 30 illuminates the analysis result in accordance with a predetermined notation to request the operation content of the electronic device requested by the user. Is identified.

  Note that the above-described “notation” is desirably as simple and intuitive as possible when considering usability as a NUI. Here, in the following, one notation that can be adopted by the present invention is proposed. It expresses the target “electronic device” and “operation instruction” with a combination of natural words (characters) and simple figures that mean itself, and the correspondence between “electronic device” and “operation instruction” Is a notation characterized by defining it with a simple diagram. In the following, in describing a specific usage scene of the electronic device remote control system 100 adopting the present notation, the contents of the present notation and the contents of the processes executed by the respective functional units will be clarified. . In the following description, FIG. 1 will be referred to as appropriate.

  In this usage scene, it is assumed that the following five operations are performed on the table lamp 72 and the liquid crystal television 74 from the initial state shown in FIG.

(Operation 1) The table lamp 72 is turned on.
(Operation 2) The table light 72 is turned off.
(Operation 3) The table light 72 and the television 74 are turned on simultaneously.
(Operation 4) The channel of the liquid crystal television 74 is changed.
(Operation 5) The liquid crystal television 74 is turned off.

(Operation 1)
First, as shown in FIG. 3A, the user draws “Light” on the paper 14 with the digital pen 12. On the other hand, the character recognition unit 22 of the handwriting drawing analysis unit 20 is configured to always refer to the machine learning discriminator 26 for the handwriting trajectory input to the handwriting input reception unit 10 and perform character recognition. . Here, in the machine learning discriminator 26, the learned data of the locus in which the character “Light” meaning the table light 72 that is the operation target device is handwritten is registered in advance. In this case, the character recognition unit 22 Then, it is recognized that the character string “Light” is drawn on the paper 14.

  Subsequently, as shown in FIG. 3B, the user draws “rectangle j” so as to surround “Light” with the digital pen 12. On the other hand, the graphic recognition unit 24 of the handwritten drawing analysis unit 20 is configured to always refer to the machine learning discriminator 26 and perform graphic recognition on the handwritten trajectory input to the handwriting input reception unit 10. . Here, in the machine learning discriminator 26, the learned data of the locus in which the “rectangle” that is the first graphic for defining the operation target device is handwritten is registered in advance. In this case, the graphic recognition unit 24 recognizes that “square” is drawn on the paper 14.

  On the other hand, the operation target device specifying unit 32 of the operation request specifying unit 30 includes a character drawing area (coordinate information) recognized by the character recognizing unit 22 and a first graphic drawing area (coordinate information) recognized by the graphic recognizing unit 24. Are always compared to determine whether the character is surrounded by the first graphic, and if it is determined that the character is surrounded by the first graphic, The character is associated with the first graphic (specifically, coordinate information of the area occupied by the first graphic) as identification information of the operation target device and registered in the operation registration unit 38.

  In the case shown in FIG. 3B, the operation target device specifying unit 32 determines that the character string “Light” is surrounded by “rectangle j”, and uses the character string “Light” as identification information of the operation target device. The information is registered in the operation registration unit 38 in association with “rectangle j” (specifically, coordinate information of an area occupied by “rectangle j”).

  Subsequently, the user draws “on” with the digital pen 12 as shown in FIG. In response to this, the character recognition unit 22 performs character recognition on the handwritten trajectory input to the handwriting input reception unit 10. Here, in the machine learning discriminator 26, the learned data of the locus in which the character “on” meaning the operation command “power ON” is handwritten is registered in advance. 14 recognizes that the character string “on” is drawn.

  Subsequently, as shown in FIG. 4D, the user draws “circular k” so as to surround “on” with the digital pen 12. In response to this, the graphic recognition unit 24 of the handwritten drawing analysis unit 20 inquires of the machine learning discriminator 26 for the handwritten trajectory input to the handwriting input reception unit 10 and performs graphic recognition. Here, in the machine learning discriminator 26, the learned data of the locus in which the “circle” that is the second graphic for defining the operation command is handwritten is registered in advance. In this case, the graphic recognition unit 24 is registered. Recognizes that “circular” is drawn on the paper 14.

  On the other hand, the operation command specifying unit 34 of the operation request specifying unit 30 displays the drawing region (coordinate information) of the character recognized by the character recognizing unit 22 and the drawing region (coordinate information) of the second graphic recognized by the graphic recognizing unit 24. It is configured to determine whether or not the character is surrounded by the second graphic, and when it is determined that the character is surrounded by the second graphic, Is associated with a second graphic (specifically, coordinate information of an area occupied by the second graphic) as identification information of the operation command and registered in the operation registration unit 38.

  In the case shown in FIG. 4D, the operation command specifying unit 34 determines that the character string “on” is surrounded by “circle k”, and uses the character string “on” as the identification information of the operation command “circle”. k ”(specifically, coordinate information of an area occupied by“ circular k ”) is registered in the operation registration unit 38.

  Subsequently, as shown in FIG. 4E, the user draws a line l with the digital pen 12 so as to connect “circle k” and “square j”. Here, the associating unit 36 of the operation request specifying unit 30 is the first graphic in which the graphic recognition unit 24 always recognizes the coordinates of the start point and the coordinates of the end point of the handwritten trajectory input to the handwriting input receiving unit 10. And the second figure occupies the first figure and the second figure in the coordinates of the start point and the end point of the handwritten trajectory. If it is determined that it is within the area, the identification information of the operation target device associated with the first graphic registered in the operation registration unit 38 is associated with the identification information of the operation command associated with the second graphic. Has been.

  In the case shown in FIG. 4E, the associating unit 36 determines that the coordinates of the start point of the line l are in the area occupied by “circular k” and the coordinates of the end point are in the area occupied by “rectangle j”. Then, the identification information “Light” of the operation target device associated with “rectangle j” registered in the operation registration unit 38 is associated with the identification information “on” of the operation command associated with “circular k”.

  On the other hand, the trigger operation detection unit 42 of the operation code generation unit 40 is configured to detect a predetermined trigger operation by the user based on a handwritten trajectory input to the handwriting input reception unit 10. The trigger operation here is an arbitrary operation meaning a start request for an operation, and is defined in advance. In this case, the description will be continued assuming that “tap” is defined as the trigger operation.

  Subsequently, as shown in FIG. 4F, the user taps an area occupied by “circular k” surrounding “on” with the digital pen 12. In response to this, the trigger operation detection unit 42 detects a tap. In the present embodiment, for example, a tap can be detected when the distance between the start point and the end point of a handwritten locus is below a predetermined threshold value and below a predetermined threshold value of the drawing time.

  In response to this determination, the operation code generation unit 40 searches the operation registration unit 38 using the tapped position (coordinates) as a key, and performs an operation associated with the second graphic including the tapped position (coordinates). The identification information of the command and the identification information of the operation target device associated with the identification information are extracted. In the case of the example shown in FIG. 4 (f), the operation code generation unit 40 includes the identification information “on” associated with the “circular k” including the tapped position (coordinates) and the identification information “on” associated therewith. The identification information “Light” of the operation target device is extracted from the operation registration unit 38.

  Subsequently, the operation code generation unit 40 inputs the extracted identification information “Light” and “on” to the operation code conversion unit 44. Here, the operation code conversion unit 44 receives the identification information related to the electronic device and the identification information related to the operation command as inputs, and is referred to as a means for outputting an operation code representing the operation command for the electronic device. For example, it can be configured as a table in which identification information and operation instructions are associated with each other. In this case, the operation code conversion unit 44 returns, as an output, an operation code related to power-on of the table lamp 72 in response to the input of the identification information (Light, on). In response to this, the operation code generation unit 40 outputs an operation code related to power-on of the table lamp 72 to the operation signal output device 60.

  The operation signal output device 60 generates an operation signal based on an operation code related to power-on of the table lamp 72 input from the operation code generation unit 40, and places the operation signal on a communication medium corresponding to the table lamp 72. Output.

  For example, when the desk lamp 72 is an infrared remote controller compatible device, the operation signal output device 60 outputs an operation signal by placing it on infrared rays from an installed infrared light emitter. In addition, when the table lamp 72 is a Bluetooth (registered trademark) standard wireless remote control compatible device, the operation signal output device 60 outputs an operation signal by placing it on a radio wave from an installed radio wave transmitter. In response to this, the table lamp 72 is turned on as shown in FIG.

(Operation 2)
As shown in FIG. 5A, the user draws the handwritten character “off” with the digital pen 12, and then draws “circular m” so as to surround it. In response to this, the character recognition unit 22 of the handwriting drawing analysis unit 20 inquires of the machine learning discriminator 26 for the handwritten trajectory input to the handwriting input reception unit 10 and performs character recognition, and the character string on the paper 14. Recognize that “off” is drawn. Further, the graphic recognition unit 24 of the handwritten drawing analysis unit 20 inquires the machine learning discriminator 26 for the handwritten trajectory input to the handwriting input reception unit 10 to perform graphic recognition, and “circle” is drawn on the paper 14. Recognize that

  At this time, the operation command specifying unit 34 of the operation request specifying unit 30 determines that the character string “off” is surrounded by “circle m”, and uses the character string “off” as “circular m” as identification information of the operation command. "(Specifically, coordinate information of an area occupied by" circular m ") is registered in the operation registration unit 38.

  Subsequently, as shown in FIG. 5B, the user draws a line n with the digital pen 12 so as to connect “circle m” and “square j”. In response, the associating unit 36 identifies the operation command identification information “off” associated with “circle m” registered in the operation registration unit 38 and the operation target device identification information “Light” associated with “rectangle j”. ”

  Subsequently, as shown in FIG. 5C, the user taps the area occupied by the “circular m” surrounding the character string “off” with the digital pen 12. In response to the trigger operation detection unit 42 detecting the tap, the operation code generation unit 40 searches the operation registration unit 38 using the tapped position (coordinate) as a key, and finds the tapped position (coordinate). The operation command identification information “off” associated with the “circle m” to be included and the operation target device identification information “Light” associated therewith are extracted.

  Subsequently, the operation code generation unit 40 inputs the extracted identification information “Light” and “off” to the operation code conversion unit 44, receives an operation code related to the power-off of the table lamp 72 as an output, and operates this The signal is output to the signal output device 60. The operation signal output device 60 generates and outputs an operation signal based on the operation code input from the operation code generation unit 40. In response to this, the table lamp 72 is turned off as shown in FIG.

(Operation 3)
As shown in FIG. 6A, the user draws “TV” with the digital pen 12, and then draws a “square p” so as to surround it. In response to this, the character recognition unit 22 of the handwriting drawing analysis unit 20 inquires of the machine learning discriminator 26 for the handwritten trajectory input to the handwriting input reception unit 10 and performs character recognition, and the character string on the paper 14. Recognize that “TV” is drawn. The graphic recognition unit 24 of the handwriting drawing analysis unit 20 inquires the machine learning discriminator 26 for the handwritten trajectory input to the handwriting input reception unit 10 to perform graphic recognition, and “rectangle” is drawn on the paper 14. Recognize that

  At this time, the operation command specifying unit 34 of the operation request specifying unit 30 determines that the character string “TV” is surrounded by “rectangle p”, and uses the character string “TV” as identification information of the operation target device as “rectangle”. p ”(specifically, coordinate information of an area occupied by“ square p ”) is registered in the operation registration unit 38.

  Subsequently, as shown in FIG. 6B, the user draws a line q so as to connect “rectangle p” and “circle k” with the digital pen 12. In response to this, the associating unit 36 identifies the operation instruction identification information “on” associated with “circular k” registered in the operation registration unit 38 and the identification information “TV” of the operation target device associated with “square p”. ”

  Subsequently, as shown in FIG. 6C, the user taps the area occupied by the “circle k” surrounding the character string “on” with the digital pen 12. In response to the trigger operation detection unit 42 detecting the tap, the operation code generation unit 40 searches the operation registration unit 38 using the tapped position (coordinate) as a key, and finds the tapped position (coordinate). Two pieces of identification information associated with the identification information “on” of the operation command associated with the “circle k” to be included, that is, identification information “Light” of the operation target device and identification information “TV” of the operation target device are extracted. .

  Subsequently, the operation code generation unit 40 inputs the identification information “Light”, “TV”, “on” to the operation code conversion unit 44, and outputs the operation code related to the power ONN of the table lamp 72 and the liquid crystal television 74. The operation code related to the power ON is received and output to the operation signal output device 60. The operation signal output device 60 generates and outputs two operation signals based on the two operation codes input from the operation code generation unit 40. In response to this, as shown in FIG. 2 (d), the table lamp 72 and the liquid crystal television 74 are turned on simultaneously.

(Operation 4)
As shown in FIG. 6C, the user draws “6” inside the “square p” with the digital pen 12, and then draws “circular r” so as to surround it. In response to this, the character recognition unit 22 of the handwritten drawing analysis unit 20 inquires of the machine learning discriminator 26 for the handwritten locus input to the handwriting input reception unit 10 and performs character recognition. Recognize that 6 ″ is drawn.

  At this time, the operation command specifying unit 34 of the operation request specifying unit 30 determines that the number “6” is surrounded by “circular r”, and uses the number “6” as “circular r” ( Specifically, it is registered in the operation registration unit 38 in association with the coordinate information of the area occupied by “circular r”.

  Here, the association unit 36 of the operation request specifying unit 30 is configured to always determine whether or not the second graphic recognized by the graphic recognition unit 24 is surrounded by the first graphic. If it is determined that the second graphic is surrounded by the first graphic, the identification information of the operation target device associated with the first graphic registered in the operation registration unit 38 and the operation command associated with the second graphic Is configured to associate the identification information.

  In the case illustrated in FIG. 6C, the associating unit 36 determines that “circular r” is surrounded by “quadrangle p”, and the operation associated with “circular r” registered in the operation registering unit 38. The identification information “6” of the command is associated with the identification information “TV” of the operation target device associated with “rectangle p”.

  Subsequently, as shown in FIG. 7D, the user taps the area occupied by the “circular r” surrounding the number “6” with the digital pen 12. In response to the trigger operation detection unit 42 detecting the tap, the operation code generation unit 40 searches the operation registration unit 38 using the tapped position (coordinate) as a key, and finds the tapped position (coordinate). The operation command identification information “6” associated with the “circle r” to be included and the operation target device identification information “TV” associated therewith are extracted.

  Subsequently, the operation code generation unit 40 inputs the extracted identification information “TV” and “6” to the operation code conversion unit 44, receives as an output an operation code related to the power-on of the liquid crystal television 74, and operates it. The signal is output to the signal output device 60. The operation signal output device 60 generates and outputs an operation signal based on the operation code input from the operation code generation unit 40. In response to this, as shown in FIG. 2E, the channel of the liquid crystal television 74 is switched from the initial “4” to “6”.

(Operation 5)
As shown in FIG. 7E, the user draws a line s so as to connect “rectangle p” and “circular m” with the digital pen 12. In response, the associating unit 36 identifies the operation instruction identification information “off” associated with “circular m” registered in the operation registration unit 38 and the identification information “TV” of the operation target device associated with “square p”. ”

  Subsequently, as shown in FIG. 7E, the user draws “x” on a line n connecting “rectangle j” and “circle m”. Here, “x” is a symbol determined in advance to mean that a line drawn so as to connect the first graphic and the second graphic is cancelled. Note that “x” is merely an example, and it goes without saying that other symbols may be adopted as symbols for canceling the line.

  Here, the associating unit 36 of the operation request specifying unit 30 always draws a handwritten trajectory input to the handwriting input receiving unit 10 on a line drawn so that “x” connects two figures. The operation registration unit 38 is configured to cancel the association of the identification information based on the line when it is determined that “x” is drawn on the line. ing. Accordingly, in the case illustrated in FIG. 7E, the associating unit 36 cancels the association between the identification information “Light” of the operation target device registered in the operation registration unit 38 and the identification information “off” of the operation command.

  Subsequently, as shown in FIG. 7F, the user taps the area occupied by the “circular m” surrounding the character string “off” with the digital pen 12. In response to the trigger operation detection unit 42 detecting the tap, the operation code generation unit 40 searches the operation registration unit 38 using the tapped position (coordinate) as a key, and finds the tapped position (coordinate). The operation command identification information “off” associated with the “circle m” to be included and the operation target device identification information “TV” associated therewith are extracted. At this time, it should be noted that the identification information “Light” is not extracted in response to the cancellation of the association between the identification information “off” and the identification information “Light” due to the cancellation of the line n.

  Subsequently, the operation code generation unit 40 inputs the operation target device “TV” and the operation command “off” to the operation code conversion unit 44. In response to this, the operation code conversion unit 44 returns an operation code for turning off the power of the liquid crystal television 74 as an output in response to the input (TV, off). In response to this, the operation signal output device 60 outputs an operation signal (for example, an infrared signal) obtained by decoding the operation code from the operation code conversion unit 44 from a corresponding signal output means (for example, an infrared LED).

  Subsequently, the operation code generation unit 40 inputs the extracted identification information “TV” and “off” to the operation code conversion unit 44, receives an operation code related to power-off of the liquid crystal television 74 as an output, and operates this The signal is output to the signal output device 60. The operation signal output device 60 generates and outputs an operation signal based on the operation code input from the operation code generation unit 40. In response to this, as shown in FIG. 2 (f), only the liquid crystal television 74 is turned off while the table lamp 72 is turned on.

  As described above, according to the electronic device remote operation system 100 of the present embodiment, a natural user interface (NUI) for remotely operating an electronic device based on a user's writing operation is realized. As a result, for example, the user can turn on the TV or turn off the cooler using the margin of the notebook while writing.

  Since the electronic device remote control system 100 of the present embodiment uses the operation of writing familiar to the user, the user's resistance is less than that of a conventional NUI that uses gestures and utterances. There is little learning burden on the user. Furthermore, since recognition of line diagrams (characters, graphics, etc.) is much easier than that of gestures and utterances, it becomes possible to specify user operation requests with high accuracy in real time. In the future, once a ubiquitous platform for recognizing the user's writing action is in place, it will be possible to operate various electronic devices just by writing action anytime and anywhere.

  In addition, according to the electronic device remote operation system 100 of the present embodiment, it is possible to easily perform complicated operations. For example, in order to reserve a washing machine, it is necessary to specify a plurality of conditions such as the start time, the amount of water, and the washing mode, but it is difficult to do this with gestures and utterances. In this respect, according to the present system using the writing operation, this can be easily performed. Here, an example of a complex operation that can be performed by the present system will be described. The operation target device in this example is an “electronic musical instrument (synthesizer)”. In this case, the user performs handwriting using the digital pen 12 in the following procedure.

Specifically, as shown in FIG.
(1) Handwriting the characters “Music”.
(2) Surround the handwritten character “Music” with a square.
(3) Draw a staff notation in the square.

  In response to the handwriting operations (1) and (2) above, the handwritten drawing analysis unit 20 recognizes the character string “Music” and “square”, and the operation command specifying unit 34 changes the character string “Music” to “square”. When it is determined that the character string is enclosed, the character string “Music” is registered in the operation registration unit 38 as the identification information of the operation target device in association with the “rectangle”.

  In this case, the handwritten drawing analysis unit 20 has a function of analyzing the notation contents of the staff notation, and analyzes the notation contents of the staff notation drawn in the square in response to the handwriting operation of (3) above. Then, the analysis result (notation content) is registered in the operation registration unit 38.

  Subsequently, the user performs handwriting using the digital pen 12 according to the following procedure.

Specifically, as shown in FIG.
(4) Draw the characters “Pf” and “VI” and enclose each in “circle”.
(5) A line is drawn so as to connect each “circular” and “stave” surrounding “Pf” and “VI”.
(6) Draw characters “play” and “stop”, and enclose each in “circle”.
(7) Draw a line so as to connect each of the “circular” and “stave” surrounding “play” and “stop”.

  In response to the handwriting operation of (4) above, the handwritten drawing analysis unit 20 recognizes the character string “Pf” and the character string “VI” and “circular”, and the operation command specifying unit 34 determines the character string “Pf” and the character string. It is determined that “VI” is surrounded by “circular”, and the character string “Pf” and the character string “VI” are associated with the respective “circular” as operation command identification information and registered in the operation registration unit 38. . Then, the associating unit 36 determines that each “circular” is surrounded by “quadrangle”, and identification information “Pf” and “Pf” of the operation instruction associated with the “circular” registered in the operation registering unit 38. The operation registration unit 38 associates the identification information “Music” of the operation target device associated with “VI” with the “square”.

  Similarly, in response to the handwriting operation of (6), the handwritten drawing analysis unit 20 recognizes the character string “play” and the character strings “stop” and “circular”, and the operation command specifying unit 34 recognizes the character string “play”. Then, it is determined that the character string “stop” is surrounded by “circular”, and the character register “play” and the character string “stop” are associated with each “circular” as identification information of the operation command, and the operation registration unit 38. Register with. Then, the associating unit 36 determines that each “circle” is surrounded by “rectangles”, and identifies the identification information (“play”, “play”, and operation command associated with the “circle” registered in the operation registration unit 38. The operation registration unit 38 associates “stop”) with the identification information “Music” of the operation target device associated with the “rectangle”.

  On the other hand, in response to the handwriting operation of (5) above, the associating unit 36 determines the association between “circular” and “stave” located at both ends of the line, and the “stave” registered in the operation registration unit 38. The operation registration unit 38 associates the notation contents of the operation instruction and the identification information (“Pf”, “VI”) of the operation instruction associated with the “circular”. Similarly, in response to the handwriting operation of (7) above, the associating unit 36 determines the association between “circular” and “stave” located at both ends of the line, and the “stave score” registered in the operation registration unit 38. The operation registration unit 38 associates the notation content of “” and the identification information (“play”, “stop”) of the operation command associated with the “circular”.

  Thereafter, in response to the “circle” surrounding the character string (“play”, “stop”) being tapped by the digital pen 12, the operation code generating unit 40 identifies the identification information associated with the tapped “circle”. ("Play", "stop"), the notation content of the staff notation associated with the identification information ("play", "stop"), and the identification information ("Pf", "VI") associated with the "stave" ) And identification information “Music” of the operation target device associated with the identification information (“play”, “stop”) are extracted from the operation registration unit 38.

  Subsequently, the operation code generation unit 40 inputs the content extracted from the operation registration unit 38 to the operation code conversion unit 44, receives an operation code related to “electronic musical instrument” as an output, and sends this to the operation signal output device 60. Output.

  For example, when the “circle” surrounding the character string “play” is tapped by the digital pen 12, the operation code generation unit 40 adds the identification information “play” associated with the “circle” and the identification information “play”. Manipulates the combination of the notation contents of the "stave" associated with the associated information, the identification information ("Pf", "VI") associated with the "stave", and the identification information "Music" associated with the identification information "play" Input to the code conversion unit 44. In response to this, the operation code conversion unit 44 uses the notation content of the input “staff” “tones of musical instruments (“ piano ”,“ violin ”) corresponding to the identification information (“ Pf ”,“ VI ”). The performance data (eg, MIDI data) to be generated at the time is output to the operation signal output device 60 as an operation code.

  The operation signal output device 60 generates an operation signal (MIDI signal) based on the operation code input from the operation code generation unit 40 and outputs it to the “electronic musical instrument”. In response to this, the “electronic musical instrument” plays the melody represented in the handwritten staff in the specified tone.

  As mentioned above, although this invention has been demonstrated with embodiment, it cannot be overemphasized that this invention is not limited to embodiment mentioned above. For example, in the above-described embodiment, the description has been given of the notation in which “rectangle” is drawn as the first graphic defining the operation target device and “circular” is drawn as the second graphic defining the operation command. It goes without saying that other figures may be adopted as the second figure. In the above-described embodiment, the notation for drawing a line so as to connect the first graphic and the second graphic in order to associate the operation target device with the operation command has been described. However, other diagram representations (for example, the first graphic representation) Needless to say, the operation target device may be associated with the operation command by drawing an arrow line between the first graphic and the second graphic). Further, it goes without saying that the present invention does not exclude other appropriate notations from being devised and adopted.

  Further, in the above-described embodiment, the aspect in which the electronic device is remotely operated using the writing operation on the paper using the digital pen has been described. However, the present invention is not limited to the writing operation using the pen tablet, the touch pad, or the touch panel. It also includes a mode in which an electronic device is remotely controlled using a drawing operation with a finger using the. Furthermore, in the present invention, all the functional means shown in FIG. 1 may be housed and integrated in one housing, or each functional means shown in FIG. 1 may be distributed and arranged on the network in appropriate units. May be. In addition, it is included in the scope of the present invention as long as the effects and effects of the present invention are exhibited within the scope of embodiments that can be considered by those skilled in the art.

  Note that each function of the above-described embodiment can be realized by a device-executable program described in C, C ++, C #, Java (registered trademark), and the like. The program of this embodiment includes a hard disk device, a CD-ROM. , MO, DVD, flexible disk, EEPROM, EPROM and the like can be stored and distributed in a device-readable recording medium, and can be transmitted via a network in a format that other devices can.

DESCRIPTION OF SYMBOLS 10 ... Handwritten input reception part 12 ... Digital pen 14 ... Paper 20 ... Handwritten drawing analysis part 22 ... Character recognition part 24 ... Graphic recognition part 26 ... Machine learning discriminator 30 ... Operation request specific | specification part 32 ... Operation object apparatus specific part 34 ... Operation command specifying unit 36 ... Association unit 38 ... Operation registration unit 40 ... Operation code generation unit 42 ... Trigger operation detection unit 44 ... Operation code conversion unit 50 ... Information processing device 60 ... Operation signal output device 72 ... Table lamp 74 ... Liquid crystal television 100: Electronic device remote control system

Claims (15)

A handwriting input receiving means for receiving a handwritten locus as an input;
Handwritten drawing analysis means for analyzing the content drawn by the user from the handwritten locus;
Operation request specifying means for specifying the operation content of the electronic device requested by the user in light of a predetermined notation of the content drawn by the user;
Operation code generation means for generating an operation code of the electronic device corresponding to the specified operation content;
Only including,
The handwritten drawing analysis means includes:
A character recognition means for recognizing a character meaning an operation target device and an operation command;
Graphic recognition means for recognizing a first graphic defining an operation target device and a second graphic defining an operation command;
Including
The operation request specifying means includes:
An operation target device specifying means for associating a character surrounded by the first graphic with the first graphic as identification information of the operation target device;
An operation command specifying means for associating a character surrounded by the second graphic with the second graphic as identification information of the operation command;
Association means for associating the identification information of the operation target device with the identification information of the operation command;
Including
The operation code generation means includes
Identification information of an operation command associated with the second graphic in response to detection of the trigger operation on the second graphic, including trigger operation detection means for detecting a trigger operation meaning an operation start request Generating an operation code representing the operation instruction for the electronic device related to the identification information of the operation target device associated with
Electronic equipment remote control system.   The predetermined notation is a notation in which an operation target device and an operation command are expressed by a combination of characters and figures meaning itself, and a correspondence relationship between the operation target device and the operation command is expressed by a diagram. Item 2. The electronic device remote control system according to Item 1.   The electronic device remote operation system according to claim 1, wherein the handwriting input reception unit receives a handwritten locus acquired by a digital pen as an input. The association means includes
In response to the drawing of a line connecting the first graphic and the second graphic, the identification information of the operation target device associated with the first graphic and the second graphic are associated with each other. The electronic device remote operation system according to claim 1, wherein the identification information of the operation command is associated. The association means includes
The electronic device remote control system according to claim 4 , wherein, in response to the drawing of a predetermined symbol that means to cancel the line, the association based on the line is canceled. The association means includes
When the second graphic is surrounded by the first graphic, the identification information of the operation target device linked to the first graphic and the identification information of the operation command linked to the second graphic are displayed. The electronic device remote control system according to claim 1 , wherein the electronic device remote control system is associated. The operation code generation means includes
Operation code conversion means for inputting identification information related to the operation target device and identification information related to the operation command and outputting an operation code representing the operation command for the operation target device is provided.
In response to detecting the trigger operation on the second graphic, the identification information related to the operation command associated with the second graphic and the identification information related to the operation target device associated with the identification information are Input to the code conversion means, and obtain an operation code representing the operation instruction for the operation target device as an output;
The electronic device remote control system according to any one of claims 1 to 6 . The operation signal output device further includes an operation signal output device that generates an operation signal based on the operation code acquired by the operation code generation unit and outputs the operation signal on a communication medium corresponding to the operation target device. The electronic device remote control system according to any one of claims 7 to 9. A computer for remote control of electronic devices,
Handwriting input accepting means for accepting a handwritten locus as an input;
Handwritten drawing analysis means for analyzing the content drawn by the user from the handwritten locus;
An operation request specifying means for specifying the operation content of the electronic device requested by the user in light of a predetermined notation of the content drawn by the user;
Operation code generation means for generating an operation code of the electronic device corresponding to the specified operation content;
Is a program for functioning as
The handwritten drawing analysis means includes:
A character recognition means for recognizing a character meaning an operation target device and an operation command;
Graphic recognition means for recognizing a first graphic defining an operation target device and a second graphic defining an operation command;
Including
The operation request specifying means includes:
An operation target device specifying means for associating a character surrounded by the first graphic with the first graphic as identification information of the operation target device;
An operation command specifying means for associating a character surrounded by the second graphic with the second graphic as identification information of the operation command;
Association means for associating the identification information of the operation target device with the identification information of the operation command;
Including
The operation code generation means includes
Identification information of an operation command associated with the second graphic in response to detection of the trigger operation on the second graphic, including trigger operation detection means for detecting a trigger operation meaning an operation start request Generating an operation code representing the operation command for the operation target device related to the identification information of the operation target device associated with
A computer executable program. The predetermined notation is a notation in which an operation target device and an operation command are expressed by a combination of characters and figures meaning itself, and a correspondence relationship between the operation target device and the operation command is expressed by a diagram. Item 10. The program according to Item 9 . The program according to claim 9 or 10 , wherein the handwriting input receiving unit receives a handwritten locus acquired by a digital pen as an input. The association means includes
In response to the drawing of a line connecting the first graphic and the second graphic, the identification information of the operation target device associated with the first graphic and the second graphic are associated with each other. The program according to any one of claims 9 to 11, which associates identification information of the operation instruction. The association means includes
13. The program according to claim 12 , wherein in response to drawing of a predetermined symbol that means to cancel the line, the association based on the line is canceled. The association means includes
When the second graphic is surrounded by the first graphic, the identification information of the operation target device linked to the first graphic and the identification information of the operation command linked to the second graphic are displayed. The program according to any one of claims 9 to 13 , which is associated. The operation code generation means includes
Comprising operation code conversion means for inputting identification information relating to an electronic device and identification information relating to an operation instruction, and outputting an operation code representing the operation instruction for the electronic device;
In response to detecting the trigger operation on the second graphic, identification information related to an operation command associated with the second graphic and identification information related to an electronic device associated with the identification information are stored in the operation code. Input to the conversion means, and obtain an operation code representing the operation instruction for the electronic device as an output;
The program as described in any one of Claims 9-14 .