JP6857757B2 - Screen display system, screen display method and screen display processing program - Google Patents

Description

An embodiment of the present invention relates to a screen display system, a screen display method, and a screen display processing program.

In recent years, when an image taken by a portable terminal equipped with an imaging function is transmitted to an image recognition server on the Web, the image taken based on the dictionary image of the database on the server is recognized, and information according to the recognition result is displayed. An image recognition service (hospitality service) obtained by returning to a terminal is considered (see, for example, Non-Patent Document 1).

FIG. 6 is a functional block diagram showing a configuration example of a conventional screen display system.
This screen display system includes a terminal management system 110, an image recognition system 130, and a translation processing system 140.

The terminal management system 110 includes a terminal communication unit 111, a terminal control unit 112, and an image inquiry unit 113. The image recognition system 130 has an image recognition unit 131 and a dictionary image storage unit 132. The portable terminal 120 owned by the user includes a management system communication unit 121, a camera (imaging unit) 122, and a display (display unit) 123.

FIG. 7 is a sequence diagram showing an example of the operation of the conventional screen display system.
When the terminal communication unit 111 of the terminal management system 110 receives the image data to be recognized taken by the camera 122 by the user operation of the portable terminal 120 (T101 to T104), this image data is sent to the terminal control unit 112 (T105). ). The terminal control unit 112 outputs a recognition request for the sent image data to the image inquiry unit 113 (T106).

Then, the image inquiry unit 113 sends the image data to be recognized to the image recognition unit 131 of the image recognition system 130 (T107). The image recognition unit 131 performs image recognition by determining the degree of similarity between the image data to be recognized and a large number of dictionary image data stored in the dictionary image storage unit 132 (T108), and this recognition result (terminal management). If the image sent from the system 110 is an image containing a character string, the character string, and if the image sent from the terminal management system 110 is an image containing no character string, the object included in this image ( For example, a character string indicating the name of a flower) is returned to the terminal control unit 112. (T109, T110).
Then, the terminal control unit 112 outputs a translation inquiry of the character string indicated by the recognition result to the translation processing system 140 (T111).

In response to this translation inquiry, the translation processing system 140 performs translation processing of the pre-translation character string, and sends the translation processing result (post-translation character string) to the terminal control unit 112 as a translation result response (T112).

When the terminal control unit 112 acquires the translation processing result, it generates terminal screen data including the translated character string, and transmits the generated terminal screen data and this display instruction signal to the portable terminal 120 via the terminal communication unit 111. (T113, T114).

The management system communication unit 121 of the portable terminal 120 displays the terminal screen data including the translated character string on the display 123. As a result, the user confirms the displayed post-translation character string (T115, T116).

NTT Technology Journal 2016 Vol.28 No.5 pp57-59 "Starting technical verification for the realization of hospitality services using simple portable terminals equipped with transmissive displays"

Related to the above image recognition service, there is a technology that translates the character string included in the image recognition result from Japanese to English by translation processing and presents this translation result to a portable terminal. When presenting, even if the translation result of a character string different from the original character string intended by the user is displayed on the portable terminal, the user may not notice this.

Specifically, for example, when the character string (original character string) of "○○" is written on the information board, the user photographs the character string of "○○" as described above. , Suppose you can translate this string and get the translation result "~ Park".

However, even if the user intends to take a picture of the original character string and obtain the translation result of this character string, another kind of character such as "**" is added to the above "○○". This other character string may be mistakenly photographed and the translation result of this character string may be obtained because the columns are adjacent to each other on the guide plate. In this case, the user may mistakenly recognize that the translation result is the translation result of the above original character string (“○○”). In particular, when the original character string and another type of character string are similar, the possibility of erroneous recognition of the above translation result increases.

An object of the present invention is to provide a screen display system, a screen display method, and a screen display processing program that can promote appropriate confirmation of a translation result according to image recognition.

In order to achieve the above object, the first aspect of the screen display system according to the embodiment of the present invention is an image receiving means for receiving an image taken by an external terminal, and information indicating the content of the image. A dictionary image storage means for storing as a dictionary image together with information indicating whether or not the character string is intended for translation, an image recognition requesting means for requesting recognition of the image received by the image receiving means, and an image recognition requesting means for requesting recognition of the image received by the image receiving means. In response to the request by the image recognition requesting means, the received image is used as the recognition target image to recognize the dictionary image which is the recognition result of the recognition target image, and the dictionary image obtained as the recognition result is a character string. When it contains information indicating that the image is intended to be translated, it indicates that the character string indicated by the recognition result and the type of the recognition result are the character strings included in the image as the recognition result. When the information is output and the dictionary image obtained as the recognition result contains information indicating that the character string is not intended to be translated, the character string indicated by the recognition result and the type of the recognition result. Is an image recognition means that outputs information indicating that is the name of an object included in the image that is the recognition result, and the type of the recognition result is a character string that is included in the image that is the recognition result. In this case, the character string output from the image recognition means is translated, and when the type of the recognition result is the name of the object included in the image as the recognition result, the name of the object is translated. The processing means and the character string after the translation processing by the translation processing means and the character string before the translation processing by the translation processing means when the type of the recognition result is the character string included in the image which is the recognition result. A terminal screen including each is generated, and a terminal screen including a character string after translation processing by the translation processing means is generated when the type of the recognition result is the name of an object included in the image as the recognition result. Provided is a system having a terminal screen generating means and a terminal screen transmitting means for transmitting the terminal screen generated by the terminal screen generating means to the external terminal.

A second aspect of the system having the above configuration is that, in the first aspect, the terminal screen generation means is based on a first terminal screen including a character string after translation processing by the translation processing means, and the translation processing means. A second terminal screen including the character string before the translation process is generated, and the terminal screen transmitting means sequentially displays the generated first terminal screen and the second terminal screen on the external terminal. To provide a system for transmitting to the external terminal.

In a third aspect of the system having the above configuration, in the second aspect, the terminal screen transmitting means alternately and repeatedly displays the generated first terminal screen and the second terminal screen on the external terminal. Provide a system for transmitting to the external terminal so as to be performed.

The first aspect of the screen display method according to the embodiment of the present invention stores an image as a dictionary image together with information indicating the content of the image and information indicating whether or not the character string is intended to be translated. A method applied to a screen display system having a dictionary image storage device, which acquires an image taken by a terminal, requests recognition of the acquired image, and in response to the request, the acquired image. When the dictionary image which is the recognition result of the recognition target image is recognized by using the image as the recognition target image and the dictionary image which is the recognition result contains information indicating that the character string is intended to be translated. , The character string indicated by the recognition result, and the information indicating that the type of the recognition result is the character string included in the image which is the recognition result are output, and the dictionary image which is the recognition result is When the character string contains information indicating that the image is not intended for translation, the character string indicated by the recognition result and the type of the recognition result are the names of objects included in the image whose recognition result is the recognition result. Information indicating that is output, and when the type of the recognition result is a character string included in the image as the recognition result, the output character string is translated and the type of the recognition result is changed to When the name of the object included in the image obtained as the recognition result is the name of the object, the name of the object is translated, and the translation when the type of the recognition result is a character string included in the image obtained as the recognition result. After the translation process, a terminal screen including the character string after the processing and the character string before the translation is generated, and the type of the recognition result is the name of the object included in the image as the recognition result. Provided is a method of generating a terminal screen including a character string and transmitting the generated terminal screen to the terminal.

In the second aspect of the above method, in the first aspect, a first terminal screen including the translated character string and a second terminal screen including the untranslated character string are generated, respectively. Then, a method is provided in which the generated first terminal screen and the second terminal screen are transmitted to the terminal so as to be sequentially displayed on the terminal.

A third aspect of the above method is a method of transmitting to the terminal so that the generated first terminal screen and the second terminal screen are alternately and repeatedly displayed on the terminal in the second aspect. I will provide a.

According to the present invention, it is possible to promote appropriate confirmation of a translation result according to image recognition.

The functional block diagram which shows the structural example of the screen display system in 1st Embodiment of this invention. The sequence diagram which shows the 1st example of the operation of the screen display system in 1st Embodiment of this invention. The sequence diagram which shows the 2nd example of the operation of the screen display system in 1st Embodiment of this invention. The sequence diagram which shows an example of the operation of the screen display system in 2nd Embodiment of this invention. The sequence diagram which shows an example of the operation of the screen display system in 3rd Embodiment of this invention. A functional block diagram showing a configuration example of a conventional screen display system. A sequence diagram showing an example of the operation of a conventional screen display system.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First Embodiment)
First, the first embodiment of the present invention will be described. FIG. 1 is a functional block diagram showing a configuration example of a screen display system according to the first embodiment of the present invention.
This screen display system includes a terminal management system 10, an image recognition system 30, and a translation processing system 40.

The terminal management system 10 includes a terminal communication unit 11, a terminal control unit 12, and an image inquiry unit 13. The image recognition system 30 has an image recognition unit 31 and a dictionary image storage unit 32.
The terminal management system 10, the image recognition system 30, and the translation processing system 40 are composed of a server device connected to a communication network such as the Internet, and a CPU built in the server device stores the server device in a storage device such as a hard disk in advance. Each part is made to function by executing a server control program (the image recognition system 30 includes an image recognition program and the translation processing system 40 includes a translation processing program) to control the operation of each part of the electronic circuit.
A part or all of the terminal management system 10, the image recognition system 30, and the translation processing system 40 may be an integrated device.

The terminal communication unit 11 has (1) a function of receiving image data captured by the portable terminal 20 connected via a communication network and sending it to the terminal control unit 12, and (2) being sent to the terminal control unit 12. It has a function of transmitting the terminal screen data generated by the terminal control unit 12 to the portable terminal 20 together with the screen display instruction according to the recognition result of the image data.

The terminal control unit 12 has (1) a function of requesting the image inquiry unit 13 to recognize the image data sent from the terminal communication unit 11, and (2) a function of acquiring the recognition result of the image data from the image recognition system 30. (3) When the type of recognition result is a character string, a function to inquire the translation processing system 40 for the translation of this character string, (4) a function to generate terminal screen data including the translation processing result, (5) a terminal screen. It has a function of sending data to the terminal communication unit 11.
The image inquiry unit 13 has a function of sending the image data to the image recognition system 30 in response to a recognition request of the image data from the terminal control unit 12.

The image recognition unit 31 of the image recognition system 30 determines (1) the degree of similarity between the image data to be recognized sent from the image inquiry unit 13 and a large number of dictionary image data stored in the dictionary image storage unit 32. It has a function of recognizing an image and responding to the recognition result to the terminal control unit 12.
The dictionary image storage unit 32 stores, for example, a large number of image data obtained by photographing a wide variety of things existing in the world as a dictionary image together with information indicating (representing) the contents of the image data.

In the terminal management system 10, the image recognition system 30, and the translation processing system 40 configured in this way, the CPU controls the operation of each part of the circuit according to the instructions described in the control program, and the software and the hardware cooperate with each other. By operating the system, various functions as described in the following operation description are realized.

The portable terminal 20 includes a management system communication unit 21, a camera (imaging unit) 22, and a display (display unit) 23, and the CPU built in the portable terminal 20 is also stored in a storage device such as a flash ROM. By executing the terminal control program stored in advance to control the operation of each part of the electronic circuit, each part of the management system communication unit 21, the camera 22, and the display 23 is made to function.

The management system communication unit 21 has (1) a function of transmitting image data taken by the camera 22 in response to a user's operation to the terminal management system 10 of a server device on the communication network, and (2) the transmitted image data. It has a function of receiving terminal screen data including information according to the recognition result from the terminal management system 10 and displaying it on the display 23.

Next, the operation of the screen display system having the above configuration will be described.
FIG. 2 is a sequence diagram showing a first example of the operation of the screen display system according to the first embodiment of the present invention.
When the terminal communication unit 11 of the terminal management system 10 receives the image data of the recognition target captured by the user operation after the start of use of the portable terminal 20 (T1 to T4), the image data is sent to the terminal control unit 12 (T1 to T4). T5). The terminal control unit 12 outputs a recognition request for the sent image data to the image inquiry unit 13 (T6).

Then, the image inquiry unit 13 sends the image data to be recognized (here, the image data including the character string “~ park”) to the image recognition unit 31 of the image recognition system (T7). The image recognition unit 31 performs image recognition by determining the degree of similarity between the image data to be recognized and a large number of dictionary image data stored in the dictionary image storage unit 32 (T8), and this recognition result (here, here). , The character string "~ park" of the image including the above character string "~ park") and its type (information indicating whether or not the recognition result is a character string included in the image, here "character string", Or "Other (name)") is responded to the terminal control unit 12. (T9).
In the present embodiment, the dictionary image data includes information indicating whether or not the image is intended to translate the character string, and the dictionary image data obtained as the recognition result in the above image recognition is intended to translate the character string. When the information indicating that the image is the image is included, the above type that is responded with the recognition result is the information (“character string”) indicating that the image is a character string included in the image. Further, when the dictionary image data used as the recognition result in the above image recognition contains information indicating that the image is not intended for translation of the character string, the above type to be responded with the recognition result is included in the image. It is not the character string that is displayed, but the information (“other (name)”) that indicates the name of the object included in this image.

Then, when the terminal control unit 12 confirms that the type of the recognition result is "character string" (T10), the terminal control unit 12 inquires about the translation of the character string (character string before translation, here "~ park") indicated by the recognition result. Is output to the translation processing system 40 (T11).

In response to this translation inquiry, the translation processing system 40 performs translation processing of the pre-translation character string, and sends the translation processing result (post-translation character string, here "~ park") to the terminal control unit 12 as a translation result response. Send (T12).

When the terminal control unit 12 acquires the translation processing result, it generates terminal screen data including the previously obtained pre-translation character string and post-translation character string, and generates the generated terminal screen data and this display instruction signal. It is transmitted to the portable terminal 20 via the terminal communication unit 11 (T13, T14).

The management system communication unit 21 of the portable terminal 20 displays the terminal screen data including the pre-translation character string and the post-translation character string by recognizing the image data captured by the user operation on the display 23. As a result, the user can quickly and easily obtain the pre-translation character string corresponding to the displayed post-translation character string, check the pre-translation character string, or compare the pre-translation character string with the post-translation character string. Can be done (T15, T16).

FIG. 3 is a sequence diagram showing a second example of the operation of the screen display system according to the first embodiment of the present invention.
In the sequence diagram of the second example shown in FIG. 3, the same operation portions as those of the sequence diagram of the first example shown in FIG. 2 are designated by the same reference numerals T1 to T9, and the description thereof will be omitted. However, it is assumed that the image data to be recognized is image data that does not include a character string.

After T9, when the terminal control unit 12 confirms that the type of the recognition result is "other (name)", that is, the image data to be recognized does not include the character string as described above (T10a), the recognition result Outputs the translation inquiry of the character string indicated by (the character string before translation as the name of the object (here, the rose flower) included in the image data not including the character string, here "rose") to the translation processing system 40. (T11a).

In response to this translation inquiry, the translation processing system 40 performs translation processing of the pre-translation character string, and sends the translation processing result (post-translation character string, here "rose") to the terminal control unit 12 as a translation result response. (T12a).

When the terminal control unit 12 acquires the translation processing result, it generates terminal screen data including the translated character string, and transmits the generated terminal screen data and this display instruction signal to the portable terminal 20 via the terminal communication unit 11. (T13a, T14a).
The management system communication unit 21 of the portable terminal 20 displays the terminal screen data including the translated character string on the display 23 (T15a, T16a).

As described above, according to the screen display system according to the first embodiment of the present invention, image data including a character string (for example, "~ park") taken by the portable terminal 20 is received and stored in a dictionary image. When image recognition processing is performed based on a large number of dictionary image data stored in unit 32, the pre-translation character string as the recognition result is translated to obtain the post-translation character string (for example, "~ park"). Together, these pre-translation character strings and post-translation character strings are displayed on the portable terminal 20 at the same time.

As a result, even if a character string different from the character string intended to be translated by the user is photographed and this character string is recognized as the pre-translation character string, this pre-translation character string is regarded as the post-translation character string. Can be confirmed at the same time, so it is possible to easily confirm whether or not the translation result of the character string intended by the user is obtained.

By this confirmation, if the character string before translation is different from the character string intended by the user, the user can notice that the wrong object has been photographed, and prompts the user to re-photograph the correct object. be able to.

(Second embodiment)
Next, the second embodiment will be described. FIG. 4 is a sequence diagram showing the operation of the screen display system according to the second embodiment of the present invention.
In the sequence diagram of the second embodiment shown in FIG. 4, the same operation portions as those of the sequence diagram of the first embodiment shown in FIG. 2 are designated by the same reference numerals T1 to T24, and the description thereof will be omitted.

However, in T24, when the terminal control unit 12 acquires the translation processing result, the terminal control unit 12 obtains the first terminal screen data including the previously obtained pre-translation character string and the second terminal screen data including the post-translation character string. The terminal communication unit 11 displays the generated terminal screen data and a display instruction signal that is an instruction to sequentially display the generated terminal screen data one by one in the order of, for example, the pre-translation character string, the post-translation character string, or the reverse order. It is transmitted to the portable terminal 20 via.

The management system communication unit 21 of the portable terminal 20 displays the first and second terminal screen data by recognizing the image data captured by the user operation on the display 23 in the order indicated by the display instruction signal. As a result, as in the first embodiment, the user can quickly and easily obtain the pre-translation character string corresponding to the displayed post-translation character string, and confirm the pre-translation character string or confirm the pre-translation character string. Can be compared with the translated character string (T15a, T16a, T15b, T16b). Further, as compared with the first embodiment, the user can clearly distinguish the displayed pre-translation character string and the post-translation character string.

(Third Embodiment)
Next, a third embodiment will be described. FIG. 5 is a sequence diagram showing the operation of the screen display system according to the third embodiment of the present invention.
In the sequence diagram of the third embodiment shown in FIG. 5, the same operation portions as those of the sequence diagram of the first embodiment shown in FIG. 2 are designated by the same reference numerals T1 to T24, and the description thereof will be omitted.

However, in T24, when the terminal control unit 12 acquires the translation processing result, the terminal control unit 12 obtains the first terminal screen data including the previously obtained pre-translation character string and the second terminal screen data including the post-translation character string. Each of them is generated, and the generated terminal screen data and these are repeatedly displayed one by one in the order of the pre-translation character string, the post-translation character string, or vice versa, for a predetermined number of times or for a predetermined time (for example, before translation). Character string-> character string after translation-> character string before translation-> character string after translation-> ...) The display instruction signal, which is an instruction, is transmitted to the portable terminal 20 via the terminal communication unit 11.

The management system communication unit 21 of the portable terminal 20 repeatedly displays the first and second terminal screen data by recognizing the image data captured by the user operation on the display 23 in the order instructed by the display instruction signal a predetermined number of times. .. As a result, as in the first embodiment, the user can quickly and easily obtain the pre-translation character string corresponding to the displayed post-translation character string, and confirm the pre-translation character string or confirm the pre-translation character string. Can be compared with the translated character string (T15c, T16d, T15c, T16d). Further, as compared with the first embodiment, the user can clearly and repeatedly distinguish the displayed pre-translation character string and the post-translation character string.

The present invention is not limited to the above-described embodiment as it is, and at the implementation stage, the components can be modified and embodied within a range that does not deviate from the gist thereof. In addition, various inventions can be formed by an appropriate combination of the plurality of components disclosed in the above-described embodiment. For example, some components may be removed from all the components shown in the embodiments. In addition, components across different embodiments may be combined as appropriate.

Further, the methods described in each embodiment include, for example, a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM,) as a program (software means) that can be executed by a computer (computer). It can be stored in a recording medium such as a DVD, MO, etc.), a semiconductor memory (ROM, RAM, flash memory, etc.), or transmitted and distributed by a communication medium. The program stored on the medium side also includes a setting program for configuring the software means (including not only the execution program but also the table and the data structure) to be executed by the computer in the computer. A computer that realizes this system reads a program recorded on a recording medium, constructs software means by a setting program in some cases, and executes the above-mentioned processing by controlling the operation by the software means. The recording medium referred to in the present specification is not limited to distribution, and includes a storage medium such as a magnetic disk or a semiconductor memory provided in a device connected inside a computer or via a network.

10,110 ... Terminal management system, 11,111 ... Terminal communication unit, 12,112 ... Terminal control unit, 13,113 ... Image inquiry unit, 20,120 ... Portable terminal, 21,121 ... Management system communication unit, 22 , 122 ... Camera (imaging unit), 23,123 ... Display (display unit), 30,130 ... Image recognition system, 31,131 ... Image recognition unit, 32,132 ... Dictionary image storage unit, 40,140 ... Translation processing system.

Claims (7)

An image receiving means for receiving an image taken by an external terminal,
A dictionary image storage means for storing an image as a dictionary image together with information indicating the content of the image and information indicating whether or not the character string is intended to be translated.
An image recognition requesting means that requests recognition of an image received by the image receiving means, and an image recognition requesting means.
In response to the request by the image recognition requesting means, the received image is used as the recognition target image to recognize the dictionary image which is the recognition result of the recognition target image.
When the dictionary image obtained as the recognition result contains information indicating that the image is intended to be translated into a character string, the character string indicated by the recognition result and the type of the recognition result are the recognition result. Outputs information indicating that it is a character string included in the image
When the dictionary image obtained as the recognition result contains information indicating that the image is not intended for translation of the character string, the character string indicated by the recognition result and the type of the recognition result are regarded as the recognition result. An image recognition means that outputs information indicating that the name of the object is included in the image, and
When the type of the recognition result is a character string included in the image obtained as the recognition result, the character string output from the image recognition means is translated.
When the type of the recognition result is the name of the object included in the image obtained as the recognition result, the translation processing means for translating the name of the object and the translation processing means.
A terminal screen including a character string after translation processing by the translation processing means and a character string before translation processing by the translation processing means when the type of the recognition result is a character string included in the image as the recognition result. To generate
When the type of the recognition result is the name of the object included in the image as the recognition result, the terminal screen generation means for generating the terminal screen including the character string after the translation processing by the translation processing means, and the terminal screen generation means.
A screen display system including a terminal screen transmitting means for transmitting a terminal screen generated by the terminal screen generating means to the external terminal. The terminal screen generation means
A first terminal screen including the character string after the translation process by the translation processing means and a second terminal screen including the character string before the translation process by the translation processing means are generated, respectively.
The terminal screen transmitting means
The screen display system according to claim 1, wherein the generated first terminal screen and the second terminal screen are transmitted to the external terminal so as to be sequentially displayed on the external terminal. The terminal screen transmitting means
The screen display system according to claim 2, wherein the generated first terminal screen and the second terminal screen are transmitted to the external terminal so as to be alternately and repeatedly displayed on the external terminal. It is a method applied to a screen display system having a dictionary image storage device that stores an image as a dictionary image together with information indicating the content of the image and information indicating whether or not the character string is intended to be translated. hand,
Get the image taken by the terminal,
Request recognition of the acquired image
In response to the request, the acquired image is used as the recognition target image, and the dictionary image which is the recognition result of the recognition target image is recognized.
When the dictionary image obtained as the recognition result contains information indicating that the image is intended to be translated into a character string, the character string indicated by the recognition result and the type of the recognition result are the recognition result. Outputs information indicating that it is a character string included in the image
When the dictionary image obtained as the recognition result contains information indicating that the image is not intended for translation of the character string, the character string indicated by the recognition result and the type of the recognition result are regarded as the recognition result. Information indicating that it is the name of the object included in the image is output.
When the type of the recognition result is a character string included in the image obtained as the recognition result, the output character string is translated.
When the type of the recognition result is the name of the object included in the image obtained as the recognition result, the name of the object is translated.
When the type of the recognition result is a character string included in the image as the recognition result, a terminal screen including the translated character string and the untranslated character string is generated.
When the type of the recognition result is the name of the object included in the image which is the recognition result, a terminal screen including the character string after the translation process is generated.
A screen display method for transmitting the generated terminal screen to the terminal. A first terminal screen containing the translated character string and a second terminal screen containing the translated character string are generated, respectively.
The screen display method according to claim 4, wherein the generated first terminal screen and the second terminal screen are transmitted to the terminal so as to be sequentially displayed on the terminal. The screen display method according to claim 5, wherein the generated first terminal screen and the second terminal screen are transmitted to the terminal so as to be alternately and repeatedly displayed on the terminal. A program used in a computer that operates as a part of the screen display system according to claim 1.
The computer
A screen display processing program for functioning as the image receiving means, the image recognition requesting means, the translation processing means, the terminal screen generating means, and the terminal screen transmitting means.

Images