JP2024075480A - METHOD FOR PROVIDING RELATED INFORMATION RELATED TO IMAGE PROCESSING, IMAGE PROCESSING SYSTEM, AND IMAGE PROCESSING APPARATUS - Google Patents

Abstract

[Problem] Providing a user with highly relevant information from among image processing executed in the past while taking into consideration the protection of information. [Solution] A related information providing method for providing related information related to a job, comprising the steps of: a processor controlling an image processing device and a data storage device executes a job related to image processing of an original or print data; a step of having a user select whether or not to perform a search using extracted information extracted from image data related to the original or print data; a step of extracting the extracted information from the image data if the search is permitted; a step of searching for related information related to the extracted information; a step of providing the obtained related information to the user; and a step of registering the extracted information and the source image data from which it was extracted in a database. [Selected Figure] Figure 2

Description

The present disclosure relates to a related information providing method for providing a user with information related to an image of a document read in a job related to an image processing device or an image when printing, and to an image processing system and an image processing device for providing related information.

There are known image processing devices such as printers, scanners, and multifunction devices that read and print images contained in one or more documents, store the images as data, or transmit the data to an external device. The image processing device handles images of one or more documents or print data consisting of one or more pages as a single process. The unit of the single process is called a job.
2. Description of the Related Art Image forming apparatuses are known that display information, such as advertisements, other than information related to the image forming apparatus on a display unit during processing of a job such as copying (see, for example, Japanese Patent Application Laid-Open No. 2003-233634).
Furthermore, an image forming device has been proposed that does not simply display a fixed advertisement, but reads an image of a document as input information and displays information related to the input image. More specifically, an image is read from a document, a predetermined area is extracted from the read image, and an item consisting of a character string is recognized from the extracted area. Then, a database memory is searched based on the extracted item, and it is determined whether there is related information related to the item. If there is related information, the related information is read and displayed (for example, see Patent Document 2).

Japanese Patent Application Laid-Open No. 7-261609 JP 2006-217276 A

In the technique disclosed in Japanese Patent Laid-Open No. 2003-233665, an image forming apparatus displays advertisements while a job is being processed, thereby making effective use of a display unit. It would be convenient if the image forming apparatus could display not only advertisements, but also information that is useful to a user or that may be of interest to the user.
Japanese Patent Laid-Open No. 2006-133663 discloses a method for notifying a user of information related to input information. However, the input information may include personal information, confidential information, or other information that the user does not want a third party to know, and the method does not disclose how to protect the information.
The present disclosure has been made in consideration of the above circumstances, and is capable of providing a user with highly relevant information from among image processing operations that have been performed in the past, while taking into consideration the protection of the information.

The present disclosure provides a method for providing related information related to image processing, comprising the steps executed by at least one processor that controls an image processing device and a data storage device individually or collectively, and controls the image processing device to execute at least one job related to image processing, which is a process of reading a document and outputting or storing image data, or a process of expanding print data and printing or storing image data; a step of having a user select whether or not to perform a search using extracted information obtained by extracting at least one of information related to characters and images from image data related to the document or the print data; a step of extracting the extracted information from the image data when a search using the extracted information is permitted; a step of searching for related information related to the extracted information from a database stored in the data storage device; a step of providing the related information obtained by the search to the user; and a step of registering the extracted information and the original image data from which it was extracted in the database.

From a different perspective, the present disclosure provides an image processing system including an image processing device and a data storage device that stores a database, the image processing device including a job execution unit that executes a job related to at least one of image processing, a process of reading a document and outputting or storing image data, or a process of receiving and expanding print data and printing or storing image data, a search usage permission selection unit that allows a user to select whether or not to perform a search using extracted information obtained by extracting at least one of information related to characters and images from image data related to the document or the print data, and a related information provision unit that provides the user with related information obtained by searching the database when a search using the extracted information is permitted, and the data storage device includes a related information search unit that searches the database for related information related to the extracted information and provides the search results to the image processing device when a search using the extracted information is permitted, and an extracted information registration unit that registers the extracted information and the original image data from which it was extracted in the database.

The present disclosure further provides an image processing device including a job execution unit that executes a job related to at least one of image processing, a process of reading a document and outputting or storing image data, or a process of receiving and expanding print data and printing or storing image data; a search usage permission selection unit that allows a user to select whether or not to perform a search using extracted information obtained by extracting at least one of information related to characters and images from image data related to the document or the print data; a data storage unit that stores a database; a related information search unit that searches the database for related information related to the extracted information in the data storage unit when a search using the extracted information is permitted; a related information provision unit that provides the user with the related information obtained by the search; and an extracted information registration unit that registers the extracted information and the original image data from which it was extracted in the database when a search using the extracted information is permitted.

The present disclosure provides an image processing device that includes a job execution unit that executes a job related to at least one of image processing, a process of reading a document and outputting or storing image data, or a process of receiving and expanding print data, and printing or storing image data, a natural language processing usage permission setting unit that accepts a setting as to whether or not natural language processing is performed using extracted information obtained by extracting character-related information from image data related to the document or the print data, and a processing result acquisition unit that accesses an external natural language processing service to perform natural language processing using the extracted information and acquires the results from the natural language processing service.

From a different perspective, the present disclosure provides a method for performing natural language processing using extraction information related to a job, the method including the steps executed by at least one processor controlling an image processing device, the steps being to execute a job related to at least one image processing of reading a document and outputting or storing image data, or developing print data and printing or storing image data, a step of having a user set whether or not natural language processing is to be performed using extracted information obtained by extracting character-related information from image data related to the document or the print data, a step of accessing an external natural language processing service to perform natural language processing using the extracted information when natural language processing using the extracted information is permitted, and a step of acquiring a result from the natural language processing service.

The related information providing method according to the present disclosure includes a step of having a user select whether or not to conduct a search using extracted information in which information related to at least either characters or images is extracted from image data related to a manuscript or the print data, a step of extracting the extracted information from the image data if a search using the extracted information is permitted, and a step of searching for related information related to the extracted information from a database stored in the data storage device, thereby making it possible to provide a user with information related to image processing while taking into consideration the protection of information.
Furthermore, the related information providing method according to the present disclosure includes a step of registering the extracted information and the original image data from which it was extracted in the database, so that past extracted information and the original image are registered in the database, and highly relevant information from among image processing performed in the past can be provided to the user.
The image processing system and image processing device according to the present disclosure also achieve the same advantageous effects.

1 is an explanatory diagram showing an example of the configuration of a system in which an image processing device and a data storage device according to the present disclosure, and an information processing device used by a user are connected via a network. 2 is a block diagram showing an example of the configuration of the image processing device and a server serving as a data storage device shown in FIG. 1 . 3 is a block diagram showing an example of the configuration of an image processing device different from that shown in FIG. 2; 4 is a flowchart illustrating an example of a process in which the image processing apparatus illustrated in FIG. 1 executes a job. 4 is a flowchart showing an example of a process in which the data storage device shown in FIG. 1 searches for related information. 4 is a flowchart showing an example of a process in which the image processing device shown in FIG. 1 provides related information to a user. 2 is an explanatory diagram showing an example in which the image processing apparatus shown in FIG. 1 executes a scan job and scan data and related information relating to the job are provided to an information processing apparatus; 4 is an explanatory diagram showing an example of a search use confirmation screen displayed on the operation unit shown in FIG. 2 or FIG. 3; 4 is an explanatory diagram showing an example of a related information selection screen displayed on the operation unit shown in FIG. 2 or FIG. 3 . 4 is an explanatory diagram showing an example of an output selection screen displayed on the operation unit shown in FIG. 2 or FIG. 3 . 10 is an explanatory diagram showing an example of a search use confirmation screen displayed on the operation unit shown in FIG. 2 or FIG. 3 , which is different from the example shown in FIG. 9 . 4 is an explanatory diagram showing an example of a search use reconfirmation screen displayed on the operation unit shown in FIG. 2 or FIG. 3; 1 is a block diagram showing an example configuration in which an image processing device according to the present disclosure is connected to a server that provides a natural language processing service. 14 is a first flowchart showing an example of a process in which the image processing apparatus shown in FIG. 13 executes a job related to natural language processing. 14 is a second flowchart showing an example of a process in which the image processing apparatus shown in FIG. 13 executes a job related to natural language processing. 14 is an explanatory diagram showing an example of a job selection screen displayed on an operation unit of the image processing apparatus shown in FIG. 13; 14 is an explanatory diagram showing an example of an extracted information provision confirmation screen that the natural language processing enable/disable setting unit shown in FIG. 13 causes to be displayed on the operation unit; 16 is a first flowchart showing an example of processing different from that in FIGS. 14 and 15. 16 is a second flowchart showing an example of processing different from that in FIGS. 14 and 15. 19 is an explanatory diagram showing an example of a connection destination setting screen that accepts settings for a natural language processing service and whether to request confirmation for providing extracted information in the flowchart of FIG. 18. FIG. 20 is a first flowchart showing an example of processing different from that in FIGS. 18 and 19; 20 is a second flowchart showing an example of processing different from that in FIGS. 18 and 19; 20 is a third flowchart showing an example of processing different from that in FIGS. 18 and 19; FIG. 22 is an explanatory diagram showing an example of a user table referred to in the flowchart of FIG. 21 . FIG. 22 is an explanatory diagram showing an example of a group table referred to in the flowchart of FIG. 21 .

The present disclosure will be described in further detail below with reference to the drawings. Note that the following description is illustrative in all respects and should not be construed as limiting the present disclosure.
FIG. 1 is an explanatory diagram showing an example of the configuration of a system in which an image processing device and a data storage device according to the present disclosure, and an information processing device used by a user are connected via a network.

In Fig. 1, the image processing device 200 is a multifunction peripheral. It can read an original document using an original reading unit and output or store image data (scanned data). It can also extract print data received from the outside using an engine unit and print or store the image data. The image processing device 201 is an image scanner. It has an original reading unit but no engine unit. The image processing device 202 is a printer. It has an engine unit but no original reading unit.
1, the data storage device is a server 300. Furthermore, information processing devices 400 and 402 are personal computers (PCs) used by users, and information processing devices 401 and 403 are smartphones used by users.
In FIG. 1, image processing devices 200, 201, and 202, a server 300 serving as a data storage device, and information processing devices 400, 401, 402, and 403 are connected via a network N, and can transmit and receive data.

Fig. 2 is a block diagram showing an example of the configuration of the image processing device and a server acting as a data storage device shown in Fig. 1. The image processing device 200 (i.e., a multifunction peripheral) shown in Fig. 1 is shown as a representative example.
The image processing device 200 includes a control unit 210 , a data storage unit 230 , an operation unit 240 , a document reading unit 250 , an engine unit 260 , and a communication circuit 270 .
The control unit 210 includes an operation control unit 212, a job execution unit 214, a search/use permission selection unit 216, an extracted information providing unit 218, a related information providing unit 219, and an image processing unit 220. The image processing unit 220 includes a document image processing unit 222 and a print data expansion processing unit 224. Furthermore, it optionally includes an information extraction unit 226.

The control unit 210 is a circuit mainly composed of a processor and a memory as hardware resources. The processor executes a control program stored in the memory to realize the functions of the operation control unit 212, the job execution unit 214, the search/use permission selection unit 216, the extracted information providing unit 218, the related information providing unit 219, and the image processing unit 220.
The data storage unit 230 is configured to be accessible from the control unit 210, and is a combination of a RAM and a non-volatile memory. A DDR SDRAM (Double-Data-Rage Synchronous Dynamic Random Access Memory) such as DDR4 or DDR5 may be applied to the RAM. A solid state disk (SSD) or a hard disk drive (HDD) may be applied to the non-volatile memory. At least a part of the memory of the control unit 210 and the data storage unit 230 may be common.

The operation unit 240 includes a display device such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Display) that displays information related to the status and operation of the image processing device 200, and an input detection device such as a touch panel that detects operations.
The document reading unit 250 includes a mechanism, a circuit, and an image sensor for scanning and reading a document, and is a part that generates image data corresponding to the document.
The engine unit 260 is the part that executes printing, and includes mechanisms and circuits for electrophotographic charging, exposure, development, transfer, and fixing, as well as a paper feed tray in the main body and a paper feed and transport mechanism that feeds and transports print paper.
The communication circuit 270 is a circuit for communicating with external devices via wired communication such as Ethernet or wireless communication such as wireless LAN or Bluetooth (registered trademark).

The operation control unit 212 controls the display of the operation unit 240 and processes the user's operation detected by the operation unit 240 .
The job execution unit 214 controls the execution of jobs. It executes jobs such as a copy job for reading an original document and printing the image data, a scan job for reading an original document and outputting or storing the image data, and a print job for expanding print data received from an external source and printing the image data. The job execution unit 214 includes a job queue for managing the execution order of these jobs. The job queue provides a mechanism for registering jobs to be executed in a queue and executing them in order so that resources such as original document reading, printing, and memory required for executing each type of job do not compete with each other.

The search use permission selection unit 216 performs processing to inquire of the user as to whether image data related to a job can be used for searching for related information, and to receive the selection.
The control unit 210 may include an information extraction unit 226 (described later) and an extracted information providing unit 218. The extracted information providing unit 218 provides the extracted information extracted by the information extraction unit 226 to a data storage device.
The related information providing unit 219 performs processing to provide the user with related information found by a search using the extracted information when the search is permitted.
The image processing unit 220 performs processes relating to reading and printing of an original document, and performs image processing on the read original image.
The document image processing unit 222 performs image processing to prepare image data by performing image processing on the document image read by the document reading unit 250. The print data expansion processing unit 224 expands print data received from the outside and prepares image data that can be provided to the engine unit 260.

As described above, the image processing device 200 may include an information extraction unit 226 that extracts information related to at least one of characters and images from image data related to a document, print data, or image data obtained by expanding the print data. The information extraction unit 226 may extract character information by applying character recognition processing to the image data, for example. Also, information related to images may be extracted by applying various appropriate image processing techniques. In this case, the control unit 210 also includes an extracted information providing unit 218. However, the data storage device (server) may include a server information extraction unit 326, which is the basic embodiment (embodiment 1). An embodiment 2 is an embodiment in which the image processing device includes the information extraction unit 226 and the extracted information providing unit 218.
Similarly, below, the basic configuration will be referred to as embodiment 1 of the present disclosure, and modified examples will be referred to as different embodiments with additional explanations added each time.

The server 300 serving as a data storage device includes a server communication unit 302 , a server data storage unit 304 , and a server control unit 310 .
The server communication unit 302 is a circuit for allowing the server 300 to communicate with external devices such as the image processing apparatus 200 via wired communication such as Ethernet or wireless communication such as wireless LAN or Bluetooth.
The server data storage unit 304 includes a large-capacity storage device for storing the database 306 and other data, and the storage device may be an HDD or SSD.
The server control unit 310 is a circuit mainly composed of a processor and a memory as hardware resources. The server control unit 310 includes a related information search unit 312 and a server information extraction unit 326. The server control unit 310 may further include a server related information providing unit 314. The processor executes a control program stored in the memory to realize the functions of the related information search unit 312, the server related information providing unit 314, and the server information extraction unit 326.

Figure 3 is a block diagram showing an example of the configuration of an image processing device different from that shown in Figure 2. In Figure 2, the database 306 is located in the server 300, and therefore the server information extraction unit 326 can be located on the server 300 side. In other words, either a configuration in which the information extraction unit 226 is located in the image processing device 200 (embodiment 2) or a configuration in which the server information extraction unit 326 is located on the server 300 side (embodiment 1) is possible. In contrast, in the configuration shown in Figure 3, the database is located in the data storage unit of the image processing device 200 (embodiment 3). Accordingly, a configuration in which the configuration corresponding to the information extraction unit 226 is located outside the image processing device 200 is not shown in Figure 3. Components corresponding to those in Figure 2 are given corresponding symbols.

<Data transmission between devices>
Data exchanged between devices will be described with reference to Figures 1 and 2. In the present disclosure, image data related to an original or print data is generated by an image processing device. For example, Figure 1 shows a case in which an image processing device 200, which is a multifunction device, executes a scan job to read an original, generates image data, and transmits the scan data (image data) to a user's information processing device 401 (smartphone). That is, an arrow in Figure 1 indicates the output of scan data from the image processing device 200, and an arrow in Figure 1 indicates the reception of scan data by the information processing device 401.

Furthermore, when a search using the extraction information is permitted, the source image data from which the extraction information is extracted is sent from the image processing device 200 to the server 300. In a mode in which the image processing device 200 includes the information extraction unit 226, the extraction information is sent together with the source image data.
The server 300 searches for related information using the extracted information and sends the search results to the image processing device 200. When a selection is made in the image processing device 200 to send the related information to the user's information processing device (e.g., the information processing device 401), the related information is sent to the information processing device 401. Note that there is also a mode in which the server 300 directly provides the related information to the user's information processing device (e.g., the information processing device 401).

For example, when a user generates print data using the information processing device 400 and causes the image processing device 200 to print the print data, the print data is sent from the information processing device 400 to the image processing device 200, and the image processing device 200 executes a print job. When a search using the extraction information is permitted, image data from which the extraction information is extracted (source image data) is sent from the image processing device 200 to the server 300.
In a mode in which the image processing device 200 includes the information extraction unit 226, the extracted information is sent together with the original image data. The server 300 searches for related information using the extracted information and sends the search results to the image processing device 200. In the image processing device 200, when a selection is made to send the related information to the user's information processing device (e.g., the information processing device 401), the related information is sent to the information processing device 401. Note that there is also a mode in which the server 300 directly provides the related information to the user's information processing device (e.g., the information processing device 401).

The same applies to the other image processing devices 201, 202 and information processing devices 402, 403. However, the image processing device 201, which is a scanner and does not have the engine unit 260, does not execute a print job. Also, the image processing device 202, which is a printer and does not have the document reading unit 250, does not execute a scan job. The image processing device generates print data and causes the image processing device to execute a print job, or receives scan data (image data) generated by a scan job. When a search using the extracted information is permitted, it is preferable that related information related to the job is provided to the information processing device. However, related information may be provided to an information processing device that is not used to execute the job. For example, a user who executed a copy job may select the information processing device used by the user as the destination of related information, rather than the operation unit of the image processing device.

<Flowchart>
An example of a processing procedure of the related information providing method according to the present disclosure will be described below with reference to Fig. 7 and a flowchart. First, an example of a process in which an image processing device executes a job and provides an image related to the job to a data storage device will be described. Fig. 4 is a flowchart showing an example of a process in which the image processing device shown in Fig. 1, more specifically, the control unit 210 shown in Fig. 2, executes a job. Fig. 7 is an explanatory diagram showing an example in which the image processing device shown in Fig. 1 executes a scan job and the scan data and related information related to the job are provided to an information processing device.

When the control unit 210, acting as the job execution unit 214, is in a state in which a job can be started (Yes in step S11 shown in FIG. 4), it refers to the job queue and searches for jobs that can be executed without conflicting resources such as document reading, printing, and memory, starting from the first job registered in the job queue (i.e., the job with the highest priority). First, it determines whether there is a job related to document reading that can be executed (step S13). Examples of jobs related to document reading include a copy job that prints scanned image data and a scan job that stores image data. Further examples include an image transmission job that transmits image data to an external device by email or the like, and a facsimile transmission job that transmits image data as a facsimile. However, the examples are not limited to these.

If there is no executable job related to reading a document (No in step S13), the job execution unit 214 then determines whether there is an executable job related to printing (step S15). Examples of jobs related to printing include a print job that acquires print data from an external source and prints it, and a facsimile reception job that prints received facsimile data. However, the job is not limited to these. A copy job is also a job related to printing, but in the example shown in FIG. 4, it is determined in step S13 to be a job related to reading a document.

If there is no executable job related to printing (No in step S15), the job executing unit 214 returns the process to the above-mentioned step S11 and waits for a job to be registered in the job queue.
If it is determined in the above-mentioned step S13 that there is an executable job and that the job is related to reading an original, the control unit 210 as the job execution unit 214 acquires user identification information related to the user authentication performed when accepting the execution request of the job and registering it in the job queue (step S17). The user identification information is information that identifies which user the job belongs to, and a typical example is a user ID and a password. However, it is not limited to this. The user identification information is not limited to information that identifies an individual user, but may be information that identifies a specific department or group, for example. In addition, for example, when the image processing device is a small machine used by an individual at a desk side, the image processing device itself can be said to correspond to an individual user, so that the user identification information may not be used as information related to the job. In that case, the process of step S17 is omitted.

The control unit 210, functioning as the job execution unit 214, reads images of one or more documents related to the job. The control unit 210, functioning as the document image processing unit 222, processes the read document images to generate image data (step S19).
Next, the control unit 210 as the search use permission selection unit 216 allows the user to select whether or not to perform a search using the extracted information obtained by extracting information related to at least one of characters and images from the generated image data (step S21). Fig. 8 is an explanatory diagram showing an example of a search use confirmation screen that the search use permission selection unit 216 causes the operation unit 240 to display in the process of step S21. The search use confirmation screen 460 shown in Fig. 8 displays a confirmation message 461 asking "Do you agree to collect information used for search from this manuscript?" and accepts the operation of either a [Do not agree] key 462 or an [Agree] key 464 as a user's choice in response to the message.

When the operation control unit 212 recognizes that the [Do not agree] key 462 has been touched, the search usage availability selection unit 216 proceeds to step S27 shown in Figure 4, executes the job without extracting information from the image data (step S27), and ends the process.
On the other hand, when the operation control unit 212 recognizes that the [Agree] key 464 has been touched on the search use confirmation screen 460 shown in FIG. 8, the search use permission selection unit 216 performs the following process.
In the example shown in Fig. 2, in the case where the image processing device 200 includes the information extraction unit 226 (embodiment 2), the control unit 210 as the information extraction unit 226 extracts information related to at least characters and images from the generated image data (step S23). In the basic configuration (embodiment 1) in which the server information extraction unit 326 extracts information, the information extraction unit 226 does not exist, and is therefore shown by a dashed line in Fig. 2. Furthermore, the process of step S23 in Fig. 4, which corresponds to the process of the information extraction unit 226, is also shown by a dashed line. The dashed lines have the same meaning for other components and processes.

Next, the control unit 210 functioning as the extracted information providing unit 218 links the image data to the user identification information and transmits it to the server 300 (step S25). In the case of the second embodiment, the extracted information is further linked to the image data and transmitted.
7 shows how image data is output as scan data Ds from the image processing device 200, sent to the destination information processing device 401, and a scan image 450 is displayed on the screen of the information processing device 401. It also shows how the scan data Ds is sent to the server 300 as an extraction source image.

A case will now be described in which it is determined in step S15 that an executable job is available and that the job is related to printing. The control unit 210, functioning as the job execution unit 214, acquires user identification information related to the user authentication performed when accepting the execution request for the job and registering it in the job queue (step S31). The user identification information is the same as that described in the process of step S17 above.
The control unit 210 acting as the job execution unit 214 develops the print data relating to the job to generate image data (step S33).

The control unit 210, functioning as the search usage permission selection unit 216, then allows the user to select whether or not to perform a search using extracted information, which is information related to at least characters and/or images extracted from the generated image data (step S35). The processing of the search usage permission selection unit 216 and the search usage confirmation screen 460 are the same as those already described in steps S21 to S25. If a search using the extracted information is permitted, the image data is linked to the user identification information and transmitted to the server 300 (step S39). In the case of the second embodiment, the extracted information is further linked to the image data and transmitted. The job is then executed (step S41), and the process ends.

The above is an example of the process for executing a job, which is shown in the flowchart of FIG.
It has been stated that the search usage confirmation screen 460 is displayed on the operation unit 240 of the image processing device 200 by the search usage availability selection unit 216, however, this is not limited to this and it is also possible to display a screen similar to the search usage confirmation screen 460 on an information processing device used by the user to allow the user to make a selection (embodiment 4).
As another embodiment, once a setting is accepted on the search use confirmation screen relating to the processing of steps S21 and S31 shown in Fig. 4 (see Fig. 8), the search use permission selection unit 216 may store the setting in the data storage unit 230 in association with the user identification information. The next time a job is executed, the setting stored in association with the user identification information is read out and processing is performed according to the setting. In this way, it becomes unnecessary for the user to select whether or not to agree to the use of the extracted information each time a job is executed.
Upon initial operation, or after the user deletes the settings stored in the data storage unit 230 by performing a specified operation, the search use availability selection unit 216 causes a search use confirmation screen such as that shown in FIG. 8 to be displayed on the operation unit 240 and accepts the user's settings (embodiment 5).

Next, the processing of the server 300 acting as a data storage device will be described.
Fig. 5 is a flow chart showing an example of a process in which the server 300 shown in Fig. 2, specifically the server control unit 310, searches for related information. As shown in Fig. 5, when the server control unit 310 receives image data from the image processing device 200 (step S51), the server control unit 310 as the server information extraction unit 326 extracts information related to at least one of characters and images from the received image data (step S53). Note that the image data received here is the image data provided to the server 300, which is a data storage device, in the process of step S25 or step S39 shown in Fig. 4.
In the second embodiment, instead of the server information extraction unit 326 of the server 300, the information extraction unit 226 of the image processing device 200 extracts information from image data and provides the extracted information linked to the image data, so that the processing of step S53 in FIG. 5 is omitted.

Next, the server control unit 310 as the related information search unit 312 preferably determines tags to be used in the search using the extracted information. A tag is known as information that serves as a mark attached to computer data to identify its structure or contents.
The tags according to the present disclosure are search tags, and when performing a search in which tags corresponding to common characteristics of search targets (objects) are added to those characteristics, objects with common characteristics (tendencies) are extracted by searching for objects to which a specific tag is added. This allows searches to be performed accurately in a short time.

The tag may be added to the entire image data, or to a part of it, such as a specific page, a specific location, a specific document, or a specific image contained therein. Therefore, the target of the search may be the entire image data, but is not limited to this, and may be a part of it.
According to this aspect, by adding tags representing the attributes to images related to jobs that each user has performed up to that point and registering them in the database, images of past jobs related to the tags of images related to the next job to be performed can be found as related information. Since related information is searched for based on the history of jobs performed by each user, highly relevant information that matches the tendencies of each user can be provided.

Regarding which tags are added or not added to which extracted information, for example, tags that are frequently used among those published in the service or site may be adopted, and the tags may be made deletable, editable, or addable. The person who can delete, edit, or add tags may be at least one of the designer, manager, and user of the related information search system.
It is possible to use the extracted information as tags as is, but adding too many tags increases the burden on the search process, so it is preferable to select and/or integrate the extracted information to use as tags for search.

The database 306 stores the extraction source image data to which tags have been added and which is linked to the user recognition information. The related information search unit 312 searches for data registered in the database 306 using search tags determined from image data related to the manuscript or print data of the executed job. That is, data to which related search tags have been added is searched for (step S57). Through the search, related information related to the current job is found from data registered in the database for jobs previously executed by the user.
According to this aspect, it is possible to search an already registered database using the tag of the extraction source image data linked to the user identification information as a clue to find and provide related information for each user.
FIG. 7 shows an example in which related information Dr 1 is found from among the data registered in the database 306 .

In addition, the related information search unit 312 may search for external data in addition to searching the database 306 (Sixth embodiment). For example, data on the Internet may be searched for. The search may be performed using a known search engine.
According to this aspect, in addition to the database related to jobs executed by the user up to that point, other external data can be searched to find related information related to the extracted information.
FIG. 7 also shows an example in which related information Dr2 is found from external data in the sixth embodiment in which external data other than the database 306 is searched for using a search tag.

For example, the source image data is a newspaper article about personnel affairs regarding the appointment of a president of a company. The server information extraction unit 326 extracts keywords related to the name of the newspaper that published the article, the name of the newspaper company, the type of article, etc. from the source image data. Furthermore, it extracts information such as the company name and the president name published in the article. It also extracts photo images published in the article. The related information search unit 312 uses the extracted information to determine tags for search. For example, it adds the newspaper company name, article classification (company personnel), company name, person's name, and photo images as search tags. Then, it searches the database 306 using the tags added to the source image data to find related information Dr1. It also searches the Internet using the tags to find related information Dr2.
The basic flow (first embodiment) does not include a configuration for using a search tag to search for data other than the database 306. Therefore, in the first embodiment, only the related information Dr1 is found, and the related information Dr2 is not found.

The basic configuration of the present disclosure is a mode in which the server control unit 310 transmits the search results of related information to the image processing device 200, and the related information providing unit 219 of the image processing device receives the search results and provides the related information to the user. Fig. 7 shows an example in which the found related information Dr1 and Dr2 are sent from the server 300 to the image processing device 200.
However, there may also be a mode in which the server 300 provides the related information directly to the user (embodiment 7). In this case, the server 300 includes a server related information providing unit 314. Then, it is determined whether the destination of the search results has been sent to the server 300 together with the image data in step S51 (step S59). If the server control unit 310 has acquired the destination of the search results (Yes in step S59), the server related information providing unit 314 transmits the search results to the user (step S61). The destination is, for example, an information processing device such as a smartphone or PC used by the user. Then, the process ends.

On the other hand, if it is determined in step S59 that the server control unit 310 has not acquired the destination to which the search results are to be provided (No in step S59), the server control unit 310 transmits the search results to the image processing device 200 (step S63) and ends the process. The same applies to the case where the server control unit 310 does not include the server-related information providing unit 314 (basic configuration of the first embodiment).
The above is an example of the related information search process shown in FIG.
Next, the process of the image processing device 200 that has received the related information from the server 300 will be described.

Figure 6 is a flowchart showing an example of a process in which the image processing device 200 provides the user with related information received from the server 300. As shown in Figure 6, when the search result for related information is received from the server 300 (Yes in step S71), the control unit 210, functioning as the related information providing unit 219, causes the operation unit 240 to display the received related information. Here, the related information providing unit 219 is provided as one or more pieces of information. When multiple pieces of related information are provided, the related information providing unit 219 causes the headings of the received related information to be displayed. Since not all related information is necessarily useful information for the user, the user is allowed to select the related information required. In addition, the mode in which the selected related information is to be provided to the user is accepted (step S73).

9 is an explanatory diagram showing an example of a related information selection screen on which headings of related information are displayed on the operation unit 240 and selection of related information is accepted. The related information selection screen 470 may be displayed as a pop-up on a screen related to job operations.
The related information selection screen 470 shown in Fig. 9 has a related information display field 472 that displays the headings of related information, selection buttons 473 that accept the selection of each piece of related information, and an [OK] key 474. The selection buttons 473 are arranged to the left of each piece of related information and individually accept whether or not it is selected. Related information with a black circle in the selection button 473 indicates that it has been selected and output. Related information with a white circle in the selection button 473 indicates that it is not yet selected. When the user touches a non-selected selection button 473, the related information providing unit 219 recognizes that the related information has been selected, and causes the output selection screen 480 shown in Fig. 10 to be displayed.

The output selection screen 480 is a screen that accepts the user's selection of the mode of outputting the selected related information. As shown in Fig. 10, the output selection screen 480 has a [Send mail] key for sending the related information by mail to an address specified by the user, and a [Display screen] key 483 for displaying the related information on the screen. In addition, the output selection screen 480 has a [Save data] key 484 for saving the related information in a USB memory connected to a USB connector (not shown) of the image processing apparatus 200, and a [Print] key 485 for printing the related information. In addition, the output selection screen 480 has a [Return] key 486 for returning the screen to the related information selection screen 470.
When one or more pieces of related information are selected by the user, the mode of providing the selected pieces of related information to the user is accepted. The mode of providing the selected pieces of related information may be accepted for each piece of related information or may be accepted all at once. In the flowchart shown in FIG. 6, the mode of providing the selected pieces of related information is accepted for each piece of related information.

The related information providing unit 219 determines whether printing of the related information has been selected as the mode of provision for any of the selected related information (step S75). When the [Print] key 485 on the output selection screen 480 is touched, the related information providing unit 219 recognizes that printing has been selected. If printing has been selected (Yes in step S75), the related information providing unit 219 registers a job to print the target related information in the job queue (step S77) and proceeds to step S91.

In step S91, the related information providing unit 219 may register the output mode of the related information selected by the user as a history related to the user (step S91). This is the eighth embodiment. However, this is not included in the basic configuration (first embodiment). Based on the history of the output mode of the related information, the order in which the option keys are arranged may be rearranged when the output selection screen 480 is subsequently displayed. Also, the initial value of the mail destination when send is selected may be presented based on the history, and it may be determined whether or not to present the initial value based on the history. If the rate at which send is selected is high and the rate at which mail is sent to the same destination is high, the initial value may be presented.
According to this aspect, not only can the user be allowed to select the manner in which related information is provided to the user, but also the manner in which the user received the related information can be kept as history.

Next, the related information providing unit 219 judges whether or not all the selected related information has been provided to the user (step S93). If not all the related information has been provided to the user (No in step S93), the process returns to the judgment in step S75.
If printing is not selected in the determination of step S75 described above (No in step S75), it is next determined whether or not transmission of related information is selected as the mode of provision (step S79). When the [Email transmission] key 482 on the output selection screen 480 is touched, the related information providing unit 219 recognizes that transmission is selected. If transmission of related information is selected (Yes in step S79), the related information providing unit 219 transmits the target related information to the specified destination (step S81), and proceeds to step S93. The destination may be requested to input from the user when transmission of related information is selected. Alternatively, when performing user authentication for execution of a job or when a search for related information using extracted information is permitted (steps S21 and S35 shown in FIG. 4), the input of the destination may also be accepted. FIG. 7 shows an example in which transmission of related information Dr1 and Dr2 is selected, the related information Dr1 and Dr2 are sent from the image processing device 200 to the information processing device 401, which is the destination, and the scanned image 450 is displayed on the screen of the information processing device 401.

If the transmission of the related information is not selected in the determination of step S79 described above (No in step S79), it is next determined whether or not storage of the related information is selected as the mode of provision (step S83). When the [Save Data] key 484 on the output selection screen 480 is touched, the related information providing unit 219 recognizes that storage has been selected. If storage of the related information has been selected (Yes in step S83), the related information providing unit 219 stores the target related information as data in the specified storage destination (step S85), and the process proceeds to step S93.
If the storage of related information is not selected in the determination of step S83 described above (No in step S83), it is next determined whether display of related information is selected as the mode of provision (step S87). When the [Screen Display] key 483 on the output selection screen 480 is touched, the related information providing unit 219 recognizes that display has been selected. If display of related information is selected (Yes in step S87), the related information providing unit 219 causes the operation unit 240 to display the target related information (step S89), and the process proceeds to step S93.

If display of related information is not selected in the determination of step S87 described above (No in step S87), the related method providing unit 219 proceeds to step S93 to determine whether all selected related information has been output. If any selected related information has not yet been output (No in step S93), the related information providing unit 219 proceeds to step S75 described above.
If it is determined in the aforementioned step S93 that all the related information has been output (Yes in step S93), the related information providing unit 219 then determines whether or not the related information selection screen 470 has disappeared due to the [OK] key 474 on the related information selection screen 470 being touched (step S95).

If the related information selection screen 470 has been removed (Yes in step S95), the process ends. If the related information selection screen 470 has not yet been removed (No in step S95), the related information providing unit 219 advances the process to step S75 described above.
As described above, the user can select from among printing, transmitting, and storing as the mode for providing each piece of related information to the user.
When all the selected related information has been provided (Yes in step S93), the process ends.
The above is the process relating to the provision of related information shown in FIG.

<<Variations>>
Here, some variations on the above-described process flow will be described.
In steps S21 and S35 shown in FIG. 4, when the user is prompted to select whether or not to use the extracted information in a search, the target of whether or not to use the extracted information in a search may be specified in detail. For example, the user may be asked whether or not to use the extracted information related to characters and the information related to images in a search separately. In this case, instead of the search use confirmation screen 460 shown in FIG. 8, the search use confirmation unit 216 displays a search use confirmation screen 490 shown in FIG. 11. The search use confirmation screen 490 shown in FIG. 11 has three keys in addition to the [Do not agree] key 492 similar to the search use confirmation screen 460 in FIG. 8. These are an [Agree to all] key 493, an [Agree to images only] key 494, and an [Agree to text only] key 495, which correspond to the [Agree] key 464 in FIG. 8.
According to this aspect, it is possible to accept a user's selection as to whether or not to use the extracted information, with respect to either character-related information or image-related information of the extracted information.

When the user is prompted to select whether or not to perform a search using the extracted information, the selection may be accepted for each type of job.
According to this aspect, it is possible to accept a user's selection regarding whether or not the extracted information can be used for each type of job, such as a copy job, a print job, or a scan job.
Alternatively, when the user is prompted to select whether or not to perform a search using the extracted information, the user may be allowed to select whether or not to specify a job or a page.
According to this aspect, it is possible to accept a user's selection regarding whether or not to allow use of extracted information for a specific job, or for one or more specific pages of a job, for example.

Furthermore, even if a search using the extracted information is permitted in steps S21 and S35 shown in Fig. 4, the permission for use may be confirmed again if the extracted information extracted in step S53 shown in Fig. 5 according to the selection includes specific information registered in advance. In the second embodiment, the extracted information extracted in steps S23 and S37 shown in Fig. 4 includes specific information. Examples of the specific information include character strings such as "confidential", "secret", and "top secret", as well as images such as a specific seal or a specific signature.
The server information extraction unit 326 (information extraction unit 226 in the case of embodiment 2) judges whether the extracted information contains one or more pieces of information that have been registered in advance. If it is judged that any of the registered information is included in the extracted information, it notifies the control unit 210 of the image processing device 200. Upon receiving the notice, the control unit 210 reconfirms with the user. The person who can register specific information may be at least one of the designer, administrator, and user of the related information search system. It is also preferable that the registered information can be added, edited, and deleted.

12 is an explanatory diagram showing an example of a search use reconfirmation screen that the control unit 210 causes the operation unit to display at that time. The search use reconfirmation screen 500 shown in FIG. 12 has a specific information display field 502, an [Agree] key 503, and a [Disagree] key 504.
The specific information display field 502 in Fig. 12 indicates that it has been determined that the extracted information contains the preregistered character string "confidential." This is presented to the user, who is then prompted to select whether or not to perform a search using the extracted information. If the [Agree] key 503 is touched, a search using the extracted information is continued. If the [Disagree] key 504 is touched, the job is executed and terminated without a search being performed.
According to this aspect, when the extracted information includes specific information that has been registered in advance, such as character strings such as "confidential" or "company-only", data protection can be made more reliable by asking the user to reconfirm permission to use the extracted information.

<First aspect of natural language processing using extracted information>
In recent years, technology using natural language processing has been attracting attention. Natural language processing refers to a technology in which a computer processes words written or spoken by a person. This is because the remarkable progress of artificial intelligence (AI), especially deep learning, has made advanced natural language processing possible. In natural language processing, a sentence is input into a "language model" consisting of a large amount of sentences prepared in advance, and the next language is predicted by an artificial intelligence using the language model. This learning is repeated to grow the language model, and the computer is made to process natural words written or spoken by a person, resulting in a situation in which the computer understands the words written or spoken by a person and responds. In this embodiment, a mode in which natural language processing is performed on extracted information extracted from image data related to image processing to obtain a result is described.

Figure 13 is a block diagram showing an example of a configuration in which an image processing device 200 similar to that in Figure 2 is connected to a server 300 that provides a natural language processing service. Components corresponding to those in Figure 2 are given the same reference numerals as in Figure 2. Figure 13 differs from Figure 2 in that the image processing device 200 includes a natural language processing usability setting unit 216N instead of the search usability selection unit 216 in Figure 2. It also includes a processing result acquisition unit 211N that acquires the results of natural language processing not shown in Figure 2, and further includes an optional user authentication processing unit 280 not shown in Figure 2. It also includes a natural language processing service providing unit 330N instead of the related information search unit 312 in Figure 2, and further includes a processing result providing unit 332N as an essential component instead of the optional server related information providing unit 314 in Figure 2.

The natural language processing availability setting unit 216N accepts and stores a setting as to whether or not to permit the extraction information to be provided to the natural language processing service. The processing result acquisition unit 211N acquires the result of the natural language processing from the processing result provision unit 332N of the server 300. The optional user authentication processing unit 280 performs a process of authenticating a user who gives an instruction related to the execution of a job to the image processing device 200. The natural language processing service provision unit 330N of the server 300 executes natural language processing using the extraction information provided from the extraction information provision unit 218 of the image processing device 200 or from the server information extraction unit 326. The processing of the natural language processing service may be included in the natural language processing service provision unit 330N, but may also be in a form in which an external natural language processing service is used via the natural language processing service provision unit 330N. The processing result provision unit 332N provides the result of the natural language processing by the natural language processing service provision unit 330N or via the natural language processing service provision unit 330N to the processing result acquisition unit 211N of the image processing device 200.

Next, an example of the processing procedure for a job related to natural language processing will be described. Figures 14 and 15 are a first flowchart showing an example of processing in which the image processing device shown in Figure 13 executes a job related to natural language processing. In Figures 14 and 15, the left side shows processing by the control unit 210 of the image processing device 200, and the right side shows processing by the server control unit 310 of the server 300. In Figure 14, the control unit 210 of the image processing device 200 displays the job selection screen 160 shown in Figure 16 on the operation unit 240, and accepts the user's selection of a job to be executed by the image processing device 200 (step S109).

Figure 16 is an explanatory diagram showing an example of a job selection screen displayed on the operation unit 240 of the image processing device 200. The job selection screen 160 has a job icon display area 161 in which icons for accepting job selection are arranged. Below the job icon display area 161 is a function icon display area in which function icons for accepting settings of various functions are arranged. As shown in Figure 16, when the [natural language processing service] icon 162 arranged in the job icon display area 161 is touched (Yes in step S111), the control unit 210 as the natural language processing availability setting unit 216N responds by executing the following process. That is, the extracted information provision confirmation screen 170 shown in Figure 17 is popped up on the operation unit 240 (step S124). On the other hand, when another icon, that is, a job icon or function icon other than the [natural language processing service] is selected (No in step S111), the control unit 210 executes the process corresponding to the selected icon (step S115) and then returns the process to the above-mentioned step S111. If no icon is selected and the operation is awaited (No in step S113), nothing is done and the process returns to step S111 described above.

The extracted information provision confirmation screen 170 displayed in response to the operation of the [Natural Language Processing Service] icon 162 includes a confirmation message 171 asking "Do you agree to provide the information contained in this manuscript to the natural language processing service?", a [Disagree] key 172, and an [Agree] key 173. The control unit 210, acting as the natural language processing usage permission setting unit 216N, displays the extracted information provision confirmation screen 170 and accepts a confirmation operation by the user (step S124). If the [Disagree] key 172 is operated (No in step S131), the control unit 210 returns the process to the aforementioned step S111 without starting execution of a job related to the natural language processing service.

When the [Agree] key 173 is operated (Yes in step S131), the control unit 210 as the image processing unit 220 acquires image data related to the job that the user instructs the image processing device 200 to execute, that is, scanned data or print data of the document (step S141). Since execution instructions and image data generation related to jobs of the image processing device are known, details will be omitted, but the image processing unit 220 acquires print data related to a print job and scan data related to a scan job or copy job. Then, in a mode in which the image processing unit 220 has an information extraction unit 226, information is extracted from the acquired image data (step S143 shown in FIG. 15). In one example, character recognition processing is applied to the image data to extract character information. Then, the control unit 210 as the extracted information providing unit 218 transmits the information extracted from the image data to the server 300 that provides natural language processing services (step S145).

When the server control unit 310 of the server 300 receives the extraction information from the image processing device 200 (Yes in step S161 shown in FIG. 15), it executes a natural language processing service for the received extraction information as the natural language processing service providing unit 330N (step S163). As the processing result providing unit 332N, the server control unit 310 transmits the result obtained by executing the natural language processing service to the image processing device 200 (step S165). Unlike the aspect in which the image processing device 200 has the information extraction unit 226, in the case of an aspect in which the server 300 has the server information extraction unit 326, the image processing unit 220 may transmit image data to the server 300 instead of the extraction information. When image data is received from the image processing device 200 (Yes in step S161), the server control unit 310 as the server information extraction unit 326 extracts information from the received image data. In one example, character recognition processing is applied to the image data to extract character information. Then, the extracted extraction information is provided to the natural language processing service providing unit 330N. The natural language processing service providing unit 330N executes a natural language processing service for the provided extracted information (step S163). The server control unit 310, acting as the processing result providing unit 332N, transmits the results obtained by executing the natural language processing service to the image processing device 200 (step S165).

When the image processing device 200 receives the results of the natural language processing from the server 300 (Yes in step S147 shown in FIG. 15), the control unit 210, functioning as the processing result acquisition unit 211N, executes the following process. That is, in accordance with settings received in advance or with user instructions received on a screen not shown, the received results are printed, transferred to an external device, or stored in the data storage unit 230 (step S149). Then, the job related to the natural language processing service is terminated. This is the first mode of performing natural language processing using extracted information.

<Second aspect of natural language processing using extracted information>
A different example of a processing procedure for a job related to natural language processing will be described. As natural language processing services, various service providers provide different services, and the number of such services is expected to increase in the future. In this second aspect, a plurality of natural language processing services are provided, and the image processing device 200 can set which natural language processing service to use. FIGS. 18 and 19 are flowcharts showing an example of processing different from that shown in FIGS. 14 and 15. As in FIGS. 14 and 15, the left side shows processing by the control unit 210 of the image processing device 200, and the right side shows processing by the server control unit 310 of the server 300. Processing similar to that shown in FIGS. 14 and 15 is denoted by the same reference numerals. Below, the processing procedure will be described mainly with respect to the points different from those shown in FIGS. 14 and 15.

In the flowchart shown in FIG. 18, when the [Natural Language Processing Service] icon is touched in the same way as in FIG. 14 (Yes in step S111), the control unit 210 performs the following process as the natural language processing availability setting unit 216N. That is, it refers to the setting of the natural language processing service to be used and whether or not to request user confirmation regarding providing the extracted information to each natural language processing service (step S125). This setting is set in advance by, for example, the administrator of the image processing device 200, and the setting is stored in the data storage unit 230. FIG. 20 is an explanatory diagram showing an example of a connection destination setting screen 520 that accepts the setting by the administrator described above via the operation unit 240.

In the example shown in FIG. 20, the connection destination setting screen 520 has a connection destination setting check box 521 for each connection destination, a connection destination setting field 522, and an extracted information provision confirmation setting field 523. The connection destination with number 001 has a check in the connection destination setting check box 521. This indicates that it is selected as the natural language processing service to be used when the [Natural Language Processing Service] icon is touched. In contrast, the connection destination with number 002 has an unchecked connection destination setting check box 521. This indicates that it is not selected as the natural language processing service to be used when the [Natural Language Processing Service] icon is touched. The URLs for accessing the natural language processing services with numbers 001 and 002 are entered in the connection destination setting field 522.

The confirmation setting drop box 524 of the number 001 is set to "confirm". If this setting is made (Yes in step S127), the control unit 210 as the natural language processing usage permission setting unit 216N displays the extraction information provision confirmation screen shown in FIG. 17 on the operation unit 240 (step S129) and provides the extraction information to use the natural language processing service only if the user's permission is obtained. On the other hand, the confirmation setting drop box 525 of the number 002 is set to "not confirm". If it is set to use the natural language processing service of the number 002, the control unit 210 as the natural language processing usage permission setting unit 216N assumes that the provision of the extraction information is permitted. Then, without displaying the extraction information provision confirmation screen shown in FIG. 17 on the operation unit 240, the extraction information is provided to the natural language processing service of the number 0002 to use the natural language processing service. This mode corresponds to the process flow that proceeds to step S141 shown in FIG. 19 via No in step S127 in FIG. 18.

Although the connection setting screen 520 in FIG. 20 shows only two natural language processing services, this is merely one example. Settings relating to a greater number of natural language processing services may be accepted. As can be seen from the fact that the natural language processing service to be used is set using a check box, the number of natural language processing services to be used is not limited to one. It is possible to use multiple natural language processing services simultaneously by checking multiple natural language processing services. This is the second mode of performing natural language processing using extracted information.

<Third aspect of natural language processing using extracted information>
Next, a further different example of the processing procedure of a job related to natural language processing will be described. In this third aspect, settings related to the use of the natural language processing service are made for each user, and the settings are stored in advance in a data table. An example is shown in a user table 530 in FIG. 24 and a group table 540 in FIG. 25. In addition, in the third aspect, the image processing device 200 is provided with a user authentication processing unit 280 so that the user who refers to the settings can be specified. FIGS. 21 to 23 are flowcharts showing an example of processing different from that in FIGS. 18 and 19. In FIGS. 21 to 23, the left side shows processing by the control unit 210 of the image processing device 200, and the right side shows processing by the server control unit 310 of the server 300. The same processes as those in FIGS. 18 and 19 are denoted by the same reference numerals. Below, the processing procedure will be described mainly with respect to the points different from those in FIGS. 18 and 19.

21, the control unit 210 as the user authentication processing unit 280 first displays a user authentication screen (not shown) on the operation unit 240 for a user who is going to use the image processing device 200 and accepts operations related to user authentication (step S101). Any means of user authentication may be used, but for example, the input of a user name and password determined for each user may be accepted on the user authentication screen. When an input related to user authentication is accepted (Yes in step S103), the control unit 210 as the user authentication processing unit 280 determines whether the input matches that related to a previously registered user (step S105). If the input does not match that related to a previously registered user (No in step S105), the control unit 210 as the user authentication processing unit 280 displays a message indicating that user authentication has failed on the operation unit 240 (step S107). Then, the process returns to the above-mentioned step S101, and accepts user authentication input again.

If the input matches that of a pre-registered user (Yes in step S105), the control unit 210 performs the following process as the user authentication processing unit 280 in the same manner as in FIG. 18. That is, the job selection screen 160 shown in FIG. 16 is displayed on the operation unit 240, and the user's selection of a job to be executed by the image processing device 200 is accepted (step S109). When the [Natural Language Processing Service] icon 162 is touched (Yes in step S111), the control unit 210 performs the following process as the natural language processing availability setting unit 216N. That is, it refers to the user table related to the user (authenticated user) who was successfully authenticated in the above-mentioned step S105 and the group table to which the authenticated user belongs (step S117).

Figure 24 is an explanatory diagram showing an example of a user table 530 referred to by the control unit 210 in step S117 described above. Figure 25 is an explanatory diagram showing an example of a group table 540 referred to by the control unit 210 in a similar manner. In this third mode, the authenticated user belongs to a group, and settings common to the group are stored in the group table 540. On the other hand, settings specific to the user are stored in the user table 530. These settings are set in advance by, for example, an administrator of the image processing device 200, and are stored in the data storage unit 230 as the user table 530 and group table 540. In the user table shown in Figure 24, items indicated by "-" indicate that the group table 540 of the group to which each user belongs should be referenced and the corresponding settings should be applied.

For example, when referring to the user table 530 in FIG. 23 for a user with a user ID of 001, the user belongs to a group with a group ID of 001. Furthermore, the settings for whether the user can execute each job of scan jobs, copy jobs, print jobs, and natural language processing services are all set to "-". In other words, they conform to the settings for the group with a group ID of 001 to which the user belongs. When referring to the group table in FIG. 24, for the group with a group ID of 001, scan jobs, copy jobs, and print jobs are all set to "○", meaning that they are executable, for both monochrome and color. Therefore, the user can execute all of these jobs. Furthermore, the AI service is set to 001. Therefore, the user can use the natural language processing service corresponding to 001 in the connection destination setting screen 520 in FIG. 20.

For example, when referring to the user table 530 in FIG. 23 for a user with a user ID of 005, the user belongs to a group with a group ID of 002. However, the user table 530 has settings for whether or not the job can be executed for each of the scan job, copy job, print job, and natural language processing service. Therefore, whether or not the job can be executed for that user is determined without referring to the group table 540. That is, since the scan job, copy job, and print job are all set to "○" for monochrome and "×" for color, only monochrome jobs are set to be executable. Color jobs are set to not be executable. Since the AI service is set to "×", the natural language processing service cannot be used.

In this way, the user table 530 and the group table 540 related to the authenticated user are referenced to obtain the settings related to the authenticated user. If the authenticated user is set to be unable to use the natural language processing service (No in step S121 shown in FIG. 22), the control unit 210 as the natural language processing usability setting unit 216N performs the following process. That is, the operation unit 240 is caused to display a message indicating that the authenticated user cannot use the natural language processing service (step S123). Then, the process returns to step S111 shown in FIG. 21, and the job selection is accepted on the job selection screen 160. On the other hand, if the authenticated user is set to be able to use the natural language processing service (Yes in step S121 shown in FIG. 22), the control unit 210 performs the following process. That is, as in the second aspect shown in FIG. 18, the natural language processing service to be used and the setting of whether or not to request the user's confirmation about providing the extracted information to each natural language processing service are referenced (step S125). Depending on the referenced setting, confirmation is requested if necessary, and the natural language processing service is used to obtain the result. The process for using the natural language processing service from step S125 onwards is similar to that shown in Figures 18 and 19. This completes the third aspect of natural language processing using extracted information.

It should be understood that the present disclosure also includes any combination of the above-described aspects.
In addition to the above-described embodiment, various modifications of the present disclosure are possible. These modifications should not be interpreted as not belonging to the scope of the present disclosure. The invention according to the present disclosure should include all modifications that are equivalent to the scope of the claims and that belong to the scope of the present disclosure.

160: job selection screen, 161: job icon display area, 162: [natural language processing service] icon, 170: extracted information provision confirmation screen, 171: confirmation message, 172: [do not agree] key, 173: [agree] key, 200, 201, 202: image processing device, 210: control unit, 211N: processing result acquisition unit, 212: operation control unit, 214: job execution unit, 216: search usage availability selection unit, 216N: natural language processing usage availability setting unit, 218: extracted information provision unit, 219: related information provision unit, 220: image processing unit, 222: original image processing unit, 224: print data expansion processing unit, 226: information extraction unit, 230: data storage unit, 240: operation unit, 250: original reading unit, 260: engine unit, 270: communication circuit, 280: User authentication processing unit, 300: Server, 302: Server communication unit, 304: Server data storage unit, 306: Database, 310: Server control unit, 312: Related information search unit, 314: Server related information providing unit, 326: Server information extraction unit, 330N: Natural language processing service providing unit, 332N: Processing result providing unit, 400, 401, 402, 403: Information processing device, 450: Scanned image, 460, 490: Search use confirmation screen, 461: Confirmation message, 462, 492, 504: [Do not agree] key, 464, 503: [Agree] key, 470: Related information selection screen, 472: Related information display field, 473: Selection button, 474: [OK] key, 480: Output selection screen, 482:: [Email send] key, 483: [Screen display] key, 484: [Save data] key, 485: [Print] key, 486: [Back] key, 493: [Agree to all] key, 494: [Agree to images only] key, 495: [Agree to text only] key, 500: Search use reconfirmation screen, 502: Specific information display field, 520: Connection destination setting screen, 521: Connection destination setting checkbox, 522: Connection destination setting field, 523: Extraction information provision confirmation setting field, 524, 525: Confirmation setting drop box, 530: User table, 540: Group table Dr1, Dr2: Related information, Ds: Scan data, N: Network

Claims (17)

Steps executed by at least one processor that controls the image processing device and the data storage device individually or collectively,
A step of controlling the image processing device to execute at least one job related to image processing of a process of reading an original document and outputting or storing image data, or a process of developing print data and printing or storing image data;
a step of having a user select whether or not to perform a search using extracted information obtained by extracting information relating to at least one of characters and images from image data relating to the document or the print data;
If a search using the extracted information is permitted,
extracting the extracted information from the image data;
retrieving relevant information related to the extracted information from a database stored in the data storage device;
providing the related information obtained by the search to the user;
registering the extracted information and the original image data from which it was extracted in the database;
A related information providing method for providing related information related to a job, comprising: authenticating the user;
providing user identification information associated with authentication of the user to the data storage device;
The step of registering the extraction information includes determining a search tag for the extraction source image data using the extraction information, adding the search tag to the extraction source image data, and registering the extraction source image data to which the search tag has been added in the database while being linked to the user identification information;
The related information providing method according to claim 1, wherein the step of searching the database is a process of using a search tag determined from the original image data related to the job to find data registered in the database to which a related tag has been added as the related information. The related information providing method according to claim 2, wherein the search tag is information that represents an attribute related to a predetermined item when information related to at least one of characters and images included in the extracted information includes the attribute. The method for providing related information according to claim 1, wherein the step of searching for related information includes searching external data in addition to searching the database. The method for providing related information according to claim 1, wherein the step of providing the related information to the user includes having the user select one of the following processes: sending the related information to the image processing device and displaying or printing the information on an operation unit of the image processing device, or sending the related information to an information processing device previously registered for the user and displaying or storing the information as data, and storing the selection made by the user as the user's history. The related information providing method according to claim 1, wherein the step of selecting whether or not to use the extracted information accepts a selection of whether or not to use the extracted information for each type of job. The related information providing method according to claim 1, wherein the step of selecting whether or not to use the extracted information accepts a selection of whether or not to use a specified job or page. The method for providing related information according to claim 1, wherein the step of selecting whether or not to use the extracted information includes accepting a selection of whether or not to use the information related to the text or the information related to the image. The method for providing related information according to claim 1, wherein the step of selecting whether or not to use the extracted information requires the user to select whether or not to use the extracted information again if the extracted information includes specific information that has been registered in advance, even if permission to use the extracted information has been selected. the data storage device is located outside the image processing device;
2. The related information providing method according to claim 1, wherein the step of providing the extraction information and the original image data to the data storage device is a step of transmitting the extraction information and the original image data to the data storage device. the data storage device is located inside the image processing device;
2. The method according to claim 1, wherein the step of providing the extraction information and the original image data to the data storage device comprises the step of outputting the extraction information and the original image data to the data storage device. An image processing system including an image processing device and a data storage device for storing a database,
The image processing device includes:
a job execution unit that executes a job relating to at least one image processing process of reading an original document and outputting or storing image data, or receiving and converting print data, and printing or storing the image data;
a search availability selection unit that allows a user to select whether or not to perform a search using extracted information obtained by extracting information related to at least one of characters and images from image data related to the document or the print data;
a related information providing unit that provides the user with related information obtained by searching the database when a search using the extracted information is permitted;
Equipped with
The data storage device includes:
a related information search unit that searches the database for related information related to the extracted information when a search using the extracted information is permitted, and provides a search result to the image processing device;
an extraction information registration unit that registers the extraction information and the original image data from which the information is extracted in the database;
An image processing system comprising: a job execution unit that executes a job relating to at least one image processing process of reading an original document and outputting or storing image data, or receiving and converting print data, and printing or storing the image data;
a search availability selection unit that allows a user to select whether or not to perform a search using extracted information obtained by extracting information related to at least one of characters and images from image data related to the document or the print data;
A data storage unit for storing a database;
a related information search unit that searches the database for related information related to the extracted information in the data storage unit when a search using the extracted information is permitted;
a related information providing unit that provides the related information obtained by the search to the user;
an extracted information registration unit that registers the extracted information and the original image data from which the extracted information is extracted in the database when a search using the extracted information is permitted;
An image processing device comprising: a job execution unit that executes a job relating to at least one image processing process of reading an original document and outputting or storing image data, or receiving and converting print data, and printing or storing the image data;
a natural language processing availability setting unit that accepts a setting for whether or not natural language processing is to be performed using extracted information obtained by extracting information related to characters from image data related to the document or the print data;
a processing result acquisition unit that accesses an external natural language processing service to perform natural language processing using the extracted information and acquires a result from the natural language processing service;
An image processing device comprising: The image processing device according to claim 14, wherein the natural language processing availability setting unit individually accepts the user's availability settings for a plurality of natural language processing services. A user authentication processing unit that authenticates a user who receives an instruction,
The image processing apparatus according to claim 14 , wherein the natural language processing availability setting unit accepts a setting for availability of the natural language processing service for each authenticated user. A step performed by at least one processor controlling the image processing device, comprising:
A step of executing a job relating to at least any one of image processing, that is, a process of reading an original and outputting or storing image data, and a process of expanding print data and printing or storing image data;
A step of having a user set whether or not natural language processing is to be performed using extracted information obtained by extracting information related to characters from image data related to the document or the print data;
If natural language processing using the extracted information is permitted,
accessing an external natural language processing service and performing natural language processing using the extracted information;
obtaining results from the natural language processing service;
A method for performing natural language processing using extracted information relating to a job comprising: