JP7553907B2 - Character recognition device, character recognition program, fisheries management support terminal device, and fisheries management support system - Google Patents

Description

The present invention relates to a character recognition device and a character recognition program that optically recognize characters displayed on a measuring instrument, and a fisheries management support terminal device and a fisheries management support system that use the same.

A character recognition device optically reads characters written on a medium using a digital camera as a still image or video, and outputs character data corresponding to the characters.
The medium may be paper on which characters are handwritten or printed from a computer, or it may be a display screen on which characters are displayed.

For example, the image processing method, image processing program, and information management system described in Patent Document 1 apply expansion and contraction processes to a seven-segment number image contained in an image obtained by photographing the measurement result displayed on a measuring device, thereby filling in the gaps between multiple segments, making it possible to recognize measurement values with high accuracy from measurement results consisting of seven-segment number images, even when using conventional OCR software.

JP 2015-197851 A

In the conventional character recognition device described in Patent Document 1, the measuring instrument is assumed to be an RI (radioisotope) measuring instrument that measures moisture and the like contained in land during civil engineering work, but it is described that the measuring instrument may be other types of measuring instruments, and that the measuring instrument may be any measuring instrument, including, for example, a gas detector, an environmental measuring instrument such as a dust meter or a light meter, a meteorological observation instrument such as a thermo-hygrometer or a rain gauge, or a water quality/soil measuring instrument such as a pH meter or a turbidity meter.
Furthermore, in the conventional character recognition device described in Patent Document 1, it is also described that in order to prevent the user from being captured on the display screen of the measuring instrument, which becomes the captured image, a box with a hole in it can be placed over the display screen of the measuring instrument, and the display screen can be photographed with the camera unit of the mobile terminal aligned with the position of the hole.
However, this requires covering the character with a box every time a photograph is taken for character recognition, which is a cumbersome operation.
However, if you try to photograph the display screen of a measuring instrument without covering it with a box and perform character recognition, the strong sunlight outdoors can cause not only the photographer to be captured in the image, but also reflected light from various surrounding objects. Also, at night, there is a risk that light from lighting can be captured in the image. This reduces the accuracy of recognition.

The present invention aims to provide a character recognition device, a character recognition program, a fisheries management support terminal device, and a fisheries management support system that can be used in harsh environments by enabling accurate character recognition even when reflected light is caught on the display screen.

The character recognition device of the present invention is a character recognition device that optically recognizes characters from a captured image of a display screen on which characters are displayed, and is characterized by comprising: a binarization means that generates a binary image in which color data of pixels other than first color data indicating the character color of the character from the captured image is replaced with second color data; and a recognition means that recognizes characters indicated by the first color data included in the binary image and outputs the characters as character information.

The character recognition program of the present invention causes a computer to function as a character recognition device that optically recognizes characters from a captured image of a display screen on which characters are displayed, and is characterized by causing the computer to function as a binarization means that generates a binary image in which color data of pixels other than first color data indicating the character color of the character is replaced with second color data from the captured image, and a recognition means that recognizes characters indicated by the first color data included in the binary image and outputs them as character information.

According to the present invention, the binarization means generates a binary image by replacing the color data of pixels other than the first color data indicating the character color of the characters from the captured image with the second color data, so that the characters displayed on the display screen become the first color data and the reflected light becomes the same second color data as the rest, and the reflected light can be removed.

The device may be equipped with an extraction means for generating a captured image in which there is no or minimal reflected light or overlapping of obstacles on the characters by removing outliers from each pixel and synthesizing the captured images based on multiple captured images or video images taken while changing the shooting direction of the display screen.
By taking the captured images while changing the shooting direction, even if reflected light or an obstacle is superimposed on the characters in the first captured image, the reflected light or obstacle will move away from the characters to be recognized as the shooting direction moves, and will not be superimposed on the characters. Therefore, the extraction means removes outliers from each pixel from multiple captured images or videos and combines them to extract captured images with no or minimal reflected light or obstacles superimposed on the characters, thereby further improving the accuracy of character recognition.

The device may be provided with a screen area detection means for estimating the screen area of the display screen from a captured image based on a deep learning model constructed by learning the screen area in the captured image according to the shooting direction using a plurality of captured images or video images captured while changing the shooting direction of the display screen and an area-designated image in which the screen area of the captured image is designated.
A deep learning model is constructed by learning the screen area in the captured image according to the shooting direction from the captured image captured while changing the shooting direction and the area-specified image in which the screen area of the captured image is specified. Then, the screen area detection means estimates the screen area of the display screen from the captured image based on this deep learning model. Therefore, even if the shooting direction is not the front, the range of the display screen can be accurately grasped.

A recognition member for recognizing the range of the display screen is provided in a frame around the display screen, and the device can be equipped with a screen area detection means for recognizing the peripheral frame of the display screen in the captured image from the recognition member and detecting the screen area of the display screen.
The screen area detection means detects the surrounding frame and notifies the screen area of the display screen from the photographed image, so that the range of the display screen can be accurately grasped even if the display screen is not photographed from the front, the display screen is photographed distorted, or the background is reflected around the display screen.

The recognition member can be a peripheral frame formed from a retroreflective material, and the screen area detection means can detect a virtual frame formed by reflected light from the peripheral frame in the captured image, and notify the screen area surrounded by the virtual frame.
By detecting the virtual frame using light reflected from a peripheral frame formed by a retroreflective material, it is possible to detect the screen area surrounded by the virtual frame.

The recognition member can be a peripheral frame formed by colored tape, and the screen area detection means can detect a virtual frame based on the color of the peripheral frame in the captured image, and notify the screen area surrounded by the virtual frame.
By detecting the virtual frame based on the color of the peripheral frame formed by the color tape, the screen area surrounded by the virtual frame can be detected.

The recognition component can be a two-dimensional barcode provided at four corners of the frame, and the screen area detection means can recognize the position of the two-dimensional barcode in the captured image, detect a virtual frame connecting the positions of the two-dimensional barcodes, and notify the screen area surrounded by the virtual frame.
By detecting a virtual frame that connects the positions of the two-dimensional barcodes provided at the four corners of the frame, the screen area surrounded by the virtual frame can be detected.

The device may include a template designation means for reading a template from a storage means in which a template is stored that defines an area including characters to be recognized and is displayed on the display screen, and designating the template for the recognition means. Since the template designation means recognizes characters in the area designated by the template, it is possible to recognize only the characters in the required range.

The storage means stores a portrait template corresponding to a portrait image captured with the display means in a position where the length direction is along the up-down direction, and a landscape template corresponding to a portrait image captured with the display means in a position where the length direction is along the left-right direction, and the template designation means can switch between the portrait template and the landscape template depending on the shooting position. In this way, a template suitable for the shooting position can be selected.

The apparatus may further comprise a template acquisition means for acquiring the template from a character recognition support server connected via a telecommunications line in which the template is stored, and storing the template in the storage means.
Since the template acquiring means acquires the template from a character recognition support server connected via an electric communication line, the template can be the latest template or another person's template with a high character recognition rate.

The template designation means can be configured to designate to the recognition means a template selected by the input means from among a plurality of templates stored in the storage means. In this way, the operator can select a template that is suitable for character recognition.

The recognition means can generate character recognition information including the character recognition rate when a character is recognized. This can help improve the template by understanding the character recognition rate of the template used for character recognition.

The character recognition information can include the date and time the template was used. The date and time of use can be used to understand the effects of sunlight and lighting, so if the character recognition information includes the date and time the template was used, it is possible to select a template that is resistant to these effects.

The device may be provided with a recognition information notification means for notifying the character recognition information to a character recognition support server connected via a telecommunications line. The character recognition support server can collect character recognition information, improving convenience.

The recognition information notification means can include character information input to the input screen by the input means in the character recognition information and notify the user. In addition to the mechanical character recognition rate, the input means can also feed back comments as character information.

The fishery management support terminal device of the present invention may include an imaging means for imaging the display screen of a tidal current meter installed on a fishing boat on which a fisherman is aboard, a character recognition device of the present invention for optically recognizing characters from the image of the display screen to read tidal current information, a positioning means for outputting the location and date and time of the fisherman's operation as location information and date and time information based on a signal from a positioning satellite, a communication means for communicating with a fishery management support server that generates a fishing performance report or an operation logbook or both based on a list of catches indicating the amount of catches input from an input terminal device based on a list issued by a production market, the location information, the date and time information, and the tidal current information, which is an invoice listing the amount of catch obtained by the fisherman's operation, and the location information, the date and time information, and the tidal current information, and an information transmission means for transmitting the tidal current information, the location information, and the date and time information to the fishery management support server.

The fisheries management support system of the present invention may include a fisheries management support terminal device that uses the character recognition device of the present invention to optically recognize characters from a photographed image of the display screen of a tidal current meter installed on a fishing boat on which a fisherman is aboard and outputs tidal current information read from the image, and outputs the location and date and time of the fisherman's operation as location information and the date and time information based on a signal from a positioning satellite; an input terminal device that is an invoice that describes the amount of catch obtained by the fisherman's operation and is used to input the amount of catch as catch information based on an invoice issued by a production market; and a fisheries management support server that generates a fishing performance report or an operation logbook, or both, based on the tidal current information, location information, and date and time information from the fisheries management support terminal device and the catch information from the input terminal device.

According to the present invention, reflected light can be removed by the binarization means, so even if reflected light is captured on the display screen, accurate character recognition is possible, making it possible to use the device in harsh environments.

1 is a diagram showing a configuration of a fisheries management support system according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of a display screen of a tidal current meter mounted on a fishing boat. 2 is a diagram showing the configuration of a fishery management support terminal device of the fishery management support system shown in FIG. 1. 4 is a diagram showing the configuration of a character recognition means of the fishery management support terminal device shown in FIG. 3. 2 is a diagram showing the configuration of a fishery management support server of the fishery management support system shown in FIG. 1. 2 is a diagram for explaining the operation and usage state of the fisheries management support system shown in FIG. 1. 7 is a flowchart for explaining the operation of the character recognition process in FIG. 6 . FIG. 4 is a diagram of an example of an operation screen displayed on the display means of the fishery management support terminal device shown in FIG. 3, and shows the state before character recognition. FIG. 9 is a diagram of an example of the operation screen shown in FIG. 8, showing the state after character recognition. 1A to 1D are diagrams showing examples of images captured by a tidal current meter, in which (A) to (D) are diagrams showing multiple captured images, and (E) is a diagram showing a composite image obtained by combining (A) to (D). 13 is a diagram showing a state in which a two-dimensional barcode is provided as a recognition member at four corners of a frame of a display screen. FIG. FIG. 13A is a diagram for explaining binarization to remove reflected light, and FIG. 13B is a diagram for explaining a template. 2 is a diagram of an example of a fishing performance report generated by the fisheries management support server shown in FIG. 1 . 2 is a diagram showing an example of an operation log generated by the fishery management support server shown in FIG. 1.

The fisheries management support system according to the embodiment of the present invention will be described with reference to the drawings. In this specification, the term "character" is used to mean characters and symbols, and is the same as the object recognized by devices and software generally called OCR (Optical character recognition).
The fisheries management support system 10 shown in FIG. 1 supports fishermen in creating fishing reports and operation logs.

The fisheries management support system 10 comprises a fisheries management support terminal device 20 operated by fishermen on board their vessels, a fisheries management support server 30 that collects information regarding fishing operations and prepares fishing performance reports and operation logs, an input terminal device 40 for inputting catch volumes, a marine weather information providing server 50, a character recognition support server 60, and a work terminal device 70.
The fisheries management support server 30 is communicatively connected to a fisheries management support terminal device 20, an input terminal device 40, a marine weather information providing server 50, a character recognition support server 60, and a work terminal device 70 via the Internet W, which is an example of a telecommunications line.

The fisheries management support server 30 and the character recognition support server 60 are servers operated by a management company that manages the fisheries management support system 10, and can be servers installed at the management company or rental servers.
The marine weather information providing server 50 is a server operated by a meteorological data providing company that provides marine weather information.

The fishing boat in which the fishermen appear is equipped with a tidal current meter, which is an example of a measuring instrument used to record measurements as reference data during fishing operations and in fishing logbooks and catch reports. A peripheral frame F made of retroreflective material is placed as a recognition component in the frame surrounding the display screen D of the tidal current meter T shown in Figure 2.

Next, the configuration of the fishery management support terminal device 20 will be described with reference to FIG.
The fishery management support terminal device 20 is a computer that transmits operation information to the fishery management support server 30 (see FIG. 1 ). The fishery management support terminal device 20 is a tablet computer or a smartphone that is easy to carry and that fishermen already own or can easily procure.
A fishery management support program runs on the fishery management support terminal device 20. The fishery management support program runs on the fishery management support terminal device 20, causing the computer to function as each of the following means.

The fisheries management support terminal device 20 includes a communication means 21, an image capture means 22, a screen control means 23, a character recognition means 24, a positioning means 25, an information transmission means 26, an input means 27, a display means 28, and a storage means 29.

The communication means 21 communicates with a mobile phone company's relay station (not shown) via wireless communication, and outputs data received from the Internet W to each means of the fisheries management support terminal device 20, or inputs data from each means and transmits it to the Internet W (see Figure 1).
The communication means 21 may be, for example, a means for communicating with a relay station of a mobile phone company using a communication method such as 4G or 5G. Also, the communication means 21 may communicate with a communication destination in a short distance using Wi-Fi (registered trademark).

The imaging means 22 captures an image of the display screen of a tidal current meter, which is an example of a measuring instrument, and stores the captured still or moving image in the storage means 29 .
The screen control means 23 reads out the screen data from the storage means 29 and displays it on the display means 28 .
The character recognition means 24 functions as a character recognition device. The character recognition means 24 optically recognizes characters from a captured image of a display screen on which characters are displayed of the tidal current meter mounted on the fishing boat shown in Figure 1, and outputs the tidal current information read from the captured image.

The positioning means 25 has a function of determining the current location as position information based on a signal from a positioning satellite. The position information may be longitude and latitude, and may also include altitude. The positioning means 25 also has a function of outputting the current time (date and hour) obtained during positioning as date and time information. The positioning satellite may be the Quasi-Zenith Satellite System (QZSS), also known as Michibiki, or the Global Positioning System (GPS).

When the communication means 21 detects that communication with the outside has become possible, the information transmission means 26 reads out the operation information from the storage means 29 and transmits it to the fisheries management support server 30.
The input means 27 can be a touch panel provided on the front surface of the display means 28 .
The display means 28 may be an LCD panel, an organic EL panel, or the like.
The storage means 29 may be a non-volatile flash memory that can be read and written at high speed. Programs such as an OS and application software, as well as various data, are stored in the storage means 29. Current information relating to currents during operation is stored in the storage means 29. Position information indicating the operation location and date and time information indicating the start date and time of operation are stored as operation time information. Screen data is also stored in the storage means 29. Furthermore, a template is stored in the storage means 29, in which an area in which characters to be recognized are displayed (hereinafter referred to as the recognition target area) is defined.
In addition to the recognition target area, the template defines the attributes of each area (flow speed, water temperature), the background color of the display screen, the character color, the area in which the recognized characters are to be displayed, and so on.

The character recognition means 24 will now be described in detail with reference to the drawings.
The character recognition means 24 reads the tidal speed, tidal direction and tidal temperature from the display screen of the tidal current meter, and outputs the tidal speed information, tidal direction information and tidal temperature information as tidal current information.
As shown in FIG. 4, the character recognition means 24 includes a screen area detection means 241 , an extraction means 242 , a binarization means 243 , a template designation means 244 , a recognition means 245 , and an attribute assignment means 246 .

The screen area detection means 241 notifies the screen area on the display screen.
The extraction means 242 removes outliers from each pixel based on multiple captured images or video images of the tidal current meter display screen taken while changing the shooting direction, and combines them to generate an image that has no or minimal overlap of reflected light from the characters or obstacles such as a steering wheel or lever that are located between the shooting means 22 (see Figure 3) and the display screen D (see Figure 2).
The binarizing means 243 generates a binary image by replacing the color data of pixels other than the first color data indicating the character color of the characters in the captured image with the second color data.

The template designation means 244 reads out and designates a template in which a portion to be character recognized is designated on the display screen from the storage means 29. In this embodiment, the template designation means 244 uses the template stored in the storage means 29 offline, but a template may also be obtained from the character recognition assistance server 60 every time character recognition is performed and used online.
Furthermore, the template designation means 244 can display a plurality of templates on the display means 28 and select a template designated by the input means 27 from among them.

The recognition unit 245 recognizes the character represented by the first color data included in the template and outputs it as character information. The recognition unit 245 generates character recognition information including a character recognition rate when the character is recognized.
The attribute assigning means 246 assigns attributes corresponding to the template designated by the template designation means 244 to the characters recognized by the recognition means 245 .

The recognition information notifying means 247 notifies the character recognition assistance server 60 of statistical information at the time of recognition, such as the character recognition rate and the usage time (number of uses), as character recognition information. The recognition information notifying means 247 can include comments in the form of character information input by the operator through the input means 27 into an input screen (not shown) in the character recognition information.
The template acquisition means 248 acquires a template from the character recognition assistance server 60. Furthermore, if a new template for improving the character recognition rate has been set, the template acquisition means 248 acquires the new template and updates the template used by the template designation means 244.

Next, the configuration of the fisheries management support server 30 will be explained with reference to the drawings.
As shown in Figure 5, the fisheries management support server 30 includes a communication means 31, a sea area analysis means 32, a marine information acquisition means 33, a catch volume registration means 34, a report generation means 35, an operation log generation means 36, and a memory means 37.

The communication means 31 outputs data received from the Internet W to each means of the fisheries management support server 30, and inputs data from each means and transmits it to the Internet W (see Figure 1).

The sea area analysis means 32 determines the fishing area (sea area) where fishing is conducted based on location information, which is the latitude and longitude information, from the fisheries management support terminal device 20 (see Figure 1), and outputs a fishing area number corresponding to the fishing area.
The marine information acquisition means 33 acquires marine weather information from the marine weather information providing server 50 based on the position information and date and time information from the fishery management support terminal device 20 (see FIG. 1).
Marine weather information includes meteorological information such as water temperature information indicating water temperature at different depths, tidal current information indicating tidal currents (east-west and north-south components of ocean currents), sea surface altitude information indicating the sea surface altitude, salinity information indicating the salinity of seawater, wave information indicating wave height, wind information indicating wind direction and wind speed, precipitation information indicating precipitation amount, and pressure information indicating air pressure.

In this embodiment, marine weather information is obtained online via the Internet W from a marine weather information providing server 50 provided by a weather data provider, but past marine weather information can also be obtained, for example, from a CD-ROM or DVD. In this case, offline data on past weather information, "sea condition analysis data," provided by the Meteorological Operations Support Center can be used.

The catch volume registration means 34 registers the catch volume to be included in the fishing report inputted from the input terminal device 40 in the storage means 37 as catch volume information.
The report generating means 35 generates a fishing report based on information related to fishing operations (fishing time information (current information, position information, date and time information), and catch volume information). The fishing report may be in a format such as PDF (Portable Document Format).
The operation log generating means 36 generates an operation log based on information related to the operation.
The storage means 37 stores programs such as an OS and application software, and various data (information), and may be, for example, an SSD or a hard disk drive.

The input terminal device 40 shown in Figure 1 is a computer that accesses the fisheries management support server 30 and inputs the catch volume based on an invoice issued by the production market, which is an invoice that lists the catch volume obtained by fishermen through their fishing operations.
The marine weather information providing server 50 provides information on marine weather at the operation start date and time at the operation location based on the position information and date and time information.

The character recognition support server 60 receives character recognition information such as the character recognition rate from the fisheries management support terminal device 20 and provides it to the manager (developer). The character recognition support server 60 also provides a new template in response to a request from the fisheries management support terminal device 20.
The character recognition support server 60 associates templates with character recognition information including identification information for identifying the fisheries management support terminal device 20, an account assigned to the operator operating the fisheries management support terminal device 20, model number information for identifying the type of tidal current meter, a character recognition rate, etc., and stores and manages the templates in a memory means not shown.

The character recognition support server 60 can charge the account of each operator. For example, a charge can be made each time a new template is provided, or each time the fisheries management support terminal device 20 performs character recognition. When charging each time character recognition is performed, a charge can be made when the template designation means 244 checks whether or not a new template is available before character recognition, or when the recognition information notification means 247 notifies character recognition information after character recognition.

The work terminal device 70 is a terminal device for creating and editing templates.

The operation and usage of the fisheries management support system 10 according to the embodiment of the present invention configured as above will be described with reference to the drawings.
As shown in Fig. 6, a fisherman arrives at a fishing ground. Then, the fisherman operates the fishery management support terminal device 20 to start the fishery management support program. By doing so, the screen control means 23 shown in Fig. 3 reads out the screen data from the storage means 29 and displays the operation screen P1 shown in Fig. 8 on the display means 28.

An operation log is displayed on the operation screen P1. In the middle of the operation screen P1, a button B11 for starting photography, labeled "input from camera", is displayed to call up the photography means 22 for photographing the display screen of the tidal current meter.
In addition, the operation screen P1 is provided with a tide speed display field R11 which displays tide speed information indicating the speed of the tide, a tide direction display field R12 which displays tide direction information indicating the direction of the tide, and a tide water temperature display field R13 which displays tide water temperature information indicating the water temperature of the tide.

First, before fishing operations, the fisherman operates the fisheries management support terminal device 20 to photograph the display screen of the tidal current meter T using the photographing means 22 while changing the shooting direction, and obtains tidal current information by performing character recognition (see step S10 in Figure 6).
Here, the character recognition process performed by the character recognition means 24 to recognize characters displayed on the display screen and output character information will be described with reference to the drawings.

(Get template)
When a template is to be used offline, the template designation means 244 shown in Fig. 4 reads the template from the storage means 29. When a template is to be used online, the template is acquired by accessing the character recognition support server 60 shown in Fig. 1 (see step S11).
At this time, in response to a request from the fisheries management support terminal device 20, the character recognition support server 60 can return a template that is linked to the requesting fisheries management support terminal device 20 or corresponds to the operator's own account, or can return a template with a high character recognition rate that is linked to another fisheries management support terminal device 20 or corresponds to another operator's account.
In this way, the character recognition assistance server 60 provides not only the user's own template but also templates of other users with high character recognition rates, thereby making it possible to improve the character recognition rate.
The template designation means 244 can designate a template selected by the input means 27 from among a plurality of templates displayed on the display means 28, so that the operator can select a template that is suitable for character recognition.

(Photo of the display screen)
When the image capturing means 22 shown in FIG. 3 captures an image of the display screen D (see FIG. 2), the captured image is stored in the storage means 29 .
At this time, the photographing means 22 can take multiple still images or can take a video. In this embodiment, the fisherman can eliminate the trouble of taking multiple still images while changing the photographing direction by having the fisherman take a video (see step S12 in FIG. 7).

A video is made up of multiple still images called frames. First, the screen area detection means 241 finds the surrounding frame F shown in Figure 2 from the captured images G1 to G4, which are the multiple still images, as shown in Figures 10 (A) to (D).

(Screen area detection)
Here, a method for the screen area detection means 241 to detect the screen area from a captured image of the display screen will be described (see step S13).

(1) First, a method for detecting a screen area using deep learning will be described.
First, a deep learning model is constructed by having the model learn the screen area in the captured image according to the shooting direction using multiple captured images captured while changing the shooting direction of the display screen, or captured images of a video captured while changing the shooting direction of the display screen, and a region-designated image that designates the screen area of the captured image.
The area-specified image may be specified by manually filling in the screen area, or may be specified by attaching a frame indicating the screen area to the display screen and automatically filling in the inside of the frame as the screen area with a photographed image in which the screen area and the frame are photographed together.
This deep learning model is stored in memory means 29.

In this way, once a deep learning model that has learned the screen area according to the shooting direction is constructed, the screen area detection means 241 estimates the screen area of the display screen from the captured image based on the deep learning model when performing character recognition.
Then, the screen area detection means 241 converts this screen area into a rectangular screen area by projective transformation and notifies the converted area.

(2) Next, a method will be described in which a virtual frame is detected based on reflected light from a peripheral frame F formed by a retroreflective material provided in the frame surrounding the display screen D shown in Figure 2, and the screen area surrounded by the virtual frame is detected from the captured image.
Since the surrounding frame F is formed from a retroreflective material as a recognition component, light from a light-emitting means (not shown) that emits light as a flash in the fisheries management support terminal device 20 is reflected by the surrounding frame F and returns straight to the fisheries management support terminal device 20.
Therefore, the light reflected from the peripheral frame F appears as a virtual frame V in the captured images G1 to G4.
The screen area detection means 241 detects this virtual frame V, converts the captured image into a rectangular display screen D surrounded by the virtual frame V by projective transformation, and notifies the screen area.

(3) Next, a method of forming the peripheral frame F as a recognition member from colored tape in addition to forming it from a retroreflective material will be described.
When the peripheral frame F is formed by color tape, the color of the color tape, for example, the yellow peripheral frame F, appears in the captured image. Therefore, the screen area detection means 241 can detect this yellow peripheral frame F and notify the screen area of the display screen surrounded by the peripheral frame F from the captured image.

(4) Furthermore, in the case where the recognition component is a two-dimensional barcode B provided at the four corners of a frame on the display screen, as shown in FIG. 11, the screen area detection means 241 recognizes the position of the two-dimensional barcode B and detects a virtual frame V defined by four virtual lines L connecting the positions of adjacent two-dimensional barcodes B, thereby being able to notify the screen area surrounded by the virtual frame V from the captured image G.
In this embodiment, ArUco, which is a type of two-dimensional marker, is used as the two-dimensional barcode B, but a QR code (registered trademark) can also be used.

According to (1) to (4), even if a fisherman does not photograph the display screen D from the front, and even if the rectangular display screen D is distorted into a trapezoid shape or the background is reflected around the display screen D, the fisherman can accurately grasp the range of the display screen D.
In addition, by estimating the screen area using deep learning (1), it is possible to accurately grasp the range of the display screen without placing markings indicating the screen area (e.g., retroreflective material, colored tape, two-dimensional barcodes, etc.) on the display screen.

(Generation of composite images)
Next, the extraction means 242 shown in FIG. 4 removes outliers from each pixel and synthesizes the pixels to generate a composite image as a captured image with no or minimal overlap of reflected light on the characters (see step S14).
Here, a method for generating a captured image by the extraction means 242 will be described in detail.
For example, in the successively captured images G1 to G4 shown in FIGS. 10A to 10D, point-like reflected light R and planar reflected light R are captured.

First, when the RGB values of each pixel (x, y) in the i-th image among the N images are expressed as Ri(x, y), Gi(x, y), and Bi(x, y), the extraction means 242 calculates the intermediate value of the RGB values of each pixel using the following formula, thereby generating a single composite image RR.
R _RR (x,y)=median (R ₀ (x,y), R ₁ (x,y), R ₂ (x,y), ..., R _N-1 (x,y))
G _RR (x,y)=median (G ₀ (x,y), G ₁ (x,y), G ₂ (x,y), ..., G _N-1 (x,y))
B _RR (x,y)=median (B ₀ (x,y), B ₁ (x,y), B ₂ (x,y), ..., B _N-1 (x,y))
In this way, the extraction means 242 generates a composite image RR from multiple captured images using the median RGB values of each pixel to remove outliers from each pixel, so that pixels in an image that contain bright reflections can have lower RGB values than pixels in other images that do not contain reflections.
In addition, since the captured images G1 to G4 were captured while changing the shooting direction, even if reflected light R is superimposed on the characters in the first captured image, the reflected light R moves away from the characters as the shooting direction is moved, and it is possible to prevent the reflected light R from being superimposed on the characters. Therefore, when these captured images G1 to G4 are combined, the reflected light R can be weakened or removed.
The resulting composite image RR is shown in Fig. 10(E) , from which it can be seen that there is no reflected light.

In the above explanation, in order to remove outliers, the extraction means 242 synthesizes the mean value by calculating the median value of each pixel, but in removing outliers, values that deviate from 2σ of the average value in the standard deviation can be regarded as outliers, and the average value can be obtained by removing the outliers.

(Generation of binary images)
Next, the binarization means 243 shown in Fig. 4 generates a binary image by replacing the color data of pixels other than the first color data indicating the character color of the character from the captured image with the second color data (see step S15). In this embodiment, the character color is white. That is, the first color data is 0 for red data, 0 for green data, and 0 for blue data. Also, the second color data is black. That is, the first color data is 255 for red data, 255 for green data, and 255 for blue data.
12A, for example, the reflected light R from a fluorescent lamp is superimposed on the character "1.0", and the reflected light R has a pale blue color. Therefore, the reflected light R can be removed by performing binarization processing to change all colors other than white to black.
In this manner, the binarization means 243 generates a binary image, thereby making it possible to remove reflected light from a photographed image in which blackish reflected light has appeared that could not be removed even by the extraction means 242 .

Therefore, the character recognition means 24 does not require a box to cover the display screen to prevent reflected light R or obstacles from appearing on the display screen, which eliminates the need for cumbersome labor. Furthermore, the character recognition means 24 can be used in harsh environments such as the ocean by enabling accurate character recognition even when reflected light appears on the display screen.

(Character Recognition)
Next, the recognition means 245 shown in FIG. 4 recognizes the character indicated by the first color data of the binary image included in the recognition target area of the template specified by the template specification means 244, and outputs it as character information (see step S16).

For example, in this embodiment, the rectangular recognition target areas defined by the template are a tide speed area S1 indicating the speed of the tide, a tide direction area S2 indicating the direction of the tide, and a tide temperature area S3 indicating the water temperature of the tide, as shown in Figure 12 (B).
Since the template designation means 244 recognizes characters in the recognition target area using the template, it is possible to recognize only the necessary range of characters, thereby improving the accuracy of character recognition.
Then, the attribute assigning means 246 assigns attributes corresponding to the recognition target areas (tidal speed area S1, tidal direction area S2, tidal temperature area S3) specified by the template designation means 244 to the characters recognized by the recognition means 245 (see step S17). For example, characters included in the tidal speed area S1 become tidal speed information, characters included in the tidal direction area S2 become tidal direction information, and characters included in the tidal temperature area S3 become tidal temperature information.

In this way, the character recognition means 24 can recognize tidal speed information corresponding to the tidal speed region S1, tidal direction information corresponding to the tidal direction region S2, and tidal temperature information corresponding to the tidal temperature region S3 from the display screen and generate them as tidal current information.

When character recognition means 24 is offline and uses a template stored in storage means 29, it stores character recognition information about the recognition result in storage means 29 and moves to the next process. When character recognition means 24 is online and uses a recognition target area obtained from character recognition assistance server 60, recognition information notification means 247 uploads the character recognition information to character recognition assistance server 60 and then moves to the next process.
Since the character recognition information is uploaded to the character recognition assistance server 60, the character recognition information can be collected by the character recognition assistance server 60, and it is possible to understand how the character recognition rate changes depending on the difference in the template definition. In addition, the date and time of use of the template can be included in the character recognition information.
Since the use date and time can be used to understand the surrounding environment, particularly the effects of sunlight and lighting, if the character recognition information includes the use date and time of the template, it is possible to select a template that is resistant to these effects. Furthermore, the operator's comments included in the character recognition information can provide feedback on the ease of use to the operator.
Therefore, the templates can be evaluated based on the character recognition rate, which contributes to improving the character recognition rate and enhances convenience.

When characters are recognized by the character recognition means 24, tide speed information is displayed in the tide speed display column R11, tide direction information is displayed in the tide direction display column R12, and tide water temperature information is displayed in the tide water temperature display column R13, all of which are arranged on the operation screen P1 shown in Figure 9.
In the example shown in Figure 9, the tide speed display field R11 displays 2.5 knots as the tide speed information, the tide direction display field R12 displays north-northeast as the tide direction information, and the tide water temperature display field R13 displays 11 degrees as the tide water temperature information.

The layout of the display screen for the recognition target area of the template specified by the template specification means 244 varies depending on the manufacturer's model of the tidal current meter (measuring instrument). Therefore, a template is defined and stored in the storage means 29 for each manufacturer's model. This template can be changed by settings, so the fisheries management support terminal device 20 can be adapted by specifying models installed on various fishing boats.

The fisheries management support terminal device 20 shown in FIG. 3 acquires the determined position information and the date and time information acquired at the time of positioning by the positioning means 25 performing positioning based on signals from positioning satellites. By acquiring the position information and date and time information, the latitude and longitude are displayed on the operation screen P1 shown in FIG. 9.

The operation screen P1 displays "Save and register new" to register a new fishing log, and also displays a button B12 indicating the start of fishing. When a fisherman starts fishing, he presses this button B12.
When the input means 27 shown in Figure 3 notifies the user that button B12 has been pressed (see step S30 in Figure 6), the information transmission means 26 attempts to transmit operation information (current information, location information, date and time information) as notification information to the fisheries management support server 30 via the communication means 21.

However, because the operation site is far from the coast, the communication means 21 is unable to communicate with the relay station of the mobile phone company (see step S40). Therefore, the information transmission means 26 stores the notification information in the storage means 29 and waits for transmission (see step S50).

Then, when fishing operations are completed and the fishing boat approaches the coast, communication between the communication means 21 and the relay station is established, and the information transmission means 26 reads the notification information from the storage means 29 and transmits it to the fisheries management support server 30 (see step S60).
At this time, character recognition information, which is the recognition result stored in memory means 29 when the template was used offline by character recognition means 24, may also be transmitted to character recognition assistance server 60 when approaching the coast.

When the fisheries management support server 30 receives operation information (current information, location information, and date and time information) from the fisheries management support terminal device 20 via the communication means 31, the operation information is stored in the memory means 37, and the sea area analysis means 32 analyzes the fishing area (sea area) that includes the operation location based on the location information and stores it in the memory means 37 as fishing area information (see step S70).

In addition, the marine information acquisition means 33 (see Figure 5) transmits location information and date and time information to the marine weather information providing server 50 to acquire information regarding marine weather (marine weather information) for the date and time indicated by the date and time information at the location indicated by the location information (see step S80).
Furthermore, a person in charge at the fisheries cooperative operates the input terminal device 40 to access the fisheries management support server 30 and inputs the catch volume of each fish species at each operation location based on the invoice (see step S90).

In the fisheries management support server 30 , the catch volume registration means 34 registers the catch volume information on the invoice from the input terminal device 40 in the memory means 37 .
When the fisheries cooperative association staff member has completed inputting all the information, he/she operates the input terminal device 40 to instruct the fisheries management support server 30 to output a fishing report.

In the fisheries management support server 30, the report generating means 35 reads out information related to the operation (location information, date and time information, tidal current information, and catch amount information) from the storage means 37, generates a fishing performance report in a predetermined format, and stores it in the storage means 37 (see step S100). At this time, the report generating means 35 generates a date and time by adding a predetermined date and time to the date and time information, as end date and time information indicating the end date and time of the operation.
The generated fishing performance report can be downloaded to the input terminal device 40 and printed on a printing device (printer, etc.) connected directly to the input terminal device 40 via USB, etc., or indirectly via LAN, etc. (see step S110).

FIG. 13 shows an example of a fishing report generated by the report generating means 35. In FIG.
In the fishing report shown in Figure 13, information to be entered in the date column R1 and the net casting start time column R2 is generated from time and date information. Information to be entered in the fishing area column R3 is generated from location information. Information to be entered in the net casting end time column R4 is automatically generated from date and time information. Information to be entered in the water temperature column R5 is generated from tide temperature information (current information). And information to be entered in the fish species catch column R6 is generated from catch volume information entered based on the invoice.
In the fishing report shown in FIG. 13, the fishing area number is entered in the fishing area column R3, but if the format also includes latitude and longitude, this is generated from the location information.

In this way, the fisheries management support server 30 can automatically generate fishing performance reports by collecting various information from fishermen, fishing cooperatives, etc.

Then, in the fisheries management support server 30, the operation log generation means 36 automatically generates the operation log shown in FIG. 14 based on the information related to the operation (location information, date and time information, and catch information) and the marine weather information. The operation log shown in FIG. 14 was viewed by accessing the fisheries management support server 30 using the fisheries management support terminal device 20.

In this way, according to the fisheries management support system 10, when the fisheries management support terminal device 20 is unable to transmit to the fisheries management support server 30 via the mobile phone company's relay station, it stores the location information and date and time information in the storage means 29 and waits. Then, when communication with the relay station is established, the fisheries management support terminal device 20 transmits the location information and date and time information to the fisheries management support server 30. Therefore, even if a fisherman is operating at sea, he or she can transmit the location information and date and time information to the fisheries management support server 30 when communication is restored.

Therefore, the fishery management support terminal device 20 can be a smartphone or tablet having a function of communicating with a mobile phone company. Moreover, the fisherman only needs to operate the operation screens shown in Figs. 8 and 9 to obtain the location information and the date and time information.
Therefore, the fisheries management support system 10 shown in FIG. 1 supports the creation of fishing performance reports, allowing even small-scale fishermen to easily create them.

In addition, the report generation means 35 of the fisheries management support server 30 can generate end date and time information indicating the end date and time of the operation by adding a specified date and time from the date and time information, so that the fisherman does not need to operate the fisheries management support terminal device 20 to input the end date and time of the operation.

Furthermore, the fishing log generation means 36 of the fisheries management support server 30 can generate a fishing log as shown in FIG. 14 based on the location information, date and time information, and tidal current information.
This makes it possible to automatically generate not only a fishing report but also a fishing log at the same time.
In addition, the fisheries management support system 10 may not only automatically generate a fishing performance report or both a fishing performance report and an operation log, but may also automatically generate only the operation log.

Next, creation and editing of a template will be described.
An operator can access the character recognition assistance server 60 by operating the operation terminal device 70 to create a new template or edit an existing template. In addition, a single template can be created by combining and composing a plurality of recognition target regions (parts) that constitute a template.
At this time, in the work terminal device 70, the template displayed on the display means by a GUI (Graphical User Interface) can be defined by input means such as a mouse, a touch pad, a touch screen, etc. For example, by operating the input means, a rectangular recognition target area can be expanded, narrowed, or moved, and a template can be created, combined, or deleted to create or edit the template.

When synthesizing a single template from multiple target regions, if the individual target regions have different scales (reduced sizes), the differences in scale can be absorbed by using relative coordinates to define each target region.

In addition, when creating a template using the work terminal device 70, it can be created based on the shooting posture of the fishery management support terminal device 20.
For example, when an image is captured with the display means 28 of the fisheries management support terminal device 20 in a position such that its length is aligned with the top-to-bottom direction, the captured image is vertically long to match a vertical screen (portrait), and when an image is captured with the display means 28 in a position such that its length is aligned with the left-to-right direction, the captured image is horizontally long to match a horizontal screen (landscape), and the aspect ratio of the captured image changes depending on the shooting position of the fisheries management support terminal device 20.

Therefore, when photographing a tidal current meter with a horizontally long display screen, if you try to photograph it in portrait mode, you will be farther away from the tidal current meter than if you photographed it in landscape mode. This will change the size of the recognition target area of the template. Also, when the recognized characters are displayed on the display means 28 of the fishery management support terminal device 20 (see FIG. 2), the display position will change between portrait and landscape modes.
Therefore, a vertical template suitable for a vertically long photographed image and a horizontal template suitable for a horizontally long photographed image are prepared, and the template designation means 244 switches between the vertical and horizontal templates according to the photographing posture of the fisheries management support terminal device 20. In this way, a template suitable for the photographing posture can be selected.

In this embodiment, the measuring instrument is a tidal current meter used in fishing, but the character recognition means (character recognition device) of the present invention can be applied to any instrument that has a display screen and can display characters on the display screen.
In this embodiment, the fisheries management support server 30, marine weather information providing server 50, character recognition support server 60, etc. are configured as separate servers, but the servers may be combined or all be integrated together.

The present invention is suitable for reading the display screens of measuring instruments used both indoors and outdoors, as it is capable of character recognition even in harsh environments. In addition, the present invention is suitable for those engaged in the fishing industry, as it can easily create fishing reports and operation logs.

10 Fisheries management support system 20 Fisheries management support terminal device 21 Communication means 22 Photography means 23 Screen control means 24 Character recognition means 241 Screen area detection means 242 Extraction means 243 Binarization means 244 Template designation means 245 Recognition means 246 Attribute assignment means 247 Recognized information notification means 248 Template acquisition means 25 Positioning means 26 Information transmission means 27 Input means 28 Display means 29 Storage means 30 Fisheries management support server 31 Communication means 32 Sea area analysis means 33 Marine information acquisition means 34 Catch volume registration means 35 Report generation means 36 Operation log generation means 37 Storage means 40 Input terminal device 50 Marine weather information providing server 60 Character recognition support server 70 Work terminal device W Internet D Display screen T Current meter G, G1 to G4 Captured image RR Composite image F Surrounding frame V Virtual frame L Virtual line B Two-dimensional barcode R Reflected light P1 Operation screen B11, B12 Button S Positioning satellite R11 Tidal speed display field R12 Tide direction display field R13 Tide water temperature display field R1 Month and day field R2 Casting start time field R3 Fishing area field R4 Casting Net end time column R5 Water temperature column R6 Catch by fish species column S1 Tidal speed area S2 Tidal direction area S3 Tidal temperature area

Claims (18)

A character recognition device that optically recognizes characters from a captured image of a display screen on which characters are displayed, comprising:
a binarization means for generating a binary image by replacing color data of pixels other than first color data indicating a character color of the character from the photographed image with second color data;
a recognition means for recognizing a character represented by the first color data included in the binary image and outputting the character as character information ;
and an extraction means for generating a captured image in which there is no or minimal reflected light or obstacles on the characters, based on a plurality of captured images or video images obtained by photographing the display screen while changing the shooting direction and projecting the display screen onto a rectangular display screen, and removing outliers from each pixel and synthesizing the images . A character recognition device that optically recognizes characters from a captured image of a display screen on which characters are displayed, comprising:
a binarization means for generating a binary image by replacing color data of pixels other than first color data indicating a character color of the character from the photographed image with second color data;
a recognition means for recognizing a character represented by the first color data included in the binary image and outputting the character as character information ;
A character recognition device comprising: a screen area detection means for estimating the screen area of the display screen from a captured image based on a deep learning model constructed by learning the screen area in the captured image according to the shooting direction using a plurality of captured images or videos captured while changing the shooting direction of the display screen and an area-designated image in which the screen area of the captured image is designated; and for converting this screen area into a rectangular screen area by projective transformation . A character recognition device that optically recognizes characters from a captured image of a display screen on which characters are displayed, comprising:
a binarization means for generating a binary image by replacing color data of pixels other than first color data indicating a character color of the character from the photographed image with second color data;
a recognition means for recognizing a character represented by the first color data included in the binary image and outputting the character as character information ;
a screen area detection means for estimating a screen area of the display screen from a captured image based on a deep learning model constructed by learning a screen area in a captured image according to a shooting direction using a plurality of captured images or a video captured while changing a shooting direction of the display screen and an area-designated image in which a screen area of the captured image is designated, and converting the screen area into a rectangular screen area by projective transformation;
and an extraction means for generating a captured image in which there is no or minimal reflected light or obstacles on the characters, based on a plurality of captured images or video images obtained by photographing the display screen while changing the shooting direction and projecting the display screen onto a rectangular display screen, and removing outliers from each pixel and synthesizing the images . a recognition member for recognizing the range of the display screen is provided on a frame around the display screen;
2. The character recognition device according to claim 1 , further comprising screen area detection means for detecting a screen area of said display screen by recognizing a peripheral frame of said display screen in said photographed image from said recognition member. The recognition member is a peripheral frame formed of a retroreflective material,
5. The character recognition device according to claim 4, wherein said screen area detection means detects a virtual frame formed by reflected light from said peripheral frame in said photographed image, and notifies said user of the screen area surrounded by said virtual frame. The identification member is a peripheral frame formed by colored tape,
5. The character recognition device according to claim 4, wherein said screen area detection means detects a virtual frame based on the color of the peripheral frame in said photographed image, and notifies the user of the screen area surrounded by said virtual frame. the recognition members are two-dimensional barcodes provided at four corners of the frame,
5. The character recognition device according to claim 4, wherein the screen area detection means recognizes the positions of the two-dimensional barcodes in the captured image, detects a virtual frame connecting the positions of the two-dimensional barcodes, and notifies the screen area surrounded by the virtual frame. The character recognition device according to any one of claims 1 to 7, further comprising a template designation means for reading a template from a storage means in which a template is stored that defines an area including a character to be recognized and is displayed on the display screen, and designating the template to the recognition means. The storage means stores a portrait template corresponding to a portrait-oriented captured image captured with the display means oriented such that the length direction of the display means is aligned with the up-down direction, and a landscape template corresponding to a landscape-oriented captured image captured with the display means oriented such that the length direction of the display means is aligned with the left-right direction,
9. The character recognition device according to claim 8, wherein the template designation means switches between the portrait template and the landscape template depending on the photographing posture. The character recognition device according to claim 8 or 9, further comprising a template acquisition means for acquiring the template from a character recognition support server connected via a telecommunications line in which the template is stored, and storing the template in the storage means. The character recognition device according to claim 10, wherein the template designation means designates a template selected by the input means from among a plurality of templates stored in the storage means to the recognition means. The character recognition device according to any one of claims 1 to 11, wherein the recognition means generates character recognition information including a character recognition rate when a character is recognized. The character recognition device according to claim 12, wherein the character recognition information includes the date and time of use of the template. The character recognition device according to claim 12 or 13, further comprising a recognition information notification means for notifying the character recognition information to a character recognition support server connected via a telecommunications line. The character recognition device according to claim 14, wherein the recognition information notification means notifies the character information input to the input screen by the input means by including it in the character recognition information. Computer,
The device functions as a character recognition device that optically recognizes characters from a captured image of a display screen on which characters are displayed,
a binarization means for generating a binary image by replacing color data of pixels other than first color data indicating a character color of the character from the photographed image with second color data;
a recognition means for recognizing a character represented by the first color data included in the binary image and outputting the character as character information ;
A character recognition program that functions as an extraction means for generating a captured image in which there is no or minimal reflected light or obstacles on the characters by photographing the display screen while changing the shooting direction and projecting the display screen onto a rectangular display screen, and then removing outliers from each pixel and synthesizing the images based on multiple captured images or video images . An imaging means for imaging a display screen of a tidal current meter installed on a fishing boat on which a fisherman is aboard;
The character recognition device according to any one of claims 1 to 15, which optically recognizes characters from a captured image of the display screen to read tide trend information;
a positioning means for outputting the position and date and time of the fisherman's fishing operation as position information and date and time information based on a signal from a positioning satellite;
a catch invoice which describes the amount of catch obtained by the fishermen through their fishing operations, and a communication means for communicating with a fishery management support server which generates a fishing report or an operation logbook or both based on catch information indicating the amount of catch input from an input terminal device based on the invoice issued by the fishery market, the location information, the date and time information, and the tidal current information;
A fisheries management support terminal device comprising an information transmitting means for transmitting the tidal current information, the location information and the date and time information to the fisheries management support server. a fishery management support terminal device that outputs tidal current information read by optically recognizing characters from a photographed image of a display screen of a tidal current meter installed on a fishing boat on which a fisherman is aboard using the character recognition device according to any one of claims 1 to 15, and outputs the location and date and time at which the fisherman operated as location information and date and time information based on a signal from a positioning satellite;
an invoice indicating the amount of catch obtained by the fishermen through their fishing operations, and an input terminal device for inputting the amount of catch as catch information based on the invoice issued by the fishery market;
A fisheries management support system comprising a fisheries management support server that generates either a fishing performance report or an operation log, or both, based on the tidal current information, the location information, and the date and time information from the fisheries management support terminal device and the catch volume information from the input terminal device.

Images