WO2023281995A1 - Personal information masking method, and personal information masking device - Google Patents

Abstract

[problem] To provide a personal information masking method which erases personal information of characters or numbers from a color image. [Solution] This personal information masking method comprises: a step (S21) for acquiring a color image; a size change step (S22) for changing the acquired color image to a first color image of a prescribed size; a step (S28) for changing the first color image to a binary value image; an area detection step (S30) for detecting, from the binary image, an area in which personal information is possibly present; a decision step (S33) for deciding whether the personal information is present with respect to the area in which the personal information is likely to be present; and an erasing step (S34) for drawing surrounding pixels of the personal information on the personal information of the first color image and erasing the personal information.

Description

Personal information masking method and personal information masking device

The present invention relates to a personal information masking method and a personal information masking apparatus for masking a portion of personal information that identifies an individual captured in an image.

  In recent years, users often report activity reports at companies, schools, etc. to SNSs (social network services) on the cloud, and post scenery including people to SNSs. In the activity reports and posts, not only sentences of text data but also image information of still images or moving images are often uploaded to the SNS.

However, in the image (still image / video), personal information (information consisting of letters and numbers such as name, address, date of birth, gender, car registration number, etc.) information is called personal information.) is often imprinted. Such personal information may infringe on the privacy of individuals, and pose a problem of being used for crimes.

In order to avoid such a problem, Patent Document 1 discloses that after an image is displayed on a display, it is determined whether or not there is personal information that the user does not want to share in the image from the viewpoint of protecting personal information. A technical idea is disclosed in which a user's determination result is accepted, and based on the determination result, a partial area of an image is masked so that personal information cannot be recognized. As a method for detecting personal information consisting of letters or numbers from an image, OCR (Optical Character Recognition) using pattern matching technology is used, as disclosed in Patent Document 2.

JP 2020-156033 A JP 2015-028735 A

However, it is up to the user to decide whether or not the personal information, etc. of Patent Document 1 exists. In addition, there is a possibility that there may be personal information, etc., that the user does not notice unless the user carefully examines it. If the user does not notice, the personal information, etc. is not masked, and there is a problem that the personal information, etc. is not protected. In addition, in the OCR described in Patent Document 2, it is difficult to recognize characters or numbers such as resolution, vertical and horizontal arrangement, handwritten characters, and broken characters, especially small characters or numbers included in the image background. , was almost unrecognizable.

In addition, in Europe, the GDPR (General Data Protection Regulation) has been enforced to protect personal information and privacy, and further regulations for the protection of personal information are being promoted outside of Europe, so there is a growing demand for the protection of personal information. .

In view of the above problems, the present invention provides a personal information masking apparatus and method for erasing personal information that appears to be characters or numbers from a color image, and for outputting an image that does not give a sense of incongruity as a color image even if the personal information is erased. I will provide a.

A personal information masking method according to the present embodiment comprises a step of obtaining a color image, a size conversion step of converting the obtained color image into a first color image of a predetermined size, and a step of converting the first color image into a binary image. and a step. Furthermore, the masking method includes an area detection step of detecting an area in which personal information may exist in a binary image, a determination step of determining whether or not personal information exists in an area in which personal information is likely to exist, and Based on the personal information determined in the determining step and the first color image, an erasing step of drawing peripheral pixels of the personal information into the personal information of the first color image to erase the personal information, and a color image after the erasing step. and outputting.
The size conversion step converts the obtained color image into a first grayscale image of a predetermined size, and the region detection step detects a region in which personal information may exist in the first grayscale image. good too.

A personal information masking method according to another embodiment includes steps of obtaining a color image, a size conversion step of converting the obtained color image into a first color image and a first grayscale image of a predetermined size, and a first grayscale image. and an area detection step of detecting areas in the image where personal information may exist. Further, the masking method comprises a determination step of determining the presence or absence of personal information in an area in which personal information is likely to exist, and a pixel surrounding the personal information based on the personal information determined in the determination step and the first color image. An erasing step of drawing in personal information of one color image and erasing the personal information, and a step of outputting the color image after the erasing step are provided.

Further, the masking method includes the steps of detecting a protected area of a person in the first color image, storing the protected area when the protected area is detected, and removing the protected area from the first color image after the erasing step. and an overwriting step of overwriting.
Further, the masking method comprises the steps of denoising the first color image, processing a color space transformation on the first color image, and after the steps of denoising and processing the color space transformation, the first masking method. and a converting step of converting the color image into a second grayscale image.
The masking method further comprises processing a local histogram equalization on the second grayscale image and converting to a third grayscale image, wherein the region detection step comprises the second grayscale image or the third grayscale image. A region in which personal information may exist in the image may be detected.

The judgment process is
(a) A filter for determining the presence or absence of personal information based on whether the logical product of the area where personal information is likely to exist and the area obtained by rotating the area by a predetermined angle is greater than a first threshold;
(b) a filter for determining the presence or absence of personal information based on whether the standard deviation of the distance from the thin line that forms the framework of the area where personal information is likely to exist to the boundary line of the area is greater than a second threshold;
(c) a filter that determines the presence or absence of personal information based on whether the vertical or horizontal length of an area where personal information is likely to exist is greater than a third threshold for a predetermined size;
(d) A filter that determines the presence or absence of personal information based on whether the ratio of the area of the thin line that forms the framework of the area where personal information is likely to exist and the area of the area where personal information is likely to exist is greater than a fourth threshold;
It is preferred to apply at least one filter treatment of

A personal information detection step may be provided for detecting personal information for the first color image.
After the overwriting step, a finishing step of finishing with an edge preserving filter may be provided.

The personal information masking apparatus according to the present embodiment includes an acquisition unit that acquires a first color image of a predetermined size, a binary conversion unit that converts the first color image into a binary image, and a personal information masking unit for the binary image. an area detection step for detecting an area in which information may exist; a determination unit for determining the presence or absence of personal information in an area in which personal information is likely to exist; and the personal information determined by the determination unit and the first color. An erasing section for erasing the personal information by drawing surrounding pixels of the personal information into the personal information of the first color image based on the image, and an outputting section for outputting the erased color image.

The personal information masking device further includes a protected area detection unit for detecting a protected area of a person in the first color image, the protected area detection unit comprising:
i) the ability to detect facial regions using deep learning;
ii) the ability to detect specific objects using deep learning;
iii) the ability to detect skin areas by skin color in the first color image;
iv) a function of setting a protection frame to be protected on the first color image and detecting an area within the protection frame;
It is preferable to have at least one of

In this specification, personal information means characters (including multilingual characters and numbers, hereinafter referred to as "characters") captured in images (still images/videos).

According to the personal information masking device and the personal information masking method of the present invention, the personal information is erased by drawing the peripheral pixels of the personal information in the personal information of the color image, so that the personal information is protected.

1 is a block diagram of one embodiment of a personal information masking device; FIG. It is a flow chart using a personal information masking device. (A) is a diagram explaining color space conversion by a color space conversion unit. (B) is an image diagram of outline detection and outer frame detection by the area detection unit. (C) is a diagram explaining a mechanism for erasing personal information. 4 is a flow chart showing the function of detection of a protected area; 4 is a conceptual diagram when a user sets a protection area on the personal computer 23 or the mobile information terminal 25. FIG. 10 is a flow chart showing a method of determining personal information (whether or not it is characters). (A) is a diagram showing a state in which a character or graphic is rotated and superimposed on the original character or graphic. (B) is a diagram showing a state in which a character or graphic is shifted by a fixed distance and rotated, and superimposed on the original character or graphic. (A) is a diagram explaining the distance from the center line of a character or figure to the boundary line. (B) is an image diagram of outline detection and outer frame detection by the center line of a character or figure and the center line area detection unit. It is an example of a color image acquired by the personal information masking device 100 . It is an example (Example 1) of the output color image. It is an example (Example 2) of the output color image. It is an example (Example 3) of the output color image. It is an example (Example 4) of the output color image.

  Hereinafter, the personal information masking device of the embodiment will be described in detail with reference to the drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

<<Overview of personal information masking device>>
A personal information masking apparatus according to the present embodiment detects characters, which are personal information, from color images such as everyday scenery captured by a camera/video camera. OCR (Optical Character Recognition) is widely used as a method of recognizing characters from an image. However, since the personal information masking apparatus according to the present embodiment does not use OCR, it is not necessary to register all characters to be recognized in advance, and multilingual personal information can be deleted. In addition, the personal information masking device can delete fine characters or corrupted characters that cannot be detected by an OCR (optical character reader), such as fine personal information contained in the background.

FIG. 1 shows an outline of a system in which a user H uploads images (still images and moving images) taken at school, for example, to a personal information masking device 100, and downloads or receives images from which personal information has been deleted from the personal information masking device 100. It is a diagram showing. Even if the user H uploads the image from which the personal situation has been deleted to an SNS or the like, the personal information within the school included in the image has been deleted, so the personal information will not be leaked.

User H captures a still image of a predetermined place in the school with the digital camera 21, for example. A still image captured by the camera 21 is recorded in the personal computer 23 . User H can use the personal computer 23 to upload a still image to the personal information masking device 100 via a communication network NET such as the Internet. Also, the user H can take a moving image using the portable information terminal 25 with a camera function such as a tablet PC or a smartphone, and upload the moving image to the personal information masking device 100 via the communication network NET. Although not specifically described, in order to access the personal information masking device 100, the user H may register a user name, password, credit card number, etc. in advance.

<<Overview of Configuration of Personal Information Masking Device 100>>
The personal information masking device 100 depicted in FIG. 1 shows a block configuration of one embodiment. A cloud server or the like is suitable for the hardware configuration of the personal information masking device 100. Specifically, it has one or more processors, one or more memories, a communication interface, and the like. A color image acquisition unit, an image size conversion unit, and the like, which will be described below, are composed of a processor, a memory, a communication interface, programs executed by the processor, and the like.

The personal information masking device 100 includes a color image acquisition unit 101 that acquires a color image (still image or moving image) from an external device such as the personal computer 23 or the mobile information terminal 25, and an image size converter that converts the size of the acquired image into a predetermined size. A conversion unit 102, a protection area detection unit 103 that detects whether there is a person's face, clock, etc. (protection area) in the color image converted to a predetermined size, and a noise removal unit that removes noise contained in the color image ( color) 104, and a color space conversion unit 105 that makes it easier to see personal information that is hidden in the background color and is difficult to see. Note that the image size conversion unit 102 also has a function of converting a color image before removing color noise into a gray scale.

Further, the personal information masking apparatus 100 includes a grayscale conversion unit 106 that converts the noise-removed color image into a grayscale image, a local histogram flattening unit 107 that flattens the brightness of the grayscale image, and a flattening of the brightness. A binary conversion unit 108 that converts the converted grayscale image into a binary image, a noise removal unit (binary) 109 that removes black and white noise included in the binary image, and personal information is extracted from the binary image or the grayscale image. and an area detection unit 110 for detecting an area in which is likely to exist. The area detection unit 110 detects boundaries and outlines due to line drawings, shadows of objects, noise, etc. existing in a binary image or a grayscale image as areas in which personal information is likely to exist. Therefore, the area detection unit 110 can detect fine characters that are difficult for the human eye to notice, and can be applied to characters of any language, not limited to a specific language. On the other hand, the area detection unit 110 also detects shadows of objects other than characters or noise as areas in which personal information is likely to exist.

Further, the personal information masking apparatus 100 includes a personal information determination unit 111 that determines whether personal information exists in an area where personal information is likely to exist, and a personal information determination unit 111 that erases the personal information determined from the color image converted to a predetermined size. Protected area overwriting for overwriting a protected area saved when a person's face, skin or clock is detected by the information erasing section 112 and the protected area detecting section 103 into a color image from which personal information has been erased. a finishing unit 114 for finishing the color image; and a color image output unit 115 for outputting the color image to the outside such as the personal computer 23 or the portable information terminal 25 . The personal information masking apparatus 100 may also include a personal information detection unit 116 such as an in-scene character detection program (EAST: An Efficient and Accurate Scene Text Detector) using a neural network.

  The personal information masking device 100 may not only have the plurality of components described above physically at the same location, but may also be a collection of a plurality of servers. Also, a part of the plurality of configurations described above may be performed by the personal computer 23 or the portable information terminal 25 to which the application software is downloaded.

<<Flow chart of personal information masking method>>
Next, the operation of the personal information masking method of this embodiment will be described. FIG. 2 is a flow chart of the personal information masking method from acquisition of a color image to output of a color image from which personal information has been erased.

First, the color image acquisition unit 101 acquires a color image from the personal computer 23 or the mobile information terminal 25 (step S21). For example, this color image includes HD (1280*720), FHD (1920*1088) and 6M wide (3264*1836) for still images, and HD (1280*720) and FHD (1920) for moving images. *1088) and 4K (3840*2160) images.

Next, the image size conversion unit 102 converts color images of various sizes into images of a predetermined size (S22). For example, in this embodiment, the image size conversion unit 102 converts the image into an FHD size color image. The color image converted to the predetermined size is stored in a memory (not shown) or the like. The image size conversion unit 102 also has a function of converting a color image into grayscale.

Next, the protected area detection unit 103 detects a protected area included in the color image of a predetermined size (S23). Also, when no protected area is detected, it is preferable to add a flag indicating that there is no protected area to the color image. The protection area detection function will be described later.

Next, the color space conversion unit 105 converts the color space so that the personal information buried in colors can be easily seen (S24). Color image expression methods include the RGB method, which expresses color images using the three primary colors of light, and the HSV method, which expresses color images using hue, saturation, and value. A color space conversion unit 105 converts a color image (RGB) of a predetermined size into a color image (HSV) and erases a dark background color to highlight personal information. FIG. 3(A) shows a color photograph (A-1) with a name written in black on a red cap, a grayscale image of the color photograph (A-2), and a saturation (S) of zero. FIG. 10 is a diagram showing an image (A-3); When the color photograph (A-1), in which personal information is drawn in black against a dark background color (especially red, blue, gray, etc.), is converted to grayscale or binary, the black personal information becomes a dark background color. This makes it difficult to detect (see A-2). On the other hand, in the image (A-3) with the saturation (S) set to zero, the personal information stands out and stands out.

Next, noise is removed from the color image by the noise removal unit (color) 104 (S25). Since the area detection unit 110 detects a line drawing, a shadow of an object, noise, or the like as an area in which personal information is likely to exist and the amount of calculation increases, it is preferable to remove noise contained in the color image as much as possible. Here, the noise included in the color image is, for example, if the subject is a leather surface or the like, the area detection unit 110 detects the texture of the leather surface or the like as a shadow of an object, etc. Roughness on the surface of the object is also processed as noise contained in the color image.

As a method of removing noise contained in color images, there is a process called Gaussian Blur processing that uses a Gaussian function to blur the entire color image. However, since Gaussian blur processing smoothes even the edge portion of personal information contained in a color image, there is a possibility that personal information cannot be detected. Therefore, the noise removal unit (color) 104 of the present embodiment preferably applies edge-preserving smoothing filter processing (bilateral filter, mean shift filter, adaptive bilateral filter).

Next, the grayscale conversion unit 106 converts the color space-converted image into a grayscale image (S26).
Subsequently, the local histogram flattening unit 107 flattens the grayscale image so that personal information buried in the dark/bright areas of the grayscale image can be easily detected (S27). For example, if a grayscale image has a bright object on an overall dark background, if the overall contrast is flattened, the bright object will become pure white, and the personal information present in the bright object will disappear. Sometimes I end up Therefore, applying the local histogram equalization process to the grayscale image makes it easier to detect personal information buried in dark and bright areas. Local histogram equalization specifically includes adaptive histogram equalization, contrast-limited adaptive histogram equalization (CLAHE), multi-peak histogram equalization (MPHE), and multi-objective beta-optimized bihistogram equalization. (MBOBHE) processing and the like.

Next, the binary conversion unit 108 converts the grayscale image whose brightness has been flattened into a binary image (S28). If the background of the binary image is dark and the personal information is drawn in black, the background and the personal information will have the same value if the binary conversion is performed with a specific threshold value. area cannot be detected. Therefore, the binarization conversion unit 108 binarizes the grayscale image using the average value of the entire grayscale image (Otsu's binarization process (Discriminant Analysis method)), or for each part of the grayscale image (comparison Adaptive Gaussian Thresholding or Adaptive Mean Thresholding is preferably used to make dark black characters stand out.

Next, a noise removal unit (binary) 109 removes noise remaining in the binary image. Specifically, the noise removal unit (binary) 109 repeats erosion for removing pixels on the boundary of the object in the binary image and dilation for adding pixels to the boundary. to remove binary noise.

Next, the area detection unit 110 detects contours in which personal information is likely to exist from the binary image (S30). Contours in which personal information is likely to exist are detected by detecting contours such as lines in a binary image. In general, the area detector 110 scans the binary image for pixels of the object and determines whether it is an outer boundary or a hole boundary. By scanning and repeating the determination, the outermost boundary or contour is detected. for example. FIG. 3B-1 is an image diagram in which a dashed-dotted line contour 50 is detected in a line drawing included in a binary image. More details are disclosed in the paper Topological structural analysis of digitized binary images by border following; by Satoshi Suzuki (Computer Vision, Graphics, and Image Processing Volume 30, Issue 1, April 1985, Pages 32-46).

At the same time as step S30, the area detection unit 110 detects an outer frame in which personal information is likely to exist from the grayscale image (S31). Any of the grayscale images generated in steps S22, S26 or S27 may be used as the grayscale image. Schematically, the area detection unit 110 is used for area division of a grayscale image, in which areas having similar luminance values in the grayscale image are grouped into one area. Based on the idea of grouping together areas with similar luminance values, it is interpreted as an outer frame with a stable distribution, and its representative points are derived as feature quantities. for example. FIG. 3B-2 is an image diagram in which the outer frame 51 of the two-dot chain line is detected in the line drawing included in the grayscale image. FIG. 3B-2 is particularly different from FIG. 3B-1 in that the inside of "0" is also detected as an outer frame. More details are disclosed in the paper Efficient Maximally Stable Extremal Region (MSER) Tracking; by M. Donoser et al (2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)).

Personal information is detected from the color image by the personal information detection unit 116 (S32). The personal information detection unit 116 detects personal information (characters) by applying an in-scene character detection program (EAST: An Efficient and Accurate Scene Text Detector) or the like to the color image of a predetermined size generated in step S22. The personal information detection unit 116 can detect relatively large characters and the like included in the color screen, but it is difficult to detect small characters and the like included in the background.

Next, the personal information judging unit 111 judges whether or not there is personal information, that is, letters and numbers, in the outline or outer frame area that is likely to contain personal information detected in steps S30 and S31 (S33). . Also for characters detected by the personal information detection unit 116 . Similarly, the personal information determining unit 111 may determine whether or not there is personal information, that is, characters (S33). A method for determining personal information will be described later with reference to FIG.

In this embodiment, a route for determining whether personal information exists from a binary image (S29, S30, S33) and a route for determining whether personal information exists from a grayscale image (S22 (or S26, S27) , S31, S33) and a route (S22, S32, S33) for determining whether personal information exists from the color image, and personal information (characters) is stored in a memory (not shown) in each route. Then, the personal information (characters) stored in each route is integrated to obtain the personal information contained in the color image. However, the present invention is not limited to this, and at least one route out of these three routes may be applied, or two routes out of the three routes may be selected and integrated.

When the personal information is specified in step S33, the personal information erasing unit 112 erases the personal information determined from the color image of the predetermined size with the peripheral image without discomfort (S34). FIG. 3(C) is a color photograph (C-1) in which personal information (a linear flaw in FIG. 3(C)) is drawn on a color image of a predetermined size. Data (C-2) in which the personal information determined in step S33 is outlined on a black background, and a color image (C-3) in which the personal information is inpainted with surrounding pixels are shown. It is a diagram. In other words, the personal information erasing unit 112 prepares a color image (C-1) of a predetermined size and an image (C-2) of the same size in which the personal information is outlined against a black background and erases the personal information. The personal information is erased (C-3) by gradually drawing in the personal information with surrounding pixels from the boundary toward the inside (Inpainting).

This is because the process of drawing (inpainting) personal information with the surrounding pixels requires a large amount of calculations, which puts a heavy burden on the processor or takes time. In step S34, it is preferable to devise ways to reduce the amount of calculation. Specifically, the personal information erasing unit 112 converts an image of a predetermined size (eg, FHD) into a reduced size image (eg, HD). Based on the color photograph (C-1), the white data (C-2), and the reduced size, pixels surrounding the personal information are drawn in (inpainting). Then, the personal information erasing unit 112 creates a reduced-size image (C-3) from which the personal information has been erased. Then, the reduced size image (C-3) from which the personal information has been deleted is returned to a predetermined size, and the color image (C-1) of the predetermined size is blanked with the image (C-3) restored to the predetermined size. The image at position (C-2) should be overwritten. Such processing can significantly reduce the amount of calculation.

To explain in more detail, there are two methods of inpainting (Inpainting) in which the surrounding pixels are gradually drawn into the personal information from the boundary of the personal information toward the inside. One is to replace the value of a pixel in the neighboring area to be erased with the weighted sum of the pixel values of pixels whose pixel values are already known among the surrounding pixels, and draw a certain pixel, FMM (Fast Marching Method) is used to move to the next nearest neighbor point, and the pixels close to the pixels without personal information or erased are drawn in order. More details are disclosed in the article An image inpainting technique based on the fast marching method; by Alexandru Telea (January 2004 Journal of Graphics Tools). Another method is to draw pixels by searching along edges from pixel values in a known area around the personal information to the area of the personal information. More details are disclosed in the paper Navier-stokes, fluid dynamics, and image and video inpainting; by Marcelo Bertalmio et al (Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.).

Next, when the protected area detection unit 103 detects a protected area, the protected area overwrite unit 113 overwrites the saved protected area in the color image from which the personal information has been deleted (S35). The area detection unit 110 detects that areas in which personal information is likely to exist include, for example, the temples of eyeglasses and the shadow of the nose. is disturbed. Therefore, the protected area is protected by overwriting the saved protected area with the color image from which the personal information has been deleted. If no protected area is detected in step S23, step S35 may be skipped based on the flag.

Then, it is preferable that the finishing section 114 performs finishing processing on the color photograph in which the protected area has been overwritten (S36). Specifically, a pattern preserving filter (non-local mean filter) that suppresses smoothing of luminance values across boundaries or Gaussian blurring is used for color images. The boundary with the overwritten protected area can be made inconspicuous, and minute personal information (characters) that could not be erased can be made illegible. There is no problem even if there is no finishing treatment.

Finally, the color image output unit 115 outputs the finished color image to the personal computer 23 or the mobile information terminal 25 (S37).

<Detection of protection area>
Here, the function (S23) of detecting the protected area by the protected area detection unit 103 will be described with reference to FIGS. 3 and 4. FIG.

First, in step S231, the protection area detection unit 103 uses an AI capable of deep learning, etc., based on the color image that has been changed to a predetermined size, from the features of the human face such as the positions of the eyes, nose, and mouth. , to detect the face area. The face area is a protected area that does not contain personal information (characters). The face area is saved in step S235 and overwritten in the color image from which the personal information has been erased as described in S35.

Next, in step S232, the protected area detection unit 103 uses AI or the like to label the objects included in the color image. Therefore, when a color image includes a clock, calendar, or table, the protected area detection unit 103 outputs a label such as "clock", "wall calendar", or "table". For example, clocks and calendars have time and date numbers (characters) drawn on them, but objects that contain characters but do not contain personal information, clocks, etc., are registered in advance. back. A clock or the like is detected as a protected area, and the protected area of the clock or the like is saved in step S235.

Next, in step S233, the protected area detection unit 103 detects whether the color image resized to the predetermined size has skin color. For example, the spaces (shadows) between the fingers are prevented from being recognized as a single line character (for example, "I" of the alphabet). The skin area is a protected area that does not contain personal information, and the skin area is saved in step S235.

The protected area detection unit 103 may detect the color of the skin using, for example, color image display methods RGB, HSV, and YCbCr for a color image (RGB) of a predetermined size. More details are disclosed in the paper Human Skin Detection Using RGB, HSV and YCbCr Color Models; by S. Kolkur et al (ICCASP/ICMMD-2016. Published by Atlantic Press). In this embodiment, a color image (RGB) is converted into a color image (HSV), and hue, saturation, and value are adjusted to detect a skin color area. For example, if adjusting with 8 bits (0-255), H = 16 to 40 (reddish to yellowish) S = 20 to 200, V = 128 to 255, the skin in the color image It can detect areas where color exists.

Next, in step S234, the protected area detection unit 103 displays a protected frame so that the user H can set a protected area on the screen of the personal computer 23 or the mobile information terminal 25. FIG. 5 is an example of an uploaded color image 51 displayed on the screen of the mobile information terminal 25, and the color image 51 is part of a school classroom. In the classroom, there are two desks and one shelf, the desk has a personal computer on which character information is displayed, and the shelf has a plurality of books and files. There are clocks, placards and windows on the walls of the classroom. The clock has a dial and the placard has the words "Congratulations on winning".

The protected area detection unit 103 causes the portable information terminal 25 of FIG. When the user H clicks on the frame 57, the frame 57 is displayed on the screen. The user H changes the size and position of the frame 57, for example, sets a frame 57a around the placard and sets a frame 57b at the bottom of the shelf. User H clicks the completion button 54 when the setting of the frame 57 is completed. The area set by the frame 57a or 57b in this manner is detected as a protected area, and the protected area is saved in step S235. As described in S35, the protected area is overwritten in the color image from which the personal information has been erased. Note that even if the clock on the wall of the classroom in FIG. 5 is not enclosed as a protected area in step S234, the clock is made a protected area when step 232 is executed. Although FIG. 5 has been described on the premise of a still image, the protection area can also be set for a moving image. For example, a frame area designated by the user H is held as a template image, and template matching processing is performed to check areas in the moving image that match the template image.

The order of steps S231 to S234 may be changed for all protected areas, or only one of the steps may be executed, such as detecting only the face area. In step S32 and the like, the personal information area detection unit 110 detects the area of personal information, and the detection area detected by the personal information detection unit 111 includes a protection area such as a face area or a skin area. You may detect whether it exists.

<Determination of personal information (characters/numbers)>
Here, a method (S33) for determining whether personal information is personal information by the personal information determining unit 111 for an area in which personal information is likely to exist will be described with reference to FIGS. 6 to 8. FIG. In the personal information determination method, a case where the personal information is characters (numbers) will be described. Therefore, personal information in an area where personal information is likely to exist is called an estimated letter.

First, the personal information determination unit 111 determines whether or not the area ratio between the area of the estimated character rotated and superimposed on the original estimated character and the area of the original estimated character is greater than a certain threshold k1 ( S331). If the area ratio is larger than the threshold k1, it is determined to be a figure (other than a character) and the process proceeds to step S336.

Fig. 7(A) shows an example of rotating the estimated characters. In FIG. 7A, alphabetic characters "C", "O" and "Z" and figures of perfect circles, squares and equilateral triangles are used as examples of estimated characters. Further, in the table shown in FIG. 7A, from the left, 0° (not rotated), 90° rotation, superposition of 0° estimated character and 90° rotated estimated character, 180° rotation, 0° rotation Superposition of the estimated character and the estimated character rotated by 180°, superposition of the estimated character rotated by 270°, and the estimated character rotated by 0° and the estimated character rotated by 270° are shown. Note that the rotation is performed around the position of the center of gravity of the character area or figure area.

First, in the superimposition of 0° and 90°, the overlap region of "C", "O" and "Z" is 50% or less. On the other hand, a perfect circle, a square and an equilateral triangle have an overlapping area of 50% or more. These letters and figures are the same in superposition of 0° and 270°.

Next, in the superposition of 0° and 180°, "O" and "Z" have 100% of their overlapping area, and "C" has about 80% of their overlapping area. A perfect circle and a square have an overlapping area of 100%, and an equilateral triangle has an overlapping area of about 50%.

In this embodiment, the personal information determination unit 111 supports not only the alphabet but also multiple languages. It has a threshold value k13 for superposition of ° and 270°, respectively, and it is determined whether or not the estimated character is a character by combining the superposition of the three rotation angles. If the personal information determination unit 111 performs character determination on an estimated character of a certain specific language, for example, only the threshold value k11 for overlapping 0° and 90° may be used. Moreover, it is preferable to make the threshold k1 variable, and it is preferable to make the threshold k1 value variable for each language.

As described above, in the superimposition of 0° and 180°, the overlapping area of "C" is about 80%, and the overlapping area of the equilateral triangle is about 50%. It's getting smaller. Since it is difficult to determine whether the estimated character is a character, the estimated character may be rotated as shown in FIG. 7B. In FIG. 7B, an alphabet "C", a perfect circle, and an equilateral triangle are examples of estimated characters. Further, in the table shown in FIG. 7B, 0° from the left (not rotated), shifted by a distance S, rotated 180° with the shift, the estimated character at 0°, and the estimated character shifted and rotated 180°. A superposition with is shown. Note that the rotation is performed around the position of the center of gravity of the character area or figure area. The overlap area of the "C" is about 10%, the overlap area of the new circle is about 70%, and the overlap area of the equilateral triangle is about 50%. Whether or not the estimated character is a character can be determined by checking whether or not the overlap region of the estimated character at 0 degrees and the estimated character shifted by the distance S and rotated by 180 degrees is larger than the threshold value k12. A technique of shifting and rotating 180°, rotating 90°, and rotating 270° may be combined. Although the alphabet is explained in FIG. 7, the same applies to Arabic numerals and Chinese characters.

In step S332, the personal information determination unit 111 calculates the standard deviation of the distance from the center line of the estimated character to the boundary line of the estimated character, compares the standard deviation with a threshold value k2, and determines that the character is a character. If the standard deviation is larger than the threshold k2, it is determined to be a figure or the like, and the process proceeds to step S336. Although it has already been determined in step S331 that the character is a character, in step S332 another method may be used to determine whether the estimated character is a character.

Characters have the characteristic that the distance (multiple points) from the center line (framework) of the character to the boundary line of the character does not change significantly. Therefore, the standard deviation of the distance from the center line of the estimated character to the boundary line of the estimated character is calculated. However, since the standard deviation also increases in proportion to the character size, it is preferable to divide by the average distance from the center line to the boundary line, that is, to calculate threshold k2=standard deviation of distance/average distance. By dividing by the average distance, the threshold k2 becomes less susceptible to the character size. Also, instead of dividing by the average distance from the center line to the boundary line, the threshold value k2 may be set by the standard deviation of (distance/average distance). Also, it is preferable to make the threshold value k2 variable in order to adjust the criteria for judging graphics and the like and characters.

FIG. 8(A) is an example showing the alphabet "J" and a trapezoidal figure similar to J. The distance L1 from the center line (framework) 61 of "J" to the boundary line 63 is almost constant and its standard deviation is small. In FIG. 8A, J is Gothic, but the standard deviation is small even in other typefaces such as New Century. The distance L2 from the center line (framework) 61 of the trapezoidal figure to the boundary line 63 varies greatly, and its standard deviation increases. For example, the threshold k2 may be set to 0.6.

In step S333, the personal information determination unit 111 calculates the area ratio of the area of the estimated character itself and the area of the entire area surrounding the estimated character, and determines that it is a character if it is smaller than the threshold value k3. If it is larger than the threshold k3, it is determined to be a figure or the like and the process proceeds to step S336. If it is smaller than the threshold k3, it is determined to be a character and the process proceeds to step S334. Although it has already been determined that the character is a character in steps S331 and S332, another method may be used to determine whether the estimated character is a character in step S333.

FIG. 8(B) is a specific example of step S333, showing an alphabet "J" and a spoon-shaped figure similar to J. In FIG. 8B, the area ratio of the area 66 of the J character itself to the area 65 of the entire area surrounding J is about 60%, and the area 66 of the spoon-shaped figure and the area of the entire area surrounding the spoon-shaped figure The area ratio with 65 is about 80%. For example, the threshold k3 may be set to 70%.

　The order of steps S331 to S333 may be changed, or it may be determined that the estimated character is a character in only one of the steps. In steps S334 and S335, it is determined whether or not the character is personal information because it was determined in step S331, S332 or S333 that the character is a character.

Returning to FIG. 6 again, in step S334, it is determined whether or not the estimated character is larger than a certain threshold k4 for an image of a predetermined size. If it is larger than the threshold k4, it is determined that the character is large, and the process proceeds to step S338. If it is smaller than the threshold k4, it is determined that the character is small, and the process proceeds to step S336. More specifically, when the image size is FHD (1920*1088), at least one of the estimated characters has a vertical pixel count or a horizontal pixel count of, for example, 5 percent (96*54) or more of the image size. It is determined whether or not. If larger, the process proceeds to step S339. The percentage of the image size (threshold k4) is preferably variable. A function that can change the threshold value k4 may be displayed on a web screen or the like for uploading a color image so that the user can determine the size of characters that the user wants to leave.

In step S335, it is determined whether or not the character is drawn with a thick line width greater than a certain threshold value k5. If it is larger than the threshold k5, it is determined that the information is not personal information, and the process proceeds to step S338. Preferably, the threshold k5 is also variable. A function that can change the threshold value k5 may be displayed on a web screen or the like for uploading a color image so that the user can determine the characters with a thick line width that the user wants to leave.

FIG. 8(C) is a specific example of step S335, showing a thick line width "I" and a thin line width "I" of the alphabet. The area of the "I" character itself 68 and the area of the three centerlines (framework) 67 of the "I" are calculated, and it is determined whether the ratio of the areas is greater than a threshold k5, for example 10.

　In step S336 of FIG. 6, the estimated characters determined as not being characters in steps S331, S332, or S333 are considered to be graphics or the like, and are therefore left as part of the color image.

In step S337, it is determined that the estimated characters are characters in step S331 or S332, and that they are neither large characters nor thick line width characters in step S334 or S335. Therefore, these characters are deleted from the color image as personal information. subject to

In step S338, the estimated characters are determined to be characters in step S332 or S333, and are determined to be large characters or characters with a thick line width in step S334 or S335. The "Graduation Ceremony" signboard hanging at the entrance of the school and the letters drawn in thick lines on the T-shirt are preserved in the color image. In the present embodiment, large characters or characters with a thick line width are not treated as personal information, even if they are names, addresses, dates of birth, sex, numbers on license plates, and the like.

9 is a color image acquired by the personal information masking device 100 from the personal computer 23. FIG. 10, 11, 12 and 13 are color images output to the personal computer 23 by the personal information masking device 100. FIG.

The color image shown in FIG. 9 includes license plates of more than 10 cars, 8 languages (English, Thai, Arabic, Cyrillic, Japanese, Georgian, Korean, Burmese) drawn on the car. ) language characters, English characters such as news titles, and reporter faces.

FIG. 10 is an output example of Example 1, and Example 1 is an example of processing a color image without using steps S31 and S32 of FIG. That is, FIG. 10 is a color image obtained by executing steps S21-S30 and S33-S37 in FIG. In step S23, only S231 (face area protection) in FIG. 3 is executed.

Thai, Japanese, Hangul, and Cyrillic are deleted from the color image shown in FIG. However, English (alphabet), Burmese and Arabic parts and numbers remain. Since many white lines were determined as figures, many white lines of the road remain. The name on the reporter's name tag, the microphone held by the reporter, and the name of the broadcasting station have also been erased. The words LIVE NEWS and the news titles are still there. The reporter's face has been clarified by the execution of steps S231) and S36.

FIG. 11 is an output example of Example 2, in which Steps S24-30 and S32 of FIG. 2 are not executed. That is, FIG. 11 is a color image obtained by performing step S31 on the grayscale image converted in step S22. In step S23, only S231 (face area protection) in FIG. 3 is executed.

In the color image shown in Fig. 11, the letters and numbers on all license plates, including those of distant cars, have been erased. Also, all eight languages painted on the car have been erased. The name on the reporter's name tag, the microphone held by the reporter, and the name of the broadcasting station have also been erased. The white lines on the road are also determined as characters and deleted. Some parts of the LIVE NEWS text and news titles are blurred, but they have not been erased. Although it is difficult to understand by comparing FIG. 10 and FIG. 11, in the color image obtained by processing the grayscale image in step S31, black characters on a red background may be particularly difficult to erase. The reporter's face is clear due to the execution of steps S231 and S36.

FIG. 12 is an output example of Example 3. In Example 3, all the flowcharts of FIG. 2 are executed except for step S32. Therefore, FIG. 12 is equivalent to the photograph in which FIGS. 10 and 11 are superimposed. In step S23, only S231 (face area protection) in FIG. 3 is executed.

In the color image shown in FIG. 12, the letters and numbers of all the license plates, including the license plate of the distant car, have been erased, while some of the white lines and the car have been erased in FIG. A part of the car body of is drawn. The reporter's face is clear due to the execution of steps S231 and S36.

FIG. 13 is an output example of Example 4. In Example 4, steps S24-30 and S32 are not executed as in Example 2, and the grayscale image converted in step S22 is executed in step S31. This is a color image. However, in step S23, only S234 (setting of protected area) in FIG. 3 is executed, and the protected area set by user H is the reporter frame in which the reporter is captured.

In the color image shown in Fig. 13, the letters and numbers on all license plates, including those of distant cars, have been erased. Also, all eight languages painted on the car have been erased. On the other hand, the name of the reporter's name tag, the characters on the microphone held by the reporter, and the broadcasting station name and date on the upper left of the reporter's frame are also maintained as they are in the original color image. Of course, the color image shown in FIG. 13 also clearly shows the reporter's face and fingertips within the reporter's frame.

The personal information masking device of this embodiment processes everything from acquiring a color image to outputting a color image from which personal information has been erased. However, the personal computer 23 or the mobile information terminal 25 may be responsible for part of the processing. For example, the personal computer 23 or the mobile information terminal 25 downloads an application from the cloud server in advance, and the personal computer 23 or the mobile information terminal 25 converts the color image into a predetermined size and protects the protected area such as the face area. may be detected and stored before being uploaded to the personal information masking device 100 . After the personal information is erased, a color image may be output from the personal information masking apparatus 100 and the protected area may be overwritten by the personal computer 23 or the portable information terminal 25 .

Also, the personal computer 23 or the portable information terminal 25 may be responsible for all software processing of the personal information masking device 100 .　

23 personal computer 25 portable information terminal 100 personal information masking device 101 color image acquisition unit 102 image size conversion unit 103 protected area detection unit;
104 Noise remover (color)
105 color space converter 106 grayscale converter 107 local histogram equalizer 108 binary converter 109 noise remover (binary)
110 Area detection unit 111 Personal information determination unit 112 Personal information deletion unit 113 Protected area overwrite unit 114 Finishing unit 115 Color image output unit 116 Personal information detection unit

Claims (12)

obtaining a color image;
a size conversion step of converting the obtained color image into a first color image of a predetermined size;
converting the first color image to a binary image;
a region detection step of detecting a region in which personal information may exist in the binary image;
a determination step of determining the presence or absence of personal information in the area where the personal information is likely to exist;
an erasing step of drawing peripheral pixels of the personal information into the personal information of the first color image and erasing the personal information based on the personal information determined in the determining step and the first color image;
a step of outputting a color image after the erasing step;
A method of masking personal information, comprising: The size conversion step converts the obtained color image into a first grayscale image of the predetermined size,
The region detection step detects regions in which personal information may exist in the first grayscale image.
The personal information masking method according to claim 1. obtaining a color image;
a size conversion step of converting the obtained color image into a first color image and a first grayscale image of a predetermined size;
an area detection step of detecting an area in which personal information may exist in the first grayscale image;
a determination step of determining the presence or absence of personal information in the area where the personal information is likely to exist;
an erasing step of drawing peripheral pixels of the personal information into the personal information of the first color image and erasing the personal information based on the personal information determined in the determining step and the first color image;
a step of outputting a color image after the erasing step;
A method of masking personal information, comprising: detecting protected areas of a person for the first color image;
storing the protected area when the protected area is detected;
an overwriting step of overwriting the protected area on the first color image after the erasing step;
A personal information masking method according to any one of claims 1 to 3, comprising: denoising the first color image;
performing a color space transformation on the first color image;
a converting step of converting the first color image into a second grayscale image after the steps of denoising and processing color space conversion;
A personal information masking method according to any one of claims 1 to 4, comprising: processing a local histogram equalization on the second grayscale image to convert to a third grayscale image;
6. The personal information masking method according to claim 5, wherein said area detection step detects areas in which personal information may exist in said second grayscale image or said third grayscale image. The determination step includes
(a) a filter for determining the presence or absence of personal information based on whether the logical product of the area where the personal information is likely to exist and the area obtained by rotating the area by a predetermined angle is greater than a first threshold;
(b) a filter that determines the presence or absence of personal information based on whether the standard deviation of the distance from the thin line that forms the framework of the area where the personal information is likely to exist to the boundary line of the area is greater than a second threshold;
(c) a filter for determining the presence or absence of personal information based on whether the area ratio of the area where the personal information is likely to exist and the area surrounding the area where the personal information is likely to exist is greater than a third threshold;
(d) a filter for determining the presence or absence of personal information based on whether the vertical or horizontal length of the area where the personal information is likely to exist is greater than a fourth threshold for the predetermined size;
(e) Determining the presence or absence of personal information based on whether the ratio of the area of the thin line that forms the framework of the area where the personal information is likely to exist and the area of the area where the personal information is likely to exist is greater than a fifth threshold. filter,
7. The method of masking personal information according to any one of claims 1 to 6, applying at least one filter process of . The personal information masking method according to any one of claims 1 to 7, comprising a personal information detection step of detecting personal information for the first color image. After the overwriting step, a finishing step of finishing with an edge preserving filter;
5. The personal information masking method of claim 4, comprising: an acquisition unit that acquires a first color image of a predetermined size;
a binary conversion unit that converts the first color image into a binary image;
a region detection step of detecting a region in which personal information may exist in the binary image;
a determination unit that determines the presence or absence of personal information in the area where the personal information is likely to exist;
an erasing unit that draws peripheral pixels of the personal information into the personal information of the first color image and erases the personal information based on the personal information determined by the determining unit and the first color image;
an output unit that outputs the erased color image;
A personal information masking device comprising: The determination unit
(a) a filter for determining the presence or absence of personal information based on whether the logical product of the area where the personal information is likely to exist and the area obtained by rotating the area by a predetermined angle is greater than a first threshold;
(b) a filter that determines the presence or absence of personal information based on whether the standard deviation of the distance from the thin line that forms the framework of the area where the personal information is likely to exist to the boundary line of the area is greater than a second threshold;
(c) a filter for determining the presence or absence of personal information based on whether the area ratio of the area where the personal information is likely to exist and the area surrounding the area where the personal information is likely to exist is greater than a third threshold;
(d) a filter for determining the presence or absence of personal information based on whether the vertical or horizontal length of the area where the personal information is likely to exist is greater than a fourth threshold for the predetermined size;
(e) Determining the presence or absence of personal information based on whether the ratio of the area of the thin line that forms the framework of the area where the personal information is likely to exist and the area of the area where the personal information is likely to exist is greater than a fifth threshold. filter,
10. The personal information masking device of claim 9, applying at least one filter of . Further comprising a protected area detection unit for detecting a protected area of a person in the first color image, the protected area detection unit comprising:
i) the ability to detect facial regions using deep learning;
ii) the ability to detect specific objects using deep learning;
iii) detecting skin areas by skin color in said first color image;
iv) a function of setting a protective frame to be protected on the first color image and detecting an area within the protective frame;
12. Personal information masking device according to claim 10 or claim 11, applying at least one function of

Images