KR102866131B1 - Method for protecting information using camera module - Google Patents

Abstract

According to one embodiment of the present disclosure, a method for protecting information utilizing a photographing module, performed by a computing device, is disclosed. The method comprises the steps of: acquiring visual data utilizing the photographing module; detecting facial data within the visual data utilizing a pre-learned first model; executing a first protection operation based on a result of the detection and a first threshold value; matching the detected facial data with facial recognition data utilizing the pre-learned first model or the pre-learned second model; and executing a second protection operation based on the result of the matching and a second threshold value, wherein the first protection operation and the second protection operation may include a visual control operation regarding information displayed on a screen of a terminal.

Description

Method for protecting information using a camera module

The present disclosure relates to a method for protecting information by utilizing a photographing module, and more specifically, to a method for protecting information by acquiring visual data by utilizing a photographing module and detecting a user through the data.

In today's information-driven society, the importance of personal information is increasingly emphasized. In particular, information displayed on devices such as computers, laptops, and mobile phones often contains sensitive personal information. Furthermore, even if not personal information, the information displayed on devices often contains confidential work-related information or information about specific groups, information whose exposure or disclosure is undesirable. To prevent unauthorized exposure or leakage of such information, various physical and methodological privacy protection technologies are being developed. For example, these technologies often focus on verifying the identity of the user in front of the device screen or safely protecting the information displayed on the screen.

Korean Patent No. 2640357 (February 20, 2024) discloses a control method of a system for non-face-to-face identification using color, exposure, and depth values of a facial image.

Meanwhile, while the present disclosure was derived at least based on the technical background discussed above, the technical task or purpose of the present disclosure is not limited to resolving the problems or shortcomings discussed above. That is, in addition to the technical issues discussed above, the present disclosure can cover various technical issues related to the content described below.

The present disclosure may utilize a camera module to protect information. For example, to protect information displayed on a terminal, the purpose may be to authenticate the user's identity or, further, perform a protection operation based on the results of the authentication. For example, information displayed on a terminal can be visually controlled to minimize interference with the operation of each app or program, while still implementing a protection function for the information displayed on the screen.

In addition, the present disclosure can implement such information protection methods in various modes so that a mode suitable for the user's situation, intention, and type of information displayed on the screen can be automatically or manually executed.

Meanwhile, the technical task to be achieved by the present disclosure is not limited to the technical task mentioned above, and may include various technical tasks within a scope obvious to a person skilled in the art from the contents described below.

According to one embodiment of the present disclosure for achieving the aforementioned task, a method for protecting information utilizing a photographing module is disclosed. The method comprises the steps of: acquiring visual data utilizing the photographing module; detecting facial data within the visual data utilizing a pre-learned first model; executing a first protection operation based on a result of the detection and a first threshold value; matching the detected facial data with facial recognition data utilizing the pre-learned first model or the pre-learned second model; and executing a second protection operation based on the result of the matching and a second threshold value. The first protection operation and the second protection operation may include a visual control operation regarding information displayed on a screen of a terminal.

In one embodiment, the step of acquiring visual data using the photographing module may be performed repeatedly at preset time intervals.

In one embodiment, the step of acquiring visual data using the photographing module can be performed in real time.

In one embodiment, the step of executing a first protection action based on the result of the detection and a first threshold value includes the step of executing the first protection action when the result of the detection is less than the first threshold value; and the step of recognizing that facial data has been detected when the result of the detection meets the first threshold value, wherein the first protection action may include at least one of a filter action or a transparency action.

In one embodiment, the step of executing the second protection operation based on the result of the matching and the second reference value may include the step of executing the second protection operation when the result of the matching falls below the second reference value.

In one embodiment, the second protection action may include at least one of a filter action, a transparency action, or a notification action.

In one embodiment, the detected facial data may include one or more facial data.

In one embodiment, the method further includes a step of setting one of a plurality of modes based on a user's selection, wherein the plurality of modes may include a first mode and a second mode.

In one embodiment, the first mode may be a mode set to perform a second protection operation when, as a result of the matching, there is one or more facial data that falls below the second reference value.

In one embodiment, the second mode may be a mode set to perform a second protection operation when, as a result of the matching, there is no facial data that satisfies the second reference value.

In one embodiment, the method further comprises the step of pre-registering facial recognition data of at least one user, wherein the first reference value and the second reference value may be preset.

In one embodiment, the method further comprises the step of standardizing the facial recognition data and the visual data.

In one embodiment, the method may further include, if, as a result of the matching, there is at least one piece of facial data satisfying the second criterion, a step of tracking the facial data satisfying the second criterion using a pre-learned model.

According to one embodiment of the present disclosure for realizing the above-described task, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, the computer program causes the one or more processors to perform the following operations for protecting information by utilizing a photographing module, the operations including: acquiring visual data by utilizing the photographing module; detecting facial data in the visual data by utilizing a pre-learned first model; executing a first protection operation based on a result of the detection and a first threshold value; matching the detected facial data with facial recognition data by utilizing the pre-learned first model or the pre-learned second model; and executing a second protection operation based on a result of the matching and a second threshold value, wherein the first protection operation and the second protection operation may include a visual control operation regarding information displayed on a screen of a terminal.

A computing device according to one embodiment of the present disclosure for achieving the above-described task is disclosed. The device comprises at least one processor; and a memory, wherein the at least one processor is configured to acquire visual data through a photographing module; detect facial data in the visual data using a pre-learned first model; execute a first protection operation based on a result of the detection and a first threshold value; match the detected facial data with facial recognition data using the pre-learned first model or the pre-learned second model; and execute a second protection operation based on the result of the matching and a second threshold value, wherein the first protection operation and the second protection operation may include a visual control operation regarding information displayed on a screen of a terminal.

The present disclosure can protect information by utilizing visual data acquired through a shooting module. For example, the present disclosure may provide a function for acquiring visual data using a shooting module installed or built into a terminal, and protecting information displayed on the terminal based on the acquired visual data.

Meanwhile, the effects of the present disclosure are not limited to the effects mentioned above, and various effects may be included within a range apparent to those skilled in the art from the contents described below.

FIG. 1 is a block diagram of a computing device that can be utilized to implement a module according to one embodiment of the present disclosure.
FIG. 2 is a flowchart briefly illustrating an information protection method utilizing a shooting module according to one embodiment of the present disclosure.
FIG. 3 is a flowchart illustrating in more detail an information protection method utilizing a shooting module according to one embodiment of the present disclosure.
FIG. 4 is a flowchart briefly illustrating information protection steps based on facial data detection according to one embodiment of the present disclosure.
FIG. 5 is a flowchart briefly illustrating an information protection step based on matching between facial data and facial recognition data according to one embodiment of the present disclosure.
FIG. 6 is a flowchart illustrating in more detail an information protection step based on matching between one or more facial data and facial recognition data according to one embodiment of the present disclosure.
FIG. 7 is a schematic diagram schematically representing an operation in a situation where there is no facial data whose matching result falls below a reference value according to one embodiment of the present disclosure.
FIG. 8 is a schematic diagram schematically representing that a first security operation is executed when a detection result falls below a reference value according to one embodiment of the present disclosure.
FIG. 9 is a schematic diagram schematically representing an operation in a situation where there is facial data whose matching result falls below a reference value, there is facial data that satisfies the reference value, and a second mode is set, according to one embodiment of the present disclosure.
FIG. 10 is a schematic diagram schematically representing an operation in a situation where there is facial data whose matching result falls below a reference value, there is facial data that satisfies the reference value, and the first mode is set, according to one embodiment of the present disclosure.
FIG. 11 is a schematic diagram schematically representing an operation in a situation where there is facial data whose matching result falls below a reference value and there is no facial data that satisfies the reference value, according to one embodiment of the present disclosure.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to provide an understanding of the present disclosure. However, it will be apparent that these embodiments may be practiced without these specific details. Furthermore, it should be understood that the present disclosure encompasses not only the specifically presented embodiments, but also all modifications, equivalents, and alternatives falling within the technical spirit and scope of the present disclosure. Various modifications to the present disclosure will be apparent to those skilled in the art, and technical ideas that can be devised by a person skilled in the art without special technical effort based on the teachings herein should be construed as being included within the technical spirit described in the present disclosure.

In describing the present invention, detailed descriptions of related known technologies will be omitted if they are deemed to obscure the gist of the invention. However, such omissions are not intended to exclude awareness of known technologies that can be utilized in the present invention. The present invention should be construed in the broadest sense consistent with the principles and novel features disclosed herein.

Terms such as "first," "second," "third," and "fourth" may be used to describe various components, but these components should not be limited by these terms. These terms are used solely to distinguish specific components from other components, and these specific components may have different interpretations depending on the embodiment. The terms "first," "second," and the like used in this disclosure are used solely to describe specific embodiments and are not intended to limit the present invention.

In this disclosure, singular expressions include plural expressions unless the context clearly indicates otherwise. In cases where the context does not clearly indicate a singular form, the singular in this disclosure should be understood to mean "one or more."

Furthermore, the terms "includes" and/or "comprising" imply the presence of the features and/or components, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, features, or combinations thereof.

Additionally, the term "and/or" should be understood to refer to every possible combination of one or more of the listed elements or to include every possible combination of one or more of the listed elements.

Additionally, the term "or" should be interpreted to mean including at least one of the listed elements, rather than exclusively referring to any one of the listed elements. For example, the expression "N utilizes A or B" could be interpreted to mean either N utilizes A; N utilizes B; or N utilizes both A and B.

Furthermore, the term "at least one of A or B" should be interpreted to mean "if it includes only A", "if it includes only B", or "if it is combined in the composition of A and B".

In the present disclosure, terms such as "module" and "system" may refer to hardware, firmware, software, a combination of software and hardware, a combination of hardware and firmware, or an execution of software, etc., related to a computing device. For example, a module may be, but is not limited to, a processor, an object, a component, a procedure running on a processor, a program, a computing device, or a combination of two or more thereof. A module may be included in a specific computing device, but may also be configured to be distributed across two or more computing devices. Such modules may be executed from various computer-readable media. For example, modules may communicate with other modules, devices, components, or elements through local or remote processes using a signal containing one or more data packets. For example, a data packet may include data from one component interacting with another component in a local system, a distributed system, and/or data transmitted through a signal to another system and/or over a network such as the Internet.

Hereinafter, the present invention will be described in detail, focusing on embodiments thereof, with reference to the attached drawings. Like reference numerals throughout the drawings represent like elements, components, steps, or operations. The drawings attached to this specification should not be construed as limiting the scope of the present invention to the embodiments depicted in the drawings, but rather should be utilized solely as a reference point in interpreting the technical concepts of the present invention.

FIG. 1 is a block diagram of a computing device that can be utilized to implement a module according to one embodiment of the present disclosure.

The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In the present disclosure, the exemplary computing device (100) may include other components for forming or performing the computing environment of the computing device (100), and only some of the disclosed components may constitute the computing device (100). For example, the computing device configuration in the present disclosure may be applied to various terminals or components thereof.

A computing device (100) may include at least one of an input/output unit (110), a processing unit (120), a memory unit (130), or a communication unit (140). In the present disclosure, the processing unit (120) may be interpreted as a processing device, including a processor. In a computing device according to one embodiment of the present disclosure, the input unit (110) and the memory unit (130) may be optional components.

The input/output unit (110) is a component that provides various input and output means so that a user can interact with a computing device in one embodiment of the present disclosure, through which a user can input commands to the device or check information generated by the device. The input/output unit (110) is a component that includes at least one of an input unit and an output unit. As an input unit, the input/output unit (110) may include various input devices such as buttons, joysticks, keyboards, mice, touchpads, or touchscreens. These input devices may include functions that allow a user to perform tasks such as inputting data, executing commands, and selecting elements within a screen. For example, a touchscreen allows a user to intuitively interact with the device by directly touching the screen, and this can be exemplified by the use of portable computing devices such as mobile devices or tablets. In terms of the output unit, the input/output unit (110) may include various types of output devices such as a CRT (Cathode Ray Tube) monitor, a screen display, a speaker, and an LED indicator. For example, a display may provide high-resolution visual output so that a user can clearly view applications or content running on the device. In addition, auditory information transmission to the user can be performed by providing auditory notifications or signals through the speaker. For example, feedback can be provided to the user through notifications or media playback. The input/output unit (110) may also include an output device such as a head-up display (HUD) or an augmented reality (AR) or virtual reality (VR) headset to provide an immersive user experience. The input/output unit (110) provides such input or output functions to enable smooth interaction between the user and the device, and may also be utilized for interaction between other devices or other parts other than the input/output unit (110). Through the input/output unit (110), the system, device, component, etc. of the present disclosure can provide an efficient interface, etc., and the input/output unit (110) can be configured in a flexible manner in various environments. The input/output unit (110) is not limited to the examples mentioned above, and various methods or configurations that can be utilized by a person skilled in the art for inputting and outputting data or signals can be utilized in implementing the input/output unit (110).

The processing unit (120) is a component responsible for calculating, controlling, or processing data, and may be composed of one or more cores within a computing device.

For example, the processing unit (120) may include a processor such as a central processing unit (CPU), a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU) of a computing device. Furthermore, at least one of a field-programmable gate array (FPGA), a cache memory, or an auxiliary security module may be additionally included. The processing unit (120) may read data or a computer program stored in the memory unit (130) to perform data processing according to an embodiment of the present disclosure. At least one of the CPU, GPU, GPGPU, or TPU of the processing unit (120) may process the execution or learning of a network function. For example, the CPU and the GPGPU may together process learning of a network function, classification or processing of data using a network function, and execution of encryption and authentication using a network function. In addition, in one embodiment of the present disclosure, the processors of a plurality of computing devices may be used together to process learning of network functions, classification or processing of data using network functions, and execution of encryption and authentication using network functions. In one embodiment of the present disclosure, the computer program executed on the computing device may be a CPU, GPU, GPGPU, or TPU executable program. In the present disclosure, the processing unit (120) may be flexibly configured according to various requirements of the device, and includes at least one of a processor, a sensor control device, an auxiliary security module, a data bus, or a cache memory of a form that can be applied without special difficulty by a person skilled in the art.

According to one embodiment of the present disclosure, the memory unit (130) can store any form of data or information generated or determined by the processing unit (120) and any form of data or information received by the communication unit (140).

For example, the memory unit (130) is a component that performs data storage and management functions, and may serve to temporarily or permanently store data within the computing device. The memory unit (130) may be composed of various types of memory, and may, for example, contribute to optimizing data processing speed through interaction with the processing unit (120). In one embodiment, the memory unit (130) may include a main memory, and the main memory may include at least one type of storage medium among Random Access Memory (RAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Non-Volatile Dual In-line Memory Module (NVDIMM), Low Power Double Data Rate Memory (LPDDR), Hybrid Memory Cube (HMC), High Bandwidth Memory (HBM), and Persistent Memory (PMEM), but is not limited to the examples listed above. In addition, by way of example, the memory unit (130) may include non-volatile memory for long-term storage of data such as major system settings or firmware. Storage devices such as Read-Only Memory (ROM) and flash memory types may be used as the non-volatile memory. In additional embodiments, the memory unit (130) may further include an auxiliary storage device for storing and managing large amounts of data. By way of example, a Solid State Drive (SSD) or a Hard Disk Drive (HDD) may be included as part of the memory unit, and this may be used in applications requiring permanent storage of large amounts of data and fast data transmission. The description of the memory unit (130) described above is merely an example, and the memory unit (130) in the present disclosure is not limited to the examples mentioned above.

In one embodiment of the present disclosure, the communication unit (140) can perform communication with various wired and wireless networks. Through this, the communication unit (140) can enable stable and efficient data transmission in various communication environments. For example, the network may be the well-known World Wide Web (WWW). For example, the communication unit (140) can use various wired communication systems such as the Public Switched Telephone Network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and local area network (LAN). In the case of wired communication, the communication unit (140) can support various DSL methods or wide area network systems for high-speed data transmission. For example, it can accommodate various forms of wired communication, from transmission technology based on public switched telephone networks to high-speed DSL methods, to enable stable data transmission and reception. As another example, in terms of wireless communication, the communication unit (140) can support various frequency and time division methods such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), and SC-FDMA (Single Carrier-FDMA). These methods are designed to maintain a stable connection in communication even while moving. In addition, the communication unit (140) can easily connect to a local area network (LAN), a personal area network (PAN), or a wide area network (WAN), and can also support short-range wireless technologies such as Bluetooth and infrared (IrDA). Through this, the communication unit (140) can provide the optimal connection option according to the situation. In this way, the communication unit (140) is compatible with various network types and can function as a universal connection unit. This facilitates information exchange with other devices within the system and supports the effective implementation of various network-based functions. Due to the connectivity and flexibility of the communication unit (140), the system of the present disclosure can selectively apply communication methods or devices appropriate for the communication environment, and the configuration and role of the communication unit (140) are not limited to the examples mentioned above.

Hereinafter, an information protection method utilizing a photographing module according to an embodiment of the present disclosure will be described with reference to FIGS. 2 and 3, and further, a facial data detection step or a facial data matching step according to an embodiment of the present disclosure will be described in detail with reference to FIGS. 4 to 6.

FIG. 2 is a flowchart briefly illustrating an information protection method utilizing a shooting module according to one embodiment of the present disclosure.

According to one embodiment of the present disclosure, an information protection method utilizing a photographing module may include a step of acquiring visual data (S130); a step of detecting facial data in the visual data using a pre-learned model (S200); a step of executing a protection action based on the detection result and a reference value (S250); a step of matching facial data and facial recognition data using the pre-learned model (S300); and a step of executing a protection action based on the matching result and the reference value (S350).

In one embodiment, the computing device (100) can acquire visual data. For example, the step (S130) of acquiring visual data may be a step of acquiring visual data for each time zone using a photographing module. For example, the present disclosure can acquire visual data repeatedly at preset time intervals using the photographing module. These preset time intervals can be set by the user, or an optimized default setting can be provided. Alternatively, the step of acquiring visual data using the photographing module can be performed in real time. When performed in real time, it has the advantage of being able to immediately respond to visual data that changes in real time from an information protection perspective. Alternatively, the step may be applied to acquire visual data in real time when the photographing module detects an event on its own. For example, when a fixed image is being captured without significant change, the photographing module may not perform the capture, but rather acquire visual data when an event occurs, such as a change in the image or movement being recognized within the captured frame. In this way, acquiring visual data based on the occurrence of an event can be implemented in a way that utilizes relatively few computing device resources from a data processing perspective.

In the present disclosure, visual data may include 2D images, 3D images, or video data obtained by using a shooting module.

In the present disclosure, a shooting module is a module capable of shooting an image or video, and may include a webcam, a CCD (Camera Charge-Coupled Device) camera, a CMOS (Complementary Metal-Oxide Semiconductor) camera, an IR camera, a thermal imaging camera, an HDR (High Dynamic Range) camera, an autofocus camera, a fixed focus camera, a stereo camera, etc., but is not limited to the examples described above.

In one embodiment, the computing device (100) can detect facial data within visual data using a pre-learned model. For example, the step (S200) of detecting facial data within visual data using a pre-learned model may include a step of extracting facial candidate data within the visual data using the pre-learned model. Such facial candidate data may be configured in various box-shaped units. The computing device (100) may determine facial candidate data exceeding a reference value among the extracted facial candidate data as facial data. In the present disclosure, the reference value may be a preset value or a value dynamically determined based on a pre-learned model or rule. In the present disclosure, the reference values for each step may be different or the same, and a person skilled in the art may experimentally apply a reference value appropriate for each step depending on the field of application.

In the present disclosure, pre-trained models utilized to detect facial data within visual data may include, but are not limited to, object detection models such as Faster R-CNN, YOLO, and SSD. Furthermore, for example, the Intersection of Union may be utilized to evaluate how closely facial candidate data matches actual facial regions for facial data detection.

In one embodiment, the computing device (100) may perform a protection operation based on the detection result and the reference value. For example, the step (S250) of executing the protection operation based on the detection result and the reference value may include a step of executing the protection operation when the result of performing the detection of facial data within the visual data is less than the reference value, i.e., when it is determined that facial data is not detected. For example, the reference value in step S250 may be set as a first reference value. In addition, the protection operation in step S250 may be set as the first protection operation, and the pre-learned model utilized in step S250 may be set as the first pre-learned model. In other words, when it is determined that facial data is not detected based on the detection result and the reference value, it is determined that there is no human face in the visual data acquired through the shooting module, and therefore, a pre-registered user may be viewed as not being within the shooting area, and therefore, there is a need to protect information displayed on the screen of the terminal among the input/output units (110) of the computing device (100). For example, there may be cases where a third party other than the registered user may view information displayed on the terminal screen outside the shooting area of the shooting module. Accordingly, the computing device (100) may perform a protection operation to protect at least some of the information displayed on the terminal screen.

Information displayed on the terminal screen may include personal or sensitive information. For example, personal information refers to information that can directly or indirectly identify a specific individual, such as a resident registration number, a specific website ID, an email address, occupation, IP address, or company. Furthermore, sensitive information refers to information regarding an individual's private life, physical/mental state, etc., and may include private messages, medical information, Internet search results, work-related information, or information stored on a computer. In one embodiment of the present disclosure, a protection operation may be an operation to protect such personal or sensitive information. For example, the protection operation may operate by interfering with or blocking visual display on the terminal screen to protect the information displayed on the terminal screen.

For example, the first protection operation may include at least one of a filter operation or a transparency operation. In one embodiment, the filter operation may be an operation of covering a window displayed on the screen with a filter layer. The filter may be a single-colored, opaque or translucent layer, or an opaque or translucent layer with an image or picture inserted therein. In addition, the transparency operation may be an operation of adjusting the transparency of the window itself displayed on the screen. This transparency operation may form the window displayed on the screen translucent or transparent based on a preset transparency. For example, the filter operation may be utilized for the purpose of clearly obscuring information displayed on the screen, or may be utilized for the purpose of deceiving the information displayed on the screen as different information by utilizing a layer containing an image. For example, the transparency operation may have the effect of making the window to be protected transparent when the transparency is set high in advance, thereby making it difficult to visually recognize the presence of the window, thereby making it difficult for a third party to determine whether the protection operation has been performed.
In this specification, the term "window" or "window displayed on the screen" includes all forms of user interface elements displayed on the screen of a display device through a graphical user interface, as understood by those of ordinary skill in the art, and may be interpreted to include software-based windows that provide interaction with a user, such as an application execution window, a web browser window, an operating system's system settings window, a pop-up window, a dialog box, etc.

In one embodiment, the computing device (100) may perform matching of facial data and facial recognition data using a pre-learned model. For example, the step (S300) of matching facial data and facial recognition data using a pre-learned model may include a step of matching facial data detected in visual data with pre-registered facial recognition data. In one embodiment of the present disclosure, the facial recognition data is facial data for recognizing a registered user, and may be data that stores the face of a pre-registered user as an image or a converted vector. For example, when an image or video including a facial image or video of a user to be registered is acquired using a photographing module, feature information may be extracted from the acquired image or video using a pre-learned model and converted into a form suitable for facial recognition, and this may be determined to be included in the facial recognition data. In this case, the pre-learned model for extracting the feature information may be a pre-learned first model or a pre-learned second model, but may also be a different pre-learned model. For example, the pre-trained model utilized in step S300 may be the pre-trained first model of step S200 or a pre-trained second model different from the first model. The step S300 may include a step of extracting features of each face from the facial data and the facial recognition data, and calculating the similarity between the facial data and the facial recognition data based on the features. At this time, the similarity calculation may be performed using cosine similarity or Euclidean distance, or various calculation methods that can be utilized without particular difficulty by a person skilled in the art may be utilized. The result of the similarity calculation may be included in the matching result between the facial data and the facial recognition data.

In one embodiment, the computing device (100) may perform a protection operation based on the matching result and the reference value. For example, the step (S350) of executing the protection operation based on the matching result and the reference value may include a step of executing the protection operation when the result of matching the facial data and the facial recognition data is less than the reference value, i.e., when it is determined that the facial data is not the face of a pre-registered user. For example, the reference value in step S350 may be set as a second reference value. In addition, the protection operation in step S350 may be set as the second protection operation. In step S350, when it is determined that the facial data and the facial recognition data do not match based on the matching result and the reference value, in other words, it can be regarded as the facial data detected in the visual data acquired through the photographing module and the pre-registered facial recognition data do not match each other, and therefore, there is a need to protect information displayed on the screen of the terminal among the input/output units (110) of the computing device (100) from a third party who is not a pre-registered user. For example, a third party who is not a registered user may attempt to manipulate the terminal or view information displayed on the terminal's screen. Accordingly, the computing device (100) may perform a protection operation based on the matching results and reference values to protect at least some of the information displayed on the terminal's screen.

For example, the second protection action may include at least one of a filter action, a transparency action, or a notification action. In one embodiment, the filter action may be an action of covering a window displayed on the screen with a filter layer. The filter may be a single-colored, opaque or translucent layer, or an opaque or translucent layer with an image or picture inserted therein. In addition, the transparency action may be an action of adjusting the transparency of the window itself displayed on the screen. This transparency action may form the window displayed on the screen translucent or transparent based on a preset transparency. In addition, the notification action may be a notification action that notifies a pre-registered user located in front of the terminal that a third party has been detected by the shooting module, even if it does not obscure information displayed on the screen. This notification action may function to notify, through a notification, that a third party can view the terminal screen when a pre-registered user is detected and recognized on the visual data and an unregistered third party is detected at the same time. For example, a filter operation can be utilized to clearly obscure information displayed on the screen, or it can be utilized to disguise information displayed on the screen as different information by utilizing a layer containing an image. For example, a transparency operation can have the effect of making it difficult to visually recognize the existence of a window to be protected by making it transparent when the transparency has been set high in advance, thereby making it difficult for a third party to determine whether a protection operation has been performed. For example, a notification operation, unlike a filter operation or a transparency operation, does not obscure information displayed on the terminal screen, but rather provides a notification to the user, thereby allowing the user to be aware of detection by a third party without interfering with the user's use of the terminal. Such a notification operation may be designed as a minimum information protection measure, taking into account that registered users may also experience interference with the use of preset protection apps when a filter operation or a transparency operation is activated.

In the present disclosure, a preset protection app refers to a program or app stored or running on a computing device, and may include apps or programs such as messengers, email, productivity programs, and work tools, among which the user has preset or set an initial value. For example, such a preset protection app may be one that a user has preset among apps or programs existing in the computing device, registered in advance.

FIG. 3 is a flowchart illustrating in more detail an information protection method utilizing a shooting module according to one embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device (100) may pre-register facial recognition data. For example, the step of pre-registering facial recognition data (S110) may include a step of pre-registering facial recognition data of a specific user in order to register the user. Such facial recognition data may be data generated using at least one of a frontal face image, a side face image, an upper and lower face image, or a facial image of the specific user. The step of pre-registering facial recognition data may include a step of the computing device (100) using a photographing module to photograph the face of the specific user; and a step of standardizing the photographed image or video. The standardization may be configured to include at least one of a step of standardizing pixels, a step of correcting differences due to an angle or lighting of the face, a vectorization step, a step of removing a background other than the face, or a step of extracting features for facial recognition from the vectorized data.

According to one embodiment of the present disclosure, the computing device (100) can standardize facial recognition data and visual data. For example, the step of standardizing the facial recognition data and the visual data (S150) may include the step of standardizing the facial recognition data to a state optimized for facial recognition; and the step of standardizing the visual data to a state optimized for facial detection. For example, the step of standardizing to a state optimized for facial recognition may include at least one of the steps of standardizing pixels, correcting differences due to angle or lighting of the face, vectorization, feature extraction, or removing a background other than the face. In addition, the step of standardizing to a state optimized for facial detection may include at least one of the steps of standardizing pixels, vectorization, or segmenting an image into multiple rectangular anchors and converting them into a tensor form.

FIG. 4 is a flowchart briefly illustrating information protection steps based on facial data detection according to one embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device (100) can detect facial data in visual data using a pre-learned model, and based on the detection result and a reference value, execute a first protection operation or match the facial data with facial recognition data. For example, if the detection result is less than the reference value, that is, if it is determined that there is no facial data in the visual data, the computing device (100) can execute the first protection operation (S410). Alternatively, if the detection result is not less than the reference value or satisfies the reference value, that is, if it is determined that facial data has been detected in the visual data, the computing device (100) can perform a step of matching the detected facial data. The step of matching the detected facial data may be a step (S300) of matching the detected facial data with facial recognition data using a pre-learned model. In this way, if the above detection result satisfies the reference value, i.e., if it is determined that facial data exists in the visual data, step S300 can be performed using the detected facial data.

In one embodiment of the present disclosure, the step (S410) of executing the first protection operation may be a step for protecting information displayed on the screen of the terminal when it is determined that no face is detected within the shooting area of the shooting module. This first protection operation may be intended to prevent information displayed on the screen of the terminal from being exposed to the outside when there is no person within the shooting area of the shooting module. In addition, the first protection operation may function to prevent unregistered third parties from viewing the information displayed on the screen of the terminal outside the shooting area of the shooting module.

As mentioned above, the reference value in the facial data detection stage can be set as the first reference value.

FIG. 5 is a flowchart briefly illustrating an information protection step based on matching between facial data and facial recognition data according to one embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device (100) may use a pre-learned model to match detected facial data with facial recognition data, and may or may not execute a second protection operation based on the matching result and a reference value. For example, if the matching result is less than the reference value, that is, if the detected facial data is determined to be a different face from the facial recognition data, the computing device (100) may execute the second protection operation (S420). Alternatively, if the matching result is not less than the reference value or satisfies the reference value, that is, if the facial data detected in the visual data is determined to be the same face as the facial recognition data, the computing device (100) may not execute the second protection operation. In this way, if the matching result satisfies the reference value, that is, if the facial data and the facial recognition data are determined to be the face of the same person, it can be seen that a registered user is within the shooting area, and therefore, a separate protection operation may not be executed for information displayed on the screen of the terminal.

As previously mentioned, a second reference value can be set as the baseline during the matching process between facial data and facial recognition data. This reference value, as a parameter that determines the sensitivity of detection and matching while also influencing the likelihood of false positives, can be specified experimentally or dynamically set using artificial intelligence, depending on the specific use case.

FIG. 6 is a flowchart illustrating in more detail an information protection step based on matching between one or more facial data and facial recognition data according to one embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device (100) may match detected facial data with facial recognition data using a pre-trained model. At this time, the detected facial data may include one or more facial data, and the facial recognition data may also include one or more facial recognition data. For example, if there is facial data whose matching result falls below a threshold, this may mean that at least one face of a third party other than the registered user has been detected. Furthermore, if there is no facial data whose matching result falls below a threshold, this may mean that only the faces of the registered users have been detected. In this case, even if there are multiple registered users and multiple facial recognition data corresponding to each registered user, if there is no facial data whose matching result falls below a threshold, this may mean that only the faces of the registered users have been detected. Consequently, if there is no facial data whose matching result falls below a threshold, it may be determined that only the registered user is viewing the screen displayed on the terminal, and therefore the computing device (100) may not perform a separate protection operation.

According to one embodiment of the present disclosure, if there is facial data whose matching result falls short of a reference value, the computing device (100) can calculate whether there is facial data that satisfies the reference value. In this case, the presence of facial data that satisfies the reference value means that there is facial data whose matching result falls short of the reference value, and at the same time, there is facial data that satisfies the reference value. Therefore, this may mean that the faces of both a registered user and an unregistered third party are detected. In this case, depending on the intention of the registered user, the operation may be applied differently based on the set mode.

In one embodiment of the present disclosure, the first mode may be a mode set to perform a second protection operation when, as a result of matching, one or more pieces of facial data fall below a second reference value. For example, the first mode may be a mode for protecting information displayed on the terminal screen as long as a third party other than the registered user is present in the shooting area, even if the registered user is present in the shooting area. For example, the first mode may be a mode in which only the registered user can view various pieces of information displayed on the terminal screen. This first mode may be useful when information that is not desired to be shared with a third party or should not be shared is displayed on the screen.

In one embodiment of the present disclosure, the second mode may be a mode set to perform a second security operation when, as a result of matching, there is no facial data that satisfies the second criterion. For example, the second mode may be a mode in which, if there is facial data that falls below the criterion as a result of matching and there is no facial data that satisfies the criterion, a second protection operation is executed (S420). For example, the second mode may be a mode in which, if a registered user is present in the shooting area, information displayed on the terminal screen is not protected even if a third party other than the registered user is determined to be present in the shooting area. However, in the case set to the second mode, even if information on the terminal screen is not actively protected, a notification operation may be performed so that the registered user can recognize the presence of a third party in the shooting area. The notification operation may be included as an example of the second protection operation, but the notification operation is not the second protection operation. This notification action can utilize various notification signals such as a notification window displayed on the terminal screen, a notification sound, and vibration to notify the registered user of the presence of an unregistered third party detected within the shooting area. For example, an image such as a character or a shape can be displayed during the notification action, and depending on whether the location of the image displayed during the notification action is on the left or right, it can have a function of notifying whether the detected unregistered third party was detected on the left or right of the shooting area. This second mode can be used to allow an unregistered third party to view information displayed on the terminal screen while the registered user is using the terminal. For example, in a case where a registered user needs to view content or work-related documents on the terminal screen together with an unregistered third party, unlike the first mode, the information displayed on the terminal screen can be viewed together through the second mode.

Through multiple modes, including the first and second modes, the present disclosure can optimize the process for protecting information displayed on the terminal screen based on each situation, the user's intent for each situation, or the characteristics of the location where the terminal is located. Registered users can switch between these multiple modes and utilize the method of the present disclosure, thereby utilizing an information protection method optimized for each situation (e.g., the location of the terminal, the user's intent, etc.).

In one embodiment of the present disclosure, the step (S150) of standardizing facial recognition data and visual data may be performed in the step (S110) of pre-registering facial recognition data or the step (S130) of acquiring visual data. For example, the facial recognition data may include an image or video of the face of the user to be registered, captured from at least one of several directions. For example, the visual data may include an image or video of a shooting area captured by a shooting module. Alternatively, it should be understood that facial recognition data and visual data that have been standardized from the perspective of optimization for the detection or matching step are also included in the facial recognition data and visual data. That is, for example, the step of standardizing facial recognition data may be a step of standardizing the original image of the registered user's face by optimizing it for facial recognition, or a step of performing a secondary standardization, such as vectorization, on the original image of the registered user's face after undergoing primary standardization, such as pixel or angle adjustment, to optimize it for facial recognition. In one embodiment, the description of the above-described step of standardizing facial recognition data may also be applied to the step of standardizing visual data.

In one embodiment of the present disclosure, if there is at least one facial data satisfying a reference value as a result of matching facial recognition data and facial data, the computing device (100) may track the facial data satisfying the reference value using a pre-trained model. For example, if there is at least one facial data satisfying the reference value as a result of matching facial recognition data and facial data, the step of tracking the facial data satisfying the reference value using a pre-trained model may be to track facial data determined to be a registered user to increase the recognition rate and decrease the false positive rate. When utilizing such tracking of facial data, continuous facial recognition can be performed with low resources in real-time webcam images, and the information protection method of the present disclosure can be implemented more efficiently than detecting and recognizing facial data from visual data newly acquired every hour.

Hereinafter, the operation of a computer program according to an embodiment of the present disclosure will be briefly described in various situations through FIGS. 7 to 11.

FIG. 7 is a schematic diagram schematically representing an operation in a situation where there is no facial data whose matching result is below a reference value according to one embodiment of the present disclosure.

In one embodiment of the present disclosure, when facial data that satisfies a reference value is detected from visual data, and a matching result of the facial data and facial recognition data satisfies the reference value, the computing device (100) can display information to be displayed on the screen of the terminal as is without visually controlling the information without executing a separate protection operation.

FIG. 8 is a schematic diagram schematically representing that a first security operation is executed when a detection result falls below a reference value according to one embodiment of the present disclosure.

In one embodiment of the present disclosure, if facial data satisfying a reference value is not detected in the visual data, the computing device (100) performs a first protection operation to visually control information to be displayed on the terminal screen, thereby preventing certain information on the terminal screen from being viewed by third parties. This allows content to be visually protected on the terminal screen even when a third party is outside the shooting area of the shooting module.

FIG. 9 is a schematic diagram schematically representing an operation in a situation where there is facial data whose matching result falls below a reference value, there is facial data that satisfies the reference value, and a second mode is set, according to one embodiment of the present disclosure.

In one embodiment of the present disclosure, when a plurality of facial data satisfying a reference value are detected, the computing device (100) may not perform a protection operation or may only perform a notification operation if, even if there is facial data satisfying the reference value as a result of matching, there is facial data falling below the reference value as a result of matching and the second mode is preset. Through this, when a registered user is present within the shooting area, the display information of the protected app or program can be displayed on the screen as is under the responsibility of the registered user. For example, the notification operation may serve to notify the registered user of the presence of an unregistered third party even if there is no visual control function for the information displayed on the screen.

FIG. 10 is a schematic diagram schematically representing an operation in a situation where there is facial data whose matching result falls below a reference value, there is facial data that satisfies the reference value, and the first mode is set, according to one embodiment of the present disclosure.

In one embodiment of the present disclosure, when multiple facial data satisfying a reference value are detected, the computing device (100) may perform a second protection operation if, even if the matching result includes facial data satisfying the reference value, the matching result includes facial data falling below the reference value and the second mode is preset. This allows for specific information to be viewed only by registered users, even if registered users are present within the capture area.

FIG. 11 is a schematic diagram schematically representing an operation in a situation where there is facial data whose matching result falls below a reference value and there is no facial data that satisfies the reference value, according to one embodiment of the present disclosure.

In one embodiment of the present disclosure, if one or more facial data satisfying a threshold value is detected, and if the matching result does not contain any facial data satisfying the threshold value, the computing device (100) may perform a second protection operation. This prevents third parties other than registered users from visually receiving information regarding the protected app or program, thereby protecting personal or sensitive information.

In FIGS. 7 to 11, the protective operations on the drawings are depicted as only filter operations with a filter applied, but the protective operations are various operations that visually control the provision of information, and are not limited to filter operations, and may include transparency operations, etc.

In one embodiment of the present disclosure, a method for protecting information utilizing a photographing module may include, as a preparatory step, a step in which a user registers his/her face as facial recognition data using the photographing module and presets an app or program to be protected. Accordingly, the method of the present disclosure can protect personal information or sensitive information while reducing the degree of interference with the user's work compared to a method of protecting information by covering the entire terminal screen by individually controlling the app or program containing the information requiring protection. Furthermore, the method of the present disclosure can be utilized in various situations and intentions by setting a mode suited to the user's intention for each situation through multiple modes. For example, when applied only to a messenger app, the method of the present disclosure can be utilized in a manner that protects personal information or sensitive information by ensuring that only registered users can view the contents of the messenger and making the contents of the messenger invisible when the registered user leaves the shooting area or a third party is detected within the shooting area. In addition to controlling directory access related to the execution or operation of the app or program to be protected, one of the technical features of the present disclosure is visually controlling the information of the app or program displayed on the screen through the protection operation. As in one embodiment of the present disclosure, when performing an information protection function by utilizing visual control over information displayed on a terminal screen, it does not directly affect the operation and processing of each app or program, thereby reducing the possibility of execution errors due to the protection operation compared to other information protection methods, and may also have advantages in terms of stability for each terminal. The present disclosure, which includes these features, may have the purpose of implementing soft security that enhances usability and stability even when security is adjusted to an appropriate level compared to an information protection method based on complete access control.
The term 'app or program' as used herein may be used to mean application software or an executable program, as understood by a person skilled in the art.

The present disclosure may be implemented through a combination of hardware and software, including computer-executable instructions and program modules that run on one or more computers. The methods of the present disclosure may be implemented on various computer devices, including single-processor systems, multi-processor systems, personal computers, handheld computing devices, or programmable electronic devices, and are not limited to the examples mentioned.

In general, program modules may include routines, programs, components, data structures, etc. that perform specific tasks or implement specific data types. Various methods of the present disclosure may also be implemented in distributed computing environments via communication between remote devices, in which program modules may be located in local or remote memory devices.

In the present disclosure, a computer-readable medium can be used to store computer commands, data structures, program modules, etc. in various forms such as volatile, non-volatile, temporary, non-transitory, removable, and non-removable. For example, the computer-readable storage medium can include RAM, ROM, EEPROM, flash memory, CD-ROM, DVD, magnetic disk, etc., and these can be media for storing data and providing desired information.

Additionally, in the present disclosure, a computer-readable transmission medium includes a medium capable of transmitting commands and data via a transmission mechanism such as a carrier wave or a modulated data signal, and includes both wired and wireless networks. A combination of these media may also be classified as a computer-readable transmission medium.

In the present disclosure, a computer may be comprised of a processing unit, system memory, and a system bus, and the system bus may be connected to various bus architectures, such as a memory bus and a peripheral bus. The system memory may include ROM or RAM, and the memory may also include a basic input/output system (BIOS) that provides the computer's basic input/output functions and information at startup.

In the present disclosure, a computer includes various storage devices, such as a hard disk drive, a floppy disk drive, and an optical disk drive, which may be connected to a system bus to provide volatile or non-volatile storage of data and instructions. The HDD, floppy disk drive, and optical drive may be connected via their respective interfaces, and may include interfaces necessary for implementing external drives, such as a Universal Serial Bus (USB) or IEEE 1394.

In the present disclosure, the computer stores various program modules, such as an operating system, application programs, other program modules, and program data, which may be cached in RAM. The computer may run on various commercial operating systems and combinations thereof.

Users can input information into a computer through input devices such as a keyboard or mouse. These input devices can be connected to a processing unit via an input device interface connected to the system bus. A display device, such as a monitor, can also be connected to the system bus, providing the computer's output.

In the present disclosure, a computer can be logically connected to a remote computer in a network environment, such as a local area network (LAN) or a wide area network (WAN). In this case, the computer can be connected to the network via a wired or wireless communication network interface or modem, and can be connected to an external network, such as the Internet. In a networked environment, portions of the program modules may be stored on a remote storage device.

The computer of the present disclosure can communicate with various wireless devices, such as printers, scanners, PDAs, or communication satellites. Data can be transmitted and received within a base station via wireless technologies such as Wi-Fi and Bluetooth, and wireless communication can be utilized to support connections between computers and Internet access. These Wi-Fi networks can provide secure, reliable, high-speed wireless connections using IEEE 802.11 technology.

These computing environments allow for the integration of various devices and functions, and the inventions described in this disclosure can be implemented or practiced in single or distributed computing environments. It should be understood that the order or hierarchical structure of the steps depicted in the presented processes may be merely examples of the present invention. Within the scope described in this disclosure, the individual steps included in the processes may be rearranged, and the order and hierarchy may be changed without departing from the scope of the present disclosure. While the present disclosure provides elements of various steps through the claims, it is not limited to the presented order or hierarchy. The descriptions of the presented embodiments are provided to enable those skilled in the art to practice the present invention, and various modifications based on these embodiments can be made by those skilled in the art. Therefore, the present disclosure is not limited to the presented embodiments, but should be interpreted in the broadest scope in consideration of the technical spirit and technical significance described in the present disclosure.

Claims (15)

As an information protection method utilizing a shooting module,
A step of acquiring visual data by using a shooting module;
A step of presetting application software or an executable program that is the target of a protection operation and receiving it;
A step of detecting facial data within the visual data by utilizing a pre-learned first model;
A step of executing a first protection action based on the result of the above detection and the first threshold;
A step of matching the detected facial data and facial recognition data using the pre-learned first model or the pre-learned second model; and
A step of executing a second protection operation based on the result of the above matching and the second reference value.
Including,
The above first protection operation and the above second protection operation are,
Includes individual visual control actions for application software or executable programs that are subject to the above protection actions with respect to information displayed on the terminal screen;
The above first protective action is,
Contains at least one of a filter action or a transparency action,
The above second protective action is,
Contains at least one of a filter action, a transparency action, or a notification action;
The above transparency operation is,
An action that adjusts the transparency of the graphical user interface window (GUI window) displayed on the screen.
method.
In the first paragraph,
The step of acquiring visual data using the above shooting module is:
Repeatedly performed at preset time intervals,
method.
In the first paragraph,
The step of acquiring visual data using the above shooting module is:
Performed in real time,
method.
In the second or third paragraph,
The step of executing the first protection action based on the result of the above detection and the first threshold value is as follows:
If the result of the above detection falls below the first reference value, a step of executing the first protection operation; and
A step of recognizing that facial data has been detected when the result of the above detection satisfies the first criterion.
including,
method.
In paragraph 4,
The step of executing the second protection operation based on the result of the above matching and the second reference value is:
A step of executing a second protection operation when the result of the above matching falls below the second reference value.
Including
method.
delete In paragraph 4,
The above detected facial data includes one or more facial data.
method.
In paragraph 7,
The method further includes a step of setting one of a plurality of modes according to a user's selection,
The above multiple modes include a first mode and a second mode,
method.
In paragraph 8,
The above first mode is,
As a result of the above matching, if there is one or more facial data that falls below the second reference value, a mode set to perform the second protection operation,
method.
In paragraph 9,
The second mode is,
As a result of the above matching, if there is no facial data that satisfies the second reference value, the mode is set to perform the second protection operation.
method.
In paragraph 10,
The above method,
Step of pre-registering facial recognition data of at least one user
Including more,
The above first reference value and the above second reference value are preset,
method.
In paragraph 10,
The above method,
A step of standardizing the above facial recognition data and the above visual data.
including more,
method.
In paragraph 12,
The above method,
As a result of the above matching, if there is one or more facial data satisfying the second criterion, a step of tracking the facial data satisfying the second criterion by utilizing the pre-learned model.
including more,
method.
A computer program stored in a computer-readable storage medium, wherein when the computer program is executed on one or more processors, the computer program causes the one or more processors to perform the following operations for protecting information by utilizing a shooting module, the operations being:
An action to acquire visual data by utilizing a shooting module;
The application software or executable program that is the target of the protection action is preset and the action of receiving it;
An operation of detecting facial data within the visual data by utilizing a pre-learned first model;
An action of executing a first protection action based on the result of the above detection and the first threshold value;
An operation of matching the detected facial data and facial recognition data using the pre-learned first model or the pre-learned second model; and
An operation for executing a second protection operation based on the result of the above matching and the second reference value.
Including,
The above first protection operation and the above second protection operation are,
Includes individual visual control actions for application software or executable programs that are subject to the above protection actions with respect to information displayed on the terminal screen;
The above first protective action is,
Contains at least one of a filter action or a transparency action,
The above second protective action is,
Contains at least one of a filter action, a transparency action, or a notification action;
The above transparency operation is,
An action that adjusts the transparency of the graphical user interface window (GUI window) displayed on the screen.
A computer program stored on a computer-readable storage medium.
As a computing device,
at least one processor; and
memory
Including,
At least one processor,
Acquire visual data through the shooting module;
The application software or executable program that is the target of the protection action is preset and receives it;
Detecting facial data within the visual data by utilizing a pre-trained first model;
Execute a first protection action based on the result of the above detection and the first threshold;
Matching the detected facial data and facial recognition data using the pre-learned first model or the pre-learned second model; and
It is configured to execute a second protection operation based on the result of the above matching and the second reference value,
The above first protection operation and the above second protection operation are,
Includes individual visual control actions for application software or executable programs that are subject to the above protection actions with respect to information displayed on the terminal screen;
The above first protective action is,
Contains at least one of a filter action or a transparency action,
The above second protective action is,
Contains at least one of a filter action, a transparency action, or a notification action;
The above transparency operation is,
An action that adjusts the transparency of the graphical user interface window (GUI window) displayed on the screen.
device.