KR102690927B1 - Appartus of providing service customized on exhibit hall and controlling method of the same - Google Patents

Abstract

A device for providing customized exhibition hall services and a method for controlling the same are disclosed. According to this, an image processing unit that acquires a first image of a user adjacent to an exhibition object and detects a second image of a face area by applying an object detection algorithm to the first image; If it is determined that the user is looking at the exhibit by applying an eye tracking algorithm to the second image, facial feature points are detected from the second image and the user's view of the exhibit is based on the feature point. an interest estimation unit that estimates the emotions of; and a control unit that analyzes the correlation between the user's emotions and the level of interest in the exhibit, and generates and quantifies the level of interest as numerical data based on the correlation. may include.

Description

Exhibition hall customized service provision device and control method {APPARTUS OF PROVIDING SERVICE CUSTOMIZED ON EXHIBIT HALL AND CONTROLLING METHOD OF THE SAME}

The present invention relates to an exhibition hall customized service provision device and a control method thereof, and more specifically, to an exhibition hall customized service provision device and a control method thereof that can provide objective indicators of visitors' level of interest in exhibition items.

Recently, various technologies and methods have been proposed in various service fields to increase the competitiveness of services by identifying users' individual needs and providing customized services. To achieve this, the individual needs of users must be accurately and effectively identified.

The most commonly used methods to identify users' individual needs are based on analyzing physical information such as gender or age, or social information such as occupation or education level. However, this information is limited to a small number of classification criteria and cannot meet the increasingly complex individual needs of users.

Accordingly, a method for analyzing the user's emotional state based on image processing was also proposed. However, current emotional state analysis methods only recognize emotions that change in real time according to the user's personal state.

When providing services at an exhibition hall, the level of interest visitors have in exhibits must be analyzed to increase visitor satisfaction and response. However, there is currently no objective way to analyze visitors' interest in exhibits.

Korean Patent Publication No. 10-2295807 (2021.08.25.)

The problem to be solved by the present invention is to provide an exhibition hall customized service provision device and a control method that can analyze interest in exhibition objects and provide objective data.

In addition, the problem to be solved by the present invention is to provide an exhibition hall customized service provision device and a control method that can more accurately and reliably analyze interest in exhibition objects by applying various parameters and analysis methods.

An exhibition hall customized service provision device according to an embodiment of the present invention acquires a first image of a user adjacent to an exhibition object and applies an object detection algorithm to the first image to detect a second image of the face area. image processing unit; If it is determined that the user is looking at the exhibit by applying an eye tracking algorithm to the second image, facial feature points are detected from the second image and the user's view of the exhibit is based on the feature point. an interest estimation unit that estimates the emotions of; and a control unit that analyzes the correlation between the user's emotions and the level of interest in the exhibit, and generates and quantifies the level of interest as numerical data based on the correlation. Includes.

In the exhibition hall customized service providing device, the interest estimator estimates the direction of the face from the second image, determines whether the direction of the face is toward the exhibition object, and calculates a viewing angle of the user's gaze. After determining whether the exhibition object is included within the viewing angle, it can be determined based on this whether the user is looking at the exhibition object.

In the exhibition hall customized service providing device, the control unit inputs the second image into a convolutional neural network related to face recognition, extracts a plurality of feature vectors from a plurality of layers constituting the convolutional neural network, and extracts a plurality of feature vectors to determine gender and age. Information is extracted, but the plurality of layers are divided into a first layer group consisting of layers lower than the predetermined layer and a second layer group consisting of the predetermined layer and layers higher than the predetermined layer based on the predetermined layer. And, gender can be recognized based on the feature vector extracted by the first layer group, and age can be recognized based on the feature vectors extracted by the first layer group and the second layer group.

In the exhibition hall customized service providing device, the control unit sets a tracking ID by applying an image tracking algorithm to the second image, generates a detection area of interest based on the tracking ID, and calculates the Kalman from the detection area of interest. A predicted region of interest estimated through a filter is generated, and the detection region of interest and the predicted region of interest can be matched one-to-one based on the size of the intersection area of the detection region of interest and the predicted region of interest.

A control method of an exhibition hall customized service provision device according to another embodiment of the present invention includes acquiring a first image of a user adjacent to an exhibition object, applying an object detection algorithm to the first image, and generating a second image of the face area. detecting; If it is determined that the user is looking at the exhibit by applying an eye tracking algorithm to the second image, facial feature points are detected from the second image and the user's view of the exhibit is based on the feature point. estimating emotions; and analyzing the correlation between the user's emotions and interest in the exhibition object; And, generating and quantifying the interest level as numerical data based on the correlation; may include.

In the control method of the exhibition hall customized service providing device, the direction of the face is estimated from the second image to determine whether the direction of the face is toward the exhibition object, the viewing angle of the user's gaze is calculated, and the viewing angle is calculated. After determining whether the exhibition object is included in the display, it can be determined based on this whether the user is looking at the exhibition object.

In the control method of the exhibition hall customized service providing device, the second image is input to a convolutional neural network related to face recognition, and a plurality of feature vectors are extracted from a plurality of layers constituting the convolutional neural network to provide gender and age information. Extracting the plurality of layers, dividing the plurality of layers into a first layer group consisting of layers lower than the predetermined layer and a second layer group consisting of the predetermined layer and layers higher than the predetermined layer, and , gender can be recognized based on the feature vector extracted by the first layer group, and age can be recognized based on the feature vectors extracted by the first layer group and the second layer group.

In the control method of the exhibition hall customized service providing device, an image tracking algorithm is applied to the second image to set a tracking ID, a detection area of interest is generated based on the tracking ID, and a Kalman filter is used from the detection area of interest. A predicted region of interest estimated through is generated, and the detection region of interest and the predicted region of interest can be matched one-to-one based on the size of the intersection area of the detection region of interest and the predicted region of interest.

According to an embodiment of the present invention, interest in exhibition objects can be analyzed and provided as objective data, and customized services can be provided to visitors based on the objective data.

Additionally, according to an embodiment of the present invention, the degree of interest in an exhibit can be analyzed more accurately and reliably by applying various parameters and analysis methods.

Furthermore, according to an embodiment of the present invention, the next exhibit or content of the exhibition hall can be determined based on the acquired objective data and customized services optimized for each class can be provided.

Figure 1 is a block diagram showing the configuration of an exhibition hall customized service provision device according to the present invention.
FIG. 2 is a diagram illustrating a process in which an exhibition hall customized service providing device generates statistical data on interest level according to an embodiment of the present invention.
Figure 3 is a diagram for explaining how the exhibition hall customized service provision device according to the present invention detects the gaze of a visitor.
Figure 4 is a diagram for explaining a case where the exhibition hall customized service provision device according to the present invention uses a learned neural network model.
Figure 5 is a diagram illustrating an algorithm for preventing overlapping interest levels for each visitor in the exhibition hall customized service provision device according to the present invention.
Figure 6 shows an example of applying an algorithm for preventing overlapping interests in the present invention.
Figure 7 is a diagram for explaining how the exhibition hall customized service provision device according to the present invention recognizes the emotions of visitors.
Figure 8 is a diagram showing the control process of the exhibition hall customized service provision device according to the present invention.
9 is a diagram illustrating a computing device according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

In this specification, duplicate descriptions of the same components are omitted.

Also, in this specification, when a component is mentioned as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but may be connected to the other component in the middle. It should be understood that may exist. On the other hand, in this specification, when it is mentioned that a component is 'directly connected' or 'directly connected' to another component, it should be understood that there are no other components in between.

Additionally, the terms used in this specification are merely used to describe specific embodiments and are not intended to limit the present invention.

Also, in this specification, singular expressions may include plural expressions, unless the context clearly dictates otherwise.

In addition, in this specification, terms such as 'include' or 'have' are only intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and one or more It should be understood that this does not preclude the presence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof.

Also, in this specification, the term 'and/or' includes a combination of a plurality of listed items or any of the plurality of listed items. In this specification, 'A or B' may include 'A', 'B', or 'both A and B'.

Additionally, in this specification, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

Figure 1 is a block diagram showing the configuration of an exhibition hall customized service provision device according to the present invention.

The exhibition hall customized service providing device 100 according to the present invention can analyze various preferences and interests of visitors visiting the exhibition hall, and for this purpose, it can be implemented in the form of an exhibition hall robot, server device, kiosk terminal, etc.

The exhibition hall customized service providing device 100 according to the present invention may be configured to include an image processing unit 110, an interest estimation unit 120, and a control unit 130.

The image processing unit 110 can obtain and process images required to provide customized services for the exhibition hall. To this end, the image processing unit 110 may be implemented as a general camera, stereo camera, depth camera, scanner, etc.

According to one example, the image processing unit 110 may include a camera (not shown). Here, the camera (not shown) may be turned on and perform photography according to any one of a control signal received from the controller 130, satisfaction of a preset condition, and a preset event.

A camera (not shown) can convert a captured image into an electrical signal and generate image data based on the converted signal. For example, a subject is converted into an electrical image signal through a semiconductor optical device (CCD; Charge Coupled Device), and the converted image signal can be amplified and converted into a digital signal and then processed.

Specifically, the image processing unit 110 may acquire a first image of a user adjacent to an exhibition object and detect a second image of the face area by applying an object detection algorithm to the first image.

According to one example, the first image is an image of a user approaching within a predetermined distance from an exhibition object or in a stationary state adjacent to an exhibition object. Preferably, the first image may be captured to include most of the user's face. The second image is an image in which only the face area is detected from the first image. Preferably, a plurality of second images may be captured from all sides of the user to determine the direction of gaze, viewing angle of gaze, face direction, etc.

The interest estimator 120 may estimate the user's interest in the exhibition object.

Specifically, the interest estimator 120 applies an eye-tracking algorithm to the second image, and when it is determined that the user is looking at the exhibition object, the interest estimator 120 detects facial feature points from the second image and displays the exhibition object based on the feature point. You can estimate the user's feelings about it.

According to one embodiment, the interest estimator 120 estimates the direction of the face from the second image, determines whether the direction of the face is toward the exhibition object, and calculates the viewing angle of the user's gaze to determine whether the exhibition object is within the viewing angle. After determining whether it is included, it is possible to determine whether the user is looking at the exhibition object based on this. For example, even when the direction of the user's face is toward an exhibition object, if the exhibition object deviates from the viewing angle of the user's gaze, the interest estimator 120 may classify this as invalid data and exclude it from the analysis target.

The control unit 130 can analyze the correlation between the user's emotions and the level of interest in the exhibit, and generate and quantify the level of interest as numerical data based on the correlation.

Since the user's emotions are temporary, although they may serve as a reference when providing current services, they cannot be helpful in planning future service directions or providing optimized services to users. Therefore, in the present invention, the correlation between emotions and interest and interest are extracted based on the user's emotions through various artificial intelligence learning methods.

Here, the level of interest can be quantified with objective data. When quantifying the level of interest, the level of interest in exhibits can be created as statistical data, and these statistical data can be used as objective reference data in providing customized services for the exhibition hall.

Furthermore, interest can be analyzed in relation to gender and age group, etc. In this case, the correlation between interest and gender and age group can be derived, and based on this, exhibits or exhibition directions can be set.

Specifically, the control unit 130 can extract necessary information from the second image by performing learning using artificial intelligence.

According to one embodiment, the control unit 130 inputs the second image into a convolutional neural network related to face recognition, extracts a plurality of feature vectors from the plurality of layers constituting the convolutional neural network, and extracts gender and age information. You can. In this case, the plurality of layers are divided into a first layer group composed of layers lower than the predetermined layer and a second layer group composed of layers higher than the predetermined layer and the predetermined layer based on the predetermined layer, and the first layer group Gender can be recognized based on the feature vector extracted by the group, and age can be recognized based on the feature vector extracted by the first layer group and the second layer group. In other words, information that can be extracted from rough features, such as gender, is learned by lower layers, and information that can be extracted from detailed features, such as age, is learned by not only low but also high layers. .

When extracting interest, the control unit 130 can prevent duplication of the same user from occurring. For example, if the interests of the same user are extracted repeatedly, the information is distorted. Therefore, in the present invention, the detected area of interest and the predicted area of interest for the user are matched one-to-one, and based on this, the level of interest is extracted to prevent duplication of the same user.

To this end, the control unit 130 sets a tracking ID by applying an image tracking algorithm to the second image, creates a detection area of interest based on the tracking ID, and predicts interest estimated from the detection area of interest through a Kalman filter. A region can be created, and the detection region of interest and the predicted region of interest can be matched one-to-one based on the size of the intersection of the detection region of interest and the prediction region of interest.

According to one embodiment, in order to execute an artificial neural network model, the control unit 130 uses a general-purpose processor such as a CPU, an AP, a digital signal processor (DSP), a graphics-specific processor such as a GPU, a vision processing unit (VPU), or an NPU. It can be implemented through a combination of the same artificial intelligence-specific processor and software. The control unit 130 may control input data to be processed according to predefined operation rules or artificial neural network models stored in memory. Alternatively, if the control unit 130 is a dedicated processor (or an artificial intelligence dedicated processor), it may be designed with a hardware structure specialized for processing a specific artificial neural network model. For example, hardware specialized for processing a specific artificial neural network model can be designed as a hardware chip such as ASIC or FPGA. When the control unit 130 is implemented with a dedicated processor, it may be implemented to include a memory for implementing an embodiment of the present disclosure, or may be implemented to include a memory processing function for using an external memory.

According to one embodiment, the control unit 130 may acquire the viewer's gaze data using a learned artificial neural network model. According to one example, the artificial neural network model may be a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a deep Q-network ( Deep Q-Networks), etc., but the present invention is not limited thereto.

In this way, the exhibition hall customized service provision device 100 according to the present invention acquires images from cameras installed on exhibition objects or directly acquires images, and obtains images in order to extract statistical data on the level of interest necessary for providing services in the exhibition hall. The level of interest is extracted by applying face detection, identification, gaze and facial expression recognition technology, and interest can be estimated by gender and age group by analyzing the level of interest in relation to gender and age group.

FIG. 2 is a diagram illustrating a process in which an exhibition hall customized service providing device generates statistical data on interest level according to an embodiment of the present invention.

In order to generate statistical data on the level of interest, the exhibition hall customized service providing device 100 may include a plurality of calculation modules. In this case, the plurality of calculation modules may be included in one component previously described in FIG. 1 or may be included in two or more components, and depending on the embodiment, may be a separate component that exists independently from the components shown in FIG. 1. It may also be included in an element.

The stereo image recognition module 210 acquires an image of a viewer.

In this case, multiple images can be obtained for all sides of the viewer by using a stereo camera or rotating a single camera.

According to one example, the image captured by the stereo image recognition module 210 may include at least a front image and a side image. In this case, by comparing the front image and the side image, it is possible to determine the viewer's gaze position and viewing angle.

The analysis module 200 analyzes gender and age group, and may be comprised of a face detection and tracking module 222, a face pose and reaction estimation module 224, and a gender and age recognition module 226.

The face detection and tracking module 222 may detect and track a face in the image of the viewer obtained by the stereo image recognition module 210. For face detection, an object detection algorithm can be applied to the image of the viewer to obtain an image of the face area. In the case of face tracking, a tracking ID can be assigned to the viewer, and interest level can be monitored based on the tracking ID.

The facial pose and reaction estimation module 224 can estimate the facial pose and user's reaction when a visitor encounters an exhibition object. Specifically, based on the facial pose, it is possible to determine whether the visitor's face is facing the exhibition object or the gaze direction is towards the exhibition object, and based on this, the user's reaction can be estimated. According to one embodiment, the present invention can analyze the user's emotions, the corresponding interest level, the user's reaction, and the correlation between the interest level, etc. In this case, the user's temporary emotions become associated with factors with objective characteristics such as reaction and interest, and the factors and analysis results with these objective characteristics can be utilized in providing customized services to visitors.

The gender and age recognition module 226 can recognize the gender and age group of the visitor. The gender and age group of the visitor can be obtained by analyzing the video of the user's face and performing learning using artificial intelligence. According to the present invention, when extracting information about gender and age, learning by optimized artificial intelligence can be performed taking into account the characteristics of gender and age. According to one embodiment, gender and age information is extracted using a convolutional neural network, but gender information is extracted by the lower layers constituting the convolutional neural network, and age information is extracted by all layers of the plurality of layers. You can. In other words, information that can be extracted from rough features, such as gender, is learned by lower layers, and information that can be extracted from detailed features, such as age, is learned by not only low but also high layers. .

Statistical data 230 is generated. Objective statistical data is generated by analyzing emotions, interest, and reactions to the exhibits.

According to this, the level of interest in the exhibition in each class (by gender/age) can be extracted through the gender/age-specific algorithm of the exhibition and the visitor's gaze analysis algorithm.

Figure 3 is a diagram for explaining how the exhibition hall customized service provision device according to the present invention detects the gaze of a visitor.

Images of visitors present in front are acquired from a camera installed in an exhibition or by the exhibition hall customized service provision device 100. An object detection algorithm is applied to the acquired image to detect only the visitor's face area from the entire image. According to one example, face detection may apply a deep learning-based detection algorithm.

According to one embodiment, an eye-tracking algorithm may be applied to the detected facial area of the visitor to determine whether the visitor is gazing at the exhibition object. The eye tracking algorithm first extracts each feature point of the face and then estimates the direction of the face. If it is determined that the user is looking at an exhibition object, the emotion can be inferred based on the extracted feature points.

In other words, if it is confirmed through the eye tracking algorithm that the visitor is looking at the exhibition, emotion recognition is possible based on the facial feature point information extracted in advance, and based on this, the level of interest in the exhibition is determined. A judgment can be made as to whether it is negative or not.

Additionally, depending on the embodiment, the face direction and gaze direction may be considered together and the degree of interest may be determined by reflecting this. As shown in FIG. 3 , the direction of the visitor's face (D _f ) is toward the exhibit 300 , but the gaze direction (D _e ) deviates from the exhibit 300 . In this case, it can be determined that the visitor's viewing level for the exhibition object 300 is not high.

According to another embodiment, the degree of interest may be set differently in response to the deviation angle at which the gaze deviates from the exhibition object 300. For example, it may be determined that as the deviation angle increases, interest in the exhibit 300 decreases.

Figure 4 is a diagram for explaining a case where the exhibition hall customized service provision device according to the present invention uses a learned neural network model.

A convolutional neural network may be composed of multiple layers capable of classifying feature vectors. One layer consists of square-shaped filters, and each filter can extract feature values from input information through a convolution operation.

The plurality of hierarchical structures continues to extract features by passing the extracted feature values through the next convolutional layer. After continuously extracting features in this way, gender and age information can be extracted by passing through the fully connected (FC) layer. For example, image -> convolution layer -> features -> convolution layer -> features -> convolution layer -> features -> fully connected layer -> gender and age information inference.

When using the neural network model learned in the present invention, the face image is input to the neural network trained using the label dataset corresponding to the image and answer sheet. In this case, detailed features are extracted while passing through layers composed of kernels that can extract feature vectors of the image, and based on this, inference about the input image is possible in the FC (Fully connected) layer.

After emotion inference is completed, the detected consumer's face information can be used to infer gender and age information by inputting the detected face area into a face recognition convolutional neural network. The inferred information can be updated to the tracking ID value assigned by the user.

As shown in FIG. 4, in the case of a plurality of layers 400 constituting the learned neural network model, the lower the layer, the more coarse feature vectors are extracted, and the higher the layer, the more detailed feature vectors are extracted.

Meanwhile, in the case of gender and age information, gender information can be extracted from rough characteristics, and age or age requires detailed characteristics to extract information. In particular, age requires more detailed characteristics than age. For example, in the case of gender, it is possible to distinguish with only rough characteristics such as hair length, facial skeleton, height, and physique, but in the case of age, detailed characteristics such as wrinkles and skin condition are needed.

Considering this, in the present invention, when extracting gender and age information using a convolutional neural network, the learning process can be designed as follows.

Specifically, gender and age information can be extracted using a convolutional neural network, but gender information can be extracted by lower layers constituting the convolutional neural network, and age information can be extracted by all layers of the plurality of layers.

Figure 5 is a diagram illustrating an algorithm for preventing overlapping interest levels for each visitor in the exhibition hall customized service provision device according to the present invention.

First, a tracking ID is set in the detected face area of the visitor.

In order to obtain accurate statistical data, only one interest information per person is needed. Therefore, to prevent double counting, tracking IDs are assigned to detected faces, and a Kalman filter-based image tracking algorithm is applied.

Image tracking obtains the size of the intersection between the predicted regions of interest estimated through a Kalman filter from information on the previously detected region of interest and the currently detected region of interest, and then applies the Hungarian algorithm to most appropriately match the previously detected region of interest. The current detection area of interest can be matched one-to-one.

As shown in FIG. 5, the detection region of interest 512 at time point i is detected, and a Kalman filter is applied to it to obtain the predicted region of interest 520 at time point i.

Additionally, the detection region of interest 514 at time point i+1 is detected.

In this case, the predicted region of interest 520 at time point i is compared with the detected region of interest 514 at time point i+1, and the size of the intersection area 530 is obtained.

The Hungarian algorithm 540, etc. is applied to the intersection area 530 to align and correspond one-to-one to the areas of interest.

Figure 6 shows an example of applying an algorithm for preventing overlapping interests in the present invention.

The first detection ROI 610 is a detection region of interest at time point i. If the Kalman filter algorithm is applied to this, the predicted ROI (620) at time point i is obtained.

The second detection ROI 620 is a detection region of interest at time point i+1.

In this case, the size of the intersection area 630 is compared by comparing the predicted ROI 620 at time point i and the second detection ROI 620 at time point i+1, and corresponding to the size of the intersection area 630. Search for one-to-one matching pairs.

Figure 7 is a diagram for explaining how the exhibition hall customized service provision device according to the present invention recognizes the emotions of visitors.

Once the state in which the visitor is looking at the exhibit is confirmed through the eye tracking algorithm, emotion recognition is performed based on the facial feature point information extracted in advance. Once an emotion is recognized, a positive/negative judgment can be made based on this to determine the level of interest in the exhibit.

To this end, as shown in Figure 7, emotions are divided into major and minor categories, and emotions belonging to each category are associated with positive and negative judgments. Positive and negative judgments can be matched in relation to interest.

Figure 8 is a diagram showing the control process of the exhibition hall customized service provision device according to the present invention.

A first image of a user adjacent to the exhibition object is acquired (S801).

An object detection algorithm is applied to the first image to detect the second image of the face area (S802).

In this case, the second image can be obtained by applying an object detection algorithm to the first image.

An eye-tracking algorithm is applied to the second image to determine whether the user is looking at the exhibit (S803).

According to one embodiment, the direction of the face can be estimated from the second image of the face area to determine whether the direction of the face is toward the exhibition object, thereby determining whether the user is looking at the exhibition object.

According to another embodiment, the direction of the face is estimated from the image of the face area to determine whether the direction of the face is toward the exhibit, the viewing angle of the user's gaze is calculated, and it is determined whether the exhibit is included in the viewing angle, Based on this, it can be determined whether the user is looking at the exhibition object.

If it is determined that the user is looking at the exhibition object, facial feature points are detected from the second image, and the user's emotion toward the exhibition object is estimated based on the feature points (S804).

The correlation between the user's emotions and the degree of interest in the exhibition is analyzed (S805).

Based on the correlation, the level of interest is generated as numerical data and quantified (S806).

9 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 9 may be the exhibition hall customized service providing device 100 described in this specification.

In the embodiment of FIG. 9, the computing device TN100 may include at least one processor TN110, a transceiver device TN120, and a memory TN130. Additionally, the computing device TN100 may further include a storage device TN140, an input interface device TN150, an output interface device TN160, etc. Components included in the computing device TN100 are connected by a bus TN170 and may communicate with each other.

The processor TN110 may execute a program command stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Processor TN110 may be configured to implement procedures, functions, and methods described in connection with embodiments of the present invention. The processor TN110 may control each component of the computing device TN100.

Each of the memory TN130 and the storage device TN140 can store various information related to the operation of the processor TN110. Each of the memory TN130 and the storage device TN140 may be comprised of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may be comprised of at least one of read only memory (ROM) and random access memory (RAM).

The transceiving device TN120 can transmit or receive wired signals or wireless signals. The transmitting and receiving device (TN120) can be connected to a network and perform communication.

Meanwhile, the embodiments of the present invention are not only implemented through the apparatus and/or method described so far, but may also be implemented through a program that realizes the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded. This implementation can be easily implemented by anyone skilled in the art from the description of the above-described embodiments.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims are also possible. It falls within the scope of invention rights.

100: Exhibition hall customized service provision device
110: Image processing unit 120: Interest estimation unit
130: Control unit 200: Analysis module
210: Stereo image recognition module 222: Face detection and tracking module
224: Face pose and reaction estimation module 226: Gender and age group recognition module

Claims (8)

In the exhibition hall customized service provision device,
an image processing unit that acquires a first image of a user adjacent to an exhibition object and detects a second image of a face area by applying an object detection algorithm to the first image;
When it is determined that the user is looking at the exhibit by applying an eye tracking algorithm to the second image, facial feature points are detected from the second image, and the user's emotion toward the exhibit is based on the feature point. an interest estimation unit that estimates; and
a control unit that analyzes the correlation between the user's emotions and the level of interest in the exhibit, and generates and quantifies the level of interest as numerical data based on the correlation; Including,
Determining the level of interest by considering both the direction of the face and the gaze direction,
Setting the interest level differently in response to the deviation angle at which the user's gaze deviates from the exhibition object,
A device that provides customized exhibition hall services. According to paragraph 1,
The interest estimation unit,
Estimating the direction of the face from the second image to determine whether the direction of the face is toward the exhibition object,
After calculating the viewing angle of the user's gaze and determining whether the exhibit is included within the viewing angle,
Based on this, it is determined whether the user is looking at the exhibit,
A device that provides customized exhibition hall services. According to paragraph 1,
The control unit,
The second image is input to a convolutional neural network related to face recognition, and gender and age information is extracted by extracting a plurality of feature vectors from a plurality of layers constituting the convolutional neural network,
Dividing the plurality of layers into a first layer group consisting of layers lower than the predetermined layer and a second layer group consisting of the predetermined layer and layers higher than the predetermined layer based on the predetermined layer,
Recognizing gender based on the feature vector extracted by the first layer group, and recognizing age based on the feature vector extracted by the first layer group and the second layer group,
A device that provides customized exhibition hall services. According to paragraph 1,
The control unit,
Setting a tracking ID by applying an image tracking algorithm to the second image,
A detection area of interest is generated based on the tracking ID, a predicted area of interest is created estimated from the detection area of interest through a Kalman filter, and the detection area is based on the size of an intersection area between the detection area of interest and the predicted area of interest. Matching the area of interest and the predicted area of interest one-to-one,
A device that provides customized exhibition hall services. In the control method of the exhibition hall customized service provision device,
Obtaining a first image of a user adjacent to an exhibition object, and detecting a second image of a face area by applying an object detection algorithm to the first image;
When it is determined that the user is looking at the exhibit by applying an eye tracking algorithm to the second image, facial feature points are detected from the second image, and the user's emotion toward the exhibit is based on the feature point. estimating;
Analyzing the correlation between the user's emotions and interest in the exhibition object; and
generating and quantifying the interest level as numerical data based on the correlation; Including,
Determining the level of interest by considering both the direction of the face and the gaze direction,
Setting the interest level differently in response to the deviation angle at which the user's gaze deviates from the exhibition object,
Control method for exhibition hall customized service provision device. According to clause 5,
Estimating the direction of the face from the second image to determine whether the direction of the face is toward the exhibition object,
After calculating the viewing angle of the user's gaze and determining whether the exhibit is included within the viewing angle,
Based on this, it is determined whether the user is looking at the exhibit,
Control method for exhibition hall customized service provision device. According to clause 5,
The second image is input to a convolutional neural network related to face recognition, and gender and age information is extracted by extracting a plurality of feature vectors from a plurality of layers constituting the convolutional neural network,
Dividing the plurality of layers into a first layer group consisting of layers lower than the predetermined layer and a second layer group consisting of the predetermined layer and layers higher than the predetermined layer based on the predetermined layer,
Recognizing gender based on the feature vector extracted by the first layer group, and recognizing age based on the feature vector extracted by the first layer group and the second layer group,
Control method for exhibition hall customized service provision device. According to clause 5,
Setting a tracking ID by applying an image tracking algorithm to the second image,
A detection area of interest is generated based on the tracking ID, a predicted area of interest is created estimated from the detection area of interest through a Kalman filter, and the detection area is based on the size of an intersection area between the detection area of interest and the predicted area of interest. Matching the area of interest and the predicted area of interest one-to-one,
Control method for exhibition hall customized service provision device.

Images