JP2018169788A - Language ratio management system and language ratio management method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To manage the ratio of languages used in a place where multiple languages are used. SOLUTION: A language ratio management system 1 detects and tracks a person from a camera 3 that captures a target area for which the ratio of the language used is to be managed and an image captured by the camera, and uses the person based on the language behavior of the person. The behavior recognition device 4 for determining the language and the language ratio management server 5 for managing the ratio of the language used in the target area by acquiring and storing information indicating the language used from the behavior recognition device. The configuration is provided. [Selection diagram] Fig. 1

Description

  The present invention relates to a language ratio management system and a language ratio management method for managing a ratio of languages used in a place where multiple languages are used.

  Conventionally, for the purpose of providing service information in various languages to visitors (persons) to an amusement facility such as a theme park, identification information for identifying each person and language information indicating the language used by the person Is previously associated and stored, and a technique for providing service information to each person in the language used by the person has been proposed (see Patent Document 1). In the prior art disclosed in Patent Document 1, person identification information is stored in a wireless tag carried by the person.

JP 2004-355087 A

  For example, in places where multiple languages are used, such as airports and sightseeing spots, it is required to broadcast various information such as communication and guidance in multiple languages (multiple languages). When broadcasting in multiple languages, it is desirable to broadcast the languages used in the broadcasting area where the languages are broadcast in an order corresponding to the ratio of the languages used in the broadcasting area.

  However, although the technique disclosed in Patent Document 1 can identify the language used by each person, the language used in the broadcast area and the ratio of the language used in the broadcast area cannot be managed. Note that the management of the ratio of the language used includes recognition of the language used in the target area and grasping the ratio of the language used in the target area. Moreover, in the technique of the above-mentioned Patent Document 1, since it is necessary to store the identification information and language information of each person in association with each other in advance, in places where an unspecified number of persons such as airports and tourist spots enter and exit, The language used in the broadcasting area could not be determined.

  The present disclosure has been devised in view of the problems of the prior art, and a language ratio management system and a language ratio management that can manage the ratio of languages used in a place where multiple languages are used. The main purpose is to provide a method. It is another object of the present disclosure to provide a language ratio management system and a language ratio management method that can determine the language used by a person at a place where an unspecified number of persons come and go.

  The language ratio management system according to the present disclosure is a language ratio management system for managing a ratio of a used language in a place where multiple languages are used, and includes a camera that captures a predetermined target area, and an image captured by the camera. Detecting and tracking a person, and determining a language used by the person based on the language behavior of the person, and acquiring and storing information indicating the language used from the action recognition device, A language ratio management server that manages the ratio of languages used in the target area is provided.

  According to the present disclosure, it is possible to manage the ratio of languages used in a place where multiple languages are used.

Schematic diagram showing a schematic configuration of a language ratio management system according to the first embodiment of the present disclosure. The figure which shows the modification of the language ratio management system which concerns on 1st Embodiment. Functional block diagram of multilingual bulletin board (A) is a figure which shows an example of the several area | region of a display part, (b) is a figure which shows an example of display language information. Functional block diagram of action recognition device The figure which shows an example of a person information management table The figure which shows an example of a language determination table Functional block diagram of language ratio management server The figure which shows an example of a language ratio management table Diagram showing an example of language ratio information Functional block diagram of the in-house broadcasting device The figure which shows an example of an area | region information table Figure showing an example of a floor map The figure which shows the setting screen of the language ratio acquisition area Figure showing the multilingual playback order setting screen The figure which shows an example of a language ratio acquisition area setting table The figure which shows an example of a multilingual reproduction order setting table The flowchart which shows a series of processes in the action recognition apparatus which concerns on 1st Embodiment. Flow chart for explaining person position information update processing in the action recognition device Flow chart for explaining the language usage determination process in the action recognition device Schematic diagram showing a schematic configuration of a language ratio management system according to the second embodiment of the present disclosure. Functional block diagram of multilingual guidance robot The figure which shows an example of the language ratio acquisition area | region of a multilingual guidance robot Flow diagram for explaining multilingual guidance processing in a multilingual guidance robot Schematic diagram showing a schematic configuration of a language ratio management system according to the third embodiment of the present disclosure. Functional block diagram of multilingual kiosk Functional block diagram of the action recognition device according to the third embodiment The flowchart for demonstrating the utilization language determination process in the action recognition apparatus which concerns on 3rd Embodiment. Schematic diagram showing a schematic configuration of a language ratio management system according to the fourth embodiment of the present disclosure. Functional block diagram of person verification device The figure which shows an example of a person language table Functional block diagram of the action recognition device according to the fourth embodiment The flowchart for demonstrating the person position information update process in the action recognition apparatus which concerns on 4th Embodiment. The flowchart for demonstrating the utilization language determination process in the action recognition apparatus which concerns on 4th Embodiment. Flow chart for explaining collation processing in person collation device Schematic diagram showing a schematic configuration of a language ratio management system according to the fifth embodiment of the present disclosure. Functional block diagram of the action recognition device according to the fifth embodiment The figure which shows an example of the person information management table which concerns on 5th Embodiment The flowchart which shows a series of processes in the action recognition apparatus which concerns on 5th Embodiment. The flowchart for demonstrating the group determination process in the action recognition apparatus which concerns on 5th Embodiment. Schematic diagram showing a schematic configuration of a language ratio management system according to a sixth embodiment of the present disclosure Functional block diagram of the action recognition apparatus according to the sixth embodiment The flowchart for demonstrating the use language determination process in the action recognition apparatus which concerns on 6th Embodiment

  A first invention made to solve the above problems is a language ratio management system for managing a ratio of a used language in a place where multiple languages are used, and a target area for which a ratio of the used language is to be managed is determined. A camera that captures an image; a person that is detected and tracked from an image captured by the camera; a behavior recognition device that determines a language used by the person based on the language behavior of the person; and the language used from the behavior recognition device A language ratio management server that manages the ratio of the language used in the target area by acquiring and storing information is provided.

  According to the language ratio management system according to the first aspect of the present invention, the ratio of the used languages used in the target area can be stored and managed by the language ratio management server. Thereby, it is possible to manage the ratio of the language used in a place where multiple languages are used. Note that the management of the ratio of the language used includes recognition of the language used in the target area and grasping the ratio of the language used in the target area.

  The second invention further includes a multilingual bulletin board having a plurality of display areas arranged in the target area and displaying different languages, respectively, in the first invention, wherein the action recognition device comprises: The line-of-sight direction of the person is detected from the captured image, and the display area in which the person is gazing out of the plurality of display areas is identified based on the line-of-sight direction, and is displayed in the identified display area The language is determined as the language used by the person.

  According to the language ratio management system according to the second aspect of the present invention, it is possible to determine the language used by the person based on the language behavior of the person viewing the display area of the multilingual bulletin board. This makes it possible to determine the language used by a person at a place where an unspecified number of persons come and go.

  The third invention further includes a multilingual kiosk arranged in the target area and operable in multiple languages in the first invention or the second invention, wherein the action recognition device The language used when operating the multilingual kiosk is determined as the language used by the person.

  According to the language ratio management system according to the third aspect of the present invention, the language used by the person can be determined based on the person's language behavior of operating the multilingual kiosk. This makes it possible to determine the language used by a person at a place where an unspecified number of persons come and go.

  Moreover, 4th invention is further provided with the microphone arrange | positioned in the said object area | region in any one of the said 1st invention thru | or 3rd invention, The said action recognition apparatus is the said sound-collected by the said microphone The language used by the person is determined based on the voice of the person.

  According to the language ratio management system according to the fourth aspect of the present invention, the language used by the person can be determined based on the language behavior of the person called speech. This makes it possible to determine the language used by a person at a place where an unspecified number of persons come and go.

  In addition, a fifth invention further includes an in-house broadcasting device that outputs multilingual audio information to the target area in any one of the first to fourth inventions, and the in-house broadcasting device includes: The language ratio information indicating the ratio of the used languages used in the target area is obtained by inquiring the language ratio management server, and based on the language ratio information, the type of multi-language to be output as a voice and the multi-language as a voice The output order is determined.

  According to the language ratio management system according to the fifth aspect of the present invention, the type of multi-language that the in-house broadcasting apparatus outputs sound and the order in which the multi-language is sound-outputted indicate the ratio of the language used in the target area. It becomes possible to determine based on the language ratio information.

  The sixth invention further includes a multilingual guide robot that outputs multilingual information in the target area in the first to fifth inventions, wherein the multilingual guide robot is the target area. Querying and obtaining the language ratio information indicating the ratio of the used language being used from the language ratio management server, and determining the type of multi-language to be output and the order of outputting the multi-language based on the language ratio information. It is characterized by.

  According to the language ratio management system according to the sixth aspect of the present invention, the language indicating the ratio of the used languages used in the target area, the type of the multilingual output by the multilingual guidance robot, and the order in which the multilingual is output. It becomes possible to determine based on the ratio information.

  In addition, a seventh invention further includes a person verification device for verifying a person detected from the captured image among the plurality of behavior recognition devices in the first to sixth inventions, The person verification device acquires and stores in advance information indicating a person image of the person detected from the captured image and a language used by the person from at least one of the action recognition apparatuses, and stores the at least one of the at least one of the action recognition devices. When a person image of a person to be collated is received from a behavior recognition apparatus different from the action recognition apparatus, the person image of the person to be collated is collated with the person image stored in advance, and if they match, the person image is stored in advance. The information indicating the language used by the person in the person image is transmitted to the another action recognition device.

  According to the language ratio management system according to the seventh aspect, since the person detected from the captured image can be collated between the plurality of action recognition devices, the language used by the person between the plurality of action recognition devices. Can be shared.

  According to an eighth aspect of the present invention, in the first to seventh aspects of the present invention, a group of persons is determined based on a tracking result of the person detected from the captured image, and a language used is found in the group. The language used by a person is the language used by a person whose language in the group is unknown.

  According to the language ratio management system according to the eighth aspect of the present invention, when at least one used language is found in a group, the used language can be changed to an unknown used language in the group. Become.

  Further, a ninth invention is a language ratio management method for managing a ratio of used languages in a place where multiple languages are used, and imaging a target area for which a ratio of used languages is to be managed by a camera; Detecting and tracking a person from a captured image of the camera, determining a language used by the person based on the language behavior of the person, and storing information indicating the language used in the target area And a step of managing a ratio of used languages.

  According to the language ratio management method according to the ninth aspect of the present invention, the ratio of the used languages used in the target area can be stored and managed. Thereby, it is possible to manage the ratio of the language used in a place where multiple languages are used.

  Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a schematic diagram illustrating a schematic configuration of a language ratio management system 1 according to the first embodiment of the present disclosure. As shown in FIG. 1, the language ratio management system 1 mainly includes a multilingual bulletin board 2, a camera 3, an action recognition device 4, a language ratio management server 5, and an in-house broadcasting device 6. The action recognition device 4 is connected to the multilingual bulletin board 2, the camera 3, and the language ratio management server 5 via a dedicated communication line or a communication network such as the Internet. The language ratio management server 5 is connected to the in-house broadcasting device 6 via a dedicated communication line or a communication network such as the Internet.

  This embodiment demonstrates the case where the language ratio management system 1 is applied to facilities, such as an airport and a station. Note that the language ratio management system 1 according to the present disclosure can be applied to various objects such as a shopping center and a sightseeing spot in addition to an airport and a station.

  FIG. 2 is a diagram illustrating a modification of the language ratio management system 1. As shown in FIG. 2, the language ratio management system 1 may include a plurality of multilingual bulletin boards 2, cameras 3, and action recognition devices 4. In the example of FIG. 2, two multilingual bulletin boards 2, two cameras 3, and two action recognition devices 4 are provided. The multilingual bulletin board 2A and the camera 3A are connected to the action recognition device 4A, and the action recognition device 4B is connected to the action recognition device 4B. A multilingual bulletin board 2B and a camera 3B are connected. The behavior recognition device 4 </ b> A and the behavior recognition device 4 </ b> B are connected to the language ratio management server 5.

  The multilingual bulletin board 2 is an electric bulletin board having a known configuration, and is arranged at an appropriate place on the floor F of a facility such as an airport or a station. The multilingual bulletin board 2 displays various information such as contact and guidance in a multilingual display for the person H existing on the floor F based on a command from an information management center (not shown).

  FIG. 3 is a functional block diagram of the multilingual bulletin board 2. As shown in FIG. 3, the multilingual bulletin board 2 includes a display unit 11 (see FIG. 4A) having a plurality of areas for displaying various types of information such as contact and guidance in multiple languages, and each of the display units 11. The display management unit 12 manages the display of the area, and the display language providing unit 13 notifies the action recognition device 4 of the display language information indicating the language displayed in each area of the display unit 11. The display unit 11 includes, for example, an LED panel or a liquid crystal panel display unit.

  FIG. 4A is a diagram illustrating an example of a plurality of areas of the display unit 11, and FIG. 4B is a diagram illustrating an example of display language information. In the example of FIG. 4A, the display unit 11 has three areas: an area 1 for displaying guidance in Japanese, an area 2 for displaying guidance in English, and an area 3 for displaying guidance in Chinese. Yes. Accordingly, the display language information in this case is “region 1 is Japanese, region 2 is English, and region 3 is Chinese”.

  The camera 3 is an imaging device having a known imaging function and communication function, and is installed on a wall or ceiling in the vicinity of the multilingual bulletin board 2 and has a predetermined imaging area (target area) around the multilingual bulletin board 2. Are captured in time series as still images or moving images. A captured image (still image or moving image) of the camera 3 is appropriately transmitted to the action recognition device 4. In addition, as long as the camera 3 can image a predetermined imaging region, the form, function, arrangement, quantity, and the like are not particularly limited, and various changes can be made.

  The behavior recognition device 4 is composed of a computer having a known configuration, detects and tracks a person from an image captured by the camera 3, and determines the language used by the person based on the language behavior of the person. A language action is an action of a person reading, writing (including input via a touch panel, etc.), talking, or listening using a language.

  FIG. 5 is a functional block diagram of the action recognition device 4. As illustrated in FIG. 5, the action recognition device 4 includes an image acquisition unit 21 that acquires a captured image from the camera 3, a person detection unit 22 that detects a person from the captured image, and a human region (hereinafter, “ A position coordinate conversion unit 23 for converting the image area (also referred to as “image person area”) to the position coordinates of the floor map which is map data of the floor F, and a person detected from the captured image (current frame image) is detected in the previous frame image The same person determination unit 24 that determines whether or not the person is the same, a language ratio management server connection unit 25 connected to the language ratio management server 5, a captured image acquired by the image acquisition unit 21, and personal information management described later And a storage unit 26 for storing the table. The captured image is stored in the storage unit 26 as a previous frame image (previous image) or a current frame image (current image).

  The action recognition device 4 includes a multilingual bulletin board connection unit 27 connected to the multilingual bulletin board 2, a gaze direction detection unit 28 that detects the gaze direction of the person detected from the captured image, and a gaze direction detection unit. A language-of-use determination unit 29 that determines the language used by the person based on the line-of-sight direction of the person detected in 28 and the display language information acquired from the multilingual bulletin board 2, and a language determination table 30 connected to the language-of-use determination unit 29. And have. The line-of-sight direction detection unit 28 detects the direction of the person's face from the captured image, and sets the detected face direction as the line-of-sight direction of the person. The use language determination unit 29 records the determination result of each frame image in the language determination table 30, and determines the use language of the person based on the recorded determination result.

  FIG. 6 is a diagram illustrating an example of the person information management table. As shown in FIG. 6, in the person information management table, an image person area of a person detected by the person detection unit 22, a position coordinate converted by the position coordinate conversion unit 23, and a use language determination are associated with the person ID. The use language determined by the unit 29 is recorded.

  FIG. 7 is a diagram illustrating an example of the language determination table 30. As shown in FIG. 7, in the language determination table 30, a region of interest, which is a region where a person gazes at the multilingual bulletin board 2 in association with a person ID, and the region of interest in successive frames are continuously observed. The number of continuations, which is the number of times, is recorded.

  The language ratio management server 5 is a server having a known configuration, acquires personal information including the position coordinates of the person and the language used from the action recognition device 4, and stores it in the language ratio management table (see FIG. 9). The ratio of the language used in the imaging area (target area) of the camera 3 is managed. Note that the management of the ratio of the language used includes recognition of the language used in the target area and grasping the ratio of the language used in the target area.

  FIG. 8 is a functional block diagram of the language ratio management server 5. As shown in FIG. 8, the language ratio management server 5 includes a person information receiving unit 31 that receives person information from the action recognition device 4 (4A, 4B), a storage unit 32 that stores a language ratio management table, and in-house broadcasting. When an inquiry about the ratio of the language used in the target area is received from the apparatus 6, the language ratio management unit 33 that creates language ratio information corresponding to the inquiry and the inquiry from the in-house broadcasting apparatus 6 It has a language rate sending unit 34 for receiving and sending the language rate information created by the language rate management unit 33 to the in-house broadcasting device 6. As described above, the language ratio management system 1 may include a plurality of action recognition devices 4. The language ratio management server 5 shown in FIG. 8 acquires and manages person information from the two action recognition devices 4A and 4B.

  FIG. 9 is a diagram illustrating an example of the language ratio management table. As shown in FIG. 9, the language ratio management table is a position obtained by converting a person area (image person area) in a captured image into position coordinates of a floor map in association with the apparatus ID assigned to the action recognition apparatus 4. The coordinates and the language used by the person are stored. In the example of FIG. 9, a device ID “1” is assigned to the behavior recognition device 4A, and a device ID “2” is assigned to the behavior recognition device 4B.

  FIG. 10 is a diagram illustrating an example of language ratio information created by the language ratio management server 5. The language ratio information is created by totaling the use languages corresponding to the position coordinates included in the target area inquired from the in-house broadcasting device 6. In the example of FIG. 10, the ratio of the language used in the target area is 20 for English, 11 for Japanese, 23 for Chinese, 3 for Korean, and 15 for unknown. In the example of FIG. 10, the ratio of the language used is expressed by the number of people, but may be expressed by a percentage. Moreover, you may make it cut off the language below a predetermined threshold value.

  The in-house broadcasting device 6 is a broadcasting device having a known configuration, and outputs audio information in various languages (various information such as contact information and guidance) to the target area.

  FIG. 11 is a functional block diagram of the in-house broadcasting device 6. As shown in FIG. 11, the in-house broadcasting device 6 includes an audio input unit 41 for the user of the in-house broadcasting device 6 to input audio, an audio recognition unit 42 for recognizing the audio input by the user, and the language ratio management server 5. A language ratio acquisition unit 43 that inquires and acquires language ratio information of the target area, a translation unit 44 that translates speech input by the user into multiple languages determined based on the language ratio information, and translated multilingual synthesized speech And a voice output unit 46 for outputting (reproducing) the generated synthesized voice as a voice.

  The in-house broadcasting device 6 inquires the language ratio management server 5 to obtain a language ratio information area (that is, a target area), and a display input panel 47 for setting a multilingual reproduction order. And a storage unit 48 storing a language ratio acquisition area setting table and a multilingual reproduction order setting table for storing the language ratio acquisition area and the multilingual reproduction order set on the display input panel 47, respectively. In addition, the storage unit 48 stores an area information table in which candidates for language ratio acquisition areas are stored in advance.

  FIG. 12 is a diagram illustrating an example of the area information table stored in the storage unit 48 of the in-house broadcasting device 6. FIG. 13 is a diagram illustrating an example of a floor map that is map data of the floor F. As illustrated in FIG. In the example of FIG. 13, the floor F is provided with two multilingual bulletin boards 2A and 2B (hereinafter also simply referred to as “bulletin board A” and “bulletin board B”). In this case, the area information table includes the area names indicating the three areas of “the entire floor”, “around the bulletin board A”, and “around the bulletin board B”, and the area information specifying each area, as the language ratio acquisition area. It is stored in advance as a candidate for (target region).

  FIG. 14 is a diagram showing a setting screen on which the user of the in-house broadcasting device 6 sets a language ratio acquisition area. In the example of FIG. 14, the entire floor is selected and set as the language ratio acquisition area. Therefore, in this case, the in-house broadcasting device 6 obtains the language ratio information about the entire floor by inquiring the language ratio management server 5.

  FIG. 15 is a diagram showing a setting screen for setting the multilingual reproduction order by the user of the in-house broadcasting device 6. In the example of FIG. 15, the first in the playback order is Japanese, the second is English, the third is the first language in the ratio information, the fourth is the second language in the ratio information, the fifth and sixth The second is set as unselected (none). Therefore, in the example of FIG. 15, the audio information is reproduced in four languages from the first to the fourth in the reproduction order. In the language ratio information, the first-ranked language and the second-ranked language are the first-ranked language and the second-ranked language in the languages other than Japanese and English set in advance as the first and second languages, respectively. Good.

  FIG. 16 is a diagram illustrating an example of a language ratio acquisition area setting table stored in the storage unit 48 of the in-house broadcasting device 6. In the language ratio acquisition area setting table, the language ratio acquisition area set in the language ratio acquisition area setting screen shown in FIG. 14 is recorded. In the example of FIG. 16, the language ratio acquisition area is stored as “entire floor”.

  FIG. 17 is a diagram illustrating an example of a multilingual reproduction order setting table stored in the storage unit 48 of the in-house broadcasting device 6. In the multilingual playback order table, the playback order set on the multilingual playback order setting screen shown in FIG. 15 is recorded. In the example of FIG. 17, the reproduction order is “first is Japanese, second is English, third is the first language in the ratio information, and fourth is the second language in the ratio information”. Is remembered.

  FIG. 18 is a flowchart showing a series of processes in the action recognition device 4 according to the first embodiment.

  First, in step ST101, a captured image of a predetermined imaging region (target region) around the multilingual bulletin board 2 is acquired from the camera 3 and stored in the storage unit 26 as a current frame image (current image).

  Next, in step ST102, person position information update processing is performed to update the image person area and position coordinates of the person information management table (see FIG. 6) based on the captured image. Details of the person position information update processing will be described later with reference to the flowchart of FIG.

  Subsequently, in step ST103, a use language determination process for determining the use language of the person detected from the captured image is performed. Details of the use language determination processing will be described later with reference to the flowchart of FIG.

  Next, in step ST104, the personal information (position coordinates and language used) stored in the language ratio management table (see FIG. 9) of the language ratio management server 5 is updated. Specifically, the person information (position coordinates and language used) stored in the person information management table (see FIG. 6) stored in the storage unit 26 is transmitted to the language ratio management server 5, and the language ratio management server The person information stored in the language ratio management table 5 is updated or added and recorded in the language ratio management table.

  In step ST105, the captured image is stored in the storage unit 26 as a previous frame image, and then the process returns to step ST101.

  FIG. 19 is a flowchart for explaining person position information update processing (step ST102 in FIG. 18) in the action recognition device 4.

  First, in step ST201, a person area (image person area) is detected from the current frame image. The detection of the person region from the current frame image may be performed using a known method such as a method based on the feature amount of the person.

  Next, in step ST202, the person area (image person area) is converted into floor map position coordinates, which are map data of the floor F. Specifically, the coordinates of the center or the center of gravity of the person area may be set as the position coordinates.

  Subsequently, in step ST203, it is determined whether or not a person detected in the current frame image exists in the previous frame image. This determination may be performed using a known method such as a method based on the feature amount of a person, as in step ST201 described above. If it is determined that the same person exists in the previous frame image (step ST203: Yes), the process proceeds to step ST204. If it is determined that the same person does not exist in the previous frame image (step ST203: No), the process proceeds to step ST205. .

  In step ST204, the position coordinates of the corresponding person ID and the image person area in the person information management table (see FIG. 6) stored in the storage unit 26 are updated, and then the process proceeds to step ST206.

  In step ST205, a person ID is added to the person information management table, the image person area and the position coordinates are recorded in association with the added person ID, and then the process proceeds to step ST206.

  In step ST206, the person ID that has not been updated in the person information management table is deleted, and the process ends. Thereby, the person information of the person who does not exist in the current frame image is deleted. The above steps ST202-205 are repeated for each person area detected from the current frame image.

  FIG. 20 is a flowchart for explaining the use language determination processing (step ST103 in FIG. 18) in the action recognition device 4. The following process is repeated for each person ID updated or added in the person position information update process.

  First, in step ST301, the gaze direction of a person is detected from the current frame image. Specifically, the face area of the person is detected from the image person area of the current frame image using a known technique such as a technique using a statistical or non-statistic classifier or a technique using template matching. A face feature amount is extracted from the region using a known method. Then, based on the extracted facial feature quantity, a face direction is detected using a statistical or non-statistical classifier prepared in advance, and the detected face direction of the person is set as the line-of-sight direction of the person.

  Next, in step ST302, the attention area in the display part of a multilingual bulletin board is detected based on the detected gaze direction of the person. A region of interest is a region where a person is gazing.

  In a succeeding step ST303, it is determined whether or not a region of interest is detected. When it is determined that the region of interest has not been detected (step ST303: No), the process proceeds to step ST304, and when it is determined that the region of interest has been detected (step ST303: Yes), the process proceeds to step ST305.

  In step ST304, since the region of interest has not been detected, the region of interest in the language determination table (see FIG. 7) is updated to “none”, the number of continuations is updated to “0”, and then the process ends.

  In step ST305, it is determined whether the region of interest is the same as the previous frame image. When it is determined that they are not the same (step ST305: No), the process proceeds to step ST306, and when it is determined that they are the same (step ST305: Yes), the process proceeds to step ST307.

  In step ST306, a new region of interest is added, linked to the region of interest, the continuation count “1” is recorded, and then the process ends.

  In step ST307, the number of continuations of the target region of interest is increased by “1”.

  In a succeeding step ST308, it is determined whether or not the number of continuations has exceeded a threshold value. If it is determined that the threshold value is not greater than or equal to the threshold value (step ST308: No), the process ends. If it is determined that the threshold value is equal to or greater than the threshold value (step ST308: Yes), the process proceeds to step ST309.

  In step ST309, the language corresponding to the region of interest is recorded in the column of language used in the person information management table, and then the process ends. The language corresponding to the area is determined based on the display language information (see FIG. 4B). As a result, the language used by the person is determined.

  Next, the operation of the language ratio management server 5 will be described.

  When the language ratio management server 5 receives the person information including the position coordinates of the person and the language used from the action recognition device 4, the language ratio management server 5 records the person information in the language ratio management table (see FIG. 9). As described above, the personal information is stored in association with the device ID of the behavior recognition device 4. And when the inquiry about the ratio of the use language used in the object area is received from the in-house broadcasting device 6, language ratio information (see FIG. 10) indicating the ratio of the use language used in the object area is received. It is created and transmitted to the in-house broadcasting device 6.

  Next, the operation of the in-house broadcasting device 6 will be described.

  First, the user of the in-house broadcasting device 6 operates each setting screen (see FIGS. 14 and 15) displayed on the display input panel 47 of the in-house broadcasting device 6 to set the language ratio acquisition area and the multilingual playback order. . The language ratio acquisition area is set by the user selecting a desired language ratio acquisition area from the language ratio acquisition area candidates displayed on the setting screen. The set language ratio acquisition area is stored in a language ratio acquisition area setting table (see FIG. 16). The multilingual playback order is set by the user operating the setting screen and specifying the playback order. The set playback order is stored in a multilingual playback order setting table (see FIG. 17).

  When the in-house broadcasting device 6 outputs multilingual audio information to the target area (broadcast area), the in-house broadcasting apparatus 6 inquires the language ratio management server 5 for the ratio of the language used in the target area, and acquires the language ratio information. Based on the acquired language ratio information, the type of multi-language to be output as speech and the order of outputting the multi-language as speech are determined. Thereby, the language used in the target area can be broadcast according to the ratio of the language used in the target area.

  As described above, according to the language ratio management system according to the first embodiment, the language ratio management server 5 stores and manages the ratio of the used languages used in the target area, so that multiple languages are used. It is possible to manage the ratio of the language used in the location.

  Further, according to the language ratio management system according to the first embodiment, it is possible to determine the language used by the person based on the language behavior of the person viewing the display area of the multilingual bulletin board 2. This makes it possible to determine the language used by a person at a place where an unspecified number of persons come and go.

  In addition, according to the language ratio management system according to the first embodiment, the ratio of the languages used in the target area is set according to the types of multi-languages that the in-house broadcasting device 6 outputs and the order in which the multi-languages are output as voices. It is possible to determine based on the language ratio information indicating.

(Second Embodiment)
Next, the language ratio management system 1 according to the second embodiment of the present disclosure will be described. In the second embodiment, items not particularly mentioned below are the same as those in the first embodiment described above.

  FIG. 21 is a schematic diagram illustrating a schematic configuration of the language ratio management system 1 according to the second embodiment. The second embodiment is different from the first embodiment in that a multilingual guide robot 7 is provided instead of the in-house broadcasting device 6.

  The multilingual guidance robot 7 (hereinafter simply referred to as “robot 7”) autonomously travels on the floor F while grasping the current position, and multilingual guidance voices (voice information) stored in advance are centered on the robot 7. The voice is output to the area (target area) of the predetermined range. As in the case of the above-mentioned in-house broadcasting device 6, the type of multi-language to be output as audio and the order of broadcasting the multi-language are determined based on the ratio of the languages used in the target area.

  FIG. 22 is a functional block diagram of the multilingual guide robot 7. As shown in FIG. 22, the multilingual guidance robot 7 includes a driving unit 51 that drives the robot 7, a current position grasping unit 52 that grasps the current position of the robot 7, a language ratio acquisition region setting table, a multilingual A storage unit 53 that stores a reproduction order setting table and multilingual guide information (voice information and character information), a language rate acquisition unit 54 that acquires language rate information from the language rate management server 5, and multilingual guide information Is provided. The output unit 55 reproduces and outputs sound through a speaker, and displays and outputs character information and the like on a display provided in the robot 7.

  The language ratio acquisition region (that is, the target region) is a circular region having a predetermined radius R around the position coordinate of the robot 7. The radius R is set as appropriate. FIG. 23 is a diagram illustrating an example of the language ratio acquisition area of the robot 7. In the example of FIG. 23, a circle area having a radius R centering on the coordinates (1500, 550) of the current position of the robot 7 is the language ratio acquisition area.

  FIG. 24 is a flowchart for explaining multilingual guidance processing in the robot 7. The following processing is based on a program preset in the storage unit 53 of the robot 7 (for example, a multilingual guidance program) or a command from an information management center (not shown). This is started when outputting various kinds of information such as voice information and character information).

  First, in step ST401, the current position grasping unit 52 grasps the current position of the robot 7, and determines a language ratio acquisition region based on the position coordinates of the current position and a predetermined radius R. As described above, the language ratio acquisition region is a circular region having a radius R around the current position coordinate.

  Next, in step ST402, the language ratio management server 5 is inquired about the ratio of the language used in the language ratio acquisition area (target area), and the language ratio management server 5 determines the language used in the target area. Language ratio information indicating the ratio is acquired.

  Subsequently, in step ST403, the language reproduction order is determined based on the language ratio information acquired from the language ratio management server 5 and the multilingual reproduction order setting. The multilingual reproduction order setting is set in advance by the user in the same manner as in the case of the in-house broadcasting device 6 described above.

  In step ST404, guidance information is output (reproduction of guidance voice or display of character information) in accordance with the determined language reproduction order, and then the process ends.

  As described above, according to the language ratio management system according to the second embodiment, the language indicating the ratio of the used languages used in the target area, the type of the multi-language output by the robot 7 and the order in which the multi-language is output. It becomes possible to determine based on the ratio information. Note that the in-house broadcasting device of the first embodiment and the multilingual guidance robot 7 of the second embodiment may be used in combination.

  Further, in the above example, the type of language output by the robot 7 is determined based on the position coordinates of the current position of the robot 7 and the ratio of the language used in the range of the radius R determined in advance. The robot 7 inquires the language ratio management server 5 about the language used by the person, and the language ratio management server 5 sets the language used to the robot 7 based on the position coordinates of the person from the language ratio management table. By transmitting, the type of language that the robot 7 guides to the person located in front may be determined.

(Third embodiment)
Next, the language ratio management system 1 according to the third embodiment of the present disclosure will be described. In the third embodiment, matters not particularly mentioned below are the same as those in the above-described first embodiment.

  FIG. 25 is a schematic diagram illustrating a schematic configuration of the language ratio management system 1 according to the third embodiment. The third embodiment differs from the first embodiment in that a multilingual kiosk 8 is provided instead of the multilingual bulletin board 2.

  The multilingual kiosk 8 is an information terminal device having a known configuration, and is arranged at an appropriate place on the floor F. The multilingual kiosk 8 has a touch panel as a display input unit, and a person existing on the floor F can access various kinds of information by operating the touch panel.

  FIG. 26 is a functional block diagram of the multilingual kiosk 8. As shown in FIG. 26, the multilingual kiosk 8 displays various information such as contact and guidance in multiple languages, and manages the display input unit (touch panel) 61 that can be operated by the user and the display of the display input unit 61. And a display language providing unit 63 that notifies the action recognition device 4 of the language displayed on the display input unit 61, that is, the display language information indicating the language used for the operation of the display input unit 61. Have.

  FIG. 27 is a functional block diagram of the action recognition device 4 according to the third embodiment. As shown in FIG. 27, the action recognition device 4 according to the third embodiment has a multilingual kiosk connection unit 64 instead of the multilingual bulletin board connection unit 27, and the multilingual kiosk connection unit 64 is used. Connected to multilingual kiosk 8. Further, the action recognition device 4 according to the third embodiment does not include the line-of-sight direction detection unit 28 and the language determination table 30. Since the other points are the same as those of the action recognition device 4 according to the first embodiment, description thereof is omitted.

  FIG. 28 is a flowchart for explaining the use language determination processing in the action recognition device 4 according to the third embodiment. The following processing is repeated for each person ID.

  First, in step ST501, a person located in front of the multilingual kiosk 8 is detected based on the position coordinates recorded in the person information management table (see FIG. 6). The action recognition device 4 is assumed to hold the position coordinates of the multilingual kiosk 8 in advance.

  Next, in step ST502, it is determined whether or not a person located in front of the multilingual kiosk 8 has been detected in the position coordinates of the person information management table (see FIG. 6). When it is determined that it has been detected (step ST502: Yes), the process proceeds to step ST503, and when it is determined that it has not been detected (no person is located in front of the multilingual kiosk 8) (step ST502: No). The process is terminated.

  In step ST503, it is determined whether or not the detected person has operated the display input unit 61 of the multilingual kiosk 8. Here, it is determined that an operation has been performed when the detected person has stayed in front of the multilingual kiosk 8 for a predetermined time or more. If it is determined that it has been operated (step ST503: Yes), the process proceeds to step ST504, and if it is determined that it has not been operated (step ST503: No), the process is terminated.

  In step ST504, the multilingual kiosk 8 is inquired about the language used when the person operates the display input unit 61 of the multilingual kiosk 8.

  In step ST505, the result of the inquiry, that is, the use language of the person is stored in the use language column of the person information management table, and then the process ends.

  As described above, according to the language ratio management system according to the third embodiment, it is possible to determine the language used by the person based on the language behavior of the person operating the multilingual kiosk 8. This makes it possible to determine the language used by a person at a place where an unspecified number of persons come and go.

  Note that the language ratio management system 1 may include both the multilingual bulletin board 2 and the multilingual kiosk 8, and may determine the language used by a person using both devices. You may make it give priority.

(Fourth embodiment)
Next, the language ratio management system 1 according to the fourth embodiment of the present disclosure will be described. In the fourth embodiment, items not particularly mentioned below are the same as those in the above-described first embodiment.

  FIG. 29 is a schematic diagram showing a schematic configuration of the language ratio management system 1 according to the fourth embodiment. The fourth embodiment is different from the first embodiment in that the person verification device 9 is further provided.

  In the fourth embodiment, a person image extracted from a captured image captured by the camera 3A connected to the action recognition apparatus 4A by the person verification device 9 and a captured image captured by the camera 3B connected to the action recognition apparatus 4B. The person image extracted from the image can be collated. For example, when a person H1 located around the display unit 11 of the multilingual bulletin board 2 and captured by the camera 3B moves in the direction of the multilingual kiosk 8 and is captured by the camera 3A, the person H1 captured by the camera 3A is displayed. The person verification device 9 can collate the person image and the person image of the person H1 captured by the camera 3B. Therefore, if the language used by the person H1 is known by the action recognition device 4B, the language used by the person H1 can be notified to the action recognition device 4B via the person verification device 9.

  FIG. 30 is a functional block diagram of the person verification device 9. As shown in FIG. 30, the person verification device 9 includes an action recognition device connection unit 71 that connects to the action recognition device 4, a storage unit 72 that stores a person language table, and a person verification unit 73 that matches a person image. Have.

  FIG. 31 is a diagram illustrating an example of a person language table. As shown in FIG. 31, in the person language table, the use language of the person of the person image is stored in association with the person image that is a binary image.

  FIG. 32 is a functional block diagram of the action recognition device 4 according to the fourth embodiment. As shown in FIG. 32, the action recognition device 4 according to the fourth embodiment further includes a person verification device connection unit 74, and is connected to the person verification device 9 via the person verification device connection unit 74. . Since the other points are the same as those of the action recognition device 4 according to the first embodiment, description thereof is omitted.

  FIG. 33 is a flowchart for explaining person position information update processing in the action recognition device 4 according to the fourth embodiment. In the person position information update process according to the fourth embodiment, a new person is detected from the captured image (current frame image), and the person image extracted from the current frame image is added after step ST205 for adding the new person ID. Is transmitted to the person verification device 9, and the usage language information indicating the usage language corresponding to the person image is acquired from the person verification device 9 (step ST601). The acquired use language information is recorded in the use language column of the person information management table (see FIG. 6) in association with the person ID assigned to the new person (step ST602). Since the other points are the same as the person position information update process according to the first embodiment, description thereof will be omitted.

  FIG. 34 is a flowchart for explaining the use language determination processing in the action recognition device 4 according to the fourth embodiment, and the same parts as ST301-307 in FIG. 20 are omitted. In the use language determination process according to the fourth embodiment, after the language corresponding to the region of interest is stored (stored) in the use language column of the person information management table in step ST309, information indicating the use language is stored in the image person region. And the person image are transmitted to the person verification device 9 (step ST603). Since the other points are the same as the person position information update process according to the first embodiment, description thereof will be omitted.

  When the person verification device 9 acquires the person image and the use language from the action recognition device 4 (behavior recognition device 4B in the example of FIG. 29), the person check device 9 records the person image and the use language in the person language table (FIG. 31).

  FIG. 35 is a flowchart for explaining the collation processing of the person collation apparatus 9. The following processing corresponds to the processing of step ST601 and step ST602 in the person position information update processing of the action recognition device 4 described with reference to FIG.

  First, in step ST701, a person image to be collated is received from the action recognition device 4 (in the example of FIG. 29, the action recognition device 4A).

  Next, in step ST702, the person image to be collated is collated with the person image stored in the person language table.

  Subsequently, in step ST703, it is determined whether or not a person image that matches the person image to be collated is stored in the person language table, that is, whether or not a corresponding person exists. When it is determined that the corresponding person does not exist (step ST703: No), the process proceeds to step ST704, and when it is determined that the corresponding person exists (step ST703: Yes), the process proceeds to step ST705.

  In step ST704, the action recognition device 4 is notified that “the language used is unknown”, and then the process ends.

  In step ST705, the use language of the corresponding person is determined with reference to the person language table, and the determined use language is notified to the action recognition device 4. Thereafter, the process ends.

  As described above, according to the language ratio management system according to the fourth embodiment, since the person detected from the captured image can be collated between the plurality of action recognition apparatuses 4, the plurality of action recognition apparatuses 4 can It is possible to share information indicating the language used by a person.

(Fifth embodiment)
Next, a language ratio management system 1 according to the fifth embodiment of the present disclosure will be described. In the fifth embodiment, matters not particularly mentioned below are the same as those in the above-described first embodiment.

  FIG. 36 is a schematic diagram showing a schematic configuration of the language ratio management system 1 according to the fifth embodiment. In the fifth embodiment, when a group of persons existing on the floor F is determined and the use language of at least one person in the group is determined, the use language of the person whose use language in the group is unknown is determined. The point which determines with a use language differs from said 1st Embodiment.

  In the example of FIG. 36, on the floor F, there are a group G1 composed of three persons H2, H3, and H4 and a group G2 composed of two persons H5 and H6. Since people in the same group are considered to use the same language, if the language used by at least one person in the same group is known, the language used is the language used by other people in the group. Can be determined.

  Therefore, in the group G1, when the usage language of the person H2 is determined, even if the usage languages of the persons H3 and H4 of the group G1 are unknown, the usage language of the persons H3 and H4 is the usage language of the person H1 (FIG. 36). In the example, it can be determined as English). Similarly, in the group G2, when the language used by the person H5 is known, even if the language used by the person H6 in the group G2 is unknown, the language used by the person H6 is the language used by the person H5 (China in the example of FIG. 36). Word).

  FIG. 37 is a functional block diagram of the action recognition device 4 according to the fifth embodiment. As shown in FIG. 37, the action recognition device 4 according to the fifth embodiment further includes a group determination unit 81 that determines a group. Since the other points are the same as those of the action recognition device 4 according to the first embodiment, description thereof is omitted.

  FIG. 38 is a diagram showing a person information management table according to the fifth embodiment. As shown in FIG. 38, the person information management table further includes a position coordinate history column in addition to the person ID, the image person area, and the language used. In the position coordinate history column, position coordinates acquired from the current frame image are added and recorded.

  FIG. 39 is a flowchart for explaining a series of processes in the action recognition device 4 according to the fifth embodiment.

  As shown in FIG. 39, in the fifth embodiment, a group determination process (step S104) is performed between the language information update process in step ST104 and the process in which the captured image in step ST105 is stored in the storage unit 26 as a previous frame image. ST106). In this group determination process, a group of persons is determined, and when the use language of one of the groups is found, the use language is set as the use language of a person whose use language is unknown in the group. Since the other points are the same as the series of processes in the action recognition device 4 according to the first embodiment, description thereof is omitted.

  FIG. 40 is a flowchart for explaining group determination processing of the action recognition device 4 according to the fifth embodiment.

  First, in step ST801, for each person ID, the position coordinates of the current frame image are added and stored in the position coordinate history column of the person information management table (see FIG. 38).

  Next, in step ST802, based on the position coordinates stored in the position coordinate history column of the person information management table, a group of persons and a person ID belonging to the group are determined. For example, a plurality of persons that exist together in a predetermined number of consecutive frame images and whose mutual distances are within a predetermined distance may be determined as a group. The group determination method is not limited to this, and various other known methods can be used.

  Subsequently, in step ST803, it is determined whether or not there is a person whose use language is unknown in the determined group. If it is determined that there is a person whose use language is unknown (step ST803: Yes), the process proceeds to step ST804. If it is determined that there is no person whose use language is unknown (step ST803: No), the process ends. To do.

  In step ST804, the use language most frequently used by the persons in the group is determined as the use language of the person whose use language is unknown. Then, the determined usage language is recorded in the usage language column of the person information management table, and then the processing is terminated.

  As described above, according to the language ratio management system according to the fifth embodiment, when the use language of at least one person in the group is found, the use language is used as the use language of a person whose use language is unknown in the group. It becomes possible to judge. Note that when the language used by at least one person in the group is found, the language used may be the language used by all persons in the group.

(Sixth embodiment)
Next, the language ratio management system 1 according to the sixth embodiment of the present disclosure will be described. In the sixth embodiment, matters not particularly mentioned below are the same as those in the first embodiment described above.

  FIG. 41 is a schematic diagram showing a schematic configuration of the language ratio management system 1 according to the sixth embodiment. The sixth embodiment is different from the first embodiment in that a microphone 10 is provided instead of the multilingual bulletin board 2.

  The microphone 10 is a sound collection microphone having a known configuration, and is arranged at an appropriate position on the floor F of the airport to collect speech such as a person's conversation.

  FIG. 42 is a functional block diagram of the action recognition device 4 according to the sixth embodiment. As shown in FIG. 42, the action recognition device 4 according to the sixth embodiment has a microphone connection unit 91 instead of the multilingual bulletin board connection unit 27, and is connected to the microphone 10 via the microphone connection unit 91. Has been. The use language determination unit 29 analyzes the sound collected by the microphone 10 using a known sound analysis technique, and determines the use language of the person. Moreover, the action recognition device 4 according to the sixth embodiment does not include the line-of-sight direction detection unit 28 and the language determination table 30. Since the other points are the same as those of the action recognition device 4 according to the first embodiment, description thereof is omitted.

  FIG. 43 is a flowchart for explaining the use language determination processing in the action recognition device 4 according to the sixth embodiment. The following processing is repeated for each person ID.

  First, in step ST901, a person located in front of the microphone 10 is detected based on the position coordinates recorded in the person information management table (see FIG. 6). The action recognition device 4 is assumed to hold the position coordinates of the microphone 10 in advance.

  Next, in step ST902, it is determined whether or not a person located in front of the microphone 10 is detected in the position coordinates of the person information management table (see FIG. 6). When it is determined that it has been detected (step ST902: Yes), the process proceeds to step ST903, and when it is determined that it has not been detected (no person is located in front of the microphone 10) (step ST902: No), processing is performed. Exit.

  In step ST903, it is determined whether or not the detected voice of the person has been acquired by the microphone 10. When it is determined that the voice has been acquired (step ST903: Yes), the process proceeds to step ST904, and when it is determined that the voice has not been acquired (step ST903: No), the process ends.

  In step ST904, the use language determination unit 29 analyzes the sound collected by the microphone 10 and determines the use language of the person.

  In step ST905, the determined use language is stored (stored) in the use language column of the person information management table, and then the process ends.

  As described above, according to the language ratio management system according to the sixth embodiment, the language used by a person can be determined based on the language behavior of the person called speech. This makes it possible to determine the language used by a person at a place where an unspecified number of persons come and go.

  The language ratio management system 1 may include an arbitrary combination of the multilingual bulletin board 2, the multilingual kiosk 8, and the microphone 10, and each device may determine the language used by the person. Moreover, you may make it give priority to the determination result of either apparatus.

  As mentioned above, although this indication was explained based on specific embodiments, these embodiments are illustrations to the last, and this indication is not limited by these embodiments. Moreover, it is good also as embodiment which arbitrarily combined 6th Embodiment from 1st Embodiment. In addition, each component of the language ratio management system and the language ratio management method according to the present disclosure shown in the above embodiment is not necessarily essential, and may be appropriately selected as long as it does not depart from the scope of the present disclosure. Is possible.

  The language ratio management system and the language ratio management method according to the present disclosure are useful as a language ratio management system, a language ratio management method, and the like that can manage the ratio of used languages in a place where multiple languages are used.

DESCRIPTION OF SYMBOLS 1 Language ratio management system 2 Multilingual bulletin board 3 Camera 4 Action recognition apparatus 5 Language ratio management server 6 In-house broadcasting apparatus 7 Multilingual broadcasting robot 8 Multilingual kiosk 9 Person collation apparatus 10 Microphone

Claims (9)

A language ratio management system for managing the ratio of languages used in a place where multiple languages are used,
A camera that captures the target area for which the percentage of languages used is to be managed;
An action recognition device that detects and tracks a person from a captured image of the camera and determines a language used by the person based on the language action of the person;
A language ratio management server that manages the ratio of the used language used in the target area by acquiring and storing information indicating the used language from the action recognition device. Percentage management system. A multilingual bulletin board having a plurality of display areas arranged in the target area and displaying different languages from each other;
The behavior recognition device detects a gaze direction of the person from the captured image, identifies a display area in which the person is gazing out of the plurality of display areas based on the gaze direction, and identifies the identified The language ratio management system according to claim 1, wherein a language displayed in a display area is determined as a language used by the person. A multilingual kiosk arranged in the target area and operable in multiple languages;
3. The language ratio management system according to claim 1, wherein the behavior recognition apparatus determines a language used when the person operates the multilingual kiosk as a language used by the person. 4. A microphone disposed in the target area;
The language ratio management system according to claim 1, wherein the behavior recognition device determines a language used by the person based on the voice of the person collected by the microphone. . Further comprising an in-house broadcasting device for outputting multilingual audio information for the target area,
The in-house broadcasting device inquires the language ratio management server for language ratio information indicating the ratio of the used language used in the target area, acquires the language ratio information and outputs the language based on the language ratio information, The language ratio management system according to any one of claims 1 to 4, wherein an order of outputting the multilingual voice is determined. A multilingual guidance robot that outputs multilingual information in the target area;
The multilingual guidance robot inquires the language ratio management server for language ratio information indicating a ratio of the used languages used in the target area, and obtains the type of multilanguage to be output based on the language ratio information and The language ratio management system according to any one of claims 1 to 5, wherein an order of outputting the multi-language is determined. A human collation device for collating a person detected from the captured image between the plurality of behavior recognition devices;
The person verification device
From at least one of the action recognition devices, a person image of a person detected from the captured image and information indicating the language used by the person are acquired and stored in advance.
When a person image of a person to be collated is received from a behavior recognition apparatus different from the at least one action recognition apparatus, the person image of the person to be collated is compared with the person image stored in advance and matches 7. The language ratio management system according to claim 1, wherein information indicating a language used by a person of the person image stored in advance is transmitted to the another action recognition device.   A group of persons is determined based on the tracking result of the person detected from the captured image, and a language used by a person whose usage language is known in the group is defined as a language used by a person whose usage language is unknown in the group. The language ratio management system according to any one of claims 1 to 7, wherein: A language ratio management method for managing the ratio of languages used in a place where multiple languages are used,
Imaging the target area for which the ratio of the language used is to be managed by the camera;
Detecting and tracking a person from a captured image of the camera, and determining a language used by the person based on the language behavior of the person;
Storing the information indicating the language used to manage the rate of the language used in the target area.

Images