[go: up one dir, main page]

WO2018025492A1 - Dispositif de surveillance de la presence d'un client, système et procédé de surveillance de service à la clientèle - Google Patents

Dispositif de surveillance de la presence d'un client, système et procédé de surveillance de service à la clientèle Download PDF

Info

Publication number
WO2018025492A1
WO2018025492A1 PCT/JP2017/020901 JP2017020901W WO2018025492A1 WO 2018025492 A1 WO2018025492 A1 WO 2018025492A1 JP 2017020901 W JP2017020901 W JP 2017020901W WO 2018025492 A1 WO2018025492 A1 WO 2018025492A1
Authority
WO
WIPO (PCT)
Prior art keywords
customer
information
customer service
monitoring device
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2017/020901
Other languages
English (en)
Japanese (ja)
Inventor
林　裕昭
寿嗣 辻
鉄平 福田
亮太 藤井
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Intellectual Property Management Co Ltd
Original Assignee
Panasonic Intellectual Property Management Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Panasonic Intellectual Property Management Co Ltd filed Critical Panasonic Intellectual Property Management Co Ltd
Publication of WO2018025492A1 publication Critical patent/WO2018025492A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/04—Segmentation; Word boundary detection

Definitions

  • This disclosure relates to a service monitoring device, a service monitoring system, and a service monitoring method.
  • a camera is installed in the store, and this camera is used for security purposes.
  • this camera may be used for improvement of store operation, for example, confirmation of the presence or absence of customer service by a store clerk.
  • the following activity status analysis system is known as a system for determining the presence or absence of customer service using a camera (see, for example, Patent Document 1).
  • This activity status analysis system acquires the staying information of a moving body from a captured image by a camera that captures a target area, and acquires the staying information of an ID moving body from a portable terminal held by a store clerk. Then, this activity state analysis system determines that there is a customer service when the ID mobile body and the mobile body stay at a nearby point. The determination result of this customer service is utilized for store management as customer service information.
  • the activity situation analysis system may determine that there is a customer service if the store clerk and the customer are nearby, and the customer service determination accuracy may decrease. For example, even when a store clerk and a customer are located close to each other, it may be determined that the customer is in service, and it is desirable that the determination accuracy of the customer service is improved.
  • This disclosure aims to improve customer service accuracy.
  • the customer service monitoring device acquires sound data related to sound collected by a sound collecting device that collects sound of a target area divided into a plurality of grids, and captures an image of the target area.
  • the utterance detection unit that detects the utterance based on the audio data, and based on the image data, the customer information or the residence information of the person including the customer is generated,
  • a residence information generation unit that generates customer residence information, and detection of person residence information, customer residence information, and utterances Based on the result, when the customer and the customer stay in the same grid for the first predetermined time and the utterance is detected for the second predetermined time or more in the same grid, the customer receives the customer in the same grid. Have And a customer determination unit determines that.
  • the customer service monitoring system of the present disclosure includes a sound collection device that collects sound of a target area divided into a plurality of grids, an imaging device that captures an image of the target area, and a customer service monitoring device.
  • the customer service monitoring device acquires sound data related to the sound collected by the sound collecting device, acquires image data related to the image picked up by the image pickup device, detects speech based on the sound data, and detects image data Based on the location information of the customer including the customer or the customer is generated, and the customer's residence information is generated based on the location information of the mobile terminal possessed by each customer in the target area.
  • the staying person and the customer stay in the same grid for a first predetermined time, and the utterance exceeds the second predetermined time in the same grid.
  • the customer service monitoring method of the present disclosure is a customer service monitoring method in the customer service monitoring device, and acquires voice data related to a sound collected by a sound collecting device that collects sound of a target area divided into a plurality of grids. , Including image data related to an image captured by an imaging device that captures an image of the target area, and an utterance detection unit that detects an utterance based on audio data, and a customer or a customer based on the image data Based on the location information of the mobile terminal possessed by each customer who is located in the target area, the customer's residence information is generated, and the person's residence information, the customer's residence information, and Based on the utterance detection result, if the customer and the customer stay in the same grid for the first predetermined time and the utterance is detected for the second predetermined time or more in the same grid, the same grid is detected. Determines that the customer who is in customer service in de.
  • FIG. 1 is a block diagram showing a customer service monitoring system according to the first embodiment.
  • FIG. 2 is a flowchart illustrating an operation example of the customer service monitoring apparatus according to the first embodiment.
  • FIG. 3 is a schematic diagram illustrating an example of a customer service viewed from the ceiling of the store.
  • FIG. 4 is a perspective view corresponding to FIG. 3 and showing an example of customer service in a store.
  • FIG. 5 is a block diagram illustrating a configuration example of the customer service monitoring system according to the second embodiment.
  • FIG. 6 is a flowchart illustrating an operation example of the customer service monitoring apparatus according to the second embodiment.
  • FIG. 7 is a schematic diagram for explaining a specific action performed by a customer.
  • FIG. 8A is a schematic diagram for explaining an example of peeping determination.
  • FIG. 8A is a schematic diagram for explaining an example of peeping determination.
  • FIG. 8B is a schematic diagram for explaining an example of the ginger determination.
  • FIG. 8C is a schematic diagram for explaining an example of tilt determination.
  • FIG. 9 is a schematic diagram for explaining a store clerk serving as a transmission destination of customer service instruction information.
  • FIG. 10 corresponds to FIG. 9 and is a schematic diagram for explaining a store clerk serving as a destination of customer service instruction information.
  • FIG. 11 is a block diagram illustrating a configuration example of the customer service monitoring system according to the third embodiment.
  • FIG. 12 is a flowchart illustrating an operation example of the customer service monitoring apparatus according to the third embodiment.
  • FIG. 13 is a schematic diagram for explaining the identity of the staying information.
  • FIG. 1 is a block diagram illustrating a configuration example of a customer service monitoring system 100 according to the first embodiment.
  • the customer service monitoring system 100 includes a mobile terminal TE, a microphone array device MA, a camera device CA, a recorder RC, and a customer service monitoring device 10.
  • the mobile terminal TE, the microphone array device MA, the camera device CA, the recorder RC, and the customer service monitoring device 10 are connected via a network NT so that data communication is possible.
  • the network NT is formed including a wired network or a wireless network.
  • the network NT widely includes various networks such as the Internet, a cellular network, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, and a dedicated line.
  • the portable terminal TE (business terminal) is carried by a store clerk SA (for example, see FIGS. 3 and 4) working in the store ST, and can communicate with another communication device (for example, the customer service monitoring device 10) via the network NT. is there.
  • another communication device for example, the customer service monitoring device 10.
  • customers who enter the store ST do not have the mobile terminal TE.
  • the mobile terminal TE detects the position of the mobile terminal TE for the service determination by the service monitoring device 10.
  • Position detection technology using GPS (Global Positioning System), wireless LAN, RFID (Radio Frequency IDentifier), or the like may be used to detect the position of the mobile terminal TE in the store ST.
  • the mobile terminal TE communicates a beacon signal with each beacon terminal installed at a predetermined interval in the in-store ST, and based on the positional relationship with the beacon terminal as the transmission source of the received beacon signal, You may detect the position in ST.
  • the beacon signal may include, for example, information on coordinates indicating a relative position with respect to a reference point in the store ST (for example, the center position of the store ST). Further, the position of the mobile terminal TE in the store ST may be detected by a method other than the above.
  • the portable terminal TE transmits, to the customer service monitoring device 10, the position information where the portable terminal TE is located, for example, by an arbitrary communication method (for example, communication using wireless LAN communication or a public line). Further, the mobile terminal TE receives from the customer service monitoring device 10 information related to customer service (for example, customer service instruction information, information indicating the necessity of customer service, customer CT position information).
  • customer service instruction information for example, customer service instruction information, information indicating the necessity of customer service, customer CT position information.
  • the mobile terminal TE may include an acceleration sensor and detect in which direction the clerk SA carrying the mobile terminal TE has moved (movement information).
  • the customer service monitoring device 10 generates flow line information and staying information for the clerk SA and the customer CT based on the position information of the mobile terminal TE acquired from the mobile terminal TE or based on the position information and movement information of the mobile terminal TE. To do.
  • the camera device CA is a fixed camera with a fixed angle of view, for example, installed on the ceiling or wall of the store ST.
  • the camera device CA functions as a monitoring camera that can image an imaging area that is an imaging space in which the device itself is installed.
  • the camera device CA may temporarily capture a video (image), or may continuously capture and accumulate video (image).
  • the camera apparatus CA is not limited to a fixed camera, and may be an omnidirectional camera or a PTZ camera capable of pan / tilt / zoom operations.
  • the camera device CA stores the time (image capturing time) when the image is captured in association with the image data. Further, the camera device CA transmits the stored video data and imaging time data to the customer service monitoring device 10 via the network NW.
  • the video includes at least one of a moving image and a still image.
  • the microphone array device MA is an omnidirectional microphone array device installed on the ceiling of the store ST, for example.
  • the microphone array device MA collects sound in all directions in the sound collection space (sound collection area) where the device itself is installed.
  • the microphone array apparatus MA may temporarily collect voice, or may continuously collect voice and accumulate it.
  • the microphone array apparatus MA includes, for example, a housing having an opening formed at the center, and a plurality of microphone units arranged concentrically around the opening in the circumferential direction.
  • a high sound quality small electret condenser microphone ECM: Electric Condenser Microphone
  • ECM Electric Condenser Microphone
  • the imaging area and the sound collection area are substantially the same.
  • the imaging area and the sound collection area are service determination target areas.
  • the microphone array device MA stores the collected sound data in association with the time of sound collection (sound collection time). Further, the microphone array apparatus MA transmits the stored voice data and sound collection time data to the customer service monitoring apparatus 10 via the network NW.
  • Recorder RC stores various data.
  • the recorder RC may store the video data captured by the camera device CA and the video time information together with the customer service monitoring device 10 or instead of the customer service monitoring device 10.
  • the recorder RC may store the voice data collected by the microphone array device MA and the sound collection time together with the customer service monitoring device 10 or instead of the customer service monitoring device 10.
  • the customer service monitoring device 10 is installed in or outside the store ST where the microphone array device MA and the camera device CA are installed, for example, in the back yard of the store ST.
  • the customer service monitoring device 10 is, for example, a stationary PC (Personal Computer) or a server.
  • the customer service monitoring device 10 may be installed as a server (for example, a cloud server) in a remote place (for example, a monitoring center for collectively monitoring each store ST) away from the store ST.
  • the customer service monitoring device 10 is monitored, for example, by a monitor (for example, a manager of the store ST, a monitor of the monitoring center), and the display is confirmed and operated.
  • the customer service monitoring device 10 is used for voice detection function (speech detection function), voice detection result accumulation function, person flow line detection function (staying detection function), and voice detection and person flow line detection. Based on customer service determination function.
  • the customer service monitoring device 10 has a position detection function (positioning measurement function) for detecting a position where the mobile terminal TE is located (corresponding to a position of an ID mobile body described later).
  • the customer service monitoring device 10 may be a communication terminal such as a mobile phone, a tablet terminal, or a smartphone instead of the PC.
  • the customer service monitoring device 10 includes a communication unit 11, a control unit 12, a memory 13, an operation unit 14, a display device 15, and a speaker device 16.
  • the customer service monitoring device 10 includes a voice processing unit 20, a person detection unit 30, and a customer service determination unit 40.
  • the voice processing unit 20 includes a voice detection unit 21 and a detection result accumulation unit 22.
  • the person detecting unit 30 includes a moving body staying information generating unit 31 and an ID staying information generating unit 32.
  • the customer service determination unit 40 includes a customer service condition setting unit 41 and a customer service action extraction unit 42.
  • the communication unit 11 receives the video data including the imaging time transmitted from the camera device CA and the audio data including the sound collection time transmitted from the microphone array device MA, and outputs them to the control unit 12. Further, the communication unit 11 receives position information of the mobile terminal TE from the mobile terminal TE.
  • the operation unit 14 is a user interface (UI: User Interface) for notifying the control unit 12 of the content of the user's input operation, and includes a pointing device such as a mouse or a keyboard.
  • UI User Interface
  • the operation unit 14 may be configured using, for example, a touch panel or a touch pad that is arranged corresponding to the screen of the display device 15 and can be input with a user's finger or stylus pen.
  • the operation unit 14 receives an input operation for designating a range of the grid GR.
  • the operation unit 14 may receive an input operation for changing the range of the grid GR.
  • the grid GR is a unit area for determining customer service.
  • the grid GR may be set according to, for example, a new product area, a seasonable area, a standard area, or an area that is classified according to the characteristics of the store.
  • the size of the grid GR may be set arbitrarily.
  • the range of one grid GR may be defined in association with the range of the area where the products of the same concept are displayed, or without being related to the range of the area where the products of the similar concept are displayed, A range of one grid GR may be defined.
  • the division of the grid GR may be determined at the beginning of the opening of the store and may not be changed, or may be changed every time the layout is changed in the store ST.
  • the memory 13 may include, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory).
  • the memory 13 functions as a program memory, a data memory, and a work memory when the customer service monitoring device 10 operates.
  • the memory 13 may include, for example, an HDD (Hard Disk Drive) or an SDD (Solid State Drive).
  • the memory 13 stores, for example, sound data of the sound collected by the microphone array device MA together with the sound collection time.
  • the memory 13 stores, for example, video data of video captured by the camera device CA together with the imaging time. Further, the memory 13 holds information on the portable ID of the portable terminal TE possessed by each clerk SA.
  • the control unit 12 supervises each component of the customer service monitoring device 10.
  • the voice processing unit 20 performs various voice processes on voice data collected by the microphone array device MA.
  • the sound detection unit 21 detects the presence or absence of sound at each position (for example, each divided area described later) in the sound collection area in the store ST.
  • the voice detection by the voice detection unit 21 is performed, for example, constantly or periodically.
  • the voice detection unit 21 performs directivity processing and directs directivity to each divided area DA (see FIG. 3). That is, the voice detection unit 21 sequentially designates each divided area DA, and forms directivity in the direction of the designated divided area DA with the microphone array device MA as a base point based on the voice data. Thereby, the customer service monitoring apparatus 10 can detect the presence or absence of the sound in each divided area DA.
  • the sound collection area (for example, the entire store ST) is divided into a predetermined number in the x direction and the y direction (see FIG. 3), for example. That is, the sound collection area is divided into divided areas DA for determining the presence / absence (utterance presence / absence) of the clerk SA or the customer CT.
  • the sound detection unit 21 detects the presence or absence of sound for each divided area DA. Note that the size of one divided area DA is smaller than the size of one grid GR.
  • the presence or absence of sound for example, when the signal level of the sound data is equal to or higher than a predetermined level, it is detected that there is sound (there is an utterance), and there is no sound when the signal level of the sound data is lower than the predetermined level. (No utterance) may be detected.
  • the detection results are accumulated in time series and can be detected at a constant rate within a predetermined time, it may be detected that there is a voice (there is an utterance). In this case, the customer service monitoring device 10 can reduce false detection due to sudden noise in the store ST.
  • the detection result of the presence or absence of audio in each divided area DA may be visualized as an audio heat map by being displayed on the display device 15 under the control of the control unit 12. Further, in consideration of the size of the sound level, it may be visualized as a sound heat map indicating each position in the store ST and the sound level at that position.
  • the utterance may be detected in one or more divided areas DA.
  • utterances may be detected in all the divided areas DA in which utterances are detected, or utterances are detected in one divided area DA having the maximum detected voice level among the divided areas DA in which utterances are detected. May be.
  • the detection result accumulating unit 22 accumulates and stores in the memory 13 the detection result (sound heat map) of the presence or absence of sound detected by the sound detecting unit 21. For example, the detection result accumulating unit 22 accumulates information on the detection result of the presence / absence of sound in each divided area DA generated every predetermined time.
  • the person detecting unit 30 detects the presence / absence of the clerk SA and the customer CT at each position (for example, each grid GR) in the store ST. For example, the person detection unit 30 accumulates detection information of the positions of the clerk SA and the customer CT, and derives flow line information indicating a trajectory of the movement of the clerk SA and the customer CT over time. Based on the flow line information, the person detection unit 30 derives retention information indicating the retention of the clerk SA and the customer CT.
  • the moving body staying information generating unit 31 generates staying information on the moving body (person) using the video data from the camera device CA.
  • This moving body includes, for example, a customer CT or a store clerk SA. If the ID residence information (corresponding to the residence information of the clerk SA) acquired by the ID residence information generation unit 32 is subtracted from the residence information generated by the mobile body residence information generation unit 31, the residence information of the customer CT is derived. Is done.
  • the moving body staying information generating unit 31 acquires real-time video data imaged by the camera device CA or past video data recorded in the recorder RC or the memory 13.
  • the moving body staying information generating unit 31 detects a person (moving body) from the acquired video data, and generates tracking information (detected position information) for each person.
  • This moving body tracking process is performed using, for example, an image recognition technique such as a person detection technique or a person tracking technique.
  • the moving object staying information generating unit 31 stores the tracking information in the memory 13.
  • the tracking information includes information on the detection position for each person detected in each frame of the video data, the detection time for each person acquired from the imaging time of the frame in which the person was detected, and the like.
  • the moving body staying information generating unit 31 generates flow line information for each person detected from the video data based on the generated tracking information.
  • the moving body staying information generation unit 31 acquires the tracking information from the memory 13 when performing the flow line generation processing after the fact, and directly uses the generated tracking information when performing the flow line generation processing in real time. .
  • the moving body staying information generating unit 31 stores the flow line information generated for each person in the memory 13.
  • the flow line information includes information relating to a movement vector indicating the locus of movement of the person, the position and time of the change point of the movement vector, and the like.
  • the moving body staying information generating unit 31 may generate the flow line information for the peripheral part of each grid GR in order to determine the intersection between the outer periphery of each grid GR and the flow line in the target area.
  • the moving body staying information generating unit 31 generates staying information for each person in the target area (information indicating how much time is staying at which position) for each grid GR based on the flow line information for each person. .
  • the staying information includes, for example, the position of the staying grid GR, its staying time or staying time.
  • the moving object staying information generating unit 31 stores the generated staying information for each person in the memory 13.
  • the ID residence information generation unit 32 derives residence information (also referred to as ID residence information) of the mobile terminal TE using an ID (mobile ID) for identifying the mobile terminal TE. Since the portable terminal TE is possessed by the store clerk SA, the ID stay information can be said to be stay information related to the store clerk SA.
  • the ID retention information generation unit 32 acquires information (ID position information) on the position where the mobile terminal TE is detected by communication with the mobile terminal TE by the communication unit 11.
  • ID position information detection technique for example, the technique described above is used.
  • the ID residence information generation unit 32 accumulates the acquired ID position information in the memory 13.
  • the ID residence information generation unit 32 generates flow line information (ID flow line information) of the mobile terminal TE based on the acquired ID position information.
  • the ID staying information generation unit 32 acquires the ID position information once accumulated in the memory 13 when the flow line generation process is performed later, and the detected ID when the flow line generation process is performed in real time. Use location information directly.
  • the ID retention information generation unit 32 stores ID flow line information in the memory 13.
  • the ID flow line information includes information regarding a movement vector indicating the movement locus of the mobile terminal TE, the position of the change point, the time, and the like.
  • the ID staying information generation unit 32 may generate flow line information for the peripheral part of each grid GR in order to determine the intersection between the outer periphery of each grid GR and the flow line in the target area.
  • the ID staying information generating unit 32 generates ID staying information in the target area for each grid GR based on the generated ID flow line information for each person (each store clerk SA).
  • the ID staying information includes, for example, the position of the staying grid GR and the staying time and staying time thereof.
  • the ID staying information generating unit 32 stores the generated ID staying information in the memory 13.
  • the customer service determination unit 40 determines whether or not the clerk SA is serving the customer CT at the store ST.
  • the customer service condition setting unit 41 sets customer service conditions based on input information received via the operation unit 14, for example.
  • the customer service condition includes, for example, selection of an area (grid GR) for performing customer service determination, operation related to customer service (for example, operation related to customer service shown in FIG. 2 described later), and setting of various threshold values used in operations related to customer service. It is.
  • the customer service condition defines, for example, in which area the store clerk SA or customer CT determines what kind of behavior (customer service behavior) is determined to be in service.
  • the customer service behavior extraction unit 42 extracts the customer service behavior that satisfies the set customer service conditions. When the customer service condition satisfying the customer service is extracted, it indicates that the clerk SA is serving the customer CT. The customer service behavior extraction unit 42 extracts presence / absence of customer service behavior in the store ST.
  • the display device 15 displays real-time or past video data captured by the camera device CA on the screen.
  • the speaker device 16 outputs the sound data picked up by the microphone array device MA or the sound data picked up by the microphone array device MA having directivity formed at the directivity angles ( ⁇ MAh, ⁇ MAv).
  • the display device 15 and the speaker device 16 may be configured as separate devices from the customer service monitoring device 10.
  • the directivity angle ⁇ MAh indicates the horizontal angle from the microphone array apparatus MA to the sound source position (for example, the divided area DA)
  • the directivity angle ⁇ MAv indicates the vertical angle from the microphone array apparatus MA to the sound source position (for example, the divided area DA). Show.
  • each constituent unit for example, the control unit 12, the voice processing unit 20, the person detection unit 30, and the service determination unit 40
  • the processor includes a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphical Processing Unit), and the like.
  • FIG. 2 is a flowchart showing an operation example of the customer service monitoring device 10.
  • the customer service condition setting unit 41 sets customer service conditions based on, for example, input information obtained via the operation unit 14 (S11).
  • the customer service conditions here include, for example, the processes shown in S14 to S19.
  • the moving body staying information generating unit 31 acquires video data from the camera device CA and generates staying information on the moving body (S12). Further, the ID residence information generation unit 32 acquires information on the position of each clerk SA from the mobile terminal TE identified by the mobile ID, and generates ID residence information (S12).
  • the staying information and ID staying information of the moving body may be real-time information or past information.
  • the generation of the stay information of the moving body and the generation of the ID stay information may be performed constantly or periodically.
  • the voice processing unit 20 acquires voice data from the microphone array device MA via the communication unit 11. Moreover, the audio
  • the customer service condition setting unit 41 selects a grid GR to be analyzed (customer determination target) in the store ST based on, for example, input information (operation information for selecting a grid) obtained via the operation unit 14. (S14).
  • the customer service behavior extraction unit 42 determines whether or not the store clerk SA and the customer CT are located in the grid GR to be analyzed based on the stay information and ID stay information of the moving body (S15). For example, the customer service behavior extraction unit 42 detects a plurality of moving bodies in the staying information of the moving body, and the ID staying information has less than the number of moving bodies in the staying information of the moving body and one or more ID moving bodies. Is detected, it is determined that the clerk SA and the customer CT are located in the grid GR to be analyzed. In S15, it may be determined whether or not the clerk SA and the customer CT exist for a certain period of time.
  • the voice processing unit 20 determines whether or not voice is detected for a certain period of time in an arbitrary divided area DA in the grid GR. Is determined (S16).
  • the divided areas DA to be subjected to sound detection are sequentially changed, and the presence or absence of sound is determined in each divided area DA.
  • the customer service action extraction unit 42 determines that the clerk SA is serving the customer CT in the grid GR (S17).
  • This voice includes, for example, the voice of the clerk SA or the voice of the customer CT.
  • the speaker may or may not be identified by voice, but depending on the position of the divided area DA where the voice is detected, it may be possible to identify the voice of the clerk SA or the voice of the customer CT. .
  • the extraction unit 42 determines that the clerk SA is not serving the customer CT (S18).
  • the customer service determination unit 40 determines whether the determination of the presence / absence of customer service is completed in all the grids GR to be analyzed (S19). When the determination of the presence / absence of customer service in all grids GR is completed (Yes in S19), the customer service monitoring device 10 ends the process of FIG. When the determination of the presence / absence of customer service in at least one grid GR has not been completed (No in S19), the process proceeds to S14, and the customer service determination unit 40 selects the next grid GR as the analysis target grid.
  • the customer service monitoring device 10 determines the position of the store clerk SA at which position in the store ST and at which time zone based on the stay information of the person (mobile body) and the stay information of the portable ID. And whether the customer CT is located. Further, the customer service monitoring device 10 can use the information on the presence / absence of voice in the divided area DA to determine the presence / absence of customer service. Therefore, the customer service monitoring device 10 can suppress the determination that there is customer service by simply detecting the approach between the clerk SA and the customer CT in the same time zone.
  • FIG. 3 is a schematic diagram showing an example of a customer service viewed from the ceiling of the store ST.
  • FIG. 4 corresponds to FIG. 3 and is a perspective view showing an example of a state of customer service in the store ST.
  • the clerk SA is located at an arbitrary position in the store ST, for example.
  • the customer CT approaches the product shelf SS in order to purchase and confirm the product displayed on the product shelf SS in the store ST.
  • the stores ST are supermarkets, home centers, furniture stores, department stores, and the like. When the store ST is a relatively wide store, the customer service monitoring system 100 is more useful.
  • the periphery of the product shelf SS1 is included in the areas of grids GR1, GR2, GR5, GR6.
  • the grid GR1 has a store clerk SA1 overlooking the entire store ST.
  • the store clerk SA1 greets the customer CT or the like who has visited the store.
  • voice is detected in each divided area DA of the area D1 in GR1.
  • the person detection unit 30 detects that one person is present in the grid GR1 based on the video data captured by the camera device CA (S12 in FIG. 2).
  • the person detection unit 30 detects that the clerk SA1 is located in the grid GR1 based on the portable ID of the portable terminal TE possessed by the clerk SA1 (S12 in FIG. 2). Therefore, the person detection unit 30 detects that there is one clerk SA1 in the grid GR1 and no customer CT (No in S15 in FIG. 2).
  • the voice processing unit 20 detects the voice generated by the clerk SA1 in the surrounding divided area DA including the location of the clerk SA1 in the grid GR1.
  • the customer service determination unit 40 determines that the customer is not in service in the grid GR1 (S18 in FIG. 2).
  • the store clerk SA2 and the customer CT1 are located in the area of the grid GR7 between the product shelves SS2 and SS3. As shown in FIG. 4, the clerk SA2 faces the product shelf SS2, the customer CT1 faces the product shelf SS3, and the clerk SA2 and the customer CT1 are back to back. Here, the clerk SA2 and the customer CT1 are not talking.
  • the person detection unit 30 detects that there are two sights on the grid GR7 based on the video data captured by the camera device CA (S12 in FIG. 2).
  • the person detection unit 30 detects that the clerk SA2 is located in the grid GR7 based on the portable ID of the portable terminal TE possessed by the clerk SA2 (S12 in FIG. 2). Therefore, the person detection unit 30 detects that one clerk SA2 and one customer CT1 are present in the grid GR7 (Yes in S15 of FIG. 2).
  • the sound processing unit 20 does not detect any sound emitted by the store clerk SA2 and the customer CT1 in the surrounding divided area DA including the locations of the store clerk SA2 and the customer CT1 (No in S16 of FIG. 2). . Therefore, the customer service determination unit 40 determines that the customer is not in service in the grid GR1 (S18 in FIG. 2).
  • the store clerk SA3 and the customer CT2 are located in the area of the grid GR14 between the product shelves SS4 and SS5. As shown in FIG. 4, the clerk SA3 and the customer CT2 face the same product shelf SS5. For example, the clerk SA3 and the customer CT2 have a conversation related to the products displayed on the product shelf SS5. In FIG. 3, voice (speech) is detected in each divided area DA of the area D2 in the GR14.
  • the person detection unit 30 detects that there are two sights on the grid GR14 based on the video data captured by the camera device CA (S12 in FIG. 2).
  • the person detection unit 30 detects that the clerk SA3 is located in the grid GR14 based on the mobile ID of the mobile terminal TE possessed by the clerk SA3 (S12 in FIG. 2). Therefore, the person detection unit 30 detects that one clerk SA3 and one customer CT2 are located in the grid GR1 (Yes in S15 in FIG. 2).
  • the voice processing unit 20 detects a voice uttered by at least one of the clerk SA3 and the customer CT2 in the surrounding divided area DA including the location of the clerk SA3 and the customer CT2 (Yes in S16 of FIG. 2). ). Therefore, the customer service determination unit 40 determines that the clerk SA3 is serving the customer CT2 in the grid GR14 (S17 in FIG. 2).
  • the customer service monitoring system 100 includes the mobile terminal TE, the microphone array device MA, the camera device CA, and the customer service monitoring device 10.
  • the customer service monitoring device 10 includes a communication unit 11, a voice processing unit 20, a person detection unit 30, and a customer service determination unit 40.
  • the communication unit 11 acquires audio data related to the sound collected by the microphone array device MA that collects the sound of the target area divided into a plurality of grids GD.
  • the communication unit 11 acquires image data relating to an image captured by the camera device CA that captures an image of the target area.
  • the voice processing unit 20 detects an utterance based on the voice data.
  • the person detection unit 30 generates residence information of a person including the clerk SA or the customer CT based on the image data.
  • the person detecting unit 30 generates the stay information of the clerk SA based on the position information of the mobile terminal TE possessed by each clerk SA located in the target area.
  • the customer service determination unit 40 stores the clerk SA and the customer CT in the same grid GR for a predetermined time t1 based on the person's stay information, the stay information of the store clerk SA, and the utterance detection result. When the utterance is detected for the time t2 or more, it is determined that the store clerk SA is in contact with the same grid GR.
  • the voice processing unit 20 is an example of an utterance detection unit.
  • the person detection unit 30 is an example of a staying information generation unit.
  • the microphone array device MA is an example of a sound collection device.
  • the camera device CA is an example of an imaging device.
  • the salesclerk SA is an example of a customer service.
  • the predetermined time t1 is, for example, 5 seconds.
  • the predetermined time t2 is, for example, 1 second.
  • the customer monitoring system 100 and the customer monitoring device 10 do not determine that the customer is in service in response to the fact that the clerk SA and the customer CT are close to each other, and the sound collected by the microphone array device MA is collected. Data can be used for customer service judgment. Therefore, the customer service monitoring system 100 and the customer service monitoring device 10 determine that the customer is not in service when, for example, the store clerk SA and the customer CT accidentally approach but do not talk and serve, so the determination accuracy of the service is increased. Can be improved. Moreover, the customer service monitoring system 100 and the customer service monitoring apparatus 10 can perform customer service determination without requiring a special operation for customer service determination.
  • FIG. 5 is a block diagram illustrating a configuration example of the customer service monitoring system 100A according to the second embodiment. 5 is different from the customer monitoring system 100 shown in FIG. 1 in that a customer monitoring device 10A is provided instead of the customer monitoring device 10.
  • the same components as those of the customer service monitoring system 100 shown in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted or simplified.
  • the customer service monitoring device 10A includes a voice processing unit 20A, a person detection unit 30, and a customer service determination unit 40A.
  • the voice processing unit 20A includes a voice detection unit 21, a detection result accumulation unit 22, a directivity synthesis unit 23, and a voice discrimination unit 24.
  • the service determination unit 40A includes a service condition setting unit 41A, a behavior determination unit 43, a service necessity determination unit 44, and a terminal notification unit 45.
  • the customer service determination unit 40A may have the function of the customer service determination unit 40 of the first embodiment. That is, the customer service determination unit 40A may determine the presence or absence of customer service at each position in the store ST.
  • the directivity synthesis unit 23 forms voice directivity in the direction where the specific customer CT in the store ST is located.
  • the specific customer CT is, for example, a customer who is not served by the store clerk SA, and includes a customer who needs an explanation about a product, a customer who is looking for a product, or another customer who needs support.
  • the voice discriminating unit 24 discriminates whether or not a specific keyword is included in the voice detected by the voice detecting unit 21 by a predetermined voice recognition process.
  • the specific keyword includes, for example, a word (including explicit or suggestion) indicating that customer service by the salesclerk SA is required. Specific examples of specific keywords will be described later.
  • the customer service condition setting unit 41A sets the customer service condition based on the input information received via the operation unit 14, for example.
  • the customer service conditions include, for example, selection of an area (grid GR) for determining customer service necessity, operation related to customer service necessity determination (for example, operation related to customer service necessity determination shown in FIG. 6 described later), customer service required.
  • the setting of various threshold values used in the negative determination is included.
  • the customer service condition here defines, for example, in which area the store clerk SA and the customer CT determine what kind of behavior the customer service is necessary.
  • the behavior determination unit 43 determines whether or not the customer CT located in the store ST has performed a specific behavior.
  • the specific behavior includes, for example, behavior indicating that the customer CT needs customer service by the store clerk SA (including explicit or suggestion). Specific examples of specific actions will be described later.
  • the customer service necessity determination unit 44 determines the necessity of customer service at each position (each grid GR) in the store ST. For example, when the customer CT located in the store ST issues a specific keyword or when the customer CT located in the store ST performs a specific action, the customer service necessity determination unit 44 needs to serve the customer. Is determined. Further, for example, the customer service necessity determination unit 44 determines that the customer service is unnecessary when the customer CT located in the store ST does not issue a specific keyword or does not perform a specific action.
  • the terminal notification unit 45 transmits, through the communication unit 11, customer service instruction information indicating that customer service is required to all or a part of the mobile terminal TE possessed by the store clerk SA.
  • the customer service instruction information includes, for example, information on the location of the customer CT that needs customer service and information that instructs the customer CT.
  • the information on the location of the customer CT may be, for example, the location information of the grid GR including the location of the customer CT, or the location information of the divided area DA including the location of the customer CT.
  • the information on the location of the customer CT may be intuitively shown using a map indicating the inside of the store ST.
  • FIG. 6 is a flowchart showing an operation example of the customer service monitoring device 10A.
  • the customer service condition setting unit 41 sets a customer service condition (customer service necessity determination condition) based on, for example, input information obtained through the operation unit 14 (S21).
  • the customer service conditions here include, for example, the processes of S23 to S31.
  • the moving body staying information generating unit 31 acquires video data from the camera device CA and generates staying information of the moving body (S22). Further, the ID staying information generating unit 32 acquires information on the position of each clerk SA from the mobile terminal TE identified by the mobile ID, and generates ID staying information (S22).
  • the staying information and ID staying information of the mobile object may be real-time information or past information. Note that the generation of the stay information of the moving body and the generation of the ID stay information may be performed constantly or periodically. That is, the process of S22 is the same as the process of S12.
  • the customer service condition setting unit 41A selects, for example, the grid GR to be analyzed (service request necessity determination target) in the store ST based on the input information obtained through the operation unit 14 (S23).
  • the customer service necessity determination unit 44 determines whether or not the customer CT is alone in the selected grid GR (S24). For example, the customer service necessity determination unit 44 can extract the stay information of the customer CT by subtracting the ID stay information from the stay information of the moving body (person), and can determine whether or not the customer CT is alone. .
  • the voice processing unit 20A acquires voice data from the microphone array device MA, the recorder RC, or the memory 13 via the communication unit 11 (S25). ).
  • the directivity synthesis unit 23 forms voice directivity in the direction in which the customer CT detected in S24 in the selected grid GR is located (S25).
  • the voice discriminating unit 24 determines whether or not a specific keyword (target keyword) issued by the customer CT in the selected grid GR has been detected (S26).
  • specific keywords include, for example, “I ’m sorry”, “Is there someone else”, “Care clerk”, “Well, what to do”, “I don't know”, “I don't know”, “ Words pronounced of the need for customer service, such as “Do n’t get lost” and “Keep me here”.
  • the specific keyword is not limited to these words.
  • the customer service determination unit 40A displays the flow line information or staying information of the customer CT in the selected grid GR as the person detection unit 30 or Read (acquire) from the memory 13 (S27).
  • the behavior determination unit 43 determines whether or not the customer CT has performed a specific behavior based on the flow line information or the stay information of the customer CT in the selected grid GR (S28). A specific example of the specific action will be described later.
  • the customer service necessity determination unit 44 determines that customer service is not required for this customer CT (S29).
  • the customer service necessity determination unit 44 needs to serve the customer CT. It is determined that there is (S30).
  • the terminal notification unit 45 transmits customer service instruction information to all or a part of the mobile terminal TE possessed by the store clerk SA via the communication unit 11 (S30). A specific example of the clerk SA that is the destination of the customer service instruction information will be described later.
  • the customer service determination unit 40A It is determined whether or not the determination of the necessity of customer service has been completed in all grids GR (S30A).
  • the customer service monitoring apparatus 10A ends the process of FIG.
  • the process proceeds to S23, and the customer service determination unit 40A selects the next grid GR as the analysis target grid.
  • the customer service monitoring apparatus 10A determines whether or not the customer needs to be served when the customer CT is alone and a specific keyword or a specific action is detected. Therefore, even if the customer CT happens to be alone in the store ST, issues a specific keyword, or performs a specific action, it is not determined that the customer service is necessary, so the accuracy of determining whether the customer service is necessary is improved. it can.
  • FIG. 7 is a schematic diagram for explaining a specific action performed by the customer CT.
  • six commodity shelves SS are installed in the store ST.
  • the area in the vicinity of the product shelf SS is assumed to be a product tour area A1 for the customer CT to tour the products on the product shelf SS.
  • the passage for moving the product shelf SS to be visited and the passage on the outer periphery in the store ST are assumed to be the store clerk search area A2 for the customer CT to search for the store clerk SA.
  • the product tour area A1 is an area where the customer CT may stay in order to tour the product.
  • the clerk search area A2 is an area where the customer CT may stay in order to search for the clerk SA.
  • the customer service determination unit 40A refers to the stay information of the customer CT, the stay time in the product tour area A1 is a predetermined time t11 (for example, 1 second) or more, then moves to the clerk search area A2, and then the product tour area again.
  • t11 for example, 1 second
  • the customer CT went to look for the clerk SA because the height of the merchandise shelf SS was high, but gave up without being found.
  • the customer service determination unit 40A May be determined to have performed a specific action. In this behavior, it can be considered that the customer CT has been thinking about the product displayed on the product shelf SS for a long time.
  • the customer service determination unit 40A determines that the customer CT has performed a specific action. May be. In this behavior, it is considered that the customer CT moves to the clerk search area A2 and searches for the clerk SA.
  • a predetermined time t13 for example, 600 seconds (10 minutes)
  • the customer service determination unit 40A When the customer service determination unit 40A is searching for a clerk SA or a product beyond the product shelf SS in the product tour area A1 (that is, the customer CT looks into the product shelf SS and extends, for example), the customer CT May be determined to have performed a specific action. In this case, the service determination unit 40A determines whether or not the position of the head HD1 of the customer CT is higher than the average position HDave in the vertical direction while staying in the product tour area A1 by a predetermined threshold th1 or more. To do. This determination can be made by, for example, image recognition processing or feature extraction processing on video data.
  • FIG. 8A is a schematic diagram for explaining an example of whether or not the customer CT has looked into from the upper side of the commodity shelf SS (looking-in determination).
  • the position of the head HD of the customer CT is at the position of the head HD1 by looking into the top of the product shelf SS, and is above the threshold th1 by the head th average position HDave.
  • the customer service determination unit 40A determines that the customer CT is looking in, and determines that the customer CT is performing a specific action.
  • the customer service determination unit 40A may determine that the customer CT has performed a specific action when the customer CT is in the product tour area A1 or the clerk search area A2. In this case, the service determination unit 40A determines that the position of the head HD or the shoulder SD of the customer CT is the left-right direction (a direction rotated 90 ° counterclockwise or right) as viewed from the top of the head (above the head HD). It is determined whether or not it has been moved a plurality of times at a certain angle (for example, a direction rotated at an arbitrary angle other than 90 ° counterclockwise or clockwise). This determination can be made by, for example, image recognition processing or feature extraction processing on video data.
  • FIG. 8B is a schematic diagram for explaining an example of determining whether or not the customer CT is in a dark state (a black and white determination).
  • the head HD is rotated counterclockwise at an angle ⁇ and rotated a plurality of times.
  • the customer service determination unit 40A determines that the customer is afraid and determines that the user is performing a specific action.
  • the fact that the number of rotations of the head HD or the shoulder SD is equal to or greater than a predetermined number may be added to a condition that satisfies the mentality determination.
  • the customer service determination unit 40A may determine that the customer CT has performed a specific action when the customer CT leans his / her head in the product tour area A1 or the clerk search area A2. In this case, the service determination unit 40A determines that the position of the head HD or shoulder SD of the customer CT is in the left-right direction (counterclockwise or clockwise) when viewed from the front direction of the head HD (that is, the front direction of the face). It is determined whether or not it is tilted at a predetermined angle), for a predetermined time t21 (for example, 5 seconds) or more or for a predetermined number of times (for example, 3 times), that is, whether the neck is tilted. This determination can be made by, for example, image recognition processing or feature extraction processing on video data.
  • FIG. 8C is a schematic diagram for explaining an example of whether or not the head is tilted (tilting determination).
  • the head HD of the customer CT is tilted leftward at an angle ⁇ 2 a predetermined number of times t22 or more.
  • the customer service determination unit 40A determines that the head is curled and determines that a specific action is being performed.
  • the customer service determination unit 40A may determine that the customer CT has performed a specific action when the customer CT has a different product in the product tour area A1 or the clerk search area A2. In this case, for example, the customer service determination unit 40A detects the area of the arm or hand of the customer CT from the video data, and determines whether or not an object (product) exists in the area around both of the areas. When the customer CT has a plurality of products, it may be possible to compare both products and want to know the characteristics of each product.
  • FIG. 9 and FIG. 10 are schematic diagrams for explaining the store clerk SA that is the transmission destination of the customer service instruction information.
  • the terminal notification unit 45 may transmit customer service instruction information with priority given to a single clerk SA (for example, clerk SA11).
  • the person detection unit 30 may detect that there is one clerk SA in the selected grid GR and that it is a single clerk SA when there is one ID residence information in the selected grid GR. Good.
  • the terminal notifying unit 45 excludes that the store clerk SA is otherwise incapable of serving customers, from the destination of the customer service instruction information. May be. In this case, the customer service instruction information is not transmitted to the portable terminal TE possessed by the clerk SA.
  • the person detection unit 30 may determine whether or not the customer CT is located near the store clerk SA, for example, by the same method as S15 in FIG. That is, in the selected grid GR, the person detection unit 30 detects a plurality of moving bodies in the staying information of the moving bodies, and the ID staying information has less than the number of moving bodies in the staying information of the moving bodies, and 1 When two or more ID moving bodies are detected, it may be determined that the customer CT is located near the clerk SA.
  • the terminal notification unit 45 transmits customer service instruction information to the mobile terminal TE owned by the clerk SA located in the backyard outside the store ST. Also good.
  • the person detection unit 30 analyzes each grid GR, and when there is no ID residence information in each grid GR, or even if there is ID residence information in each grid GR, the customer CT is located near the clerk SA. In this case, it may be detected that there is no single clerk SA.
  • the terminal notification unit 45 may transmit the customer service instruction information to the mobile terminal TE identified by the mobile ID for which the ID staying information is not detected by the person detection unit 30. In this case, the mobile ID information is held in advance in the memory 13 for each mobile terminal TE possessed by the store clerk SA located inside and outside the store ST.
  • the terminal notification unit 45 provides customer service instruction information to all the mobile terminals TE owned by the clerk SA (all clerk) located inside and outside the store ST. May be sent. In this case, the customer service instruction information is notified to all the clerk SA, so that the clerk SA can cooperate with each other as much as possible.
  • the person detection unit 30 detects one first customer CT that stays for a predetermined time t1 or longer in an arbitrary grid GR.
  • the customer service determination unit 40A needs to serve the first customer CT. Is determined.
  • the communication unit 11 transmits customer service instruction information for instructing customer service for the first customer CT to the mobile terminal TE possessed by at least a part of the store clerk SA when customer service for the first customer CT is required.
  • the customer service monitoring device 10A detects a mobile body (that is, the first customer CT) that is not the store clerk SA in an arbitrary grid GR, and determines whether the first customer CT needs to support the store clerk SA. Inferred from the word issued by the customer CT and the behavior of the first customer CT. Then, when the first customer CT needs the support of the store clerk SA, the customer service monitoring apparatus 10A notifies the mobile terminal TE held by the store clerk SA to that effect.
  • a mobile body that is, the first customer CT
  • the customer service monitoring apparatus 10A notifies the mobile terminal TE held by the store clerk SA to that effect.
  • the customer service monitoring system 100A and the customer service monitoring device 10A utilize the ward issued by the first customer CT that requires customer service and the behavior of the first customer CT to determine whether or not the customer needs to be served. Loss can be reduced. Further, regardless of whether or not the first customer CT directly talks to the clerk SA, the clerk SA can recognize that the customer service is required, and the satisfaction of the first customer CT can be improved.
  • the service determination unit 40A performs a specific action based on the detection position of the head HD or shoulder SD of the first customer CT and the detection time or the number of detections of the head HD or shoulder SD. It may be determined whether or not.
  • the customer service monitoring system 100A and the customer service monitoring device 10A can easily determine whether or not a specific action is being performed by detecting the movement of each part of the body of the first customer CT. It can be used for judgment.
  • the customer service determination unit 40A may determine that a specific action is being performed when the stay information of the first customer CT includes a predetermined stay pattern (for example, first to third stay patterns).
  • the customer service monitoring system 100A and the customer service monitoring device 10A can easily determine whether or not a specific action is being performed by detecting a staying pattern based on the trajectory of the movement of the first customer CT. It can be used to determine necessity.
  • the communication unit 11 may transmit the customer service instruction information to the clerk SA located by one person in an arbitrary grid based on the stay information of the clerk SA.
  • the clerk SA located by himself / herself can check the customer service instruction information and can quickly serve the customer 1 for the first customer CT.
  • the communication unit 11 may transmit customer service instruction information to a non-customer (for example, a customer clerk SA who is not in customer service) based on the customer clerk SA residence information and person residence information.
  • a non-customer for example, a customer clerk SA who is not in customer service
  • the clerk SA who has confirmed the customer service instruction information is not in service, and can quickly serve the first customer CT.
  • the communication unit 11 transmits the customer service instruction information to the mobile terminal TE possessed by the clerk SA for which no stay information of the clerk SA has been generated. Also good.
  • the store clerk SA who is not in the store ST and has not generated the stay information and is waiting in the backyard confirms the customer service instruction information with the mobile terminal TE, and quickly becomes the first customer CT. You can serve customers.
  • the communication unit 11 may transmit the customer service instruction information to the mobile terminals TE possessed by all the clerk SA when there is no clerk SA who is alone in any grid GR.
  • each clerk SA confirms the customer service instruction information received by the mobile terminal TE, and each clerk SA consults with each other. For example, according to the current customer service situation, any clerk SA serves the first customer CT. can do.
  • the customer service monitoring device 10A determines the behavior of the customer CT who is about to purchase the product but is expected to be confused, the behavior of the customer CT that is expected to explain the product, and the clerk SA.
  • the behavior of the customer CT expected to be searched is regarded as the behavior that the customer needs, and the customer clerk SA is notified of the customer service instruction.
  • the customer service instruction information can be notified to the clerk SA who is expected to be able to serve customers.
  • the notified clerk SA can recognize the necessity of customer service for the customer CT.
  • the customer service monitoring device 10A can promptly notify the appropriate store clerk SA of the necessity of customer service, and can suppress customer service loss.
  • the customer service monitoring device 10A is based on person detection and flow line detection based on an image captured by the camera device CA, position detection of the clerk SA using radio, sound detection based on sound collected by the microphone array device MA, and the like.
  • the customer CT that requires the store clerk SA can be detected, and an alert can be output to the store clerk SA and other monitoring personnel that the customer service is required.
  • the customer service monitoring device 10A can suppress the loss of the customer service opportunity because the store clerk SA cannot recognize the customer service requirement even though the customer CT needs the customer service.
  • the customer service monitoring device 10A can analyze the behavior of the customer CT without forcing the customer CT, and can realize customer service with higher accuracy by associating the customer CT with the clerk SA.
  • the customer service monitoring device in the customer service monitoring system uses voice recognition to determine whether or not the person is the same person when the person's flow line indicated by the person's flow line information is interrupted by an obstacle or the like. Explain that Judgment.
  • FIG. 11 is a block diagram illustrating a configuration example of the customer service monitoring system 100B according to the third embodiment. 11 is different from the customer monitoring system 100 shown in FIG. 1 in that a customer monitoring device 10B is provided instead of the customer monitoring device 10.
  • the same components as those of the customer service monitoring system 100 shown in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted or simplified.
  • the customer service monitoring device 10B includes a voice processing unit 20B, a person detection unit 30B, and a customer service determination unit 40.
  • the voice processing unit 20B includes a voice detection unit 21, a detection result accumulation unit 22, and a speaker identification unit 25.
  • the person detection unit 30 ⁇ / b> B includes a moving body residence information generation unit 31, an ID residence information generation unit 32, and a residence information adjustment unit 33.
  • the speaker identification unit 25 identifies a speaker that utters the voice detected by the voice detection unit 21 by a predetermined speaker recognition process (for example, voiceprint identification).
  • the staying information adjusting unit 33 integrates staying information determined to be the same person by speaker identification or the like, and stores the staying information in the memory 13. For example, if it is determined that the same person as the person who has been generated and retained in the memory 13 has appeared again in the store ST, the staying information adjustment unit 33 continues the staying information ( As part of the information, information related to this person's stay is accumulated.
  • FIG. 12 is a flowchart showing an operation example of the customer service monitoring device 10B.
  • the customer service monitoring device 10B executes the processing of S11 to S14 shown in FIG.
  • the customer service behavior extraction unit 42 determines whether or not the store clerk SA or the customer CT is located in the grid GR to be analyzed (S31). For example, the customer service behavior extraction unit 42 can determine that the store clerk SA or the customer CT is located when the moving object is detected in the staying information of the moving object.
  • the speaker identification unit 25 acquires speaker identification information for identifying the speaker by speaker authentication processing (S32). ). For example, the speaker identification unit 25 acquires information on a spectrum of a speech waveform related to a salesclerk SA or a customer CT as a speaker as speaker identification information.
  • the speaker identification unit 25 determines whether or not speaker identification of this clerk SA or customer CT has been performed before (S33). Whether or not it has been performed before can be determined by whether or not the same speaker identification information is held in the memory 13.
  • the same here does not need to be completely the same, and when it exceeds a certain likelihood, for example, when the correlation value or similarity between the speech spectra as speaker identification information is a predetermined value or more May be included identically.
  • the position in the store ST where the speaker identification information was obtained in the past may be an arbitrary position, and may not be limited to the same grid as the grid GR selected in S14.
  • the speaker identification unit 25 Holds (registers) the acquired speaker identification information in the memory 13 (S34). By registering the speaker identification information in the memory 13 for the first time, speaker identification can be performed using the speaker identification information for the second and subsequent times.
  • the staying information adjustment unit 33 When speaker identification of this clerk SA or customer CT has been performed before, that is, when the same speaker identification information as the speaker identification information acquired in S32 is held in the memory 13, the staying information adjustment unit 33 The staying information related to the person with the same speaker identification information is integrated (merged) and held in the memory 13 as staying information of the same person (S35).
  • the stay information adjusting unit 33 integrates the stay information of the mobile body related to the store clerk SA, and further integrates the ID stay information related to the store clerk SA.
  • the staying information adjustment unit 33 integrates the staying information of the moving body related to the customer CT.
  • the customer service determination unit 40 performs the analysis in all the grids GR to be analyzed. It is determined whether the determination of the identity of the staying information has been completed (S36). When the determination of the identity of the staying information in all grids GR is completed (Yes in S36), the customer service monitoring device 10B ends the process of FIG. When the determination of the identity of the stay information in at least one grid GR has not been completed (No in S36), the process proceeds to S14, and the customer service determination unit 40 analyzes the next grid GR (the identity of the stay information). Select as the judgment target grid.
  • the service monitoring apparatus 10 ⁇ / b> B corresponds to the person staying information related to the speaker identification information when the speaker identification information related to the newly appearing person has existed in the past.
  • the existing staying information is integrated and stored in the memory 13.
  • the customer service monitoring device 10B uses the speaker identification processing even when the flow line information or staying information of the person is interrupted, for example, when the person moves through a shield such as the product shelf SS. Thus, it is possible to determine the identity of the sound produced by each person.
  • the customer service monitoring device 10B can be used for customer service activities as flow line information and staying information of the same person when a plurality of persons are identified as being the same person by exceeding the certain likelihood by the identification process using voice.
  • FIG. 13 is a schematic diagram for explaining the identity of the staying information.
  • the first person P1 is hidden behind the product shelf SS21 from the left side of the product shelf SS21, and the second person P2 appears from the right side of the product shelf SS21.
  • the flow line information and the staying information relating to the first person P1 and the second person P2 are derived by the person detection unit 30, the flow line is blocked by the product shelf SS21 as a shield. Therefore, the flow line information and the staying information are obtained separately for each of the first person P1 and the second person P2.
  • the voice processing unit 20B obtains voice data of the first person P1 and the second person P2, and performs voice identification processing on the voice data of the both persons, thereby generating voice generated by the same person. Or not.
  • the customer service monitoring apparatus 10B can handle the flow line information and the staying information as the same person information in an integrated manner even if the flow line information and the staying information of a plurality of persons are divided by the shield. Therefore, for example, when a customer CT appears from the back of the product shelf SS21 instead of a new customer, it can be linked to whether or not the customer is a customer from past residence information, and the customer service activities are made more efficient it can.
  • the speaker identification unit 25 identifies the first speaker related to the utterance detected by the voice detection unit 21 by the speaker identification process.
  • the first speaker identification information related to the first speaker is stored in the memory 13.
  • the staying information adjustment unit 33 includes first speaker identification information, speaker identification information related to the second speaker identified after the first speaker identification information is obtained by the speaker identification unit 25, and Are substantially the same, the retention information of the person related to the second speaker or the retention information of the clerk SA is integrated into the retention information of the person related to the first speaker or the retention information of the clerk SA.
  • the service monitoring system 100B and the service monitoring device 10B are provided separately if the flow line information and the stay information are once blocked and the stay information can be linked by speaker identification even if another person is detected. Can be handled as flow line information and staying information of the same person.
  • the customer service monitoring system 100B and the customer service monitoring device 10B determine whether or not they are the same person using the audio data, so compared with the case where it is determined whether or not they are the same person using face matching or the like using the image data. , Processing load can be reduced. Therefore, since the customer service monitoring system 100B and the customer service monitoring device 10B can improve the person identification accuracy related to the flow line information and the stay information, the customer service determination accuracy can be improved.
  • the processor may be physically configured in any manner. Further, if a programmable processor is used, the processing contents can be changed by changing the program, so that the degree of freedom in designing the processor can be increased.
  • the processor may be composed of one semiconductor chip or physically composed of a plurality of semiconductor chips. When configured by a plurality of semiconductor chips, each control of the first to third embodiments may be realized by different semiconductor chips. In this case, it can be considered that a plurality of semiconductor chips constitute one processor.
  • the processor may be configured by a member (capacitor or the like) having a function different from that of the semiconductor chip. Further, one semiconductor chip may be configured so as to realize the functions of the processor and other functions.
  • a plurality of processors may be configured by one processor.
  • each configuration may be realized by hardware or software. Also good.
  • This disclosure is useful for a customer service monitoring device, a customer service monitoring system, and a customer service monitoring method that can improve the accuracy of customer service determination.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Finance (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Accounting & Taxation (AREA)
  • Economics (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Computational Linguistics (AREA)
  • Development Economics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Image Analysis (AREA)
  • Alarm Systems (AREA)

Abstract

Un dispositif de surveillance de la présence d'un client comprend : une unité de communication pour acquérir des données vocales concernant une parole saisie par un dispositif de prise de son qui capte la parole dans une zone cible divisée en une pluralité de grilles, et l'acquisition de données d'image relatives à une image capturée par un dispositif d'imagerie qui forme une image de la zone cible; une unité de génération d'informations de séjour destinée à générer des informations de séjour concernant des personnes comprenant un accompagnateur ou un client et à générer des informations de séjour concernant l'accompagnateur; et une unité de détermination de présence de client qui, lorsque l'accompagnateur et le client restent pendant un premier temps prescrit dans le système la même grille et un énoncé d'une durée d'un second temps prescrit ou plus est détecté dans la même grille, déterminer, sur la base des informations de séjour concernant les personnes, les informations de séjour concernant l'accompagnateur, et un résultat de détection d'énoncé, que l'accompagnateur participe au client dans ladite même grille.
PCT/JP2017/020901 2016-08-02 2017-06-06 Dispositif de surveillance de la presence d'un client, système et procédé de surveillance de service à la clientèle Ceased WO2018025492A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2016152135A JP2018022284A (ja) 2016-08-02 2016-08-02 接客モニタリング装置、接客モニタリングシステム、及び接客モニタリング方法
JP2016-152135 2016-08-02

Publications (1)

Publication Number Publication Date
WO2018025492A1 true WO2018025492A1 (fr) 2018-02-08

Family

ID=61073861

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2017/020901 Ceased WO2018025492A1 (fr) 2016-08-02 2017-06-06 Dispositif de surveillance de la presence d'un client, système et procédé de surveillance de service à la clientèle

Country Status (2)

Country Link
JP (1) JP2018022284A (fr)
WO (1) WO2018025492A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021536046A (ja) * 2019-08-14 2021-12-23 北京市商▲湯▼科技▲開▼▲發▼有限公司Beijing Sensetime Technology Development Co., Ltd. データ処理方法、装置、及び記憶媒体

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7336747B2 (ja) * 2018-07-05 2023-09-01 パナソニックＩｐマネジメント株式会社 販売促進システムおよび販売促進方法
JP7366547B2 (ja) * 2019-01-10 2023-10-23 東芝テック株式会社 情報処理装置及び情報処理プログラム
JP7336674B2 (ja) * 2019-02-07 2023-09-01 パナソニックＩｐマネジメント株式会社 接客状況分析装置、接客状況分析システム、及び接客状況分析方法
WO2021038814A1 (fr) * 2019-08-29 2021-03-04 株式会社オプティム Système informatique, procédé d'amélioration de situation spécifique, et programme
JP6954687B1 (ja) * 2020-06-17 2021-10-27 Ｎｅｃプラットフォームズ株式会社 接客管理システム、接客管理方法、管理サーバ及びプログラム
JP7618990B2 (ja) * 2020-09-23 2025-01-22 株式会社Ｊｖｃケンウッド 画像処理装置、及び画像処理プログラム
CN116157833A (zh) * 2020-09-23 2023-05-23 Jvc建伍株式会社 图像处理装置及图像处理程序
JP7540307B2 (ja) * 2020-09-23 2024-08-27 株式会社Ｊｖｃケンウッド 画像処理装置、及び画像処理プログラム
JP2023086466A (ja) * 2021-12-10 2023-06-22 コニカミノルタ株式会社 対応要否判定装置、対応要否判定プログラム
JP7315049B1 (ja) 2022-02-22 2023-07-26 富士通株式会社 情報処理プログラム、情報処理方法、および情報処理装置
WO2024184987A1 (fr) * 2023-03-06 2024-09-12 日本電気株式会社 Système d'aide au stockage, procédé d'aide au stockage et support d'enregistrement
CN119444154B (zh) * 2025-01-07 2025-09-19 深圳市妙想互联科技有限公司 一种基于智慧园区的员工考勤数据管理系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07287781A (ja) * 1994-04-19 1995-10-31 Hitachi Ltd 移動体の識別・追跡装置
JP2009238044A (ja) * 2008-03-27 2009-10-15 Brother Ind Ltd 接客支援システム
JP2011237966A (ja) * 2010-05-10 2011-11-24 Seiko Epson Corp 接客支援装置、接客支援方法およびプログラム
JP5879616B1 (ja) * 2014-10-07 2016-03-08 パナソニックＩｐマネジメント株式会社 活動状況分析システム、活動状況分析装置、活動状況分析方法、活動状況分析プログラムおよびそのプログラムを格納する記憶媒体

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07287781A (ja) * 1994-04-19 1995-10-31 Hitachi Ltd 移動体の識別・追跡装置
JP2009238044A (ja) * 2008-03-27 2009-10-15 Brother Ind Ltd 接客支援システム
JP2011237966A (ja) * 2010-05-10 2011-11-24 Seiko Epson Corp 接客支援装置、接客支援方法およびプログラム
JP5879616B1 (ja) * 2014-10-07 2016-03-08 パナソニックＩｐマネジメント株式会社 活動状況分析システム、活動状況分析装置、活動状況分析方法、活動状況分析プログラムおよびそのプログラムを格納する記憶媒体

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021536046A (ja) * 2019-08-14 2021-12-23 北京市商▲湯▼科技▲開▼▲發▼有限公司Beijing Sensetime Technology Development Co., Ltd. データ処理方法、装置、及び記憶媒体
JP7082688B2 (ja) 2019-08-14 2022-06-08 北京市商▲湯▼科技▲開▼▲發▼有限公司 データ処理方法、装置、及び記憶媒体

Also Published As

Publication number Publication date
JP2018022284A (ja) 2018-02-08

Similar Documents

Publication Publication Date Title
WO2018025492A1 (fr) Dispositif de surveillance de la presence d'un client, système et procédé de surveillance de service à la clientèle
JP6135880B2 (ja) 音声処理方法、音声処理システム、及び記憶媒体
US10182280B2 (en) Sound processing apparatus, sound processing system and sound processing method
JP5495235B2 (ja) 監視対象者の行動を監視する装置及び方法
JP6674584B2 (ja) 映像監視システム
US20160241818A1 (en) Automatic alerts for video surveillance systems
US20150281832A1 (en) Sound processing apparatus, sound processing system and sound processing method
US20180349710A1 (en) Method and arrangement for receiving data about site traffic derived from imaging processing
US20170330208A1 (en) Customer service monitoring device, customer service monitoring system, and customer service monitoring method
US9198225B2 (en) Ad-hoc surveillance network
WO2017029718A1 (fr) Système, procédé et programme numériques à sécurité intelligente
EP3121799A1 (fr) Système, procédé et programme de sécurité antivol numérique
JP2014155159A (ja) 情報処理システム、情報処理方法及びプログラム
US20170262725A1 (en) Method and arrangement for receiving data about site traffic derived from imaging processing
CN110291516A (zh) 信息处理设备、信息处理方法和程序
JPWO2015151130A1 (ja) 音声処理方法、音声処理システム、及び記憶媒体
JP7138547B2 (ja) 店舗装置
JP2015228626A (ja) 画像出力装置、画像管理システム、画像処理方法及びプログラム
JPWO2018143341A1 (ja) 監視システム、監視方法及びプログラム
JP7729455B2 (ja) 監視システム、監視装置、監視方法、およびプログラム
JP7256082B2 (ja) 監視システム、プログラム、およびリスト作成方法
JP7316580B2 (ja) 接客状況分析装置、接客状況分析システム、及び接客状況分析方法
JP2017215735A (ja) 振り込め詐欺防止システム及び振り込め詐欺防止方法
JP2023020307A (ja) 情報処理装置、情報処理方法及びプログラム
EP2927885A1 (fr) Appareil de traitement du son, système de traitement sonore et procédé de traitement sonore

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17836599

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17836599

Country of ref document: EP

Kind code of ref document: A1