JP2024079370A - Information processing device, information processing system, device management method and program - Google Patents

Abstract

A second electronic device is operated using information used in a first electronic device.
[Solution] The information processing device comprises a first image receiving unit that receives a first image taken of a first user using a first electronic device, a first user identification unit that identifies the first user based on usage information of the first electronic device and the first image, an input information receiving unit that receives from the first electronic device authentication information entered by the first user on the first electronic device, a second image receiving unit that receives a second image taken of a second user present in an operable position of the second electronic device, a second user identification unit that identifies the second user from the second image, and an operation instruction unit that instructs the second electronic device to perform an operation using the authentication information or screen setting information in the authentication information when predetermined conditions are met based on information on the first and second users and the authentication information.
[Selected Figure] Figure 6

Description

This invention relates to an information processing device, an information processing system, a device management method, and a program.

In cloud computing, there is technology that realizes single sign-on by linking software. For example, Patent Document 1 discloses an invention that uses a common authentication infrastructure across multiple services, allowing a user authenticated in one service to omit authentication in other services.

In authentication using hardware such as electronic devices, there is a technology that links the authentication process of each piece of hardware. For example, Patent Document 2 discloses an invention that links a system that manages entry and exit to an area where confidential information is handled with electronic devices in the area, thereby not allowing authentication on the electronic devices unless the user has been authenticated by the entry and exit management system.

It is difficult to determine the identity of a user when linking hardware devices. The invention disclosed in Patent Document 2 uses the authentication status of an entrance/exit management system for authentication on electronic devices, but each user must perform authentication operations separately.

One embodiment of the present invention aims to operate a second electronic device using information used in a first electronic device.

An information processing device according to an embodiment of the present invention is an information processing device capable of communicating with a first electronic device, a second electronic device, and a photographing device via a network, and includes a first image receiving unit that receives a first image of a first user using the first electronic device from the photographing device, a first user identification unit that identifies the first user using the first electronic device based on usage information of the first electronic device and the first image, an input information receiving unit that receives from the first electronic device authentication information input by the first user identified by the first user identification unit on the first electronic device, a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the photographing device, a second user identification unit that identifies the second user present at an operable position of the second electronic device from the second image, and an operation instruction unit that instructs the second electronic device to perform an operation using the authentication information or screen setting information in the authentication information when a predetermined condition is met based on the information of the first user identified by the first user identification unit and the second user identified by the second user identification unit and the authentication information.

According to one embodiment of the present invention, a second electronic device can be operated using information used in a first electronic device.

1 is a diagram illustrating an example of an overall configuration of a device management system according to an embodiment. FIG. 2 illustrates an example of a hardware configuration of a computer according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the omnidirectional imaging device according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of an MFP according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a self-service terminal/electronic whiteboard according to an embodiment. FIG. 2 illustrates an example of a functional configuration of a device management system according to an embodiment. FIG. 11 is a diagram illustrating an example of state management information according to an embodiment. FIG. 11 is a diagram illustrating an example of a first basic flowchart according to an embodiment. FIG. 11 is a diagram illustrating an example of a second basic flowchart according to an embodiment. FIG. 1 illustrates an example of a device management method according to an embodiment. FIG. 13 is a diagram illustrating an example of an object state update process according to an embodiment. FIG. 11 is a diagram illustrating an example of a detected object list creation process according to an embodiment. FIG. 11 is a diagram illustrating an example of a registered object list update process according to an embodiment. FIG. 11 is a diagram illustrating an example of a registered object list update process according to an embodiment. FIG. 11 is a diagram illustrating a first example of a group ID assignment process according to an embodiment. FIG. 11 is a diagram illustrating a second example of a group ID assignment process according to an embodiment. FIG. 11 illustrates an example of a device status update process according to an embodiment. FIG. 11 is a diagram illustrating an example of an output information determination process according to an embodiment. FIG. 11 is a diagram illustrating an example of an output condition determination process according to an embodiment. FIG. 11 is a diagram illustrating a first example of device output information according to an embodiment. FIG. 11 is a diagram illustrating a second example of device output information according to an embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings. Note that components having the same functions in the drawings are given the same numbers, and duplicate explanations will be omitted.

[Embodiment]
An embodiment of the present invention is a device management system that manages a plurality of electronic devices. The device management system in this embodiment realizes cooperation between electronic devices by instructing another electronic device to perform an operation using information inputted in one electronic device.

Conventionally, in hardware linkage, the identity of a user was determined by whether the user possessed the same device (for example, an integrated circuit (IC) card or a mobile information terminal, etc.). As a result, even when hardware linkage was performed, authentication operations were required for each piece of hardware. With single sign-on through software linkage, when a user authenticated in one service accesses another service, the operations are performed from the same device, making it easy to determine the identity of the user.

In the device management system of this embodiment, the management device tracks people moving between electronic devices based on images captured by the management device of a space in which multiple electronic devices are installed. This allows the management device to determine whether a user who operates an electronic device is the same as a user who is in the vicinity of another electronic device.

If the identity of a user can be determined, it is possible to have another electronic device execute an operation using information entered on one electronic device. For example, if an electronic device in the vicinity of the user performs authentication using authentication information entered by the user on another electronic device, single sign-on through hardware linkage can be realized. Even if authentication is not performed, if the identity of a person can be confirmed based on a captured image, device setting information and data can be linked between hardware used by the same person.

<Overall configuration of the equipment management system>
First, the overall configuration of the device management system in this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the overall configuration of the device management system in this embodiment.

As shown in FIG. 1, the device management system 1 in this embodiment includes a management device 10, one or more monitoring devices 20, and multiple electronic devices 30 (examples of a first electronic device and a second electronic device). The management device 10, the monitoring device 20, and the electronic devices 30 are each connected to a communication network N1.

The communication network N1 is configured so that the connected devices can communicate with each other. The communication network N1 is constructed by a wired communication network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

The communication network N1 may include not only wired communication, but also wireless communication such as wireless LAN or short-range wireless communication, or a network using mobile communication such as WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or 5G (5th Generation).

The monitoring device 20 and the electronic device 30 are installed in the managed space R1. The managed space R1 may be a single space such as a room in a building, or may be multiple spaces connected by accessible spaces such as doors or corridors.

An example of a managed space R1 is a sales floor or back yard in a small store such as a convenience store. Another example of a managed space R1 is a conference room or office. Another example of a managed space R1 is a hotel lobby or guest room. Another example of a managed space R1 is an airport, a bookstore, or a factory. The managed space R1 is not limited to these, and may be any space in which electronic devices that can be operated by multiple users are installed.

The management device 10 is an information processing device such as a PC (Personal Computer), a workstation, or a server that manages the electronic device 30. The management device 10 transmits operation instructions to the electronic device 30 based on images acquired by the monitoring device 20. An example of the management device 10 is a computer.

The monitoring device 20 is an electronic device that acquires images including the vicinity of the electronic devices 30 installed in the managed space R1. The monitoring device 20 may acquire video (i.e., a time series of images). The monitoring device 20 is installed in a position where it can capture images of all of the multiple electronic devices 30.

One example of the monitoring device 20 is a spherical imaging device. Another example of the monitoring device 20 is a plurality of network cameras. When the monitoring device 20 is a plurality of network cameras, each network camera is positioned with its angle of view adjusted so that no blind spots are created within the managed space R1.

Hereinafter, when there are multiple monitoring devices 20, they will be distinguished from one another by using sub-numbers such as "monitoring device 20-1," "monitoring device 20-2," etc.

The electronic device 30 is a variety of electronic devices used by users (examples of a first user and a second user). The electronic device 30 is configured to enable authentication using hardware such as an IC (Integrated Circuit) card.

One example of electronic device 30 is an image forming device (such as a printer, a fax machine, a digital multifunction machine (MFP (Multifunction Peripheral/Product/Printer)), a scanner device, etc.). Another example of electronic device 30 is a self-service terminal with a charge settlement function, or an electronic whiteboard (IWB (Interactive White Board: an electronic whiteboard with a whiteboard function that allows intercommunication)).

Hereinafter, when distinguishing between multiple electronic devices 30, they will be referred to using sub-numbers such as "electronic device 30-1," "electronic device 30-2," etc.

The electronic device 30 is not limited to an image forming device, a self-service terminal, or an electronic whiteboard, so long as it is a device equipped with a communication function. That is, the electronic device 30 may be, for example, an output device such as a PJ (Projector), digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, a medical device, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, or a desktop PC.

<Hardware configuration of the device management system>
Next, the hardware configuration of each device included in the device management system in this embodiment will be described with reference to FIGS.

<Computer hardware configuration>
FIG. 2 is a diagram illustrating an example of a hardware configuration in which the management device 10 is realized by a computer.

As shown in FIG. 2, the computer in one embodiment includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I/F (Interface) 508, a network I/F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I/F 516.

Of these, the CPU 501 controls the operation of the entire computer. The ROM 502 stores programs used to drive the CPU 501, such as the IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501.

The display 506 displays various information such as a cursor, a menu, a window, characters, or an image. The external device connection I/F 508 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I/F 509 is an interface for data communication using the communication network N1. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 2.

The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, etc. The DVD-RW drive 514 controls the reading and writing of various data from the DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I/F 516 controls the reading and writing (storing) of data from the recording medium 515, such as a flash memory.

<Hardware configuration of the spherical imaging device>
3 is a diagram showing an example of a hardware configuration in the case where the monitoring device 20 is realized by an omnidirectional imaging device. In the following, the omnidirectional imaging device is assumed to be an omnidirectional (all-directional) imaging device using two imaging elements, but any number of imaging elements may be used as long as it is two or more. In addition, the omnidirectional imaging device does not necessarily have to be a device dedicated to omnidirectional imaging, and a normal digital camera, a smartphone, or the like may be provided with an omnidirectional imaging unit as a retrofit to have substantially the same functions as the omnidirectional imaging device.

As shown in FIG. 3, the omnidirectional imaging device in one embodiment is composed of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I/F 616, a communication unit 617, an antenna 617a, an acceleration/direction sensor 618, a gyro sensor 619, an acceleration sensor 620, and a concave terminal 621 for a Micro USB.

The imaging unit 601 includes wide-angle lenses (so-called fisheye lenses) 602a and 602b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 603a and 603b provided corresponding to each wide-angle lens. The imaging elements 603a and 603b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the fisheye lenses 602a and 602b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the imaging elements are set.

The imaging elements 603a and 603b of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I/F bus. On the other hand, the imaging elements 603a and 603b of the imaging unit 601 are connected to the imaging control unit 605 via a serial I/F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via a bus 610. In addition, the bus 610 is also connected to a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I/F (Interface) 616, a communication unit 617, and an acceleration/direction sensor 618.

The image processing unit 604 takes in the image data output from the image sensors 603a and 603b via a parallel I/F bus, performs a predetermined process on each piece of image data, and then synthesizes the image data to create equirectangular projection image data.

The imaging control unit 605 generally sets commands and the like in the registers of the imaging elements 603a and 603b using the I2C bus, with the imaging control unit 605 acting as a master device and the imaging elements 603a and 603b acting as slave devices. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 603a and 603b, and send it to the CPU 611.

The imaging control unit 605 also instructs the imaging elements 603a and 603b to output image data when the shutter button of the operation unit 615 is pressed. Some omnidirectional imaging devices have a preview display function or a function corresponding to video display on a display (e.g., a smartphone display). In this case, the image data is output from the imaging elements 603a and 603b continuously at a predetermined frame rate (frames/minute).

The imaging control unit 605 also functions as a synchronization control unit that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging elements 603a and 603b, as described below. Note that in this embodiment, the omnidirectional imaging device is not provided with a display, but a display unit may be provided.

The microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the sound data output from the microphone 608 via the I/F bus and performs a specified process on the sound data.

The CPU 611 controls the overall operation of the omnidirectional imaging device and executes necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data in the middle of processing. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604 and data of a processed equirectangular projection image.

The operation unit 615 is a general term for operation buttons such as the shutter button 615a. The user operates the operation unit 615 to input various shooting modes, shooting conditions, and the like.

The external device connection I/F 616 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a PC (Personal Computer). The data of the equirectangular projection image stored in the DRAM 614 is recorded on an external medium via the external device connection I/F 616, and is transmitted to an external terminal (device) such as a smartphone via the external device connection I/F 616 as necessary.

The communication unit 617 communicates with an external terminal (device) such as a smartphone by short-range wireless communication technology such as Wi-Fi, NFC (Near Field Communication), or Bluetooth (registered trademark) via an antenna 617a provided in the omnidirectional imaging device. This communication unit 617 can also transmit data of the equirectangular projection image to an external terminal (device) such as a smartphone.

The acceleration and orientation sensor 618 calculates the orientation of the omnidirectional imaging device from the Earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data on the capture date and time of the image and the data capacity of the image data. The acceleration and orientation sensor 618 is a sensor that detects angle changes (roll angle, pitch angle, yaw angle) that accompany the movement of the omnidirectional imaging device 6. The angle changes are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of the captured image.

Furthermore, the acceleration/orientation sensor 618 is a sensor that detects acceleration in three axial directions. The omnidirectional imaging device calculates the attitude (angle with respect to the direction of gravity) of its own device (the omnidirectional imaging device) based on the acceleration detected by the acceleration/orientation sensor 618. Providing the acceleration/orientation sensor 618 in the omnidirectional imaging device improves the accuracy of image correction.

<MFP hardware configuration>
FIG. 4 is a diagram showing an example of a hardware configuration in the case where the electronic device 30 is realized by an MFP.

As shown in FIG. 4, the MFP in one embodiment includes a controller 910, a short-range communication circuit 920, an engine control unit 930, an operation panel 940, and a network I/F (Interface) 950.

Of these, the controller 910 has a CPU (Central Processing Unit) 901, which is the main part of the computer, a system memory (MEM-P) 902, a north bridge (NB) 903, a south bridge (SB) 904, an ASIC (Application Specific Integrated Circuit) 906, a local memory (MEM-C) 907, which is a storage unit, a HDD (Hard Disk Drive) controller 908, and a HD (Hard Disk) 909, which is also a storage unit, and is configured such that the NB 903 and the ASIC 906 are connected by an AGP (Accelerated Graphics Port) bus 921.

Of these, the CPU 901 is a control unit that performs overall control of the MFP. The NB 903 is a bridge that connects the CPU 901 with the MEM-P 902, SB 904, and AGP bus 921, and includes a memory controller that controls reading and writing to the MEM-P 902, a PCI (Peripheral Component Interconnect) master, and an AGP target.

The MEM-P 902 is made up of a ROM (Read Only Memory) 902a, which is memory for storing programs and data that realize the various functions of the controller 910, and a RAM (Random Access Memory) 902b, which is used for expanding programs and data, and as a drawing memory during memory printing. The programs stored in the RAM 902b may be provided by recording them in an installable or executable format on a computer-readable recording medium such as a CD-ROM, CD-R, or DVD.

The SB904 is a bridge for connecting the NB903 to PCI devices and peripheral devices. The ASIC906 is an integrated circuit (IC) for image processing applications that has hardware elements for image processing, and serves as a bridge connecting the AGP bus 921, the PCI bus 922, the HDD 908, and the MEM-C 907. The ASIC906 is made up of a PCI target and an AGP master, an arbiter (ARB) that is the core of the ASIC906, a memory controller that controls the MEM-C 907, a number of direct memory access controllers (DMACs) that rotate image data using hardware logic, and a PCI unit that transfers data between the scanner unit 931 and the printer unit 932 via the PCI bus 922. The ASIC906 may be connected to a universal serial bus (USB) interface or an Institute of Electrical and Electronics Engineers 1394 (IEEE1394) interface.

MEM-C907 is a local memory used as an image buffer for copying and a code buffer. HD909 is a storage for storing image data, font data used during printing, and forms. HD909 controls the reading and writing of data from and to HD909 under the control of CPU901. AGP bus921 is a bus interface for a graphics accelerator card proposed to speed up graphic processing, and by directly accessing MEM-P902 with high throughput, the graphics accelerator card can be made faster.

The short-range communication circuit 920 also includes a short-range communication circuit 920a. The short-range communication circuit 920 is a communication circuit such as NFC or Bluetooth.

The engine control unit 930 is further made up of a scanner unit 931 and a printer unit 932. The operation panel 940 has a panel display unit 940a, such as a touch panel, which displays the current settings and selection screens and receives input from the operator, and an operation panel 940b consisting of a numeric keypad that receives settings for image formation conditions such as density settings and a start key that receives a copy start command. The controller 910 controls the entire MFP, and controls, for example, drawing, communication, and input from the operation panel 940. The scanner unit 931 or the printer unit 932 includes an image processing unit such as error diffusion and gamma conversion.

The MFP can sequentially switch between the document box function, copy function, printer function, and facsimile function using the application switching key on the operation panel 940. When the document box function is selected, the mode becomes document box mode, when the copy function is selected, the mode becomes copy mode, when the printer function is selected, the mode becomes printer mode, and when the facsimile mode is selected, the mode becomes facsimile mode.

The network I/F 950 is an interface for data communication using the communication network N1. The short-range communication circuit 920 and the network I/F 950 are electrically connected to the ASIC 906 via the PCI bus 922.

<Hardware configuration of self-service terminals/electronic whiteboards>
5 is a diagram showing an example of a hardware configuration when the electronic device 30 is realized as a self-service terminal or an electronic whiteboard. The hardware of the self-service terminal and the electronic whiteboard are close to each other in terms of display function, user input function, etc., so the hardware configuration of this example can cover both.

As shown in FIG. 5, the self-service terminal/electronic whiteboard in one embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a SSD (Solid State Drive) 204, a network I/F (Interface) 205, and an external device connection I/F 206.

Of these, the CPU 201 controls the overall operation of the self-service terminal/electronic whiteboard. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data, such as programs for the self-service terminal/electronic whiteboard. The network controller 205 controls communication with the communication network N1. The external device connection I/F 206 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 230 and external devices (a microphone 240, a speaker 250, and a camera 260).

The self-service terminal/electronic whiteboard also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a for the short-range communication circuit 219, a power switch 222, and selection switches 223.

Of these, the capture device 211 displays video information as still images or moving images on the display of an external PC (Personal Computer) 270. The GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages the screen display in order to output the output image from the GPU 212 to a display 280, etc.

The contact sensor 214 detects that the electronic pen 290 or the user's hand H has come into contact with the display 280. The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 inputs and detects coordinates using an infrared blocking method. The method of inputting and detecting coordinates is that two light receiving and emitting devices installed at both ends of the upper side of the display 280 emit multiple infrared rays parallel to the display 280, and receive the light that is reflected by a reflecting member installed around the display 280 and returns along the same optical path as the light emitted by the light receiving element. The contact sensor 214 outputs the ID of the infrared rays emitted by the two light receiving and emitting devices that have been blocked by an object to the sensor controller 215, and the sensor controller 215 identifies the coordinate position that is the contact position of the object.

The electronic pen controller 216 communicates with the electronic pen 290 to determine whether the tip or the end of the pen is touching the display 280. The short-range communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for switching the power of the self-service terminal/electronic whiteboard ON/OFF. The selection switches 223 are, for example, a group of switches for adjusting the brightness and color of the display 280.

Furthermore, the self-service terminal/electronic whiteboard is equipped with a bus line 210. The bus line 210 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 201 shown in FIG. 5.

The contact sensor 214 is not limited to an infrared blocking type, and may use various detection means such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by a voltage change between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contacting object touching the display unit. The electronic pen controller 216 may also determine whether or not the tip and end of the electronic pen 290 have been touched, as well as the part of the electronic pen 290 that the user holds, or other parts of the electronic pen.

<Functional configuration of the equipment management system>
Next, an example of the functional configuration of the device management system in this embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram illustrating the functional configuration of each device included in the device management system in this embodiment.

<Functional configuration of management device>
6 , the management device 10 in this embodiment includes an image receiving unit 11, an operation information receiving unit 12, a status management unit 13, an operation instruction unit 14, and a status storage unit 100. The status management unit 13 in this embodiment includes an object status management unit 131, a device status management unit 132, and an output information determination unit 133.

The image receiving unit 11, the operation information receiving unit 12, the status management unit 13, and the operation instruction unit 14 are realized, for example, by a process executed by the CPU 501 and the HDD controller 505 of a program loaded onto the RAM 503 from the HDD 504 shown in FIG. 2.

The state storage unit 100 is realized, for example, by using the HD 504 shown in FIG. 2. Reading or writing of data stored in the HD 504 is performed, for example, via the HDD controller 505.

The state storage unit 100 stores state management information that manages the states of devices and objects present in the managed space R1. In this embodiment, the state management information includes a registered object list, a registered device list, and a detected object list.

The registered object list is a list that manages information about objects that exist in the managed space R1. In this embodiment, the object is a person.

The registered device list is a list that manages information about devices that exist in the managed space R1. In this embodiment, the devices are electronic devices 30.

The detected object list is a list that manages objects detected in the managed space R1. The detected object list is a temporary list used to update the registered object list.

Here, the state management information in this embodiment will be described with reference to FIG. 7. FIG. 7(A) is a conceptual diagram showing an example of a registered object list.

As shown in FIG. 7(A), the registered object list in this embodiment has the following data items: registered object ID, group ID, existence confirmation flag, attribute information, and device operation information.

The registered object ID is identification information that identifies an object included in the registered object list.

A group ID is identification information that identifies a group to which multiple registered objects belong. A group ID is assigned when some of the registered objects included in the registered object list are determined to belong to the same group. A default value (e.g., zero) is set as the group ID for registered objects that are not determined to belong to a group.

A group can be, for example, colleagues, parents and children, or friends. Members of a group may hold meetings or carry out jobs such as shopping together. Therefore, by recognizing groups, it is expected that the behavior of people can be understood more accurately. For example, when a parent and child are in an unmanned store, it would be very convenient if the parent could pay for any products the child has picked up at the same time as they check out. Also, when multiple employees are holding a meeting in a conference room equipped with an MFP and an electronic whiteboard, it would be very convenient if one employee authenticates himself/herself on the MFP and other employees could omit authentication when using the electronic whiteboard.

The existence confirmation flag is a flag that indicates whether the registered object exists in the managed space R1. For example, the existence confirmation flag is set to 1 if the object exists, and 0 if the object does not exist.

The attribute information is information associated with a registered object and is necessary for management. In this embodiment, it is the position of the registered object and the time when the position was confirmed (hereinafter, also referred to as the "image acquisition time"). The position of the registered object is expressed in three-dimensional coordinates (i.e., numerical values on each axis of a Cartesian coordinate system, such as the X-axis, Y-axis, and Z-axis).

The method of acquiring location information varies depending on the type of monitoring device 20. For example, if the monitoring device 20 is a network camera, known techniques can be used, such as acquiring the distance to an object with a stereo camera and mapping it together with the direction onto three-dimensional coordinates.

The attribute information may include various information indicating the characteristics of the registered object in addition to the position of the registered object. For example, the color or shape of the registered object can be used. If the object is a person, it is preferable to use color because the shape changes depending on the posture. The number of different colors or the area of each color also changes depending on the posture, so when determining whether the objects are the same, it is preferable to use as a condition whether at least one color is continuous.

Device operation information is information about device operations performed by a registered object. Device operation information may include input information entered by the user during the device operation. Device operation information may be, for example, the function executed by the user on the electronic device, the time of the operation, fee information as payment for the operation, payment flag information (described below) that is linked one-to-one with each fee information and indicates that the fee has already been paid, the language used, etc. Input information may be authentication information during authentication operations or setting information during device setting operations, etc.

The number of objects present in the managed space R1 changes from moment to moment. Therefore, the number of registered objects included in the registered object list is variable.

Figure 7 (B) is a conceptual diagram showing an example of a registered device list. As shown in Figure 7 (B), the registered device list in this embodiment has the following data items: registered device ID, attribute information, device input information, and device output information.

The registered device ID is identification information that identifies a device included in the registered device list.

The attribute information is information associated with a registered device and is necessary for management. In this embodiment, the attribute information includes the location of the registered device, power on/off status, whether it is in sleep mode, whether authentication is required, whether options are installed, and status information such as maintenance in progress/scheduled maintenance time/in energy saving mode. The location of the registered device is expressed in three-dimensional coordinates, just like the registered object.

Device input information (hereinafter also referred to as "usage information") is specific information that is managed when a registered device is operated. Device input information may include input information entered by the user when operating the device. Device input information is, for example, the function executed by the user on the registered device, the time of operation, fee information to be charged as compensation for the operation, etc. Device input information may also include screen setting information set by the user when operating the device. Screen setting information is, for example, the language used, the font of characters used, the size of characters used, etc.

The device output information is information that is preregistered for a registered device. The device output information is a list that associates the output conditions that trigger the transmission of an operation instruction with the operation instruction to be sent to the registered device when the output conditions are satisfied.

The number of devices installed in the managed space R1 may change. Therefore, the number of registered devices included in the registered device list is variable.

Figure 7 (C) is a conceptual diagram showing an example of a detected object list. As shown in Figure 7 (C), the detected object list in this embodiment has the detected object ID, group ID, and attribute information as data items.

The detected object ID is identification information that identifies the object included in the detected object list.

A group ID is identification information that identifies a group that includes multiple detected objects. A group ID is assigned when some of the detected objects included in the detected object list are determined to belong to the same group. An initial value (e.g., zero) is set as the group ID for detected objects that are not determined to belong to a group.

The attribute information is information associated with the detected object and is necessary for management. In this embodiment, it is the position of the detected object and the time of image capture. The position of the detected object is expressed in three-dimensional coordinates, just like registered objects.

The number of objects detected in the managed space R1 changes from moment to moment. Therefore, the number of detected objects included in the detected object list is variable.

Returning to FIG. 6, the image receiving unit 11 receives an image from the monitoring device 20. In response to a request from the state management unit 13, the image receiving unit 11 sends the received image to the state management unit 13. The image receiving unit 11 is an example of the first image receiving unit and the second image receiving unit. The first image receiving unit and the second image receiving unit may be composed of one software module or multiple software modules. The image receiving unit 11 receives, for example, a first image captured by the first image receiving unit of a first user using a first electronic device. The image receiving unit 11 receives, for example, a second image captured by a photographing device of a second user present in an operable position of the second electronic device by the second image receiving unit.

The operation information receiving unit 12 receives device operation information from the electronic device 30. In response to a request from the state management unit 13, the operation information receiving unit 12 sends the received device operation information to the state management unit 13. Note that the operation information receiving unit 12 is an example of an input information receiving unit. For example, the operation information receiving unit 12 receives from the first electronic device authentication information input on the first electronic device by the first user identified by the first user identification unit. Also, for example, the operation information receiving unit 12 receives from the first electronic device usage information of the first user identified by the first user identification unit using the first electronic device.

The state management unit 13 manages the state of the electronic device 30 and the object using the object state management unit 131 and the device state management unit 132. The state management unit 13 also uses the output information determination unit 133 to determine the content of the operation instruction to be sent to the electronic device 30. The state management unit 13 is an example of the first user identification unit and the second user identification unit. The first user identification unit and the second user identification unit may be composed of one software module or multiple software modules. The state management unit 13 identifies the first user who used the first electronic device based on the usage information of the first electronic device and the first image. The state management unit 13 also identifies the second user who is present in an operable position of the second electronic device from the second image. This process is performed using the following object state management unit 131 and device state management unit 132.

The object state management unit 131 updates the registered object list stored in the state storage unit 100 based on the image received from the image receiving unit 11. For example, the object state management unit 131 identifies a first user based on a first image. Also, for example, the object state management unit 131 identifies a second user who is present in an operable position of a second electronic device from a second image. Details of the process will be described later.

The device status management unit 132 updates the registered device list and the registered object list stored in the status storage unit 100 based on the device operation information received from the operation information receiving unit 12. For example, the device status management unit 132 identifies the first user who used the first electronic device based on the usage information of the first electronic device and the first image. Details of the process will be described later.

The output information determination unit 133 determines the content of the operation instruction to be sent to the electronic device 30 based on the state management information stored in the state storage unit 100. The output information determination unit 133 sends the determined operation instruction to the operation instruction unit 14.

The operation instruction unit 14 transmits the operation instruction received from the state management unit 13 to the electronic device 30. For example, based on the information of the first user identified by the first user identification unit and the second user identified by the second user identification unit and the authentication information, when a predetermined condition is satisfied, the operation instruction unit 14 instructs the second electronic device to perform an operation using the authentication information or the screen setting information in the authentication information. Also, based on the information of the first user identified by the first user identification unit and the second user identified by the second user identification unit and the usage information, when a predetermined condition is satisfied, the operation instruction unit 14 instructs the second electronic device to perform an operation using the usage information.

For example, the specified condition is when device operation information exists in which the first user performed an operation on the first electronic device that incurs a fee, and the fee is unpaid. At this time, the operation instruction unit 14 instructs the second electronic device to display or add the usage fee for the first electronic device to the billing amount presented to the first user.

For example, the specified condition is when device operation information exists in which another user with the same group ID as the first user performed an operation on the first electronic device that generates a fee, and the fee is unpaid. At this time, the operation instruction unit 14 instructs the second electronic device to add the usage fee of the first electronic device related to the other user to the bill amount presented to the first user and display it, or to display an option indicating that the usage fee of the first electronic device related to the other user will be paid by the other user, not the first user.

Also, for example, the specified condition is when device operation information exists indicating that the first user performed an operation on the first electronic device that generates a fee, and the fee is unpaid. At this time, the operation instruction unit 14 instructs the second electronic device to display the billing amount for the first user, excluding the amount already paid.

Also, for example, the predetermined condition is when device operation information exists in which a first user has performed an operation to log in on a first electronic device, authentication information input by the first user on the first electronic device can be authenticated on a second electronic device, and the second electronic device has not yet been logged in. At this time, the operation instruction unit 14 instructs the second electronic device to perform an operation to log in using the authentication information input by the first user on the first electronic device.

Also, for example, the specified condition is when device operation information exists in which another user having the same group ID as the first user has performed an operation to log in on the first electronic device, the authentication information input by the other user on the first electronic device can be authenticated on the second electronic device, and the second electronic device is not yet logged in. At this time, the operation instruction unit 14 instructs the second electronic device to perform an operation to log in using the authentication information input by the other user on the first electronic device.

Furthermore, when the device operation information includes an operation to input information to the first electronic device, the operation instruction unit 14 instructs the second electronic device to perform an operation to display an option to display the information input to the first electronic device.

<Functional configuration of the monitoring device>
As shown in FIG. 6, a monitoring device 20 in this embodiment includes an image acquisition unit 21 and an image transmission unit 22 .

The image acquisition unit 21 acquires an image including the vicinity of the electronic device 30 installed in the managed space R1. The image acquisition unit 21 is realized, for example, by a process in which a program expanded from the ROM 612 to the SRAM 613 shown in FIG. 3 is executed by the CPU 611 and the imaging control unit 605.

The image transmission unit 22 transmits the images acquired by the image acquisition unit 21 to the management device 10. The image transmission unit 22 is realized, for example, by a process in which a program expanded from the ROM 612 to the SRAM 613 shown in FIG. 3 is executed by the CPU 611 and the external device connection I/F 616.

<Functional configuration of electronic devices>
As shown in FIG. 6 , an electronic device 30 in this embodiment includes an operation information transmission unit 31 and a device control unit 32 .

The operation information transmission unit 31 and the device control unit 32 are realized, for example, by a program loaded from the HD 909 shown in FIG. 4 onto the RAM 902b, which is executed by the CPU 901 and the HDD controller 908.

The operation information transmission unit 31 transmits device operation information regarding operations performed by the user to the management device 10.

The device control unit 32 receives an operation instruction from the management device 10. The device control unit 32 controls the operation of the electronic device 30 based on the received operation instruction. For example, the device control unit 32 authenticates a user using authentication information included in the operation instruction. Also, for example, the device control unit 32 changes the settings of the electronic device 30 using setting information included in the operation instruction.

<Processing procedure for device management method>
Next, a device management method executed by the device management system in this embodiment will be described with reference to FIGS.

<Basic Flowchart>
The device management method in this embodiment includes many double-loop processes that process all combinations of data included in a list and data included in another list. Therefore, in this embodiment, a basic flowchart showing the framework of the double-loop process is introduced, and the following explanation focuses on the process in the basic flowchart. Note that each process in the basic flowchart is a process performed for one combination of two data.

The basic flowchart in this embodiment includes two basic flowcharts. The first basic flowchart is used when two data items included in different lists are processed in a double loop. The second basic flowchart is used when two data items included in the same list are processed in a double loop.

Figure 8 is a flowchart showing an example of the first basic flowchart in this embodiment.

In step S101, list A is read. The number of data items in list A is set to N. In step S102, list B is read. The number of data items in list B is set to M.

In step S103, variable n is initialized to 1. In step S104, variable m is initialized to 1. In step S105, process A1 is executed.

In step S106, it is determined whether the variable m is equal to the number of data items M. This means whether the combinations of the nth data item in list A with all data items in list B have been processed. If the variable m is different from the number of data items M (NO), the process proceeds to step S107. If the variable m is equal to the number of data items M (YES), the process proceeds to step S109.

In step S107, process A3 is executed. In step S108, variable m is incremented. Then, the process returns to step S106.

In step S109, process A2 is executed. In step S110, it is determined whether or not the variable n is equal to the number of data N. This means that for all data in list A, combinations with all data in list B have been processed. If the variable n is different from the number of data N (NO), proceed to step S111. If the variable n is equal to the number of data N (YES), proceed to step S113.

In step S111, process A5 is executed. In step S112, variable n is incremented. Then, the process returns to step S104.

In step S113, process A4 is executed. This completes the processing for all combinations of data contained in list A and list B.

Figure 9 is a flowchart showing an example of the second basic flowchart in this embodiment.

In step S201, list A is read. The number of data items in list A is N.

In step S202, variable n is initialized to 1. In step S203, variable m is assigned n+1. In step S204, process B1 is executed.

In step S205, it is determined whether the variable m is equal to N. This means whether the combinations of the nth data in list A with all data in list A have been processed. If the variable m is different from N (NO), the process proceeds to step S206. If the variable m is equal to N (YES), the process proceeds to step S208.

In step S206, process B3 is executed. In step S207, variable m is incremented. Then, the process returns to step S204.

In step S208, process B2 is executed. In step S209, it is determined whether variable n is equal to N-1. This means that all combinations of all data in list A with all other data in list A have been processed. If variable n is different from N-1 (NO), proceed to step S210. If variable n is equal to N-1 (YES), proceed to step S212.

In step S210, process B5 is executed. In step S211, variable n is incremented. Then, the process returns to step S203.

In step S212, process B4 is executed. This completes the processing for all combinations of data included in list A.

<Device management method>
FIG. 10 is a flowchart showing an example of a device management method executed by the device management system in this embodiment.

The device management system repeatedly executes the flowchart shown in FIG. 10 at a predetermined time interval. The time interval can be set arbitrarily, but is, for example, 3 seconds.

In step S1, the image acquisition unit 21 included in the monitoring device 20 acquires a first image capturing an area near the electronic device 30-1. The image acquisition unit 21 also acquires a second image capturing an area near the electronic device 30-2. The monitoring device 20 is installed so that the areas near the electronic devices 30-1 and 30-2 installed in the managed space R1 are included in the angle of view. Therefore, the first image acquired by the image acquisition unit 21 captures an area near the electronic device 30-1. The second image acquired by the image acquisition unit 21 captures an area near the electronic device 30-2.

The vicinity of electronic device 30-1 or electronic device 30-2 refers to the range of positions where the user can operate electronic device 30-1 or electronic device 30-2 (hereinafter also referred to as the "operable position"). The operable position is a position where the user can reach electronic device 30-1 or electronic device 30-2. The operable position is, for example, a position within one meter of electronic device 30-1 or electronic device 30-2. Therefore, the first image is an image of electronic device 30-1 and the user using electronic device 30-1. The second image is an image of electronic device 30-2 and the user using electronic device 30-2.

If the operable positions of the electronic devices 30-1 and 30-2 can be captured in a single image, the operable positions of the electronic devices 30-1 and 30-2 may be captured in a single image. In this case, the first image and the second image are the same image.

The first image and the second image acquired by the image acquisition unit 21 are not limited to still images, but may be videos.

The image acquisition unit 21 then sends the acquired first image and second image to the image transmission unit 22. The image transmission unit 22 receives the first image and the second image from the image acquisition unit 21. The image transmission unit 22 then transmits the received first image and second image to the management device 10. In the management device 10, the image reception unit 11 receives the first image and the second image from the monitoring device 20.

In step S2, the state management unit 13 included in the management device 10 requests the first image and the second image from the image receiving unit 11. Next, the state management unit 13 inputs the first image and the second image received from the image receiving unit 11 to the object state management unit 131.

Next, the object state management unit 131 executes an object state update process, which will be described later, and updates the registered object list of the state management information stored in the state storage unit 100. If there are multiple monitoring devices 20, the object state management unit 131 executes an object state update process for each of the multiple images received from each of the multiple monitoring devices 20.

<Object state update process>
The object state update process in this embodiment will now be described with reference to Fig. 11 to Fig. 16. Fig. 11 is a flowchart showing an example of the object state update process (step S2 in Fig. 10) executed by the object state management unit 131 in this embodiment.

In step S21, the object state management unit 131 executes the detection object list creation process described below and creates a detection object list based on the image input from the state management unit 13.

Figure 12 is a flowchart showing an example of the detected object list creation process (step S21 in Figure 11) executed by the object state management unit 131 in this embodiment.

In step S21-1, the object state management unit 131 clears the detected object list stored in the state storage unit 100. In other words, it deletes all data contained in the detected object list.

In step S21-2, the object state management unit 131 acquires an image input from the state management unit 13. The image may be one image received from one monitoring device 20, or multiple images received from multiple monitoring devices 20.

Next, the object state management unit 131 divides the image into a predetermined number of blocks. Next, for each block, the object state management unit 131 measures the distance between the object captured in the image and the camera. In addition, the object state management unit 131 calculates three-dimensional coordinates from the measured distances. A publicly known method may be used to convert from distance to three-dimensional coordinates.

In step S21-3, the object state management unit 131 acquires the image acquisition time T from the acquired image. If the image acquisition time cannot be acquired from the image, the current time may be acquired as the image acquisition time T.

In step S21-4, the object state management unit 131 analyzes the acquired image and detects the object captured in the image. In this embodiment, it detects the person captured in the image.

People can be detected using known methods. For example, pattern matching using a machine learning model can be used. Specifically, people can be extracted from an image using known Region Based Convolutional Neural Networks (R-CNN) or the like.

Furthermore, the accuracy of detection can be improved by pattern matching the extracted image of the person with a comparison image stored in advance. In addition to these, various other well-known person detection methods can also be used.

In step S21-5, the object state management unit 131 assigns a detected object ID that identifies the detected object.

In step S21-6, the object state management unit 131 registers the detected object ID and attribute information (the position of the detected object and the image acquisition time) in the detected object list. Note that at this stage, the group ID is set to an initial value indicating that the object does not belong to a group.

Returning to FIG. 11, in step S22, the object state management unit 131 executes the registered object list update process described below, and updates the registered object list based on the detected object list.

Figures 13 and 14 are a flowchart showing an example of the registered object list update process (step S22 in Figure 11) executed by the object state management unit 131 in this embodiment.

The registered object list update process is performed within the framework of the first basic flowchart (see FIG. 8). Specifically, the detected object list is list A, the registered object list is list B, and process A1 (FIG. 13) and process A4 (FIG. 14) are executed for all combinations of detected objects and registered objects. Note that there are no processes equivalent to processes A2, A3, and A5.

Figure 13 is a flowchart showing an example of process A1 of the registered object list update process. Process A1 is a process for determining the identity of a registered object and a detected object and updating the position of the registered object.

Here, processing is performed on the combination of the nth detected object in the detected object list and the mth registered object in the registered object list.

In step S22-1, the object state management unit 131 clears the existence confirmation flag of the mth registered object. In other words, the existence confirmation flag of the mth registered object is set to 0. This means that it is unknown whether the registered object exists in the managed space R1.

In step S22-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the mth registered object and the three-dimensional coordinates of the nth detected object.

In step S22-3, the object state management unit 131 determines whether the distance X is equal to or less than a predetermined threshold (e.g., 1 meter). If the distance X exceeds the threshold (NO), the object state management unit 131 ends process A1. This means that the two are far apart and therefore it has been determined that they are not the same person. If the distance X is equal to or less than the threshold (YES), the object state management unit 131 proceeds to step S22-4.

In step S22-4, the object state management unit 131 adds the three-dimensional position and image acquisition time of the nth detected object to the attribute information of the mth registered object. This means that because the distance between the two is close, it is determined that they are the same person, and that the position of the nth detected object is the position of the mth registered object at the image acquisition time T.

The attribute information in the registered object list increases each time the object state update process is executed, as long as the registered object is detected in the image. In other words, the attribute information in the registered object list represents the time series of the positions where the registered object was detected in the managed space. Therefore, the movement of the registered object can be tracked using the attribute information in the registered object list.

In step S22-5, the object state management unit 131 sets the presence confirmation flag for the mth registered object. That is, the presence confirmation flag for the mth registered object is set to 1. This means that an object matching the registered object was detected in step S22-3, and therefore the registered object was determined to exist in the managed space R1 even at the image acquisition time T.

Figure 14 is a flowchart showing an example of process A4 of the registered object list update process. Process A4 is a process for deleting registered objects whose existence could not be confirmed in process A1 from the registered object list.

In step S22-6, the object state management unit 131 initializes the variable m to 1.

In step S22-7, the object state management unit 131 determines whether the existence of the mth registered object has been confirmed. Specifically, it determines whether the existence confirmation flag is 1 or 0. If the existence confirmation flag is 1 (YES), the object state management unit 131 proceeds to step S22-9. If the existence confirmation flag is 0 (NO), the object state management unit 131 proceeds to step S22-8.

In step S22-8, the object state management unit 131 deletes the mth registered object from the registered object list.

In step S22-9, the object state management unit 131 determines whether the variable m is equal to the number of data M. If the variable m is different from the number of data M (NO), the object state management unit 131 proceeds to step S22-10. If the variable m is equal to the number of data M (YES), the object state management unit 131 ends the process.

In step S22-10, the object state management unit 131 increments the variable m. After that, the object state management unit 131 returns the process to step S22-7.

Returning to FIG. 11, in step S23, the object state management unit 131 executes a group ID assignment process, which will be described later, and assigns group IDs to the registered objects included in the registered object list.

The group ID assignment process is performed within the framework of the second basic flowchart (see FIG. 9). Specifically, the registered object list is List A, and process B1 (FIG. 15 or FIG. 16) is executed for all combinations of two registered objects. Note that there are no processes equivalent to processes B2-B5.

There are various possible methods for determining whether an object and another object are included in the same group. In this embodiment, group determination based on positional proximity and group determination based on group behavior will be described. However, the group determination methods are not limited to these, and any technology that can perform group determination of objects from an image may be used.

Figure 15 is a flowchart showing a first example of the group ID assignment process (step S23 in Figure 11) executed by the object state management unit 131 in this embodiment. The first example of the group ID assignment process is group determination based on positional proximity.

Group determination based on locational proximity is a process of registering as a group two objects that have been detected in close proximity over a recent predetermined number of consecutive times. The predetermined number is, for example, five times. If the time interval for executing the device management method is three seconds, two objects that have been detected in close proximity over a continuous period of 15 seconds will be determined to be in the same group.

Here, processing is performed on the combination of the mth registered object and the nth registered object in the registered object list.

In step S23A-1, the object state management unit 131 initializes variables k and j to 1. Variable k is a counter that indicates the number of times it has been determined whether or not two objects are in close proximity. Variable j is a counter that indicates the number of times it has been determined that two objects are in close proximity.

In step S23A-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the mth registered object k times before and the three-dimensional coordinates of the nth registered object k times before.

In step S23A-3, the object state management unit 131 determines whether the distance X is less than a predetermined threshold (e.g., 1 meter). If the distance X is less than the threshold (YES), the object state management unit 131 proceeds to step S23A-4. If the distance X is equal to or greater than the threshold (NO), the object state management unit 131 proceeds to step S23A-5.

In step S23A-4, the object state management unit 131 increments the variable j.

In step S23A-5, the object state management unit 131 increments the variable k.

In step S23A-6, the object state management unit 131 determines whether the variable k is equal to the predetermined number of times K. If the variable k is different from the predetermined number of times K (NO), the object state management unit 131 returns the process to step S23A-2. If the variable k is equal to the predetermined number of times K (YES), the object state management unit 131 proceeds to step S23A-7.

In step S23A-7, the object state management unit 131 determines whether the variable j is equal to the predetermined number of times K. If the variable j is different from the predetermined number of times K (NO), the object state management unit 131 ends the process. If the variable j is equal to the predetermined number of times K (YES), the object state management unit 131 proceeds to step S23A-8.

In step S23A-8, the object state management unit 131 determines whether a group ID has been assigned to either the mth registered object or the nth registered object. If a group ID has been assigned (YES), the object state management unit 131 proceeds to step S23A-9. If a group ID has not been assigned (NO), the object state management unit 131 proceeds to step S23A-10.

In step S23A-9, the object state management unit 131 determines one of the group IDs assigned to the mth registered object and the nth registered object as the group ID to be assigned. Next, the object state management unit 131 sets the determined group ID as the group ID for the mth registered object and the nth registered object in the registered object list.

The object state management unit 131 determines the group ID to be assigned as follows. If a group ID is assigned to only one of the registered objects, that group ID is set as the group ID to be assigned. In other words, a registered object that has not been assigned a group ID is added as a member of an existing group.

If both registered objects have been assigned group IDs, first, the m+1th and subsequent registered objects are sorted by group ID to identify the registered object that has the same group ID as the mth registered object. Next, the group IDs of the mth registered object and all identified registered objects are updated with the group ID of the nth registered object. As a result, all members of the group to which the mth registered object belongs belong to the same group as the nth registered object. This makes it possible to identify groups to which three or more registered objects belong.

In step S23A-10, the object state management unit 131 issues a new group ID that does not overlap with other group IDs. Next, the object state management unit 131 sets the issued new group ID as the group ID of the mth registered object and the nth registered object in the registered object list.

Figure 16 is a flowchart showing a second example of the group ID assignment process (step S23 in Figure 11) executed by the object state management unit 131 in this embodiment. The second example of the group ID assignment process is group determination based on group actions.

Group determination based on group behavior is a process in which a detected object list is created and then group determination is performed by image analysis. Therefore, when performing group determination based on group behavior, step S23 is executed between step S21 and step S22.

Here, processing is performed on the combination of the mth detected object and the nth detected object in the detected object list.

In step S23B-1, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the nth detected object and the three-dimensional coordinates of the mth detected object.

In step S23B-2, the object state management unit 131 determines whether the distance X is less than a predetermined threshold (e.g., 1 meter). If the distance X is equal to or greater than the threshold (NO), the object state management unit 131 ends the process. If the distance X is less than the threshold (YES), the object state management unit 131 proceeds to step S23B-3.

In step S23B-3, the object state management unit 131 calculates the midpoint between the three-dimensional coordinates of the nth detected object and the three-dimensional coordinates of the mth detected object. Specifically, the object state management unit 131 divides the sum of the X coordinate, Y coordinate, and Z coordinate of the two detected objects by 2.

In step S23B-4, the object state management unit 131 extracts an image P with a radius of Y pixels centered on the midpoint. The radius Y is set according to the resolution of the image so that the range included in the image P is approximately 1 m in distance in real space.

In step S23B-5, the object state management unit 131 reads out a group action image that has been stored in advance. A group action image is a collection of images that represent actions that are determined to belong to the same group. For example, it is an image of two people greeting or shaking hands, etc.

In step S23B-6, the object state management unit 131 calculates the similarity between the video P and each of the group action images by pattern matching or the like.

In step S23B-7, the object state management unit 131 determines whether any of the calculated similarities exceeds a predetermined threshold. If any of the similarities exceeds the threshold (YES), the object state management unit 131 proceeds to step S23B-8. If no similarity exceeds the threshold (NO), the object state management unit 131 ends the process.

In step S23B-8, the object state management unit 131 issues a new group ID that does not overlap with other group IDs. Next, the object state management unit 131 sets the issued new group ID as the group ID for the mth detected object and the nth detected object in the detected object list.

In step S22-4 (see FIG. 13) of the registered object list update process that is executed after that, the object state management unit 131 sets the group ID of the registered object to the group ID of the detected object that was determined to be the same object. Note that if an existing group ID has been assigned to the registered object, steps S23A-8 to S23A-10 (see FIG. 15) of the first example of the group ID assignment process are executed.

Referring back to FIG. 10, in step S3, the operation information transmission unit 31 included in the electronic device 30-1 determines whether the operation content is information that should be notified to the management device 10, depending on the operation performed by the user on the electronic device 30-1. This determination is made based on whether the operation content matches any of the predetermined operation contents.

When the operation information transmission unit 31 determines that the operation content is one that should be notified, it transmits device operation information regarding the device operation to the management device 10. In the management device 10, the operation information reception unit 12 receives the device operation information from the electronic device 30-1.

The operation contents and operation information in this embodiment are shown below as examples.

<Example 1>
Operation details: Device login Operation information: Registered device ID, device authentication ID (if authentication is required), login time

<Example 2>
Operation details: Processing that incurs charges (copy output, use of pay-per-use software, etc.)
Operation information: registered device ID, device authentication ID (if authentication is required), charge, input data, language used

<Example 3>
Operation details: None Operation information: Maintenance, energy saving mode, etc.

The registered device ID is identification information included in the registered device list stored in the state storage unit 100 of the management device 10. The registered device ID is assigned to each electronic device 30 installed in the managed space R1. The device authentication ID is authentication information used by the user of the electronic device 30 to authenticate the use of the electronic device 30.

In step S4, the status management unit 13 included in the management device 10 requests operation information from the operation information receiving unit 12. Next, the status management unit 13 inputs the operation information received from the operation information receiving unit 12 to the device status management unit 132.

Then, the device status management unit 132 executes the device status update process described below, and updates the registered device list of the status management information stored in the status storage unit 100.

<Device status update process>
Here, the device status update process in this embodiment will be described with reference to Fig. 17. Fig. 17 is a flow chart showing an example of the device status update process (step S4 in Fig. 10) executed by the device status management unit 132 in this embodiment.

In step S41, the device status management unit 132 identifies a registered device included in the registered device list by the registered device ID included in the device operation information. Next, the device status management unit 132 sets the received device operation information to the device input information of the identified registered device.

In step S42, the device status management unit 132 initializes the variable n to 1.

In step S43, the device status management unit 132 calculates the distance X between the three-dimensional coordinates of the identified registered device and the latest three-dimensional coordinates of the nth registered object. The three-dimensional coordinates of the registered device are set in advance using a method similar to that for the three-dimensional coordinates of the registered object. Since the installation location of the registered device may be moved, it is necessary to update the three-dimensional coordinates of the registered device periodically, but the update frequency may be low.

In step S44, the device status management unit 132 determines whether the distance X is less than a predetermined threshold (e.g., 1 meter). If the distance X is equal to or greater than the threshold (NO), the device status management unit 132 proceeds to step S47. If the distance X is less than the threshold (YES), the device status management unit 132 proceeds to step S45.

In step S45, the device status management unit 132 adds the device input information of the identified registered device to the device operation information of the nth registered object. The device operation information in the registered object list is configured to be able to store a predetermined number of pieces of device input information.

In step S46, the device status management unit 132 determines whether the variable n is equal to the number of data N. If the variable n is different from the number of data N (NO), the device status management unit 132 proceeds to step S47. If the variable n is equal to the number of data N (YES), the device status management unit 132 ends the process.

In step S47, the device status management unit 132 increments the variable n. The device status management unit 132 then returns to step S43.

Returning to FIG. 10, in step S5, the output information determination unit 133 of the management device 10 executes an output information determination process, which will be described later, and determines the operation instructions to be sent to the electronic device 30 based on the state management information stored in the state storage unit 100.

<Output information determination process>
The output information determination process in this embodiment will now be described with reference to Fig. 18 and Fig. 19. Fig. 18 is a flowchart showing an example of the output information determination process (step S5 in Fig. 10) executed by the output information determination unit 133 in this embodiment.

The output information determination process is performed within the framework of the first basic flowchart (see FIG. 8). Specifically, the registered device list is List A, the registered object list is List B, and process A1 (FIG. 18) is executed for all combinations of registered devices and registered objects. Note that there are no processes equivalent to processes A2-A5.

Here, processing is performed on the combination of the nth registered device in the registered device list and the mth registered object in the registered object list.

In step S51, the output information determination unit 133 calculates the distance X between the latest three-dimensional coordinates of the nth registered device and the latest three-dimensional coordinates of the mth registered object.

In step S52, the output information determination unit 133 determines whether the distance X is less than a predetermined threshold (e.g., 1 meter). If the distance X is equal to or greater than the threshold (NO), the device status management unit 132 ends the process. If the distance X is less than the threshold (YES), the device status management unit 132 proceeds to step S53.

In step S53, the output information determination unit 133 obtains the output conditions included in the device output information of the nth registered device from the registered device list stored in the state storage unit 100.

In step S54, the output information determination unit 133 obtains device operation information for the mth registered object from the registered object list stored in the state storage unit 100.

In step S55, the output information determination unit 133 determines whether any of the device operation information acquired in step S54 satisfies the output condition acquired in step S53. If none of the device operation information satisfies the output condition (NO), the device status management unit 132 ends the process. If any of the device operation information satisfies the output condition (YES), the device status management unit 132 proceeds to step S56.

In step S56, the output information determination unit 133 obtains an operation instruction corresponding to the output condition obtained in step S53 from the device output information of the nth registered device. The operation instruction of the registered device may include an instruction to perform adaptive processing according to the attribute information of the registered device. For example, the operation instruction may include a condition that the operation instruction is not sent when the registered device is in an inoperable state. An inoperable state is, for example, a power-off state or a hibernation state.

Figure 19 is a flowchart showing a modified example of the output condition determination process (step S55 in Figure 18) executed by the output information determination unit 133 in this embodiment.

In the output information determination process shown in FIG. 18, in step S55, it was determined whether the output conditions were met based only on the device operation information of the mth registered object. In the output information determination process shown in FIG. 19, it is determined whether the output conditions are met by referring to device operation information other than the mth registered device.

In step S55-1, the output information determination unit 133 determines whether any of the device operation information of the mth registered object satisfies the output condition of the nth registered device. If any of the device operation information satisfies the output condition (YES), the output information determination unit 133 proceeds to step S55-10. If none of the device operation information satisfies the output condition (NO), the output information determination unit 133 proceeds to step S55-2.

In step S55-2, the output information determination unit 133 determines whether or not device operation information of other registered objects is necessary based on the output conditions of the nth registered device. If device operation information of other registered objects is necessary (YES), the output information determination unit 133 proceeds to step S55-3. If device operation information of other registered objects is not necessary (NO), the output information determination unit 133 proceeds to step S55-9.

In step S55-3, the output information determination unit 133 initializes the variable k to 1.

In step S55-4, the output information determination unit 133 determines whether the variable k is equal to the variable m. If the variable k is equal to the variable m (YES), the output information determination unit 133 proceeds to step S55-5. If the variable k is not equal to the variable m (NO), the output information determination unit 133 proceeds to step S55-7.

In step S55-5, the output information determination unit 133 obtains device operation information for the kth registered object from the registered object list stored in the state storage unit 100.

In step S55-6, the output information determination unit 133 determines whether any of the device operation information of the kth registered object satisfies the output condition of the nth registered device. If any of the device operation information satisfies the output condition (YES), the output information determination unit 133 proceeds to step S55-10. If none of the device operation information satisfies the output condition (NO), the output information determination unit 133 proceeds to step S55-7.

In step S55-7, the output information determination unit 133 determines whether the variable k is equal to the number of data items M. If the variable k is different from the number of data items M (NO), the output information determination unit 133 proceeds to step S55-8. If the variable m is equal to the number of data items M (YES), the output information determination unit 133 proceeds to step S55-9.

In step S55-8, the output information determination unit 133 increments the variable k. After that, the output information determination unit 133 returns the process to step S55-4.

In step S55-9, the output information determination unit 133 determines that the device operation information acquired in step S54 does not satisfy the output conditions acquired in step S53, and ends the process.

In step S55-10, the output information determination unit 133 determines that the device operation information acquired in step S54 satisfies the output conditions acquired in step S53, and ends the process.

Figure 20 is a diagram showing a first example of device output information in this embodiment. The first example of device output information in this embodiment includes output conditions related to payment at an unmanned store. The first example of device output information includes both output conditions that do not require device operation information of other registered objects, and output conditions that require device operation information of other registered objects (using group IDs).

The first example of device output information is an application example to a usage scene in which electronic device 30-1 is an MFP, electronic device 30-2 is a payment device (self-service terminal), and the MFP usage fee is paid at the payment device.

As shown in FIG. 20, the output condition 1-1 in this embodiment is that device operation information exists in which the mth registered object (hereinafter also referred to as "person α") performed an operation that incurs a fee on electronic device 30-1 (here, an MFP), and the payment flag is 0 (=false), indicating unpaid.

As shown in FIG. 20, if it is determined that output condition 1-1 is satisfied, one or more of the following operation instructions are determined as the operation instructions to be sent to the registered device (electronic device 30-2) or management device 10. The first operation instruction is to display the MFP usage fee as the billing amount to be presented to person α, or to display the MFP usage fee in addition to the price for purchasing other products, etc. The second operation instruction is to instruct the management device 10 to update the paid flag in person α's device operation information to 1 (= true), which indicates that payment has been made.

In this embodiment, output condition 1-2 is that device operation information exists in which the kth registered object (hereinafter also referred to as "person β") performed an operation that incurs a fee on electronic device 30-1 (here, an MFP), and the payment flag is 0 (=false), indicating unpaid. Note that person β has the same group ID as person α.

When it is determined that the output condition 1-2 is satisfied, one or more of the following operation instructions are determined as operation instructions to be sent to the registered device (electronic device 30-2) or the management device 10. The first operation instruction is to add the MFP usage fee for person β to the billing amount presented to person α. The second operation instruction is to display an option indicating that "person β, not person α, pays the MFP usage fee for person β" and a selection button for "Yes" or "No". If "Yes" is selected, the billing amount presented to person α is the billing amount including the MFP usage fee for person β. If "No" is selected, the billing amount presented to person α is the amount not including the MFP usage fee for person β. The third operation instruction is to instruct the management device 10 to update the paid flag in the device operation information of person β to 1 (= true), indicating that payment has been made, when person α pays the billing amount including the MFP usage fee for person β.

In this embodiment, output condition 1-3 is that device operation information exists in which the mth registered object (hereinafter also referred to as "person α") has performed an operation on electronic device 30-1 (here, an MFP) that incurs a fee, but the already paid flag is 1 (= true), indicating that payment has already been made. For example, this situation occurs when the kth registered object (person β) belonging to the same group has already paid the MFP usage fee for the mth registered object (person α) (paid flag = 1).

If it is determined that output condition 1-3 is met, the billing amount for person α is calculated without including any fees for which the paid flag is set to 1. This process can be easily performed using the well-known if operation of adding only fees for which the paid flag is not set to 1. If person α's only use within the store is the MFP usage fee, a message is displayed stating that "all fees have already been paid." By issuing operational instructions only when certain conditions are met, as in output condition 1-3, it is possible to improve customer convenience.

Returning to FIG. 10, in step S6, the output information determination unit 133 included in the management device 10 sends operation instruction information representing an operation instruction to the operation instruction unit 14. The operation instruction information includes information indicating the determined operation instruction.

The operation instruction unit 14 receives the operation instruction information from the output information determination unit 133. Next, the operation instruction unit 14 transmits the operation instruction contained in the operation instruction information to the electronic device 30-2, which is a settlement device (self-service terminal).

In the electronic device 30-2 (payment settlement device), the device control unit 32 receives an operation instruction from the management device 10. Next, the device control unit 32 executes an operation using the fee information included in the operation instruction in accordance with the received operation instruction. For example, the device control unit 32 displays the fee information included in the operation instruction on the display 506 in addition to the price of the product purchased by the user.

For example, when the device control unit 32 receives fee information for all members of a group to which the user belongs, it displays the fee to be paid by the user on the display 506 in a selectable format. If the user wishes to pay individually, they can make the payment excluding fees based on the operation of electronic devices by other users.

Furthermore, the device control unit 32 determines whether the fee has already been paid by the group members to which the user belongs based on the paid flag, and removes fees for which the paid flag is set to 1 in advance so that they are not displayed on the display 506.

Figure 21 is a diagram showing a second example of device output information in this embodiment. The second example of device output information in this embodiment includes output conditions related to information sharing between electronic devices in an office. As with the first example, the second example of device output information includes both output conditions that do not require device operation information of other registered objects, and output conditions that require device operation information of other registered objects (using group IDs).

The second example of device output information is an application example to a usage scenario in which electronic device 30-1 is an MFP, electronic device 30-2 is an electronic whiteboard, and authentication information input on the MFP is used to log in to the electronic whiteboard.

As shown in FIG. 21, output condition 2-1 in this embodiment is that the authentication information input by the mth registered object (hereinafter also referred to as "person α") on another electronic device (here, an MFP) is authentication information that can be authenticated on the nth electronic device (here, an electronic whiteboard), and that the nth electronic device is not already being used by another person.

By checking that the board is not being used by another person, it is possible to prevent the display of the electronic whiteboard being changed without permission when person α approaches while it is being used by another person. Checking that the board is not being used by another person can be appropriately performed by the electronic whiteboard's CPU using known programming methods based on conditions such as whether the electronic whiteboard is being used through authentication or whether writing has been made on the electronic whiteboard within the past minute.

As in output condition 2-1, by issuing an operational command only when certain conditions are met, customer convenience can be improved.

As shown in FIG. 21, if it is determined that output condition 2-1 is satisfied, one or more of the following operation instructions are determined as the operation instructions to be transmitted to the registered device (electronic device 30-2). The first operation instruction is to automatically authenticate person α on the electronic whiteboard. In this case, the operation instruction to be transmitted includes the authentication information entered by person α on the MFP. The second operation instruction is to display "Mr. α, automatic authentication has been performed" on the electronic whiteboard for several seconds.

The third operation instruction is to display the message "Do you want to display the information just entered on device XX (name of MFP)?" along with a selection button for "Yes" or "No" when person α is entering information on an MFP (e.g., scanning operation). In this case, if "Yes" is selected, the information entered on the scanner is displayed on the electronic whiteboard.

The fourth operation instruction is to identify the language set in the MFP from person α's device operation information and reflect it in the display on the electronic whiteboard.

As a variation of output condition 2-1, an output condition may be used in which the attribute information of the electronic whiteboard is not in a dormant state such as "in energy saving mode." In this way, an electronic whiteboard that is in a dormant state will not be turned on by the approach of person α, and energy usage efficiency will be improved.

As shown in FIG. 21, output condition 2-2 in this embodiment is that the authentication information input by the kth registered object (hereinafter also referred to as "person β") in another electronic device (here, an MFP) is authentication information that can be authenticated in the nth electronic device (here, an electronic whiteboard). Note that person β has the same group ID as person α.

As shown in FIG. 21, if it is determined that output condition 2-2 is satisfied, one or more of the following operation instructions are determined as the operation instructions to be transmitted to the registered device (electronic device 30-2). The first operation instruction is to automatically authenticate person α on the electronic whiteboard. In this case, the operation instruction to be transmitted includes the authentication information entered by person β on the MFP. The second operation instruction is to display on the electronic whiteboard for several seconds, "Automatic authentication has been performed using β's authentication information."

The third operational instruction is to display the text "Do you want to display the information entered by person β on device ○○ (name of MFP)?" along with a selection button for "Yes" or "No" when person β is entering information on an MFP (e.g., scanning operation). In this case, if "Yes" is selected, the information entered on the scanner is displayed on the electronic whiteboard.

Returning to FIG. 10, in step S6, the output information determination unit 133 included in the management device 10 sends operation instruction information representing an operation instruction to the operation instruction unit 14. The operation instruction information includes the determined operation instruction and information indicating the destination electronic device 30-2.

The operation instruction unit 14 receives the operation instruction information from the output information determination unit 133. Next, the operation instruction unit 14 transmits the operation instruction included in the operation instruction information to the destination electronic device 30-2 included in the operation instruction information.

In the electronic device 30-2, the device control unit 32 receives an operation instruction from the management device 10. Next, the device control unit 32 executes an operation using the input information included in the operation instruction in accordance with the received operation instruction. For example, the device control unit 32 authenticates the user using the authentication information included in the operation instruction. Also, for example, the device control unit 32 changes the settings of the electronic device 30-2 using the setting information included in the operation instruction.

Effects of the embodiment
The device management system 1 in this embodiment tracks the movement of a person based on images captured in the vicinity of the electronic device 30, and transmits an operation instruction using information input by a user to a first electronic device 30-1 to a second electronic device 30-2 that detects the user's presence in the vicinity. Therefore, according to the device management system 1 in this embodiment, it is possible to cause the second electronic device to execute an operation using information input to the first electronic device.

In particular, by sending an authentication instruction using authentication information authenticated by the first electronic device 30-1 to the second electronic device 30-2, authentication can be automatically performed when a user authenticated by the first electronic device 30-1 moves near the second electronic device 30-2. In other words, the device management system in this embodiment can achieve single sign-on through hardware linkage.

[Application example]
In the above embodiment, the managed space is assumed to be a conference room or a work room in an office, and the electronic device 30 is managed as an OA (Office Automation) device. However, the device management system 1 can be applied to a variety of usage scenarios, and is not limited to these.

For example, the device management system 1 can be configured to manage electronic devices installed in a hotel. Various electronic devices are installed in a hotel. For example, an automatic check-in machine may be installed in the lobby. Also, a set-top box capable of playing television broadcasts and on-demand videos may be installed in the guest rooms. By managing these devices with the device management system 1, it becomes possible to automatically set the language used by a hotel guest at the automatic check-in machine as the language used by the set-top box. For example, the automatic check-in machine is an example of a first electronic device, and the set-top box is an example of a second electronic device. Note that the first electronic device and the second electronic device may be other devices, or vice versa.

For example, the equipment management system 1 can be configured to manage electronic devices installed at an airport. At an airport, passengers must go through procedures such as ticket issuing, check-in, and baggage inspection before boarding an aircraft, and boarding passes are processed by dedicated electronic devices at each procedure. By managing these devices with the equipment management system 1, it becomes possible to automatically set the language set in the electronic device performing the previous procedure as the language set in the electronic device performing the next procedure. For example, an automatic check-in machine for ticket issuing and check-in is an example of a first electronic device, and various service devices used after ticket issuing and check-in (devices used for shops, lounges, currency exchange, and other procedures) are an example of a second electronic device. Note that the first electronic device and the second electronic device may be other devices, or vice versa.

For example, the equipment management system 1 can be configured to manage electronic devices such as search terminals installed in bookstores or libraries. By linking searches performed by customers on the search terminal with information on the bookshelf to which they are subsequently moved, this can be used to strengthen marketing. For example, a search terminal is an example of a first electronic device, and a payment device or lending machine (such as a terminal with a barcode reader) is an example of a second electronic device. Note that the first electronic device and the second electronic device may be other devices, or vice versa.

For example, the equipment management system 1 can be configured to manage electronic devices such as PCs installed in a factory. By correlating and verifying the actions of workers with the operation of equipment installed at the work site, the validity of the workers' work can be monitored. For example, an information processing terminal (such as a PC installed on a worker) is an example of a first electronic device, and work equipment (product manufacturing equipment, inspection equipment, printing equipment, etc.) is an example of a second electronic device. Note that the first electronic device and second electronic device may be other devices, or vice versa.

For example, the device management system 1 can be configured to manage electronic devices installed in an unmanned convenience store. Electronic devices such as MFPs and payment devices such as self-service terminals are installed in the convenience store. By managing these devices with the device management system 1, for example, it becomes possible to set the language used by the customer set on the MFP as the language used on the payment device. For example, the MFP is an example of the first electronic device, and a payment device such as a self-service terminal is an example of the second electronic device. Note that the first electronic device and the second electronic device may be other devices. Conversely, the payment device may be set as the first electronic device, and the MFP as the second electronic device. In this case, the device management system 1 tracks the movement of a customer who has paid a printing fee in advance at the payment device, and when the customer approaches the MFP, it can allow the customer to print or send a facsimile according to the fee.

For example, the equipment management system 1 can be configured to manage a ticket issuing machine (a self-service terminal which is an example of a first electronic device) and a settlement device (an example of a second electronic device) installed in an unmanned convenience store. In this way, even if the ticket issuing machine and the settlement device cannot directly share information via a network, the equipment management system 1 can obtain the ticket purchase information entered at the ticket issuing machine and track the customer's movements to display the fare when the customer approaches the settlement device, thereby making it possible to request the customer to make an appropriate settlement.

[supplement]
In each of the above embodiments, the management device 10 is an example of an information processing device. The device management system 1 is an example of an information processing system. The monitoring device 20 is an example of an imaging device. The electronic device 30 is an example of a first electronic device and a second electronic device. The image receiving unit 11 is an example of a first image receiving unit and a second image receiving unit. The operation information receiving unit 12 is an example of an input information receiving unit. The status management unit 13 is an example of a first user identification unit and a second user identification unit.

Each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and conventional circuit modules designed to execute each function described above.

The devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the management device 10 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and perform the processes disclosed herein.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications and alterations are possible within the scope of the gist of the present invention described in the claims.

REFERENCE SIGNS LIST 1 Equipment management system 10 Management device 11 Image receiving unit 12 Operation information receiving unit 13 State management unit 14 Operation instruction unit 100 State storage unit 131 Object state management unit 132 Equipment state management unit 133 Output information determination unit 20 Monitoring device 21 Image acquisition unit 22 Image transmission unit 30 Electronic device 31 Operation information transmission unit 32 Equipment control unit

Japanese Patent No. 6064636 Patent No. 5238409

Claims (14)

An information processing device capable of communicating with a first electronic device, a second electronic device, and an image capture device via a network,
a first image receiving unit that receives a first image captured of a first user using the first electronic device from the photographing device;
a first user identification unit that identifies the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving unit that receives, from the first electronic device, authentication information input by the first user identified by the first user identifying unit at the first electronic device;
a second image receiving unit that receives a second image captured by the photographing device of a second user who is present at an operable position of the second electronic device;
a second user identification unit that identifies the second user who is present at an operable position of the second electronic device from the second image;
an operation instruction unit that instructs the second electronic device to perform an operation using the authentication information or screen setting information in the authentication information when a predetermined condition is satisfied based on information of the first user identified by the first user identification unit and the second user identified by the second user identification unit and the authentication information;
An information processing device comprising: An information processing device capable of communicating with a first electronic device, a second electronic device, and an image capture device via a network,
a first image receiving unit that receives a first image captured of a first user using the first electronic device from the photographing device;
a first user identification unit that identifies the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving unit that receives usage information from the first electronic device, the usage information being generated by the first user identified by the first user identifying unit using the first electronic device;
a second image receiving unit that receives a second image captured by the photographing device of a second user who is present at an operable position of the second electronic device;
a second user identification unit that identifies the second user who is present at an operable position of the second electronic device from the second image;
an operation instruction unit that instructs the second electronic device to perform an operation using the usage information when a predetermined condition is satisfied based on information of the first user identified by the first user identification unit and the second user identified by the second user identification unit and the usage information;
An information processing device comprising: When the predetermined condition is that device operation information exists in which the first user has performed an operation on the first electronic device that generates a fee, and the fee is unpaid,
the operation to be instructed to the second electronic device is an operation of displaying or adding a usage fee of the first electronic device to a billing amount to be presented to the first user;
The information processing device according to claim 2 . The predetermined condition is that device operation information exists indicating that another user having the same group ID as the first user performed an operation on the first electronic device that generates a fee, and the fee is unpaid.
the operation to be instructed to the second electronic device is an operation of adding a usage fee of the first electronic device related to the other user to a billing amount to be presented to the first user and displaying the result, or an operation of displaying an option indicating that the usage fee of the first electronic device related to the other user will be paid by the other user, not the first user;
The information processing device according to claim 2 . When the predetermined condition is that device operation information exists in which the first user has performed an operation on the first electronic device that generates a fee, and the fee is unpaid,
the operation of instructing the second electronic device to display a bill amount to the first user, excluding a fee already paid;
The information processing device according to claim 2 . When the predetermined condition is that device operation information in which the first user has performed an operation to log in on the first electronic device exists, the authentication information input by the first user on the first electronic device can be authenticated on the second electronic device, and the second electronic device is not yet logged in,
the operation to be instructed to the second electronic device is an operation of logging in using the authentication information input by the first user on the first electronic device;
The information processing device according to claim 1 . The predetermined condition is that device operation information exists in which another user having the same group ID as the first user has performed an operation to log in to the first electronic device, the authentication information input by the other user to the first electronic device can be authenticated by the second electronic device, and the second electronic device is not yet logged in.
the operation to be instructed to the second electronic device is an operation of logging in by the other user using the authentication information inputted on the first electronic device;
The information processing device according to claim 1 . When the device operation information includes an operation for inputting information into the first electronic device,
the operation of instructing the second electronic device is an operation of displaying an option indicating that the information input to the first electronic device is to be displayed;
8. The information processing device according to claim 6 or 7. An information processing system in which a first electronic device, a second electronic device, a photographing device, and an information processing device can communicate with each other via a network,
The information processing device includes:
a first image receiving unit that receives a first image captured of a first user using the first electronic device from the photographing device;
a first user identification unit that identifies the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving unit that receives, from the first electronic device, authentication information input by the first user identified by the first user identifying unit at the first electronic device;
a second image receiving unit that receives a second image captured by the photographing device of a second user who is present at an operable position of the second electronic device;
a second user identification unit that identifies the second user who is present at an operable position of the second electronic device from the second image;
an operation instruction unit that instructs the second electronic device to perform an operation using the authentication information or screen setting information in the authentication information when a predetermined condition is satisfied based on information of the first user identified by the first user identification unit and the second user identified by the second user identification unit and the authentication information;
An information processing system comprising: An information processing system in which a first electronic device, a second electronic device, a photographing device, and an information processing device can communicate with each other via a network,
The information processing device includes:
a first image receiving unit that receives a first image captured of a first user using the first electronic device from the photographing device;
a first user identification unit that identifies the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving unit that receives usage information from the first electronic device, the usage information being generated by the first user identified by the first user identifying unit using the first electronic device;
a second image receiving unit that receives a second image captured by the photographing device of a second user who is present at an operable position of the second electronic device;
a second user identification unit that identifies the second user who is present at an operable position of the second electronic device from the second image;
an operation instruction unit that instructs the second electronic device to perform an operation using the usage information when a predetermined condition is satisfied based on information of the first user identified by the first user identification unit and the second user identified by the second user identification unit and the usage information;
An information processing system comprising: a computer capable of communicating with the first electronic device, the second electronic device, and the photographing device via a network;
a first image receiving step of receiving a first image captured by the image capturing device of a first user using the first electronic device;
a first user identification step of identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving step of receiving, from the first electronic device, authentication information input by the first user identified by the first user identifying step, at the first electronic device;
a second image receiving step of receiving, from the photographing device, a second image of a second user who is present in an operable position of the second electronic device;
a second user identification step of identifying the second user who is present at an operable position of the second electronic device from the second image;
an operation instruction step of instructing the second electronic device to perform an operation using the authentication information or screen setting information in the authentication information when a predetermined condition is satisfied based on information of the first user identified by the first user identification step and the second user identified by the second user identification step and the authentication information;
A device management method for performing the above. a computer capable of communicating with the first electronic device, the second electronic device, and the photographing device via a network;
a first image receiving step of receiving a first image captured by the image capturing device of a first user using the first electronic device;
a first user identification step of identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving step of receiving, from the first electronic device, usage information on the first user identified by the first user identification step using the first electronic device;
a second image receiving step of receiving, from the photographing device, a second image of a second user who is present at an operable position of the second electronic device;
a second user identification step of identifying the second user who is present at an operable position of the second electronic device from the second image;
an operation instruction step of instructing the second electronic device to perform an operation using the usage information when a predetermined condition is satisfied based on information of the first user identified by the first user identification step and the second user identified by the second user identification step and the usage information;
A device management method for performing the above. a computer capable of communicating with the first electronic device, the second electronic device, and the photographing device via a network;
a first image receiving step of receiving a first image captured by the image capturing device of a first user using the first electronic device;
a first user identification step of identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving step of receiving, from the first electronic device, authentication information input by the first user identified by the first user identifying step, at the first electronic device;
a second image receiving step of receiving, from the photographing device, a second image of a second user who is present at an operable position of the second electronic device;
a second user identification step of identifying the second user who is present at an operable position of the second electronic device from the second image;
an operation instruction step of instructing the second electronic device to perform an operation using the authentication information or screen setting information in the authentication information when a predetermined condition is satisfied based on information of the first user identified by the first user identification step and the second user identified by the second user identification step and the authentication information;
A program for executing. a computer capable of communicating with the first electronic device, the second electronic device, and the photographing device via a network;
a first image receiving step of receiving a first image captured by the image capturing device of a first user using the first electronic device;
a first user identification step of identifying the first user who used the first electronic device based on usage information of the first electronic device and the first image;
an input information receiving step of receiving, from the first electronic device, usage information on the first user identified by the first user identification step using the first electronic device;
a second image receiving step of receiving, from the photographing device, a second image of a second user who is present in an operable position of the second electronic device;
a second user identification step of identifying the second user who is present at an operable position of the second electronic device from the second image;
an operation instruction step of instructing the second electronic device to perform an operation using the usage information when a predetermined condition is satisfied based on information of the first user identified by the first user identification step and the second user identified by the second user identification step and the usage information;
A program for executing.

Images