JP2019176471A - Communication system and communication method - Google Patents

Abstract

A technique for detecting an object from an image captured by a fish-eye camera, cutting out a region including the object, and further performing a projection method conversion is already known. However, conventional fish-eye camera devices have been built with a specific application in mind, and are subject to changes in the purpose (recognition, blurring, etc.) of post-processing performed using output images, and in the performance of servers that perform post-processing. There is a problem that the image size and the projection method cannot be changed according to the situation. In a communication system, a distributed data processing terminal (6) requests image processing content for one of an image acquisition terminal (2) and a centralized data processing server (7) to the other in association with a service provided from a service providing server (8). The image processing contents obtained later as a response are transmitted to the one side, and the image data received after the image processing of the image acquisition terminal 2 is transmitted to the distributed data processing terminal 6, so that the image processing for the service is transmitted to the centralized data processing server 7. Let it be processed. [Selection diagram] Fig. 1

Description

  The present disclosure relates to a communication system and a communication method.

  In recent years, IoT (Internet of Things) has attracted attention. Among them, in particular, an edge computing communication system is attracting attention. In this communication system, image or audio data is acquired on the terminal side, transmitted to a server via the Internet, and recognition processing is performed on the server side. In Patent Document 1, for the purpose of executing processing for recognizing a person's face, a camera for face recognition is installed on the terminal (edge node) side, and the image acquisition terminal receives image data acquired from the camera on a server (face recognition). A cloud service platform is disclosed that performs facial authentication processing on the server.

  Further, a technique for detecting an object from an image picked up by a fisheye camera, cutting out a region including the object, and further performing projection system conversion is already known.

  However, conventional fisheye cameras have been constructed assuming specific (fixed) applications, and the purpose of post-processing (face recognition, blurring) performed using captured image data obtained by imaging with a fisheye camera or the like. Etc.) and there is a difference in the performance of the server that executes post-processing, etc., and there is a problem that it is not possible to respond to various needs of users such as changes in image size and projection method.

  The invention according to claim 1 is a service providing server that provides a service via a communication network, an image acquisition terminal that obtains captured image data generated by predetermined image processing, and a predetermined image for the image data. A communication system having a centralized data processing server that performs processing, and a distributed data processing terminal that mediates the image acquisition terminal and the centralized data processing server, wherein the distributed data processing terminal is provided from the service providing server The image processing content obtained as a response after requesting the image processing content for one of the image acquisition terminal or the centralized data processing server to the other is transmitted to the one in association with the service to be performed, and the image of the image acquisition terminal By transmitting image data received after processing to the distributed data processing terminal, It is a communication system characterized by having a reception means for processing that image processing to the centralized data processing server.

  As described above, according to the present invention, there is an effect that post-processing on captured image data obtained by imaging can meet various needs of users.

It is the schematic of the communication system which concerns on this embodiment. It is a hardware block diagram of a real-time data processing terminal. It is a hardware block diagram of an imaging unit. 4A is a hemispherical image (front side) captured by the imaging unit 40b, FIG. 4B is a hemispherical image captured by the imaging unit 40b (rear side), and FIG. 4C is an equirectangular projection. It is the figure which showed the image represented. It is a hardware block diagram of a near terminal data processing apparatus or a distributed data processing terminal. It is a hardware block diagram of a centralized data processing server. It is a software block diagram of the real-time data processing terminal and near terminal data processing apparatus in an image acquisition terminal. It is a functional block diagram of a communication system. It is a functional block diagram of a communication system. (A) is a conceptual diagram of an image sensor information management table, and (b) is a conceptual diagram of a cycle value management table. (A) is a conceptual diagram of an image acquisition program management table, (b) is a conceptual diagram of a composition processing program management table, (c) is a conceptual diagram of a distortion correction program management table, and (d) is a conceptual diagram of a service program management table. It is. It is a figure which shows the table of (a) object recognition service information, (b) object blurring service information, (c) object blurring service information. It is a conceptual diagram of a collation data management table. It is a conceptual diagram of a session management table. It is a conceptual diagram of terminal ID. It is a conceptual diagram of an authentication server management table. It is a conceptual diagram of an authentication management table. It is the sequence diagram which showed the authentication process. It is the sequence diagram which showed the authentication process. It is a figure which shows the example of a screen of a distributed data processing terminal. It is a figure which shows the example of a screen of a distributed data processing terminal. It is the sequence figure which showed the processing of the start request of image recognition. It is the sequence figure which showed the process of the object blurring service at the time of using a process setting table (Fig.13 (a)) for the setting of a real time data processing terminal. It is the sequence diagram which showed the process of the object blurring service at the time of using a process setting table (FIG.13 (b)) for the setting of a real time data processing terminal. It is the sequence diagram which showed the process of the object blurring service at the time of using a process setting table (FIG.13 (c)) for the setting of a real time data processing terminal. It is the figure which showed the example of a display of the image in a distributed data processing terminal. It is a processing information management table setting example. It is the sequence diagram which showed the other example of the process of a start request | requirement of image recognition.

● First embodiment [Overall outline]
The overall outline of the first embodiment of the present invention will be described below with reference to the drawings. The present embodiment discloses a communication system that realizes edge computing. That is, this embodiment discloses an invention that realizes a service provided from the service providing server 8 at the image acquisition terminal 2 by cooperation of the processing at the service providing server 8 and the processing at the image acquisition terminal 2.

<Overall Configuration of Embodiment>
FIG. 1 is a schematic diagram of a communication system according to the present embodiment. As shown in FIG. 1, the communication system of this embodiment includes a real-time data processing terminal 3, a near-terminal data processing device 5, a distributed data processing terminal 6, a centralized data processing server 7, a service providing server 8, and It is constructed by authentication servers 9a, 9b, 9c. The real time data processing terminal 3, the near terminal data processing device 5, and the distributed data processing terminal 6 constitute a distributed processing system 100. Although there are a plurality of distributed processing systems 100, only one is shown in FIG. 1 to simplify the drawing.

  The near terminal data processing device 5 is connected to the distributed data processing terminal 6 through the intranet 200 so as to be communicable. The distributed data processing terminal 6 is connected to the centralized data processing server 7, the service providing server 8, and the authentication servers 9a, 9b, and 9c via the Internet 600 so as to be communicable. The “authentication server 9” described below is a general term for the authentication servers 9a, 9b, and 9c.

  Among these, the real-time data processing terminal 3 is a terminal that performs real-time processing (real-time processing) for obtaining captured image data (hereinafter referred to as “captured image data”). The real-time data processing terminal 3 is detachably connected with an imaging unit 40 including an image sensor for imaging a subject such as a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor. As a result, the real-time data processing terminal 3 digitizes the captured image data input from the imaging unit 40 and detects an image of an object (here, a face) in real time (for example, every 1/60 seconds). The real-time data processing terminal 3 uses partial image data (hereinafter referred to as “partial image data”), which is an area of an object part (here, a face part), of a captured image including a background or the like as near-terminal data. Transmit to the processing device 5.

  The near-terminal data processing device 5 is disposed at a position closest to the real-time data processing terminal 3, and is closely connected in a one-to-one manner by, for example, a data bus or a USB (Universal Serial Bus). The near-terminal data processing device 5 encodes the partial image data received from the real-time data processing terminal 3 into a general-purpose format such as JPEG (Joint Photographic Experts Group), and then the distributed data processing terminal 6 via the intranet 200. Send to. The partial image data is handled as “data to be verified” when the centralized data processing server 7 performs verification with the verification data. Usually, the real-time data processing terminal 3 and the near-terminal data processing device 5 are connected and integrated. Here, the real-time data processing terminal 3 and the near-terminal data processing device 5 connect the image acquisition terminal 2 to each other. It is composed. Therefore, the function of encoding the partial image data (data to be verified) into a general-purpose format such as JPEG may be provided in either the real time data processing terminal 3 or the near terminal data processing terminal device 5.

  The distributed data processing terminal 6 is a computer that is used by a user at a position relatively close to the near-terminal data processing device 5 and can accept various operations of the user. The distributed data processing terminal 6 registers and stores “collation data” in the case where collation of face images is performed in advance. The distributed data processing terminal 6 can request the central data processing server 7 to collate the collation data with the collated data via the Internet 600. In this case, the distributed data processing terminal 6 also transmits the collated data received from the near terminal data processing device 5 and the collation data registered in advance in the own terminal (distributed data processing terminal 6). Furthermore, the distributed data processing terminal 6 can receive the verification result information indicating the verification result from the centralized data processing server 7 as a response. Further, the distributed data processing terminal 6 can display the collation result via the graphic interface.

  The central data processing server 7 is installed at a position relatively far from the near terminal data processing device 5 and can communicate with the distributed data processing terminal 6 via a communication network such as the Internet 600. When the centralized data processing server 7 receives the collation request information indicating the collation request, the collation data, and the collated data, the collation data and the collated data are collated and the similarity is determined. Then, the centralized data processing server 7 transmits collation result information indicating the collation result including the similarity to the distributed data processing terminal 6.

  The service providing server 8 is a server that provides various services to the image acquisition terminal 2.

  The authentication server 9a authenticates the terminal ID (or user ID) in order to determine whether the image acquisition terminal 2 (or user) has a legitimate authority to receive services from the service providing system 8. The same applies to the authentication servers 9b and 9c.

[Hardware configuration]
Next, each hardware configuration of the communication system according to the present embodiment will be described with reference to FIGS.

<Hardware configuration of real-time data processing terminal>
FIG. 2 is a hardware configuration diagram of the real-time data processing terminal. The real-time data processing terminal 3 includes a CPU 301, ROM 302, RAM 303, EEPROM 304, CMOS sensor 305, acceleration / direction sensor 306, media I / F 308, and GPS receiver 309.

  Among these, the CPU 301 controls the operation of the entire real-time data processing terminal 3. The ROM 302 stores a program used for driving the CPU 301. The RAM 303 is used as a work area for the CPU 301. The EEPROM 304 reads or writes various data such as a real-time data processing terminal program under the control of the CPU 301. The CMOS sensor 305 images a subject (mainly the blind spot of the imaging unit 40) under the control of the CPU 301 to obtain captured image data. The acceleration / direction sensor 306 is various sensors such as an electronic magnetic compass, a gyrocompass, and an acceleration sensor that detect geomagnetism. A media I / F 308 controls reading or writing (storage) of data with respect to a recording medium 307 such as a flash memory. The GPS receiver 309 receives GPS signals from GPS satellites.

  The real-time data processing terminal 3 includes an imaging unit I / F 313, a microphone 314, a speaker 315, a sound input / output I / F 316, a display 317, an external device connection I / F 318, and a touch panel 321.

  Among these, the imaging unit I / F 313 is a circuit that controls driving of the imaging unit 40 when an external imaging unit 40 is connected. The microphone 314 is a kind of built-in sound collecting means for inputting sound. The sound input / output I / F 316 is a circuit that processes input / output of a sound signal between the microphone 314 and the speaker 315 under the control of the CPU 301. The display 317 is a kind of display means such as a liquid crystal display or an organic EL display that displays a subject image, various icons, and the like. The external device connection I / F 318 is an interface for connecting various external devices. The touch panel 321 is a kind of input means for operating the real-time data processing terminal 3 when the user presses the display 317.

  The real time data processing terminal 3 includes a bus line 310. The bus line 310 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 301 shown in FIG.

<Hardware configuration of imaging unit>
FIG. 3 is a hardware configuration diagram of the imaging unit. In particular, FIG. 3A is a hardware configuration diagram of a monocular imaging unit 40 a among the imaging units 40. FIG. 3B is a hardware configuration diagram of the compound-eye imaging unit 40 b in the imaging unit 40. The “imaging unit 40” is a general term for a plurality of types of imaging units (imaging units 40a, 40b, etc.) having different numbers of imaging elements or different types of lenses.

  As shown in FIG. 3A, the imaging unit 40a is electrically connected to the imaging element 401a such as CMOS or CCD, the lens 402a, and the imaging unit I / F 313 of the real-time data processing terminal 3. Connection I / F 408a. When the imaging unit 40a is connected to the imaging unit I / F 313 of the real-time data processing terminal 3, the imaging element 401a sends the imaging control signal sent from the imaging unit I / F 313 via the connection I / F 408a. Then, the captured image data is transmitted to the imaging unit I / F 313 via the connection I / F 408a.

  Also, as shown in FIG. 3B, the imaging unit 40b includes imaging elements 401b1 and 401b2, such as CMOS and CCD, lenses 402b1 and 402b2, and the imaging unit I / F 313 of the real-time data processing terminal 3. A connection I / F 408b for electrical connection is provided. The lenses 402b1 and 402b2 are, for example, fisheye lenses. When the imaging unit 40b is connected to the imaging unit I / F 313 of the real-time data processing terminal 3, the imaging element 401b sends an imaging control signal sent from the imaging unit I / F 313 via the connection I / F 408b. In this manner, the image is captured, and a plurality of captured image data is transmitted to the imaging unit I / F 313 via the connection I / F 408b.

  The imaging unit 40a shown in FIG. 3 (a) obtains a general planar image, but the imaging unit 40b shown in FIG. 3 (b) can obtain an omnidirectional image. Here, with reference to FIG. 4, an outline of processing until an equirectangular projection image EC is created from an image captured by the imaging unit 40 b will be described. 4A is a hemispherical image generated by imaging with the imaging unit 40b (front side), and FIG. 4B is a hemispherical image generated by imaging with the imaging unit 40b (rear side). FIG. 4C is a diagram showing an image represented by an equirectangular projection (hereinafter referred to as an “equal-distance cylindrical projection image”).

  As shown in FIG. 4A, the image obtained by the image sensor 401b1 is a hemispherical image (front side) curved by the lens 402b1. Also, as shown in FIG. 4B, the image obtained by the image sensor 403b is a hemispherical image (rear side) curved by the lens 402b2. Then, the hemispherical image (front side) and the hemispherical image inverted by 180 degrees (rear side) are synthesized, and an equirectangular projection image EC is created as shown in FIG.

<Hardware configuration of near terminal data processing device and distributed data processing terminal>
FIG. 5 is a hardware configuration diagram of the near-terminal data processing device or the distributed data processing terminal. Here, since the near terminal data processing device 5 and the distributed data processing terminal 6 are shown to have the same hardware configuration, the near terminal data processing device 5 will be described, and the distributed data processing terminal 6 will be described. Is omitted.

  As shown in FIG. 5, the near-terminal data processing device 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an acceleration / direction sensor 506, a media I / F 508, and a GPS receiving unit 509.

  Among these, the CPU 501 controls the operation of the entire near-terminal data processing device 5. The ROM 502 stores a program used for driving the CPU 501. The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as the near-terminal data processing device program under the control of the CPU 501. The CMOS sensor 505 captures a subject (mainly a self-portrait of a user who operates the near-terminal data processing device 5) according to the control of the CPU 501, and obtains captured image data. The acceleration / direction sensor 506 is various sensors such as an electronic magnetic compass, a gyrocompass, and an acceleration sensor that detect geomagnetism. A media I / F 508 controls reading or writing (storage) of data with respect to a recording medium 507 such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.

  The near-terminal data processing device 5 includes a long-distance communication circuit 511, an antenna 511a of the long-distance communication circuit 511, a CMOS sensor 512, an image sensor I / F 513, a microphone 514, a speaker 515, a sound input / output I / F 516, and a display 517. , An external device connection I / F 518, a near field communication circuit 519, an antenna 519 a of the near field communication circuit 519, and a touch panel 521.

  Among these, the long-distance communication circuit 511 is a circuit that communicates with other devices via the intranet 200. The CMOS sensor 512 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 501 and obtains captured image data. The image sensor I / F 513 is a circuit that controls driving of the CMOS sensor 512. The microphone 514 is a kind of built-in sound collecting means for inputting sound. The sound input / output I / F 516 is a circuit that processes input / output of a sound signal between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a kind of display means such as a liquid crystal display or an organic EL display that displays a subject image, various icons, and the like. The external device connection I / F 518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as NFC (Near Field Radio Communication) or Bluetooth (registered trademark). The touch panel 521 is a kind of input means for operating the near-terminal data processing device 5 when the user presses the display 517.

  The near terminal data processing device 5 includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG.

<Hardware configuration of centralized data processing server, service providing server, and authentication server>
FIG. 6 is a hardware configuration diagram of a centralized data processing server, a service providing server, or an authentication server. Here, since the central data processing server 7, the service providing server 8, and the authentication server 9 are shown to have the same hardware configuration, the central data processing server 7 will be described, and the service providing server 8 and the authentication server 9 will be described. Description of the server 9 is omitted.

  FIG. 6 is a hardware configuration diagram of the centralized data processing server. The central data processing server 7 is constructed by a computer, and as shown in FIG. 6, a CPU 701, ROM 702, RAM 703, HD 704, HDD (Hard Disk Drive) 705, recording medium 706, media I / F 707, display 708, a network I / F 709, a keyboard 711, a mouse 712, a drive 714 corresponding to a CD-RW, DVD, Blu-ray storage medium, and a bus line 710. Since the central data processing server 7 functions as a server, there is no need for input devices such as a keyboard 711 and a mouse 712 and output devices such as a display 708.

  Among these, the CPU 701 controls the operation of the centralized data processing server 7 as a whole. The ROM 702 stores a program used for driving the CPU 701. The RAM 703 is used as a work area for the CPU 701. The HD 704 stores various data such as programs. The HDD 705 controls reading or writing of various data with respect to the HD 704 according to the control of the CPU 701. The media I / F 707 controls reading or writing (storage) of data with respect to a recording medium 706 such as a flash memory. The display 708 displays various information such as a cursor, menu, window, character, or image. A network I / F 709 is an interface for performing data communication using the Internet 600. The keyboard 711 is a kind of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 712 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The drive 714 controls reading of various data with respect to a removable recording medium.

  The centralized data processing server 7 includes a bus line 710. The bus line 710 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 701 shown in FIG.

[Software configuration]
FIG. 7 is a software configuration diagram of the real-time data processing terminal and the near-terminal data processing device in the image acquisition terminal.

  As shown in FIG. 7, the real-time data processing terminal 3 has an OS 300 and an image recognition application (hereinafter, “application” is indicated as “application”) AP1. The image recognition application AP1 operates on the work of the RAM 303 of the real-time data processing terminal 3. Among these, the OS 300 is basic software that provides basic functions and manages the entire real-time data processing terminal 3. The image recognition application AP1 is an application for recognizing a face such as a person or an animal from a captured image.

  The near terminal data processing device 5 has an OS 500 and a communication application AP2. The communication application AP2 operates on the work of the RAM 503 of the near-terminal data processing device 5. Among these, the OS 500 is basic software that provides basic functions and manages the entire near-terminal data processing device 5. The communication application AP2 is an application for communicating with other terminals (devices) such as the distributed data processing terminal 6.

  Thus, in the image acquisition terminal 2, the real-time data processing terminal 3 performs image recognition, whereas the near-terminal data processing device 5 communicates with the distributed data processing terminal 6 via the intranet 200. Distributed processing can be performed.

  Note that the real-time data processing terminal 3 and the near-terminal data processing device 5 include not only an OS but also software such as a driver, a software development kit (SDK), or an application programming interface (API). The real-time data processing terminal 3 and the near-terminal data processing device 5 may have different installed software, for example, each having a different OS.

  Next, the functional configuration and processing of an embodiment of the present invention will be described using the drawings.

[Functional Configuration of Embodiment]
First, the functional configurations of the terminal, the device, and the server that constitute the communication system according to the present embodiment will be described with reference to FIGS. FIG. 8 is a functional block diagram of the communication system according to the embodiment. FIG. 8 particularly shows functional blocks of the image acquisition terminal 2.

<Functional configuration of real-time data processing terminal>
As shown in FIG. 8, the real-time data processing terminal 3 includes a determination unit 33, an image processing unit 34, an object detection unit 35, an event generation unit 36, a display control unit 37, a connection unit 38, and a storage / reading unit. 39, a blur processing unit 47, and a communication unit 48. Each of these units is a function or means realized by any one of the constituent elements shown in FIG. 2 operating according to a command from the CPU 301 according to a program expanded from the EEPROM 304 onto the RAM 303.

  The real-time data processing terminal 3 has a storage unit 3000 constructed by the ROM 302, the RAM 303, and the EEPROM 304 shown in FIG. The storage unit 3000 stores shape model data described later. Further, in the storage unit 3000, an image sensor information management DB 3001, a cycle value management DB 3002, an image acquisition program management DB 3003, a composition processing program management DB 3004, a distortion correction program management DB 3005, and a service program management DB 3006 are constructed.

  The image sensor information management DB 3001 is configured by an image sensor information management table described later. The cycle value management DB 3002 is configured by a cycle value management table described later. The image acquisition program management DB 3003 is configured by an image acquisition program management table described later. The synthesis processing program management DB 3004 is configured by a synthesis processing program management table described later. The distortion correction program management DB 3005 is configured by a distortion correction program management table described later. The service program management DB 3006 is configured by a service program management table described later.

(Image sensor information management table)
FIG. 10A is a conceptual diagram showing an image sensor information management table. In the imaging element information management table, the model number of the imaging unit 40, the number of imaging elements included in the imaging unit 40, and the type of lens used in the imaging element are managed in association with each other. The model number is an example of type information indicating the type of the imaging unit corresponding to the number of imaging elements and the difference in the type of lens used in the imaging element. In addition, as long as the number of imaging elements and the type of lens can be specified, identification information such as a product number or a bar code is not limited to a model number.

(Cycle value management table)
FIG. 10B is a conceptual diagram showing a cycle value management table. In this cycle value management table, the number of image sensors included in the imaging unit 40 and a cycle value (Frames Per Second) of an object recognition process described later are managed in association with each other.

  Each table shown in FIG. 11 is a table for managing whether or not each program is installed in the real-time data processing terminal 3.

(Image acquisition program management table)
FIG. 11A is a conceptual diagram showing an image acquisition program management table. In the image acquisition program management table, the number of image pickup elements included in the image pickup unit 40 and the image acquisition program (or this name) are managed in association with each other. For example, when the image acquisition program for the number of image sensors “1” is installed in the real-time data processing terminal 3, the program “ProgC01 (one system image acquisition)” is associated and managed. When the image acquisition program for the number of image pickup devices “2” is installed in the real-time data processing terminal 3, the program “ProgC02 (2 systems of image acquisition)” is associated and managed.

(Synthesis processing program management table)
FIG. 11B is a conceptual diagram showing a synthesis processing program management table. In the synthesis processing program management table, the number of image sensors included in the imaging unit 40 and the name of the synthesis processing program are managed in association with each other.

(Distortion correction program management table)
FIG. 11C is a conceptual diagram showing a distortion correction program management table. In this distortion correction program management table, the lens type of the imaging unit 40 and the distortion correction program (or this name) are managed in association with each other. For example, when a distortion correction program for the lens type “wide angle” is installed in the real-time data processing terminal 3, the program “ProgW01 (wide angle distortion correction)” is associated and managed. When the distortion correction program for the lens type “fisheye” is installed in the real-time data processing terminal 3, the program “ProgW02 (fisheye distortion correction)” is associated and managed.

(Service program management table)
FIG. 11D is a conceptual diagram showing a service program management table. In this service program management table, the authentication server ID for identifying the authentication server 9 and the name of the service program for executing the service provided by the image acquisition terminal 2 are managed in association with each other. For example, when the service program for the authentication server ID “a01” is installed in the real-time data processing terminal 3, the program “ProgD01 (object recognition)” is associated and managed.

  When the service program for the authentication server ID “a02” is installed in the real-time data processing terminal 3, the program “ProgD02 (object blurring)” is associated and managed. The object blurring service content is that the centralized data processing server 7 stores the image data after the blurring of the object. Further, when the processing of the object blurring is performed on the centralized data processing server 7 side (see S108). In some cases, the processing is performed by the real-time data processing terminal 3 (see S125).

(Functional configuration of real-time data processing terminal)
Next, each functional configuration of the real-time data processing terminal 3 will be described in more detail with reference to FIG.

  The determination unit 33 of the real-time data processing terminal 3 is realized by the processing of the CPU 301 and makes various determinations. For example, the determination unit 33 determines the number of imaging elements based on the model number sent from the imaging unit 40 with reference to the imaging element information management DB 3001.

  The image processing unit 34 is realized by the processing of the CPU 301, and performs various types of image processing by executing the respective programs (image acquisition program, image synthesis program, distortion correction program, and service program) shown in FIG. Specifically, the image processing unit 34 executes a first program (for example, an image acquisition program, an image synthesis program, and a distortion correction program) that does not require authentication in order to acquire the image data. First image processing (for example, image acquisition, image synthesis, distortion correction) is performed. In addition, the image processing unit 34 executes a second program (for example, a service program) that requires authentication in order to acquire the second image processing (for example, object recognition processing, object (Blur processing).

  The object detection unit 35 is realized by the processing of the CPU 301, and detects feature points that are candidates for a predetermined specific object such as a face in the captured image data acquired from the imaging unit 40 via the connection unit 38. Then, by referring to shape model data indicating a predetermined shape model of a specific object such as a face, and detecting the position of the object in the captured image, the rectangle of the region of the object such as the face from the captured image An image (hereinafter referred to as “object region rectangular image”) is detected (or extracted).

  The event generation unit 36 is realized by the processing of the CPU 301, and generates detection information (event information) indicating that the position of the object has been detected by the image processing unit 34.

  The display control unit 37 is realized by the processing of the CPU 301 and displays various screens on the display 317 which is an example of a display unit.

  The connection unit 38 is realized by processing of the imaging unit I / F 313 and the CPU 301, and is an interface for mechanically and electrically connecting the imaging unit 40 to the real-time data processing terminal 3.

  The storage / reading unit 39 is realized by processing of the CPU 301, and stores various data (or information) in the storage unit 3000 and reads out various data (or information) from the storage unit 3000.

  The blur processing unit 47 is realized by the processing of the CPU 301 and executes a process of blurring part or all of the image received by the transmission / reception unit 41.

  The communication unit 48 is realized by processing of the external device connection I / F 318 and the CPU 301, and transmits / receives various data (or information) to / from a communication unit 58 described later of the near-terminal data processing device 5 through one-to-one communication. Do. Note that this communication may be wireless as well as wired.

<Functional configuration of near-terminal data processing device>
As shown in FIG. 8, the near-terminal data processing device 5 includes a transmission / reception unit 51, a data detection unit 56, a display control unit 57, a communication unit 58, and a storage / readout unit 59. Each of these components is realized by any one of the constituent elements shown in FIG. 5 operating in the near-terminal data processing device 5 according to a command from the CPU 501 according to a program expanded from the EEPROM 504 to the RAM 503. Function or means to be performed.

  Further, the near-terminal data processing device 5 has a storage unit 5000 constructed by the ROM 502, the RAM 503, and the EEPROM 504 shown in FIG.

(Each functional configuration of near terminal data processing device)
Next, each functional configuration of the near-terminal data processing device 5 will be described in more detail with reference to FIG.

  The transmission / reception unit 51 of the near-terminal data processing device 5 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501, and is connected to the distributed data processing terminal 6 via the communication network (here, the intranet 200). Send and receive various data (or information).

  The data detection unit 56 is realized by the processing of the CPU 501, and detects whether an event for receiving data from the real-time data processing terminal 3 has occurred, or whether reception of data has been completed. In addition, the data detection unit 56 identifies the service content based on the content of the authentication server ID sent from the distributed data processing terminal 6.

  The display control unit 57 is realized by the processing of the CPU 501 and displays various screens on the display 517 which is an example of a display unit.

  The communication unit 58 is realized by the processing of the external device connection I / F 518 and the CPU 501, and transmits / receives various data (or information) to / from the communication unit 48 of the real-time data processing terminal 3 through one-to-one communication. Note that this communication may be wireless as well as wired.

  The storage / reading unit 59 is realized by processing of the CPU 501, and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000.

<Functional configuration of distributed data processing terminal>
As illustrated in FIG. 9, the distributed data processing terminal 6 includes a transmission / reception unit 61, a reception unit 62, a determination unit 63, a display control unit 67, and a storage / readout unit 69. Each of these units is realized by any one of the constituent elements shown in FIG. 5 operating in the distributed data processing terminal 6 according to a command from the CPU 501 according to a program expanded from the EEPROM 504 to the RAM 503. Function or means.

  The distributed data processing terminal 6 has a storage unit 6000 constructed by the ROM 502, the RAM 503, and the EEPROM 504 shown in FIG. The storage unit 6000 stores collation data. Furthermore, a collation data management DB 6001 is constructed in the storage unit 6000. The collation data management DB 6001 is configured by a collation data management table described later. The collation data may be stored in a data management server other than the distributed data processing terminal 6.

(Verification data management table)
FIG. 13 is a conceptual diagram showing a collation data management table. In this collation data management table, an image file indicating the collation data file name and the name of the person indicated by the collation image are managed in association with each other.

(Functional configuration of distributed data processing terminal)
The transmission / reception unit 61 of the distributed data processing terminal 6 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501 in the distributed data processing terminal 6, and centralized data via a communication network (here, the Internet 600). Various data (or information) is transmitted / received to / from the processing server 7. For example, the transmission / reception unit 61 transmits a collation request between the collation data and the data to be collated to the central data processing server 7 or performs a process on the collation result sent from the central data processing server 7.

  The accepting unit 62 is realized by processing of the touch panel 521 and the CPU 501 in the distributed data processing terminal 6 and accepts various operations of the user.

  The determination unit 63 is realized by the processing of the CPU 501 in the distributed data processing terminal 6 and makes various determinations.

  The display control unit 67 is realized by the processing of the CPU 501 in the distributed data processing terminal 6 and displays various screens on the display 517 which is an example of a display unit.

  The storage / reading unit 69 is realized by processing of the CPU 501 in the distributed data processing terminal 6, and stores various data (or information) in the storage unit 6000 and reads various data (or information) from the storage unit 6000. For example, the storage / reading unit 69 stores the collation data (here, the face image data) in the storage unit 6000 and registers it according to the registration request received by the receiving unit 62.

<Functional configuration of centralized data processing server>
As shown in FIG. 9, the centralized data processing server 7 includes a transmission / reception unit 71, a reception unit 72, a feature amount generation unit 74, a collation unit 75, an object detection unit 76, a blur processing unit 77, and a storage / reading unit. 79. Each of these units is realized by any one of the components shown in FIG. 6 operating in the distributed data processing terminal 6 according to a command from the CPU 701 according to a program expanded from the HD 704 to the RAM 703. Function or means.

  The central data processing server 7 has a storage unit 7000 constructed by the ROM 702, the RAM 703, and the HD 704 shown in FIG. In the storage unit 7000, a processing information management DB 7001 is constructed. The processing information management DB 7001 is configured by a processing information management table described later. Further, the storage unit 7000 stores characteristic data on the collation side described later.

(Processing information management table)
Each table shown in FIG. 12 is stored in the storage unit 7000 of the centralized data processing server 7 and is a table for managing the processing contents performed by the real-time data processing terminal 3.

  FIG. 12A is a table for executing the face recognition service. This table is managed by the real-time data processing terminal 3 in association with the service name “object recognition” and processing information. The processing information in this case includes whether to execute object detection processing (for example, face detection processing), the number of horizontal pixels and vertical pixels of the generated face area cut-out image, and the generated face area cut-out image. Each information of the projection method is included.

  FIGS. 12B and 12C are tables for executing the object blurring service. In this table, the service name “object blur” and the processing information are managed in association with each other. The processing information in this case includes whether the real-time data processing terminal 3 performs object blurring processing (for example, face blurring processing), the number of horizontal pixels and the number of vertical pixels of the generated captured image, and the generated imaging Each information of the image projection method is included. When the real-time data processing terminal 3 does not execute the object blurring process (see OFF in FIG. 12B), the centralized data processing server 7 provides the object blurring service instead. When the real-time data processing terminal 3 performs the object blurring process (see ON in FIG. 12C), the real-time data processing terminal 3 performs the object blurring process. In the case of the object blur service, a service in which the central data processing server 7 stores the image data after the object blur process regardless of whether or not the real-time data processing terminal 3 executes the object blur process (for example, the face blur process). I do.

  When the processing capability of the centralized data processing server 7 is relatively high, the tables shown in FIGS. 12A and 12B are managed in advance in the processing information management DB 7001. On the other hand, when the processing capability of the centralized data processing server 7 is relatively low, the tables shown in FIGS. 12A and 12C are managed in the processing information management DB 7001 in advance.

  The detailed contents of the perspective projection are disclosed in Japanese Patent 3012142, and the detailed contents of the equirectangular projection are disclosed in Japanese Patent 3025255.

(Functional configuration of centralized data processing server)
The transmission / reception unit 71 of the centralized data processing server 7 is realized by the processing of the network I / F 709 and the CPU 701, and communicates with the distributed data processing terminal 6 and various data (or information) via a communication network (in this case, the Internet 600). Send and receive. The transmission / reception unit 71 receives a collation request between the collation data and the collated data from the distributed data processing terminal 6 or transmits collation result information indicating the collation result to the distributed data processing terminal 6.

  The accepting unit 72 is realized by processing of the keyboard 711, the mouse 712, and the CPU 701, and accepts various operations of the user.

  The feature quantity generation unit 74 is realized by the processing of the CPU 701, and generates a feature quantity parameter from the collated data (partial image data) and the collation data received by the transmission / reception unit 71.

  The collation unit 75 is realized by the processing of the CPU 701, and collates the feature amount on the collation data side with the feature amount on the collation data side using the feature amount generated by the feature amount generation unit 74, and shows a score indicating similarity (Point) is calculated.

  The object detection unit 76 detects an object (for example, a face) from the image received by the transmission / reception unit 71.

  The blur processing unit 77 executes a process of blurring a part or all of the image received by the transmission / reception unit 71.

  The storage / reading unit 79 is realized by the processing of the CPU 701, stores various data (or information) in the storage unit 7000, and reads out various data (or information) from the storage unit 7000.

<Functional configuration of service providing server>
Next, the functional configuration of the service providing server 8 will be described in detail with reference to FIGS. 9 and 14 to 16. As illustrated in FIG. 9, the service providing server 8 includes a transmission / reception unit 81, a determination unit 82, an extraction unit 87, and a storage / reading unit 89. Each of these units is a function realized by any one of the constituent elements shown in FIG. 6 being operated by an instruction from the CPU 701 according to the program for the service providing server 8 expanded from the HD 704 to the RAM 703, or Means.

  Further, the service providing server 8 includes a storage unit 8000 constructed by the RAM 703 and the HD 704 shown in FIG. The storage unit 8000 stores various data sent from the distributed data processing terminal 6 or the authentication server 9. The storage unit 8000 stores all the programs shown in FIG. 11, and the service providing server 8 transmits the requested program in response to a request from the real-time data processing terminal 3. Can do.

  Further, the storage unit 8000 includes a session management DB 8001 and an authentication server management DB 8002. Among these, the session management DB 8001 is configured by a session management table described later. The authentication server management DB 8002 is configured by an authentication server management table described later. Hereinafter, each table will be described in detail.

(Session management table)
FIG. 14 is a conceptual diagram showing a session management table. In this session management table, in order to provide a service to the distributed data processing terminal 6, a session ID for identifying a communication session established with the distributed data processing terminal 6, a terminal ID for identifying the distributed data processing terminal 6, And the IP address of the user's distributed data processing terminal 6 indicated by the terminal ID is stored and managed in association with each other.

  FIGS. 15A, 15B, and 15C show an e-mail address as an example of a terminal ID (identification), each of which includes an authentication target part and an authentication target non-part. The authentication target portion is a user ID used when being authenticated by the authentication server 9. The non-authentication target part is a part that is not used when being authenticated by the authentication server 9.

  Among these, in the first pattern shown in FIG. 15A, the authentication target part is composed of the account name “asai”, the host name “myhost”, and the domain name “ricoo.com”. ing. On the other hand, the non-authentication part is composed of the latter part “theta1” of the domain name. In this case, the extraction unit 87 distinguishes between the authentication target part and the non-authentication part by “/”.

  FIG. 15B also shows the first pattern, but the non-authentication part is different from FIG. 15A. That is, since the authentication target part is the same between the terminal ID shown in FIG. 15 (a) and the terminal ID shown in FIG. 15 (b), the authentication server 9 authenticates as the same ID.

  The terminal ID may be the second pattern shown in FIG. In this second pattern, the authentication target part is composed of the preceding part “asai” of the account name. On the other hand, the non-authenticated part is composed of the latter part “theta2” of the account name, the host name “myhost”, and the domain name “ricoo.com”. In this case, the extraction unit 87 distinguishes the authentication target part and the non-authentication part by “+”.

(Authentication server management table)
FIG. 16 is a conceptual diagram showing an authentication server management table. In this authentication server management table, a URL (Uniform Resource Locator) for accessing each authentication server 9 is stored and managed in association with each authentication server ID for identifying each authentication server 9.

(Functional configuration of service providing server)
Next, each functional configuration of the service providing server 8 will be described in detail with reference to FIG.

  The transmission / reception unit 81 of the service providing server 8 is mainly realized by a command from the CPU 701 shown in FIG. 6 and a network I / F 709, and via the communication network 9, the distributed data processing terminal 6 or the authentication server 9. Send and receive various data (or information).

  The determination unit 82 is realized mainly by a command from the CPU 701 shown in FIG. 6. For example, the determination unit 82 determines whether a communication session for providing a service to the distributed data processing terminal 6 has already been established. .

  The extracting unit 87 is realized mainly by a command from the CPU 701 shown in FIG. 6 and performs a process of extracting a user ID (authentication target part) as shown in FIG. 15 from the terminal ID.

  The storage / reading unit 89 is realized mainly by the instruction from the CPU 701 and the HDD 705 shown in FIG. 6, and stores various data in the storage unit 8000 and reads various data from the storage unit 8000.

<Functional configuration of authentication server>
Next, the functional configuration of the authentication server 9 will be described in detail with reference to FIGS. 9 and 11. The authentication server 9 includes a transmission / reception unit 91, an authentication unit 92, and a storage / reading unit 99. Each of these units is a function or means realized by any of the constituent elements shown in FIG. 6 being operated by a command from the CPU 701 according to the authentication server 9 program expanded from the HD 704 onto the RAM 703. It is.

  Further, the authentication server 9 includes a storage unit 9000 constructed by the RAM 703 and the HD 704 shown in FIG. The storage unit 9000 stores various data sent from the distributed data processing terminal 6 or the service providing server 8.

  Further, an authentication management DB 9001 is constructed in the storage unit 9000. The authentication management DB 9001 is configured by an authentication management table described later. Hereinafter, this table will be described in detail.

(Authentication management table)
17A is a conceptual diagram showing an authentication management table possessed by the authentication server 9a, FIG. 17B is a conceptual diagram showing an authentication management table possessed by the authentication server 9b, and FIG. 17C is an authentication possessed by the authentication server 9c. It is a conceptual diagram which shows a management table.

  In each authentication management table, the user ID (part to be authenticated) of the terminal ID and the password are stored and managed in association with each other.

(Functional configuration of authentication server)
Next, each functional configuration of the authentication server 9 will be described in detail with reference to FIG.

  The transmission / reception unit 91 of the authentication server 9 is realized mainly by a command from the CPU 701 shown in FIG. 6 and a network I / F 709, and is distributed with the distributed data processing terminal 6 and the service providing server 8 via the Internet 600. Send and receive data (or information).

  The authentication unit 92 is realized mainly by a command from the CPU 701 shown in FIG. 6 and determines whether the image acquisition terminal 2 that has transmitted the authentication request has a legitimate authority to receive the service. Then, ID authentication is performed. The authentication server 9 performs such authentication because the distributed data processing terminal 6 manages the image acquisition terminal 2.

  The storage / reading unit 99 is mainly realized by the command from the CPU 701 shown in FIG. 6 and the HDD 705, and stores various data (or information) in the storage unit 9000, and stores various data (or information) from the storage unit 9000 ( Or information).

[Processing or operation of this embodiment]
Next, the processing or operation of this embodiment will be described with reference to FIGS.

<Authentication process>
First, the authentication process will be described with reference to FIGS. 18 and 19 are sequence diagrams illustrating the authentication process. 20 and 21 are diagrams showing examples of screens of the distributed data processing terminal.

  As shown in FIG. 18, a request for an authentication destination selection screen is transmitted from the transmission / reception unit 61 of the distributed data processing terminal 6 to the service providing server 8 (step S21). This request includes the terminal ID of the distributed data processing terminal 6. At this time, the transmission / reception unit 61 transmits the IP address of the terminal itself. Thereby, the transmission / reception unit 81 of the service providing server 8 receives the request for the authentication destination selection screen and the IP address of the distributed data processing terminal 6.

  Next, the determination unit 82 of the service providing server 8 determines whether or not the terminal ID received in Step S21 is managed in association with a predetermined session ID in the session management table (see FIG. 14) (Step S21). S22). Hereinafter, a case where the terminal ID is not managed will be described.

  The transmission / reception unit 81 of the service providing server 8 transmits the authentication destination selection screen data to the distributed data processing terminal 6 (step S23). Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the data of the authentication destination selection screen.

  Next, the display control unit 63 of the distributed data processing terminal 6 causes the display 517 to display an authentication destination selection screen s1 as shown in FIG. 20 (step S24). FIG. 20 shows a screen example as the distributed data processing terminal 6. On the authentication destination selection screen s1, a terminal ID input field b1, a password input field b2, and a login button b3 for requesting login (authentication request) are displayed. Furthermore, authentication server selection buttons a1, a2, and a3 for selecting authentication servers 9a, 9b, and 9c are displayed on the authentication destination selection screen s1. For example, the authentication server selection button a1 is a button used when the user receives an object authentication processing service. The authentication server selection button a2 is a button when the user receives an object blurring service.

  Here, the user inputs his / her terminal ID in the input field b1 and his / her password in the input field b2, presses a desired button among the authentication server selection buttons a1, a2 and a3, and presses the login button b3. Then, the reception part 62 receives each input and selection (step S25). Here, a case will be described in which object detection processing is executed by the object detection processing program ProgD01 in the image recognition processing by selecting the authentication server selection button a1.

  The transmission / reception unit 61 transmits an authentication request for an ID (here, a terminal ID or a user ID) to the service providing server 8 (step S26). This authentication request includes the terminal ID and password received in step S25, the selection result of the authentication server 9, and the URL of the distributed data processing terminal 6. This authentication result indicates an authentication server ID for identifying the authentication server 9. Thereby, the transmission / reception unit 81 of the service providing server 8 receives the ID authentication request.

  Next, the storage / reading unit 89 of the service providing server 8 responds by searching the authentication server management table (see FIG. 16) using the authentication server ID as the selection result received in step S26 as a search key. The URL of the authentication server is read (step S27).

  Next, the extraction unit 87 extracts only the user ID (authentication target part) from the terminal ID received in step S26 (step S28). Then, the transmission / reception unit 81 transmits an ID authentication request to the authentication server 9 indicated by the URL read in step S27 (step S29). This ID authentication request includes the user ID (authentication target part) extracted in step S28, the password received in step S26, and the URL of the distributed data processing terminal 6 received in step 26. . Thereby, the transmission / reception part 71 of the authentication server 9 receives a user's authentication request.

  Next, the storage / reading unit 99 of the authentication server 9 uses the combination of the user ID (authentication target part) and the password received in step S29 as a search key, and the same set of authentications in the authentication management table (see FIG. 17). The authentication unit 92 performs authentication using the search result of the target part and the password (step S30). When the same set is managed, the authentication unit 92 determines that the distributed data processing terminal 6 is a valid terminal for receiving a service from the service providing server 8, and when the same set is not managed. Determines that the distributed data processing terminal 6 is not a valid terminal for receiving a service from the service providing server 8.

  In step S28, the extraction unit 87 extracts the authentication target part from the terminal ID, but the present invention is not limited to this. For example, the service providing server 8 does not have the extraction unit 87, and in step S29, the transmission / reception unit 81 transmits only the user ID (authentication target portion) of the terminal ID in addition to the password and the URL. It may be.

  Subsequently, as shown in FIG. 19, the authentication unit 92 of the authentication server 9 encrypts the token (transmission right) (step S41). Then, based on the URL of the distributed data processing terminal 6 received in step S29, the transmission / reception unit 91 transmits an authentication result to the distributed data processing terminal 6 (step S42). This authentication result indicates whether or not the distributed data processing terminal 6 is valid, and includes the token encrypted in step S41. Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the authentication result of the user. Hereinafter, a case where the user has a valid authority will be described.

  The transmission / reception unit 61 of the distributed data processing terminal 6 transmits a session establishment request to the service providing server 8 (step S43). This session establishment request includes the terminal ID and the encrypted token received in step S42. Thereby, the transmission / reception unit 81 of the service providing server 8 receives the session establishment request.

  Next, the service providing server 8 confirms that the distributed data processing terminal 6 that has transmitted the session establishment request is legitimate in step S30. An authentication request is transmitted (step S44). This token authentication request includes the encrypted token received in step S43. Thereby, the transmission / reception unit 91 of the authentication server 9 receives the token authentication request.

  Next, the authentication unit 92 decrypts the encrypted token received in step S44 (step S45). Then, the authentication unit 92 authenticates the token by comparing the token before encryption in step S41 with the token after decryption in step S45 (step S46). Then, the transmission / reception unit 91 of the authentication server 9 transmits the authentication result of step S46 to the service providing server 8 (step S47). Thereby, the transmission / reception unit 81 of the service providing server 8 receives the authentication result. Hereinafter, the case where it is determined in step S46 that the token is valid will be described.

  Next, the storage / reading unit 89 of the service providing server 8 assigns a new session ID in the session management table (see FIG. 14), and the terminal ID and IP received in step S26 for this session ID. Addresses are managed in association with each other (step S48). Then, the transmission / reception unit 81 transmits the service provision screen data to the distributed data processing terminal 6 (step S49). Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the data of the service providing screen.

  Next, the display control unit 63 of the distributed data processing terminal 6 displays the service provision screen s2 as shown in FIG. 21 on the display 517 (step S50). FIG. 21 shows a screen example of the distributed data processing terminal 6. Here, as a service providing example, a remote operation service for realizing remote operation from the distributed data processing terminal 6 to the image acquisition terminal 2 will be described. In the service providing screen s2 shown in FIG. 21, a terminal ID input field c1 for identifying a remote operation target and a “remote operation start” button c2 are displayed.

<Image recognition preparation process>
(Service startup processing)
It is the sequence diagram which showed the process of the start request | requirement of image recognition using FIG.

  First, in the distributed data processing terminal 6, the accepting unit 62 accepts a service start request relating to the service name designated by the user (step S61). In this case, the GUI (Graphical User Interface) of the distributed data processing terminal 6 is used. Then, the transmission / reception unit 61 of the distributed data terminal 6 transmits a request for processing information to the centralized data processing server 7 (step S62). At this time, a service name for specifying a service to start (object recognition, face blurring, etc.) is transmitted. Thereby, the transmission / reception unit 71 of the centralized data processing server 7 receives a request for processing information.

  The storage / reading unit 79 of the centralized data processing server 7 reads the corresponding processing information by searching the processing information management DB 7001 using the service name received in step S62 as a search key (step S63). For example, when the face recognition service is selected, the table of FIG. 12A is read, and when the object blurring service is selected, the table of FIG. 12B or FIG. 12C is read. In FIG. 12B and FIG. 12C, the contents of the table differ depending on the performance of the server on which the object blurring service is executed. FIG. 12B is used when the performance of the centralized data processing server 7 is relatively high, and the real-time data processing terminal 3 does not execute the face blurring process. On the other hand, FIG. 12C is used when the performance of the centralized data processing server 7 is relatively low, and the real-time data processing terminal 3 executes the face blurring process.

  Next, the transmission / reception unit 71 of the centralized data processing server 7 transmits processing information to the distributed data processing terminal 6 (step S64). As a result, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the processing information.

  The transmission / reception unit 61 of the distributed data processing terminal 6 transmits a processing preparation request to the near-terminal data processing device 5 of the image acquisition terminal 2 (step S65). As a result, the transmission / reception unit 21 of the image acquisition terminal 2 receives the processing preparation request. This request includes the service name specified in step S61 and the processing information received in step S64. The communication unit 58 of the near-terminal data processing device 5 transfers the processing information to the communication unit 48 of the real-time data processing terminal 3 (Step S66). Then, the storage / reading unit 39 of the real-time data processing terminal 3 sets a process to be executed according to the received process information (step S67). Details of the execution process setting will be described later for each service.

  In addition, after the setting in step S67, the communication unit 48 of the real time data processing terminal 3 transmits a setting completion notification to the near terminal data processing device 5 (step S68). As a result, the transmission / reception unit 51 of the near-terminal data processing device 5 receives the setting completion notification.

  Next, the transmission / reception unit 51 of the near-terminal data processing device 5 transmits a processing preparation completion notification to the distributed data processing terminal 6 (step S69). As a result, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the processing preparation completion notification.

  The transmission / reception unit 61 of the distributed data processing terminal 6 transmits a service start request to the centralized data processing server 7 (step S70). Thereby, the transmission / reception unit 71 of the centralized data processing server 7 receives the service start request. Further, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits a processing start request to the near-terminal data processing device 5 (step S71). The communication unit 58 of the near-terminal data processing device 5 transfers a process start request to the communication unit 48 of the real-time data processing terminal 3 (step S72). As a result, the entire service is started. Details of the service processing will be described later for each service.

  Next, processing for a total of three patterns, one pattern for the face recognition service and two patterns for the object blurring service, will be described.

(Real-time data processing terminal setting process)
The setting in the image acquisition terminal 2, specifically, the sequence of step S67 in FIG. 22 will be described for each service.

[A] Example of Object Recognition Service When the transmission / reception unit 51 of the near terminal data processing device 5 receives the processing information (see S65), the service specifying unit 54 of the near terminal data processing device 5 sets the service name (object recognition). A service to be executed (here, a face recognition service) is specified based on the information. Then, the communication unit 58 of the near-terminal data processing device 5 sends the following information to the communication unit 48 of the real-time data processing terminal 3 according to the processing information indicating the contents of the processing setting table (see FIG. 12A). Set up.
(1) The object detection unit 35 completes preparation for processing.
(2) The image processing unit 34 sets the number of horizontal pixels and the number of vertical pixels of the detected object area rectangular image to 200. In this example, an optimum size is set for an algorithm of object (for example, face) recognition processing performed by the collation unit 75 of the centralized data processing server 7.
(3) The image processing unit 34 sets the projection method of the object area rectangular image detected by the object detection unit 35 to the perspective projection method. This is a setting for obtaining a correct collation result in the collation unit 75 of the centralized data processing server 7.

[B] Example of Object Blur Service 1 When the transmitting / receiving unit 51 of the near terminal data processing device 5 receives the processing information (see S65), the service specifying unit 54 of the near terminal data processing device 5 reads the service name (object blur ( Server)) to identify the service to be executed (here, the object blurring and storage service by the centralized data processing server 7). Then, the communication unit 58 of the near-terminal data processing device 5 sends the following information to the communication unit 48 of the real-time data processing terminal 3 according to the processing information indicating the contents of the processing setting table (see FIG. 12B). Set up.
(1) The blur processing unit 47 sets invalidity (OFF) of the object blur processing.
(2) The image processing unit 34 sets the number of horizontal pixels of the entire image to be captured to 3840 and the number of vertical pixels to 2160 for the imaging unit 40 via the connection unit 38. In this case, the maximum size that can be captured by the imaging unit 40 is set, but the size may be reduced.
(3) The image processing unit 34 sets the projection method of the captured image data acquired from the imaging unit to equirectangular projection. The conversion of the projection method may be performed by the service providing server 8.

[C] Example of Object Blur Service 2 When the transmitting / receiving unit 51 of the near terminal data processing device 5 receives the processing information (see S65), the service specifying unit 54 of the near terminal data processing device 5 reads the service name (object blur ( Terminal)) to be executed (here, object blurring by the real-time data processing terminal 3 and storage service by the centralized data processing server 7). Then, the communication unit 58 of the near-terminal data processing device 5 sends the following information to the communication unit 48 of the real-time data processing terminal 3 according to the processing information indicating the contents of the processing setting table (see FIG. 12C). Set up.
(1) The blur processing unit 47 sets the validity (ON) of the object blur processing.
(2) The image processing unit 34 sets the number of horizontal pixels of the entire image to be captured to 3840 and the number of vertical pixels to 2160 for the imaging unit 40 via the connection unit 38. In this case, the maximum size that can be captured by the imaging unit 40 is set, but the size may be reduced.
(3) The image processing unit 34 sets the projection method of the captured image data acquired from the imaging unit 40 to equirectangular projection. The conversion of the projection method may be performed by the service providing server 8.

<Processing performed by the real-time data processing terminal during service execution>
A process in which each service is set and executed will be described with reference to FIGS.

[A] Example of Object Recognition Service FIG. 23 is a sequence diagram showing processing of the object recognition service when the processing setting table (FIG. 12A) is used for setting the real-time data processing terminal 3.

  First, captured image data is generated by being captured by the imaging unit 40, and the connection unit 38 of the real-time data processing terminal 3 acquires captured image data from the imaging unit 40 (step S81).

  Next, the object detection unit 35 of the real-time data processing terminal 3 performs an object (here, face) detection process on the captured image (step S82). When a target detection target (here, a face) is detected in this process, the image processing unit 34 performs projection method conversion on the captured image data (step S83). In this case, the image processing unit 34 performs conversion from the equidistant projection obtained by the imaging unit 40 to the above-described already set perspective projection.

  Next, the image processing unit 34 performs resizing processing so that the object region rectangular image, which is a partial image of the object (face), has the number of horizontal pixels and the number of vertical pixels set by the processing setting table. Data to be verified, which is a partial image of (face), is created (step S84). The number of pixels to be resized is determined in advance so that the centralized data processing server 7 can easily collate with the collation data later.

  The image processing unit 34 encodes (encodes) the partial image data that is the object region rectangular image of the object (face) (step S85).

  The communication unit 48 of the real-time data processing terminal 3 transmits an imaging process completion notification to the communication unit 58 of the near-terminal data processing device 5 (step S86). This processing completion notification includes data to be verified as partial image data encoded in step S85. As a result, the communication unit 58 of the near-terminal data processing device 5 receives the processing completion notification.

  Next, the transmission / reception unit 51 of the near-terminal data processing device 5 transmits an imaging process completion notification to the distributed data processing terminal 6 (step S87). This process completion notification includes the collated data received in step S86. Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the processing completion notification.

  Next, the storage / reading unit 69 of the distributed data processing terminal 6 reads the collation data from the storage unit 6000 (step S88). And the transmission / reception part 61 transmits a collation request | requirement with respect to the centralized data processing server 7 (step S89). This collation request includes the collated data received in step S87 and the collation data read in step S88. Thereby, the transmission / reception unit 71 of the centralized data processing server 7 receives the collation request.

  Next, the collation unit 75 of the centralized data processing server 7 collates the collation data received in step S89 with the data to be collated, and calculates a score indicating the degree of similarity between the data (step S90). After the similarity is calculated by the collation unit 75, the transmission / reception unit 7 transmits a collation result notification to the distributed data terminal 6 (step S91). This collation result notification includes the similarity calculated in step S90. Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the verification result notification.

  Next, the display control unit 67 of the distributed data processing terminal 6 displays a matching result on the display 517 of the distributed data processing terminal 6 based on the similarity received in step S91 (step S92).

[B] Example of Face Blur Processing Pattern 1 FIG. 24 is a sequence diagram showing the processing of the object blur service when the processing setting table (FIG. 12B) is used for setting the real-time data processing terminal 3. . FIG. 26 is a diagram showing a display example of an image on the distributed data processing terminal.

  First, captured image data is generated by being captured by the imaging unit 40, and the connection unit 38 of the real-time data processing terminal 3 acquires captured image data from the imaging unit 40 (step S101). In the case of an object (face) blur service, an interval imaging function or the like is used.

  Next, the image processing unit 34 of the real-time data processing terminal 3 performs projection method conversion on the captured image data (step S102). In this case, the image processing unit 34 performs conversion from the equidistant projection obtained by the imaging unit 40 to the above-mentioned equirectangular projection described above.

  The image processing unit 34 encodes the captured image data before the blurring process (step S103).

  The communication unit 48 of the real-time data processing terminal 3 transmits an imaging process completion notification to the communication unit 58 of the near-terminal data processing device 5 (step S104). This processing completion notification includes the captured image data encoded in step S103. Thereby, the communication unit 58 of the near-terminal data processing device 5 receives the processing completion notification.

  The transmission / reception 51 of the near-terminal data processing device 5 transmits an imaging process completion notification to the distributed data processing terminal 6 (step S105). This processing completion notification includes the captured image data received in step S104. Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the processing completion notification.

  Next, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits an imaging process completion notification to the centralized data processing server 7 (step S106). This processing completion notification includes the captured image data received in step S105. Thereby, the transmission / reception unit 71 of the centralized data processing server 7 receives the processing completion notification.

  The object detection unit 76 of the centralized data processing server 7 performs an object (face) detection process on the captured image data (step S107). When an object (face) is detected in this process, the blurring processing unit 77 of the centralized data processing server 7 performs a blurring process on the object area rectangular image that is a partial image of the detected object (face). (Step S108). Finally, the storage / reading unit 79 of the centralized data processing server 7 stores the captured image data after the blurring process in the storage unit 7000 (step S109). Note that, as shown in FIG. 26, the captured image data after the blurring process is not subjected to the blurring process in the blurred area part (object area rectangular image part) and the captured image other than this part. An area part (chest part and background part) is included.

[C] Example of Face Blur Processing Pattern 2 FIG. 25 is a sequence showing the processing of the object blur service when the processing setting table (FIG. 12C) is used for setting the real time data processing terminal 3.

  First, captured image data is generated by being captured by the imaging unit 40, and the connection unit 38 of the real-time data processing terminal 3 acquires captured image data from the imaging unit 40 (step S121). In the case of the object blurring service, an interval imaging function or the like is used.

  Next, the object detection unit 35 of the real-time data processing terminal 3 performs object (here, face) detection processing on the captured image (step 122). When an object (face) is detected in this process, the blurring processing unit 47 of the real-time data processing terminal 3 performs a blurring process on the object region rectangular image that is a partial image of the detected object (face). This is performed (step S123).

  Next, the image processing unit 34 of the real-time data processing terminal 3 performs projection method conversion on the captured image data after the blurring process (step S124). In this case, the image processing unit 34 performs conversion from the equidistant projection obtained by the imaging unit 40 to the above-mentioned equirectangular projection described above. Note that, as shown in FIG. 26, the captured image data after the blurring process is not subjected to the blurring process in the blurred area part (object area rectangular image part) and the captured image other than this part. An area part (chest part and background part) is included.

  The image processing unit 34 encodes (encodes) the partial image data (data to be verified) of the object (face) (step S125).

  The communication unit 48 of the real-time data processing terminal 3 transmits an imaging process completion notification to the communication unit 58 of the near-terminal data processing device 5 (step S126). This processing completion notification includes the captured image data after the blurring process encoded in step S125. Thereby, the communication unit 58 of the near-terminal data processing device 5 receives the processing completion notification.

  The transmission / reception 51 of the near-terminal data processing device 5 transmits an imaging process completion notification to the distributed data processing terminal 6 (step S127). This processing completion notification includes the captured image data after the blurring process received in step S126. Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the processing completion notification.

  Next, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits an imaging process completion notification to the centralized data processing server 7 (step S128). This processing completion notification includes the captured image data after the blurring process received in step S127. Thereby, the transmission / reception unit 71 of the centralized data processing server 7 receives the processing completion notification. Finally, the storage / reading unit 79 of the centralized data processing server 7 stores the image data after the blurring process in the storage unit 7000 (step S109).

  As described above, when the performance of the centralized data processing server 7 is relatively high, the centralized data processing server 7 executes the object blurring process (see FIG. 24). On the other hand, when the performance of the central data processing server 7 is relatively low, the real-time data processing terminal 3 executes the object blurring process (see FIG. 25).

  Further, since the central data processing server 7 includes the processing information management DB 7001 corresponding to the service, there is a merit that the manager who manages the central data processing server 7 can easily manage and operate the processing information management DB 7001. is there.

[Main effects of the embodiment]
As described above, according to the present embodiment, the purpose of post-processing (face recognition, face blurring, etc.) performed using captured image data obtained by imaging with a fisheye camera or the like, or post-processing is executed. Even if there is a difference in performance of the server or the like, since the processing sharing can be changed between the image acquisition terminal 2 side and the central data processing server 7 side, there is an effect that various needs of users can be met.

  For example, when the service is an object recognition service (see FIGS. 11D and 23), the real-time data processing terminal 3 of the image acquisition terminal 2 performs object detection (see S81), projection method conversion (see S82), and By performing up to resizing (see S83), the centralized data processing server 7 has only to perform the matching process (see S89). Thereby, even if the performance of the centralized data processing server 7 is relatively low, the service providing server 8 can provide the object recognition service.

  When the service is the object blurring service (pattern 1) (see FIG. 11D and FIG. 24), the distributed data processing terminal 6 performs the process from the captured image to object detection (see S107) and blurring processing (see S108). By doing so, the image acquisition terminal 2 only needs to transmit the captured image data (see S105). Thereby, even if the performance of the image acquisition terminal 2 is relatively low, the service providing server 8 can provide the object blurring service.

Further, when the service is an object blurring service (pattern 2) (see FIG. 11D and FIG. 25),
The real-time data processing terminal 3 on the image acquisition terminal 2 side performs from the captured image to the object detection (see S124) and the blurring process (see S125), so that the distributed data processing terminal 6 performs the blurring process. Data storage (see S131) is sufficient. Thereby, even if the performance of the centralized data processing server 7 is relatively low, the object providing service can be provided by the service providing server 8.

  In addition, the real-time data processing terminal 3 of the image acquisition terminal 2 converts the projection method of the captured image data (S82, S102, S126), thereby correcting the distortion when captured with the fisheye lens and reducing the uncomfortable feeling during browsing. can do.

Second Embodiment Subsequently, a second embodiment will be described with reference to FIGS. This embodiment shows another processing example of steps S61 to S69 in the first embodiment. Since other processes, hardware configuration, and functional configuration are the same as those in the first embodiment, description thereof is omitted.

  In the first embodiment, the processing information management DB 7001 is stored in the storage unit 7000 of the centralized data processing server 7. However, in the second embodiment, the processing information management DB 7001 is stored in the storage unit 7000 of the centralized data processing server 7. Not remembered. Instead, a processing information management DB composed of the table shown in FIG. 27 is stored in the storage unit 5000 of the near-terminal data processing device 5.

(Processing information table)
Each table shown in FIG. 27 is stored in the storage unit 5000 of the near-terminal data processing device 5 and is a table for managing the processing contents performed by the centralized data processing server 7. FIGS. 27A and 27B are tables specialized for executing the object blurring service. With these tables, it is possible to set whether or not to cause the central data processing server to execute the object blurring process (ON / OFF) and whether to save the captured image data after the blurring process (ON / OFF).

  When the processing capability of the real time data processing terminal 3 is relatively high, the table of FIG. 27A is managed in the storage unit 5000 in advance. On the other hand, when the processing capability of the real-time data processing terminal 3 is relatively low, the table in FIG. 27C is managed in the storage unit 5000 in advance.

(Service startup processing)
First, in the distributed data processing terminal 6, the receiving unit 62 receives a service start request from the user (step S161). In this case, the GUI (Graphical User Interface) of the distributed data processing terminal 6 is used.

  Next, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits a request for processing information to the near-terminal data processing device 5 (step S162). In this case, a service name for specifying the service to be started is transmitted. Thereby, the transmission / reception unit 51 of the near-terminal data processing device 5 receives the request for the processing information.

  Next, the storage / reading unit 59 of the near-terminal data processing device 5 reads the corresponding processing information by searching the storage unit 5000 using the service name received in step S162 as a search key (step S163).

  The transmitter / receiver 51 of the near-terminal data processing device 5 transmits processing information notification to the distributed data processing terminal 6 (step S164). This processing information notification includes the processing information (information in the table of FIG. 27) read in step S163. Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the notification of the processing information.

  Next, the transmission / reception unit 61 of the distributed data processing terminal 6 transmits a service preparation request to the centralized data processing server 7 (step S165). In this case, the service name for specifying the service to be started and the processing information received in step S164 are transmitted. Thereby, the transmission / reception unit 71 of the centralized data processing server 7 receives the service preparation request.

  Next, the storage / reading unit 79 of the centralized data processing server 7 performs various settings according to the processing information (step S166). Specifically, the setting is valid (ON) or invalid (OFF) of the object blurring process, and valid (ON) or invalid (OFF) of the image storage.

  Next, the transmission / reception unit 71 of the centralized data processing server 7 transmits a service preparation completion notification to the distributed data processing terminal 6 (step S167). Thereby, the transmission / reception unit 61 of the distributed data processing terminal 6 receives the service preparation completion notification. Since the subsequent processing is the same as the processing after step S69 in FIG. 22, the description thereof is omitted.

  Further, the setting of the processing executed by the real-time data processing terminal 3 or the central data processing server 7 during the service execution and the processing are the same as the case where the central data processing server 7 includes the processing information table. Hereinafter, a simple description will be given.

  That is, when the processing information (table information) shown in FIG. 27A is set in step S166, the processing shown in FIG. 24 is executed. On the other hand, when the processing information (table information) shown in FIG. 27B is set in step S166, the processing shown in FIG. 25 is executed.

  In the example of this embodiment, a case has been described in which the near-terminal data processing device 5 includes a processing information table corresponding to a service. For the administrator who manages the system, the merit in management and operation is less than that in the centralized data processing server. However, the load on the centralized data processing server is reduced, and the desired processing content can be flexibly changed according to the processing capability of the real-time data processing terminal 3 by being managed by the individual near-terminal data processing devices 5. Will be able to.

Supplementary Note In the above embodiment, the projection method conversion is performed by the real-time data processing terminal 3 of the image acquisition terminal 2, but may be performed by the centralized data processing server 7 side.

  In each of the above embodiments, the captured image data has been described. However, the present invention is not limited to this, and may be generated image data created by a user without being captured. In this case, the captured image data and the created image data are examples of “image data”. Also. The image acquisition terminal 2 is an example of a communication terminal, and may be created by the own terminal without acquiring image data from the outside. Furthermore, the communication terminal may acquire sound data by collecting sound, may acquire temperature data by a temperature sensor, or may acquire humidity data by a humidity sensor.

  Moreover, each component, such as each CPU 301, 501, 701, etc. may be single or plural. Further, there may be a plurality of image acquisition terminals 2, distributed data processing terminals 6, and centralized data processing servers 7, respectively. Further, the distributed data processing terminal 6 may be a server.

Further, each function in the above-described embodiment can be realized by one or a plurality of processing circuits. Here, the `` processing circuit '' in the present embodiment is a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or an ASIC (designed to execute each function described above). Application Specific Integrated
Circuit, DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU, and devices such as conventional circuit modules.

2 Image acquisition terminal (an example of a communication terminal)
3 Real-time data processing terminal 5 Near-terminal data processing device 6 Distributed data processing terminal 7 Centralized data processing server 34 Image processing unit (an example of image processing means)
35 Object detection unit (an example of object detection means)
38 connection part (an example of an acquisition means)
47 Blur processor (an example of blur processor)
51 Transmission / Reception Unit 61 Transmission / Reception Unit 62 Reception Unit 67 Display Control Unit 69 Storage / Reading Unit 71 Transmission / Reception Unit 74 Feature Quantity Generation Unit 75 Collation Unit 76 Object Detection Unit (Example of Object Detection Unit)
77 Blur processor (an example of blur processor)
200 Intranet 600 Internet 3001 Image sensor information management DB (an example of image sensor information management means)
3002 Cycle value management DB
3003 Image acquisition program DB
3004 Composition processing program management DB
3005 Distortion correction program management DB
3006 Service program management DB (an example of program management means)
6000 storage unit 6001 collation data management DB
8000 storage unit 8001 session management DB
8002 Authentication server management DB
9000 storage unit 9001 authentication management DB

Special table 2006-519812 gazette

Claims (9)

A service providing server for providing a service via a communication network;
An image acquisition terminal for obtaining captured image data generated by predetermined image processing;
A centralized data processing server that performs predetermined image processing on the image data;
A distributed data processing terminal that mediates between the image acquisition terminal and the centralized data processing server,
The distributed data processing terminal is associated with a service provided from the service providing server,
The image processing content obtained as a response after requesting the image processing content for either the image acquisition terminal or the centralized data processing server to the other is transmitted to the one, and the image data received after the image processing of the image acquisition terminal is received. A communication system comprising transmission / reception means for causing the centralized data processing server to process image processing for the service by transmitting to the distributed data processing terminal.   The service is an object recognition service, and when the transmission / reception unit of the distributed data processing terminal requests the centralized data processing server for image processing content for the image acquisition terminal, the image processing content is at least object detection. 2. The communication system according to claim 1, wherein the central data processing server processes object recognition for the received image data, including projection method conversion and resizing.   The service is an object blurring service, and when the transmission / reception means of the distributed data processing terminal requests the centralized data processing server for image processing content for the image acquisition terminal, the image processing content is at least object detection. The communication system according to claim 1, wherein the centralized data processing server performs a storage process for the received image data.   The service is an object blurring service, and when the transmission / reception unit of the distributed data processing terminal requests the centralized data processing server for the image processing content for the image acquisition terminal, the image processing content is the received image 2. The communication system according to claim 1, wherein the central data processing server processes object detection, object blurring, and storage processing for data.   5. The communication system according to claim 3, wherein the image processing content for the image acquisition terminal further adds projection method conversion.   The service is an object blur service, and the image processing content when the transmission / reception unit of the distributed data processing terminal requests image processing content for the centralized data processing server to the image acquisition terminal includes storage processing, The communication system according to claim 1, wherein at least image detection and object blurring are performed on the image data received from the image acquisition terminal.   The service is an object blur service, and the image processing content when the transmission / reception means of the distributed data processing terminal requests the image acquisition terminal to perform image processing on the centralized data processing server is object blurring and storage processing The communication system according to claim 1, wherein at least image processing for object detection is performed on the image data received from the image acquisition terminal.   8. The communication system according to claim 6, wherein a projection system conversion process is further added to the image data received from the image acquisition terminal. A service providing server for providing a service via a communication network;
An image acquisition terminal for obtaining captured image data generated by predetermined image processing;
A centralized data processing server that performs predetermined image processing on the image data;
A communication method executed by a communication system having a distributed data processing terminal that mediates between the image acquisition terminal and the centralized data processing server,
The distributed data processing terminal is associated with a service provided from the service providing server,
The image processing content obtained as a response after requesting the image processing content for either the image acquisition terminal or the centralized data processing server to the other is transmitted to the one, and the image data received after the image processing of the image acquisition terminal is received. A communication method characterized by causing the centralized data processing server to process image processing for the service by transmitting to the distributed data processing terminal.