JP2018182569A - IMAGE PROCESSING APPARATUS, IMAGE PROCESSING SYSTEM, INFORMATION PROCESSING SYSTEM, AND IMAGE PROCESSING METHOD - Google Patents

Abstract

A technology capable of reducing power consumption of an image processing apparatus is provided. An image processing apparatus includes a first generation unit, a second generation unit, a first determination unit, an encoding unit, and a first transmission unit. The first generation unit generates hierarchized hierarchical data indicating a frame image. The second generation unit generates difference layered data indicating a difference between layered data for two frame images. A 1st determination part determines transmission object data from difference hierarchization data based on 1st data whose absolute value of the value is more than a threshold value or larger than the said threshold value among difference hierarchization data. The encoding unit compresses and encodes the transmission target data to generate encoded data. The first transmission unit transmits encoded data. [Selection] Figure 3

Description

  The present invention relates to image processing.

  Patent Documents 1 and 2 disclose techniques related to image processing.

JP, 2015-192321, A Unexamined-Japanese-Patent No. 2003-219386

  For the image processing apparatus, reduction of the power consumption is desired.

  Therefore, the present invention has been made in view of the above-described point, and an object thereof is to provide a technology capable of reducing the power consumption of an image processing apparatus.

  One aspect of the image processing apparatus includes: a first generation unit that generates hierarchically layered data indicating a frame image; and differential hierarchical data indicating differences between the hierarchical data of two frame images The data to be transmitted is determined from the differentially-layered data based on first generation data, of the differentially-layered data, and an absolute value of the value is greater than or equal to a threshold value. A first determination unit, an encoding unit that compression-codes the transmission target data to generate encoded data, and a first transmission unit that transmits the encoded data.

  Further, one aspect of the image processing apparatus is an image processing apparatus which is a second apparatus which communicates with the first apparatus which is the above-mentioned image processing apparatus, and receives the encoded data transmitted from the first apparatus And a processing unit that performs processing based on the encoded data.

  Further, one aspect of the image processing system includes the first device described above and the second device described above.

  Further, one aspect of the information processing system includes: the image processing system described above; and a third device that receives a result of processing in the processing unit from the second device of the image processing system.

  Further, one aspect of the image processing method is the image processing method in the image processing apparatus, which includes the steps of generating hierarchical data showing a frame image and the hierarchical data of the two frame images. From the step of generating difference hierarchical data indicating a difference, and based on data of the difference hierarchical data, the absolute value of the value is greater than or equal to a threshold value or larger than the threshold value. The method includes the steps of: determining transmission target data; compressing and encoding the transmission target data to generate encoded data; and transmitting the encoded data.

  It is possible to reduce the power consumption of the image processing apparatus.

It is a figure showing an example of the composition of an information processing system. It is a figure which shows an example of the application example of an information processing system. It is a figure which shows an example of a structure of an IoT terminal. It is a figure which shows an example of a structure of a gateway. It is a figure which shows an example of a structure of a hierarchy part. It is a figure which shows an example of a wavelet plane. It is a figure which shows an example of a wavelet plane. It is a figure which shows an example of a wavelet plane. It is a figure which shows an example of a frame image. It is a figure which shows an example of a wavelet plane. It is a figure which shows an example of a structure of a difference production | generation part. It is a figure which shows an example of a structure of a determination part. It is a figure which shows an example of a frame image. It is a figure which shows an example of a difference frame mask. It is a figure for demonstrating an example of the production | generation method of an integrated sub-band mask. It is a figure for demonstrating an example of the production | generation method of an integrated sub-band mask. It is a figure which shows an example of the correspondence of a quantization difference wavelet plane and a code block. It is a figure which shows an example of the correspondence of an integrated sub-band mask and a code block. It is a figure which shows an example of the correspondence of an integrated sub-band mask and a code block. It is a figure which shows an example of a structure of an encoding apparatus. It is a figure for demonstrating an example of operation | movement of a coefficient bit modeling part. It is a figure for demonstrating an example of operation | movement of a coefficient bit modeling part. It is a figure which shows an example of a structure of a decoding apparatus. It is a figure which shows an example of a structure of a data processing part. It is a figure which shows an example of a structure of the data generation part for recognition. It is a figure which shows an example of a structure of an image recognition part. It is a figure for demonstrating an example of operation | movement of an image recognition part. It is a figure which shows an example of a structure of a transcoder. It is a flow chart which shows an example of operation of a gateway. It is a flowchart which shows an example of an IoT terminal. It is a figure which shows an example of the integrated sub-band mask by which the labeling process was carried out. It is a figure which shows an example of the integrated sub-band mask by which the labeling process was carried out. It is a flow chart which shows an example of operation of a gateway. It is a flow chart which shows an example of operation of a gateway. It is a figure which shows an example of the correspondence of the code block between several sub bands. It is a flowchart which shows an example of operation | movement of an IoT terminal. It is a figure which shows an example of a structure of an IoT terminal. It is a figure which shows an example of a structure of a motion correction part. It is a flowchart which shows an example of operation | movement of a motion correction part. It is a figure for demonstrating a motion correction process. It is a figure which shows an example of a structure of a hierarchy part. It is a figure which shows an example of a structure of a motion correction part. It is a figure which shows an example of a structure of an IoT terminal. It is a figure which shows an example of a structure of a data processing part. It is a figure for demonstrating an example of operation | movement of an IoT terminal. It is a figure for demonstrating an example of operation | movement of an IoT terminal. It is a figure for demonstrating an example of operation | movement of an image processing system. It is a figure for demonstrating an example of operation | movement of an image processing system. It is a figure which shows an example of a structure of an encoding apparatus. It is a figure which shows an example of a structure of a decoding apparatus. It is a figure which shows an example of a structure of a layer division | segmentation process part. It is a figure which shows an example of the priority set to each sub-band. It is a figure which shows an example of a mode that the several bit which comprises a coefficient is bit-shifted. It is a figure which shows an example of a structure of a layer synthetic | combination process part.

<System outline>
FIG. 1 is a diagram showing an example of the configuration of the information processing system 1. As shown in FIG. 1, the information processing system 1 includes an image processing system 4 including image processing devices 2 and 3, and an information processing device 5. The image processing device 2 generates compression encoded data representing an image and transmits it to the image processing device 3. The image processing device 3 performs processing based on the encoded data from the image processing device 2, and transmits the result of the processing to the information processing device 5. The information processing device 5 stores information from the image processing device 3 and performs various processing using the stored information.

  Such an information processing system 1 can be used in various situations. FIG. 2 is a diagram showing an application example of the information processing system 1. In the example of FIG. 2, the information processing system 1 is used as an IoT (Internet of Things) system 1. The image processing device 2, the image processing device 3 and the information processing device 5 are respectively used as, for example, the IoT terminal 2, the gateway 3 and the cloud server 5. The gateway 3 is also called an edge gateway. Each of the IoT terminal 2, the gateway 3 and the cloud server 5 is, for example, a type of computer device.

  As shown in FIG. 2, the IoT system 1 includes an image processing system 4 having a plurality of IoT terminals 2 and a gateway 3, and a cloud server 5. The plurality of IoT terminals 2 are connected to the gateway 3 by a local network. The gateway 3 and the cloud server 5 are connected by the Internet. The IoT terminal 2 and the gateway 3 may be connected by wireless or may be connected by wire. When the IoT terminal 2 wirelessly communicates with the gateway 3, for example, ZigBee (registered trademark) is adopted as a communication standard between the IoT terminal 2 and the gateway 3. The communication standard between the IoT terminal 2 and the gateway 3 is not limited to this.

  The processing capacity of the IoT terminal 2 is, for example, lower than that of the gateway 3. Also, the data transmission rate between the IoT terminal 2 and the gateway 3 is lower than, for example, the data transmission rate between the gateway 3 and the cloud server 5. The data transmission rate between the IoT terminal 2 and the gateway 3 is, for example, several tenths of the data transmission rate between the gateway 3 and the cloud server 5.

  Each IoT terminal 2 has, for example, a camera capable of capturing a moving image. Each IoT terminal 2 can generate compression-coded encoded data indicating at least a part of a frame image of a moving image captured by a camera. In addition, each IoT terminal 2 can generate a difference image indicating the difference between two frame images of a moving image captured by a camera. Then, each IoT terminal 2 can generate compression-coded encoded data indicating at least a part of the generated difference image. Each IoT terminal 2 transmits the generated encoded data to the gateway 3. Hereinafter, the term “difference image” means a difference image indicating the difference between two frame images.

  The IoT terminal 2 is, for example, a mobile phone such as a smartphone, a wearable device such as a smart glass, a network camera, or a television phone. The plurality of IoT terminals 2 may be the same type of device or may be different types of devices.

  The gateway 3 performs, for example, an image recognition process on an image based on encoded data from the IoT terminal 2. Then, the gateway 3 transmits information indicating the result of the image recognition process to the cloud server 5 through the Internet. The cloud server 5 stores information from the gateway 3 and executes various processes based on the stored information. The gateway 3 may transmit streaming data indicating at least a part of the image obtained by the IoT terminal 2 to the cloud server 5. In this case, the cloud server 5 may display streaming data from the gateway 3. Alternatively, the cloud server 5 may transmit information indicating the result of the image recognition process or streaming data to another gateway 3 or the IoT terminal 2 through the Internet. In this case, another gateway 3 or the IoT terminal 2 may display the information or streaming data received from the cloud server 5.

  The IoT system 1 can be used, for example, in a smart home system. In this case, the plurality of IoT terminals 2 and the gateway 3 are provided in the house, and the cloud server 5 is provided at a place away from the house. Each IoT terminal 2 captures a picture of the inside of a house with a camera, and indicates coded data indicating at least a part of a frame image of a moving picture showing the state or coding indicating at least a part of a difference image for the moving picture Send data to the gateway 3 The gateway 3 performs an image recognition process on an image based on encoded data from the IoT terminal 2 to detect, for example, a person in a house. Then, the gateway 3 transmits the detection result to the cloud server 5. This makes it possible, for example, to watch and manage the children or the elderly in the house.

  The IoT terminal 2 can be used, for example, in a smart factory. In this case, the plurality of IoT terminals 2 and the gateway 3 are provided in the factory, and the cloud server 5 is provided at a location distant from the factory. Each IoT terminal 2 takes a picture of the situation in the factory, and encodes the coded data indicating at least a part of the frame image of the moving image showing the situation or the encoded data indicating at least a part of the difference image for the moving image Send to 3 The gateway 3 performs an image recognition process on the image based on the encoded data from the IoT terminal 2 to detect, for example, an empty space where a package can be placed. Then, the gateway 3 transmits the detection result to the cloud server 5. This makes it possible, for example, to manage the transport of luggage within the factory.

  The scene where the IoT system 1 is used is not limited to the above. The IoT terminal 2 may also receive image data from a separate camera. Further, the image handled by the IoT terminal 2 may be not only an image captured by a camera but also an animation image.

  Hereinafter, the information processing system 1 will be described in detail by taking the case where the information processing system 1 is the IoT system 1 as an example.

<Configuration of IoT terminal>
FIG. 3 is a diagram showing an example of the configuration of the IoT terminal 2. As shown in FIG. 3, the IoT terminal 2 includes a camera 20, an image memory 21, a layering unit 22, a difference generation unit 23, an encoding device 24, a communication unit 25, a determination unit 26, a coordinate memory 27, and the like. Circuit configuration. The IoT terminal 2 is, for example, a battery-driven terminal, and includes a battery that outputs power of the IoT terminal 2.

  The camera 20 captures a moving image and outputs image data 500 indicating a frame image of the moving image to be captured. The image data 500 is composed of a plurality of pixel values. Each time the camera 20 captures a frame image, the camera 20 sequentially outputs image data 500 indicating the captured frame image. In this example, the position of the camera 20 is fixed. Hereinafter, the image data 500 may be referred to as frame image data 500.

  The image memory 21 stores image data 500 output from the camera 20. It can also be said that the image memory 21 stores frame images taken by the camera 20.

  The hierarchization unit 22 hierarchizes the image data 500 in the image memory 21 and outputs hierarchized data 501 obtained thereby. It can be said that the hierarchical data 501 is data indicating a frame image. The layering unit 22 performs, for example, wavelet transform (more specifically, discrete wavelet transform) on the image data 500 to layer the image data 500. The layering unit 22 performs wavelet transformation on the image data 500 in the same manner as, for example, wavelet transformation in compression encoding adopted in JPEG (Joint Photographic Experts Group) 2000. At least a part of the layering unit 22 may be realized by a hardware circuit that does not require software to realize its function. In addition, at least a part of the layering unit 22 may be a functional block realized by the computer executing a program. Hereinafter, the wavelet may be denoted as WT.

  The difference generation unit 23 generates and outputs difference hierarchical data 502 indicating the difference between the hierarchical data 501 for two frame images. The difference generation unit 23 can store the hierarchical data 501 output from the hierarchical unit 22. When the hierarchical data 501 is output from the hierarchical unit 22, the difference generation unit 23 generates differential hierarchical data 502 indicating the difference between the hierarchical data 501 and the hierarchical data 501 already stored. . As a result, difference hierarchical data 502 indicating the difference between the hierarchical data 501 of two frame images captured at different timings in the camera 20 is generated. It can be said that difference layered data 502 which shows the difference of layered data 501 about two frame pictures is hierarchized layered data which shows a difference picture which shows a difference of the two frame pictures concerned. At least a part of the difference generation unit 23 may be realized by a hardware circuit which does not require software for realizing the function. In addition, at least a part of the difference generation unit 23 may be a functional block realized by the computer executing a program.

  Hereinafter, the hierarchical data 501 may be referred to as non-differential hierarchical data 501. Also, the non-difference hierarchical data 501 and the difference hierarchical data 502 are collectively referred to simply as hierarchical data without using a code.

  The determination unit 26 determines transmission target data 503 from the difference hierarchical data 502 based on a predetermined standard. In this example, the result of the process in the IoT terminal 2 and the instruction information 520 transmitted from the gateway 3 are used as the predetermined reference. As a result of processing in the IoT terminal 2, for example, integrated mask data generated by a mask generation unit described later is used.

  The determination unit 26 determines transmission target data 503 from the difference hierarchical data 502 based on the integrated mask data and the instruction information 520. When the determination unit 26 receives the new instruction information 520 from the gateway 3, the determination unit 26 determines the transmission target data 503 based on the new instruction information 520. The determination unit 26 inputs the transmission target data 503 determined from the difference hierarchical data 502 to the encoding device 24.

  The determination unit 26 can also determine the transmission target data 503 from the non-difference layered data 501. The determination unit 26 inputs the transmission target data 503 determined from the non-difference layered data 501 to the encoding device 24.

  Further, the determination unit 26 generates and outputs coordinate data 504 related to the determined transmission target data 503. The coordinate data 504 will be described in detail later. At least a part of the determination unit 26 may be realized by a hardware circuit which does not require software for realizing the function. In addition, at least a part of the determination unit 26 may be a functional block realized by the computer executing a program.

  The encoding device 24 compresses and encodes the input transmission target data 503 to generate encoded data 505. Then, the encoding device 24 generates and outputs a bit stream 506 including the generated encoded data 505. Since the transmission target data 503 determined by the determination unit 26 is compressed and encoded by the encoding device 24, it can be said that the determination unit 26 performs a process of determining data to be compressed and encoded. At least a part of the encoding device 24 may be implemented by a hardware circuit that does not require software to implement its function. In addition, at least a part of the encoding device 24 may be a functional block realized by a computer executing a program.

  Hereinafter, the encoded data 505 generated by compressing and encoding the transmission target data 503 determined from the difference layered data 502 may be referred to as encoded difference data 505. Also, encoded data 505 generated by compression encoding the transmission target data 503 determined from the non-differential layered data 501 may be referred to as encoded non-differential data 505.

  The coordinate memory 27 stores a coordinate table 27a in which coordinate data 504 is registered. The coordinate memory 27 registers the coordinate data 504 output from the determination unit 26 in the coordinate table 27 a.

  The communication unit 25 is a communication circuit that communicates with the gateway 3. The communication unit 25 performs wireless communication with the gateway 3 in accordance with ZigBee, for example. The communication method of the communication unit 25 is not limited to this. The communication unit 25 includes a receiving unit 25 b that receives a signal from the gateway 3 and a transmitting unit 25 a that transmits a signal to the gateway 3. The transmitting unit 25 a transmits the bit stream 506 generated by the encoding device 24 to the gateway 3. The transmitting unit 25 a also transmits the coordinate data 504 output from the determining unit 26 to the gateway 3. The receiving unit 25 b receives the instruction information 520 transmitted by the gateway 3, and outputs the received instruction information 520 to the determining unit 26.

  The IoT terminal 2 may not have the camera 20. In this case, image data (image data indicating a photographed image) output from a camera separate from the IoT terminal 2 may be input to the image memory 21, or image data indicating an animation image may be input. .

  The image memory 21 and the coordinate memory 27 may be separate memories independent of each other, or a part of the storage area of one memory is used as the image memory 21 and the other part of the storage area is a coordinate. The memory 27 may be used.

<Configuration of gateway>
FIG. 4 is a diagram showing an example of the configuration of the gateway 3. As shown in FIG. 4, the gateway 3 has a circuit configuration including a communication unit 30, a decoding device 31, a data processing unit 32, an image recognition unit 33, a transcoder 34, a communication unit 35, a coordinate memory 36, and the like. Prepare. The gateway 3 operates using, for example, a commercial power supply as a power supply. In the gateway 3, the decoding device 31, the data processing unit 32, the image recognition unit 33, and the transcoder 34 constitute a processing unit that performs predetermined processing based on the encoded data from the IoT terminal 2.

  The communication unit 30 is a communication circuit that communicates with the IoT terminal 2. The communication unit 30 includes a receiving unit 30 a that receives a signal from the IoT terminal 2 and a transmitting unit 30 b that transmits a signal to the IoT terminal 2. The receiving unit 30 a receives the bit stream 506 and the coordinate data 504 transmitted from the IoT terminal 2. The transmitting unit 30 b transmits, to the IoT terminal 2, instruction information 520 for instructing the IoT terminal 2 to transmit data to be transmitted by the IoT terminal 2. The instruction information 520 is generated by the data processing unit 32.

  The coordinate memory 36 stores a coordinate table 36 a in which coordinate data 504 from the IoT terminal 2 is registered. The coordinate memory 36 registers the coordinate data 504 received by the receiving unit 30 a in the coordinate table 36 a.

  The decoding device 31 extracts the coded data 505 from the bit stream 506 received by the receiving unit 30a. The decoding device 31 outputs the extracted encoded data 505 to the data processing unit 32. Also, the decoding device 31 decompresses and decodes the extracted encoded data 505 to generate decoded data 521. This decoded data 521 is data that has not been wavelet inverse transformed (more specifically, discrete wavelet inverse transformation). That is, when the decoded data 521 is obtained by expanding and decoding the coded difference data 505, at least a part of the difference layered data 502 (WT converted data) generated by the IoT terminal 2 Is restored. Further, when the decoded data 521 is obtained by expanding and decoding the encoded non-differential data 505, at least at least the non-differential layered data 501 (WT converted data) generated by the IoT terminal 2 It is a partial restoration. When the receiving unit 30 a receives a new bitstream 506, the decoding device 31 decompresses and decodes the encoded data 505 included in the new bitstream 506 to generate decoded data 521. At least a part of the decoding device 31 may be realized by a hardware circuit that does not require software to realize its function. In addition, at least a part of the decryption device 31 may be a functional block realized by the computer executing a program.

  Hereinafter, the decoded data 521 obtained by expanding and decoding the coded differential data 505 may be referred to as decoded differential data 521. Also, decoded data 521 obtained by expanding and decoding the encoded non-difference data 505 may be referred to as decoded non-difference data 521.

  The data processing unit 32 stores the decoded non-differential data 521 generated by the decoding device 31 in a first memory described later. Further, the data processing unit 32 restores the data included in the non-difference layered data 501 corresponding to the decoded difference data 521 from the decoded difference data 521 generated by the decoding device 31 to the first memory. Remember. The data processing unit 32 also stores the encoded data 505 from the decoding device 31 in a second memory described later.

  The image recognition unit 33 performs an image recognition process on the recognition target image indicated by the recognition data 522 generated by the data processing unit 32. The image recognition unit 33 detects an object to be detected such as a whole person or a human face from the recognition target image in the image recognition process. The object to be detected is not limited to this. The image recognition unit 33 generates recognition result information 523 that is used by the data processing unit 32 and indicates the result of the image recognition process. Further, the image recognition unit 33 outputs, to the communication unit 35, recognition result information 524 transmitted from the gateway 3 to the cloud server 5 and indicating the result of the image recognition processing.

  The data processing unit 32 generates recognition data 522 used by the image recognition unit 33 based on the data in the first memory. The data processing unit 32 generates mask data 525 used by the transcoder 34 based on the recognition result information 523 generated by the image recognition unit 33. The mask data 525 is mask data for specifying an ROI (region of interest) in an image based on data in the first memory. In other words, the mask data 525 is mask data for specifying an ROI in an image based on the encoded data 505 transmitted from the IoT terminal 2. The data processing unit 32 generates the instruction information 520 based on the recognition result information 523 and the like.

  The data processing unit 32 also generates and outputs transcoder decoded data 526 to be used by the transcoder 34 based on the data in the first memory. The data processing unit 32 also generates and outputs transcoder encoded data 527 used by the transcoder 34 based on the data in the second memory.

  The transcoder 34 identifies an ROI from the image indicated by the transcoder decoded data 526 based on the mask data 525 generated by the data processing unit 32, and encodes the compressed and encoded data indicating the identified ROI. Generate as data. The gateway 3 can generate coded data indicating ROIs of various shapes by changing the mask data 525. The transcoder 34 generates and outputs a bit stream 529 including the generated encoded data. In addition, the transcoder 34 generates and outputs a bit stream 529 including transcoder encoded data 527.

  The communication unit 35 is a communication circuit that communicates with the cloud server 5. The communication unit 35 transmits the bit stream 529 output from the transcoder 34 to the cloud server 5. The communication unit 35 also transmits the recognition result information 524 output from the image recognition unit 33 to the cloud server 5 as metadata. The gateway 3 may compress and encode the recognition result information 524 and transmit encoded data obtained thereby to the cloud server 5.

  The cloud server 5 decompresses and decodes the encoded data included in the bit stream 529 received from the gateway 3 and stores the decoded data (image data) obtained thereby. Also, the cloud server 5 stores metadata from the gateway 3. Then, the cloud server 5 performs image search or image analysis based on the stored decoded data and metadata.

  As described above, in the IoT system 1 according to this example, the IoT terminal 2 generates hierarchical data (non-differential hierarchical data 501 indicating a frame image and differential hierarchical data 502 indicating a differential image) indicating an image. From the generated hierarchical data, transmission target data 503 is determined based on a predetermined standard. Then, the IoT terminal 2 compresses and encodes the determined transmission target data 503, and transmits the encoded data 505 obtained thereby to the gateway 3. As a result, the power consumption of the IoT terminal 2 can be reduced compared to the case where all the hierarchical data representing the image are compression-coded and the encoded data obtained thereby is transmitted from the IoT terminal 2 it can. Further, even when the data transmission rate between the IoT terminal 2 and the gateway 3 is low, the transmission delay of the encoded data transmitted from the IoT terminal 2 can be reduced.

  In addition, the data amount of the difference hierarchical data 502 can be smaller than the data amount of the non-difference hierarchical data 501. Therefore, the IoT terminal 2 can reduce the amount of data to be transmitted by the IoT terminal 2 by determining the transmission target data 503 from the differential layered data 502. Therefore, the power consumption of the IoT terminal 2 can be further reduced. Moreover, the transmission delay of the coding data transmitted from the IoT terminal 2 can be further reduced.

<Detailed explanation of IoT terminal>
<Detailed Description of Layering Unit>
FIG. 5 is a diagram showing an example of the configuration of the layering unit 22 of the IoT terminal 2. As shown in FIG. 5, the layering unit 22 includes a DC level shift unit 221, a color space conversion unit 222, a tiling unit 223, a wavelet conversion unit 224 (WT conversion unit 224), and a quantization unit 225. .

  The DC level shift unit 221 converts the DC level of the frame image data 500 as needed. The color space conversion unit 222 converts the color space of the frame image data 500 after DC level conversion. For example, RGB components are converted to YCbCr components (composed of a luminance component Y and color difference components Cb and Cr). The tiling unit 223 divides the frame image data 500 after color space conversion into a plurality of rectangular area components called tiles. Then, the tiling unit 223 inputs the frame image data 500 to the wavelet transform unit 224 for each tile. The frame image data 500 need not necessarily be divided into tiles, and the frame image data 500 output from the color space converter 222 may be input to the wavelet converter 224 as it is.

  The wavelet transform unit 224 subjects the frame image data 500 processed by the tiling unit 223 to wavelet transform to generate hierarchical data 510. Then, the quantization unit 225 quantizes and outputs the hierarchical data 510. The hierarchization unit 22 outputs the quantized hierarchized data 510 as the hierarchized data 501.

Wavelet Transform Unit
The wavelet transform unit 224 performs integer or real discrete wavelet transform (DWT) on the input frame image data 500, and outputs a plurality of transform coefficients obtained as a result. The transform coefficients may be referred to as wavelet coefficients below. Also, data (a group of wavelet coefficients) generated by performing wavelet transform on the frame image data 500 may be referred to as first wavelet coefficient data. The wavelet transform unit 224 outputs the generated first wavelet coefficient data as hierarchical data 510 indicating a frame image.

  In wavelet transformation, two-dimensional image data is decomposed into high frequency components (in other words, high frequency components) and low frequency components (in other words, low frequency components). This frequency resolution is also called, for example, band division. In addition, each band component (that is, each of the low band component and the high band component) obtained by frequency decomposition is also called a sub-band. Here, in accordance with the basic scheme of JPEG 2000, an octave division scheme is adopted in which only the subbands divided to the low frequency side in both the vertical direction and the horizontal direction are recursively divided into bands. The number of times of recursive band division is called decomposition level. The decomposition level information is associated with the first wavelet coefficient data.

  The wavelet transform unit 224 decomposes the frame image data 500 to a predetermined decomposition level. In general, good coding efficiency is obtained when the decomposition level is about 3 to 5. The predetermined decomposition level in the wavelet transform unit 224 may be referred to as a maximum decomposition level. In this example, the maximum decomposition level is set to three.

  6 to 8 show Mallat-type wavelet planes 551 to 553 for wavelet transform in two dimensions. According to the example of FIGS. 6-8, the input image (two-dimensional image) is subjected to frequency decomposition in the vertical direction and the horizontal direction at the decomposition level 1 (see FIG. 6). Thereby, as shown in the wavelet plane 551 of FIG. 6, the signal is decomposed into four sub-bands HH1, HL1, LH1, and LL1. The subband LL1 obtained at decomposition level 1 is further decomposed into four subbands HH2, HL2, LH2, LL2 at decomposition level 2 (see wavelet plane 552 in FIG. 7). The subband LL2 obtained at decomposition level 2 is further decomposed into four subbands HH3, HL3, LH3 and LL3 at decomposition level 3 (see the wavelet plane 553 in FIG. 8).

  Regarding the notation relating to the two-dimensional wavelet transform, for example, HL1 is a sub-band composed of a high frequency component H in the horizontal direction and a low frequency component L in the vertical direction at the decomposition level 1. The notation is generalized as XYm (X and Y are either H or L, respectively, and m is an integer of 1 or more). That is, the sub-band consisting of the sub-band X in the horizontal direction and the sub-band Y in the vertical direction at the decomposition level m is denoted as XYm. Further, when the decomposition level is not specified, a sub-band consisting of the sub-band X in the horizontal direction and the sub-band Y in the vertical direction is denoted as XY.

  In the following, each of the subbands LL1, LL2,... May be abbreviated as LL subband. Also, the subband LL1 may be referred to as the LL1 subband. The same is true for the other subbands.

  Here, a wavelet plane (see FIGS. 6 to 8) is a data group in which calculation result data of wavelet transform are two-dimensionally arranged in correspondence with an array of pixels in an original image (image in a state where wavelet transform is not performed). It is. For example, in the region shown as sub-band LL1 in the wavelet plane, the operation result data obtained with a certain pixel in the original image as the target pixel is arranged corresponding to the position of the target pixel in the original image. It is done. The wavelet coefficient can be said to be a value corresponding to a pixel.

  The wavelet plane may also be called wavelet space or wavelet domain. Also, assuming the two-dimensionally arrayed coefficients as pixel values, the wavelet plane is called a wavelet image, the subbands are called subband images, the XYm subbands are called XYm subband images, and the XY subbands are XY subbands. Sometimes called an image.

  At decomposition level 1, subband LL1 corresponds to the essential information of the image. According to the subband LL1, it is possible to provide an image of 1⁄4 the size of the image before decomposition (in other words, an image with a reduction ratio of 1⁄2 to the image before decomposition). The subband HL1 corresponds to the information of the vertically extending edge, and the subband LH1 corresponds to the information of the horizontally extending edge. The sub-band HH corresponds to the information of the diagonally extending edge. These points are the same for other decomposition levels. For example, the subbands LL2, HL2, LH2, and HH2 at decomposition level 2 are in the same relationship as the subbands LL1, HL1, LH1, and HH1 when the subband LL1 before decomposition is regarded as the original image.

  In the following, the original image may be expressed as a wavelet image at decomposition level 0 or a wavelet plane at decomposition level 0 by making the original image in a state in which wavelet transformation is not performed correspond to decomposition level 0.

  FIG. 9 is a view showing an example of a frame image 5000 (original image). FIG. 10 is a view showing a wavelet plane 553 at decomposition level 3 obtained as a schematic image, which is obtained by wavelet transformation of the frame image data 500 showing the frame image 5000 shown in FIG. The frame image 5000 shown in FIG. 9 includes an image 5001 in which a moving person is photographed and an image 5002 in which a tree is photographed. As shown in FIG. 10, each sub-band on the wavelet plane includes a portion 5531 indicating an image 5001 and a portion 5532 indicating an image 5002.

  Each subband on the wavelet plane 553 can be said to be data indicating a frame image 5000. The resolution of the frame image 5000 represented by the subbands LL3, HL3, LH3, and HH3 with three subband divisions is higher than the resolution of the frame image 5000 represented by the subbands HL2, LH2, and HH2 with two subband divisions. It can be said that it is low. Further, the resolution of the frame image 5000 indicated by the subbands HL2, LH2, and HH2 with two subband divisions is higher than the resolution of the frame image 5000 indicated by the subbands HL1, LH1, and HH1 with one subband division. It can be said that it is low. Then, it can be said that the resolution of the frame image 5000 indicated by the subbands HL1, LH1, and HH1 with one subband division number is lower than the resolution of the frame image 5000 not subjected to the subband division.

  As described above, it can be said that the first wavelet coefficient data (hierarchized data 510) is composed of multiple layer data indicating frame images having different resolutions.

  In the Mallat type, as described above, the LL subband is decomposed recursively the same number of times in each of the horizontal direction and the vertical direction. Also, as described later, in the Mallat type, sub-bands are synthesized in the reverse procedure of decomposition. However, it is not necessary to decompose and combine the L component and the H component in the horizontal direction and the vertical direction in the same number of times. That is, a wavelet transform of a type different from the Mallat type may be used. Also, the sizes on the wavelet plane of the sub-bands having the same number of decompositions may not be the same.

  Also, in the original image and wavelet image, the upper left end is taken as the origin of the orthogonal coordinate system, the origin is treated as 0, the L component output of wavelet transform is treated as even, and the H component output is treated as odd. However, it is also possible to treat the L component output as an odd number and the H component output as an even number. The two orthogonal axes in this orthogonal coordinate system are set, for example, in the horizontal direction and the vertical direction of the wavelet plane. The wavelet plane (see FIGS. 6 to 8) is a conceptual plane in which even-numbered and odd-numbered outputs of the wavelet transform are rearranged for each subband.

  Further, the position of each coefficient of the wavelet plane (wavelet image) may be represented by coordinates (x, y) in the orthogonal coordinate system. The coordinates (x, y) of the coefficient located at the origin of the orthogonal coordinate system are (0, 0).

<Quantizer>
The quantization unit 225 performs scalar quantization on the first wavelet coefficient data output from the wavelet transform unit 224 based on the quantization step size, whereby first quantized wavelet coefficient data (first quantum And generate WT coefficients data). The quantization unit 225 outputs the generated first quantized wavelet coefficient data as hierarchical data 501. The first quantized wavelet coefficient data is quantized hierarchical data 510. The quantization step size is set, for example, according to the target image quality. By changing the quantization step size, the resolution of the image represented by the first quantized wavelet coefficient data can be adjusted. By quantizing the layered data 510, the amount of data of the high band sub-bands LH, HL, HH particularly decreases.

  The first quantized wavelet coefficient data has the same data structure as the first wavelet coefficient data. Thus, according to the first quantized wavelet coefficient data, a wavelet plane is provided as well as the first wavelet coefficient data. The wavelet plane provided by the first quantized wavelet coefficient data may be referred to as a quantized wavelet plane.

<Detailed Description of Difference Generation Unit>
FIG. 11 is a diagram showing an example of the configuration of the difference generation unit 23. As shown in FIG. As shown in FIG. 11, the difference generation unit 23 includes a generation unit 230 and a frame buffer 231. The generation unit 230 stores the hierarchical data 501 in the frame buffer 231. The generation unit 230 generates hierarchical data 501 indicating the frame image to be processed at present by the hierarchical unit 22, and stores the hierarchical data 501 and the hierarchy generated in the past, which are stored in the frame buffer 231. Based on the conversion data 501, the difference hierarchical data 502 is generated.

  Hereinafter, the current frame image to be processed may be referred to as a target frame image or a current frame image. Also, hierarchical data 501 indicating a target frame image may be referred to as current hierarchical data 501. Also, the quantized wavelet coefficient (quantized WT coefficient) of the current hierarchical data 501 may be referred to as a currently quantized wavelet coefficient (currently quantized WT coefficient). Also, the image data 500 of the target frame image may be referred to as target frame image data 500 or current frame image data 500. Also, among the two hierarchical data 501 used by the generation unit 230 in the generation of the difference hierarchical data 502, the previous hierarchical data 501 read from the frame buffer 231 may be referred to as the past hierarchical data 501. Also, a frame image indicated by the past hierarchical data 501 may be referred to as a past frame image. In addition, the quantized wavelet coefficients of the past hierarchical data 501 may be referred to as past quantized wavelet coefficients (past quantized WT coefficients).

  The generation unit 230 is a difference hierarchy indicating a difference image indicating a difference between the target frame image and the past frame image based on the current hierarchy data 501 indicating the target frame image and the past hierarchy data 501 indicating the past frame image. To generate the conversion data 502. The past frame image is a frame image P frames before the target frame image. P is an integer of 1 or more, and is set to 1, for example. P may be 2 or more.

  For each current quantized WT coefficient of the current hierarchical data 501, the generation unit 230 generates, from the current quantized WT coefficient, a past quantized WT coefficient corresponding to the current quantized WT coefficient in the past hierarchical data 501. The subtracted value is obtained as a quantization difference WT coefficient. Here, the past quantized WT coefficient corresponding to the current quantized WT coefficient is a past quantized WT coefficient existing at the same position as the current quantized WT coefficient in the quantized wavelet plane. The quantized differential WT coefficients may be simply referred to as differential WT coefficients.

  Hereinafter, data composed of quantized difference WT coefficients obtained for each current quantized WT coefficient of the current hierarchical data 501 may be referred to as “first quantized difference WT coefficient data”. The first quantized differential WT coefficient data has the same data structure as the first wavelet coefficient data. For this reason, according to the first quantization difference WT coefficient data, the wavelet plane is provided similarly to the first wavelet coefficient data. The wavelet plane provided by the first quantized difference WT coefficient data may be referred to as a quantized difference wavelet plane (quantized difference WT plane).

  Here, it is assumed that the current quantized WT coefficient and the past quantized WT coefficient at the position of the coordinates (x, y) are C1 (x, y) and C2 (x, y), respectively. In addition, the differential WT coefficient at the position of the coordinates (x, y) obtained from the current quantized WT coefficient C 1 (x, y) and the past quantized WT coefficient C 2 (x, y) is DC (x, y) Do. In this case, the difference WT coefficient DC (x, y) = C1 (x, y) -C2 (x, y).

  When the generation unit 230 generates quantized difference WT coefficient data based on the current hierarchical data 501 and the past hierarchical data 501, the generation unit 230 outputs the generated quantized difference WT coefficient data as the difference hierarchical data 502. The generation unit 230 generates the difference hierarchical data 502 by using the hierarchical data 501 as the current hierarchical data 501 each time the hierarchical data 501 is generated by the hierarchical unit 22.

<Detailed Description of Determination Unit>
FIG. 12 is a diagram showing an example of the configuration of the determination unit 26. As shown in FIG. As shown in FIG. 12, the determination unit 26 includes a mask generation unit 260 and a transmission target determination unit 263.

<Mask generation unit>
The mask generation unit 260 includes a generation unit 261 and an integration unit 262. The generation unit 261 generates difference frame mask data 512 (hereinafter, may be referred to as difference frame mask data 512) for specifying a difference WT coefficient whose absolute value is larger than the threshold value in the difference hierarchical data 502. Do. The threshold is set to, for example, zero. The threshold may be greater than zero.

  Hereinafter, in the difference hierarchical data 502, the difference WT coefficient whose absolute value is larger than the threshold is referred to as a large difference WT coefficient, and the difference WT coefficient whose absolute value is equal to or less than the threshold is the small difference WT Sometimes called a coefficient. The difference layered data 502 can be divided into large difference data consisting of large difference WT coefficients of large difference and small difference data consisting of small differential WT coefficients of small difference.

  The difference frame mask data 512 is composed of a plurality of coefficients respectively corresponding to a plurality of difference WT coefficients constituting the difference layered data 502. The plurality of coefficients are two-dimensionally arranged in correspondence with the arrangement of pixels in the frame image (in the difference image) to form a difference frame mask.

  The plurality of coefficients constituting the difference frame mask data 512 include a first ROI mask coefficient corresponding to a large difference WT coefficient and a first non-ROI mask coefficient corresponding to a small difference WT coefficient. The first ROI mask factor is, for example, one, and the first non-ROI mask factor is, for example, zero. The generation unit 261 obtains the absolute value of each difference WT coefficient of the difference hierarchical data 502. Then, if the absolute value of the difference WT coefficient located at the coordinates (x, y) in the quantization difference WT plane is larger than the threshold (in the present example, it is larger than zero), the generation unit 261 A coefficient located at the coordinates (x, y) is set as a first ROI mask coefficient. On the other hand, if the absolute value of the difference WT coefficient located at the coordinates (x, y) in the quantized difference WT plane is equal to or less than the threshold (if it is zero in this example), the generation unit 261 A coefficient located at the coordinates (x, y) is set as a first non-ROI mask coefficient. The difference frame mask can be divided into a first ROI mask portion consisting of first ROI mask coefficients and a first non-ROI mask portion consisting of first non-ROI mask coefficients.

  FIG. 13 is a view schematically showing an example of the frame image 5010. As shown in FIG. The frame image 5010 includes an image 5011 in which a person is photographed. FIG. 14 shows an example of the difference frame mask 5120 for specifying the difference WT coefficient of the difference large in the difference hierarchical data 502 generated based on the current hierarchical data 501 indicating the frame image 5010 shown in FIG. FIG. As shown in FIG. 14, the differential frame mask 5120 is configured of a first ROI mask portion 5121 shown in white and a first non-ROI mask portion 5122 shown in black. In the difference frame mask 5120, a portion corresponding to the image 5011 in which a person is photographed in the frame image 5010 is a first ROI mask portion 5121.

  As can be understood from FIG. 14, the differential frame mask can be understood as a collection of masks for each subband included in the quantized differential wavelet plane. That is, when the mask for each subband is called a subband mask, the differential frame mask can be understood as a collection of subband masks. For example, the portion for the LL sub-band in the differential frame mask will be referred to as the LL sub-band mask. The same applies to portions corresponding to other subbands. Also, there is data of the subband mask included in the difference frame mask data 512 as subband mask data. Also, data of the LL subband mask may be referred to as LL subband mask data. The same applies to data of other subband masks. The subband mask data can be said to be data for specifying a large difference WT coefficient in the subband corresponding to the subband mask data in the quantization differential wavelet plane.

  The integration unit 262 integrates a plurality of subband masks respectively corresponding to a plurality of subbands of the decomposition level for each decomposition level of the subbands of the quantization difference wavelet plane, and a sub-common to the plurality of subbands. Generate a band mask. Hereinafter, the common subband mask may be referred to as an integrated subband mask.

  15 and 16 are diagrams for explaining an example of a method of generating an integrated subband mask. FIG. 15 is a diagram for describing a method of generating an integrated subband mask for decomposition levels in which four subbands are present. In other words, FIG. 15 is a diagram for describing a method of generating an integrated subband mask for decomposition levels in which LL subbands are present. FIG. 16 is a diagram for describing a method of generating an integrated subband mask for decomposition levels in which three subbands are present. In other words, FIG. 16 is a diagram for describing a method of generating an integrated subband mask for decomposition levels in which LL subbands do not exist.

  Referring to FIG. 15, with respect to the decomposition level in which LL subbands, LH subbands, HL subbands and HH subbands are present as in decomposition level 3, combining section 262 combines LL subbands, LH subbands, LL subband mask 5125LL, LH subband mask 5125LH, HL subband mask 5125HL, and HH subband mask 5125HH corresponding to the HL subband and the HH subband, respectively, are integrated to generate an integrated subband mask 5126. Specifically, the integration unit 262 obtains the logical sum of the coefficients at the same position in the LL subband mask 5125LL, the LH subband mask 5125LH, the HL subband mask 5125HL, and the HH subband mask 5125HH, This value is taken as the value of the coefficient at the same position as that of integrated subband mask 5126. Therefore, if at least one of the coefficients at the same position in LL subband mask 5125LL, LH subband mask 5125LH, HL subband mask 5125HL and HH subband mask 5125HH is 1, it is at the same position as in integrated subband mask 5126 The factor is one. On the other hand, if all the coefficients at the same position in LL subband mask 5125LL, LH subband mask 5125LH, HL subband mask 5125HL and HH subband mask 5125HH are zero, the coefficients at the same position as in integrated subband mask 5126 Is zero. In this manner, the integration unit 262 obtains the coefficient of each position in the integrated subband mask 5126.

  As shown in FIG. 16, with respect to the decomposition level in which the LL sub-band is not present, as in the decomposition levels 1 and 2, the merging unit 262 selects the LH sub-band corresponding to the LH sub-band, the HL sub-band and the HH sub-band respectively. Band mask 5125 LH, HL sub-band mask 5 125 HL and HH sub-band mask 5 125 HH are integrated to generate integrated sub-band mask 5126. Specifically, integration section 262 calculates the logical sum of the coefficients at the same position in LH subband mask 5125 LH, HL subband mask 5125 HL and HH subband mask 5125 HH, and integrates the values obtained thereby. The value of the coefficient at the same position as that of the band mask 5126 is used. In this manner, the integration unit 262 obtains the coefficient of each position in the integrated subband mask 5126.

  As described above, the integration unit 262 generates a sub-band mask (unification sub-band mask) common to a plurality of sub-bands of the decomposition level for each decomposition bell of the sub-bands. By the sub-band mask common to a plurality of sub-bands of the same decomposition level, it is possible to substantially identify difference large differential WT coefficients in each of the plurality of sub-bands. The determination unit 26 determines the transmission target data 503 using the integrated subband mask.

  Hereinafter, the integrated subband mask common to the four subbands at decomposition level 3 may be referred to as an integrated subband mask corresponding to decomposition level 3. Similarly, an integrated subband mask common to the three subband masks of decomposition level 2 may be referred to as an integrated subband mask corresponding to decomposition level 2. Similarly, the integrated subband mask common to the three subband masks at decomposition level 1 may be referred to as an integrated subband mask corresponding to decomposition level 1. In addition, integrated sub-band masks corresponding to decomposition levels 1 to 3 may be collectively referred to as an integrated mask. Also, data of the integrated subband mask may be referred to as integrated subband mask data, and data of the integrated mask may be referred to as integrated mask data. Also, in the integrated subband mask, a portion where the coefficient is 1 may be referred to as a second ROI mask portion, and a portion where the coefficient is 0 may be referred to as a second non-ROI mask portion. In the integrated sub-band mask 5126 shown in FIGS. 15 and 16, the white part is the second ROI mask part and the black part is the second non-ROI mask part.

  The integration unit 262 generates and outputs integrated mask data 513 including a plurality of integrated subband mask data respectively corresponding to a plurality of decomposition levels of subbands based on the difference frame mask data 512.

<Transmission Target Determination Unit>
The transmission target determination unit 263 determines the transmission target data 503 from the difference hierarchical data 502 based on the integrated mask data 513 generated by the mask generation unit 260 and the instruction information 520 from the gateway 3. Further, the transmission target determination unit 263 determines the transmission target data 503 from the non-difference layered data 501 based on the instruction information 520.

  For example, the transmission target determination unit 263 divides each sub-band of the difference layered data 502 into an area called a “code block” of about 32 × 32 or 64 × 64 as in JPEG2000. FIG. 17 is a diagram showing an example of how each subband of the quantization difference wavelet plane 5020 is divided into a plurality of code blocks 5021. As shown in FIG. 17, the sub-band is divided into a plurality of code blocks 5021 based on the upper left end thereof. Then, the transmission target determination unit 263 identifies a differential code block corresponding to the instruction information 520 in the differential layered data 502 based on the integrated mask data 513, and sets the identified differential code block as transmission target data 503.

  Here, the differential code block includes at least a part of the second ROI mask portion of the integrated subband mask in the subband when the integrated subband mask is superimposed on the subband of the corresponding decomposition level. Stands for code block.

  FIG. 18 shows an example of how integrated sub-band mask 5126 including second ROI mask portion 5126a and second non-ROI mask portion 5126b corresponding to decomposition level 2 is superimposed on the subbands of decomposition level 2. It is. In the example of FIG. 18, among the 30 code blocks 5021 constituting the subband of decomposition level 2, each of 9 code blocks 5021 including at least a part of the second ROI mask portion 5126 a of the integrated subband mask 5126 is The difference code block 5021 is obtained. The diagonal lines in the differential code block 5021 are as shown in FIG.

  For example, when the subband LL3 at decomposition level 3 is designated as data to be transmitted by the IoT terminal 2 by the instruction information 520, the transmission target determining unit 263 performs subband processing on the integrated subband mask corresponding to the decomposition level 3 When overlapping with LL3, in the subband LL3, a differential code block including at least a part of the second ROI mask portion of the integrated subband mask is set as transmission target data 503.

  Further, the transmission target determination unit 263 divides each sub-band of the non-difference layered data 501 into code blocks of approximately 32 × 32 or 64 × 64, as with the difference layered data 502. Then, the transmission target determination unit 263 identifies a code block corresponding to the instruction information 520 in the plurality of code blocks constituting the non-difference layered data 501, and sets the identified code block as transmission target data 503.

  Hereinafter, the code block may be called CB. Also, the code block of the non-differential layered data 501 may be referred to as a non-differential code block (non-differential CB).

  Difference CB data composed of a plurality of difference code blocks included in difference layered data 502 is large difference data (data composed of difference WT coefficients whose absolute value is larger than a threshold value) included in difference layered data 502 It almost agrees. The difference CB data can be said to be data indicating a partial image indicating a moving object, which is included in the difference image.

  The gateway 3 can freely designate differential code blocks to be transmitted by the IoT terminal 2 using the instruction information 520. For example, the gateway 3 can specify a differential code block to be transmitted by the IoT terminal 2 in code block units. Also, the gateway 3 can specify the differential code block to be transmitted by the IoT terminal 2 in units of subbands. For example, the gateway 3 can specify the differential code block of the sub-band LL3 as the differential code block to be transmitted by the IoT terminal 2. Also, the gateway 3 can specify, for example, differential code blocks of the sub-bands HH3 and HL3 as differential code blocks to be transmitted by the IoT terminal 2. The gateway 3 can also specify data to be transmitted by the IoT terminal 2 in units of decomposition levels. For example, the gateway 3 can designate the decomposition level 2 difference code block as the difference code block to be transmitted by the IoT terminal 2. In this case, differential code blocks of the sub-bands HH2, LH2, and HL2 are designated. Also, the gateway 3 can specify, for example, differential code blocks at decomposition levels 1 and 2 as differential code blocks to be transmitted by the IoT terminal 2. In this case, differential code blocks of the sub-bands HH1, LH1, HL1, HH2, LH2, and HL2 are designated.

  Also, the gateway 3 can freely designate the non-differential code block to be transmitted by the IoT terminal 2 using the instruction information 520. The gateway 3 can specify non-differential code blocks to be transmitted by the IoT terminal 2 in units of code blocks, units of subbands, and units of decomposition level, for example.

  Hereinafter, the instruction information 520 in the case where the gateway 3 designates a differential code block as data to be transmitted by the IoT terminal 2 may be referred to as differential transmission instruction information 520. Further, the instruction information 520 when the gateway 3 designates a non-differential code block as data to be transmitted by the IoT terminal 2 may be referred to as non-differential transmission instruction information 520. Also, data designated by the gateway 3 in the instruction information 520 may be referred to as designated data. The specified data can be said to be data to be transmitted by the IoT terminal 2 which is instructed to the IoT terminal 2 by the instruction information 520.

  When the transmission target determination unit 263 receives the difference transmission instruction information 520 from the gateway 3, the transmission object determination unit 263 transmits the difference code block corresponding to the designated data specified by the difference transmission instruction information 520 in the difference hierarchical data 502. The target data 503 is assumed. On the other hand, when the transmission target determination unit 263 receives the non-differential transmission instruction information 520 from the gateway 3, the transmission target determination unit 263 corresponds to the designation data specified in the non-differential transmission instruction information 520 among the non-differential hierarchical data 501. A non-differential code block is set as transmission target data 503.

  When the transmission target determination unit 263 determines the transmission target data 503, the transmission target determination unit 263 generates and outputs coordinate data 504 including coordinates indicating the position on the wavelet plane for each code block included in the transmission target data 503. The coordinate data 504 is registered in the coordinate table 27 a of the coordinate memory 27. Thereby, the coordinates of the code block transmitted from the IoT terminal 2 are registered in the coordinate table 27a.

  In this example, for example, the coordinates of the code block are represented in an orthogonal coordinate system in which the upper left end of the wavelet plane is set as the origin and two axes orthogonal to each other are set in the horizontal direction and the vertical direction of the wavelet plane. Hereinafter, the coordinates of the code block on the wavelet plane may be represented by (i, j).

  When the transmission target data 503 includes the difference CB, the transmission target determination unit 263 includes coordinate data including coordinates (i, j) on the quantized difference wavelet plane for each difference CB included in the transmission target data 503. Generate 504. On the other hand, when the transmission target data 503 includes the non-difference CB, the transmission target determination unit 263 determines the coordinates (i, j) on the quantized wavelet plane for each non-difference CB included in the transmission target data 503. To generate coordinate data 504 including.

  As described above, the determination unit 26 determines, based on the difference large data whose absolute value of the value is larger than the threshold value among the difference hierarchical data 502 and the difference transmission instruction information 520 from the gateway 3. The transmission target data 503 can be determined from the difference hierarchical data 502. Further, the determination unit 26 can determine the transmission target data 503 from the non-differential layered data 501 based on the non-differential transmission instruction information 520 from the gateway 3.

  In the above example, in the differential layered data 502, the differential WT coefficient whose absolute value is larger than the threshold is used as the differential WT coefficient of the large difference, but the differential WT coefficient whose absolute value is equal to or larger than the threshold is used. A difference large difference WT coefficient may be used. In this case, in the difference hierarchical data 502, the difference WT coefficient whose absolute value is less than the threshold value becomes the difference WT coefficient whose difference is small.

<Detailed Description of Encoding Device>
FIG. 20 is a diagram showing an example of the configuration of the encoding device 24. As shown in FIG. As shown in FIG. 20, the encoding device 24 includes an encoding unit 240 and a bit stream generation unit 243. The encoding unit 240 compresses and encodes the transmission target data 503 to generate encoded data 505. The bit stream generation unit 243 generates a bit stream 506 including the encoded data 505 and transmits the bit stream 506 to the transmission unit 25 a. The transmitting unit 25 a transmits the bit stream 506 to the gateway 3.

<Encoding unit>
The encoding unit 240 performs entropy coding, for example, in accordance with Embedded Block Coding with Optimized Truncation (EBCOT) that performs bit-plane coding. In this example, the encoding unit 240 includes a coefficient bit modeling unit 241 and an entropy encoding unit 242.

  The coefficient bit modeling unit 241 performs bit modeling processing on the transmission target data 503. In the bit modeling process, first, the coefficient bit modeling unit 241 decomposes each code block included in the transmission target data 503 into a plurality of bit planes configured by a two-dimensional array of each bit. When the transmission target data 503 is configured by the difference CB, the coefficient bit modeling unit 241 decomposes each difference CB included in the transmission target data 503 into a plurality of bit planes. On the other hand, when the transmission target data 503 includes non-differential CBs, the coefficient bit modeling unit 241 decomposes each non-differential CB included in the transmission target data 503 into a plurality of bit planes. Hereinafter, the differential CB and the non-differential CB included in the transmission target data 503 may be collectively referred to as a transmission target code block.

FIG. 21 is a diagram illustrating an example of _n bit planes 571 _{0 to} 571 _n-1 (n is a natural number) that configure the transmission target code block 570. The coefficient bit modeling unit 241 assigns each bit constituting the binary value of each coefficient in the transmission target code block 570 to a separate bit plane. As shown in FIG. 21, when the binary value 572 of the coefficient of one point in the code block 570 is “011... 0”, the plurality of bits constituting the binary value 572 are each a bit plane _{571 n-1, 571 n-} 2, 571 n-3, ···, is decomposed to belong to 571 _0. Bit plane 571 _n-1 in the figure represents the most significant bit plane consisting only of the most significant bit (MSB) of the coefficient, and bit plane 571 ₁₀ represents the least significant bit plane consisting only of the least significant bit (LSB) It represents.

Furthermore, the coefficient bit modeling unit 241 performs context determination of each bit in each bit plane 571 _k (k = 0 to n−1), and as shown in FIG. According to the result, the bit plane 571 _k is decomposed into three types of coding passes: CL pass, CL pass, Magnification Refinement pass, and SIG pass (SIGnificance propagation pass). The algorithm of context determination for each coding pass is defined in the JPEG 2000 standard. According to it, "significant" means a state in which the coefficient of interest is known not to be zero in the coding process so far. Also, "not significant" means that the coefficient may be zero or may be zero.

  The coefficient bit modeling unit 241 sets the SIG pass (coding pass of non-significant coefficient with significant coefficient around), the MR pass (coding pass of significant coefficient), and the CL pass (remaining non-SIG pass and MR pass). Bit plane coding is performed in three types of coding passes of Bit plane coding is performed by scanning the bits of each bit plane in units of 4 bits from the most significant bit plane to the least significant bit plane, and determining whether a significant coefficient is present or not. The number of bit planes composed only of insignificant coefficients (0 bits) is included in the below-described packet header generated by the bit stream generation unit 243 as zero bit plane information. In bit plane coding, actual coding starts from the bit plane where the significant coefficient first appeared. The bit plane at the start of encoding is encoded only by the CL pass, and bit planes lower than the bit plane are encoded sequentially by the above three types of encoding passes.

  The entropy coding unit 242 performs entropy coding on the data generated by the coefficient bit modeling unit 241 to generate coded data 505. The encoded data 505 is composed of a compression-coded transmission target code block. For example, arithmetic coding is used as entropy coding.

  The encoding unit 240 may perform rate control on the encoded data 505 generated by the entropy encoding unit 242 to control the code amount. Hereinafter, the difference CB included in the coded data 505 may be referred to as a coded difference CB. The encoded data 505 configured by the encoded difference CB becomes the encoded difference data 505. Also, the non-differential CB included in the encoded data 505 may be referred to as an encoded non-differential CB. The encoded data 505 configured by the encoded non-differential CB becomes the encoded non-differential data 505. Also, the coding difference CB and the coding non-difference CB may be collectively referred to as coding CB.

<Bitstream Generator>
The bitstream generation unit 243 generates a bitstream 506 including the encoded data 505. Specifically, the bit stream generation unit 243 packetizes the coded data 505 and generates a bit stream 506 including packet data generated thereby and additional information. The additional information includes a packet header, layer configuration information, scalability information, and a quantization table. The packet header includes zero-length packet information, code block inclusion information, zero bit plane information, coding pass number information, and code amount information of the code block (compressed data length of the code block). The packet header is encoded and included in bitstream 506. The bit stream 506 generated by the bit stream generation unit 243 is transmitted together with coordinate data 504 indicating the coordinates of each coded CB included in the coded data 504 included in the bit stream 506, which is output from the determination unit 26. It is transmitted to the gateway 3 from the unit 25a.

<Detailed explanation of gateway>
<Detailed Description of Decryption Device>
FIG. 23 is a diagram showing an example of the configuration of the decryption device 31 of the gateway 3. As shown in FIG. 23, the decoding device 31 includes a bit stream analysis unit 310, a decoding unit 311, and an inverse quantization unit 314.

  The bit stream analysis unit 310 analyzes the bit stream 506 from the IoT terminal 2 and extracts the encoded data 505 and additional information from the bit stream 506. The bit stream analysis unit 310 outputs the extracted encoded data 505 to the decoding unit 311 and the data processing unit 32. Also, the bit stream analysis unit 310 decodes the encoded packet header included in the extracted additional information. The additional information is used by the decoding unit 311, the inverse quantization unit 314, and the like.

  The decoding unit 311 performs predetermined decompression decoding on the encoded data 505. The predetermined decompression decoding corresponds to processing reverse to compression encoding in the encoding unit 240 of FIG. 20 except code amount control. In this example, the decoding unit 311 includes an entropy decoding unit 312 and a coefficient bit modeling unit 313.

  The entropy decoding unit 312 performs entropy decoding on the coded data 505 to generate bit data. Entropy decoding corresponds to processing opposite to the entropy coding in the entropy coding unit 242 of FIG.

  The coefficient bit modeling unit 313 performs bit modeling processing on the bit data generated by the entropy decoding unit 312, and restores a plurality of coefficients constituting each transmission target code block included in the encoded data 505. The bit modeling process here is the reverse process to that in the coefficient bit modeling unit 241 of FIG. The coefficient bit modeling unit 313 inputs the restored coefficient to the inverse quantization unit 314.

  When the coded difference data 505 is extracted by the bit stream analysis unit 310, the coefficient bit modeling unit 313 determines quantized difference WT coefficients included in each coded difference CB included in the extracted coded difference data 505. Is restored. On the other hand, when the encoded non-differential data 505 is extracted by the bit stream analysis unit 310, the coefficient bit modeling unit 313 includes each of the encoded non-differential CBs included in the extracted encoded non-differential data 505. The quantized WT coefficients are recovered.

  Hereinafter, a group of quantized differential WT coefficients generated by the coefficient bit modeling unit 313 may be referred to as second quantized differential WT coefficient data. The second quantized differential WT coefficient data is composed of the differential CB. Also, a group of quantized WT coefficients generated by the coefficient bit modeling unit 313 may be referred to as second quantized wavelet coefficient data. The second quantized wavelet coefficient data is composed of non-differential CB. Then, the second quantization difference WT coefficient data and the second quantization wavelet coefficient data may be collectively referred to as quantization coefficient data.

  The inverse quantization unit 314 performs inverse quantization on the quantization coefficient data generated by the decoding unit 311. The inverse quantization here corresponds to the process opposite to the quantization in the quantization unit 225 of FIG. The second quantization differential WT coefficient data is converted into differential WT coefficient data by inverse quantization. In addition, the second quantization wavelet coefficient data is converted to second wavelet coefficient data by inverse quantization. The quantized differential WT coefficients included in the differential WT coefficient data may be simply referred to as differential WT coefficients.

  When the inverse quantization unit 314 generates differential WT coefficient data, the inverse quantization unit 314 outputs it as decoded data 521. Further, when the second quantization coefficient data is generated, the inverse quantization unit 314 outputs it as decoded data 521. Hereinafter, the difference WT coefficient data may be referred to as decoded difference data 521, and the second wavelet coefficient data may be referred to as decoded non-difference data 521. Also, a code block included in the decoded difference data 521 may be referred to as a decoded difference CB, and a code block included in the decoded non-difference data 521 may be referred to as a decoded non-difference CB.

<Detailed Description of Data Processing Unit>
FIG. 24 is a diagram showing an example of the configuration of the data processing unit 32. As shown in FIG. As shown in FIG. 24, the data processing unit 32 includes a recognition data generation unit 320, a first processing unit 321, a second processing unit 322, a first memory 323, a second memory 324, a selection unit 325, and a restoration unit 326. Equipped with In the first memory 323, reading and writing of data are performed by the first processing unit 321. The second memory 324 stores the encoded data 505 output from the decoding device 31. The data in the second memory 324 is read by the second processing unit 322.

<Selection section>
The selection unit 325 selects whether the decoded data 521 input to the data processing unit 32 is input to the first processing unit 321 or to the restoration unit 326. When the decoding difference data 521 is input to the data processing unit 32, the selection unit 325 inputs the decoded difference data 521 to the restoration unit 326. On the other hand, when the decoding non-difference data 521 is input to the data processing unit 32, the selection unit 325 inputs it to the first processing unit 321. The first processing unit 321 stores the input decoded non-difference data 521 in the first memory 323. Thus, the first memory 323 stores the non-differential CB included in the non-differential hierarchical data 501 restored by the gateway 3.

<Restoration unit>
The restoration unit 326 restores, for each decoded difference CB included in the decoded difference data 521, the non-difference CB included in the non-difference layered data 501 corresponding to the decoded difference CB.

  Here, as can be understood from the above description, the decoding difference CB of a certain coordinate subtracts the code block of the certain coordinate in the past hierarchical data 501 from the code block of the certain coordinate in the current hierarchical data 501. Data obtained by Assuming that the code block of the current hierarchical data 501 is a current non-differential CB and the code block of the past hierarchical data 501 is a past non-differential CB, the decoded differential CB of coordinates (i, j) is coordinates (i, j) Is a data obtained by subtracting the past non-difference CB of coordinates (i, j) from the current non-difference CB of. The restoration unit 326 adds the past non-difference CB of the coordinates (i, j) stored in the first memory 323 to the decoded difference CB of the coordinates (i, j) to obtain the coordinates (i, j). , J) are restored. The restoration unit 326 receives the past non-differential CB in the first memory 323 from the first processing unit 321. The restoration unit 326 restores the current non-difference CB from each decoding difference CB included in the decoding difference data 521. The restored current non-differential CB is stored in the first memory 323 by the first processing unit 321.

  When the restoration unit 326 restores the current non-differential CB of the coordinate (i, j) from the decoded difference CB of the coordinate (i, j), the coordinate (i, j) at the decoded differential CB The wavelet coefficients located at coordinates (x, y) in the past non-difference CB of coordinates (i, j) are added to the differential WT coefficients located at x, y). Then, the restoration unit 326 takes the value obtained thereby as the value of the coefficient located at the coordinates (x, y) at the current non-difference CB of the coordinates (i, j). The restoration unit 326 performs the same process on each difference WT coefficient included in the decoded difference CB at the coordinates (i, j). Thereby, each wavelet coefficient of the current non-difference CB of the coordinates (i, j) is restored. The wavelet coefficients of the current non-difference CB restored by the restoration unit 326 are input to the first processing unit 321. The first processing unit 321 stores the wavelet coefficients of the current non-difference CB in the first memory 323. The wavelet coefficients in the first memory 323 are used as wavelet coefficients of the past non-differential CB in a later frame.

<Recognition data generation unit>
FIG. 25 is a diagram showing an example of the configuration of the recognition data generation unit 320. As shown in FIG. As shown in FIG. 25, the recognition data generation unit 320 includes an inverse wavelet transform unit 3201, a color space conversion unit 3202, and a DC level shift unit 3203.

  The inverse wavelet transform unit 3201 performs inverse wavelet transform (specifically, inverse discrete wavelet transform) on input data composed of wavelet coefficients.

  Here, the inverse conversion non-target data 531 and the inverse conversion target data 532 generated by the first processing unit 321 are input to the recognition data generation unit 320. The inverse transform non-target data 531 is data that is not inverse wavelet transformed by the inverse wavelet transform unit 3201. The inverse transformation non-target data 531 is constituted of only wavelet coefficients of one sub-band of the current hierarchical data 501, for example. The inverse transformation non-target data 531 is composed of only wavelet coefficients of the LL3 sub-band of the current hierarchical data 501, for example. Further, the inverse transformation non-target data 531 is configured by, for example, wavelet coefficients of the HH3 sub-band of the current hierarchical data 501. Further, the inverse transformation non-target data 531 is configured by, for example, wavelet coefficients of the HL1 sub-band of the current hierarchical data 501. It can be said that the inverse transform non-target data 531 is data that can not be inverse wavelet transform.

  On the other hand, the inverse transform target data 532 is data that can be inverse wavelet transformed, and is inverse wavelet transformed by the inverse wavelet transformer 3201. The inverse transformation target data 532 includes, for example, only wavelet coefficients of the LL subband, LH subband, HL subband and HH subband at the same decomposition level in the current hierarchical data 501. Further, the inverse transformation target data 532 can be, for example, wavelet coefficients of the same decomposition level in the current hierarchical data 501, wavelet coefficients of the HL subband and the HH subband, and LL bands of the same decomposition level, It consists only of wavelet coefficients of multiple sub-bands of lower decomposition level.

  When the lowest decomposition level among the decomposition levels of wavelet coefficients included in the inverse conversion target data 532 is not 1, the inverse wavelet transformation unit 3201 generates LL subdivisions of the decomposition level one lower than the lowest decomposition level. Inverse wavelet transform is performed on the inverse transform target data 532 so as to obtain band wavelet coefficients. On the other hand, the inverse wavelet transform unit 3201 performs inverse transformation so that the pixel value of the original image can be obtained when the lowest decomposition level among the decomposition levels of wavelet coefficients included in the inverse transformation target data 532 is 1. Inverse wavelet transform is performed on the target data 532.

  For example, it is assumed that the inverse transformation target data 532 is composed of wavelet coefficients of the LL3 subband, the LH3 subband, the HL3 subband, and the HH3 subband of the decomposition level 3. In this case, in the inverse wavelet transform unit 3201, wavelet coefficients of the LL2 sub-band of the decomposition level 2 are obtained.

  Also, inverse transform target data 532 includes wavelet coefficients of LH2 subband, HL2 subband and HH2 subband at decomposition level 2, and wavelet of LL3 subband, LH3 subband, HL3 subband and HH3 subband at decomposition level 3. Consider the case of being composed of coefficients. In this case, in the inverse wavelet transform unit 3201, wavelet coefficients of the LL1 sub-band of the decomposition level 1 are obtained. The LL3 sub-band, the LH3 sub-band, the HL3 sub-band and the HH3 sub-band of the decomposition level 3 are sub-bands capable of recovering the LL2 sub-band of the decomposition level 2.

  In addition, inverse transformation target data 532 includes wavelet coefficients of LH1 subband, HL1 subband and HH1 subband at decomposition level 1, LH2 subband of HL2 decomposition level, HL2 subband and HH2 subband, and LL3 of decomposition level 3. A case is considered where it is composed of wavelets of subbands, LH3 subbands, HL3 subbands and HH3 subbands. In this case, the inverse wavelet transform unit 3201 obtains pixel values of the original image (frame image). The LH2 subband, the HL2 subband and the HH subband at decomposition level 2 and the LL3 subband, the LH3 subband, the HL3 subband and the HH3 subband at decomposition level 3 are subbands capable of restoring the LL1 subband at decomposition level 1 It is.

  The wavelet coefficients of the LL sub-band generated by the inverse wavelet transform unit 3201 are output to the first processing unit 321 as LL data 530. That is, the wavelet coefficient of the LL3 subband, the wavelet coefficient of the LL2 subband, and the wavelet coefficient of the LL1 subband, which are generated by the inverse wavelet transform unit 3201, are input to the first processing unit 321 as the LL data 530, respectively.

  Here, in the gateway 3, wavelet coefficients output from the inverse wavelet transform unit 3201 are treated as pixel values. Therefore, it can be said that the inverse wavelet transform unit 3201 outputs image data including a plurality of pixel values. In addition, wavelet coefficients included in the inverse transformation non-target data 531 are also treated as pixel values. Therefore, it can be said that the inverse conversion non-target data 531 is a kind of image data.

  The color space conversion unit 3202 performs processing reverse to that of the color space conversion unit 222 in FIG. 5 on the image data output from the inverse wavelet conversion unit 3201. The color space conversion unit 3202 performs processing reverse to the processing in the color space conversion unit 222 on the inverse conversion non-target data 531 (image data) input to the recognition data generation unit 320. The DC level shift unit 3203 converts the DC level of the image data output from the color space conversion unit 3202 as necessary. The image data output from the DC level shift unit 3203 is the recognition data 522.

  As can be understood from the above description, the recognition data 522 is at least part of the restored image data 500 or at least part of the restored subbands of the current hierarchical data 501. Therefore, the recognition target image indicated by the recognition data 522 indicates at least a part of a frame image (frame image which is not divided into sub-bands) or at least a part of sub-band images.

<First processing unit>
The first processing unit 321 performs data writing processing, mask generation processing, input data generation processing, and instruction information generation processing.

<Data writing process>
The first processing unit 321 stores, in the first memory 323, the decoded non-difference data 521 input from the selection unit 325. Further, the first processing unit 321 stores the restored current non-difference CB, which is input from the restoration unit 326, in the first memory 323. Further, the first processing unit 321 stores the LL data 530 input from the recognition data generation unit 320 in the first memory 323.

<Mask generation processing>
The first processing unit 321 generates mask data 525 based on the recognition result information 523 output from the image recognition unit 33.

  Here, when the image recognition unit 33 detects a detection target from the recognition target image indicated by the recognition data 522, the image recognition unit 33 outputs recognition result information 523 including detection information indicating that the detection target has been detected. On the other hand, when the detection target is not detected from the recognition target image, the image recognition unit 33 outputs the recognition result information 523 including undetected information indicating that the detection target is not detected.

  When receiving the recognition result information 523 including the detection information, the first processing unit 321 specifies a detection target object image in which the detection target object detected by the image recognition unit 33 is captured in the target frame image. The first processing unit 321 can specify the detection target image based on the coordinate table 36 a in the coordinate memory 36. The first processing unit 321 sets the identified detection target image as an ROI in the target frame image, and sets the other region as a non-ROI. When the image recognition unit 33 detects a plurality of detection target objects from the target frame image, a plurality of detection target object images in which the plurality of detection target objects respectively appear in the target frame image are set as ROIs.

  The first processing unit 321 generates a use mask for determining an ROI and a non-ROI in the target frame image. This use mask is a mask corresponding to the wavelet plane, similar to the differential frame mask 5120 shown in FIG. 14 described above. The mask used distinguishes wavelet coefficients (referred to as ROI coefficients) involved in the ROI and wavelet coefficients (referred to as non-ROI coefficients) involved in the non-ROI in the wavelet plane obtained by wavelet transforming the entire target frame image. It can be said that it is a mask to The used mask can be grasped as an aggregate of masks for each sub-band included in the wavelet plane, similarly to the differential frame mask 5120. The first processing unit 321 inputs the generated data of the used mask as the mask data 525 to the transcoder 34.

  The mask used may be a mask for discriminating between the ROI coefficient and the non-ROI coefficient in the wavelet plane obtained by wavelet transformation of only a part of the target frame image.

  In the above-described example, the first processing unit 321 sets the image of the detection target detected in the image recognition process as the ROI, but may use another part of the target frame image as the ROI. The first processing unit 321 can freely set the ROI in the target frame image. For example, the first processing unit 321 may set a portion designated by the cloud server 5 as the ROI in the target frame image.

<Input data generation process>
The first processing unit 321 uses the data in the first memory 323 to generate inverse conversion non-target data 531 and inverse conversion target data 532 which are input data input to the recognition data generation unit 320. The first processing unit 321 determines what kind of image the recognition target image is to be, and generates inverse conversion non-target data 531 or inverse conversion target data 532 accordingly. For example, when the entire LL3 sub-band image is to be recognized as a recognition target image, the first processing unit 321 generates inverse conversion non-target data 531 configured by the LL3 sub-band of the current hierarchical data 501, and uses the recognition data The data is input to the generation unit 320. When the first processing unit 321 sets the entire LL2 subband image as a recognition target image, the first processing unit 321 is an inverse transform composed of the LL3 subband, the LH3 subband, the HL3 subband, and the HH3 subband of the current hierarchical data 501. The target data 532 is generated and input to the recognition data generation unit 320. The image to be recognized is determined based on the past recognition result information 523, the type of detection object, the shooting range of the camera 20 of the IoT terminal 2, and the like.

<Instruction information generation process>
The first processing unit 321 determines data to be transmitted by the IoT terminal 2 based on the recognition result information 523 and the like, and generates instruction information 520 for instructing to transmit the determined data (designated data). . The first processing unit 321 generates difference transmission instruction information 520 when specifying the difference CB as data to be transmitted by the IoT terminal 2. On the other hand, the first processing unit 321 generates the non-differential transmission instruction information 520 when designating the non-differential CB as the data to be transmitted by the IoT terminal 2.

<Second processing unit>
The second processing unit 322 generates transcoder decoded data 526 based on the data read from the first memory 323. In this example, since the mask data 525 generated by the first processing unit 321 corresponds to the wavelet plane of the maximum decomposition level obtained by wavelet transformation of the entire target frame image, the second processing unit 322 The first memory 323 reads a plurality of wavelet coefficients constituting the wavelet plane of the maximum decomposition level, that is, first wavelet coefficient data. The first wavelet coefficient data is data indicating the entire target frame image. Then, the second processing unit 322 outputs the read first wavelet coefficient data as transcoder decoded data 526 to the transcoder 34.

  If the mask data 525 is data corresponding to a part of the target frame image, a plurality of wavelet coefficients for restoring the part are read from the first memory 323, and the plurality of read wavelet coefficients are read. It may be the decoded data 526 for transcoder.

  The second processing unit 322 also generates transcoder encoded data 527 based on the data read from the second memory 324. For example, the second processing unit 322 reads, from the second memory 324, a plurality of encoded non-differential CBs constituting the wavelet plane of the maximum decomposition level, that is, encoded first quantized wavelet coefficient data. It can be said that the plurality of coded non-differential CBs constituting the wavelet plane of the maximum decomposition level are the plurality of coded non-differential CBs for restoring the entire target frame image. Then, the second processing unit 322 outputs the plurality of read encoded non-differential CBs as transcoder encoded data 527 to the transcoder 34.

  The second processing unit 322 reads a plurality of coded non-differential CBs for restoring a part of the target frame image from the second memory 324, and reads the plurality of read coded non-differential CBs for a transcoder. Alternatively, the conversion data 527 may be used.

<Detailed Description of Image Recognition Unit>
FIG. 26 shows an example of the configuration of the image recognition unit 33. As shown in FIG. FIG. 27 shows the operation of the image recognition unit 33. As shown in FIG. As shown in FIG. 26, the image recognition unit 33 includes a preprocessing unit 330 and an image recognition engine 334. The preprocessing unit 330 includes a memory 331, a separation unit 332, and a normalization unit 333.

  The memory 331 stores the recognition data 522 from the data processing unit 32 (see <data storage> in FIG. 27). The separation unit 332 selects a plurality of partial images 601 from the recognition target image 600 indicated by the recognition data 522 in the memory 331 (see <selection> in FIG. 27). In the example of FIG. 27, each partial image 601 overlaps with at least one other partial image 601. Then, the separation unit 332 separates the plurality of selected partial images 601 from one another (see <separation> in FIG. 27). The normalization unit 333 normalizes each of the plurality of partial images 601 separated by the separation unit 332 to generate a plurality of normalized partial images 602 (see <Normalization> in FIG. 27). Data indicating each normalized partial image 602 generated by the normalization unit 333 is input to the image recognition engine 334.

  The image recognition engine 334 performs an image recognition process on each normalized partial image 602 based on the data input from the pre-processing unit 330. For example, when a detection target is detected from at least one of the plurality of input normalized partial images 602, the image recognition engine 334 inputs recognition result information 523 including detection information to the data processing unit 32. On the other hand, when the detection target is not detected from all of the plurality of input normalized partial images 602, the image recognition engine 334 inputs the recognition result information 523 including non-detection information to the data processing unit 32. Further, the image recognition engine 334 generates recognition result information 524 including target object information related to the detected detection target and inputs it to the communication unit 35. For example, when the detection target is a person, the object information includes, for example, the gender and age of the detected person. The object information may also include information indicating the position of the detection object within the imaging range of the camera 20. The information contained in the object information depends on the information that the image recognition engine can specify by image recognition. The communication unit 35 transmits metadata including the recognition result information 524 to the cloud server 5. The recognition result information 524 may be the same as the recognition result information 523.

  Various methods can be considered as a method for the image recognition engine 334 to detect a detection target from the normalized partial image 602. For example, the image recognition engine 334 extracts, from the normalized partial image 602, a feature amount indicating the feature of the detection target. As this feature amount, for example, an edge, color, Haar-like, HOG (Histogram of Oriented Gradients) or LBP (Local Binary Pattern) can be considered. When the image recognition engine 334 extracts the feature amount, the image recognition engine 334 inputs the extracted feature amount to the classifier included in the image recognition engine 334. The discriminator determines whether or not a detection target image is present in the normalized partial image 602 based on the input feature amount, and outputs the determination result. As a classifier, for example, a neural network, SVM (Support Vector Machine) or Adaboost is used. The image recognition engine 334 extracts a plurality of types of feature amounts from the normalized partial image 602, and based on the extracted plurality of types of feature amounts, whether or not a detection target object image exists in the normalized partial image 602 May be determined.

  Further, the image recognition unit 33 may extract the feature amount from the recognition target image 600 indicated by the recognition data 522 instead of extracting the feature amount from the normalized partial image 602. In this case, for example, the image recognition unit 33 selects the plurality of partial images 601 from the recognition target image 600 based on the extracted feature amount, normalizes each of the plurality of selected partial images 601, and sets the plurality of partial images 601. A normalized partial image 602 is generated. Then, the image recognition unit 33 specifies a feature corresponding to each normalized partial image 602 in the feature extracted from the recognition target image 600, and inputs the specified feature to the classifier.

  The image recognition engine 334 may use a neural network of a multilayer structure capable of detecting a detection target without performing extraction of feature amounts, as in deep learning.

<Detailed Description of Transcoder>
FIG. 28 shows an example of the transcoder 34. As shown in FIG. As shown in FIG. 28, the transcoder 34 includes a quantization unit 340, an encoding unit 341, and a bit stream generation unit 344. The transcoder 34 converts the input data into a bit stream without wavelet transform and inputs the bit stream to the communication unit 35.

  The quantization unit 340 performs scalar quantization on the decoded data for transcoder 526 composed of a plurality of wavelet coefficients, which is output from the data processing unit 32, based on the quantization step size, thereby the third quantization process. Generate quantized wavelet coefficient data. At this time, the quantization unit 340 determines, based on the mask data 525 from the data processing unit 32, the ROI coefficient and the non-ROI coefficient for each wavelet coefficient of the transcoder decoded data 526. Then, the quantization unit 340 quantizes each wavelet coefficient of the transcoder decoded data 526 such that the non-ROI coefficient after quantization becomes 0. As a result, the third quantized wavelet coefficient data will show only the ROI.

  The encoding unit 341 compresses and encodes the third quantized wavelet coefficient data generated by the quantization unit 340 to generate encoded data 590. The encoding unit 341 includes a coefficient bit modeling unit 342 and an entropy encoding unit 343.

  The coefficient bit modeling unit 342 performs bit modeling processing on the third quantized wavelet coefficient data. This bit modeling process is the same as the bit modeling process in the coefficient bit modeling unit 241 of the IoT terminal 2. The entropy coding unit 343 performs entropy coding on the data generated by the coefficient bit modeling unit 342 to generate coded data 590. The encoded data 590 is data indicating only the ROI specified by the mask data 525. In the present example, the encoded data 590 is data indicating an object image to be detected. For example, arithmetic coding is used as entropy coding. The encoding unit 341 may perform rate control on the encoded data 590 generated by the entropy encoding unit 343 to control the code amount.

  The bitstream generation unit 344 generates a bitstream 529 including the encoded data 590. Specifically, the bit stream generation unit 344 packetizes the encoded data 590 and generates a bit stream 529 including packet data and additional information generated thereby. The additional information includes a packet header, layer configuration information, scalability information, and a quantization table. The packet header includes zero-length packet information, code block inclusion information, zero bit plane information, coding path number information, and code amount information of the code block.

  Further, the bitstream generation unit 344 generates a bitstream 529 including transcoder encoded data 527 output from the data processing unit 32. The bitstream generator 344 packetizes the transcoder encoded data 527 and generates a bitstream 529 including packet data and additional information generated thereby.

  The bit stream 529 generated by the bit stream generation unit 344 is transmitted from the communication unit 35 to the cloud server 5. If the bitstream 529 includes the encoded data 590 indicating the ROI, the gateway 3 can transmit the data indicating the ROI to the cloud server 5. The gateway 3 can freely set the ROI, so that, for example, the ROI desired by the cloud server 5 can be transmitted to the cloud server 5.

  In addition, when the bit stream 529 includes transcoder encoded data 527 indicating, for example, the entire frame image, the gateway 3 can transmit data indicating the entire frame image to the cloud server 5. Thereby, the gateway 3 can stream and transmit the moving image captured by the camera 20 of the IoT terminal 2 to the cloud server 5. The gateway 3 can stream and transmit a moving image, for example, in response to a request from the cloud server 5.

  The second processing unit 322 of the data processing unit 32 inputs the decoded data for transcoder 526 into the transcoder 34 in response to a request from the cloud server 5, etc. Decide if to enter.

  When the image of the detection object detected by the image recognition process is set as the ROI, the communication unit 35 recognizes the bitstream 529 including the encoded data 590 and the recognition result information 524 including the object information on the detection object. And to the cloud server 5.

  Thus, the transcoder 34 receives the transcoder decoded data 516 and the transcoder encoded data 527 which are data after wavelet transform. Therefore, unlike the IoT terminal 2, the transcoder 34 can generate the bitstream 529 without wavelet transformation of input data representing an image. Therefore, the bit stream 529 can be generated by simple processing.

  Also, the transcoder encoded data 527 is data that has been compression encoded. Therefore, when the transcoder encoded data 527 is input, the transcoder 34 can generate the bit stream 529 without compressing and encoding the input data, unlike the IoT terminal 2. Therefore, the bit stream 529 can be generated by simpler processing.

<Operation Example of Image Processing System>
Next, an operation example of the entire image processing system 4 will be described. Hereinafter, as an example, an operation of the image processing system 4 in the case where the gateway 3 performs the image recognition processing on the LL sub-band image in the descending order of the decomposition level until the detection target is detected will be described.

  When the camera 20 of the IoT terminal 2 starts shooting a moving image, the image processing system 4 performs preprocessing. In the pre-processing, using the non-differential transmission instruction information 520, the gateway 3 first transmits, to the IoT terminal 2, the non-differential layered data 501 indicating the entire first frame image of the moving image for which imaging has started. To direct. In the IoT terminal 2 that has received the non-differential transmission instruction information 520, the determination unit 26 inputs the non-differential layered data 501 indicating the entire first frame image as the transmission target data 503 to the encoding device 24. Thereby, the bit stream 506 including the encoded non-differential data 505 indicating the entire first frame image is transmitted from the IoT terminal 2. In the gateway 3 having received the bit stream 506, the decoding device 31 performs decompression decoding on the encoded non-differential data 505 included in the bit stream 506, and decodes the non-differential to indicate the entire first frame image. Data 521 is generated. The data processing unit 32 stores the decoded non-differential data 521 generated by the decoding device 31 in the first memory 323. This completes the preprocessing. When the preprocessing is completed, each wavelet coefficient of the non-differential layered data 501 indicating the entire first frame image is stored in the first memory 323.

  When the preprocessing is completed, the image processing system 4 performs the operations shown in FIGS. FIGS. 29 and 30 are diagrams respectively showing an example of the operation of the gateway 3 and the IoT terminal 2 of the image processing system 4 that performs processing on the target frame image after preprocessing. In the example of FIGS. 29 and 30, the gateway 3 transmits the differential transmission instruction information 520 to the IoT terminal 2, and the IoT terminal 2 transmits the differential code block according to the differential transmission instruction information 520 to the gateway 3.

  When processing on the target frame image starts, as shown in FIG. 29, in step s11, the first processing unit 321 of the gateway 3 targets the LL subband of the maximum decomposition level, in this example, the LL3 subband. . Then, the first processing unit 321 sets a variable LV indicating the decomposition level of the LL sub-band to be processed to 3. Hereinafter, the LL subband to be processed is referred to as a target LL subband. Also, the target LL subband may be regarded as an image and may be referred to as a target LL subband image. Also, the decomposition level of the target LL subband may be referred to as the target decomposition level. The first processing unit 321 determines the differential code block of the target LL subband as designated data.

  Next, in step s12, the first processing unit 321 generates difference transmission instruction information 520 for notifying the IoT terminal 2 of the designation data determined in step s11, and inputs the information to the transmission unit 30b. The transmitting unit 30 b transmits the input difference transmission instruction information 520 to the IoT terminal 2.

  In the IoT terminal 2, as shown in FIG. 30, the reception unit 25b receives the difference transmission instruction information 520 in step s31 and inputs the information to the determination unit 26. Next, in step s32, the determination unit 26 determines transmission target data 503 from the difference hierarchical data 502 based on the input difference transmission instruction information 520 and the integrated mask data 513. In the first step s32 after processing of the target frame image starts, the determination unit 26 performs grouping and expansion processing on the second ROI mask portion of each integrated subband mask indicated by the integrated mask data 513. And the labeling process are sequentially performed. As a result, a unique label is assigned to the independent region (island region) included in the second ROI mask portion. Hereinafter, this independent area may be called a "label area".

  FIG. 31 is a diagram showing the grouping process and the labeling process performed on the integrated subband mask 5126 corresponding to the decomposition level 3. In FIG. 31, a plurality of code blocks 5021a to 5021i of the subband LL3 are superimposed on the integrated subband mask 5126. In FIG. 31, an integrated subband mask 5126 different from the example shown in FIGS. 18 and 19 described above is shown. In the example of FIG. 31, the second ROI mask portion (opened portion) is divided into a label area L0 of label 0 and a label area L1 of label 1. The second ROI mask portion of the integrated subband mask 5126 corresponding to the other decomposition levels is also divided into label regions L0 and L1.

  After performing the labeling process, the determining unit 26 performs a plurality of code blocks of the target LL subbands on the integrated subband mask after the labeling process corresponding to the target LL subbands, as shown in FIG. 31 described above. Layer Hereinafter, the integrated subband mask corresponding to the target LL subband may be referred to as a “target integrated subband mask”.

  Next, the determination unit 26 sets the label with the smallest number as the label to be processed (hereinafter, may be referred to as a target label). Then, the determination unit 26 sets, as transmission target data 503, a differential code block corresponding to the label area of the target label among the plurality of code blocks of the target LL subband. Specifically, the determination unit 26 sets, as transmission target data 503, a differential code block including at least a part of the label area of the target label among the plurality of code blocks of the target LL subband. Hereinafter, the label area of the target label may be referred to as a target label area.

  In the example of FIG. 31, the label 0 is the target label. Then, differential code blocks 5021b, 5021c, 5021e, and 5021f corresponding to the label area L0 are set as transmission target data 503. As will be described later, the differential code block corresponding to the label area of the other label is set as transmission target data 503 later.

  When the determination unit 26 determines the transmission target data 503, in step s33, the determination unit 26 generates coordinate data 504 including coordinates indicating the position on the wavelet plane for each differential code block included in the transmission target data 503.

  Next, in step s34, the encoding device 24 compresses and encodes the transmission target data 503 to generate encoded data 505. Next, in step s35, the encoding device 24 generates a bit stream 506 including the encoded data 505. The transmitting unit 25a transmits the bit stream 506 and the coordinate data 504 generated in step s33 to the gateway 3. At this time, the IoT terminal 2 notifies the gateway 3 of the label number obtained in the labeling process, and notifies the gateway 3 of the current target label. In the example of FIG. 31, while the label numbers 0 and 1 are notified to the gateway 3, the gateway 3 is notified that the current target label is the label 0.

  Referring back to FIG. 29, after step s12, the receiving unit 30a of the gateway 3 receives the bit stream 506 and the coordinate data 504 from the IoT terminal 2 in step s13. Then, in step s14, the encoded data 505 included in the bit stream 506 is stored in the second memory 324, and the coordinate data 504 is stored in the coordinate table 36a of the coordinate memory 36. Further, the encoded data 505 is expanded and decoded by the decoding device 31 to generate decoded difference data 521.

  Next, in step s15, the data processing unit 32 restores the current non-differential CB corresponding to the respective decoded differences CB included in the decoded differential data 521. Then, the data processing unit 32 stores the wavelet coefficients of the restored current non-difference CB in the first memory 323. Here, the wavelet coefficients of the non-differential layered data 501 indicating the entire first frame image are stored in the first memory 323 by the above-described pre-processing. That is, in the first memory 323, each non-difference CB of the non-difference hierarchical data 501 indicating the entire first frame image is stored. The data processing unit 32 restores the current non-differential CB using this non-differential CB as a past non-differential CB.

  Next, in step s16, the first processing unit 321 generates input data to the recognition data generation unit 320. Here, the first processing unit 321 corresponds to the portion corresponding to the target label area in the target LL sub-bands of the wavelet plane (hereinafter may be referred to as the target wavelet plane) obtained by wavelet transforming the target frame image. , As a recognition target image. Then, the first processing unit 321 reads, from the first memory 323, the non-differential CB (the restored non-differential CB) corresponding to the target label area in the target LL subband of the target wavelet plane. That is, when the first processing unit 321 superimposes the integrated subband mask corresponding to the target decomposition level on the target LL subband of the target wavelet plane, at least a part of the target label region of the integrated subband mask Are read from the first memory 323.

  Here, as described above, the IoT terminal 2 transmits, to the gateway 3, coordinate data 504 including the coordinates of each code block included in the transmission target data 503. Then, the gateway 3 registers the received coordinate data 504 in the coordinate table 36a. Therefore, the first processing unit 321 can specify the coordinates of the non-difference CB corresponding to the target label area in the target LL subband of the target wavelet plane by referring to the coordinate table 36a. Therefore, the first processing unit 321 can read out the non-difference CB from the first memory 323. The first processing unit 321 inputs the inverse conversion non-target data 531 configured by the read non-difference CB to the recognition data generation unit 320.

  FIG. 32 is a diagram showing an example of a state in which the integrated subband mask 5126 corresponding to the target decomposition level is superimposed on the target LL subband including the plurality of non-differential CBs 5011 a to 5011 i. In the example of FIG. 32, the target decomposition level is three. In the example of FIG. 32, assuming that the target label area is the label area L0, the non-differential CB read by the first processing unit 321 from the first memory 323 is non-differential CB 5011 b, 5011 c, including at least a part of the label area L0. It becomes 5011e and 5011f.

  Next, in step s17, the recognition data generation unit 320 generates recognition data 522 based on the inverse conversion non-target data 531. The inverse transformation non-target data 531 is input to the color space conversion unit 3202 without being inverse wavelet transformed. The recognition data 522 indicates an image corresponding to the target label area in the target LL subband image of the target wavelet plane.

  Next, in step s18, the image recognition unit 33 performs an image recognition process on the image indicated by the recognition data 522 generated in step s17. When a detection target is detected in the image recognition process, the data processing unit 32 sets the target label as an end label.

  Next, in step s19, the data processing unit 32 performs termination determination to determine whether or not the processing on the target frame image is to be terminated.

  FIG. 33 is a flowchart showing an example of the end determination. As shown in FIG. 33, in step s191, the data processing unit 32 detects an object to be detected for all labels defined by the IoT terminal 2 based on the result of the image recognition process in the past performed by the image recognition unit 33. Identify whether or not is detected. That is, the data processing unit 32 specifies, for each label defined by the IoT terminal 2, whether or not the detection target is detected from the code block corresponding to the label area of the label. When the data processing unit 32 specifies that the detection target has been detected for all the labels, in step s192, the data processing unit 32 determines to end the processing on the target frame image. Thus, the end determination ends.

  On the other hand, when the data processing unit 32 determines NO in step s191, it determines whether or not the value of the variable LN indicating the target label matches the maximum value max1 in step s193. Here, the maximum value max1 means the largest label among the labels excluding the end label in the label determined by the IoT terminal 2. When the label excluding the end label in the label defined by the IoT terminal 2 is called the label of the processing target candidate, the maximum value max1 means the maximum value among the labels of the processing target candidate. For example, as in the example of FIG. 31, when the IoT terminal 2 defines labels 0 and 1 and the current end label (the label in which the object to be detected is detected) is 0, the maximum value max1 is 1. Further, for example, when the IoT terminal 2 defines labels 0 to 3 and the current end labels are 0 and 3, the maximum value max1 is 2. When the end label does not exist, the maximum value max1 matches the maximum value of the labels determined by the IoT terminal 2.

  When it is determined in step s193 that the value of the variable LN matches the maximum value max1, in step s194, the data processing unit 32 determines whether the value of the variable LV indicating the decomposition level of the target LL subband is 1 or not. Determine If the data processing unit 32 determines that the value of the variable LV is 1, that is, if the target LL sub-band is the LL1 sub-band, the data processing unit 32 executes step s182 and determines to end the processing for the target frame image. .

  When it is determined in step s194 that the value of the variable LV is not 1, the data processing unit 32 decreases the value of the variable LV by one in step s195. From this, the LL subband of the decomposition level which is smaller by one than the decomposition level of the target LL subband up to now is the target LL subband. After step s195, in step s196, the data processing unit 32 sets the value of the variable LN to the minimum value min1. Here, the minimum value min1 means the smallest label among the labels of the processing target candidates. For example, as in the example of FIG. 31, when the IoT terminal 2 defines labels 0 and 1, and the label of the current end label area is 0, the minimum value min1 is 1. Also, for example, when the IoT terminal 2 defines labels 0 to 3 and the label of the current end label area is 0 or 3, the minimum value min1 is 1. By setting the value of the variable LN to the minimum value min1, the minimum label among the labels of the processing target candidate is set as a new target label. When there is no end label, the minimum value min1 matches the minimum value of the labels defined by the IoT terminal 2.

  After step s196, in step s197, the data processing unit 32 determines to continue the process on the target frame image. Thus, the end determination ends.

  If the value of the variable LN is not the maximum value max1 in step s193, the data processing unit 32 changes the value of the variable LN to the next value in step s198. Specifically, the data processing unit 32 changes the value of the variable LN to the label of the next larger value of the labels of the processing target candidates than the current value of the variable LN. Thereafter, step s197 is performed to determine continuation of the process on the target frame image.

  Referring back to FIG. 29, if it is determined that the data processing unit 32 ends the process on the target frame image in the end determination of step s19, the gateway 3 notifies in step s20 that the process on the target frame image is ended. End notification to the IoT terminal 2. When the detection target is detected from the target frame image, the gateway 3 includes, in step s21 after step s20, a bitstream including encoded data 590 indicating a detection target image (ROI) in which the detection target is captured. 529 is generated by the transcoder 34. When a plurality of detection targets are detected from the target frame image, a bit stream 529 including encoded data 590 indicating a plurality of detection target images in which the plurality of detection targets are respectively captured is generated. Then, the gateway 3 transmits the generated bit stream 529 and the recognition result information 524 on the target frame image generated by the image recognition unit 33 from the communication unit 35 to the cloud server 5. Thus, the process on the target frame image is completed.

  If no detection target is detected from the target frame image, the process on the target frame image is ended without executing step s21. Alternatively, after the recognition result information 524 including the information indicating that the detection target is not detected is transmitted from the communication unit 35 to the cloud server 5, the processing on the target frame image may end.

  If it is determined in step s19 that the data processing unit 32 continues the process on the target frame image in the end determination of step s19, the gateway 3 executes step s11 again to determine designated data. In step s11, designated data is determined based on the values currently indicated by the variables LN and LV.

  When the target decomposition level currently indicated by the variable LV is the maximum decomposition level, ie, 3, the first processing unit 321 indicates by the variable LN in the integrated subband mask corresponding to the decomposition level 3 among the LL3 subbands. The differential code block corresponding to the label area of the target label is set as designated data.

  In addition, when the target decomposition level currently indicated by the variable LV is other than the maximum decomposition level, that is, smaller than 3, the first processing unit 321 performs an LH sub-band with one decomposition level higher than the target decomposition level, HL sub The differential CB corresponding to the label area of the target label in the band and the HH subband is designated as designated data. This label area is the label area of the integrated subband mask corresponding to the decomposition level one higher than the target decomposition level.

  Here, in the present example, as can be understood from the above description and the following description, when the target decomposition level is smaller than 3, the gateway 3 sets the LL subband one decomposition level higher than the target decomposition level. In the first memory 323, the non-differential CB corresponding to the target label area is stored.

  In addition, the inverse wavelet transform unit 3201 of the recognition data generation unit 320 of the gateway 3 performs inverse wavelet transform of the LL3 subband, the LH3 subband, the HL3 subband, and the non-differential CB of the HH3 subband to generate LL2 subbands. Non-differential CB can be restored. Similarly, the inverse wavelet transform unit 3201 restores the non-differential CB of the LL1 subband by inverse wavelet transforming the non-differential CB of the LL2 subband, the LH2 subband, the HL2 subband, and the HH2 subband.

  Then, the restoration unit 326 of the data processing unit 32 of the gateway 3 can restore the current non-difference CB from the difference CB (decoding difference CB) from the IoT terminal 2.

  From the above, when the target decomposition level is smaller than 3, the gateway 3 is required to restore the non-differential CB corresponding to the target label area in the target LL subband by determining the designated data as described above. Data can be obtained. That is, the gateway 3 can obtain non-differential CBs corresponding to the target label area in the LL sub-band, the LH sub-band, the HL sub-band, and the HH sub-band whose decomposition level is one higher than the target decomposition level. Therefore, the gateway 3 can restore the differential CB corresponding to the label area of the target label in the LL subband of the target decomposition level by determining the designated data as described above.

  For example, when the target LL sub-band is the LL2 sub-band and the target label is the label 1, the first processing unit 321 determines that the label 1 in the LH3 sub-band, the HL3 sub-band, and the HH3 sub-band of the decomposition level 3 is included. The differential CB corresponding to the label area L1 is designated as designated data. When the target LL subband is the LL2 subband, the gateway 3 has already received from the IoT terminal 2 the non-differential CB corresponding to the label region L1 of the label 1 in the LL3 subband and stored in the first memory 323 . Also, in the restoration unit 326, the gateway 3 generates the LH3 subband of the decomposition level 3 and the HL3 from the difference CB corresponding to the label area L1 of the label 1 in the LH3 subband, the HL3 subband and the HH3 subband of the decomposition level 3. A non-differential CB corresponding to the label region L1 of label 1 in the subband and the HH3 subband can be restored. Therefore, the gateway 3 can obtain the non-differential CB corresponding to the label area L1 in the LL3 subband, the LH3 subband, the HL3 subband, and the HH3 subband by determining the designated data in this manner. Thus, the gateway 3 can restore the non-differential CB corresponding to the label region L1 in the LL2 subband by inverse wavelet transform.

  After executing step s11, the gateway 3 generates difference transmission instruction information 520 for notifying of the designated data determined in step s11 in step s12. Then, the gateway 3 transmits the generated difference transmission instruction information 520 to the IoT terminal 2.

  When the IoT terminal 2 receives the difference transmission instruction information 520 in step s31, in step s32, the IoT terminal 2 determines the transmission target data 503 from the difference hierarchical data 502 based on the received difference transmission instruction information 520 and the integrated mask data 513. Do. The determination unit 26 of the IoT terminal 2 specifies the differential CB (decoding differential CB) specified by the differential transmission instruction information 520 in the differential layered data 502 based on the integrated mask data 513. Then, the determination unit 26 sets the identified difference CB as the transmission target data 503.

  When the transmission target data 503 is determined in step s32, the IoT terminal 2 executes steps s33, s34, and s35 in the same manner as described above. After step s35, in step s36, when receiving the end notification from the gateway 3, the IoT terminal 2 ends the process on the target frame image. On the other hand, when the IoT terminal 2 receives the difference transmission instruction information 520 without receiving the end notification from the gateway 3 after step s35 (step s31), it executes step s32 in the same manner as described above, The same applies thereafter.

  In step s32, the determination unit 26 specifies the difference CB specified by the difference transmission instruction information 520 in the difference hierarchical data 502 based on the integrated mask data 513, and of the specified differences CB, has already been transmitted to the gateway 3. The difference CB other than the transmitted difference CB may be used as the transmission target data 503. In this case, in step s33, the determination unit 26 also includes, in the coordinate data 504, the coordinates of the differential CB (the differential CB already transmitted) which is not included in the transmission target data 503 among the identified differential CBs. Thus, even if all of the differential CBs corresponding to the target label area are not transmitted in step s35, the gateway 3 specifies the non-differential CBs corresponding to the target label area in the sub-band in step s16. Can.

  As described above, by using the difference CB other than the difference CB already transmitted to the gateway 3 among the difference CBs specified by the difference transmission instruction information 520 as the transmission target data 503, the determination unit 26 The amount of data to be sent to the gateway 3 can be reduced.

  For example, in the example of FIG. 31, the designated data designated by the difference transmission instruction information 520 is the difference CB corresponding to the label region L1 in the LL3 subband, and the four differences CB 5021b, 5021c, corresponding to the label region L0. 5021 e. It is assumed that 5021 f has already been sent to the gateway 3. In this case, the determination unit 26 specifies four differences CB 5021 d, 5021 e, 5021 g, 5021 h corresponding to the label area L1 based on the integrated subband mask 5126 corresponding to the decomposition level 3. Then, the determination unit 26 sets three differences CB 5021 d, 5021 g, 5021 h other than the difference CB 5021 e already transmitted among the differences CB 50 21 d, 5021 e, 5021 g, 5021 h specified as transmission target data 503.

  Similarly, when the designated data designated by differential transmission instruction information 520 is differential CB corresponding to the label area of a label in the LH, HL, and HH subbands of a decomposition level, the determination unit 26. , Identifies the difference CB based on the integrated mask data 513. Then, the determination unit 26 sets, as transmission target data 503, the difference CB excluding the difference CB already transmitted to the gateway 3 among the specified difference CBs.

  Referring back to FIG. 29, when the gateway 3 receives the bit stream 506 and the coordinate data 504 from the IoT terminal 2 in step s13, the gateway 3 executes the above-described steps s14 and s15. Then, in step s16, the gateway 3 generates input data to the recognition data generation unit 320. In step s16, when the current target decomposition level is the maximum decomposition level, ie, 3 in the first processing unit 321, the LL3 sub-band is read from the first memory 323 by referring to the coordinate table 36a as described above. Read out the non-differential CB corresponding to the target label area. Then, the first processing unit 321 inputs the inverse conversion non-target data 531 configured by the read non-difference CB to the recognition data generation unit 320.

  On the other hand, when the current target decomposition level is smaller than 3, the first processing unit 321 determines from the first memory 323 that the LL sub-band, the LH sub-band, and the HL sub-band have a decomposition level higher by one than the target decomposition level. The non-differential CB corresponding to the target label area in the H.sub.H and H.sub.H subbands is read out with reference to the coordinate table 36a. Then, the first processing unit 321 inputs the inverse conversion target data 532 configured by the read non-difference CB to the recognition data generation unit 320.

  Next, in step s17, the recognition data generation unit 320 generates recognition data 522. When the inverse conversion non-target data 531 is input to the recognition data generation unit 320 in the previous step s16, the inverse conversion non-target data 531 is not subjected to the inverse wavelet conversion as described above, and the color space conversion unit Input to 3202 On the other hand, when the inverse transformation target data 532 is input to the recognition data generation unit 320 in the previous step s16, the inverse wavelet transformation unit 3201 performs inverse wavelet transformation on the inverse transformation target data 532. As a result, the inverse wavelet transform unit 3201 generates a non-differential CB composed of wavelet coefficients corresponding to the target label area in the LL subband of the target decomposition level. The non-differential CB is stored in the first memory 323 as the LL data 530. Thereby, the first memory 323 stores the non-differential CB corresponding to the target label area in the LL sub-band of the target decomposition level. If the target decomposition level is 2, the non-differential CB corresponding to the target label area in the LL2 subband is stored, and if the target decomposition level is 1, the non-differential CB corresponding to the target label area in the LL1 subband Is stored.

  The non-difference CB generated by the inverse wavelet transform unit 3201 is input to the color space transform unit 3202. Data output from the color space conversion unit 3202 is input to the DC level shift unit 3203. Then, data output from the DC level shift unit 3203 becomes recognition data 522.

  When the recognition data 522 is generated in step s17, the gateway 3 executes steps s18 and s19 in the same manner as described above. The gateway 3 then operates in the same manner.

  As can be understood from the above description, when labels 0 and 1 shown in FIGS. 31 and 32 are determined in the IoT terminal 2, first, for the non-differential CB corresponding to the label area L0 of label 0 in the LL3 sub-band Image recognition processing is performed. Next, an image recognition process is performed on the non-differential CB corresponding to the label area L1 of the label 1 in the LL3 sub-band.

  When the detection target is not detected from the non-differential CB corresponding to the label region L0 in the LL3 subband and the detection target is not detected from the non-differential CB corresponding to the label region L1 in the LL3 subband, LL2 sub The image recognition process is performed on the non-differential CB corresponding to the label area L0 in the band, and then the image recognition process is performed on the non-differential CB corresponding to the label area L1 in the LL2 subband.

  When the detection target is not detected from the non-differential CB corresponding to the label region L0 in the LL2 subband and the detection target is not detected from the non-differential CB corresponding to the label region L1 in the LL2 subband, LL1 sub The image recognition process is performed on the non-differential CB corresponding to the label area L0 in the band, and then the image recognition process is performed on the non-differential CB corresponding to the label area L1 in the LL1 subband.

  When the processing on the target frame image is completed, the image processing system 4 performing the above-described operation performs the same processing as described above with the new frame image as the target frame image. When the processing on the frame image is completed with the frame image as the target frame image, the image processing system 4 starts processing as a new target frame image a frame image captured later than the frame image. The image processing system 4 may perform processing on frame images taken by the camera 20 for each frame, or may perform processing on a plurality of frames.

  As can be understood from the above description, in the example of FIGS. 29 and 30, the LL sub-bands are subjected to the image recognition processing in order from the one with the highest decomposition level. In other words, the LL sub-bands are subjected to the image recognition process in order from the one with the lowest resolution.

  Here, with respect to the detection target having a large range in the frame image, the detection target is likely to be detected even from the LL sub-band having a high decomposition level (LL sub-band having a low resolution). On the other hand, with respect to a detection target having a small range captured in a frame image, there is a possibility that the detection target can not be detected from the LL sub-band having a high decomposition level. As in the present example, the LL subbands are processed in order from the one with the highest decomposition level to use the LL subband with a large amount of data and the low decomposition level (LL subband with high resolution) Instead, in the frame image, it is possible to detect a detection target having a large coverage. In the example of FIGS. 31 and 32 described above, when the detection target corresponding to the label region L0 of the label 0 is detectable from, for example, the LL3 sub-band, the detection level of the detection target is the resolution level of the LL3 sub-band. Higher resolution levels of the LL2 and LL1 subbands are not used. Therefore, the amount of data transmitted to the gateway 3 by the IoT terminal 2 can be reduced, and the image recognition process at the gateway 3 can be simplified.

  The operations of the image processing system 4 shown in FIGS. 29, 30, and 33 are merely examples, and the image processing system 4 may perform processing different from the processing shown in FIGS. 29, 30, and 33.

  The gateway 3 may use the encoded data 505 from the IoT terminal 2 in processing other than the image recognition processing.

  In the above example, in the determination of the transmission target data 503, although the integrated subband mask common to a plurality of subbands of the same decomposition level is used, the subband mask unique to each subband (subband of the difference frame mask You may use a mask). In this case, in the sub-band, a code block including at least a part of the first ROI mask portion of the corresponding sub-band mask becomes a differential code block.

  Here, since a plurality of sub-bands of the same decomposition level indicate different information from each other, the range of the first ROI mask portion may not completely match among the plurality of sub-bands. Therefore, the IoT terminal 2 gateways the code block of the same coordinates among a plurality of sub-bands of the same decomposition level only by using a sub-band mask specific to each sub-band in the determination of the transmission target data 503. May not be able to send to

  On the other hand, in order for the gateway 3 to generate LL subbands of a certain decomposition level by inverse wavelet transformation, code blocks of the same coordinates are present among a plurality of subbands of a decomposition level lower than the certain decomposition level. It will be necessary.

  As described above, when a common integrated subband mask is used for multiple subbands of the same decomposition level, the IoT terminal 2 transmits, to the gateway 3, a code block of the same coordinates among the multiple subbands. It can be easily sent. Therefore, the gateway 3 can use the code block of the same coordinates among a plurality of subbands of the decomposition level lower than the certain decomposition level in generating the LL subbands of the certain decomposition level.

  In addition, the determination unit 26 of the IoT terminal 2 may determine the transmission target data 503 without using the instruction information 520. In this case, the processing of the IoT terminal 2 is simplified. Moreover, since the gateway 3 does not need to generate the instruction information 520, the processing of the gateway 3 is simplified.

  When the IoT terminal 2 determines the transmission target data 503 without using the instruction information 520, when the processing on the target frame image starts, for example, the entire difference CB included in the LL3 sub-band is set as the transmission target data 503. Send. The gateway 3 restores all non-differential CBs of the LL3 sub-band from all received differential CBs of the LL3 sub-band, and performs an image recognition process using data composed of all restored non-differential CBs as recognition data 522.

  When the gateway 3 detects a detection target, the processing on the target frame image ends. On the other hand, when the gateway 3 does not detect the detection target, the IoT terminal 2 sets the LH3 sub-band and HL3 because the image represented by the data consisting of all non-differential CBs of the LL2 sub-band is the recognition target image. The entire difference CB of the subband and the HH3 subband is transmitted as data to be transmitted. The gateway 3 restores all non-differential CBs of the LH3 subband, the HL3 subband and the HH3 subband from the total differential CB of the LH3 subband, the HL3 subband and the HH3 subband. Then, the gateway 3 performs inverse wavelet transform on inverse transform target data 532 including all non-differential CBs of LL3 subbands already acquired and all non-differential CBs of LH3 subbands, HL3 subbands and HH3 subbands. And generate all non-differential CBs of the LL2 subbands. Then, the gateway 3 performs image recognition processing using data including all non-difference CBs of the LL2 subband as the recognition data 522.

  When the gateway 3 detects a detection target, the processing on the target frame image ends. On the other hand, when the gateway 3 does not detect the detection target, the IoT terminal 2 sets the LH2 sub-band and HL2 because the image represented by the data consisting of all non-differential CBs of the LL1 sub-band is the recognition target image. The entire difference CB of the subband and the HH2 subband is transmitted as transmission target data 503. The gateway 3 restores all non-differential CBs of the LH2 subband, the HL2 subband and the HH2 subband from the total differential CB of the LH2 subband, the HL2 subband and the HH2 subband. Then, the gateway 3 performs inverse wavelet transform on inverse transform target data 532 including all non-differential CBs of LL2 subbands already acquired and all non-differential CBs of LH2 subbands, HL2 subbands and HH2 subbands. Thus, all non-differential CBs of LL1 subbands are generated. Then, the gateway 3 performs an image recognition process using data consisting of all non-differential CBs of the LL1 subband as the recognition data 522. Thereafter, the process on the target frame image is completed.

  The operation of the image processing system 4 in the case where the IoT terminal 2 determines the transmission target data 503 without using the instruction information 520 is not limited to the above example.

  Further, in the examples of FIGS. 29, 30, and 33, the image recognition process is performed in the order of decomposition levels, but the image recognition process may be performed in the order of labels. It is a flowchart which shows an example of completion | finish determination (step s19) in this case of FIG.

  As shown in FIG. 34, in step s201, the data processing unit 32 specifies whether or not a detection target is detected for the target label. That is, the data processing unit 32 specifies whether or not a detection target has been detected in the image recognition process of step s18 immediately before. When the data processing unit 32 specifies that the detection target has been detected regarding the target label, in step s204, the data processing unit 32 determines whether or not the value of the variable LN matches the maximum value max2. Here, unlike the maximum value max1 described above, the maximum value max2 means the maximum value of the labels defined in the IoT terminal 2. In the example of FIGS. 31 and 32, the maximum value max2 = 2.

  When the data processing unit 32 determines in step s204 that the value of the variable LN matches the maximum value max2, the data processing unit 32 determines to end the process on the target frame image. Thus, the end determination ends.

  On the other hand, when the data processing unit 32 determines in step s204 that the value of the variable LN does not match the maximum value max2, in step s205, the value of the variable LN is increased by one. As a result, a label that is one larger than the previous target label is the target label. Then, in step s206, the data processing unit 32 sets the value of the variable LV to the maximum decomposition level, that is, 3. As a result, the target decomposition level becomes the maximum decomposition level. Then, in step s207, the data processing unit 32 determines to continue the process on the target frame image. Thus, the end determination ends.

  When it is determined in step s201 that the detection target is not detected regarding the target label, the data processing unit 32 determines whether the value of the variable LV is 1 in step s202. If the value of the variable LV is 1, that is, if the target decomposition level is 1, the data processing unit 32 executes the above-described step s204, and so on. On the other hand, when the value of the variable LV is not 1, that is, when the target decomposition level is larger than 1, the data processing unit 32 decreases the value of the variable LV by one in step s203. As a result, the decomposition level that is one less than the previous object decomposition levels becomes the object decomposition level. Thereafter, the data processing unit 32 executes step s207 to decide to continue the process on the target frame image.

  When the end process shown in FIG. 34 is executed, in the example of FIGS. 31 and 32, first, the image recognition process is performed on the non-differential CB corresponding to the label area L0 in the LL3 subband. When the detection target is not detected, the image recognition process is performed on the non-differential CB corresponding to the label region L0 in the LL2 subband. When the detection target is not detected, the image recognition processing is performed on the non-differential CB corresponding to the label region L0 in the LL1 subband.

  When an object to be detected is detected in the image recognition process related to label 0, the image recognition process is performed on the non-differential CB corresponding to the label area L1 in the LL3 sub-band. When the detection target is not detected, the image recognition processing is performed on the non-differential CB corresponding to the label region L1 in the LL2 subband. When the detection target is not detected, the image recognition process is performed on the non-differential CB corresponding to the label region L1 in the LL1 subband.

  In the above example, the image recognition processing is performed on the image represented by the recognition data 522 composed of the non-difference CB, but the image represented by the recognition data 522 composed of the difference CB (at least a part of the difference image) Image recognition processing may be performed on the image. In this case, the gateway 3 to the restoration unit 326 become unnecessary, and the configuration of the gateway 3 is simplified.

  Further, in the above example, the transcoder decoded data 526 is configured by the non-differential CB, but may be configured by the differential CB. In this case, the data processing unit 32 generates mask data 525 for determining the ROI and the non-ROI in the difference image. Then, the transcoder 34 generates encoded data 590 indicating the ROI in the difference image, and transmits a bit stream 529 including the encoded data 590 to the cloud server 5. Also, the transcoder encoded data 527 may be configured by the encoded difference CB.

  Further, the IoT terminal 2 may determine the transmission target data 503 from the difference layered data 502 without using the integrated mask data 513. In this case, for example, the IoT terminal 2 sets the data specified by the instruction information 520 from the gateway 3 as the transmission target data 503 as it is.

  Further, when the high-frequency component of the wavelet plane is input to the image recognition unit 33 as the recognition data 522, the data processing unit 32 of the gateway 3 directly recognizes the LH, HL, or HH subbands as the recognition data 522. Alternatively, integrated sub-bands obtained by integrating the LH sub-band, the HL sub-band, and the HH sub-band at the same decomposition level may be used as the recognition data 522. When integrating the LH sub-band, the HL sub-band, and the HH sub-band, for example, the data processing unit 32 obtains an average value of coefficients present at the same position in the LH sub-band, the HL sub-band and the HH sub-band , The determined average value is set as a coefficient at the same position in the integrated subband. Also, the data processing unit 32 may set the maximum value among the coefficients present at the same position in the LH subband, the HL subband and the HH subband as a coefficient at the same position in the integrated subband. Note that the method of combining the LH sub-band, the HL sub-band and the HH sub-band is not limited to this.

  In addition, when instructing the data to be transmitted to the IoT terminal 2, the gateway 3 uses CB correspondence information indicating the correspondence between code blocks among a plurality of sub-bands constituting the wavelet plane. It is also good. FIG. 35 is a diagram showing an example of the correspondence of code blocks among a plurality of subbands. The hatched portions in FIG. 35 are code blocks 5021 corresponding to each other.

  Here, code blocks 5021 corresponding to each other among a plurality of sub-bands indicate the same part of the difference image. In FIG. 35, one hatched code block 5021 LL3 of the LL3 subband, one hatched code block 5021 LH3 of the LH3 subband, one hatched code block 5021 HL3 of the HL3 subband, and one hatched HH3 subband. One code block 5021HH3 indicates the same part of the difference image. Also, the four code blocks 5021 LH2 of oblique lines in the LH 2 subband, the four code blocks 5021 HL2 of oblique lines in the HL 2 subband, and the four code blocks 5021 HH 2 of oblique lines in the HH 2 subband indicate the same part of the difference image. There is. In addition, the 16 code blocks 5021 LH 1 hatched in the LH 1 subband, the 16 code blocks 50 21 HL 1 hatched in the HL 1 subband, and the 16 code blocks 50 21 HH 1 hatched in the HH 1 subband are the same parts of the difference image. Is shown. Also, one code block 5021 of diagonal lines of each subband of decomposition level 3, four code blocks 5021 of diagonal lines of each subband of decomposition level 2, and 16 codes of diagonal lines of each subband of decomposition level 1 Block 5021 shows the same part of the difference image.

  The gateway 3 stores CB correspondence information indicating the correspondence between code blocks among a plurality of subbands as shown in FIG. When the gateway 3 determines the designated data using the CB correspondence information, when the process on the target frame image starts, the gateway 3 designates the code block 5021LL3 of the LL3 subband indicated by hatching in FIG. 35, for example. The instruction information 520 is transmitted as The IoT terminal 2 having received the instruction information 520 transmits the code block 5021LL3 as the transmission target data 503. The gateway 3 restores the non-differential CB corresponding to the received code block 5021LL 3, and performs the image recognition processing with the restored non-differential CB as the recognition data 522.

  When the gateway 3 detects a detection target, the processing on the target frame image ends. On the other hand, when the gateway 3 does not detect the detection target, the code block 5021LH3 of LH3 subband and the code block of HL3 subband which are indicated by hatching in FIG. 35 in order to process the LL2 subband. The instruction information 520 is transmitted with the code blocks 5021HH3 of the 5021 HL3 and HH3 subbands as specification data. The IoT terminal 2 that has received the instruction information 520 transmits the code blocks 5021 LH 3, 5021 HL 3, 502 1 HH 3 as the transmission target data 503. The gateway 3 restores the corresponding non-differential CBs from the code blocks 5021 LH 3, 5021 HL 3, 502 1 HH 3 received from the IoT terminal 2. Then, the gateway 3 performs inverse wavelet transform on the inverse transformation target data 522 including the non-differential CB corresponding to the code block 5021LL3 and the non-differential CB corresponding to the code block 5021LH3, 5021HL3, 5021HH3 which has already been acquired. Convert. As a result, non-differential CBs corresponding to four code blocks 5021 of the LL2 sub-band corresponding to the code blocks 5021 LL3, 5021 LH 3, 5021 HL 3, 502 1 HH 3 are generated. The gateway 3 performs an image recognition process with the generated non-differential CB as the recognition data 522.

  When the gateway 3 detects a detection target, the processing on the target frame image ends. On the other hand, when the gateway 3 does not detect the detection target, in order to process the LL1 sub-band, the four code blocks 5021 LH 2 and L H 2 sub-bands of the LH 2 sub-band are hatched in FIG. The instruction information 520 is transmitted with four code blocks 5021HH2 of four code blocks 5021HL2 and HH2 subbands as specification data. The IoT terminal 2 having received the instruction information 520 transmits the four code blocks 5021LH2, the four code blocks 5021HL2 and the four code blocks 5021HH2 as the transmission target data 503. The gateway 3 receives the non-differential CBs corresponding to the four code blocks 5021LH2 of the LH2 subband, the four code blocks 5021HL2 of the HL2 subband, and the four code blocks 5021HH2 of the HH2 subband received from the IoT terminal 2 Restore. Then, the gateway 3 performs inverse wavelet transform on the inverse transformation target data 522 including the already acquired non-differential CBs corresponding to the four code blocks of the LL2 subband and the restored non-differential CB. Thereby, non-differential CBs corresponding to 16 code blocks of LL1 subbands corresponding to 4 code blocks of each subband of decomposition level 2 are generated. The gateway 3 performs an image recognition process with the generated non-differential CB as the recognition data 522. Thereafter, the process on the target frame image is completed.

  The operation of the image processing system 4 when the gateway 3 determines the designation data using the CB correspondence information is not limited to the above example.

<Various modifications>
Hereinafter, various modifications of the image processing system 4 will be described.

First Modified Example
In the above example, although the IoT terminal 2 determines whether to transmit the differential CB or transmit the non-differential CB according to the instruction from the gateway 3, it determines itself without the instruction from the gateway 3 It is also good. In this case, the gateway 3 designates the code block to be transmitted by the IoT terminal 2 using the instruction information 520 without specifying the differential CB or the non-differential CB. The operation of the image processing system 4 according to the present modification will be described below with reference to FIGS.

  When the processing on the target frame image starts, as shown in FIG. 29, in step s11, the gateway 3 targets the LL subband of the maximum decomposition level as described above. Then, the first processing unit 321 sets a variable LV indicating the decomposition level of the target LL subband to 3. The first processing unit 321 determines the code block of the target LL subband as specification data.

  Next, in step s12, the gateway 3 generates instruction information 520 for notifying the IoT terminal 2 of the designated data determined in step s11, and transmits the instruction information 520 to the IoT terminal 2.

  The IoT terminal 2 receives the instruction information 520 in step s31, as shown in FIG. Next, in step s32, the determination unit 26 of the IoT terminal 2 determines transmission target data 503 based on the instruction information 520 and the integrated mask data 513.

  In step s32, the determination unit 26 sets the label with the smallest number as the target label, as described above. The transmission target determination unit 263 of the determination unit 26 determines the target in the target CB of the differential hierarchical data 502, the differential CB corresponding to the label area of the target label, and the target LL subband of the non-differential hierarchical data 501. Identify the non-differential CB corresponding to the label area of the label. The identified differential CB and non-differential CB are candidates to be included in the transmission target data 503.

  The transmission target determination unit 263 performs the CB comparison processing with the identified difference CB and the non-difference CB as the candidate difference CB and the candidate non-difference CB, respectively. In this CB comparison processing, the data amount of the candidate difference CB and the data amount of the candidate non-difference CB at the same coordinates are compared. FIG. 36 is a flowchart showing an example of the CB comparison process.

  As shown in FIG. 36, the transmission target determining unit 263 obtains the data amount of the candidate difference CB in step s320. This data amount is taken as a first data amount. For example, the transmission target determining unit 263 obtains the sum of the absolute values of a plurality of coefficients (difference WT coefficients) included in the candidate difference CB, and sets the obtained sum as the first data amount. Next, in step s321, the transmission target determination unit 263 obtains the data amount of the candidate non-difference CB having the same coordinates as the coordinates of the candidate difference CB for which the data amount has been obtained in step s320. This data amount is taken as a second data amount. For example, the transmission target determination unit 263 obtains a sum of absolute values of a plurality of coefficients (wavelet coefficients) included in the candidate non-difference CB, and sets the obtained sum as a second data amount.

  Next, in step s322, the transmission target determination unit 263 determines whether the first data amount is larger than the second data amount. If the first data amount is larger than the second data amount, the transmission target determination unit 263 includes the candidate non-differential CB in the transmission target data 503 in step s323. On the other hand, if the first data amount is equal to or less than the second data amount, the transmission target determining unit 263 includes the candidate difference CB in the transmission target data 503 in step s324. The transmission target determination unit 263 includes the candidate non-differential CB in the transmission target data 503 when the first data amount is equal to or greater than the second data amount, and the candidate difference CB when the first data amount is less than the second data. It may be included in the transmission target data 503.

  The transmission target determination unit 263 executes the above-described CB comparison process for each candidate difference CB to determine the transmission target data 503.

  As described above, in the present modification, the code block of the smaller data amount among the candidate difference CB and the candidate non-difference CB is included in the transmission target data 503. That is, the code block of the smaller data amount among the candidate difference CB and the candidate non-difference CB is transmitted to the gateway 3. Depending on the shooting environment of the camera 20, the amount of data of the candidate difference CB is not always smaller than the amount of data of the candidate non-difference CB. The code block is transmitted to the gateway 3 to reduce the amount of data transmitted by the IoT terminal 2. Therefore, the power consumption of the IoT terminal 2 can be further reduced. Moreover, the transmission delay of the coding data transmitted from the IoT terminal 2 can be further reduced.

  When the determination unit 26 determines the transmission target data 503, the CB specifying signal for the gateway 3 to specify whether each code block included in the transmission target data 503 is a difference CB or a non-difference CB. Generate

  When step s32 is executed, the IoT terminal 2 executes steps s33 and s34 in the same manner as described above. Then, in step s35, the IoT terminal 2 adds the bit stream 506 including the encoded data 505 generated in step s34, the coordinate data 504 generated in step s33, and the CB identification signal generated in step s32. Send to gateway 3. At this time, the IoT terminal 2 notifies the gateway 3 of the label number obtained in the labeling process, and notifies the gateway 3 of the current target label.

  Referring back to FIG. 29, after step s12, the reception unit 30a of the gateway 3 receives the bit stream 506, the coordinate data 504, and the CB identification signal from the IoT terminal 2 in step s13. Then, in step s14, the encoded data 505 included in the bit stream 506 is stored in the second memory 324, and the coordinate data 504 is stored in the coordinate table 36a of the coordinate memory 36. Further, the encoded data 505 is decompressed and decoded by the decoding device 31 to generate decoded data 521.

  Next, in step s15, based on the CB identification signal from the IoT terminal 2, the selection unit 325 of the data processing unit 32 determines whether each code block included in the decoded data 521 is a difference CB or a non-difference CB. Identify if there is. Then, the selection unit 325 outputs the difference CB included in the decoded data 521 to the restoration unit 326. On the other hand, the selection unit 325 outputs the non-differential CB included in the decoded data 521 to the first processing unit 321. In the same manner as described above, the restoration unit 326 restores the non-difference CB from the input difference CB and inputs the non-difference CB to the first processing unit 321. The first processing unit 321 stores the non-differential CB received from the restoration unit 326 in the first memory 323. The first processing unit 321 also stores the non-difference CB received from the selection unit 325 in the first memory 323.

  Next, the gateway 3 executes steps s16 to s19 in the same manner as described above. If it is determined in step s19 that the data processing unit 32 ends processing on the target frame image in the end determination of step s19, the gateway 3 sends an end notification to the IoT terminal 2 in step s20. When the detection target is detected from the target frame image, in step s21 after step s20, the gateway 3 transcodes the bit stream 529 including the encoded data 590 indicating the detection target image in which the detection target is captured. It is generated by the coder 34. Then, the gateway 3 transmits the generated bit stream 529 and the recognition result information 524 on the target frame image generated by the image recognition unit 33 from the communication unit 35 to the cloud server 5. Thus, the process on the target frame image is completed.

  If it is determined in step s19 that the data processing unit 32 continues the process on the target frame image in the end determination of step s19, the gateway 3 executes step s11 again to determine designated data. In step s11, designated data is determined based on the values currently indicated by the variables LN and LV.

  When the target decomposition level currently indicated by the variable LV is the maximum decomposition level, the first processing unit 321 designates, as designated data, a code block corresponding to the label area of the target label currently indicated by the variable LN in the LL3 subband. .

  In addition, when the target decomposition level currently indicated by the variable LV is other than the maximum decomposition level, the first processing unit 321 applies to the LH, HL, and HH subbands whose decomposition level is one higher than the target decomposition level, A code block corresponding to the label area of the target label is set as designated data.

  After executing step s11, the gateway 3 generates instruction information 520 for notifying of the designated data determined in step s11 in step s12. Then, the gateway 3 transmits the generated instruction information 520 to the IoT terminal 2.

  When receiving the instruction information 520 in step s31, the IoT terminal 2 determines transmission target data 503 on the basis of the received instruction information 520 and the integrated mask data 513 in step s32. In step s31, the determination unit 26 of the IoT terminal 2 specifies the code block (difference CB) designated by the instruction information 520 in the difference layered data 502 based on the integrated mask data 513, and makes the non-difference layering. In the data 501, the code block (non-differential CB) specified by the instruction information 520 is specified. Then, the determination unit 26 performs the above-described CB comparison processing with the identified difference CB and the non-difference CB as the candidate difference CB and the candidate non-difference CB, respectively. Thus, transmission target data 503 is determined. Thereafter, the image processing system 4 operates in the same manner.

  Also in the present modification, the end determination shown in FIG. 34 may be executed in step s19.

Second Modified Example
When the IoT terminal 2 is mounted on, for example, a moving robot or a drone, the position of the camera 20 of the IoT terminal 2 changes. When the IoT terminal 2 is held by human hands, the position of the camera 20 changes due to camera shake. When the position of the camera 20 changes, the position of the image of the entire subject in the frame image (hereinafter, may be referred to as the entire object image) changes, so the influence of the change in the position of the camera 20 appears in the frame image. . As a result, in the imaging range of the camera 20, although the area in which a moving object such as a person is present is small, unlike in FIG. 18 described above, most of the integrated sub-band mask 5126 is the second ROI mask portion 5126a ( There is a possibility of becoming a white part). As a result, most of the plurality of code blocks constituting the difference hierarchical data 502 may become the difference CB. As a result, the amount of transmission data of the IoT terminal 2 may not be reduced significantly.

  Therefore, in the present modification, the IoT terminal 2 performs a process of correcting the influence of the change of the position of the camera 20 on the frame image data 500. As a result, although the area where the moving object such as a person is present is small in the imaging range of the camera 20, most of the plurality of code blocks constituting the differential layered data 502 may become the difference CB. It can be reduced. As a result, the amount of transmission data of the IoT terminal 2 can be reduced more reliably.

  FIG. 37 is a diagram showing an example of the configuration of the IoT terminal 2 according to the present modification. As shown in FIG. 37, the IoT terminal 2 according to the present modification further includes a motion correction unit 28 in the IoT terminal 2 shown in FIG. 3 described above.

  The motion correction unit 28 performs a motion correction process on the frame image data 500 to correct the influence of the change in the position of the camera 20. The hierarchization unit 22 hierarchizes the frame image data 500a which is the frame image data 500 after the motion correction processing, and outputs the hierarchized data 501 obtained thereby. At least a part of the motion correction unit 28 may be realized by a hardware circuit that does not require software to realize its function. In addition, at least a part of the motion correction unit 28 may be a functional block realized by the computer executing a program. The other operations of the IoT terminal 2 are the same as described above.

  FIG. 38 is a diagram showing an example of the configuration of the motion correction unit 28. As shown in FIG. As shown in FIG. 38, the motion correction unit 28 includes a correction unit 280 and a frame buffer 281. The correction unit 280 reads the frame image data 500 from the image memory 21 and stores the frame image data 500 in the frame buffer 281. When the motion correction unit 280 reads the current frame image data 500 (target frame image data 500) from the image memory 21, the current frame image data 500 and the frame image data generated in the past and stored in the frame buffer 231 The motion correction process is performed on the current frame image data 500 based on 500 (sometimes referred to as past frame image data 500). The past frame image data 500 is frame image data 500 Q frames before the current frame image data 500. The value of Q may be the same as or different from the value of P described above.

  FIG. 39 is a flowchart showing an example of the motion correction process. As shown in FIG. 39, in the motion correction process, first, at step s51, the correction unit 280 performs correction within a frame image based on the current frame image data 500 and the past frame image data 500 in the frame buffer 281. The motion vector of the entire image of the subject is obtained.

  The motion vector can be determined in various ways. For example, the correction unit 280 can obtain a motion vector based on the position of the background area of the current frame image and the position of the background area of the past frame image. The background area of the frame image is a portion other than the image of the detection target (for example, a person) detected in the image recognition processing in the gateway 3 in the frame image. The background area of the frame image can be identified based on, for example, the result of the past image recognition processing in the gateway 3. Also, the IoT terminal 2 may store in advance background information for specifying a background area of a frame image.

  After obtaining the motion vector in step s51, the correction unit 280 corrects the current frame image based on the obtained motion vector in step s52. In step s52, the correction unit 280 first sets the horizontal direction component of the motion vector as the horizontal correction amount CX, which is the horizontal correction amount for the current frame image, and the vertical direction component of the obtained motion vector, as the current frame image The vertical correction amount CY, which is the vertical correction amount for. Then, the correction unit 280 moves the entire subject image in the horizontal direction by the horizontal correction amount CX and moves in the vertical direction by the vertical correction amount CY in the current frame image to correct the current frame image. Frame image data 500 indicating the current frame image after correction becomes current frame image data 500 after motion correction processing.

  FIG. 40 is a diagram showing an outline of the motion correction process. On the upper side of FIG. 40, an example of the past frame image 5010a and the current frame image 5010b is shown. At the center of FIG. 40, the horizontal correction amount CX and the vertical correction amount CY obtained from the past frame image 5010a and the current frame image 5010b shown in FIG. 40 are shown. At the lower side of FIG. 40, a current frame image 5010b corrected based on the horizontal correction amount CX and the vertical correction amount CY shown in FIG. 40 is shown. In the current frame image 5010b after correction, the position of the entire object image 5011b changes so as to cancel the influence of the change of the position of the camera 20, and the influence is corrected. Note that the position of the entire subject image 5011b in the current frame image 5010b is corrected, so that an area 5012b in which the entire subject image 5011b does not exist is generated in the corrected current frame image 5010b. Each pixel value of this area 5012 b is set to, for example, zero.

  As described above, since the IoT terminal 2 performs processing for correcting the influence of the change in the position of the camera 20 on the frame image data 500, there is a moving object such as a person in the imaging range of the camera 20. Even though the region to be processed is small, it is possible to reduce the possibility that most of the plurality of code blocks constituting the difference layered data 502 become the difference CB. As a result, the amount of transmission data of the IoT terminal 2 can be reduced more reliably.

  The IoT terminal 2 may perform the motion correction process not on the frame image data 500 but on the hierarchical data generated by the hierarchical unit 22. FIG. 41 is a diagram showing an example of the configuration of the layering unit 22 provided in the IoT terminal 2 in this case. Hereinafter, the above-described motion correction processing performed on the frame image data 500 may be referred to as a first motion correction processing, and the motion correction processing performed on the hierarchical data may be referred to as a second motion correction processing.

  As shown in FIG. 41, the layering unit 22 according to this modification further includes a motion correction unit 226 in the layering unit 22 shown in FIG. 5 described above.

  The motion correction unit 226 performs a second motion correction process on the hierarchical data 510 output from the wavelet transform unit 224. The quantization unit 225 quantizes the hierarchical data 510 a which is the hierarchical data 510 after the second motion correction processing, and outputs the quantized hierarchical data 510 a as hierarchical data 501.

  Note that at least a part of the motion correction unit 226 may be realized by a hardware circuit that does not require software to realize its function. In addition, at least a part of the motion correction unit 226 may be a functional block realized by a computer executing a program. The other operations of the IoT terminal 2 are the same as described above. Hereinafter, the current hierarchical data 510 to be processed may be referred to as current hierarchical data 510.

  FIG. 42 is a diagram showing an example of the configuration of the motion correction unit 226. As shown in FIG. As shown in FIG. 42, the motion correction unit 226 includes a correction unit 2260 and a frame buffer 2261. The correction unit 2260 stores the layered data 510 output from the wavelet transform unit 224 in the frame buffer 2261. When the correction unit 2260 receives the current hierarchical data 510 from the wavelet transform unit 224, the correction unit 2260 stores the current hierarchical data 510 and the hierarchical data 510 (past hierarchical data 510) generated in the past and stored in the frame buffer 2261. The second motion correction process is performed on the current hierarchical data 510 on the basis of (sometimes called). The past hierarchical data 510 is hierarchical data 510 Q frames before the current hierarchical data 510.

  In the second motion correction process, unlike the first motion correction process, correction is individually performed on each sub-band XYm of the current hierarchical data 510. The second correction process will be described in detail below.

  The correction unit 2260 compares, with the current hierarchical data 510, the motion vector of the portion (sometimes referred to as the entire object portion) indicating the entire subject within the corresponding subband XYm for each subband XYm of the hierarchical data 510. It is determined based on the hierarchical data 510. This motion vector can be determined in the same manner as described above. For example, when the correction unit 2260 obtains a motion vector for a certain sub-band XYm, for example, the position of a portion indicating the background in the sub-band XYm of the current hierarchical data 510 and the sub-band of the past hierarchical data 510 A motion vector can be determined based on the position of the portion indicating the background in XYm.

After obtaining the motion vector for each subband XYm, the correction unit 2260 corrects each subband XYm of the current hierarchical data based on the motion vector corresponding thereto. When correcting a certain sub-band XYm, the correction unit 2260 is a horizontal correction component of the horizontal direction component of the current hierarchical data 510 for the sub-band XYm of the motion vector determined for the sub-band XYm. The correction amount CX _XYm is used, and the vertical component of the motion vector is set as a vertical correction amount CY _XYm that is a correction amount in the vertical direction for the sub-band XYm of the current hierarchical data 510. Then, the correction unit 280 moves the entire object in the horizontal direction by the horizontal correction amount CX _XYm in the sub-band XYm of the current hierarchical data 510 and moves the vertical _object by the vertical correction amount CY _XYm in the vertical direction. Correct band XYm. The correction unit 2260 performs this correction on each sub-band XY. The current hierarchical data 510 composed of each sub-band XYm after the correction becomes the current hierarchical data 510 after the second motion correction processing. When the horizontal correction amount CX _XYm and the vertical correction amount CY _XYm for the sub-band XYm are _expressed by (CX _XYm , CY _XYm ), the correction unit 2260 can _calculate the LL3 sub-band, the LH3 sub-band, the HL3 sub-band, the HH3 sub-band, and the LH2 Corresponding to the sub-band, HL2 sub-band, HH2 sub-band, LH1 sub-band, HL1 sub-band and HH sub-band 1 (CX _LL3 , CY _LL3 ), (CX _LH3 , CY _LH3 ), (CX _HL3 , CY _HL3 ) _{, (CX HH3, CY HH3)} , (CX LH2, CY LH2), (CX HL2, CY HL2), (CX HH2, CY HH2), (CX LH1, CY LH1), (CX HL1, CY HL1) and ( Generate CX _HH1 and CY _HH1 ).

  As described above, even when the IoT terminal 2 performs processing for correcting the influence of the change in the position of the camera 20 on the hierarchical data 510, an object such as a person moving in the imaging range of the camera 20 Even though the area in which exists is small, it is possible to reduce the possibility that most of the plurality of code blocks constituting the difference layered data 502 become the difference CB. As a result, the amount of transmission data of the IoT terminal 2 can be reduced more reliably.

Third Modified Example
In each of the above examples, the gateway 3 transmits the bit stream 529 including the image data to the cloud server 5, but the bit stream 529 may not be transmitted to the cloud server 5. That is, the gateway 3 may transmit only the recognition result information 524 which is metadata to the cloud server 5. In this case, as shown in FIG. 43, the transcoder 34 becomes unnecessary, and as shown in FIG. 44, the second memory 324 and the second processing unit 322 of the data processing unit 32 become unnecessary.

  Thus, the possibility that the image data generated by the IoT terminal 2 may flow to the Internet can be reduced by the gateway 3 not transmitting the image data to the cloud server 5. Therefore, for example, the possibility that the privacy of the person appearing in the image data generated by the IoT terminal 2 is violated can be reduced.

Fourth Modified Example
The IoT terminal 2 according to each of the above examples packetized the encoded data 505 and transmitted it, but may transmit the encoded data 505 without packetizing. This eliminates the need for packet header generation. Furthermore, of the zero-length packet information, the code block inclusion information, the zero bit plane information, the coding path number information, and the code amount information of the code block included in the packet header, it is necessary for decoding the packet header at the gateway 3. It becomes unnecessary to generate the zero-length packet information and the code block content information. The bitstream generation unit 243 of the encoding device 24 generates a bitstream 506 including encoded data 505 not packetized and additional information. In this additional information, zero bit plane information, encoding pass number information and code amount information of the code block are included without being encoded, instead of the packet header.

  As described above, in the IoT terminal 2 according to the present modification, since the encoded data 505 is not packetized, packetization of data is unnecessary in the IoT terminal 2. This simplifies the processing of the IoT terminal 2. Therefore, the power consumption of the IoT terminal 2 can be reduced, and the transmission delay of data transmitted from the IoT terminal 2 can be reduced.

  In addition, in the IoT terminal 2, since the generation of the packet header is not necessary, the processing is further simplified. Therefore, the power consumption of the IoT terminal 2 can be further reduced, and the transmission delay of data transmitted from the IoT terminal 2 can be further reduced.

  Further, in the gateway 3 that processes the bit stream 506 from the IoT terminal 2, the bit stream analysis unit 310 of the decoding device 31 does not need to decode the packet header. Thus, the processing of the gateway 3 can be simplified.

Fifth Modified Example
Depending on the type of detection object detected by the image recognition unit 33 of the gateway 3, the image recognition unit 33 can detect the detection object from an image with low resolution.

  In addition, depending on the type of processing performed by the gateway 3 using the encoded data 505 received from the IoT terminal 2, the encoded data 505 indicating an image with high resolution may be required. The resolution of the image shown by may be low. For example, when the gateway 3 streaming-transmits a high resolution moving image to the cloud server 5 based on the encoded data 505, the encoded data 505 indicating an image with high resolution is required.

  Therefore, the image processing system 4 according to the present modification is a code by adjusting the bits to be transmitted to the gateway 3 in a plurality of bits constituting the binary value of the coefficient in the code block generated by the IoT terminal 2 It is possible to adjust the resolution of the image represented by the conversion data 505. The image processing system 4 according to the present modification will be described in detail below. Hereinafter, the number of bits included in the code block and constituting the binary value of the coefficient corresponding to the pixel of the image is represented by L (L is an integer of 2 or more).

  In the present modification, the encoding unit 240 of the encoding device 24 included in the IoT terminal 2 looks at L bits constituting the coefficient of the code block in the transmission target data 503 determined by the determination unit 26 from the most significant (MSB) In this case, the bit position higher than the bit position where 1 appears for the first time is the first bit position. In addition, the encoding unit 240 positions the second bit position of the bit position lower than the first bit position by M bit numbers (M is an integer of 1 or more). The encoding unit 240 sets bits from the most significant bit to the second bit position in the L bits as target bits, and sets other bits as non-target bits. The encoding unit 240 compresses and encodes the target bit of the L bits. That is, the encoding unit 240 performs bit-plane encoding and entropy encoding only on the target bit. Then, the encoding unit 240 rounds out non-target bits of L bits and does not perform compression encoding. As a result, for each coefficient in the transmission target data 503, only the compression-coded target bit of the L bits constituting it is transmitted to the gateway 3. When the difference CB is included in the transmission target data 503, only the compression-coded object bit of L bits constituting the difference WT coefficient of the difference CB is transmitted to the gateway 3. In addition, when the non-differential CB is included in the transmission target data 503, only the compression-coded target bit of the L bits constituting the quantized wavelet coefficient of the non-differential CB is the gateway 3. Sent to

  When the most significant bit of the L bit is “1”, the encoding unit 240 determines the target bit with the virtual bit position higher by one bit than the most significant bit as the first bit position. When the bit position lower than the first bit position by the number of M bits exceeds the least significant bit (LSB) of L bits, the second bit position is set to the least significant bit.

  45 and 46 are diagrams showing an example of target bits to be compression-coded. In the example of FIGS. 45 and 46, L = 11 and M = 3. The binary value of the coefficient (differential WT coefficient or quantized wavelet coefficient) shown in FIG. 45 is composed of “000 1101 0111”. In this case, the target bit is 6 bits of "000110", and the non-target bit is 5 bits of "10111". The encoding unit 240 compresses and encodes only the target bit “000110”. Further, the binary value of the coefficient shown in FIG. 46 is composed of "010 0011 0101". In this case, the target bit is 4 bits of “0100”, and the non-target bit is 7 bits of “0110101”. The encoding unit 240 compresses and encodes only the target bit “0100”.

  The value of M that determines the target bit to be compressed and encoded is determined by, for example, the gateway 3. The gateway 3 determines the value of M according to the process performed using the coded data 505. Therefore, the value of M is a value according to the process performed by the gateway 3 using the encoded data 505.

  For example, as shown in FIG. 29 described above, the gateway 3 sets the value of M to, for example, 3 as in the examples of FIGS. 45 and 46 when performing the image recognition process using the encoded data 505. . The gateway 3 sets the value of M to, for example, 5 when processing is performed using the encoded data 505 indicating an image with high resolution. The gateway 3 notifies the IoT terminal 2 of the set M value. The encoding unit 240 of the IoT terminal 2 determines the target bit based on the value of M notified from the gateway 3.

  In the gateway 3 having received the bit stream 506 from the IoT terminal 2, the decoding unit 311 of the decoding apparatus 31 decompresses and decodes the encoded data 505 from the IoT terminal 2 to restore the coefficients in the transmission target data 503. . Then, the decoding unit 311 adds bits to target bits that constitute the restored coefficient to generate an L-bit coefficient.

  Here, it is assumed that the number of target bits constituting the restored coefficient is N (an integer of 1 or more). The decoding unit 311 adds (L−N) 0's to the target bit as bits lower than the target bit constituting the restored coefficient. This yields a coefficient consisting of L bits. The gateway 3 can specify the number N of target bits based on the value of M and the zero bit plane information transmitted from the IoT terminal 2. The coefficient composed of L bits generated by the decoding unit 311 is input to the inverse quantization unit 314.

  FIGS. 47 and 48 are diagrams showing an example of a coefficient composed of L bits, which is generated by the decoding unit 311. FIG. FIG. 47 shows an example in which the IoT terminal 2 transmits the target bit "000110" shown in FIG. 45 described above. FIG. 48 shows an example in which the IoT terminal 2 transmits the target bit "0100" shown in FIG. 46 described above.

  As shown in FIG. 47, when the decoding unit 311 restores the coefficient made up of the target bits “000110” of 6 bits, five 0s are set as target bits “000110” as lower bits than the target bit “000110”. Add to As a result, a coefficient consisting of 11 bits (L bits) of "000 1100 0000" is obtained.

  Further, as shown in FIG. 48, when the decoding unit 311 restores the coefficient made up of the 4-bit target bit “0100”, seven 0s are set as target bits “0” as bits lower than the target bit “0100”. Add to "0100". As a result, a coefficient consisting of 11 bits (L bits) of "010 0000 0000" is obtained.

  In the example of FIGS. 45 to 48, the information indicated by the 11-bit coefficients generated by the decoding unit 311 is generated by the IoT terminal 2 so that FIGS. 45 and 46 and FIGS. 47 and 48 can be compared and understood, respectively. The information of lower bits may be lost compared to the information indicated by the 11-bit coefficient. Therefore, the resolution of the image indicated by the decoded data 521 output from the inverse quantization unit 314 may be low.

  Thus, in the present modification, processing in gateway 3 from the highest order of L bits constituting coefficients (quantized difference WT coefficients or quantized wavelet coefficients) corresponding to pixels in transmission target data 503 Only the bits up to the second bit position determined by the value of M according to are compressed and transmitted to the gateway 3. Therefore, the amount of data that the IoT terminal 2 transmits to the gateway 3 can be adjusted according to the processing in the gateway 3. Therefore, the power consumption of the IoT terminal 2 can be reduced, and the transmission delay of data transmitted from the IoT terminal 2 can be reduced.

  Also, by adjusting the value of M that determines the target bit, the resolution of the image used by the gateway 3 can be easily adjusted. That is, by increasing the value of M, the resolution of the image used by the gateway 3 can be increased, and by decreasing the value of M, the resolution of the image used by the gateway 3 can be decreased. it can.

  In addition, when the value of M is not notified from the gateway 3, the IoT terminal 2 may compress and encode all of the L bits constituting the coefficient and transmit it to the gateway 3.

  The gateway 3 also wants to perform processing using an image with high resolution when there is an upper bit received as an object bit in the past and stored in the first memory 323 in L bits constituting a coefficient. At this time, only the necessary bits lower than the upper bits stored in the first memory 323 may be transmitted to the IoT terminal 2. In this case, the gateway 3 can generate an L-bit coefficient with less information loss by adding the lower bits received later to the upper bits in the first memory 323. Therefore, the gateway 3 can perform processing using an image with high resolution. In addition, since the IoT terminal 2 does not transmit the already transmitted data again, the processing of the IoT terminal 2 is simplified.

  For example, it is assumed that all the L bits constituting the coefficients generated by the IoT terminal 2 are required in order for the gateway 3 to perform processing using an image with high resolution. Also, it is assumed that the gateway 3 has already received the upper 6 bits "000110" shown in FIG. 44, for example, as the target bits among the L bits constituting the coefficient, and has stored them in the first memory 323. In such a case, the gateway 3 instructs the IoT terminal 2 to transmit only the necessary remaining lower 5 bits, ie, "10111". The IoT terminal 2 receiving this instruction compresses / decodes only the lower 5 bits “10111” and transmits it to the gateway 3. The gateway 3 adds the newly received lower “10111” to the upper “000110” in the first memory 323 to generate a coefficient consisting of 11 bits “000 1101 0111”.

  Further, in order for the gateway 3 to perform processing using an image with a high resolution, it is assumed that the upper 9 bits of L bits constituting the coefficient generated by the IoT terminal 2 are required. Also, it is assumed that the gateway 3 has already received the upper 4 bits “0100” shown in FIG. 45, for example, as the target bits among the L bits constituting the quantized wavelet coefficient and stored in the first memory 323. In such a case, the gateway 3 instructs the IoT terminal 2 to transmit only the necessary remaining 5 bits “01101” (5 bits from the bit position next to the second bit position). The IoT terminal 2 receiving this instruction compresses / decodes only 5 bits “01101” and transmits it to the gateway 3. The gateway 3 adds the newly received 5-bit “01101” to the low-order “0100” in the first memory 323 to the low order, and adds two 0's to the 11-bit “010 0011”. Generate a coefficient consisting of 0100 ′ ′.

<Sixth Modified Example>
In this modification, the IoT terminal 2 can multilayer the bit stream 506. FIG. 49 is a diagram showing an example of a configuration of the encoding device 24 of the IoT terminal 2 according to the present modification. FIG. 50 is a diagram showing an example of a configuration of the decryption device 31 of the gateway 3 according to the present modification.

  As shown in FIG. 49, the encoding device 24 according to the present modification further includes a layer division processing unit 250 in the encoding device 24 shown in FIG. 20 described above. The layer division processing unit 250 can multi-layer the bit stream 506 output from the bit stream generation unit 243. In the bit stream 506, data is arranged in code block units. When the bit stream 506 is multilayered, the layer division processing unit 250 divides the encoded data 505 included in the bit stream 506 into a plurality of layers, and includes data of at least one layer of the plurality of layers. The bitstream 506a is output. On the other hand, the layer division processing unit 250 can also output the bit stream 506 as it is without multi-layering. In this case, the layer division processing unit 250 outputs the bit stream 506 as it is as a bit stream 506a. The bit stream 506 a generated by the layer division processing unit 250 is transmitted from the transmission unit 25 a to the gateway 3.

  Note that at least a part of the layer division processing unit 250 may be realized by a hardware circuit that does not require software to realize its function. Further, at least a part of the layer division processing unit 250 may be a functional block realized by execution of a program by a computer.

  As shown in FIG. 50, the decoding device 31 according to the present modification further includes a layer combining processing unit 315 in the decoding device 31 shown in FIG. 23 described above. When the gateway 3 receives a bit stream 506a including data of a plurality of layers from the IoT terminal 2, the layer composition processing unit 315 generates data of a plurality of layers of the same subband included in the bit stream 506a. To generate and output a non-multilayer bit stream 506b, that is, a bit stream 506b in which data is arranged in code block units. On the other hand, when the gateway 3 receives a bitstream 506a (bitstream 506) that is not multi-layered from the IoT terminal 2, the layer composition processing unit 315 receives a bitstream 506a including data of only one layer. In this case, the bit stream 506a is output as it is as a bit stream 506b.

  In the same manner as described above, the bit stream analysis unit 310 analyzes the bit stream 506b and extracts the encoded data 505a and the additional information from the bit stream 506b. The encoded data 505 a is hierarchically encoded data indicating at least a part of a frame image or at least a part of a difference image, as with the encoded data 505. The bit stream analysis unit 310 outputs the extracted encoded data 505 a to the decoding unit 311 and the data processing unit 32. Each of the decoding unit 311 and the data processing unit 32 processes the encoded data 505 a in the same manner as the processing for the encoded data 505. When the gateway 3 receives a bitstream 506 a (bit stream 506) that is not multi-layered from the IoT terminal 2, the encoded data 505 a matches the encoded data 505 generated by the IoT terminal 2. .

<Detailed Description of Layer Division Processing Unit>
FIG. 51 is a diagram showing an example of the configuration of the layer division processing unit 250. As shown in FIG. As shown in FIG. 51, the layer division processing unit 250 includes a memory 251, a layer division control unit 252, a multiplexing unit 253, and a priority table 254.

  The priority table 254 stores the priorities set for each of the plurality of sub-bands constituting the wavelet plane. The priority of each subband is set according to the decomposition level of that subband. In the layer division processing unit 250, the bit stream 506 is multilayered based on the priorities in the priority table 255.

  The layer division control unit 252 stores the bit stream 506 generated by the bit stream generation unit 243 in the memory 251. When the bit stream 506 is multilayered, the layer division control unit 252 reads the encoded data 505 included in the bit stream 506 from the memory 251, and divides the read encoded data 505 into a plurality of layers. Then, the layer division control unit 252 outputs data of at least one layer of the plurality of layers to the multiplexing unit 253. The multiplexing unit 523 multiplexes the data output from the layer division control unit 252, and generates and outputs a bit stream 506a including data of at least one layer.

  On the other hand, when the bit stream 506 is not multilayered, the layer division control unit 252 reads the bit stream 506 from the memory 251 and outputs the bit stream 506 to the multiplexing unit 253 as it is. The multiplexing unit 253 outputs the input bit stream 506 as it is as a bit stream 506a. Whether or not the IoT terminal 2 multilayers the bit stream 506 is specified by the instruction information 520 from the gateway 3.

  FIG. 52 is a diagram showing an example of the priority set to each subband. In the example of FIG. 52, priority 4 is set for the LL3 subband and priority 3 is set for the LH3 subband and the HL3 subband. Also, priority 2 is set for the HH3 subband, the LH2 subband, and the HL2 subband, and priority 1 is set for the HH2 subband, the LH1 subband, and the HL1 subband. Then, priority 0 is set for the HH1 subband. The value of priority of each subband is not limited to the example of FIG.

  The layer division control unit 252 performs bit shift processing on each code block included in the encoded data 505 when the bit stream 506 is multi-layered. The bit shift processing will be described in detail below. In this modification, the code block to be described is referred to as a target code block.

  In the bit shift process for the target code block, the layer division processing unit 252 first acquires, from the priority table 254, the priority set to the sub-band to which the target code block belongs. The layer division control unit 252 shifts, for each coefficient of the target code block, bit data of L bits constituting the coefficient by the same number of bits as the acquired priority. Thereby, bit shift processing is performed on the target code block.

  In the case where the target code block belongs to, for example, the LL3 sub-band, the layer division control unit 252 bit-shifts L bit data constituting the coefficient in the same direction by 4 bits for each coefficient of the target code block. In addition, when the target code block belongs to, for example, the HH3 sub-band, the layer division control unit 252 bit-shifts L-bit data constituting the coefficient in the same direction by 2 bits for each coefficient of the target code block. Do. Note that since the priority set to the HH1 subband is 0, even if the bit shift process is performed on the target code block belonging to the HH1 subband, each coefficient of the target code block is actually a bit It is not shifted. Hereinafter, a code block subjected to bit shift processing may be referred to as a shifted code block.

  When the layer division control unit 252 performs bit shift processing on each code block included in the encoded data 505, the layer division control unit 252 performs layer division processing for dividing each shift-processed code block into a plurality of layers.

  FIG. 53 is a diagram for describing an example of layer division processing. FIG. 53 shows L-bit (11 bits in the example of FIG. 53) data 5700 constituting the coefficients (bit-shifted coefficients) of the shifted code block. The numbers 0 to 10 shown in the L-bit data 5700 indicate the bit position of each bit in the data 570. The number 0 indicates the LSB, and the number 10 indicates the MSB.

  As shown in FIG. 53, L-bit data 5700 constituting the coefficients of code blocks belonging to the LL3 subband are bit shifted by 4 bits. The L-bit data 5700 constituting the coefficients of code blocks belonging to the LH3 subband and the HL3 subband are bit shifted by 3 bits. The L-bit data 5700 constituting the coefficients of code blocks belonging to the HH3 subband, the LH2 subband, and the HL2 subband are bit shifted by 2 bits. The L-bit data 5700 constituting the coefficients of the code blocks belonging to the HH2 subband, the LH1 subband, and the HL1 subband are bit shifted by one bit. Then, L-bit data 5700 constituting the coefficient of the code block belonging to the HH1 subband is not bit shifted.

  In this modification, as shown on the lower side of FIG. 53, with respect to the shifted code block, bit shift of LL3 subband from the least significant bit position of L bit data 5700 constituting the coefficient of HH1 subband. The numbers 0 to 14 are respectively assigned to the most significant bit positions of the L-bit data 5700 constituting the coefficients.

  The layer division control unit 252 sets the bit positions 12 to 14 of the L bit data 5700 constituting the coefficient of the code block after bit shift processing to be layer 0, and the bit positions 9 to 11 Let the bits up to the number be layer 1. Further, the layer division control unit 252 sets the bit positions 6 to 8 of the L bit data 5700 constituting the coefficient of the code block after bit shift processing as layer 2 and the bit positions 3 to Let the bits up to the fifth be layer 3. Then, the layer division control unit 252 sets the bit from the 0th to the 2nd bit position to the layer 4 in the L-bit data 5700 constituting the coefficient of the code block after the bit shift processing.

  As described above, when the encoded data 505 is divided into a plurality of layers, the layer division control unit 252 outputs the data of the layer to be transmitted to the gateway 3 among the plurality of layers to the multiplexing unit 253. The layer that the IoT terminal 2 transmits to the gateway 3 is designated by the instruction information 520 from the gateway 3. The gateway 3 can freely instruct the IoT terminal 2 as to which layer data should be transmitted among the plurality of layers. The multiplexing unit 253 multiplexes the data from the layer division control unit 252 to generate a bit stream 506 a including data of a layer to be transmitted to the gateway 3. Note that the method of dividing the L-bit data 5700 into a plurality of layers is not limited to the example shown in FIG.

<Detailed Description of Layer Composition Processing Unit>
FIG. 54 is a diagram showing an example of the configuration of the layer combining processing unit 315. As shown in FIG. As shown in FIG. 54, the layer composition processing unit 315 includes a memory 316 and a layer composition control unit 317.

  The layer composition control unit 317 stores the bit stream 506 a from the IoT terminal 2 in the memory 316. Further, when the bit stream 506a received by the gateway 3 is not multilayered, the layer combining control unit 317 outputs the bit stream 506a as it is as the bit stream 506b. The layer combining control unit 317 is a bit when the bit stream 506a received by the gateway 3 is multi-layered and the bit stream 506a includes only data of one layer, the bit The stream 506a is output as it is as a bit stream 506b.

  On the other hand, in the case where the bit stream 506a received by the gateway 3 is multi-layered and the bit stream 506a includes data of a plurality of layers, the layer composition control unit 317 is the same. Synthesize multiple layers of data for subbands, and generate and output a non-multilayer bit stream 506 b (a bit stream 506 b in which data is arranged in code block units like bit stream 506) .

<Operation Example of Image Processing System According to this Modification>
Next, an operation example of the entire image processing system 4 according to the present modification when the bit stream 506 is multilayered will be described. Here, unless otherwise specified, speaking of a code block means a code block of a quantized difference wavelet plane.

  When processing on the target frame image starts, the gateway 3 transmits instruction information 520, for example, as data of layer 0 of each code block of the LL3 subband of the quantized difference wavelet plane as specification data. The IoT terminal 2 having received the instruction information 520 generates coded data 504 by using each code block of the LL3 sub-band as the transmission target data 503. Then, in the IoT terminal 2, the layer division processing unit 250 multi-layers the bit stream 506 including the encoded data 505 to generate layer 0 data of each code block of the LL3 sub-band. Then, the layer division processing unit 250 generates a bit stream 506 a including data of layer 0 of each code block of the LL 3 subband and transmits the bit stream 506 a to the gateway 3.

  In the gateway 3, the layer combining processing unit 315 outputs the bit stream 506a received from the IoT terminal 2 as it is as the bit stream 506b. After that, the gateway 3 applies data to each coefficient data so that the data (layer 0) of each coefficient of the LL3 sub-band contained in the bit stream 506b becomes L-bit (L = 11) data. Eight zeros are added as lower bits (see FIGS. 47 and 48 above). As a result, LL3 subbands in which the coefficients of each code block are configured by L-bit data are obtained. The gateway 3 restores the corresponding non-differential CB from each code block of the obtained LL3 sub-band, and uses the restored non-differential CB as recognition data 522. The gateway 3 performs image recognition processing on the recognition data 522.

  When the gateway 3 detects a detection target, the processing on the target frame image ends. On the other hand, when the gateway 3 does not detect the detection target, data of layers 0 and 1 of each code block of the LH3 subband, the HL3 subband, and the HH3 subband is used to process the LL2 subband. And the layer 1 data of each code block of the LL3 subband as designation data, and transmits the instruction information 520. The IoT terminal 2 having received the instruction information 520 generates the coded data 505 by using the code blocks of the LL3 subband, the LH3 subband, the HL3 subband, and the HH3 subband as a transmission target code block. Then, in the IoT terminal 2, the layer division processing unit 250 multilayers the bit stream 506 including the encoded data 505, and sets layers 0 and 1 of each code block of the LH3 subband, the HL3 subband, and the HH3 subband. Data and data of Layer 1 of each code block of LL3 subband are generated. Then, the layer division processing unit 250 is a bit stream including data of layers 0 and 1 of each code block of LH3 subband, HL3 subband and HH3 subband, and data of layer 1 of each code block of LL3 subband. It generates 506 a and sends it to the gateway 3. When transmitting the multi-layered bit stream 506a, the IoT terminal 2 sequentially transmits data of the upper layer in order. Here, the IoT terminal 2 transmits layer 0 data of LH3 subband, HL3 subband and HH3 subband, and then transmits layer 1 data of LL3 subband, LH3 subband, HL3 subband and HH3 subband. Send.

  In the gateway 3, the layer composition control unit 317 of the layer composition processing unit 315 stores the bit stream 506 a received from the IoT terminal 2 in the memory 316. Then, the layer combining control unit 317 reads out data of a plurality of layers of the same sub-band from the memory 316 and combines them to generate a single-layer bit stream 506b.

  Specifically, the layer composition control unit 317 reads the data of layers 0 and 1 of the LH3 sub-band from the memory 316. Then, the layer combining control unit 317 combines the read data of layers 0 and 1 of the LH3 subband, and generates data transmitted by the IoT terminal 2 in the LH3 subband in which the data are arranged in code block units. . Hereinafter, this data is referred to as single-layered LH3 subband data.

  Similarly, the layer composition control unit 317 reads, from the memory 316, data of layers 0 and 1 of the HL3 sub-band. Then, the layer combining control unit 317 combines the read data of layers 0 and 1 of the HL3 subband, and generates data transmitted by the IoT terminal 2 in the HL3 subband, in which data are arranged in code block units. . Hereinafter, this data is referred to as single-layered HL3 sub-band data.

  Similarly, the layer composition control unit 317 reads the data of layers 0 and 1 of the HH3 sub-band from the memory 316. Then, the layer combining control unit 317 combines the read data of layers 0 and 1 of the HH3 sub-band, and generates data transmitted by the IoT terminal 2 in the HH3 sub-band in which data is arranged in code block units. . Hereinafter, this data is referred to as single-layered HH3 sub-band data.

  Then, the layer combining control unit 317 performs each of the single-layered LH3 sub-band data, the single-layered HL3 sub-band data, the single-layered HH3 sub-band data, and the LL3 sub-band. A bitstream 506b is generated that includes Layer 1 data of the code block. In this bit stream 506b, data is arranged in code block units.

  As described above, since the gateway 3 generates the single-layered bit stream 506b, even if the multi-layered bit stream 506a is transmitted from the IoT terminal 2, the multi-layering is performed. Even in the case where no bitstream 506a is transmitted, the decoding unit 311 having the same configuration can be used. Thus, the configuration of the gateway 3 can be simplified.

  Next, the gateway 3 needs the data of each coefficient so that the data of each coefficient of the LH3 subband, the HL3 subband and the HH3 subband included in the bit stream 506b becomes L-bit data. The number of 0s is added as the lower bits by the number (see FIGS. 47 and 48 described above). As a result, LH3 subbands, HL3 subbands and HH3 subbands are obtained in which the coefficients of each code block are composed of L-bit data. Also, the gateway 3 combines the already acquired data of layer 0 of the LL3 subband and the data of layer 1 of the LL3 subband LL3 included in the bit stream 506b, and the respective coefficients of each code block Generates an LL3 subband composed of 6-bit data (3 bits of layer 0 + 3 bits of layer 1). Then, the gateway 3 sets only five 0s as low-order bits to the data of each coefficient so that the data of each coefficient included in the generated LL3 sub-band becomes L-bit (L = 11) data. to add. As a result, LL3 subbands in which the coefficients of each code block are configured by L-bit data are obtained. The gateway 3 restores the corresponding non-differential CB from each code block of the obtained LL3 subband, LH3 subband, HL3 subband and HH3 subband, and inverse-transformed data 522 composed of the recovered non-differential CB. Inverse wavelet transform. This generates a non-differential CB corresponding to each code block of the LL2 subband. The gateway 3 performs an image recognition process with the generated non-differential CB as the recognition data 522.

  Here, when the inverse transformation target data 522 is subjected to inverse wavelet transformation, distortion occurs in the image represented by the data generated by the inverse transformation, and the quality of the image may be degraded. In this modification, when generating LL2 sub-band data using inverse wavelet transform, not only layer 0 data but also layer 1 data is used, so only layer 0 data is used. The quality of the image represented by the data generated by the inverse wavelet transform can be improved as compared to the case of Note that inverse wavelet transform may be performed using only layer 0 data. Also, data of layers 0 to 3 may be used to perform inverse wavelet transform, and data of layers 0 to 4 may be used to perform inverse wavelet transform.

  When the detection target is detected as a result of the gateway 3 performing the image recognition processing, the processing on the target frame image ends. On the other hand, when the gateway 3 does not detect the detection target, in order to process the LL1 subband, data of layers 0 and 1 of each code block of the LH2 subband and the HL2 subband, and the HH2 sub The instruction information 520 is transmitted with the data of layer 1 of each code block of the band (the data of layer 0 does not exist in the HH2 subband) as the designated data. The IoT terminal 2 having received the instruction information 520 generates the encoded data 505 by using each code block of the LH2 subband, the HL2 subband and the HH2 subband as a transmission target code block. Then, in the IoT terminal 2, the layer division processing unit 250 multilayers the bit stream 506 including the encoded data 505 to obtain data of layers 0 and 1 of each code block of the LH 2 subband and the HL 2 subband, and HH 2 Generate layer 1 data of each code block of the subband. Then, the layer division processing unit 250 generates a bit stream 506a including data of layers 0 and 1 of each code block of LH2 subband and HL2 subband and data of layer 1 of each code block of HH2 subband. To the gateway 3. At this time, the IoT terminal 2 transmits layer 0 data of the LH 2 subband and the HL 2 subband, and then transmits layer 1 data of the LH 2 subband, the HL 2 subband and the HH 2 subband.

  In the gateway 3, the layer composition control unit 317 stores the bit stream 506 a received from the IoT terminal 2 in the memory 316. Then, the layer combining control unit 317 reads out data of a plurality of layers of the same sub-band from the memory 316 and combines them to generate a single-layer bit stream 506b.

  Specifically, the layer composition control unit 317 reads the data of layers 0 and 1 of the LH2 sub-band from the memory 316. Then, the layer combining control unit 317 combines the read data of layers 0 and 1 of the LH2 subband, and generates data transmitted by the IoT terminal 2 in the LH2 subband in which the data are arranged in code block units. . Hereinafter, this data is referred to as single-layered LH2 subband data.

  Similarly, the layer composition control unit 317 reads, from the memory 316, data of layers 0 and 1 of the HL2 subband. Then, the layer combining control unit 317 combines the read data of layers 0 and 1 of the HL2 subband, and generates data transmitted by the IoT terminal 2 in the HL2 subband, in which data are arranged in code block units. . Hereinafter, this data is referred to as single-layered HL2 subband data.

  Then, the layer combining control unit 317 is a bitstream including data of LH2 subbands in a single layer, data of HL2 subbands in a single layer, and data of layer 1 of each code block of HH2 subbands. Generate 506b. In this bit stream 506b, data is arranged in code block units.

  Next, the gateway 3 adds the necessary number of data to each coefficient data so that the data of each coefficient of the subbands LH2, HL2, HH2 included in the bit stream 506b becomes L-bit data. Are added as lower bits (see FIGS. 47 and 48 described above). As a result, LH2, HL2, and HH2 subbands in which the coefficients of each code block are composed of L-bit data are obtained. Then, the gateway 3 restores the corresponding non-differential CB from each code block of the obtained LH2 subband, HL2 subband and HH2 subband.

  Next, the gateway 3 performs inverse wavelet transformation on the inverse transformation target data 522 including the restored non-differential CB and the non-differential CB corresponding to each code block of the LL2 subband that has already been acquired. From this, a non-differential CB corresponding to each code block of the LL1 subband is generated. The gateway 3 performs image recognition processing with the non-differential CB obtained by the inverse wavelet transform as the recognition data 522. Thereafter, the process on the target frame image is completed.

  As described above, in the image processing system 4 according to the present modification, since the IoT terminal 2 can transmit data in units of layers, the power consumption of the IoT terminal 2 can be reduced.

  The operation of the image processing system 4 when the bit stream 506 is multilayered is not limited to the above example. For example, the gateway 3 may determine designated data using the above-described CB correspondence information.

<Other Modifications>
In each of the above examples, the information processing system 1 is used as an IoT system, but may be used as another system.

  Further, in each of the above examples, data is hierarchized based on JPEG 2000. However, data may be hierarchized based on another standard for hierarchizing data in the same manner as sub-band division in JPEG 2000.

  As described above, the information processing system 1, the image processing system 4, and the image processing apparatuses 2 and 3 have been described in detail, but the above description is an exemplification in all aspects, and the present invention is limited thereto It is not a thing. Further, the various modifications described above can be combined and applied as long as no contradiction arises. And, it is understood that innumerable modifications not illustrated may be assumed without departing from the scope of the present invention.

1 Information processing system 2 Image processing device (IoT terminal)
3 Image processing device (gateway)
4 image processing system 22 hierarchization unit 23 difference generation unit 25a, 30b transmission unit 26 determination unit 28, 226 motion correction unit 30a reception unit 33 image recognition unit 240 encoding unit 311 decoding unit 321 first processing unit 326 restoration unit

Claims (24)

A first generation unit configured to generate hierarchical data hierarchically showing a frame image;
A second generation unit that generates difference hierarchical data indicating a difference between the hierarchical data of two frame images;
A first determination unit that determines transmission target data from the differentially-layered data based on first data of the differentially-layered data whose absolute value is greater than or equal to a threshold value or greater than the threshold value; ,
An encoding unit that compresses and encodes the transmission target data to generate encoded data;
And a first transmission unit configured to transmit the encoded data. The image processing apparatus according to claim 1, wherein
The image processing device, wherein the second generation unit generates the difference hierarchical data indicating a difference between the hierarchical data after quantization of the two frame images. An image processing apparatus according to any one of claims 1 and 2.
The first determination unit receives the encoded data transmitted from the transmission unit, and based on the first data and instruction information from a device that performs processing based on the received encoded data. An image processing apparatus that determines transmission target data. An image processing apparatus according to any one of claims 1 to 3, wherein
The image processing apparatus, wherein the first determination unit generates mask data for specifying the first data based on the difference hierarchical data, and determines the transmission target data based on the mask data. The image processing apparatus according to claim 4,
The hierarchical data is wavelet-transformed data, and
The layered data includes a plurality of subbands for each of a plurality of decomposition levels,
The mask data includes sub-band mask data for specifying second data whose absolute value is greater than or equal to the threshold value in the sub-bands,
The image processing device, wherein the first determination unit generates, for each decomposition level, the subband mask data common to a plurality of subbands of the decomposition level. An image processing apparatus according to any one of claims 1 to 5, wherein
The image processing apparatus further includes a correction unit that performs correction processing on image data indicating the frame image to correct an influence of a change in a position of a camera that captures the frame image.
The image processing apparatus, wherein the first generation unit generates the hierarchical data based on the image data subjected to the correction processing. An image processing apparatus according to any one of claims 1 to 5, wherein
The hierarchical data further includes a correction unit that performs correction processing to correct an influence of a change in a position of a camera that captures the frame image.
The image processing apparatus, wherein the second generation unit generates the difference hierarchical data indicating the difference between the hierarchical data of the two frame images on which the correction process has been performed. The image processing apparatus according to any one of claims 1 to 7, wherein
The image processing apparatus, wherein the first determination unit determines the transmission target data from the difference layered data and the layered data. The image processing apparatus according to claim 8,
The first determination unit is
Based on the first data, first candidate data to be candidates to be included in the transmission target data is determined from the differential layering data,
If the data amount of the first candidate data is smaller than the data amount of second candidate data corresponding to the first candidate data in the hierarchical data, the first candidate data is included in the transmission target data,
The image processing apparatus, wherein the second candidate data is included in the transmission target data when the data amount of the second candidate data is smaller than the data amount of the first candidate data. The image processing apparatus according to any one of claims 1 to 9, wherein
The image processing apparatus, wherein the transmission unit transmits the encoded data that is not packetized. An image processing apparatus according to any one of claims 1 to 10, wherein
The encoded data transmitted from the transmission unit is received by a device that performs processing based on the encoded data,
A bit position higher by 1 bit than a bit position where 1 appears for the first time when a plurality of bits constituting a value corresponding to a pixel included in the transmission target data is viewed from the highest position is a first bit position When the bit position lower than the first bit position by the number of bits according to the predetermined processing is set as the second bit position, the encoding unit is configured to select the second most significant bit from the plurality of bits. An image processing apparatus, which compresses and encodes a bit up to a position and does not compress and code other bits of the plurality of bits. An image processing apparatus as a second apparatus for communicating with a first apparatus as the image processing apparatus according to any one of claims 1 to 11,
A receiver configured to receive the encoded data transmitted from the first device;
And a processing unit that performs processing based on the encoded data. The image processing apparatus according to claim 12, wherein
The processing unit is
A decoding unit that performs expansion decoding on the encoded data to restore difference data that is data included in the difference layered data;
Of the two frame images, based on the difference data and first non-difference data that is data corresponding to the difference data, which is included in the layered data for one of the two frame images. An image processing apparatus, comprising: a restoration unit that restores second non-difference data corresponding to the difference data, which is included in the hierarchical data of the other of the two. The image processing apparatus according to claim 13, wherein
The image processing apparatus, wherein the processing unit further includes an image recognition unit that performs an image recognition process on an image based on the second non-difference data. An image processing apparatus as a second apparatus for communicating with a first apparatus as the image processing apparatus according to any one of claims 8 and 9.
A receiver configured to receive the encoded data transmitted from the first device;
A processing unit that performs processing based on the encoded data;
The processing unit is
A decoding unit that performs expansion decoding on the encoded data;
Data corresponding to the difference data included in the hierarchized data for one of the two frame images, the difference data being data included in the difference hierarchized data, restored by the expansion decoding The second non-differential data corresponding to the difference data included in the layered data of the other of the two frame images based on the first non-difference data being And
The image processing apparatus, wherein the processing unit performs processing based on third non-difference data which is data included in the layered data and which is restored by the decompression decoding and the second non-difference data. An image processing apparatus according to claim 15.
The image processing apparatus, wherein the processing unit performs an image recognition process on an image based on the second and third non-difference data. An image processing apparatus, which is a second apparatus that communicates with a first apparatus that is the image processing apparatus according to claim 3.
A receiver configured to receive the encoded data transmitted from the first device;
A processing unit that performs processing based on the encoded data;
A third generation unit configured to generate instruction information for instructing the first device to transmit data to be transmitted by the first device, based on a result of processing in the processing unit;
And a second transmission unit that transmits the instruction information to the first device.
The image processing apparatus, wherein the first device determines the transmission target data based on the instruction information. An image processing apparatus as a second apparatus that communicates with a first apparatus as the image processing apparatus according to claim 11;
A receiver configured to receive the encoded data transmitted from the first device;
A processing unit that performs processing based on the encoded data;
An image processing apparatus comprising: a second determination unit configured to determine the number of bits for determining the second bit position according to processing in the processing unit; An image processing apparatus according to any one of claims 12 to 18, wherein
The image processing apparatus, wherein the second device transmits the result of processing in the processing unit. A first apparatus, which is the image processing apparatus according to any one of claims 1 to 11.
An image processing system comprising: a second apparatus which is the image processing apparatus according to any one of claims 12 to 19. An image processing system according to claim 20,
An information processing system, comprising: a third device that receives a result of processing in the processing unit from the second device of the image processing system. The information processing system according to claim 21, wherein
The first and second devices are connected by a local network,
An information processing system, wherein the second and third devices are connected via the Internet. The information processing system according to claim 22, wherein
An information processing system, wherein a plurality of the first devices are connected to the second device by the local network. An image processing method in an image processing apparatus
Generating hierarchical data representing a frame image;
Generating difference hierarchical data indicating a difference between the hierarchical data of two frame images;
Determining transmission target data from the differentially-layered data based on data of the differentially-layered data whose absolute value is greater than or equal to a threshold value or greater than the threshold value;
Compression coding the transmission target data to generate coded data;
Transmitting the encoded data.