JP2006092450A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor and an image processing method capable of obtaining a highly minute composite image while suppressing costs. <P>SOLUTION: In a highly minute image creating part 200, an image storage part 210 successively stores attention images (R0, R1, R2, ..., Rn) included in photography frames (F0, F1, F2, ..., Fn) sent consecutively from a camera 10 in an incorporated buffer. A part 220 for calculating the number of composition calculates the number of composition on the basis of the size of an attention area. An image change amount detecting part 240 receives attention images of the number of composition from the image storage part 210, obtains a change amount of the attention image, and by using the change amount, composes attention images. As a result, a highly minute image (moving image) is generated. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and an image processing method.

  Recently, the price of network cameras that can be connected to the Internet or an intranet and PC cameras connected via a PC (personal computer) has been reduced, and surveillance systems using these cameras are widely used even at the individual level. It has become. However, there is generally a correlation between price and function, and when a low-priced camera is used in a surveillance system, a sufficient level of image quality cannot be obtained for surveillance, and moving objects appear in the surveillance image. Even if it can be confirmed, it is difficult to determine whether it is a person or a thing, or if it is a person. In other words, the visibility of the monitoring target may be lowered.

  For example, if the camera is provided with an optical zoom function and a predetermined target is zoomed up, the visibility of the monitoring target can be improved. However, adding a high-magnification optical zoom function to the camera increases the cost.

  Patent Document 1 below discloses an electronic zoom mechanism and pixel shifting technology that can be mounted on a camera instead of an optical zoom or in combination with a low-magnification optical zoom.

Japanese Patent Laid-Open No. 2002-238058

  However, the electronic zoom simply enlarges the image and does not increase the resolution. Therefore, the visibility is not always improved. Also, the resolution can be improved by using the pixel shifting technique. However, it is necessary to provide the camera with a mechanism for moving the image sensor minutely or a special mechanism for changing the thickness of the optical transmission glass. As a result, it becomes difficult to control the price of the camera.

  The present invention has been made in view of such problems, and an object thereof is to provide a new and improved image processing apparatus and image processing method capable of obtaining a high-definition composite image while suppressing cost. There is.

  In order to solve the above problems, according to a first aspect of the present invention, an image storage unit that stores a plurality of images, a composite number calculation unit that calculates a composite number in accordance with the size of the image, and a composite number An image change amount detecting unit for acquiring a change amount between the combined images and an image combining unit for combining the combined images based on the change amount. An image processing apparatus characterized by the above is provided. According to this configuration, the number of combined images increases or decreases according to the image size. Therefore, if the image size is large, the number of images to be combined is reduced to reduce image composition processing. Conversely, if the image size is small, the number of images to be combined is increased to generate a finer composite image. It becomes possible to do.

  In order to solve the above problems, according to a second aspect of the present invention, an image storage unit that stores a plurality of images, a composite number calculation unit that calculates the composite number according to the size of the image, and a composite number The change amount is larger than a predetermined value among a plurality of images acquired by the image change amount detection unit and the image change amount detection unit that acquires the image of the image from the image storage unit and detects a change amount between the acquired images. There is provided an image processing apparatus including a composite image determination unit that uses an image as a composite image, and an image composition unit that synthesizes each composite image based on a change amount. According to this configuration, the number of combined images increases or decreases according to the image size. Moreover, an image with a small amount of change is not used for the composition process. This is because it is difficult to obtain the effect of synthesis even if an image with a small change amount is synthesized. Therefore, a high-definition composite image can be obtained without consuming unnecessary resources.

  Alternatively, a specific area may be set in all areas of each image, and an image part belonging to the specific area may be combined as a combined image. As a result, the size of the synthesized image becomes smaller, so that the synthesis process can be further reduced.

  It is preferable that the image composition unit synthesizes each image to be synthesized by enlarging the images to a predetermined magnification. As a result, a composite image enlarged at a predetermined magnification is obtained, and the visibility of the composite image is improved.

  A plurality of images (base images) may be input to the image processing apparatus via a network. In this case, the plurality of images are generated by an imaging device / imaging unit such as a network camera connected to the network, for example. The imaging device / imaging unit preferably has an optical zoom function. Since the image enlarged by the optical zoom is enlarged and synthesized in the image processing apparatus, a higher-definition synthesized image can be obtained.

  In order to solve the above-described problem, according to the third aspect of the present invention, the number of acquired images is determined according to the step of storing a plurality of images in the image storage unit and the size of a specific area in the entire area of each image. A step of calculating, a step of acquiring images for the number of acquired images from the image storage unit, detecting a change amount between image portions belonging to a specific area of each acquired image, and a plurality of images acquired from the image storage unit. An image processing comprising: a step of setting an image portion having an amount of change larger than a predetermined value among the included image portions as a composite image; and a step of combining the composite image based on the amount of change. A method is provided. Here, the specific area may be a part of the entire area of the image, or may coincide with the entire area of the image. Similarly, the image portion may be a part of the specific area or may coincide with the specific area. According to this method, the number of combined images increases or decreases according to the image size. Therefore, if the image size is large, the number of synthesized images is reduced to reduce the image synthesis process. Conversely, if the image size is small, the number of synthesized images is increased to generate a finer synthesized image. It becomes possible. Further, according to this method, an image with a small change amount is not used for the composition process. Even if an image with a small change amount is synthesized, the effect is low. Therefore, a high-definition composite image can be obtained without consuming unnecessary resources. Note that it is also possible to synthesize all image portions by appropriately setting a predetermined value.

  According to the present invention, it is possible to obtain a high-definition composite image with smaller resources.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

<First Embodiment>
FIG. 1 is a block diagram schematically showing the configuration of the image processing apparatus 100 according to the first embodiment of the present invention. As shown in the figure, the image processing apparatus 100 is connected to a camera 10 (imaging apparatus, imaging unit) via a network 20.

  The image processing apparatus 100 includes a control unit 110, an image decompression unit 120, an image display unit 130, a region of interest designation unit 140 (specific region setting unit), a magnification designation unit 150, a recording unit 190, and a high-definition image creation unit 200. Has been.

  The control unit 110 controls operations of various functional units of the image processing apparatus 100. The image decompression unit 120 decompresses (decompresses / decompresses) a plurality of compressed shooting frames (base image, moving image stream, or a plurality of continuous still images) sent from the camera 10, and extracts the plurality of shooting frames. It has a function of converting to an uncompressed still image. The image display unit 130 displays the shooting frame sent from the camera 10 and the high definition image created by the high definition image creation unit 200. The attention area designation unit 140 has a function of designating a part of the shooting frame as an attention area (specific area). The magnification designation unit 150 has a function of designating the zoom magnification of the attention area designated by the attention area designation unit 140. The recording unit 190 has a function of recording a shooting frame and a high-definition image. The high-definition image creation unit 200 has a function of creating a high-definition image based on a plurality of shooting frames, and includes an image storage unit 210, a composite number calculation unit 220, an image change amount detection unit 240, and an image composition unit. 250.

  The image storage unit 210 has a function of storing the image of the attention area designated by the attention area designation unit 140. The composite number calculation unit 220 has a function of detecting the composite number of images necessary for creating a high-definition image according to the size of the region of interest. The image change amount detection unit 240 detects the change amount between images. The image synthesizing unit 250 has a function of synthesizing each image using an amount of change between images.

  The operation of the image processing apparatus 100 according to this embodiment configured as described above will be described.

  The camera 10 is set to be monitored and transmits a local moving image to the image processing apparatus 100 in real time through the network 20 sequentially. At this time, the camera 10 compresses the image data and transmits the stream in order to ensure the data transfer speed.

  The image processing apparatus 100 is set in an environment where a user (monitoring person) performs monitoring work. In the image processing apparatus 100, first, the control unit 110 receives stream data (compressed moving image data) and sends it to the image decompression unit 120. The image decompression unit 120 decompresses moving image data. As a result, moving images captured by the camera 10 are extracted as a plurality of captured frames.

  Next, the control unit 110 sends the extracted shooting frame to the image display unit 130. The image display unit 130 displays the captured frame on the screen of a display device (not shown) built in the image processing apparatus 100 or externally connected. FIG. 2 shows the entire screen area A100 of this display device. Generally, the size of the display screen is larger than the image size of a shooting frame shot with a network camera or the like. Therefore, the image display unit 130 can display the shooting frame at the same magnification in the first sub-region A110 in the entire screen region A100.

  The user can monitor a predetermined target while viewing an image (photographing frame) displayed in the first sub area A110. However, if the size of the shooting frame is small (the image is rough), there is a possibility that the monitoring target cannot be visually recognized with certainty. In this regard, the image processing apparatus 100 according to the present embodiment increases the resolution of the area that the user particularly pays attention to (that is, the image area including the monitoring target), and displays a high-definition image in the second sub-area A120. The area A121 can be enlarged and displayed.

  For example, the user uses a mouse (not shown) connected to the image processing apparatus 100 to define an area including the monitoring target from the first sub area A110. This definition information is input to the attention area designation unit 140. The attention area designation unit 140 designates the image area defined by the user as the attention area A111 based on the demarcation information. Then, the attention area designation unit 140 transmits information regarding the size and position of the attention area A111 (hereinafter referred to as “size / position information”) to the control unit 110. Hereinafter, the present embodiment will be described in the case where the attention area A111 is rectangular.

  The attention area designation unit 140 defines the position of the attention area A111 with the coordinates of the upper left corner. The origin of the coordinate system used here is, for example, the upper left corner of the first sub-region A110. Also, the attention area designation unit 140 defines the size of the attention area A111 by the number of pixels in the vertical direction and the horizontal direction.

  The user uses, for example, the slider S displayed in the third sub area A130 in the entire screen area A100 to display an image displayed in the attention area A111 (hereinafter referred to as “attention image”) in a high-definition image display area. It is possible to adjust the enlargement magnification when displaying on A121. The magnification designation unit 150 notifies the control unit 110 of the magnification adjusted by the user. In the present embodiment, the magnification can be adjusted in units of 1/10 from 1 to n times. When the user does not adjust the magnification, the magnification designating unit 150 notifies the control unit 110 of a predetermined default magnification (for example, 2 times).

  The control unit 110 transmits the size / position information acquired from the attention area specifying unit 140 and the magnification information acquired from the magnification specifying unit 150 to the high-definition image creating unit 200 together with the captured frame received from the camera 10.

  The high-definition image creation unit 200 performs super-resolution processing (processing for obtaining a change amount of a plurality of images and synthesizing a high-resolution image) to make the attention image (image portion) high-definition. In general, in super-resolution processing, the use of as many images as possible can provide a higher definition effect. However, in order to create high-definition images in real time using many images, appropriate resources are required. Therefore, the number of images to be used is determined according to the image processing capability of the high-definition image creation unit 200. In addition, according to the present embodiment, the target to be refined is the attention image. Therefore, the number of images that the high-definition image creation unit 200 can use for the super-resolution processing also depends on the size of the attention area A111. That is, if the attention area A111 is large, the amount of processing for each image increases, so the number of usable images decreases. However, if the attention area A111 is small, the processing amount for each image decreases, so The number increases.

  In this regard, the high-definition image creation unit 200 according to the present embodiment determines an appropriate number of images according to the size of the attention area A111 and executes super-resolution processing. As a result, the target image is enlarged and displayed in the high-definition image display area A121 in real time and with extremely high definition. Hereinafter, the details of the image processing performed by the high-definition image creation unit 200 will be described in detail.

  First, in the high-definition image creation unit 200, the image storage unit 210 includes an image of interest included in shooting frames (F0, F1, F2,..., Fn) continuously sent from the camera 10 to a built-in buffer. (R0, R1, R2,..., Rn) are sequentially stored. Although sufficient capacity is allocated to this buffer, if the capacity is insufficient, the attention image of the oldest photographed frame is erased.

  Next, the composite number calculation unit 220 receives the size of the attention area A111 from the control unit 110, and uses the number of images to be used for super-resolution processing based on (Equation 1) below (hereinafter referred to as "composite number"). SN is determined.

SN = (PP / (W × H)) × N (Formula 1)

W: the number of horizontal pixels of the attention area A111,
H: Number of vertical pixels of the attention area A111
PP: Super-resolution processing capability of the high-definition image creation unit 200 (processor clock frequency MHz),
N: Conversion constant

  For example, when the clock frequency of the super-resolution processor included in the high-definition image creation unit 200 is 1 GHz, PP = 1000. In this embodiment, N = 100.

  The number of horizontal pixels W of the attention area A111 and the number of vertical pixels H of the attention area A111 are given from the attention area designation section 140 to the composite number calculation section 220. The super-resolution processing capability PP and the conversion constant N of the high-definition image creation unit 200 are registered in advance in the composite number calculation unit 220.

  For example, when W = 160, H = 120, and PP = 1000, SN = 5.208333... The number of composites is determined to be 5 by rounding off the decimal places. Thus, it is preferable that the number of combined images is determined according to the size of the attention area A111, and the determination method is not limited to the method using Equation 1 above.

  The composite number calculation unit 220 notifies the image change amount detection unit 240 of the calculated composite number (here, “5”).

  The image change amount detection unit 240 receives, from the image storage unit 210, attention images for the composite number notified from the composite number calculation unit 220. At this time, the image change amount detection unit 240 preferentially acquires a noticed image with a new shooting time. Here, since the number of combined images is “5”, the image change amount detection unit 240 performs the attention image (R0), the attention image (R1), the attention image (R2), the attention image (R3), and the attention image (R4). ) To get.

  Subsequently, the image change amount detection unit 240 obtains a change amount of the acquired target image. This operation will be described with reference to FIG.

  The image change amount detection unit 240 sets the target image R0 as the reference target image, and focuses on one pixel PX0 in the target image R0. Further, the image change amount detection unit 240 extracts a pixel PX1 corresponding to the pixel PX0 in the target image R0 from all the pixels in the target image R1 using, for example, an optical flow method. Then, the image change amount detection unit 240 calculates a change amount between the pixel PX0 in the target image R0 and the pixel PX1 in the target image R1.

  For example, the position coordinate of the pixel PX0 is (10, 30) in the xy coordinate system of the target image R0, and the position coordinate of the pixel PX1 is (7.5, 30.5) in the xy coordinate system of the target image R1. ), The change amounts from the pixel PX0 to the pixel PX1 are Δx (change amount in the x direction) = − 2.5 and Δy (change amount in the y direction) = 0.5.

  The image change amount detection unit 240 calculates the change amount in the same manner for all the pixels in the target image R1, and defines the calculated change amounts of all the pixels as “R0-R1 change amount between the target images”. Further, the image change amount detection unit 240 performs the same calculation for the attention image R0, the attention image R2, the attention image R0, the attention image R3, and the attention image R0 and the attention image R4. As a result, “the amount of change between R0-R2 attention images”, “the amount of change between R0-R3 attention images”, and “the amount of change between R0-R4 attention images” are obtained.

  Next, the image synthesizing unit 250 receives the four inter-image change amounts (“R0-R1 inter-image change amount”, “R0-R2 inter-image change amount”, “R0-R3 attention” from the image change amount detection unit 240. "Inter-image change amount" and "R0-R4 inter-image change amount") and five target images (R0) to (R4), and one image so that the five target images have a specified magnification. To synthesize. This combining process will be described with reference to FIG.

  First, the image composition unit 250 uses a linear enlargement method (for example, a bilinear method) to enlarge the reference image of interest R0 so as to have a magnification n designated by the magnification designation unit 150. As a result, an enlarged attention image R0ex is obtained. Note that the pixel PX0 belonging to the target image R0 corresponds to the pixel (PX0ex) belonging to the target image R0ex and its peripheral pixels (not shown) by enlarging the target image R0 n times in this way. become.

  Next, the image synthesizing unit 250 synthesizes (interpolates) each pixel constituting the target image (R1) into the enlarged target image R0ex using the R0-R1 target image change amount. For example, the pixel PX1 (see FIG. 3) belonging to the target image R1 is combined with the enlarged target image R0ex as follows.

  As described above, the pixel PX1 corresponds to the pixel PX0 belonging to the target image R0, and the amounts of change thereof are Δx = −2.5 and Δy = 0.5. Further, since the position coordinate of the pixel PX0 belonging to the target image R0 is (10, 30), the position coordinate of the pixel (PX0ex) belonging to the enlarged target image R0ex is represented by (10n, 30n) which is n times. Therefore, the pixel PX1 is magnified n times and synthesized at the position of coordinates (10n−2.5n, 30n + 0.5n).

  After synthesizing each pixel constituting the target image (R1) with the enlarged target image R0ex, the image composition unit 250 sequentially converts each pixel constituting the target image (R2) to (R4) into the enlarged target image R0ex. Synthesize. The image composition unit 250 creates a high-definition image (moving image) that has been magnified n times by repeating the composition process described above. Then, the image composition unit 250 passes the created high-definition image to the control unit 110.

  The image display unit 130 receives the high-definition image from the control unit 110 and displays it in the high-definition image display area A121 (see FIG. 2) on the screen of the display device.

  The recording unit 190 receives and / or records a captured frame and / or a high-definition image from the control unit 110. It should be noted that the object to be recorded by the recording unit 190 may be configured to be specified by the user, or may be fixed.

  By sequentially performing the above processing in all the shooting frames, a high-definition moving image can be displayed and recorded in real time.

  As described so far, according to the present embodiment, a high-definition image can be created in real time without using a large resource. Therefore, even when the camera 10 is not equipped with the optical zoom function, the image including the monitoring target is displayed and recorded in a state of being enlarged to a magnification desired by the user. As a result, the visibility of the monitoring target is extremely high.

<Second Embodiment>
FIG. 5 is a block diagram schematically showing the configuration of the image processing apparatus 101 according to the second embodiment of the present invention.

  The image processing apparatus 101 has a configuration in which the high-definition image creation unit 200 is replaced with a high-definition image creation unit 201 with respect to the image processing apparatus 100 according to the first embodiment. That is, the image processing apparatus 100 includes a control unit 110, an image decompression unit 120, an image display unit 130, an attention area designation unit 140, a magnification designation unit 150, a recording unit 190, and a high-definition image creation unit 201.

  The high-definition image creation unit 201 has a configuration in which a composite image determination unit 260 is added to the high-definition image creation unit 200 according to the first embodiment. That is, the high-definition image creation unit 201 has a function of creating a high-definition image based on a plurality of shooting frames, and includes an image storage unit 210, a composite number calculation unit 220, an image change amount detection unit 240, an image composition. Section 250 and a composite image determination section 260.

  The combined image determination unit 260 has a function of determining whether or not to use each image for the combining process based on the change amount detected by the image change amount detection unit 240.

  The operation of the image processing apparatus 101 according to this embodiment configured as described above will be described.

  In the image processing apparatus 101, each operation of the control unit 110, the image decompression unit 120, the image display unit 130, the attention area designation unit 140, the magnification designation unit 150, the recording unit 190, the image storage unit 210, and the composite number calculation unit 220 is described. Is the same as that of the image processing apparatus 100 according to the first embodiment described above.

  The image processing apparatus 101 according to the present embodiment brings a remarkable effect particularly when the amount of change between shooting frames is small. For example, when the shooting frame sent from the camera 10 is at a high rate (for example, 30 fps), or when the monitoring target is moving very slowly, the amount of change between the shooting frames is small. Super-resolution processing is a technique for obtaining a high-definition image by synthesizing pixels acquired at various positions (coordinates). However, when pixels with extremely small changes are combined, the effect obtained on the amount of computation may be reduced.

  In this regard, in the present embodiment, when the amount of change between the reference image of interest and a certain image of interest is sufficiently small, the image of interest is not used for image synthesis processing, and is more time-wise than the reference image of interest. Another distant image of interest is employed in the image composition process. Hereinafter, features of the image processing apparatus 101 according to the present embodiment will be described in detail.

  Here, as in the first embodiment, the number of combined images is “5”. The image change amount detection unit 240 receives the attention image (R1) closest in time to the attention image (R0), which is the reference attention image, from the image storage unit 210, and uses the same method as in the first embodiment. The amount of change between R0-R1 attention images is calculated. Then, the image change amount detection unit 240 notifies the combined image determination unit 260 of the calculated change amount between R0-R1 attention images.

  The synthesized image determination unit 260 extracts pixels whose change amount is not “0” from all pixels belonging to the target image (R1) based on the change amount between the R0-R1 target images. Then, the synthesized image determination unit 260 obtains an average value of the amount of change of each extracted pixel (hereinafter referred to as “average amount of change”).

  Here, the reason for excluding the pixels whose change amount is “0” when the composite image determination unit 260 calculates the average change amount is as follows.

  A pixel whose change amount is not “0” corresponds to a moving object whose position changes between frames. Conversely, a pixel whose change amount is “0” is a static object whose position does not change between frames (for example, background). ). When a minute moving object existing in a stationary environment is photographed and the average of the amount of change of each pixel constituting the photographed image is obtained, a value close to “0” can be obtained even if the momentum of the moving object is large. Thus, the average value of the change amount of each pixel is affected by the ratio of the still body region and the moving body region in the image. Therefore, the synthesized image determination unit 260 excludes pixels whose change amount is “0” from the calculation for obtaining the average change amount in order to accurately grasp the change amount of the pixel corresponding to the moving object.

  When the average change amount is smaller than the predetermined determination threshold, the synthesized image determination unit 260 has a small change amount with respect to the reference attention image (R0), and the attention image (R1) is determined as the reference attention image (R1). R0) determines that the effect of high definition is small, and instructs the image composition unit 250 not to execute the composition processing using the target image (R1).

  Thereafter, the image change amount detection unit 240 receives the attention image (R2) belonging to the next shooting frame from the image storage unit 210, and calculates the amount of change between R0-R2 attention images. The synthesized image determination unit 260 calculates the average change amount of the target image (R2) based on the change amount between the R0-R2 target images. When the average change amount equal to or greater than the determination threshold is calculated, the combined image determination unit 260 instructs the image combining unit 250 to execute a combining process using the target image (R2).

  Thereafter, the image change amount detection unit 240 and the combined image determination unit 260 sequentially execute the same processing for all the target images for the composite number (here, the target image (R3) and the target image (R4)). .

  As described above, according to the present embodiment, the same effect as in the first embodiment can be obtained. In addition, according to the present embodiment, the image synthesis process is not executed for the target image with little change between frames, so that the resources necessary for creating a high-definition image can be further reduced. In addition, since the time required for image composition processing can be shortened, the real-time performance of high-definition image reproduction is improved. In addition, since the high-definition images can be smoothly played back and recorded, the visibility of the monitoring target is further improved.

<Third Embodiment>
FIG. 3 is a block diagram schematically showing the configuration of the image processing apparatus 102 according to the third embodiment of the present invention.

  The image processing apparatus 102 is different from the image processing apparatus 100 according to the first embodiment in that a display area designating unit 160 is added, an attention area designating unit 140 and a magnification designating unit 150 are omitted, and high-definition image creation is performed. The unit 200 has a configuration in which the high-definition image creation unit 202 is replaced. In other words, the image processing apparatus 102 includes a control unit 110, an image decompression unit 120, an image display unit 130, a display area designation unit 160, a recording unit 190, and a high-definition image creation unit 202.

  The display area designating unit 160 has a function of designating a display area of a high-definition image from the entire area of the screen of a display device (not shown) built in the image processing apparatus 102 or externally connected. .

  The high-definition image creation unit 202 is different from the high-definition image creation unit 200 according to the first embodiment in that the image storage unit 210 is replaced with an image storage unit 212, and the composite number calculation unit 220 is a composite number calculation unit 222. The attention area setting unit 270 is added. That is, the high-definition image creation unit 202 has a function of creating a high-definition image based on a plurality of shooting frames, and includes an image storage unit 212, a composite number calculation unit 222, an image change amount detection unit 240, an image composition. Part 250 and an attention area setting part 270.

  The image storage unit 212 has a function of storing a shooting frame sent from the camera 10. The attention area setting section 270 has a function of setting the attention area in accordance with the amount of change detected by the image change amount detection section 240. The composite number calculation unit 222 has a function of calculating the composite number of images necessary for creating a high-definition image in accordance with the size of the attention area set by the attention area setting unit 270.

  The operation of the image processing apparatus 102 according to this embodiment configured as described above will be described.

  In the image processing apparatus 102, the operations of the control unit 110, the image decompression unit 120, and the recording unit 190 are the same as those of the image processing apparatus 100 according to the first embodiment described above.

  The image display unit 130 displays a shooting frame sent from the camera 10 on a screen of a display device built in the image processing apparatus 102 or externally connected. FIG. 7 shows the entire screen area B100 of this display device. Generally, the size of the display screen is larger than the image size of a shooting frame shot with a network camera or the like. Therefore, the image display unit 130 can display the shooting frame at the same magnification in the first sub-region B110 in the entire screen region B100.

  The user can monitor a predetermined target (for example, a moving object such as a suspicious person or a suspicious vehicle) while viewing an image (photographing frame) displayed in the first sub-region B110. However, if the size of the shooting frame is small (the image is rough), there is a possibility that the monitoring target cannot be visually recognized with certainty. In this regard, according to the image processing apparatus 102 according to the present embodiment, the second sub-region B120 for displaying a high-definition image is secured in the entire screen region B100. The user can specify the high-definition image display area B121 in the second sub-area B120. Then, the moving object in the image displayed in the first sub-region B110 and its peripheral portion are enlarged and displayed in the high-definition image display region B121.

  For example, the user uses a mouse (not shown) connected to the image processing apparatus 102 to define an image area in which the enlarged high-definition image is displayed in the second sub-area B120. This definition information is input to the display area designating unit 160. The display area designating unit 160 designates the image area defined by the user as the high-definition image display area B121 based on the definition information. Then, the display area designating unit 160 transmits information on the size and position of the high-definition image display area B121 (hereinafter referred to as “size / position information”) to the control unit 110. Hereinafter, the present embodiment will be described in the case where the high-definition image display region B121 is rectangular.

  The display area designating unit 160 defines the position of the high-definition image display area B121 with the coordinates of its upper left corner. The origin of the coordinate system used here is, for example, the upper left corner of the second sub-region B120. The display area designating unit 160 defines the size of the high-definition image display area B121 by the number of pixels in the vertical direction and the horizontal direction.

  The control unit 110 transmits the size / position information acquired from the display area designating unit 160 to the high-definition image creating unit 202 together with the captured frame received from the camera 10.

  The high-definition image creation unit 202 creates a high-definition image using super-resolution processing technology, as with the high-definition image creation unit 200 according to the first embodiment. However, the operation differs between the following two points.

  First, the operation related to the identification of the target image is different. According to the first embodiment, the attention area in the shooting frame is designated by the user. On the other hand, according to the present embodiment, the image processing apparatus 102 automatically detects a moving object in a shooting frame and designates a region including this object as a region of interest. Then, the image of the attention area is enlarged and displayed with a high resolution.

  Second, the operation relating to the designation of magnification when enlarging the image of the attention area is different. According to the first embodiment, this magnification is adjusted by the user. On the other hand, according to the present embodiment, the image processing apparatus 102 compares the size of the region of interest with the size of the high-definition image display region B121 to determine the enlargement magnification.

  Hereinafter, the details of the image processing by the high-definition image creation unit 202 will be described in detail.

  First, in the high-definition image creation unit 202, the image storage unit 212 sequentially stores captured frames (F0, F1, F2,..., Fn) continuously sent from the camera 10 in a built-in buffer. A sufficient capacity is allocated to this buffer. If the capacity is insufficient, the oldest shooting frame is deleted.

  Next, the image change amount detection unit 240 receives the shooting frame (F0) and the shooting frame (F1) from the image storage unit 210. Then, the image change amount detection unit 240 calculates the change amount of all the pixels in the shooting frame (F1) with the corresponding pixel in the shooting frame (F0). It is defined as “F0-F1 change amount between photographing frames”. The image change amount detection unit 240 sends this F0-F1 change amount between photographing frames to the attention area setting unit 270.

  The attention area setting unit 270 detects a moving object from the images displayed in the first sub-region B110 based on the F0-F1 inter-frame change amount acquired from the image change amount detecting unit 240, and the moving object is detected. The region of the smallest size that is completely included is set as the region of interest. At this time, the attention area setting section 270 acquires the size / position information of the high-definition image display area B121 from the control section 110, so that the shapes of the attention area and the high-definition image display area B121 are similar (vertical / horizontal ratio). Adjust the size of the region of interest (so that they are the same rectangle).

  By the way, it is preferable to refer to the amount of change of each pixel in order to detect a moving object from the image displayed in the first sub-region B110. That is, it can be determined that pixels whose change amount is not “0” constitute a part of the moving image.

  Here, the attention area setting operation by the attention area setting unit 270 will be specifically described with reference to FIG.

  When the shooting frame (F0) and the shooting frame (F1) include the pedestrian (moving object) P, the attention area setting unit 270 sets the minimum rectangular area including the entire pedestrian P as the moving object area C100. . As described above, the change amount of the pixel corresponding to the pedestrian P is other than “0”, and the change amount of the pixel corresponding to the background of the pedestrian P is “0”. The attention area setting section 270 can reliably extract the moving object area C100 based on the change amount of each pixel.

  Next, the attention area setting section 270 sets an area of the minimum size that completely includes the moving object area C100 as the attention area C200. However, the attention area setting section 270 adjusts the size of the attention area C200 so that the shapes of the attention area C200 and the high-definition image display area B121 are similar (so that the aspect ratio is the same rectangle). Further, the attention area setting unit 270 compares the sizes of the attention area C200 and the high-definition image display area B121 (for example, the number of pixels in the vertical direction or the number of pixels in the horizontal direction) to enlarge the image of the attention area C200. Determine the magnification.

  The attention area setting section 270 sends the set size information of the attention area C200 to the composite number calculation section 222.

  Similar to the first embodiment, the composite number calculation unit 222 calculates the composite number of images that can be super-resolution processed in real time based on the size of the attention area C200 and the capability of the processor.

  The image composition unit 250 is based on the enlargement ratio of the image of the attention area C200 set by the attention area setting unit 270 and the composite number of images calculated by the composite number calculation unit 222. Similarly, the image of the attention area C200 is enlarged to create a high-definition image, and this is sent to the control unit 110.

  The image display unit 130 receives the high-definition image from the control unit 110 and displays the high-definition image of the moving object in the high-definition image display region B121 as shown in FIG.

  As described above, according to the present embodiment, the same effect as in the first embodiment can be obtained. Moreover, according to the present embodiment, it is possible to automatically enlarge and display the moving object included in the shooting frame in the area desired by the user (high-definition image display area B121).

<Fourth embodiment>
FIG. 9 is a block diagram schematically showing the configuration of the image processing apparatus 103 according to the fourth embodiment of the present invention.

  The image processing apparatus 103 is different from the image processing apparatus 100 according to the first embodiment in that the attention area specifying unit 140 is omitted and the high-definition image creating unit 200 is replaced with a high-definition image creating unit 203. have. That is, the image processing apparatus 103 includes a control unit 110, an image decompression unit 120, an image display unit 130, a magnification designation unit 150, a recording unit 190, and a high-definition image creation unit 203.

  The high-definition image creation unit 203 has a configuration in which an attention area extraction unit 280 is added to the high-definition image creation unit 200 according to the first embodiment. That is, the high-definition image creation unit 203 has a function of creating a high-definition image based on a plurality of shooting frames, and includes an image storage unit 210, a composite number calculation unit 220, an image change amount detection unit 240, an image composition. Part 250 and a region of interest extraction part 280.

  The attention area extraction unit 280 has a function of extracting the attention area from the shooting frame.

  The operation of the image processing apparatus 103 according to this embodiment configured as described above will be described.

  In the image processing apparatus 103, each operation of the control unit 110, the image decompression unit 120, the magnification designation unit 150, the recording unit 190, the image storage unit 210, the composite number calculation unit 220, the image change amount detection unit 240, and the image composition unit 250. Is the same as the image processing apparatus 100 according to the first embodiment described above.

  The camera 15 is set for a monitoring target, and sequentially transmits a local moving image to the image processing apparatus 103 in real time through the network 20. At this time, the camera 15 compresses the image data and transmits the stream in order to ensure the data transfer speed. The camera 15 has an optical zoom function, and performs an optical zoom operation in response to a zoom instruction and an enlargement magnification designation from the image processing apparatus 103.

  The image display unit 130 displays the captured frame on a screen of a display device (not shown) built in the image processing apparatus 103 or externally connected. FIG. 10 shows the entire screen area D100 of this display device. In the entire screen area D100, a first sub area D110 and a second sub area D120 are secured. The size of the shooting frame is adjusted according to the size of the first sub-region D110. A slider S is displayed in the second sub-region D120.

  The user can use the slider S displayed in the second sub area D120 to adjust the enlargement magnification when the captured frame is displayed in the first sub area D110. Thus, the slider S displayed in the second sub-region D120 functions in the same manner as a zoom slide switch mounted on a general video camera. The magnification specifying unit 150 notifies the control unit 110 of the magnification of the shooting frame adjusted by the user. In the present embodiment, the magnification can be adjusted in units of 1/10 from 1 to n times. When the user does not adjust the magnification, the magnification designating unit 150 notifies the control unit 110 of a preset default magnification (for example, 1 time).

  Here, the relationship between the position of the slider S displayed in the second sub area D120 and the magnification of the shooting frame displayed in the first sub area D110 will be described.

  As shown in FIG. 10, when the slider S is at the left end, the enlargement magnification is adjusted to “1 (default)”. When the slider S is adjusted from the left end to the center range, the photographing frame is enlarged by the optical zoom function provided in the camera 10. The magnification at this time changes according to the position of the slider S. As shown in FIG. 11, when the position of the slider S reaches the center, the photographing frame is enlarged to the maximum by the optical zoom function provided in the camera 10.

  When the slider S is adjusted in the range from the center to the right end, the high-definition image creation unit 203 performs super-resolution processing on the captured frame to create an enlarged high-definition image. The enlarged high-definition image is displayed in the first sub-region D110 and recorded by the recording unit 190.

  Hereinafter, the shooting frame enlargement process using the optical zoom function of the camera 15 and the shooting frame super-resolution process performed by the high-definition image creation unit 203 will be described in more detail.

  As described above, the magnification specifying unit 150 notifies the control unit 110 of the magnification adjusted by the user. At this time, if the position of the slider S is in the range from the left end to the center, the control unit 110 gives the magnification designation information obtained from the magnification designation unit 150 to the camera 15 via the network 20. The camera 15 operates the optical zoom function according to the magnification designation information. As a result, an image near the center of the shooting frame is enlarged and displayed in the first sub-region D110 of the screen of the display device.

  When the position of the slider S is to the right of the center (see FIG. 11), the control unit 110 sends the shooting frame and the magnification designation information obtained from the magnification designation unit 150 to the high-definition image creation unit 203. At this time, the photographing frame sent to the high-definition image creating unit 203 is photographed with maximum magnification by the optical zoom function of the camera 15. In the following, the present embodiment will be described in the case where the photographic frame enlarged to the maximum by the optical zoom function provided in the camera 15 is further magnified m times. Here, “m-magnification” means that each of the vertical and horizontal directions of the image is expanded m times.

  In the high-definition image creation unit 203, first, the attention area extraction unit 280 extracts the attention area from the captured frame based on the magnification designation information (m-magnification). For example, as shown in FIG. 12, the attention area extraction unit 280 sets a rectangular area whose width and height are 1 / m of the shooting frame E100, and matches the center of the rectangular area with the center of the shooting frame E100. Let The attention area extraction unit 280 determines this rectangular area as the attention area E200 of the shooting frame E100.

  The composite number calculation unit 220 receives the size of the attention area E200 from the attention area extraction unit 280, and determines the number of images to be used for the super-resolution processing, that is, the composite number. In addition, the image storage unit 210 sequentially stores the images of the attention area E200 (the attention image) included in the captured frames continuously sent from the camera 15 in the built-in buffer. Then, the image change amount detection unit 240 and the image composition unit 250 operate in the same manner as in the first embodiment, and a high-definition image is sent to the control unit 110.

  The image display unit 130 receives a high-definition image from the control unit 110 and displays it in the first sub-region D110 of the screen of the display device as shown in FIG.

  As described above, according to the present embodiment, the optical zoom function of the camera 15 is combined with a high-definition image creation function by super-resolution processing. As a result, the monitoring target can be displayed and recorded at a higher magnification than the optical zoom function of the camera 15 and the visibility of the monitoring target is further improved.

<Fifth embodiment>
FIG. 14 is a block diagram schematically showing a configuration of a camera device 300 according to the fifth embodiment of the present invention.

  The camera device 300 includes a sensor unit 310, a zoom unit 320, and an image processing device 103 according to the fourth embodiment. Note that since the shooting frame is transmitted to the image processing apparatus 103 via a circuit having a wide bandwidth inside the camera apparatus 300, the compression process for the shooting frame can be omitted. In this case, the image processing apparatus 103 need not include the image decompression unit 120.

  The sensor unit 310 has a photoelectric conversion function that takes in an optical signal from the outside and converts it into an electrical signal (digital data). The zoom unit 180 is a functional unit that controls the zoom operation of an optical mechanism (not shown) included in the camera device 300.

  An operation of the camera apparatus 300 according to the present embodiment configured as described above will be described.

  A photographing frame is generated by the photoelectric conversion operation of the sensor unit 170. The captured frame is sent to the image display unit 130 via the control unit 110.

  The image display unit 130 displays a shooting frame on a liquid crystal finder (not shown) mounted on the camera device 300. Further, when a high-definition image is sent from the high-definition image creation unit 203, the image display unit 130 displays the high-definition image on the liquid crystal finder.

  The magnification designation unit 150 designates an enlargement magnification of the shooting frame in accordance with, for example, the operation of a slide switch mounted on the camera device 300. The zoom unit 320 controls the zoom operation of the optical mechanism provided in the camera device 300 according to the magnification designated by the magnification designation unit 150. However, when the magnification designated by the magnification designation unit 150 exceeds the limit value of the optical zoom, the zoom unit 320 controls the optical mechanism so as to maintain the state where the optical zoom is maximized.

  Then, when the magnification designated by the magnification designation unit 150 exceeds the limit value of the optical zoom, the high-definition image creation unit 203 enlarges the height by super-resolution processing as in the fourth embodiment described above. The creation operation of the refined image is executed.

  As described above, according to the present embodiment, since the image processing apparatus 103 capable of super-resolution processing is incorporated in the camera apparatus 300, even if the optical zoom is low, the shooting frame is increased. The zoom magnification can be increased. In addition, in the present embodiment, the captured frame is enlarged by super-resolution processing rather than mere digital zoom, and thus an image obtained by this has extremely high definition.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, this invention is not limited to the example which concerns. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are of course within the technical scope of the present invention. Understood.

  For example, the image processing apparatuses 100 to 103 can be configured by software.

  The present invention is applicable to a camera monitoring system, for example.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. It is explanatory drawing which shows the whole screen area | region of the display apparatus with which the image processing apparatus of FIG. 1 is provided. It is operation | movement explanatory drawing of the image variation detection part with which the image processing apparatus of FIG. 1 is provided. It is operation | movement explanatory drawing of the image synthetic | combination part with which the image processing apparatus of FIG. 1 is provided. It is a block diagram which shows the structure of the image processing apparatus concerning the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the image processing apparatus concerning the 3rd Embodiment of this invention. It is explanatory drawing which shows the whole screen area | region of the display apparatus with which the image processing apparatus of FIG. 6 is provided. It is operation | movement explanatory drawing of the attention area setting part with which the image processing apparatus of FIG. 6 is provided. It is a block diagram which shows the structure of the image processing apparatus concerning the 4th Embodiment of this invention. It is explanatory drawing (the 1) which shows the whole screen area | region of the display apparatus with which the image processing apparatus of FIG. 9 is provided. FIG. 10 is an explanatory diagram (part 2) illustrating the entire screen area of the display device included in the image processing apparatus of FIG. 9; It is operation | movement explanatory drawing of the attention area extraction part with which the image processing apparatus of FIG. 9 is provided. FIG. 10 is an explanatory diagram (part 3) illustrating the entire screen area of the display device included in the image processing apparatus of FIG. 9; It is a block diagram which shows the structure of the image processing apparatus concerning the 5th Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10,15 Camera 20 Network 100-103 Image processing apparatus 110 Control part 120 Image decompression part 130 Image display part 140 Attention area designation part 150 Magnification designation part 160 Display area designation part 190 Recording part 200 High-definition image creation part 210,212 Image Storage unit 220, 222 Composite number calculation unit 240 Image change amount detection unit 250 Image composition unit 260 Composite image determination unit 270 Attention region setting unit 280 Attention region extraction unit 300 Camera device 310 Sensor unit 320 Zoom unit

Claims (8)

An image storage unit for storing a plurality of images;
A composite number calculator for calculating the composite number according to the size of the image;
An image change amount detection unit that acquires the number of images as the composite number from the image storage unit as a composite image, and detects a change amount between the composite images;
An image synthesis unit that synthesizes the synthesized images based on the amount of change;
An image processing apparatus comprising: An image storage unit for storing a plurality of images;
A composite number calculator for calculating the composite number according to the size of the image;
An image change amount detection unit that acquires the number of images for the composite number from the image storage unit and detects a change amount between the acquired images;
A combined image determination unit that uses, as a combined image, an image in which the change amount is greater than a predetermined value among a plurality of images acquired by the image change amount detection unit;
An image synthesis unit that synthesizes the synthesized images based on the amount of change;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the image synthesis unit synthesizes the respective synthesized images by enlarging them to a predetermined magnification. The image processing according to claim 1, wherein each image stored in the image storage unit is an image belonging to a specific area set in all areas of a plurality of base images. apparatus. An image storage unit for storing a plurality of images;
A specific area setting unit for setting a specific area in the entire area of each image;
A composite number calculating unit for calculating the composite number according to the size of the specific area;
An image change amount detection unit that acquires the number of images corresponding to the composite number from the image storage unit, and detects a change amount between combined images belonging to a specific region of each acquired image;
An image synthesis unit that synthesizes the synthesized images based on the amount of change;
An image processing apparatus comprising: An image storage unit for storing a plurality of images;
A specific area setting unit for setting a specific area in the entire area of each image;
A composite number calculating unit for calculating the composite number according to the size of the specific area;
An image change amount detection unit that acquires the number of images corresponding to the composite number from the image storage unit, and detects a change amount between image portions belonging to a specific region of each acquired image;
A combined image determination unit that uses, as an image to be combined, an image portion in which the change amount is greater than a predetermined value among image portions included in a plurality of images acquired by the image change amount detection unit;
An image synthesis unit that synthesizes the synthesized image based on the change amount;
An image processing apparatus comprising: The image processing apparatus according to claim 5, wherein the image synthesis unit synthesizes each of the synthesized images by enlarging the images to a predetermined magnification. Storing a plurality of images in an image storage unit;
Calculating the number of acquired images according to the size of a specific area in the entire area of each image;
Acquiring images for the number of acquired images from the image storage unit, and detecting a change amount between image portions belonging to a specific area of each acquired image;
Of the image parts included in the plurality of images acquired from the image storage unit, the step of setting an image part whose change amount is larger than a predetermined value as a synthesized image;
Combining the image to be combined based on the amount of change;
An image processing method comprising:

Images