JPH10112866A - Image processing apparatus and method - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To enable a receiving side to recognize a high-definition image at an early stage. An imaging device includes a CMGY complementary color mosaic filter. The optical axis shifter 14 shifts the photographing optical axis by one pixel to the right, down, and left, and the color processing circuit 32
RGB image data is generated from images captured at two optical axis positions. The generated image data (frame 1) is stored in the memory 3
4 is encoded by the encoding circuit 38 and output to the communication line 40. Further, the frames 2, 3 and 4 are similarly generated by one-pixel shift processing with reference to the position obtained by shifting the optical axis to the right, obliquely and downward by half a pixel by the optical axis shifter 14. The encoding circuit 38 encodes the frames 2, 3 and 4 by taking the difference of the frame 1 and sends the result to the communication line 40. When the receiving side receives each of the frames 1 to 4, it combines the frames with the previously received frames and displays the images.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method for transmitting a high-resolution image.

[0002]

2. Description of the Related Art When transmitting an image, it is common that a transmitting side compresses image information by a well-known high-efficiency compression method and a receiving side expands the compressed image information. As a compression method for natural images, for example, the JPEG method is known. .

[0003]

The JPEG system includes D
An irreversible method using CT (Discrete Cosine Transform) and S
There is a reversible method called a “patial method”. Neither compression algorithm can proceed to the compression encoding process unless the entire image of one screen to be transmitted is obtained. During that time, the receiving side is in a state of waiting for reception and checks the image (for example, monitor Cannot be displayed).

[0004] It is an object of the present invention to solve such a problem and to provide an image processing apparatus and method capable of confirming a transmission image on a receiving side more quickly.

It is another object of the present invention to provide an image processing apparatus and method capable of starting compression encoding earlier.

[0006]

According to the present invention, the same object image is picked up a plurality of times by shifting the pixels by the pixel shifting mechanism, and a plurality of image data obtained thereby are encoded.
Transmission via a transmission path. As a result, high-definition image information can be transmitted at an early stage, and can be confirmed at the receiving side at an early stage.

[0007] The first processing by a predetermined number of n-pixel shift processings
And a second frame image obtained by shifting the n-pixels the predetermined number of times after shifting the m-pixels in a predetermined direction from the reference position of the first frame image to obtain a resolution exceeding the resolution of the frame alone. Image can be obtained. Also, by encoding the second frame image as a difference from the first frame image, the second frame image is encoded.
The amount of transmission data of the frame image can be reduced, and the transmission time can be reduced.

At the receiving end, each time each frame is received, one or more previously received frames are combined to reconstruct the original image, so that at the receiving end, the original image is gradually refined. You can check.

[0009]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a schematic block diagram of a transmitting side according to an embodiment of the present invention, and FIG. 2 is a schematic block diagram of a receiving side.

First, the configuration of the transmitting side shown in FIG. 1 will be described. Reference numeral 10 denotes a photographing lens, 12 denotes an aperture, and 14 denotes a parallel plate, which is an optical axis shifter that shifts the optical axis of the photographing lens 10 in the lateral direction by an arbitrary amount. Reference numeral 16 denotes a drive circuit for driving the photographing lens 10 and the aperture 12, and reference numeral 18 denotes a control circuit for controlling the shift amount of the optical axis shifter 14 with respect to the inclination amount of the optical axis shifter 14 with respect to the optical axis of the photographing lens 10.

20 comprises a complementary color mosaic filter,
An image sensor that converts an optical image of a subject that has passed through the photographing lens 10, the aperture 12, and the optical axis shifter 14 into an electric signal. Note that the complementary color mosaic filter of the image sensor 20 includes magenta (M), green (G), cyan (C), and yellow (Y). S / 22 samples and holds the output signal of the image sensor 20 and automatically controls the gain thereof.
An H / AGC circuit; 24, an A / D converter for converting an analog signal output of the S / H / AGC circuit into a digital signal;
Reference numeral 6 denotes an image sensor 20, an S / H / AGC circuit 22, and an A / D
It is a timing generation circuit that supplies the converter 24 with a timing signal that defines these operation timings.

Reference numeral 28 denotes a memory for temporarily storing output data of the A / D converter 24, reference numeral 30 denotes a memory control circuit for controlling writing and reading of the memory 28, and reference numeral 32 denotes output memory of the A / D converter 24 or the memory 28. A color processing circuit for converting the data read from the CMYG space into an RGB space; 34, a memory for temporarily storing output data of the color processing circuit 32; 36, a memory control circuit for controlling writing and reading of the memory 34; An encoding circuit 38 compresses and encodes the data read from the memory 34 and outputs the data to the communication line 40.

Reference numeral 42 denotes a synchronization signal generation circuit for supplying a synchronization signal to the timing generation circuit 26 and the memory control circuits 30 and 36, and reference numeral 44 denotes a system control circuit for controlling the whole.

The configuration of the receiving side shown in FIG. 2 will be described. 50
Is a decoding circuit for expanding compressed image data received from the communication line 40, 52 is a memory for temporarily storing output data of the decoding circuit 50, 54 is a memory control circuit for controlling writing and reading of the memory 52, 56 is a memory A frame buffer 58 stores the image data read from the monitor 52 for display on a monitor, a monitor 58 displays the image data from the frame buffer 56 as a video, and a system control circuit 60 controls the whole.

Before specifically describing the operation of this embodiment,
A brief description will be given of a method of generating an image signal having a resolution higher than the original resolution of the image sensor 20 by laterally shifting the imaging optical axis.

Since the optical axis shifter 14 is a parallel flat plate,
As is well known, by changing the angle of the optical axis shifter 14 with respect to the photographing optical axis for two axes perpendicular to the photographing optical axis, the photographing optical axis can be shifted in the horizontal direction. That is, the optical axis shifter 14
The optical image on the photoelectric conversion surface of the image sensor 20 can be laterally moved by, for example, one pixel pitch or １ ／ pixel pitch in the horizontal direction and the vertical direction of the photoelectric conversion surface depending on the angle. The details of this pixel shifting process are described in
No. 06643.

In the present embodiment, a high-resolution image signal exceeding the basic resolution of the image sensor 20 is generated by utilizing such a pixel shifting process. For example, a resolution equivalent to a three-panel camera can be obtained by the one-pixel shifting process. FIG. 3 is an explanatory diagram illustrating the one-pixel shifting process. In FIG. 3, for simplicity, the imaging optical axis is not shifted laterally, but is illustrated as if the image sensor 20 is moved laterally. The pixels hatched in FIGS. 3A to 3D are at the same position in the subject optical image. Fig. 3 (2)
Is obtained by shifting the image sensor 20 by one pixel in the horizontal right direction relative to (1), and (3) is obtained by shifting the image sensor 20 by one pixel in the vertically lower direction relative to (2). In (4), the image sensor 20 is shifted by one pixel in the horizontal left direction relative to (3).
In this way, signals of the complementary colors M, G, C, and Y are obtained for the same position of the subject optical image, and RGB signals at that position can be obtained by color conversion. An image obtained by this series of one-pixel shift processing is referred to as frame 1.

Further, as shown in FIG. 4, a similar one-pixel shifting process is executed based on a position shifted by half a pixel in the horizontal direction relatively from the position shown in FIG. 3A. The image obtained in this way is referred to as frame 2. FIG.
A state in which the image sensor 20 is relatively shifted to the right in the horizontal direction by a half pixel from the state of (1) is shown. FIG. 4 (2),
(3) and (4) show a state in which one pixel shift has been executed based on FIG. 4 (1) in the same manner as in FIG.

This time, as shown in FIG. 5, a similar one-pixel shifting process is executed based on a position shifted by half a pixel in the horizontal and vertical directions relative to the position shown in FIG. 3A.
The image obtained in this way is referred to as frame 3. Fig. 5 (1)
3 shows a state in which the image sensor 20 is relatively shifted by half a pixel to the right in the horizontal direction and the lower side in the vertical direction from the state of FIG. FIGS. 5 (2), (3) and (4)
FIG. 5B shows a state in which one pixel shift has been performed on the basis of FIG.

Finally, as shown in FIG. 6, a similar one-pixel shifting process is performed based on a position shifted by half a pixel in the vertical direction relatively from the position shown in FIG. 3A. The image obtained in this way is referred to as frame 4. FIG.
A state in which the image sensor 20 is shifted downward by a half pixel relatively vertically from the state of (1) is shown. FIG. 6 (2),
(3) and (4) show a state in which one pixel shift has been executed based on FIG. 6 (1) in the same manner as in FIG.

The operation of this embodiment will be specifically described. The timing generation circuit 26 receives the image sensor 20 and the S / H AGC circuit 22 according to the synchronization signal from the synchronization signal generation circuit 42.
And a timing signal of a predetermined frequency to the A / D converter 24. In accordance with this timing signal, the imaging device 20 converts the charge signal corresponding to the optical image on the photoelectric conversion surface into S / HAG
The signal is output to the C circuit 22, the S / H AGC circuit 22 correlatively double-samples the output signal of the image sensor 20, and adjusts the gain.
2 is converted to a digital signal. A /
The output data of the D converter 24 is temporarily stored in the memory 28.

The system control circuit 44 causes the optical axis shifter 14 to laterally shift the photographing optical axis via the control circuit 18 as shown in FIGS. 3 (2) to 3 (4). FIG. 3 (2) and FIG. 3 (3)
Is stored in the memory 28, and at the stage when the image data corresponding to FIG. 3D is output from the A / D converter 24, the memory control circuit 30 3 (1) to 3 (3) are read out and supplied to the color processing circuit 32. As a result, the four complementary color data at the same subject position are input to the color processing circuit 32 at the same time.
2 calculates RGB data from the complementary color data. The obtained RGB image data (that is, frame 1) is temporarily stored in the memory 34.

The encoding circuit 38 compresses and encodes the image data of the frame 1 (using the output data of the color processing circuit 32 as it is or reading it out from the memory 34) by a lossless encoding method. Send out. In consideration of the fact that the image data of frame 1 is used on the receiving side to restore the image data of frame 2 and thereafter, the reversible method is ideal, but if the image quality of the image is small, An irreversible coding method may be used.

The transmitting side further generates image data of frame 2 by half-pixel shift processing and subsequent one-pixel shift processing as shown in FIG. The processing from the image sensor 20 to the color processing circuit 32 is the same as that of the frame 1. The RGB data of frame 1 output from the color processing circuit 32 is supplied to an encoding circuit 38. The encoding circuit 38 reads the image data of frame 1 from the memory 34, calculates the difference between the image data of frame 2 from the color processing circuit 32, encodes the difference image data by a lossless encoding method, and Send out. At this time, if the signal deterioration is small, the irreversible coding method may be used.

Similarly, the image data of frame 3 and frame 4 described in FIGS. 5 and 6 are encoded as difference image data between frame 1 and transmitted to communication line 40.

On the receiving side, the decoding circuit 50 receives the compressed image data of the frames 1, 2, 3 and 4 from the communication line 40, decompresses them, and restores the restored frames 1, 2, 3 and 4
Are applied to the memory 52. Since the image data of frame 1 is stored in the memory 52 as it is, and the image data of frames 2, 3, and 4 are differential image data, the image data of frame 1 previously stored in the memory 52 is added. Are stored in the corresponding storage locations.

The image data stored in the memory 52 is read out as appropriate and written into the frame buffer 56.
The image data stored in the frame buffer 56 is displayed on the screen of the monitor 58 as an image. At the stage when the received image data of frame 1 is written to the frame buffer 56, the resolution of the image displayed on the screen of the monitor 58 is the resolution corresponding to frame 1. Frame 2,
As the image data is received as 3 and 4 and the image data is written into the frame buffer 56, the resolution of the image displayed on the screen of the monitor 58 improves.

Since the display of the composite image of frames 1 to 4 is the final purpose, the display of the image may be stopped until the reception of frame 4. However, when the user looks directly at the screen of monitor 58, In this case, displaying an image whose resolution gradually increases gives a more favorable impression to the user than stopping the video display until the last data (frame 4) is received.

FIG. 7 is a conceptual diagram showing a method of transmitting frames 1 to 4 according to this embodiment. As shown in FIG. 7, in the present embodiment, frame 1 is compressed and coded only in frame 1 and transmitted, and is expanded on the receiving side. For frames 2, 3, and 4, the difference from frame 1 is encoded and transmitted, and the receiving side adds frame 1 to restore the original image data. Each pixel data of the frames 1 to 4 restored on the receiving side is written to the imaging position of each frame, that is, the storage position on the frame buffer 56 corresponding to the subject position. Specifically, the image data of frame 1 is synthesized by shifting the image data of frame 1 by a half pixel pitch in the horizontal direction, and the image data of frame 3 is shifted by a half pixel pitch in both the horizontal and vertical directions. Then, the image data of the frame 4 is synthesized by being shifted by a half pixel pitch in the vertical direction.

As a result, a high-resolution image twice as high in the horizontal and vertical directions as the resolution of the frame 1 is displayed on the screen of the monitor 58. In other words, each of the frames 1 to 4 is obtained by subsampling the final high-resolution image into quarters. Since the frame 1 itself has a resolution in which each complementary color of the complementary color mosaic filter of the image sensor 20 is regarded as one pixel, the image finally displayed on the monitor 58 has a higher resolution than that normally obtained by the image sensor 20. The resolution is much higher. Since the difference values are encoded for frames 2 to 4, the amount of transmission codes can be significantly reduced.

Instead of using frame 1 as a reference frame, frames 2 to 4 may be differentially coded using the immediately preceding frame as a reference frame. FIG. 8 shows frames 1-4 in that case.
FIG. For differential encoding using the immediately preceding frame as a reference frame, the frame stored in the memory 34 may be updated for each frame. For example, when the color processing circuit 32 outputs the image data of the frame 2, the memory 34 outputs the stored image data of the frame 1 to the encoding circuit 38, and outputs the image data of the frame 2 output from the color processing circuit 32. Will be remembered.
Similarly, the contents stored in the memory 34 are updated for the frames 3 and 4. If the memory 34 has a storage capacity for two frames, it can be easily realized.

Also in this case, the frame 1 is compressed and coded only in the frame 1 and transmitted, and is expanded on the receiving side.
Frame 2 is differentially encoded based on frame 1 and transmitted, frame 3 is differentially encoded based on frame 2 and transmitted, and frame 4 is differentially encoded based on frame 3 and transmitted. Each of the differentially coded data is restored to the original image data by adding the restored data of the reference frame on the receiving side.

The pixel data of frames 1 to 4 restored on the receiving side are stored in the frame buffer 56 corresponding to the imaging position of each frame, that is, the object position, as in the case of FIG.
Written to the upper memory location. Specifically, frame 1
The image data of frame 2 is synthesized by shifting the image data by half a pixel pitch in the horizontal direction, and the image data of frame 3 is synthesized by shifting the image data by a half pixel pitch in both the horizontal direction and the vertical direction. The image data of the frame 4 is synthesized by shifting the pitch. Thus, a high-resolution image is displayed on the screen of the monitor 58. Also in this case, since the difference value is encoded for frames 2 to 4, the amount of transmission codes can be significantly reduced.

[0035]

As can be easily understood from the above description, according to the present invention, image information is compressed and transmitted as a plurality of sub-sampling images by a pixel shifting process of an image pickup system. It is possible to grasp and confirm the outline of the image at an early stage.

[Brief description of the drawings]

FIG. 1 is a schematic block diagram of a transmitting side according to an embodiment of the present invention.

FIG. 2 is a schematic block diagram of a receiving side according to an embodiment of the present invention.

FIG. 3 is an explanatory diagram of a one-pixel shifting process.

FIG. 4 is an explanatory diagram of a one-pixel shift process after a right half-pixel shift.

FIG. 5 is an explanatory diagram of one-pixel shift processing after lower-right half-pixel shift.

FIG. 6 is an explanatory diagram of one pixel shift processing after lower half pixel shift.

FIG. 7 is a schematic functional block diagram when frames 2 to 4 are transmitted as a difference from frame 1.

FIG. 8 is a schematic functional block diagram when frames 2 to 4 are transmitted as differences from previous frames 1 to 3;

[Explanation of symbols]

 10: Photographing lens 12: Aperture 14: Optical axis shifter 16: Drive circuit 18: Control circuit 20: Image sensor 22: S / H / AGC circuit 24: A / D converter 26: Timing generation circuit 28: Memory 30: Memory Control circuit 32: Color processing circuit 34: Memory 36: Memory control circuit 38: Encoding circuit 40: Communication line 42: Synchronous signal generation circuit 44: System control circuit 50: Decoding circuit 52: Memory 54: Memory control circuit 56: Frame buffer 58: Monitor 60: System control circuit

Claims (11)

[Claims] 1. An image pickup device comprising a pixel shift mechanism, wherein a plurality of image data of the same subject image can be obtained by the pixel shift mechanism, and a plurality of images obtained by the image pickup means using the pixel shift mechanism. An image processing apparatus comprising: an encoding unit that encodes data; and a transmission unit that transmits a plurality of image data encoded by the encoding unit via a transmission medium. 2. The imaging device according to claim 1, wherein the first frame image is processed by a predetermined number of n-pixel shifts, and the first frame image is shifted by m pixels in a predetermined direction from a reference position of the first frame image. The image processing apparatus according to claim 1, wherein a second frame image is captured by pixel shift. 3. The image processing apparatus according to claim 2, wherein the encoding unit encodes the second frame image as a difference from the first frame image. 4. The image processing apparatus according to claim 1, further comprising decoding means for decoding the encoded data from the transmission medium. 5. An input means for capturing a plurality of images of the same subject using a pixel shifting mechanism and inputting a plurality of encoded image data, and a decoding means for decoding the plurality of image data input by the input means. An image processing apparatus comprising: 6. A plurality of image data of the same subject image obtained by using a pixel shifting mechanism of an imaging unit, encoding the plurality of input image data, and encoding the plurality of encoded image data. An image processing method characterized by transmitting via a transmission medium. 7. An image processing method comprising: capturing a plurality of images of the same subject using a pixel shifting mechanism; inputting a plurality of encoded image data; and decoding the input plurality of image data. 8. An image processing method comprising encoding an original image as a plurality of frame images constituting the original image, transmitting the encoded image, and reconstructing the original image on a receiving side. 9. The image processing method according to claim 8, wherein, out of the plurality of frame images, the second and subsequent frame images are differentially encoded with the previous frame image and transmitted. 10. The apparatus according to claim 8, wherein each time the receiving side receives each frame, the receiving side reconstructs an original image by combining one or more previously received frames. The image processing method described in the above. 11. The image processing apparatus according to claim 8, wherein the plurality of frame images are formed by a pixel shifting process of an imaging unit that converts an optical image into an electric signal.
The image processing method according to the item.