JP2010268343A - Imaging apparatus and imaging method - Google Patents

Abstract

A compact photographing apparatus capable of efficiently assembling with high accuracy, capable of photographing a moving image of an opposite direction with a large angle of view and a small distortion, and providing the image without a sense of incongruity. Propose.
One imaging device that converts an object into an electrical signal at a predetermined period and a second imaging optical system having a focal length different from that of the first imaging optical system and the first imaging optical system. Imaging means for forming subject images from different directions in a predetermined region on one image sensor, moving image generation means for generating a first moving image based on an electrical signal output from the image sensor, The captured image dividing means for dividing the first moving image generated by the moving image generating means into partial images corresponding to the respective areas, and the partial images divided by the captured image dividing means are combined in a predetermined arrangement and second. Image synthesizing means for generating the moving image, and display control means for causing the display device to display the second moving image synthesized by the image synthesizing means.
[Selection] Figure 7

Description

  The present invention relates to an image capturing apparatus and an image capturing method that capture an image of a subject in at least two directions with a single image sensor, have a large angle of view, have little distortion, and are easily compact.

  In recent years, various sensing systems have been proposed and implemented by active safety (preventive safety) in the automobile industry. For example, many cameras are installed and realized, such as inter-vehicle sensors, white line detection, parking assistance, back monitor, face orientation detection system to prevent accidents by looking away, photographing the culprit at the time of vehicle theft, etc. Yes. In commercial vehicles, camera systems such as drive recorders are becoming widespread.

  In addition, in a camera that captures images in a plurality of directions on a single image sensor, a system that captures left and right blind corners (see, for example, Patent Document 1), or the same image with a slight shift in parallax for distance measurement. A stereo adapter to be acquired has been proposed (see, for example, Patent Document 2).

JP 2004-341509 A JP 2004-258266 A

  However, since the systems as described above are configured independently, when trying to realize each of the effective systems, a camera is individually prepared for each purpose, ensuring the installation location, There was a problem in terms of cost increase. Therefore, the current situation is that each function cannot be used in a complex manner.

Further, in this type of conventional optical imaging unit, it is difficult to shorten the position of the light receiving surface of the image sensor and the lens due to its structure, and it is difficult to reduce the size of the system.
Furthermore, since a fisheye lens is used because it is necessary to cover a wide angle of view, the necessary angle of view is cut out and the image information after distortion correction is used, and only images with low visibility due to the fisheye lens can be obtained. Even if correction was performed, an image far from the actual situation was visible due to lack of surrounding information. There is also a problem that the subsequent image processing due to the correction may be complicated.

  In particular, white line detection cameras, inter-vehicle sensors, and drive recorders that are usually installed in the vehicle compartment are facing the front of the vehicle, and a camera that captures driver images of stolen vehicles is used in the vehicle interior. Therefore, it was difficult to fit each camera in the same housing.

  The present invention has been made in view of the above-described circumstances, and can be efficiently assembled with high accuracy by simplifying the structure. The image in the opposite direction has a large angle of view and a small distortion. It is an object of the present invention to propose a compact photographing apparatus that can take a moving image and provides the image without a sense of incongruity.

The present invention employs the following configuration in order to solve the above problems.
That is, according to one aspect of the present invention, the imaging apparatus of the present invention includes one imaging device that converts a subject into an electrical signal at a predetermined period, the first imaging optical system, and the first imaging optical. An imaging means for forming a subject image from a different direction in a second imaging optical system having a focal length different from that of the system in a predetermined region on the one image sensor, and an electric signal output from the image sensor A moving image generating means for generating a first moving image based on the image, and a captured image dividing means for dividing the first moving image generated by the moving image generating means into partial images corresponding to the respective areas; Image synthesizing means for synthesizing the partial images divided by the photographed image dividing means in a predetermined arrangement to generate a second moving image, and a display device for displaying the second moving image synthesized by the image synthesizing means Display control means to display Characterized in that it has a.

In the photographing apparatus of the present invention, it is preferable that the first imaging optical system and the second imaging optical system are configured by free-form surface prisms.
In the photographing apparatus of the present invention, when the photographing apparatus is disposed at a predetermined position in the vehicle, the first imaging optical system is focused on a position separated by a predetermined distance in the traveling direction of the vehicle. It is desirable that the second imaging optical system is configured to focus on the driver of the vehicle.

  In the photographing apparatus of the present invention, the first imaging optical system has a first spectral filter having spectral characteristics that transmit light in a visible light region, and the second imaging optical system is infrared. It is desirable to have a second spectral filter having spectral characteristics that transmit light in the region.

In the photographing apparatus of the present invention, it is preferable that the first spectral filter and the second spectral filter are disposed immediately before the image sensor.
According to another aspect of the present invention, the imaging method of the present invention includes a single imaging device that converts a subject into an electrical signal at a predetermined period, a first imaging optical system, and the first imaging optical. An imaging method for controlling an imaging apparatus comprising imaging means for imaging a subject image from a different direction in a second imaging optical system having a focal length different from that of the system in a predetermined region on the one image sensor The first moving image is generated based on the electrical signal output from the image sensor, the generated first moving image is divided into partial images corresponding to the regions, and the divided The partial images are synthesized in a predetermined arrangement to generate a second moving image, and the synthesized second moving image is displayed on a display device.

  According to the present invention, it is possible to shoot a moving image of a contradictory direction with a large angle of view and a small distortion, and it is possible to efficiently perform highly accurate assembly by simplifying the structure. .

It is a figure which shows the structural example of the imaging device in 1st Embodiment to which this invention is applied. It is a figure which shows the structural example of the imaging device in 2nd Embodiment to which this invention is applied. It is a figure which shows the external appearance of the imaging device in 2nd Embodiment to which this invention is applied. FIG. 4 is a diagram showing YZ planes of free-form curved prisms 2 and 3. FIG. 3 is a perspective view illustrating an outer shape of free-form curved prisms 2 and 3. 2 is a diagram illustrating an example of a subject imaged on an image sensor 1. FIG. It is a figure which shows the structure of the camera system of 2nd Embodiment to which this invention is applied. It is a flowchart which shows the flow of the process which the 1st acceleration sensor 65 performs. It is a figure which shows the functional block of the camera system of 2nd Embodiment. It is a flowchart which shows the flow of the process which the 2nd acceleration sensor 66 performs. It is a flowchart which shows the flow of the camera control process in 2nd Embodiment. It is a figure which shows the structure of a 1st frame memory. It is a figure which shows the structure of a 2nd frame memory. It is a flowchart which shows the flow of the subroutine "display setting process" of a camera control process. It is a figure which shows the example of a display. It is a flowchart which shows the flow of a process from image detection to warning generation. It is a figure which shows the example which mounted the imaging device by this invention in the vehicle.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, an outline of the present invention will be described.
The present invention has been made to solve the above-described problems, and includes two free-form surface prisms and a single image sensor that forms an image via the two free-form surface prisms. Yes. One of these two free-form surface prisms has a difference in that the focal length is relatively long and the other one has a relatively short focal length, while the second has a light transmission effect from the object side. There are three optical surfaces, one surface, a second surface having internal reflection and transmission of light, and a third surface having light reflection, and at least one of the optical surfaces having the reflection function has a lens function. In common. A frame in which the two free-form surface prisms having different characteristics are fixedly held for different fields of view on the first surface, and the light itself emitted from each free-form surface prism is placed on the frame itself. It is characterized in that a holder provided with a single image sensor that forms an image in opposite directions as a field of view and a cover body that accommodates the holder and houses the frame together with a free-form curved prism are combined.

  In the photographic optical unit configured as described above, the free-form surface prism includes a first free-form surface prism whose entrance surface is directed to the front view with reference to the front field, and the first free-form surface prism and the entrance surface. The first free-form surface prism arranged in the opposite direction and the second free-form surface prism having different characteristics in focal length and horizontal / vertical transmission range.

  The free-form surface prism indispensable for constituting the present invention refers to an optical element having a lens function by fusing a free-form lens surface and a prism that are not rotationally symmetric. For example, when images are captured in a single image sensor using the first and second free-form surface prisms, the front image of the vehicle is rotated 90 ° to the left in the longitudinal direction of the image sensor using the first free-form surface prism. An image (for example, 480 pixels × 480 pixels when an image sensor of 750 pixels × 480 pixels is used) is connected to the position 2/3 on the left side of the image sensor, and a second free-form surface prism is used in the longitudinal direction of the image sensor An image obtained by rotating the driver's image 90 ° to the right (for example, the remaining 270 pixels × 480 pixels) is connected to the remaining 1/3 of the right side of the image sensor.

  Then, two images connected to the image sensor are generated separately, and a moving image is displayed in a predetermined arrangement. For example, the display method includes display means for displaying the two screens side by side simultaneously in order to display the two front images and the driver's image.

  In addition, in order to display either the front image or the driver image of the two front images and the driver image, the display device has display means for selectively enlarging and displaying at least one screen without reducing the two screens.

Of the two acquired images, the front video is used for functions such as white line detection, a drive recorder, and an inter-vehicle sensor, and the driver's image is used for doze detection and video theft of the criminals. In order to realize this function, it has white line detection calculation function, moving image recording memory, inter-vehicle distance calculation function, snooze detection, warning function, and detection means for detecting fraudulent acts when the car is fraudulent. Yes. In the display method, a display format that is selected by the operator is fixedly adopted.
(First embodiment)
FIG. 1 is a diagram illustrating a configuration example of a photographing apparatus according to a first embodiment to which the present invention is applied.

  In FIG. 1, the imaging apparatus includes a first imaging optical system L100 using a coaxial spherical lens, a second imaging optical system L101 using a coaxial spherical lens, and a predetermined period. One imaging device 1 for converting a subject into an electrical signal is provided, and two screens in two opposite directions are photographed.

  The imaging optical system L100 includes imaging lenses L1, L2, L3, and L4. For example, the imaging optical system L100 is arranged so as to form a subject image in front of the vehicle on the image sensor 1. Further, the imaging optical system L101 is provided with reflecting members L5 and L8 behind the same imaging lenses L1, L2, L3, and L4 as the imaging optical system L100, so that the optical axis is 180 ° (90 °). ° × 2) Inverted and forms, for example, an image of the driver's face or the like on the image sensor 1 via relay lenses L6 and L7 using a relay optical system and a field lens.

The first imaging optical system L100 and the second imaging optical system L101 have different focal lengths, and subject images from different directions are placed in predetermined areas on the one image sensor 1. Make an image. For example, by disposing the distance between the imaging lens L4 included in the imaging optical system L101 and the reflecting member L5 to be shorter than the distance between the imaging lens L4 included in the imaging optical system L100 and the imaging element 1, The focal distance on the first imaging optical system L100 side that images the front of the vehicle is set far away, and the focal distance on the second imaging optical system L101 side that images the driver is short and has a wide angle of view. Install to be.
(Second Embodiment)
FIG. 2 is a diagram illustrating a configuration example of a photographing apparatus according to the second embodiment to which the present invention is applied, and FIG. 3 is a diagram illustrating an appearance of the photographing apparatus according to the second embodiment to which the present invention is applied. 4 is a diagram showing the YZ plane of the free-form surface prisms 2 and 3, FIG. 5 is a perspective view showing the outer shape of the free-form surface prisms 2 and 3, and FIG. 2 is a diagram illustrating an example of a subject imaged in FIG.

The free-form surface prism 2 (hereinafter also referred to as a first optical element) is a free-form surface prism that enables photographing within a range of an angle θ ₁ = 40 degrees with the front visual field facing slightly in front to the bottom. Yes (see FIG. 2). The first optical element 2 has a first surface (incident surface) 2a having a light transmitting action, as shown in FIGS. 4A and 4B and FIGS. There are three optical surfaces, a second surface (outgoing surface) 2b having a reflection and transmission action and a third surface 2c having a light reflection action, and the second surface 2b and the third surface 2c have a lens function. ing.

The free-form surface prism 3 (hereinafter sometimes referred to as a second optical element) has substantially the same external shape except for the size of the first optical element 2 and the difference in curvature between the second surface and the third surface. (See FIGS. 4 and 5). The second optical element 3 is configured such that the first optical element 2 is the axis of the imaging surface of the imaging element 1 in FIG. 4 so that the first surface, which is the incident surface, faces in the opposite direction to the first optical element 2. The rear visual field can be photographed in an angle range of θ ₂ = 60 degrees from the rear to the lower side.

The first and second optical elements 2 and 3 used in the second embodiment are free-form surface prisms of the same medium, and are composed of a medium having a refractive index larger than 1.31 times. The optical surface is also composed of a free-form surface having the YZ plane as the only symmetry plane, and is configured to have a rotationally asymmetric surface shape that gives power to the light beam and corrects aberrations caused by decentration, and has three surfaces. Among them, the first surface and the third surface may be rotationally asymmetric surfaces. Here, the YZ plane is a plane as shown in FIG. 4 (here, the directions of all the prism optical elements are the same), and the first and second prism optical elements 2, By using 3, an imaging lens or the like is unnecessary, and a high-definition image can be taken with a wide angle of view. In FIG. 4, l ₁ is the axial principal ray, and 0 is the center of the stop.

  In the photographing unit configured as described above, a light image with a visual field of 40 degrees forward focused by the first optical element 2 is formed on the image sensor 1 and is focused on the rear 60 through the second optical element 3. A light image having a field of view of 2 degrees is formed in parallel with the light image of the front field of view in the image sensor 1. That is, two front and rear visual fields are imaged in parallel in the single image sensor 1 (see FIG. 6).

  In a normal imaging device 1 such as a general camera, the aspect ratio of the screen is 4 to 3, and the long 4 direction of the screen size faces the subject in the horizontal direction, and the short 3 direction faces the subject in the vertical direction. However, in the second embodiment of the present invention, as shown in FIG. 6, the vertical direction of the image divided into two is arranged in the direction of 4 with the long screen size and the short direction of 3 is divided into two. It is comprised so that the horizontal direction of may be imaged. That is, the image 51a illustrated in FIG. 6 is an image captured in the vehicle front direction by the first optical element 2, and the image 51b is an image captured in the driver direction by the second optical element 3. .

  For example, in the image shown in FIG. 6, when 750 pixels × 480 pixels of the image sensor 1 is used, an image of 480 pixels × 480 pixels is formed at the position 2/3 on the left side of the image sensor 1, and the second optical An example is shown in which the remaining 270 pixels × 480 pixels of the image obtained by rotating the driver's image 90 ° to the right in the longitudinal direction of the image sensor 1 is formed at the position of the remaining right third of the image sensor 1 by the element 3. Yes. The pixel ratio of the first and second optical elements 2 and 3 is not fixed.

As will be described in detail later, the near-infrared light emitting LED 4 shown in FIG. 3 lights up according to a threshold value for luminance calculation in order to compensate for a lack of luminance at night.
In the front (road side) shooting, a spectral filter 24 that transmits only the visible light region is provided so that light in the near infrared region is not incident. Further, in the rear (driver's side) photographing, the spectral filter 25 that allows the light to enter the light emitting region of the near-infrared LED 4 is installed in order to enable the above-described night photographing. .

  In FIG. 1, the spectral filter 24 is installed between the image sensor 1 and the first optical element 2, and the spectral filter 25 is installed between the image sensor 1 and the second optical element 3. Show.

The covers 26 and 27 attached to the front surface of the optical element 1 can also serve as the spectral filters 24 and 25.
In the second embodiment, the imaging unit that forms an image of the field of view 40 degrees forward and 60 degrees backward has been described as an example. However, the range of the field of view that can be photographed is that of the free-form curved prisms 2 and 3 themselves. The field of view of the free-form surface prism 2 is not limited to 40 degrees and the field of view of the free-form surface prism 3 is not limited to 60 degrees.

FIG. 7 is a diagram showing a configuration of a camera system according to the second embodiment to which the present invention is applied.
In FIG. 7, the camera system includes a camera unit 52 that is an imaging device, a camera controller 71, and a driving controller 72.

  The camera unit 52 includes a TG / SSG unit 60, a CDS / AGC / A / D unit 61, a digital signal processing unit 62, and a serial I / F 63 in addition to the free-form curved prisms 2 and 3 and the image sensor 1. , A communication I / F 74, an LED driver 70, and a near-infrared light emitting LED 4.

The camera controller 71 includes a CPU 53, an operation input unit 54, an image processing MPU 57, a memory 58, a communication I / F 75, a communication I / F 77, and a first alarm device 73.
The driving controller 72 includes a display 55, a display controller 56, a second alarm device 64, a first acceleration sensor 65, a second acceleration sensor 66, a driver monitor system 67, an external memory 68, a GPS / navigation 69, A communication I / F 78 is connected via a bus 79.

Note that settings such as white balance, exposure correction, and shutter speed of the captured image are the same as those of a general on-vehicle camera, and details thereof are omitted.
The camera unit 52 can capture moving images and still images, and the CPU 53 of the camera controller 71 and the image processing MPU 57 are connected to the camera unit 52 via communication I / Fs 74 and 75. The image processing MPU 57 performs processing such as image inversion, rotation, mirror image, reduction, enlargement, distortion correction, and image pasting. Then, the image processed by the image processing MPU 57 is output to the display 55 used as a monitor device via the communication I / Fs 77 and 78 and the display controller 56.

  The camera unit 52 generates a captured image as shown in FIG. 6 from the signal output from the image sensor 1, and the camera controller 71 generates a display image from the captured image. The display image generated by the camera controller 71 is output to the display 55 of the driving controller 72.

  The TG / SSG unit 60 of the camera unit 52 generates an imaging timing pulse and drives the image sensor 1. The CDS / AGC / AD 61 of the camera unit 52 performs correlated double sampling on the image signal output from the image sensor 1, adjusts the gain according to the signal amount, performs A / D conversion, and outputs it to the digital signal processing unit 62. To do. The digital signal processing unit 62 performs processing such as color signal interpolation, color correction, and white balance on the input digital image signal, and outputs color image data generated as captured image data to the camera controller 71.

The serial I / F 63 controls each part of the camera unit 52 in accordance with commands such as exposure setting and gain adjustment from the CPU 53.
The photographed image data input to the camera controller 71 is displayed on the display 55 via the display controller 56 after performing image processing to be described later based on an image display method instruction from the CPU 53.

The image data transfer to the display controller 56 is an example using an in-vehicle LAN in FIG.
Here, each device connected to the in-vehicle LAN will be described.

  The GPS / navigator 69 processes the map information captured by the CPU 53 through the in-vehicle LAN in the car navigation system based on the GPS positioning information by the image processing MPU 57 according to the image display format, and displays the image on the display 55.

  The driver monitor system 67 has recently been used to detect the direction in which the driver's head is facing and to detect dozing or looking away. In the second embodiment, the second optical element is used. 3, information on the direction of the driver's head imaged on the image sensor 1 is obtained, and depending on the direction of the head, any (handle) from the first alarm device 73 and the second alarm device 64 to the driver is obtained. The information is transmitted, and the image processed by the image processing MPU 57 is displayed on the display 55 in accordance with the image display format.

Here, operations of the LED driver 70 and the near-infrared light emitting LED 4 provided in the camera unit 52 will be described.
The driver's photographed image formed on the image sensor 1 through the second optical element 3 is signal-processed, and the average brightness obtained is calculated, so that the brightness around the driver is the brightness necessary for image recognition. The CPU 60 determines whether or not. Then, if necessary, the LED driver 70 is driven to turn on the near-infrared light emitting LED 4 to illuminate the driver.

  In recent years, a system called a drive recorder that records the vehicle and the surrounding situation from the viewpoint (view) of the driver has been utilized mainly for business vehicles. When receiving an impact such as an accident, sudden braking, or sudden steering wheel, images for several seconds before and after the impact are stored in the external memory 68.

  In the second embodiment, when the first acceleration sensor 65 detects that a collision or a large vibration has been applied, it outputs to the CPU 60 that an abnormal signal has been detected. Then, the CPU 60 determines the threshold of the impact from the abnormal signal, and the viewpoint of the driver imaged on the image sensor 1 through the first optical element 2 for a few seconds before and after the moment when the impact is applied ( The recorded images of the vehicle and the surrounding situation are recorded and held in the memory 56 from the field of view.

  The second acceleration sensor 66 is disposed at a position where it can be detected that a window glass has collided or a large vibration has been applied, and outputs to the CPU 60 that an abnormal signal has been detected. Then, the CPU 60 judges the threshold of impact from the abnormal signal, and the driver and the driver's surroundings imaged on the image sensor 1 through the second optical element 3 for an arbitrary several seconds from the moment the impact is applied. Are recorded and stored in the memory 56.

  In addition, if the first acceleration sensor 65 described above is used, the inclination of the entire vehicle can be detected. An inclination at the time of theft of the vehicle is detected, an impact threshold is determined from the abnormal signal, and an image is formed on the image sensor 1 through the first and second optical elements 2 and 3 for an arbitrary number of seconds from the moment the impact is applied. In addition, from the viewpoint (field of view) of the driver, the situation of the vehicle and the surroundings, and the captured images of the driver and the surroundings of the driver are recorded and held in the external memory 68.

Here, the flow from the detection of impact by the first acceleration sensor 65 and the second acceleration sensor 66 to the image recording will be described.
FIG. 8 is a flowchart showing a flow of processing executed by the first acceleration sensor 65, and FIG. 9 is a diagram showing functional blocks of the camera system according to the second embodiment.

  First, in step S901, when camera moving image shooting is being performed, an image is read from the second frame memory 707 according to a predetermined address value (forward direction of the car), and the image format unit 920 reads a predetermined image format (for example, A still image such as JPEG or TIFF, or a moving image such as Motion-JPEG) is recorded in the external memory 930. In this recording, a process of overwriting an image of a predetermined period in the external memory 930 is repeatedly executed.

  Next, in step S902, the first acceleration sensor 65 detects an impact, and in step S903, it is determined whether the impact (a) is equal to or greater than a predetermined threshold (G).

  In step S904, if the impact signal detected in step S902 is greater than or equal to a certain threshold (G) (step S903: YES), the captured image is stored in step S904 without performing new recording. .

  That is, when the first acceleration sensor 900 is activated and the threshold detection unit 910 determines that the threshold is exceeded, an arbitrary number of seconds are stored from the moment the impact is applied, and a new video is not overwritten.

FIG. 10 is a flowchart showing a flow of processing executed by the second acceleration sensor 66.
Since the second acceleration sensor 970 does not normally operate during parking, the second acceleration sensor 970 detects an impact in step S1101, and if the second acceleration sensor 970 exceeds the threshold of the threshold detection unit 910, the camera Start shooting. Then, an image is read out from the second frame memory 707 according to a predetermined address value (driver side), and the image format unit 920 performs row format conversion with a predetermined image format (jpg, tif, etc.), and the external memory 930 To record. In this recording, video is recorded and stored as much as possible in the recording capacity of the external memory 930.

  In other words, if the impact signal detected in step S1101 exceeds a certain threshold (G) in the next step S1102, the process moves to the next step S1103, and the camera starts shooting after the power is turned on.

  Then, in step S1104, the exposure amount of the photographic subject is calculated and determined in step S1105 so that an appropriate amount of light can be obtained. If the amount of light is insufficient, the near infrared light emitting LED 4 is turned on in the next step S1106. Make up for the lack of light.

In step S1107, the video is recorded and stored as much as possible in the recording capacity of the external memory 930.
Returning to the description of FIG.

  The GPS / navigator 69 is a car navigation system, sends position / map information from this system to the display controller 56, and displays it on the display 55 together with the image processed by the image processing MPU 57 in accordance with the image display format.

Next, image processing / processing for displaying the acquired image on the display 55 will be described.
FIG. 6 described above represents an image formed on the image sensor 1 through the first and second optical elements 2 and 3. If the image sensor 1 is assumed to be 750 pixels × 480 pixels, FIG. As shown in FIG. 6, the image is divided into two parts of 480 pixels × 480 pixels and 270 pixels × 480 pixels.

  On the other hand, when the screen divided into two is displayed on the display 55 with 270 pixels in the vertical direction, 480 pixels in the horizontal direction, and 480 pixels in the vertical direction, there is no problem in the vertical direction, but 960 pixels are required in the horizontal direction. For example, two screens are not lined up on the display 55 having a size of WVGA (750 horizontal pixels × 480 vertical pixels) which is the mainstream.

Therefore, in the image processing MPU 57, the display 55 has a function of performing pixel thinning, reduction, enlargement, and cut of an image according to various display methods described later.
Next, the flow from image capture, image processing, and image display after image processing will be described.

  FIG. 11 is a flowchart showing the flow of camera control processing in the second embodiment, FIG. 12 is a diagram showing the configuration of the first frame memory, and FIG. 13 is the configuration of the second frame memory. FIG.

  First, in step S701, the imaging unit 701 in FIG. 9 captures an image, and in the next step S702, an image signal obtained by imaging is converted into a digital image signal by an ADC (A / D converter) 702. . In step S703, the digital image signal is subjected to processing such as color signal interpolation, color correction, white balance, and gamma correction by the image generation unit 703, and then in the next step S704, the first frame memory 704 is stored. Stored as color photographed image data.

At this time, the image stored in the first frame memory 704 stores the same color image data as the image formed on the image sensor 1.
The camera unit 52 includes an exposure detection unit 717 and an exposure controller 718 as appropriate, similar to a general video camera, a digital still camera, and the like, and shoots according to the exposure value calculated from the luminance information from the image sensor 1. The exposure is adjusted automatically.

  In step S705, the captured image data stored in the first frame memory 704 is read by the read / write controller 705 with reference to the address conversion table stored in the address memory 706, and in step S706. Are stored in the second frame memory 707. In the example shown in FIGS. 12 and 13, the addresses (1, 1) to (750, 1) of the first frame memory 704, that is, the addresses (1, 1) to (480, 1) The image data read in the order of addresses (481, 1) to (750, 1) corresponding to 1B are the addresses (1, 1) to (1, 480) of the second frame memory 707, respectively. , And then from addresses (481, 1) to (481, 211). That is, the 1A line of FIG. 12 is written to 1A of FIG. 13, and the 1B line of FIG. 12 is written to the 1B line of FIG. Such processing is performed on the entire captured image data stored in the first frame memory 704.

  Then, the “display setting process” subroutine of step S707 is executed. In the second embodiment, an image can be displayed on the display 55 in at least two formats as described above.

FIG. 14 is a flowchart showing the flow of a subroutine “display setting process” of the camera control process, and FIG. 15 is a view showing a display example.
(A) is an example in which an image in front of the vehicle is displayed on the left third of the screen, a driver image is displayed below the right third, and map information is displayed above it. (B) is an example in which warning information is displayed instead of the map information in (A), (C) is an example in which the driver's image is displayed in the center, and (D) is FIG. 5E is an example in which an image ahead of the vehicle is displayed in the center, FIG. 5E is an example in which an image ahead of the vehicle is enlarged in the horizontal direction and displayed in the center, and FIG. Is an example of displaying.

  First, in step S710, the setting of the monitor display format by the user is confirmed. That is, it is detected whether or not there is a switching instruction with the display mode switch 714 (see FIG. 9).

  If there is a change in the setting of the monitor display format (step S710: YES), the process proceeds to step S711, and the corresponding address pattern is set with reference to the display pattern. On the other hand, if there is no change in the setting of the monitor display format (step S710: NO), the process proceeds to the next step S712 while keeping the display pattern before the change.

  In step S712, it is determined whether the display pattern is single-screen display. Here, in the case of single screen display, the image to be displayed is reduced, and in the case of non-single screen display, the pixel is left as it is without reduction.

When the display pattern is a single screen display (step S712: YES), in step S714, the address memory 706 is referred to and an operation for reduction is executed.
In step S715, the data is read from the second frame memory 707. In step S716, the interpolation unit 709 interpolates the read image. For example, a method is also disclosed in which four pixels around a certain pixel are formed by using a linear or three-dimensional function to perform interpolation.

  Next, in step S717, the horizontal start address, horizontal end address, vertical start address, and vertical end address write address to the display 55 are added to the interpolated image, and stored in the third frame memory 710 in step S718. .

  In step S719, it is determined whether or not writing of the entire screen has been completed. If not completed (step S719: NO), the process returns to step S714 and the subsequent steps are repeated, and if completed (step S719). In step S720, display update processing is performed by the display controller 711, and a screen as shown in FIGS. 15A and 15B is displayed on the display panel 712.

On the other hand, if it is determined in step S712 of FIG. 14 that the display pattern is not a single screen display (step S712: NO), the image is displayed as it is without reducing the image to be displayed.
That is, when the display pattern is not a single screen display (step S712: NO), for example, when a screen as shown in (C) or (D) of FIG. 15 is displayed, the address memory 706 is referred to in step S722. In step S723, a predetermined address is read from the second frame memory 707.

  Next, in step S724, the horizontal start address, horizontal end address, vertical start address, and vertical end address write address to the display 55 are added to the image to be displayed, and the image is stored in the third frame memory 710 in step S725. .

  In step S726, it is determined whether or not screen writing to the third frame memory 710 is completed. If not completed (step S726: NO), the process returns to step S722 and the subsequent steps are repeated and completed. If so (step S726: YES), display update processing is performed by the display controller 711 in step S720, and a screen as shown in FIGS. 15C and 15D is displayed on the display panel 712.

Returning to the description of FIG.
The CPU 53 has a role of supervising image information and vehicle information. For example, the CPU 53 displays the result processed by the camera unit 52 on the display 55, or reads the road separation band from the information obtained by the camera unit 52. Thus, it is possible to prompt the driver to drive safely by using the system to automatically control driving, or by causing the first warning device 73 and the second warning device 64 to issue a warning. Yes.

  The first warning device 73 is composed of a voice device. For example, the voice device issues a warning to the driver by voice from a speaker or the like. The second warning device 64 is composed of a vibration device. For example, the vibration device issues a warning by vibrating the handle or the seat to the driver by the vibration of the driver seat.

Next, the flow of processing from image detection to warning generation will be described using the functional block diagram of the camera system of the second embodiment shown in FIG.
FIG. 16 is a flowchart showing the flow of processing from image detection to warning generation.

White line detection, inter-vehicle distance detection, eyeball, eyelid, head position detection, and the like described below are already disclosed techniques, and this processing also uses these known means.
For the image being shot, the image is read from the second frame memory 707 according to a predetermined address value of the read controller 708, and the write address of the horizontal / vertical start / end address to the display 55 is added to the third frame. Stored in the memory 710. Thereafter, it is sent to the display controller 711 to perform display update processing.

  First, in step S1701, the image stored in the third frame memory 710 as described above is selectively cut out by the reading controller 800 in the forward direction of the vehicle. In step S1702, the binarization unit 810 Binarization is performed, and in step S1703, the inter-vehicle / white line detection unit 820 detects inter-vehicle and white lines.

  In step S1704, the inter-vehicle distance calculation / white line approach / leaving determination unit 830 determines the inter-vehicle distance, determines the approach / leaving of the white line, and if an alarm is necessary (step S1704: YES), in step S1705, The first alarm device 840 is alerted by sound, and the second alarm device 880 is informed of the danger to the driver by applying vibration to the seat and the handle. At the same time, a signal is sent to the display controller 711 to display the danger on the display 55.

  Further, the image stored in the third frame memory 710 is selectively cut out by the read controller 800 in the step S1701, and the binarization unit 850 similarly binarizes the driver side image in the step S1702. In step S1703, the eyeball / eyelid position / face orientation detection unit 860 detects the eyeball / eyelid / face.

  Then, in step S1704, the dozing / looking away determination unit 860 determines dozing / looking away, and if an alarm is required (step S1704: YES), in step S1705, the first alarm device 840 is sounded. The second alarm device 880 is informed of the danger by applying vibration to the seat and the steering wheel. At the same time, a signal is sent to the display controller 711 to display the danger on the display 55.

FIG. 17 is a diagram showing an example in which the imaging apparatus according to the present invention is mounted on a vehicle.
As shown in FIG. 17, a camera U capable of simultaneously photographing a front having a relatively long focal length and a rear having a relatively short focal length is attached to a ceiling portion in the vicinity of the windshield of the vehicle. ) Can be used as a monitor.

  In the second embodiment, cameras that use free-form surface prisms 2 and 3 having different shapes to capture a vertical viewing angle of 40 degrees in the front view and a vertical viewing angle of 60 degrees in the rear view. However, for example, a free-form surface prism having the same shape may be used, and a stop may be disposed immediately before the spectral filter 24 or 25 of the imaging device 1 to limit the viewing angle. In that case, as shown in FIG. 6, the front field and the rear field captured by the image sensor 1 are not imaged on the entire image sensor 1, but the front field and the rear field are imaged on a part of the image sensor 1. Since it becomes an image, the address memory 706 stores in advance in the address conversion table read addresses for reading the ranges corresponding to the front visual field and the rear visual field. Then, the read / write controller 705 refers to the address conversion table and writes the images of the front visual field and the rear visual field in the second frame memory 707.

  Further, by setting the read address stored in the address conversion table stored in the address memory 706 so as to read out a desired visual field portion of the image without disposing an aperture immediately before the spectral filter 24 or 25, You may make it designate a viewing angle and the viewing angle of a back visual field.

  As described above, the embodiments of the present invention have been described with reference to the drawings. However, the imaging apparatus to which the present invention is applied is limited to the above-described embodiments and the like as long as the function is executed. It goes without saying that a single device, a system composed of a plurality of devices, an integrated device, or a system that performs processing via a network such as a LAN may be used.

  In addition, the imaging apparatus according to the present invention, for example, observes the outside of the vehicle, is used for front and rear traveling vehicles, obstacles, white line detection, etc., or detects the position and orientation of the driver in the vehicle, the passenger's face, It can also be used as a sensor to safely operate the airbag by judging the position of the face, whether it is an adult or a child, at the time of side-by-side driving, detection of dozing, or ignition of an airbag.

Further, the photographing apparatus according to the present invention can be applied to a robot, a railroad, an airplane, a ship, a surveillance camera, a camera for a teleconference system, etc. in addition to the in-vehicle camera system.
It can also be realized by a system comprising a CPU, ROM or RAM memory connected to a bus, an input device, an output device, an external recording device, a medium drive device, a portable storage medium, and a network connection device. That is, a ROM or RAM memory, an external recording device, or a portable storage medium in which a program code of software for realizing the system of each embodiment described above is recorded is supplied to the photographing device, and the computer of the photographing device stores the program code. Needless to say, this can also be achieved by reading and executing.

  In this case, the program code itself read from the portable storage medium or the like realizes the novel function of the present invention, and the portable storage medium or the like on which the program code is recorded constitutes the present invention. .

  Examples of portable storage media for supplying the program code include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, DVD-RAMs, magnetic tapes, and nonvolatile memories. Various storage media recorded through a network connection device (in other words, a communication line) such as a card, a ROM card, electronic mail or personal computer communication can be used.

  In addition, the functions of the above-described embodiments are realized by executing the program code read out on the memory by the computer, and the OS running on the computer is actually executed based on the instruction of the program code. The functions of the above-described embodiments are also realized by performing part or all of the process.

  Furthermore, a program code read from a portable storage medium or a program (data) provided by a program provider is provided in a function expansion board inserted into a computer or a function expansion unit connected to a computer. The CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are also performed by the processing. Can be realized.

  That is, the present invention is not limited to the above-described embodiments and the like, and can take various configurations or shapes without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 Image pick-up element 2 1st optical element (free-form surface prism)
2a First surface 2b (of the first optical element 2) 2b Second surface (of the first optical element 2) 2c Third surface (of the first optical element 2) 3 Second optical element (free curved prism)
3a First surface 3b (of the second optical element 3) Second surface 3c (of the second optical element 3) Third surface (of the second optical element 3) 4 Near-infrared light emitting LED
24, 25 Spectral filter 26, 27 Cover 51a, 51b Image 52 Camera unit 53 CPU
54 Operation Input Unit 55 Display 56 Display Controller 57 Image Processing MPU
58 Memory 60 TG / SSG part 61 CDS / AGC / A / D part 62 Digital signal processing part 63 Serial I / F
64 Second alarm device 65 First acceleration sensor 66 Second acceleration sensor 67 Driver monitor system 68 External memory 69 GPS / Navi 70 LED driver 71 Camera controller 72 Driving controller 73 First alarm device 74 Communication I / F
75 Communication I / F
77 Communication I / F
78 Communication I / F
79 Bus 701 Imaging unit 702 ADC (A / D converter)
703 Image generator 704 First frame memory 705 Read / write controller 706 Address memory 707 Second frame memory 708 Read controller 709 Interpolator 710 Third frame memory 711 Display controller 712 Display panel 714 Display mode switch 717 Exposure detection Unit 718 exposure controller 800 reading controller 810 binarization unit 820 inter-vehicle / white line detection unit 830 inter-vehicle calculation / white line approach / leaving determination unit 840 first alarm device 850 binarization unit 860 eyeball / eyelid position / face orientation detection unit 870 Dozing / looking away determination unit 880 Second alarm device 900 First acceleration sensor 910 Threshold detection unit 920 Image format unit 930 External memory 970 Second acceleration sensor L1, L2, L3, L4 Imaging lens 5, L8 reflecting member L6, L7 relay lens L100 first imaging optical system L101 second imaging optical system U camera

Claims (6)

One image sensor that converts a subject into an electrical signal at a predetermined period;
A subject image from a different direction is connected to a predetermined region on the one image sensor by the first imaging optical system and the second imaging optical system having a focal length different from that of the first imaging optical system. Imaging means for imaging;
A moving image generating means for generating a first moving image based on an electric signal output from the image sensor;
Captured image dividing means for dividing the first moving image generated by the moving image generating means into partial images corresponding to the respective areas;
Image synthesizing means for synthesizing the partial images divided by the photographed image dividing means in a predetermined arrangement to generate a second moving image;
Display control means for causing the display device to display the second moving image synthesized by the image synthesis means;
A photographing apparatus comprising: The first imaging optical system and the second imaging optical system are configured by free-form surface prisms,
The imaging apparatus according to claim 1, wherein: The photographing device is arranged in a vehicle,
The first imaging optical system is configured to focus on a position separated by a predetermined distance in the traveling direction of the vehicle when the imaging device is disposed at a predetermined position in the vehicle.
The second imaging optical system is configured to focus on a driver of the vehicle.
The photographing apparatus according to claim 1 or 2. The first imaging optical system has a first spectral filter having spectral characteristics that transmit light in the visible light region;
The second imaging optical system has a second spectral filter having spectral characteristics that transmit light in the infrared region.
The imaging device according to claim 3.   The imaging apparatus according to claim 4, wherein the first spectral filter and the second spectral filter are disposed immediately before the imaging element. One image sensor that converts an object into an electrical signal at a predetermined period, a first imaging optical system, and a second imaging optical system having a focal length different from that of the first imaging optical system, respectively, have different directions. An imaging device comprising an imaging means for forming an image of a subject from the image on a predetermined area on the one image sensor,
Generating a first moving image based on the electrical signal output from the image sensor;
Dividing the generated first moving image into partial images corresponding to the respective areas;
Generating the second moving image by combining the divided partial images in a predetermined arrangement;
Displaying the synthesized second moving image on a display device;
An imaging method characterized by the above.