JP2008278515A - Image processing camera system and image processing camera control method - Google Patents

Abstract

One imaging device is reliably shared among a plurality of applications that dynamically change camera parameters.
An imaging means for acquiring an image, an imaging device control unit for controlling an imaging device from a plurality of applications, and an application scheduling unit for selecting an application are provided. Means 17 for storing the required number of image data and image data capture frequency at 15N, means for selecting a plurality of applications based on the number of image data and the frequency of image data capture, and the plurality of applications from one imaging device The image capturing scheduling unit 14 determines the timing and interval of capturing image data without overlapping. An executable application is clearly indicated to the user, and an operation unit for instructing activation is displayed on the navigation screen.
[Selection] Figure 3

Description

  The present invention relates to an image processing camera system that shares one imaging device for a plurality of uses, and a control method thereof.

  Image processing technology has begun to be applied to monitoring systems that detect intrusions and abnormalities of suspicious persons and in-vehicle systems that support safe driving of vehicles. In an outdoor image processing system such as an in-vehicle system, in order to acquire images necessary for a plurality of applications from one imaging device, camera control according to the external environment is required. It is necessary to control the orientation and exposure of the camera depending on the location where the user wants to see and the time zone in which the application is used. For example, in a camera system mounted on an automobile for lane recognition, the brightness of an image obtained by a tunnel entrance / exit, sunlight from a western sun, headlights of an oncoming vehicle, and the like rapidly change. Even in such a case, the camera system must stably recognize the lane with one camera installed in the forward direction. As an exposure control technique for stably acquiring an image required by an application, for example, there are techniques disclosed in Patent Document 1, Patent Document 2, and the like.

  In order to spread the image processing technology to these systems, it is required to reduce the space and cost of the image processing apparatus. In order to realize this small space and low price, there is a technology for sharing one imaging device among a plurality of applications. For example, as disclosed in Patent Document 3, a plurality of applications can efficiently use one camera. Specifically, first, when a plurality of applications have the same camera control (exposure, viewing angle, camera direction) and a sharable video can be acquired, a plurality of applications share one camera. Also, if the difference between the parameters for camera control is within a predetermined range and the settings can be switched within a very short time when images in multiple applications are not interrupted, one camera can be shared. ing. When the difference between the parameters deviates from a predetermined range, the corresponding application is notified that photographing is not possible, and appropriate processing is executed on the corresponding application side.

JP-A-8-240833 Japanese Patent Laid-Open No. 9-181962 JP 7-46568 A

  Here, an application of a monitoring system or an in-vehicle system that is required to perform image processing at a predetermined cycle is assumed. For example, as an example of an in-vehicle system, there is a lane keeping system that recognizes a traveling lane by image processing and controls steering so that the vehicle does not protrude from the traveling lane. In this system, it is necessary to control the steering by acquiring the lane recognition result by image processing within a certain time (for example, within 200 ms). In such a system in which the image processing result is used for subsequent control, the processing cycle of the image processing must be strictly observed. If this period cannot be observed, control by image processing cannot be performed, the system will not function, and malfunctions may be caused. As such systems that require high reliability, there are monitoring systems such as intruder detection, human flow / traffic flow measurement devices, and abnormality detection devices. There are many safety support systems, various alarm systems, etc. in in-vehicle systems. To do. In cameras used in these monitoring systems and in-vehicle systems, it is necessary to recognize the outside world, and therefore camera parameters change every moment even in one application for image processing. Therefore, it is not possible to share a camera by approximating camera parameters for a period of time. For example, in the lane keeping system, when a camera parameter is accidentally approximated to another application for a certain period of time, the system is activated. However, at the next moment, the camera parameters change and the camera cannot be shared, and the lane keeping system stops. The system used for control must not have such a situation, and a reliable system cannot be obtained.

  An object of the present invention is to provide an image processing camera system capable of sharing one imaging device with a plurality of applications with high reliability.

  Another object of the present invention is to provide an easy-to-use image processing camera system in which a plurality of applications that can be executed are clearly indicated to the user and an application to be executed by the user can be selected without hesitation.

  In one aspect, the present invention includes an imaging unit that acquires an image, an imaging device control unit that receives an image acquisition request from a plurality of applications and controls the imaging device, and an application scheduling unit that selects an application to be executed. A means for selecting a plurality of applications that can be executed in parallel based on the number of image data read from the means for storing the number of required image data and the frequency of taking in the image data and the frequency of taking in the image data in the plurality of applications. And a plurality of executable applications have an image capture scheduling unit that determines the timing and interval of image data capture that repeats the capture of image data from one imaging device without overlapping in time And features.

  In another aspect, the present invention is characterized by comprising means for displaying a plurality of applications that can be executed in parallel, and operation means for a user to instruct activation of the displayed executable applications. .

  According to one aspect of the present invention, it is possible to share one imaging device with high reliability among a plurality of applications that dynamically change camera parameters.

  According to another aspect of the present invention, it is possible to provide a user-friendly image processing camera system that clearly shows a plurality of applications that can be executed to the user and can select applications to be executed without hesitation.

  Other objects and features of the present invention will become apparent from the following description of embodiments.

  FIG. 1 is a schematic block diagram of an image processing camera system mounted on an automobile according to an embodiment of the present invention and a hardware configuration diagram of the image processing camera. This image processing camera system realizes many applications as will be described later by using cameras 2 and 3 installed in the front direction and the rear direction of the automobile 1. The image processing cameras 2 and 3 are connected to the interface means 5 via the network 4. In the case of the automobile 1, the network 4 can use a standardized network such as CAN (Control Area Network) or a connection method that simply represents ON / OFF with a power line. The interface unit 5 is a device that exchanges information with a driver such as a navigation system, a steering wheel, and a brake, and a device that exchanges information with a control device such as an engine control unit, GPS, and various sensors. When the interface means 5 is a navigation system, a large amount of information is transmitted through the network 4 such as CAN. When the interface means 5 is a side brake or the like, only the necessary minimum ON / OFF information is transmitted.

  An embodiment of the hardware configuration of the image processing cameras 2 and 3 will be described with reference to FIG. An image acquired by the imaging device 6 is stored in the RAM 8 via the imaging device interface 7. A number of application programs, which will be described later, and a control program for the imaging device 6 are stored in the ROM 9 in advance, and are executed by the CPU 10 as necessary. Inside the image processing cameras 2 and 3, the external interface unit 11 mediates with an external device via the network 4. That is, these elements 7 to 11 constitute the microcomputer 12, and the ROM 9 stores a program for starting the image processing cameras 2 and 3 and information about executable applications. The RAM 8 stores information necessary for executing the application. Information necessary for executing the application includes environment information to be described later, information obtained from the interface unit 5 in FIG. 1A, image data, and the like. The imaging device 6 is controlled by a program processed by the microcomputer 12, that is, the CPU 10, and the control information is transmitted via the imaging device interface 7.

  FIG. 2 is an explanatory diagram of many applications and required image quality, rate, camera control, and image processing functions in an automobile. As shown in FIG. 2, examples of applications include the following. (1) Monitoring around the vehicle, (2) Drive recorder that records the driving situation, (3) Video album that records the surrounding video during driving, (4) Realized by recognizing the driving lane with the camera Lane departure warnings, (5) Obstacle warnings that issue warnings for running obstacles, (6) Interrupt / passing vehicle warnings, (7) Automatic light turn-off and light intensity Automatic light control function that controls the direction and direction, (8) parking assistance / lane change assistance function when parking or changing lanes, and (9) collision mitigation / avoidance function that reduces collision damage as much as possible before collision.

  The (1) image quality, (2) rate, (3) camera control, and (4) image processing function required by these many applications will be described.

  First, with regard to “(1) image quality”, it is desirable that the image quality be higher, but if the resolution is increased, the amount of data increases and processing is burdened. As a result, processing at a predetermined rate may not be possible. . Therefore, image quality suitable for the application exists. In FIG. 2, the image quality used only for the video album is specified, but other applications have suitable image quality similar to the video album. For example, the image quality required for lane recognition is only required to be able to distinguish a lane mark such as a white line, and it is not necessary to acquire an image with higher image quality than necessary by consuming time and memory.

  Next, as for “(2) rate”, there is a rate suitable for each application as in the case of image quality. Here, the rate is the frequency of processing. The higher the rate, the smaller the number of processing cycles, and the processing is repeated at short time intervals. Generally, when trying to acquire a high-quality video or control related to safety, it is necessary to acquire an image at a short interval, and the rate becomes high. For example, in applications that require high reliability and high-speed response such as collision mitigation and collision avoidance, it is necessary to increase the number of times of image processing to improve reliability and response. In this way, some applications can satisfy the function if they can be processed once per second, and some applications require processing at intervals of 16 milliseconds. Therefore, in order to share an imaging device, the rate is considered. Must.

  “(3) Camera control” is mainly divided into an application in which a person views video and an application in which a computer processes an image. The monitoring, drive recorder, and video album functions in FIG. 2 are viewing applications, and applications below automatic light control perform recognition processing. As the contents of camera control, the monitoring control only needs to acquire a natural image with a human appearance, and the image processing control performs control so that an image that can be processed by the image processing portion can be acquired. . The image for image processing includes an image acquired with a low-speed shutter for processing a dark portion of an image, an image acquired with a high-speed shutter for processing a bright portion. An image subjected to the shutter control is often different from the camera control for monitoring because it is too dark or too bright to look. Color control is also different from monitoring camera control for reproducing natural colors because an image in which red or yellow is emphasized is acquired in order to detect a signal.

  Finally, FIG. 2 shows “(4) Image processing function” required for each application. Basic image processing functions include image compression, color correction, lane recognition, vehicle detection, and the like, but some functions require the same function in a plurality of applications. If the same function is required in a plurality of applications, the camera control required by the function can be shared between the applications. For example, if a lane departure warning is required, a lane recognition function is required. Similarly, if an interrupted vehicle warning that requires a lane recognition function and a lane recognition function are shared, the use of the imaging device is also shared. can do.

  Hereinafter, an embodiment of the present invention that realizes a large number of applications by sharing one camera (imaging device) will be described.

  FIG. 3 is a functional block diagram showing an outline of an embodiment of the image processing system according to the present invention. Most of the functions shown here are executed by the microcomputer 12 described with reference to FIG. First, an imaging device control unit 13 that controls the imaging device 6 that acquires an image and an application scheduling unit 14 that controls execution and stop of an application are provided. The system includes N applications 151 to 15N, and each application operates using necessary ones of basic image processing functions (A to M) 16A to 16M. Each application includes environment information 17 used for processing and the application scheduling unit 14 used for application control. In the present embodiment, in addition to the imaging device 6 called a normal camera, the imaging device control unit 13 that controls the imaging device 6 and the part that executes each of the applications A to M are the image processing camera 2 shown in FIG. , 3 parts. Therefore, cameras (for example, cameras 2 and 3) having advanced processing functions for realizing the application described above are called image processing cameras in order to distinguish them from cameras that simply acquire images.

  First, a case where one of the applications 151 to N shown in FIG. 3 is executed alone will be described. The description of this single execution is the same as the operations of Patent Document 1 and Patent Document 2 of the prior art. The application 151 refers to environment information 17 such as ambient brightness, an image processing range, and a currently acquired image state in order to acquire an image necessary for processing from the imaging device 6. Based on this, camera parameters (camera orientation, aperture value, shutter speed, etc.) are determined, and the imaging device control unit 13 is requested to control the imaging device 6. The imaging device control unit 13 sets camera parameters desired by the application 151 in consideration of the exposure timing of the imaging device 6 and the like. Here, the environmental information 17 includes map information, date and time (season), illuminance outside the vehicle, weather, and gaze range.

  As described above, when only a single application exists, a function for the imaging device control unit 13 to set the camera parameter desired by the application in the imaging device 6 is sufficient. However, since there are a plurality of applications in FIG. 3, it is necessary to have a function of accepting camera parameters required from each application and controlling the imaging device 6 to acquire each image at a limited time interval. In this embodiment, the application scheduling unit 14 performs its function. That is, the application scheduling unit 14 has a function that is requested by the plurality of applications 1 to N and adjusts and executes control on the imaging device 6.

  Depending on the type of application, the imaging device 6 may not be shared. For example, there are an application that wants to record a beautiful image every frame in order to record video with high image quality, and an application that wants to acquire an image necessary for image processing every time as short as possible, such as collision avoidance. Executing these two applications in parallel using one imaging device is difficult unless the camera control of the two applications is exactly the same. For this reason, applications to be executed in parallel are limited, but the application scheduling unit 14 also controls the applications. The application scheduling unit 14 determines an executable application from the image control information and processing amount required for each application, and the traveling state of the automobile 1. If it can be executed, the image acquisition timing and the processing timing are adjusted. The application scheduling unit 14 can efficiently share one imaging device even in a plurality of applications that dynamically change camera parameters. As described with reference to FIG. 1, the interface unit 5 serves as an intermediary with a driver, such as a navigation system, a steering wheel, and a brake, and a travel control system of the automobile 1.

  Next, control of camera parameters of the imaging device 1 will be described with reference to FIGS.

  FIG. 4 is a functional block diagram of the imaging device control unit 13. As shown in the figure, there are the following five control function units for controlling an image obtained from the imaging device 6. First, an exposure control unit 131 that controls the amount of light incident on the image sensor for converting the amount of light into an electrical signal, a gain control unit 132 that controls brightness with the electrical signal after converting the amount of light into an electrical signal, and color information control There is a color control unit 133 to perform. The scanning range control unit 134 and the video input control unit 135 scan the screen at a high speed by limiting the data transfer range. Each control is executed by the program processing of the microcomputer 12 as described with reference to FIG. There are static parameters that can be changed in real time depending on the object of control, and dynamic parameters that are changed by mechanically controlling the device. In the latter case, it may take time to obtain the desired image. is there.

  FIG. 5 is a hardware configuration diagram of the imaging device 6. Inside the imaging device 6, the signal processing unit 61 executes static control such as gain control and color control that can change camera parameters. On the other hand, in an optical system such as the lens 62, the diaphragm 63, and the image pickup device 64, there is an optical system controller 65 that dynamically controls the focus and shutter speed. In the present embodiment, scheduling relating to these changes is also executed, and as described above, the signal processing parameters can be changed instantaneously, but the optical system often cannot be changed instantaneously.

  Next, a scheduling method will be described. As described above, each application needs to be processed at a predetermined rate. For example, in the lane departure warning, if it is necessary to issue an alarm within 300 [ms] after the departure from the lane, the processing cycle of the lane departure warning processing must be 300 [ms] or less at the longest. If the predetermined rate cannot be realized in relation to the running application, the lane departure warning process should not be started. For this reason, each application must be started and scheduled so as to realize a predetermined rate. The scheduling will be described in detail below.

  FIG. 6 is a time chart showing an example of operation scheduling of a plurality of applications, and shows scheduling when two applications of an interrupted vehicle alarm and a drive recorder share one imaging device and are executed in parallel. The video frame 300 defines the timing at which an image can be acquired, and each timing is expressed by the frame numbers of the frames F0 to F5. For example, in a normal imaging device, one frame is 33 [ms] or 16 [ms]. In the interrupt vehicle alarm 301, as shown in the camera control column of FIG. 2, an interrupted vehicle recognition (vehicle detection) process is performed using two images subjected to high-speed and low-speed shutter control. For this reason, it is necessary to acquire and process two images 1 and 2. In this figure, since the processing cycle (one cycle) of the interrupt vehicle warning 301 is 6 frames, acquisition of two images and respective image processing are performed within a period of 6 frames. On the other hand, the drive recorder 302 is a process of recording an image, but this processing cycle is also 6 frames, and one image is acquired. The application scheduling unit 14 of FIG. 3 calculates the acquisition time of the images 1 and 2 for the interrupted vehicle alarm 301 and the processing times of the processing 1 and 2. Since 6 frames are required for the interrupt vehicle alarm 301, the capture of the image 1 is assigned to the frame F0, the process 1 is assigned to the frames F1 to F2, the capture of the image 2 is assigned to the frame F3, and the process 2 is performed to the frames F4 to F5. Assign. On the other hand, the processing of the drive recorder 302 is only image acquisition, and the capture of the image 3 is assigned to the frame F1 that does not use the imaging device 6 in the interrupt vehicle alarm 301. Here, since the capture timing of images is different in both applications, the camera parameters of the images 1 to 3 can be set completely differently, and one imaging device 6 can be shared by these applications.

  FIG. 7 is a process flow diagram of application scheduling according to an embodiment of the present invention. The period of the scheduling process will be described using this. In FIG. 6 described above, the processing cycle (cycle) is 6 frames including frames F0 to F5, and this processing cycle is repeated. First, in the first step 71 of the periodically repeated processing cycle, it is confirmed whether the processing schedule being executed is updated. If the schedule has not been changed, the process proceeds to the process of the frame F0 in step 72. In step 72, first, in step 721, an acquisition command for image data to be captured within the period of frame F0 is issued. Since the capture of the image data is realized by a transfer method that does not place a load on the CPU, which is generally called DMA transfer, the processing of the CPU proceeds to the next step 722. In step 722, camera parameters to be captured within the next frame F1, for example, exposure control settings such as shutter speed are set. The setting of camera parameters is executed by the microcomputer 12 shown in FIG. 1B as a function of the imaging device control unit 13 shown in FIG. That is, the setting is made at a timing suitable for the imaging device 6 via the imaging device interface 7. When this process ends, the process proceeds to step 723. In step 723, software processing executed in the period of the frame F0 is performed. After all the processes to be executed in the frame F0 period are executed, in step 73, the next process is set to be executed in the frame F1, and the current process is terminated. Thereafter, when the time of frame F1 comes and this processing is started again by a timer interrupt, the processing in frame F1 is executed in the same manner. In this way, the processing of the frames F0 to F5 is sequentially repeated by the timer interruption.

  Even if the schedule is updated in step 71, the subsequent processing is the same as the processing content described above, but each processing timing is initialized as necessary according to the new schedule. Application of the new schedule is performed at step 71.

  FIG. 8 is another example time chart for operation scheduling of a plurality of applications, and is an explanatory diagram regarding parallel operations of the lane departure warning 310, the automatic light control 311, and the drive recorder 302. The lane departure warning 310 has a processing cycle of 6 frames, the required images are two high-speed / low-speed shutters, and the processing amount is one frame for each image, for a total of two frames. The automatic light control 311 also has six processing cycles, two images with a high-speed / low-speed shutter, and a processing amount of one frame for each image. The drive recorder 302 may record one image at regular intervals (here, 6 frames). The application execution scheduling in this case is shown in FIGS. FIG. 6A shows a case where the images 1 and 2 to which the high-speed / low-speed shutter used in the lane departure warning 310 is applied can be shared with the image used in the automatic light control 311. At this time, it is assumed that the camera control of the image processing functions of vehicle detection and lane recognition is the same. Since the same image can be used for the lane departure warning 310 and the automatic light control 311, the processing 1 of the lane departure warning 310 and the processing 3 of the automatic light control 311 are performed on the image 1, and each of the images 2 is performed. Process 2 and process 4 are executed. As a result, scheduling as shown in FIG. 3A can be executed by the function of the application scheduling unit 14 in FIG. 3, and one imaging device 6 can be shared by three applications.

  A case where the lane departure warning 310 and the automatic light control 311 cannot share each image will be described with reference to FIG. At this time, it is assumed that the camera parameter control used in each application is different and is not necessarily the same control. Since the images of the lane departure warning 310 and the automatic light control 311 cannot be shared, the automatic light control 311 needs to acquire two images 4 and 5 independently. Scheduling is performed so that the images 4 and 5 are acquired in the frames F2 and F5 that are not used by the lane departure warning 310 and the drive recorder 302, respectively. Also, the processes 3 and 4 for the automatic light control 311 are scheduled in the frames F0 and F3 in accordance with the timing of acquiring an image. Thus, these three applications can be operated in parallel while sharing one imaging device 6.

  FIG. 9 is a scheduling time chart when the video album application is operated. In the high-quality mode 312 of the video album, a process for acquiring an image every frame and compressing it in real time is required. Since image acquisition is performed every frame, the processing period (cycle) is one frame, as shown in FIG. On the other hand, in the low-quality mode 313 of the video album, it is not necessary to record the image every frame, and the video is dropped. If the processing cycle (cycle) at this time is assumed to be 6 frames, scheduling as shown in FIG. As can be seen from the figure, when the high-quality video album function is executed, there is no room for an application that newly uses the imaging device 6 to operate, and the video recorder (high-quality) 312 such as the drive recorder 302 is completely connected. Only applications that can share images can be executed in parallel.

  On the other hand, in the low-quality video album function, the image pickup device 6 can be newly controlled with another camera parameter to acquire an image, so that various applications can be selected and executed.

  FIG. 10 is a scheduling time chart of a procedure for additionally starting an application, and is a diagram for explaining a case where the automatic light control 311 is additionally started while the low-quality image album function 313 is operating. In FIG. 9A, since only the low-quality video album function 313 is operating, there are many periods when the imaging device 6 is not used. Here, when the automatic light control 311 is added, the processing cycle (cycle) is a minimum of 4 frames (2 times of image capture and 2 times of processing), and therefore, within 6 frames which are the processing cycle of the video album function 313. It will fit. If the number of cycles is 4 frames or more, the video album (low image quality) 313 and the automatic light control 311 can be executed in parallel. In this embodiment, the number of cycles of the automatic write control 311 is 6 frames.

  As a result of executing the scheduling as shown in FIG. 10B in the application scheduling unit 14 in FIG. 3, the two applications of the video album function 313 and the automatic light control 311 can be operated in parallel. At this time, the start of the processing of the added automatic light control 311 is controlled so as to start in accordance with the processing cycle of the video album. In addition, since image acquisition is performed in synchronization with a video synchronization signal in image capture, the processing cycle and start timing are all matched to the video synchronization signal.

  According to the above-described embodiment of the present invention, the imaging device 6 can be shared among applications that dynamically change camera parameters.

  FIG. 11 is an example of grouping of applications that can operate in parallel. The operable applications can be grouped in advance, and control can be performed so as to perform optimum scheduling for each group. By grouping the applications that operate in advance, the user can save the time and effort of selecting each application, and the optimal scheduling can always be performed, so that the imaging device 6 can be used efficiently. In consideration of the degree of coincidence of the basic image processing function (FIG. 2) between the applications, the grouping is determined based on whether or not the image capturing described in FIGS. In this example, groups 1 to 4 are grouped so as to include three, two, five, and two applications, respectively.

  Next, a user interface according to an embodiment of the present invention will be described. According to the present invention, by sharing one imaging device 6, a plurality of applications can be switched or processed in parallel. Therefore, a user interface is provided that allows the driver to switch the functions of the image processing cameras 2 and 3 or start a new one while driving.

  FIG. 12 is an example of a navigation screen for selecting and switching applications. A procedure when the driver selects or switches an application on the navigation screen 19 will be described. In FIG. 12, a large number of applications that can be selected are displayed on the screen 19, and the driver can select an execution application by touching the application display section of the navigation screen 19. For example, a case where high image quality of a video album is selected will be described. In FIG. 12 and subsequent screens 19, some of the many applications shown in FIG. 2 are omitted due to space limitations, but in reality, all applications are displayed.

  FIG. 13 shows the screen after selecting the high image quality of the video album. First, since the high-quality video album function 312 is selected and operating, “video album (high quality)” is brightly displayed on the screen 19 as shown by a bold line. When the video album (high quality) 312 is in operation, as described with reference to FIG. 9A, applications that can operate in parallel are greatly limited. Here, it is assumed that the monitoring function 303 can be operated in parallel as shown in the group 4 in which parallel operation is possible in FIG. Therefore, the “monitoring” display is displayed in a color or brightness indicating that selection is possible as illustrated by a solid line. The display of other applications is switched to a state display in which selection is not possible, as indicated by broken lines.

  If the driver wants to stop the video album (high image quality) 312, the application is terminated when the “reset” button unit is touched or the “video album (high image quality)” display unit is touched again. The activation or stop of the application is immediately reflected in the control of the imaging device, and the image obtained from the imaging device changes. At this time, two types of changes occur: the obtained image itself changes due to exposure control or the like, and the processing cycle changes and the timing of the output image changes. Since one imaging device 6 is shared by a plurality of applications, it is necessary to change the output data of the imaging device when the operating application changes.

  FIG. 14 is an example of a navigation screen for adding an application. When the video album (low image quality) is in operation, many other applications can be executed than when the video album (high image quality) is in operation. As shown in the group 4 capable of parallel operation in FIG. 11, in addition to the video album (low image quality) 313, automatic light control 311, interrupt vehicle warning 301, lane departure warning 310, and parking assistance 304 can be processed. In the navigation screen 19 in FIG. 14A, it is clearly shown that the video album (low image quality) is operating, and in addition, it is displayed that the above four applications can be activated. Here, a case where the driver additionally activates the automatic light control 311 will be described. In this case, it is assumed that the “light automatic control” display unit is touched on the screen of FIG.

  FIG. 14B shows a screen where the automatic light control 311 is additionally activated. As shown in the drawing, the display unit of “light automatic control” is switched to a display indicating that it is in operation, and this application is activated.

  In the navigation screen 19, it is assumed that the application selection or switching is based on an instruction from the driver, but the application can be selected or switched using information other than the driver. For example, the parking support function 304 is not selected during parking at high speed because parking is impossible. In this case, it is desirable that the selection range of the application is limited by the vehicle speed so that the parking support function cannot be selected. FIG. 14B illustrates a state where the display unit of “parking assistance” is switched to non-selectable because it is traveling at high speed. Also, even if the driver has selected parking assistance in advance, if the planned high speed is reached, similarly, the display of “parking assistance” is turned off and selection is impossible. During high-speed movement, applications such as the interrupt vehicle warning 301 and the lane departure warning 310 can be operated, and only these possible applications are displayed for selection.

  As an example of application switching, while traveling in an urban area, safety functions such as collision reduction operate preferentially, and can be operated with priority over lane departure warning and video album functions. On the other hand, in a sightseeing spot or a scenic place, the video album function can be operated with priority and other functions can be stopped. Furthermore, in addition to driver operations, information from various sensors, such as brakes, GPS, and vehicle speed, can be used to limit the selection of applications that operate in response to changes in the surrounding environment, and application operations themselves can be switched. Can be.

  As described above, if the application scheduling unit 14 in FIG. 3 executes an application, additionally starts and stops the application, the operation application can be switched from the navigation screen according to the driver's will.

  In FIGS. 12 to 14, the application switching procedure on the navigation screen 19 has been described. However, an application to be executed on a screen other than the navigation screen may be selected. For example, a button may be attached to the steering wheel to select the lane maintenance traveling function. In this case, it is not known which application is operating when the driver presses the button. Therefore, as a scheduling method, it is conceivable that the lane maintenance traveling function is set to the highest priority, and an application that does not operate in parallel with the lane maintenance traveling function is controlled to be forcibly terminated. In such a mechanism, not only buttons but also various sensors mounted on the automobile may play the role. For example, when a signal indicating that there is an obstacle is input by connecting to a distance sensor that detects an obstacle, the obstacle avoidance application may be activated with the highest priority. In this case, if the obstacle avoidance application operates while the application that has been operating is operating, the operating application may be left as it is. In addition, if it is not possible to operate in parallel, it is possible to control so that the function for safety support begins to operate by interrupting the running application and immediately starting the obstacle avoidance application. These functions are realized by the application scheduling unit 14 shown in FIG.

  FIG. 15 is a process flow diagram for performing additional activation of an application according to an embodiment of the present invention. For example, as described in FIGS. 10 and 14, it is assumed that the driver additionally activates the automatic light control 311 on the navigation screen 19 during the operation of the video album (low image quality) 313. When “light automatic control” is selected by the driver using the touch panel, in step 201, an event for adding an application is accepted from the navigation, and this process is started by an interrupt process. After receiving the event, in step 202, the program acquires information on the application being executed and the application to be added. These pieces of information are stored in the environment information database 17 of FIG. 3 and are image acquisition information including the schedule being executed, the current frame, the processing cycle of each application, and camera parameters. Further, there is grouping information for investigating which applications can be executed in parallel as described in FIG. In the scheduling process of step 203, it is confirmed whether the selected automatic light control is in the same group as the operating video album (low image quality) function. Even if there is no grouping information, whether or not the selected application is an executable application can be confirmed by referring to the application information. Whether or not execution is possible is determined based on whether or not necessary processing can be allocated within the processing cycle as described with reference to FIGS. 6 to 10 based on the number of processing cycles, the number of acquired images, and information on necessary image processing functions. Is done. After the executable application and the non-executable application are classified, the information is reflected in the application information. If it can be executed, the schedule is updated using the application information and scheduling information in the environment information 17. In other words, the schedule is updated so that the application for which additional activation is requested and the application in operation are repeatedly fetched of image data from one imaging device without overlapping in time. In step 204, the application that can be added is analyzed again with the updated new schedule, and the driver is notified or prepared for the next application addition event, and then the process ends in step 205.

  This additional activation processing of the application is executed by interrupt processing upon detection of an event occurrence, so it is not known at which step of the flow described in FIG. 7 the interrupt processing is performed. In order to ensure the consistency of information being processed, in this embodiment, the schedule is updated in step 71 of FIG.

  The image processing camera system according to this embodiment is summarized as follows. First, a step (202) of selecting a plurality of applications that can be executed in parallel is provided. Next, a plurality of executable applications includes a scheduling step (203) for determining an image data capturing timing and interval for repeatedly capturing image data from one imaging device without overlapping in time. The scheduling step (203) includes a step of determining timing including processing using the captured image data in each application. Further, the method includes a step of reading out the number of image data necessary for a plurality of applications and the necessary capture frequency from the storage means (17) storing them. Then, based on the number of read image data and the capture frequency, a plurality of executable applications includes a step of determining an image data capture timing and interval at which the capture of the image data from one imaging device is repeated.

  FIG. 16 is an example of a screen for executing addition of a new application and deletion of an existing application. The application can specify information necessary for the operation such as a processing cycle (cycle), an image processing function, an image to be used, a processing amount, and add the information to the menu. This can be selected or deleted from the screen as in the conventional application. In FIG. 16, in order to add a new application, a touch panel unit 191 to be downloaded and a touch panel unit 192 to delete an existing application are displayed.

  FIG. 17 is a processing flowchart for downloading a new application. The download process will be described using the screen of FIG. New applications are obtained from recording media such as the Internet and CompactFlash (registered trademark). If the user chooses to select a new application from the download, this process is started from step 171. In step 172, the downloadable application is presented as a menu to the user. In step 173, the user selects a required new application from the menu. Under this result, when selection of a new application to be added in step 174 is completed, in step 175, analysis of applications that can operate in parallel is executed. As described above, the combination of applications that can be executed in parallel is determined from the number of processing cycles, the number of acquired images, a necessary image processing function, and the like. Based on this result, the existing grouping shown in FIG. 11 is updated. In step 176, after confirming combinations in all groups, the information is stored as application information. In step 177, the download process is terminated.

  FIG. 18 is a process flow diagram of operation scheduling including a new application. This scheduling process is a process performed in the scheduling of step 203 in FIG. First, in step 181, the scheduling process is activated, and in step 182, information on the application being executed and the application to be added is acquired. In step 183, the application having the shortest processing cycle is mapped on the schedule. At this time, mapping is performed with a margin as much as possible so that other applications can operate. For example, mapping is not performed in which an application that needs to be completed within 6 frames is processed in 3 frames. After mapping the application having the shortest processing cycle number, in step 184, the application having the next shortest processing cycle number is mapped. Here, if the mapping fails, that is, if the imaging device 6 cannot be used or the time for the software process cannot be taken, the process returns to the application scheduling process in the previous step 183. Then, the use timing of the imaging device 6 and the timing of the software processing are changed, and the mapping of the application having the second shortest processing cycle number in step 184 is performed again. When the mapping is completed, the process proceeds to step 185, and the application is sequentially mapped to the schedule. In step 186, it is confirmed that the application having the longest number of processing cycles can be mapped on the schedule. When an application that cannot be mapped on the schedule appears on the way, it is determined that the addition of the requested application is not an application that can be executed in parallel, the user is notified of this, and the additional activation is abandoned.

  In the above embodiment, the imaging device (6) that acquires image data and a plurality of applications (151 to 15N) that are set to have different functions using the image data obtained from the imaging device. It has. An imaging device control unit (13) that controls the imaging device in response to image data acquisition requests corresponding to a plurality of applications is provided. Furthermore, it is assumed that the image processing camera system includes a control unit (14) that allows a plurality of applications to capture image data from one imaging device and execute the plurality of applications in parallel. Here, a means (17) is provided for storing the number of required image data and the frequency of capturing image data in a plurality of applications. Further, there is provided means (14) for selecting a plurality of applications that can be executed in parallel based on the number of image data and the frequency of taking in the image data. Next, the plurality of executable applications include an image capture scheduling unit (14) that determines the timing and interval of image data capture that repeats the capture of image data from one imaging device without overlapping in time. The scheduling unit (14) is configured to determine timing including processing using captured image data in each application.

  Also provided is an application group storage unit (17) for storing a combination of a plurality of applications that are executed in parallel using image data from one imaging device. Here, the selection means reads data on application groups that can be executed in parallel from the application group storage means.

  Furthermore, a plurality of basic image processing function units (16A to 16M) for controlling the imaging device to execute a plurality of applications are provided. Based on the required degree of coincidence of these basic image processing functions, a means for determining a plurality of applications to be executed in parallel using image data from one imaging device is provided.

  The means (14) for selecting a plurality of applications that can be executed in parallel based on the number of image data and the frequency of taking in the image belongs to the same application group as the application that is being executed while the application is being executed. Based on that, the other application that can be executed is selected. In addition, the necessary basic image processing function helps to select another executable application based on the degree of coincidence with the application being executed. In addition, another application is executed at the timing between the image data acquisition period from the imaging device by the application being executed, depending on whether image data acquisition from the imaging device required by the other application is possible I choose it as a possible application.

  Next, the man-machine interface includes means (19) for displaying an executable application and operation means for the user to instruct activation of the displayed executable application. In addition, there are provided means for displaying a running application and an application that can be additionally executed, and an operation means for the user to instruct the activation of the additionally executable application and the stop of the running application. And the control means which starts and stops an application based on the instruction | indication input by this operation means is provided.

  Further, an executable application selection unit that switches an application that can be additionally executed according to a change in the surrounding environment, and a unit that displays the executable application selected by the selection unit are provided.

  Further, an operation means for requesting addition by downloading a new application is provided.

  With these features, it is possible to share one imaging device with a plurality of applications with high reliability and efficiency. In addition, the user can select an executable application, and the application selected by the user can be immediately executed. Furthermore, an application to be preferentially executed can be selected and executed according to the system status.

  So far, the embodiment in which the present invention is applied to an image processing camera system mounted on an automobile has been described in detail. However, the present invention can be applied to various image processing camera systems. For example, the present invention can be applied to an intruder monitoring camera system. Rather than installing multiple cameras that realize a specific function, the present invention is applied when there is an image processing function that can be shared, and the camera is reliably shared for each function required by the application. Can be reduced.

  According to the present invention, it is possible to provide an image processing camera system that can share one imaging device with a plurality of applications with high reliability.

  Further, according to the embodiment of the present invention, it is possible to provide an image processing camera system excellent in usability in which a plurality of applications that can be executed are clearly indicated to the user and the user can select an application to be executed without hesitation.

The block diagram of the image processing camera system in one Embodiment which applied this invention to the motor vehicle, and the hardware block diagram of an image processing camera. The figure which shows the classification of the image quality, rate, camera control, and image processing function which many applications in a motor vehicle require. The functional block diagram of the image processing camera system in one Embodiment of this invention. The functional block diagram of an imaging device control part. The hardware block diagram of an imaging device. An example time chart of operation scheduling of multiple applications. The processing flow figure which schedules the application in one Embodiment of this invention. Another example time chart of operation scheduling of a plurality of applications. Scheduling time chart when operating the video album application. Scheduling time chart of the procedure for starting additional applications. FIG. 5 is a diagram illustrating an example of grouping of applications that can operate in parallel. An example of a navigation screen for selecting and switching applications. Screen example after selecting high-quality video album. The figure which shows an example of the change of the navigation screen before and behind adding an application. The processing flow figure which performs additional starting of the application by one Embodiment of this invention. An example figure of a screen which performs addition of a new application and deletion of an existing application. The download processing flow figure of the new application by one Embodiment of this invention. The processing flow figure of operation scheduling of a plurality of applications including a new application.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Automobile, 2, 3 ... Camera, 4 ... Network, 5 ... Interface means, 6 ... Imaging device, 7 ... Imaging device interface, 8 ... RAM, 9 ... ROM, 10 ... CPU, 11 ... External interface means, 12 ... Microcomputer, 13 ... Imaging device control unit, 14 ... Application scheduling unit, 151-15N ... N applications, 16A-16M ... Image processing function (A-M), 17 ... Environmental information, 18 ... Interface, 19 ... Screen .

Claims (16)

  An imaging device that acquires image data, a plurality of applications that are set to have different functions using the image data, and the imaging device is controlled according to image data acquisition requests that correspond to the plurality of applications Image data required for a plurality of applications in an image processing camera system including an imaging device control unit and a control unit that allows a plurality of applications to capture image data from one imaging device and execute the plurality of applications in parallel Means for storing the number and frequency of capturing image data, means for selecting a plurality of applications executable in parallel based on the number of image data and the frequency of capturing image data, and the plurality of executable applications Overlap from device Image processing camera system comprising the image capture scheduling unit that determines the timing and interval of the image data acquisition repeated uptake Ku image data.   The image processing camera system according to claim 1, further comprising a scheduling unit that determines a timing including processing using the captured image data in each application.   Application group storage means for storing a combination of a plurality of applications to be executed in parallel using image data from one imaging device, wherein the selection means is an application that can be executed in parallel from the application group storage means. 2. The image processing camera system according to claim 1, wherein data about the group is read out.   Based on the degree of coincidence between the plurality of basic image processing function units that control the imaging device to execute a plurality of applications and the required basic image processing function, the image data from one imaging device is used in parallel. The image processing camera system according to claim 1, further comprising means for determining a plurality of applications to be executed.   2. The image processing camera system according to claim 1, further comprising means for selecting another executable application based on belonging to the same application group as the application being executed during execution of the application. .   2. A means for selecting another executable application based on a degree of coincidence with a running application for a basic image processing function required during execution of an application. The image processing camera system described.   While executing an application, it is possible to acquire image data from an imaging device required by another application at the timing between image data acquisition periods from the imaging device by the executing application. 2. The image processing camera system according to claim 1, further comprising means for selecting the other application as an executable application.   A plurality of applications set to have different functions using image data, an imaging device for acquiring image data, and controlling the imaging device in response to an image data acquisition request corresponding to the plurality of applications An image processing camera system including an imaging device control unit includes means for displaying an executable application and operation means for a user to instruct activation of the displayed executable application. Image processing camera system.   Means for displaying a running application during execution of an application, operating means for a user to stop the running application, and control means for stopping the application based on an instruction input by the operating means 9. The image processing camera system according to claim 8, further comprising:   Means for displaying a running application and an application that can be additionally executed during execution of an application, an operation means for the user to instruct to start an additional executable application and stop the running application, and this operation 9. The image processing camera system according to claim 8, further comprising control means for starting and stopping an application based on an instruction input by the means.   9. The image according to claim 8, further comprising: an executable application selection unit that switches an application that can be additionally executed according to a change in the surrounding environment; and a unit that displays the executable application selected by the selection unit. Processing camera system.   9. The image processing camera system according to claim 8, further comprising operation means for requesting addition of a new application.   9. The image processing camera system according to claim 8, further comprising operation means for requesting addition by downloading a new application.   An imaging device that acquires image data, a plurality of applications each having different functions using image data from the imaging device, and an imaging that controls the imaging device in response to an image data acquisition request corresponding to the plurality of applications In an image processing camera system including a device control unit and a control unit that allows a plurality of applications to capture image data from one imaging device and execute the plurality of applications in parallel, And a scheduling step for determining an image data capturing timing and an interval for repeatedly capturing image data from one imaging device without overlapping in time. Image processing camera control method.   The image processing camera control method according to claim 14, further comprising a step of determining a timing including processing using captured image data in each application.   Based on the step of reading from the storage means storing the number of image data required in a plurality of applications and the required capture frequency, and the plurality of executable applications based on the number of read image data and the capture frequency, The image processing camera control method according to claim 14, further comprising a step of determining an image data capturing timing and an interval for repeatedly capturing image data from one imaging device.

Images