TW202303517A - Image controller, image processing system and image modifying method - Google Patents

Abstract

An image processing system and an image modifying method are provided. The first controller obtains the first image from the image capturing apparatus. The first controller converts the first image into the second image according to a converting operation. The converting operation includes distortion correction, and the distortion correction is used for correcting the distortion of the target object. The second controller detects the target object in the second image, to generate a detected result. The first controller corrects the converting operation according to the detected result. Accordingly, the vision experiment can be improved.

Description

Image controller, image processing system and image correction method

The present invention relates to an image processing technology, and in particular relates to an image controller, an image processing system and an image correction method.

In the conventional technology, although a camera equipped with a wide-angle lens or a fisheye lens can capture an image with a wider field of view (FoV), the edge of the image may be curved and form an unnatural appearance. Distortion in wide-angle or fisheye images can make content difficult to read and is more likely to cause discomfort to the user's eyes.

On the other hand, such cameras are usually installed in products such as rearview mirrors, IP cameras, surveillance systems, Internet of Things cameras, and machine vision related products. In some application scenarios, the object in the image is the target that the viewer wants to track. However, there may be more than one object in the image and the object may move, but such products usually cannot provide an appropriate image in response to the movement or number of objects.

In view of this, the embodiments of the present invention provide an image controller, an image processing system, and an image correction method, which can simply and effectively correct distorted images, and can improve the recognition of a specific tracking target in the image.

The image correction method according to the embodiment of the present invention includes (but not limited to) the following steps: obtaining a first image from an image capturing device. Convert the first image into the second image according to the conversion operation. The conversion operation includes a deformation correction, and the deformation correction is used to correct the deformation of one or more objects in the first image. Detect the target object in the second image to generate a detection result. Correct the conversion operation according to the detection result.

The image processing system of the embodiment of the present invention includes (but not limited to) an image capture device, a first controller and a second controller. The image capturing device includes a lens and an image sensor. The first image is captured through the lens and the image sensor. The first controller is coupled to the image capture device and used for converting the first image into a second image according to the conversion operation. Conversion jobs include distortion corrections. The deformation correction is used to correct the deformation of one or more objects in the first image. The second controller is coupled to the first controller and used for detecting the targets in the second image to generate detection results. The first controller is further used for correcting the conversion operation according to the detection result.

The image controller of the embodiment of the present invention includes (but not limited to) a memory and a processor. Memory is used to store code. The processor is coupled to the memory. The processor is configured to load and execute code to obtain the first image, convert the first image into a second image according to the conversion operation, and detect one or more targets in the second image to generate a detection result , and correct the transformation based on the detection results. The conversion operation includes a deformation correction, and the deformation correction is used to correct the deformation of one or more objects in the first image.

Based on the above, the image controller, image processing system, and image correction method according to the embodiments of the present invention mainly correct the conversion operation of the first controller according to the detection result of the target object by the second controller. In this way, the distortion can be corrected and the target object can be highlighted in the second image.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail together with the accompanying drawings.

FIG. 1 is a block diagram of components of an image processing system 1 according to an embodiment of the present invention. Please refer to FIG. 1 , the image processing system 1 includes (but not limited to) an image capture device 10 , a first controller 30 and a second controller 50 .

The image capturing device 10 may be a camera, a video camera, a monitor or a device with similar functions. The image capture device 10 may include (but not limited to) a lens 11 and an image sensor 15 (for example, a Charge Coupled Device (CCD) or a Complementary Metal-Oxide-Semiconductor (CMOS), etc. ). In one embodiment, the image can be captured through the lens 11 and the image sensor 15 . For example, the light is imaged on the image sensor 15 through the lens 11 .

In some embodiments, the specifications of the image capture device 10 (for example, image capture aperture, magnification, focal length, image capture viewing angle, size of the image sensor 15 , etc.) and its quantity can be adjusted according to actual needs. For example, the lens 11 is a fisheye or wide-angle lens, and is used to generate a fisheye image or a wide-angle image.

The first controller 30 can be coupled to the image capture device 10 through a camera interface, I2C and/or other transmission interfaces. The first controller 30 includes (but not limited to) a memory 31 and a processor 35 . Memory 31 can be any type of fixed or removable random access memory (Radom Access Memory, RAM), read only memory (Read Only Memory, ROM), flash memory (flash memory), traditional hard disk (Hard Disk Drive, HDD), Solid-State Drive (Solid-State Drive, SSD) or similar components. In one embodiment, the memory 31 is used to store program codes, software modules, configurations, data or files. The processor 35 can be an image processor or a graphics processing unit (Graphic Processing unit, GPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP) ), programmable controller, Field Programmable Gate Array (Field Programmable Gate Array, FPGA), application-specific integrated circuit (Application-Specific Integrated Circuit, ASIC) or other similar components or a combination of the above components. In one embodiment, the processor 35 is used to execute all or part of the operations of the first controller 30 , and can load and execute various program codes, software modules, files and data stored in the memory 31 .

The second controller 50 can be coupled to the first controller 30 through a camera interface (eg, Mobile Industry Processor Interface (MIPI)), I2C, USB and/or other transmission interfaces. The second controller 50 includes (but not limited to) a memory 51 and a processor 55 . The implementation and functions of the memory 51 can refer to the description of the memory 31 , and will not be repeated here. For the implementation and functions of the processor 55, reference may be made to the description of the processor 35, which will not be repeated here. In one embodiment, the processor 55 is used to execute all or part of the operations of the second controller 50 , and can load and execute various program codes, software modules, files and data stored in the memory 51 .

In one embodiment, the image capturing device 10 , the first controller 30 and the second controller 50 can be integrated into an independent device. For example, the image processing system 1 is a camera system, wherein the first controller 30 may be a fisheye controller, a wide-angle lens controller or other image-related controllers, and the second controller 50 is a microcontroller or SoC. In another embodiment, the image capture device 10 and the first controller 30 can be integrated into a module, and the second controller 50 is, for example, a computer system (for example, a desktop computer, a notebook computer, a server, a smart phone) mobile phone or tablet) or set on one of them. In yet another embodiment, the first controller 30 and the second controller 50 can be integrated into an image controller or an appropriate controller module, and can be coupled with the image capture device 10 .

In the following, the method described in the embodiment of the present invention will be described in conjunction with various devices, components and modules in the image processing system 1 . Each process of the method can be adjusted accordingly according to the implementation situation, and is not limited thereto.

FIG. 2 is a flowchart of an image correction method according to an embodiment of the invention. Referring to FIG. 2 , the first controller 30 acquires a first image from the image capture device 10 (step S210 ). Specifically, the first image is an image obtained by shooting one or more targets through the image capture device 10 or other external image capture devices. In one embodiment, the target is, for example, a human body. In some embodiments, the first image may also be for the upper body of the human (eg, waist, shoulders, or above the chest). In other embodiments, the targets may also be various types of organisms or non-organisms. The first controller 30 can obtain the first image captured by the image capture device 10 through the camera interface and/or I2C.

The first controller 30 can convert the first image into the second image according to a conversion operation (step S230). Specifically, in one embodiment, the conversion operation includes distortion correction. The deformation correction is used to correct the deformation of one or more objects in the first image. In another embodiment, the transformation operation includes position adjustment. The position adjustment is used to adjust the position of one or more objects in the first image. In yet another embodiment, the conversion operation includes deformation correction and position adjustment. That is, the appearance and/or location of the object in the first image may be different from the same object in the second image.

For example, FIG. 3A is a schematic diagram of dewarp according to an embodiment of the present invention. Please refer to FIG. 3A , in this embodiment, the deformation correction is, for example, dewarping processing, wherein the first image FIM1 is, for example, an image obtained through a fisheye lens, and the image is, for example, a distorted image. The first controller 30 may perform a conversion operation, such as dewarping and unfolding, on the first image FIM1 to generate the second image SIM1 , so as to make the second image SIM1 closer to the real image. Thereby, an object image with a better ratio or a normal ratio can be generated.

In some application scenarios, in order to adapt to the size (eg, resolution 1920×1080 or 480×272) or ratio (eg, 16:9 or 4:3) of the display device, the viewing angle of the image can be adjusted. FIG. 3B is a schematic diagram of viewing angle adjustment according to an embodiment of the invention. Please refer to FIG. 3B , in this embodiment, the position adjustment included in the conversion operation is, for example, viewing angle adjustment, and the imaging viewing angle of the lens 11 of the image capture device 10 is, for example, 180 degrees. Wherein, the first controller 30 of the present embodiment can change or adjust, for example, the viewing angle of the first image FIM1 to an angle of view FOV1 (for example, 140 degrees) (ie, conversion operation) to generate the second image SIM2. Alternatively, the first controller 30 may change the viewing angle of the first image FIM1 to an angle of view FOV2 (for example, 110 degrees) (ie, a conversion operation) to generate the second image SIM3. In this way, the imaging angle can be directed towards the target object at a specific position in front of the lens 11 .

In some application scenarios, the image processing application has the requirement of zooming the image. For example, if the image is insufficient for image recognition, the image needs to be zoomed in. In addition, zooming in on the image will cause the viewing angle to be relatively narrowed. If you want to browse the original size of the image, you can restore the viewing angle by zooming out the image. FIG. 3C is a schematic diagram of scaling adjustment according to an embodiment of the invention. Referring to FIG. 3C , in this embodiment, the conversion operation may also include scaling adjustment. In this embodiment, the first image FIM1 can be enlarged according to the scaling factor SR1 (eg, 120%) (ie, conversion operation) to generate the second image SIM5. Alternatively, in this embodiment, the first image FIM1 can be reduced according to the scaling factor SR2 (eg, 80%) (ie, conversion operation) to generate the second image SIM6. In this way, the target object can be enlarged or reduced.

In some application scenarios, after the image is zoomed in, the region beyond the visible range in the image can be browsed by shifting. FIG. 3D is a schematic diagram of translation according to an embodiment of the invention. Referring to FIG. 3D , in this embodiment, the position adjustment included in the conversion operation is, for example, image translation. In the figure, the dotted box represents the visible range of the first image, and the solid line box represents the visible range of the second image. For example, the first controller 30 can notify the image capture device 10 that the first image FIM1 can be translated according to the direction SH1 (toward the upper right) (ie, the conversion operation) to generate the second image SIM7 . Alternatively, the first controller 30 can, for example, notify the image capturing device 10 that the first image FIM1 can be translated according to the direction SH2 (toward the bottom left) (that is, the conversion operation) to generate the second image SIM8. In this way, the position of the target object in the image can be changed.

In some application scenarios, when the image of interest is located above or below the image capture device 10 , the angle can be adjusted by adjusting the vertical viewing angle (or called tilt adjustment (tilt)), so as to obtain a better imaging viewing angle. For example, when the image capture device 10 is integrated into an electronic doorbell and installed on a wall, the height of the image capture device 10 may be higher or lower than the standing height of a person, so the image capture angle can be changed by adjusting the vertical angle of view. FIG. 3E is a schematic diagram of adjusting up and down viewing angles according to an embodiment of the invention. Referring to FIG. 3E , in this embodiment, the position adjustment included in the conversion operation is, for example, the adjustment of the up and down viewing angles. The image capture device 10 is arranged upright and facing the y-axis. For example, the first controller 30 can notify the image capture device 10 to adjust the viewing angle of the first image FIM1 of TI1 according to the axis x (ie, a conversion operation) to generate the second image SIM9 . Or, for example, the first controller 30 can notify the image capturing device 10 that the viewing angle of the first image FIM1 of the TI2 can be adjusted downwards according to the axis x (ie, a conversion operation) to generate the second image SIM10 . In this way, the position of the target object in the image can be changed.

In some application scenarios, when the target of interest is located on the left or right of the image capture device 10, the angle can be adjusted by adjusting the left and right viewing angles (or called left and right rotation angle adjustment (pan)), so as to obtain better results. angle of view. For example, when the image capture device 10 is integrated into the electronic doorbell and installed on the wall, the lens 11 may not be facing the visitor, so the image capturing angle of view can be adjusted towards the visitor by adjusting the left and right viewing angles. FIG. 3F is a schematic diagram of adjusting left and right viewing angles according to an embodiment of the present invention. Referring to FIG. 3F , the position adjustment included in the conversion operation is, for example, left and right viewing angle adjustment (or called left and right rotation angle adjustment (pan)). The image capture device 10 is arranged upright and facing the y-axis. For example, the first controller 30 can notify the image capture device 10 to adjust the viewing angle of the first image FIM1 of the PA1 to the right according to the axis z (ie, a transformation operation) to generate the second image SIM11 . Or, for example, the first controller 30 can notify the image capture device 10 to adjust the viewing angle of the first image FIM1 of the PA2 to the left according to the axis z (ie, a conversion operation), so as to generate the second image SIM12 . In this way, the position of the target object in the image can be changed.

FIG. 3G is a schematic diagram of plane viewing angle adjustment according to an embodiment of the present invention. Please refer to FIG. 3G , the position adjustment included in the conversion operation is, for example, plane viewing angle adjustment (or called rotation (rotate)). The image capture device 10 is placed horizontally and faces the y-axis. For example, the first controller 30 can notify the image capturing device 10 that the viewing angle of the first image FIM1 of the RO1 can be rotated clockwise according to the axis y (ie, a conversion operation) to generate the second image SIM13 . Or, for example, the first controller 30 can notify the image capture device 10 that the viewing angle of the first image FIM1 of RO2 can be rotated counterclockwise according to the axis y (ie, conversion operation), so as to generate the second image SIM14 . In this way, the position of the target object in the image can be changed.

With regard to the application scenario of plane viewing angle adjustment, FIG. 3H is a schematic diagram of the arrangement of the image capture device 10 and the captured images according to an embodiment of the present invention. Please refer to FIG. 3H , the image capture device 10 is placed horizontally. Assuming that the lens 11 is a fisheye lens, light passes through the lens 11 and is projected on the image sensor 13 to generate a fisheye image IMe. According to application requirements, the first controller 30 can, for example, notify the image capture device 10 to capture only the outer ring image IMo corresponding to the outer ring of the lens 11 .

FIG. 3I is a schematic diagram of the arrangement of the image capture device 10 and the captured images according to an embodiment of the present invention. Please refer to FIG. 3I , as shown in the upper half of the figure, it is assumed that the image capture device 10 is placed horizontally among the characters P1 , P2 , P3 , and P4 . If only the outer ring of the lens 11 is captured and divided into two halves, the outer ring images IMo1 and IMo2 can be generated. The outer ring image IMo1 captures the characters P1 and P2, and the outer ring image IMo2 captures the characters P3 and P4.

FIG. 3J is a schematic diagram of the arrangement of the image capture device 10 and the captured images according to an embodiment of the present invention. Referring to FIG. 3J , it is assumed that the first controller 30 adjusts the plane viewing angle of the outer ring images IMo1 and IMo2 according to the direction of the arrow shown in the left figure. The characters P1-P4 shown in light-colored lines in the figure represent their original positions, and the characters P1-P4 in dark-colored lines represent their positions after adjustment of the plane viewing angle. The characters P1 and P2 captured by the outer ring image IMo1 will be shifted to the left, and the characters P3 and P4 captured by the outer ring image IMo2 will be shifted to the right.

FIG. 4 is a schematic diagram of fisheye image expansion according to an embodiment of the present invention. Referring to FIG. 4 , it is assumed that the first image FIM2 is a fisheye image, and the characters in the image are deformed. The second image SIM15 is the image corrected by the first controller 30 , and the ratio of the characters in the image is normal.

In one embodiment, the conversion operation further includes target layout (or called window layout, multi-split windows). For example, the second image includes a plurality of windows. In addition, it is assumed that the second image includes one or more objects. For example, a first target, a second target, a third target and/or a fourth target. Object programming is used to adjust the object window (ie, one of those windows) of the first object in the second image. In this embodiment, the second image can be divided into a plurality of windows, the distortion-corrected image can be cropped, and the cropped part of the image can be arranged in a specific window. In a preferred embodiment, in this embodiment, an application program is used to divide the second image into multiple windows.

For example, FIG. 5 is a schematic diagram of object arrangement according to an embodiment of the present invention. Please refer to FIG. 5 , after the first controller 30 performs distortion correction on the first image FIM2 in FIG. 4 , the scaled and/or cropped images are respectively arranged in different windows TW1 and TW2 , for example. The images arranged in different windows TW1, TW2 are, for example, images of personnel participating in a conference or images of participants who have undergone specific screening. The specifically filtered images of the participants are, for example, the participants speaking in the conference.

FIG. 6 is a schematic diagram of target arrangement of multiple modes M1-M15 according to an embodiment of the present invention. Please refer to FIG. 6, these multiple modes M1~M15 may include one window TW1, two windows TW1, TW2, three windows TW1~TW3, four windows TW1~TW4, five windows TW1~TW5 or six windows TW1~ TW6. The dividing line in each mode M2~M15 is the window range. And TW1~TW6 are used to represent the numbers of different windows. Furthermore, even though the number of windows is the same, the positions, sizes and/or shapes of the windows may be different in different objects. For example, the window TW1 of the mode M6 is larger than the window TW1 of the mode 2 .

It should be noted that there may be other changes in the number, size and shape of the windows, which are not limited by the embodiments of the present invention. In addition, symbols "TW1", "TW2", "TW3", "TW4", "TW5", and "TW6" are merely used as numbers.

Here are three examples:

FIG. 7A is a schematic diagram of an object layout of a pattern according to an embodiment of the present invention. Please refer to FIG. 4, FIG. 6 and FIG. 7A, assuming that the first controller 30 selects the mode M1 in FIG. 6, the first image FIM2 in FIG. 4 can be converted into the second image SIM16 in FIG. 7A, wherein in FIG. The second image SIM16 of is, for example, a single-window mode.

FIG. 7B is a schematic diagram of an object layout of a pattern according to an embodiment of the present invention. Referring to FIG. 4 , FIG. 6 and FIG. 7B , assuming that the first controller 30 selects the mode M5 in FIG. 6 , the first image FIM2 in FIG. 4 can be converted into the second image SIM17 in FIG. 7B . The second image SIM17 in FIG. 7B is, for example, a two-part window. The window TW1 is for the character P3, and the window TW2 is for the four characters P1, P2, P3, and P4.

FIG. 7C is a schematic diagram of an object layout of a pattern according to an embodiment of the invention. Referring to FIG. 4 , FIG. 6 and FIG. 7C , assuming that the first controller 30 selects the mode M11 in FIG. 6 , the first image FIM2 in FIG. 4 can be converted into the second image SIM18 in FIG. 7C . The second image SIM18 is, for example, three divided windows. The window TW1 is for the four characters P1, P2, P3, P4, the window TW2 is for the character P3, and the window TW3 is for the character P2.

Referring to FIG. 2 again, the second controller 50 of this embodiment can detect one or more targets in the second image to generate detection results (step S250 ). Specifically, the second controller 50 can obtain the second image converted by the first controller 30 through the USB and/or I2C interface.

In one embodiment, the second controller 50 can perform object detection on the second image. The object detection is, for example, determining a bounding box (bounding box) or a representative point (pinot) (possibly located at the contour, center, or any position above), and then identify the type of object (for example, human, male or female, dog or cat, table or chair, etc.). The detection result includes the bounding box (or representative point) of the target and/or the type of the target. The object detection described in the present invention may also be determining a Region of Interest (ROI) or a bounding rectangle corresponding to the target in the second image, which is not limited herein.

In one embodiment, the second controller 50 may, for example, apply a neural network-based algorithm (for example, YOLO, Region Based Convolutional Neural Networks (R-CNN), or Fast R-CNN). CNN (Fast CNN)) or algorithm based on feature matching (for example, Histogram of Oriented Gradient (HOG), Harr, or feature comparison of Speeded Up Robust Features (SURF)) Object detection.

It should be noted that the embodiment of the present invention does not limit the algorithm used for object detection. Additionally, in some embodiments, the second controller 50 may specify a specific type of object.

In one embodiment, the second controller 50 determines the position of the target object in the second image. That is, the detection result includes the position of the target. For example, whether the object is in the middle of the second image. Another example is whether the target object appears in the second image. In some embodiments, the second controller 50 can define a reference axis (for example, a horizontal axis or a vertical axis) in the second image, and determine an angle of the target object in the second image relative to the reference axis.

In one embodiment, the second controller 50 can also determine whether the object in the second image moves. That is, the detection result includes the movement of the target. For example, the second controller 50 can determine the relationship and change in position or posture of the same target in the front and rear image frames in the continuous second image through object tracking. The consecutive second images represent those consecutive image frames of the video or video stream. Object tracking is, for example, judging the correlation of the position, movement, direction and other motions of the same target object in the adjacent second image (its position can be determined by a bounding box or a representative point), and then locate the moving target object . In one embodiment, the second controller 50 can apply optical flow method, sorting method (Simple Online And Realtime Tracking, SORT), depth sorting method (Deep SORT), joint detection and embedding vector (Joint Detection and Embedding, JDE ) model or other tracking algorithms for object tracking. It should be noted that the embodiment of the present invention does not limit the algorithm used for object tracking.

Based on the above, in a preferred embodiment, the second controller 50 can determine the relationship and change in position or posture of the same target in the front and back image frames in the continuous second image through object tracking. Preferably, when the second controller 50 detects the target object in the second image and the detection result is that the target object in the second image does not move, this embodiment can only focus on demarcation The object corresponding to the frame selection, Region of Interest (ROI) or bounding rectangle (bounding rectangle) is subjected to a conversion operation such as dewarp expansion, so that, for example, the object selected by the bounding box An image of the target object with a better ratio or with a normal ratio. In other words, when the detection result is that a position of the at least one target in the second image does not change, this embodiment may not need to perform a conversion operation such as dewarp on the whole image, and It is only necessary to correct the deformation of the range corresponding to the position in the first image, thereby improving the efficiency of the image processing operation.

In contrast, when the same target in the front and rear image frames (or bounding boxes) in the continuous second image changes in position or posture, the second controller 50 detects the object in the second image. The detection result generated by the target is the movement of the target in the second image. In this embodiment, the conversion operation will be corrected based on the detection result, so that the overall image can undergo conversion such as dewarp expansion. Operation, and then detect the target again.

In one embodiment, the second controller 50 judges the integrity of the object in the second image. That is, the detection result includes the integrity of the target. For example, the second controller 50 can identify the key points (eg, eyes, nose or mouth) of the object in the second image, and confirm whether the parts corresponding to these key points are complete or whether the number is correct.

It should be noted that, in some embodiments, the first controller 30 also detects one or more targets in the second image to generate a detection result. For example, the first controller 30 performs object detection, object tracking, or integrity detection on the second image, and the same or similar content can refer to the foregoing description and will not be repeated here.

The first controller 30 can modify the conversion operation according to the detection result (step S270). Specifically, the target object and/or the image capture device 10 may change positions. If the conversion operation remains unchanged, the position of the object in the second image may not be centered or part of the object may be cropped and incomplete, thereby affecting viewing experience. In one embodiment, the second controller 50 can return the detection result to the first controller 30, and judge whether to adjust the conversion operation based on it.

In one embodiment, the position of the target in the detection result is in a bounding box format. The bounding box format includes coordinates of the bounding box on the horizontal axis and vertical axis in the second image and the size (eg, width and height) of the second image.

In another embodiment, the position of the target in the detection result is in a representative point format. The representative point format includes coordinates of the representative point on the horizontal axis and vertical axis in the second image and a scaling factor.

In an embodiment, the coordinate systems defined by the first controller 30 and the second controller 50 for the second image may be different. FIG. 8 is a schematic diagram of coordinate system transformation of a model according to an embodiment of the invention. Referring to FIG. 8 , the second controller 50 locates pixels in the second image with a coordinate system CS1 such as image coordinates. The coordinate system CS1 has its upper left corner as its origin. The first controller 30 positions the pixels in the second image with the coordinate system CS2 (eg, polar coordinate system). The coordinate system CS2 takes the center point as the origin. In other preferred embodiments, neither the first controller 30 nor the second controller 50 can locate the pixels in the second image in an appropriate coordinate system, which is not limited herein.

If the detection result includes the target position of the fourth target in the second image, the first controller 30 or the second controller 50 can set the coordinates (x _o , y _o ) of the target position of the fourth target The coordinates (x _t , y _t ) converted from the coordinate system CS1 to the coordinate system CS2 are: x _t =x _o -w/2…(1) y _t =y _o -h/2…(2) x _o is the coordinate of the fourth target on the horizontal axis in the coordinate system CS1, x _t is the coordinate of the fourth target on the horizontal axis in the coordinate system CS2, y _o is the coordinate of the fourth target on the vertical axis in the coordinate system CS1 , _yt is the coordinate of the fourth object on the vertical axis in the coordinate system CS2, w is the width of the second image, and h is the height of the second image.

It should be noted that if the first controller 30 and the second controller 50 use the same coordinate system, the coordinate conversion can be ignored.

It should be noted that the object may not be centered in the second image or the window of the second image. For example, FIG. 9 is a schematic diagram of the second image SIM19 under the imaging angle of view FOV3 according to an embodiment of the present invention. Referring to FIG. 9 , it is assumed that the maximum viewing angle FOV4 of the image capture device 10 is 180 degrees. That is, the first image includes a field of view of 180 degrees. The viewing angle FOV3 set by the conversion job is 140 degrees. Assume that an imaginary line extending straight forward from the position of the image capture device 10 is the reference axis (ie, the vertical centerline of the image capturing angle of view FOV3 ). The target T1 is located directly in front of the image capture device 10 , that is, the offset angle relative to the reference axis is zero. Therefore, the target T1 is located in the middle of the second image SIM19.

FIG. 10 is a schematic diagram of a second image SIM20 with a rotated viewing angle according to an embodiment of the present invention. Please refer to FIG. 10 , different from FIG. 9 , the offset angle of the target T2 relative to the reference axis is θ1 (for example, 45 degrees). Therefore, the target T2 is located on the left side of the second image SIM20.

In one embodiment, the first controller 30 can center the target object according to the offset angle. The detection result is the deviation angle of the first target in the second image relative to the reference axis. The reference axis is the horizontal or vertical centerline of the original viewing angle of the second image. The first controller 30 can set the conversion operation as rotating the viewing angle of the first image according to the offset angle, so as to reduce the offset angle. For example, the rotation viewing angle includes the adjustment of the up and down viewing angles described in FIG. 3E , the adjustment of the left and right viewing angles described in FIG. 3F , and the adjustment of the plane viewing angle shown in FIG. 3G .

In one embodiment, the first controller 30 can rotate the direction corresponding to the first imaging angle of view of the first image according to the angle of view of the first image and the length of the first image (for example, its width or height). Axial scale transforms the coordinates of the second image. If the coordinate system CS2 in Figure 6 is used, the rotations of the imaging angles of view on different axes are: Pan(x _t )=x _t ×fov _H /w...(3) Tilt(y _t )=y _t ×fov _V / h...(4) Rotate(x _t )=x _t ×fov _H /w...(5) fov _H is the viewing angle of the second image in the horizontal direction, and fov _V is the viewing angle of the second image in the vertical direction. Pan() is a function to adjust the angle of view of the up and down imaging. For example, the direction of turning left and right corresponds to the x-axis, so the viewing angle of view in the horizontal direction and the width of the second image are concerned. Tilt() is a function for adjusting the angle of view of the left and right images. For example, the direction of turning up and down corresponds to the y-axis, so the angle of view in the vertical direction and the height of the second image are concerned. Rotate() is a function to adjust the viewing angle of the plane. As shown in FIG. 3J , the outer ring images IMo1 and IMo2 show the effect of shifting left and right respectively, so the angle of view in the horizontal direction and the width of the second image are aimed at.

In one embodiment, the detection result includes that the captured viewing angle of the rotated second image exceeds the (maximum) viewing angle of the image capture device 10 . Taking FIG. 10 as an example, after the FOV4 of the second image SIM21 is rotated, its left boundary has exceeded the maximum FOV3 of the image capture device 10 . Therefore, even though the target T2 is located in the middle of the second image SIM21, the left side of the second image SIM21 is unable to obtain a visible image and a black block is generated.

The first controller 30 can set the rotation angle of the second image (hereinafter referred to as the first imaging angle of view) to exceed the imaging angle of view of the image capture device 10 (hereinafter referred to as the second imaging angle of view). Second capture the edge of the viewing angle. For example, if the coordinate system CS2 in FIG. 8 is used, the edge correction of the imaging angle of view is as follows: If the coordinate x _t2 of the horizontal axis rotated by the imaging angle of view is between the left edge and the right edge of the second imaging angle of view , the coordinate x _t2 remains unchanged; if the coordinate x _t2 of the horizontal axis rotated by the imaging angle of view is smaller than the coordinate of the left edge of the second imaging angle of view, the coordinate x _t2 is corrected to the coordinate of the left edge of the second imaging angle of view; If the coordinate x _t2 of the horizontal axis rotated by the imaging angle of view is greater than the coordinates of the right edge of the second imaging angle of view, the coordinate x _t2 is corrected to the coordinates of the right edge of the second imaging angle of view; The coordinate y _t2 of the axis is between the upper edge and the lower edge of the second viewing angle of view, then the coordinate y _t2 remains unchanged; if the coordinate y _t2 of the vertical axis rotated by the viewing angle of view is smaller than the lower edge of the second viewing angle of view coordinates _, then the coordinate y _t2 is corrected to be the coordinates of the lower edge _of the second viewing angle; is the coordinate of the upper edge of the second imaging angle of view.

For example, FIG. 11 is a schematic diagram of the corrected second images SIM20, SIM22 according to an embodiment of the present invention. Referring to FIG. 11 , the difference from FIG. 10 is that the first viewing angle of the second image SIM22 is corrected, so the target T2 is close to the middle of the second image (for example, the angle θ2 is 20 degrees).

FIG. 12A is a schematic diagram of a second image SIM23 of a conference scene according to an embodiment of the present invention. Please refer to FIG. 12A , in the application scenario of the meeting mode, it is assumed that the lens 10 is a fisheye lens and can obtain a 360-degree first image, and splice the images in a manner of 180 degrees up and down. Since the targets T3 and T4 are located within the upper 180-degree viewing angle, and the targets T5 and T6 are located within the lower 180-degree viewing angle, the targets T3-T6 are located at appropriate positions in the second image.

FIG. 12B is a schematic diagram of a second image SIM24 of a meeting situation according to another embodiment of the present invention. Please refer to FIG. 12B , the difference from FIG. 12A is that the targets T3 - T6 in this embodiment are located in non-ideal positions or in a moving state. Therefore, parts of the objects T3, T5 in the second image SIM24 are cropped (ie, incomplete).

FIG. 12C is a schematic diagram of the corrected second image SIM25 according to another embodiment of the present invention. Please refer to FIG. 12C, if the detection result of the second controller 50 is the integrity of the target object, after judging the integrity of the target object, the first controller 30 may be triggered to rotate the imaging angle of the first image (for example, Rotation angle θ3). Thereby, compared with FIG. 12B , the targets T3 - T6 can be fully presented in the second image SIM25 .

In one embodiment, the detection result includes the size ratio of the second object in the target object in the second image. The image processing system of this embodiment can set the transformation operation according to the size ratio to change the zoom ratio of all or part of the second image, so as to maintain the proportion of the second object in the second image. Taking FIG. 6 as an example, the size of the second object in the window TW1 of the mode M1 may be larger than that of the second object in the window TW1 of the mode M15 to enhance the visual experience.

In one embodiment, if the detection result is to locate the second target with a bounding frame, the image processing system of this embodiment may set the scaling factor to be the smallest of the width ratio, height ratio and maximum ratio. Size ratio includes height ratio and width ratio. The width ratio is the ratio of the width of the second image to the width of the bounding box of the second object, and the height ratio is the ratio of the height of the second image to the height of the bounding box.

In another embodiment, if the detection result is based on the representative point to locate the second target, the image processing system of this embodiment can set the zoom ratio as a reference ratio. That is, directly enlarge the second target with the specified reference magnification.

In one embodiment, conversion jobs are scheduled for the target. The image processing system of this embodiment will determine the target window according to the bounding box or the representative point in the original window. The image processing system of this embodiment can set the reference axis for evaluating the offset as the central axis of the bounding box of the third object among the multiple objects or the extension line of the representative point, and rotate the first imaging angle of view to This third target is aligned to the target window. That is, center the viewing angle towards the third target. The viewing angles in functions (3)~(4) can be replaced with the viewing angles for the target window. In addition, if the rotated viewing angle exceeds the bounding frame, the first controller 30 may modify the viewing angle used for cropping the image, and limit the portion beyond the bounding frame to the edge of the bounding frame.

In one embodiment, the image processing system can determine the size ratio of the object window of the third object in the second image, and set the conversion operation as changing the scaling ratio of the third object according to the size ratio. For example, the image processing system may set the scaling factor to be the smallest of the second width ratio, the second height ratio and the maximum ratio. The size ratio includes a second height ratio and a second width ratio. The second width ratio is the ratio of the width of the target window to the width of the bounding box of the third object, and the second height ratio is the ratio of the height of the target window to the height of the bounding box.

For example, FIG. 13 is a schematic diagram of a multi-object window image according to an embodiment of the present invention. Please refer to FIG. 13 , taking the mode M11 in FIG. 6 as an example, the window TW1 is the original window, and the windows TW2 and TW3 are the target windows. The original window is the maximum viewing angle FOV5 of the image capture device 10 or the image within the preset viewing angle of the second image in the single window mode. The other two target windows are the images in the viewing angle FOV6 and the viewing angle FOV7 respectively. The second controller 50 obtains the bounding boxes BB1 and BB2 according to the detection results, and arranges the images in the two BB1 and BB2 in the windows TW2 and TW3 respectively. Compared with the viewing angle FOV5, the viewing angles FOV6, FOV7 are rotated and their left and right boundaries are the left and right boundaries of the bounding boxes BB1, BB2. Therefore, the faces in each target window can be automatically centered, and the faces in the target windows can be automatically scaled to an appropriate size. In addition, if the face in the target window moves, the present embodiment can still center the face in the target window and maintain the size of the face according to the detection result of the second controller 50 .

FIG. 14A is a schematic diagram of a second image SIM27 of a multi-object window according to an embodiment of the present invention. Please refer to FIG. 14A , taking the mode M4 in FIG. 6 as an example, before the conversion operation is corrected, the faces of the targets T3-T6 in the second image SIM27 are all cut by the window, which makes the visual experience not good.

FIG. 14B is a schematic diagram of the corrected second image SIM28 according to an embodiment of the invention. Referring to FIG. 14B , the first controller 30 can obtain that the targets T3 - T6 deviate from the center of the window according to the detection results, and correct the conversion operation accordingly. Compared with FIG. 14A , after the correction and conversion operation, the faces of the targets T3~T6 in the second image SIM28 have been centered in the target window, and the second image SIM28 presents a face of a suitable size to enhance the visual experience . In addition, no matter where the objects T3-T6 are located in the meeting room or move, the second image SIM28 can present a suitable four-divided image (that is, the face remains centered and the size of the face is consistent).

To sum up, in the image controller, image processing system, and image correction method of the embodiments of the present invention, the conversion operations related to deformation correction and/or object layout are corrected based on the detection results of image recognition. Wherein, the embodiment of the present invention can direct the viewing angle towards the target object, and change the size of the target object in the image. In this way, the target object can be automatically centered in the image or the specified target window, and the size of the target object in the image can be automatically adjusted, thereby improving the visual experience. Even if the target moves, the centering and size of the target can still be maintained through the detection result and operation correction.

Although the present invention has been disclosed above with the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field may make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be defined by the scope of the appended patent application.

1: Image processing system 10: Image capture device 11: Lens 13: Image sensor 30: First controller 31: Memory 35: Processor 50: Second controller 51: Memory 55: Processor S210~ S270: Steps FIM1-FIM2: first image SIM1-SIM28: second image FOV1-FOV7: angle of view SR1, SR2: scaling factor SH1, SH2: direction TI1: upward adjustment TI2: downward adjustment PA1: rightward adjustment PA2: direction Left adjustment RO1: turn clockwise RO2: turn counterclockwise x, y, z: axis P1, P2, P3, P4: character IMe: fisheye image IMo, IMo1, IMo2: outer ring image TW1~TW6: window M1~M15 : Mode CS1, CS2: Coordinate system (x _o , y _o ), (x _t , y _t ): Coordinates T1~T6: Object θ1, θ2, θ3: Angle BB1, BB2: Bounding box

FIG. 1 is a block diagram of components of an image processing system according to an embodiment of the invention. FIG. 2 is a flowchart of an image correction method according to an embodiment of the invention. FIG. 3A is a schematic diagram of dewarp according to an embodiment of the invention. FIG. 3B is a schematic diagram of viewing angle adjustment according to an embodiment of the invention. FIG. 3C is a schematic diagram of zoom adjustment according to an embodiment of the invention. FIG. 3D is a schematic diagram of a shift according to an embodiment of the invention. FIG. 3E is a schematic diagram of adjusting up and down viewing angles according to an embodiment of the invention. FIG. 3F is a schematic diagram of adjusting left and right viewing angles according to an embodiment of the present invention. FIG. 3G is a schematic diagram of plane viewing angle adjustment according to an embodiment of the present invention. FIG. 3H is a schematic diagram of an arrangement of an image capture device and captured images according to an embodiment of the present invention. FIG. 3I is a schematic diagram of the arrangement of the image capture device and the captured images according to an embodiment of the present invention. FIG. 3J is a schematic diagram of an arrangement of an image capture device and captured images according to an embodiment of the present invention. FIG. 4 is a schematic diagram of fisheye image expansion according to an embodiment of the present invention. FIG. 5 is a schematic diagram of object arrangement according to an embodiment of the present invention. FIG. 6 is a schematic diagram of object arrangement of multiple modes according to an embodiment of the present invention. FIG. 7A is a schematic diagram of an object layout of a pattern according to an embodiment of the present invention. FIG. 7B is a schematic diagram of an object layout of a pattern according to an embodiment of the present invention. FIG. 7C is a schematic diagram of an object layout of a pattern according to an embodiment of the invention. FIG. 8 is a schematic diagram of coordinate system transformation of a model according to an embodiment of the invention. FIG. 9 is a schematic diagram of a second image under a viewing angle according to an embodiment of the present invention. FIG. 10 is a schematic diagram of a second image with a rotated viewing angle according to an embodiment of the present invention. FIG. 11 is a schematic diagram of a corrected second image according to an embodiment of the invention. FIG. 12A is a schematic diagram of a second image of a meeting situation according to an embodiment of the present invention. FIG. 12B is a schematic diagram of a second image of a meeting situation according to another embodiment of the present invention. FIG. 12C is a schematic diagram of a corrected second image according to another embodiment of the present invention. FIG. 13 is a schematic diagram of a multi-object window image according to an embodiment of the present invention. FIG. 14A is a schematic diagram of a second image of a multi-object window according to an embodiment of the invention. FIG. 14B is a schematic diagram of a corrected second image according to an embodiment of the present invention.

S210~S270: steps

Claims (31)

An image correction method, comprising: obtain a first image; transforming the first image into a second image according to a transformation operation, wherein the transformation operation includes at least one deformation correction, and the deformation correction is used to correct the deformation of at least one object in the first image; detecting the at least one target in the second image to generate a detection result; and The conversion operation is corrected according to the detection result. The image correction method as described in claim 1, comprising: obtaining the first image from an image capture device through a first controller; converting the first image into the second image according to the conversion operation through the first controller; Detecting the at least one target in the second image through a second controller to generate the detection result; and The conversion operation is corrected according to the detection result through the first controller. The image correction method as described in claim 1, wherein correcting the conversion operation according to the detection result includes: The conversion operation further includes a position adjustment to adjust the position of the at least one target in the second image. The image correction method as described in claim 3, wherein the detection result includes an offset angle of a first object in the at least one object relative to a reference axis in the second image, and the position adjustment includes : The conversion operation is set to rotate a first Field of View (FoV) of the first image according to the offset angle to reduce the offset angle. The image correction method as described in claim 4, wherein the conversion operation further includes an object layout, the second image includes a plurality of windows, and the object layout is used to adjust an object window of the first object in the second image , the target window is one of the windows, and the step of setting the conversion operation as rotating the first viewing angle of the first image according to the offset angle includes: Rotate the first imaging angle to align the first object with the object window. The image correction method as described in claim 4, wherein the detection result includes that the rotated first imaging angle exceeds a second imaging angle of the image capture device, and the position adjustment further includes: A portion of the rotated first viewing angle beyond the second viewing angle is limited to an edge of the second viewing angle. The image correction method as described in claim 4, wherein the step of setting the conversion operation as rotating the first imaging angle of view of the first image according to the offset angle includes: The coordinates of the first image are transformed according to the ratio between the first viewing angle and the length of the first image on the axis corresponding to the direction in which the first viewing angle is rotated. The image correction method as described in claim 1, wherein the detection result includes a size ratio of a second target in the at least one target object in the second image, and the conversion operation is set to change according to the size ratio A zoom ratio from the first image to the second image. The image correction method as described in claim 8, wherein the step of setting the conversion operation as changing the zoom ratio from the first image to the second image according to the size ratio includes: Setting the scaling ratio to the smallest of a width ratio, a height ratio and a maximum ratio, wherein the second target is positioned with a bounding box, the size ratio includes the height ratio and the width ratio, and the width The ratio is the ratio of the width of the second image to the width of the bounding box of the second object, and the height ratio is the ratio of the height of the second image to the height of the bounding box. The image correction method as described in claim 8, wherein the step of setting the conversion operation as changing the zoom ratio from the first image to the second image according to the size ratio includes: The zoom ratio is set as a reference ratio, wherein the second object is positioned with a representative point. The image correction method as described in claim 5, wherein the detection result includes a size ratio of a third target in the at least one target object in a target window in the second image, and the target window is the windows One of them, and the steps of modifying the conversion operation according to the detection result include: The conversion operation is set to change a scaling factor of the third object according to the size ratio. The image correction method as described in claim 5, wherein the detection result includes that a position of the at least one target object in the second image does not change, and the deformation correction only corrects the range corresponding to the position in the first image deformation. The image correction method as described in claim 1, wherein the second controller uses a first coordinate system to locate pixels in the second image, and the first controller uses a second coordinate system to locate pixels in the second image pixel, the detection result includes a target position of a fourth target of the at least one target object in the second image, and the step of correcting the conversion operation according to the detection result further includes: The coordinates of the target position are converted from the first coordinate system to the coordinates of the second coordinate system, wherein the second coordinate system takes the center point as the origin, and the first coordinate system takes the upper left corner as the origin. An image processing system, comprising: An image capture device, including a lens and an image sensor, and used to capture a first image through the lens and the image sensor; a first controller, coupled to the image capture device, and used for converting the first image into a second image according to a conversion operation, wherein the conversion operation includes a deformation correction, and the deformation correction is used to correct the first deformation of at least one object in the image; and a second controller, coupled to the first controller, and used to detect the at least one target object in the second image to generate a detection result, wherein the first controller is further used to detect according to the detection The result fixes the conversion job. The image processing system as described in claim 14, wherein the first controller modifying the conversion operation includes: The conversion operation further includes a position adjustment to adjust the position of the at least one target in the second image. The image processing system according to claim 15, wherein the detection result includes an offset angle of a first object in the at least one object relative to a reference axis in the second image, and the first control The device is more used to: The conversion operation is set to rotate a first viewing angle of the first image according to the offset angle so as to reduce the offset angle. The image processing system as claimed in claim 16, wherein the conversion operation further includes an object layout, the second image includes a plurality of windows, and the object layout is used to adjust an object window of the first object in the second image , the target window is one of the windows, and the first controller is further used to: Rotate the first imaging angle to align the first object with the object window. The image processing system as claimed in claim 16, wherein the detection result includes that the rotated first imaging angle exceeds a second imaging angle of the image capture device, and the first controller is further configured to: A portion of the rotated first viewing angle beyond the second viewing angle is limited to an edge of the second viewing angle. The image processing system as claimed in claim 16, wherein the first controller is further used for: The coordinates of the first image are transformed according to the ratio between the first viewing angle and the length of the first image on the axis corresponding to the direction in which the first viewing angle is rotated. The image processing system as described in claim 14, wherein the detection result includes a size ratio of a second target in the at least one target object in the second image, and the conversion operation is set to change according to the size ratio A zoom ratio from the first image to the second image. The image processing system as described in claim 20, wherein the zoom ratio is set to the smallest of a width ratio, a height ratio and a maximum ratio, the second object is positioned with a certain bounding box, and the size ratio includes the height ratio and the width ratio, the width ratio is the ratio of the width of the second image to the width of the bounding box of the second object, and the height ratio is the ratio of the height of the second image to the height of the bounding box. The image processing system as claimed in claim 20, wherein the zoom ratio is set as a reference ratio, and the second object is positioned with a representative point. The image processing system as claimed in claim 17, wherein the detection result includes a size ratio of a third object in the at least one object in a target window in the second image, and the target window is the windows one of, and the first controller is further used to: The conversion operation is set to change a scaling factor of the third object according to the size ratio. The image processing system as claimed in claim 17, wherein the detection result includes that a position of the at least one target object in the second image does not change, and the deformation correction only corrects the range corresponding to the position in the first image deformation. The image processing system according to claim 14, wherein the second controller positions pixels in the second image with a first coordinate system, and the first controller positions pixels in the second image with a second coordinate system pixels, the detection result includes a target position of a fourth target of the at least one target in the second image, and the first controller or the second controller is further used to: The coordinates of the target position are converted from the first coordinate system to the coordinates of the second coordinate system, wherein the second coordinate system takes the center point as the origin, and the first coordinate system takes the upper left corner as the origin. An image controller, suitable for coupling with an image capture device, the image capture device includes a lens and an image sensor, the image capture device captures a first image through the lens and the image sensor , the image controller consists of: a memory for storing a program code; and A processor, coupled to the memory, configured to load and execute the program code to: obtain the first image; transforming the first image into a second image according to a transformation operation, wherein the transformation operation includes a deformation correction, and the deformation correction is used to correct deformation of at least one object in the first image; detecting the at least one target in the second image to generate a detection result; The conversion operation is corrected according to the detection result. The image controller as claimed in claim 26, wherein modifying the conversion operation according to the detection result includes: making the conversion operation further include a position adjustment to adjust the position of the at least one target in the second image. The image controller according to claim 27, wherein the detection result includes an offset angle of a first object in the at least one object relative to a reference axis in the second image, and the image controller The conversion operation is set to rotate a first viewing angle of the first image according to the offset angle so as to reduce the offset angle. The image controller as recited in claim 28, wherein the conversion operation further includes an object layout, the second image includes a plurality of windows, the object layout is used to adjust an object window of the first object in the second image , the target window is one of the windows, and the image controller is further used to: rotating the first imaging angle to align the first object with the object window; Wherein, the detection result includes a size ratio of a third target in the at least one target object in a target window in the second image, the target window is one of the windows, and the image controller More to: The conversion operation is set to change a scaling factor of the third object according to the size ratio. The image controller as claimed in claim 28, wherein the detection result includes that the rotated first viewing angle exceeds a second viewing angle of the image capture device, and the image controller is further configured to: A portion of the rotated first viewing angle beyond the second viewing angle is limited to an edge of the second viewing angle. The image controller as described in claim 28 further converts the coordinates of the first image according to the ratio between the first viewing angle and the length of the first image on the axis corresponding to the direction in which the first viewing angle is rotated .

Images