WO2018014730A1 - Method for adjusting parameters of camera, broadcast-directing camera, and broadcast-directing filming system - Google Patents

Abstract

Disclosed in the embodiments of the present invention are a method for adjusting the parameters of a camera, a broadcast-directing camera, and a broadcast-directing filming system. The method comprises: determining a target video object to be filmed; according to a preset directing policy, screening a target camera for filming the target video object, from cameras of a broadcast-directing filming system in which the broadcast-directing camera is located; acquiring the first three-dimensional coordinates of the target video object, the first three-dimensional coordinates being the three-dimensional coordinates of the target video object in a first coordinate system corresponding to the target camera; and adjusting the filming parameters of the target camera to be filming parameters corresponding to the first three-dimensional coordinates, and outputting video images after the filming parameters have been adjusted. Using the present solution, the efficiency of camera parameter adjustment can be improved, and the filming effect of the camera can be enhanced.

Description

Camera parameter adjustment method, guide camera and system

This application claims priority to Chinese Patent Application No. 201610562671.6, entitled "A Camera Parameter Adjustment Method, Guide Camera and System", filed on July 18, 2016, the entire contents of which are incorporated by reference. In this application.

Technical field

The present invention relates to the field of image processing technologies, and in particular, to a camera parameter adjustment method, a navigation camera, and a system.

Background technique

With the continuous development of image processing technology and the Internet, there are more and more scenes requiring video conferencing, which brings great convenience for remote communication between users. Currently, when video conferencing is performed, it is often necessary to deploy multiple cameras for shooting to obtain a frontal image of the participants. For example, refer to FIG. 1. FIG. 1 is a schematic diagram of a video conference. As shown in Fig. 1, in this scenario, the conference room adopts a long elliptical conference table, the participants' seats surround the conference table, the participants include A and B, and the participants A and B sit opposite each other. Cameras C0 and C1 are arranged on both sides of the projection screen in front of B and B. For A, only the front image of A can be captured by camera C0, and camera C1 cannot capture the front of A. For B, the front image of B can only be captured by camera C1, and camera C0 cannot capture the front of B. It can be seen that multiple cameras are required for shooting to achieve video conferencing.

When deploying multiple cameras for conference shooting, the camera parameters are usually adjusted manually by remote control or other means to obtain a better shooting result. However, the manual adjustment method requires the operator to have certain camera expertise and the operation process is cumbersome, which makes the adjustment efficiency low, and the better shooting effect cannot be guaranteed in time. In addition, the camera can be determined by sound source positioning and the camera shooting effect can be adjusted. The sound source is positioned by locating and tracking the participant who is speaking (ie, "speaker"), while using a camera to capture the close-up of the speaker, while tracking the speaker's face position and performing PTZ (for the lens) Pan Tilt Zoom, which is the “Pan, Tilt, Zoom” adjustment, so that the speaker's face is in the middle of the image. However, the sound source localization method only considers adjusting the front side of the speaker to the center of the image, and does not consider the effect of the image being taken, nor can it guarantee a better shooting effect.

Summary of the invention

The embodiment of the invention provides a camera parameter adjustment method, a guide camera and a system, which can improve the efficiency of camera parameter adjustment and improve the camera shooting effect.

In a first aspect, an embodiment of the present invention provides a camera parameter adjustment method, where the method is applied to a navigation camera, including:

Determining the target video object that needs to be captured;

Filtering from each camera of the navigation camera system where the navigation camera is located according to a preset navigation strategy Shooting a target camera of the target video object;

Obtaining a first three-dimensional coordinate of the target video object, where the first three-dimensional coordinate is a three-dimensional coordinate of the target video object in a first coordinate system corresponding to the target camera;

And adjusting an imaging parameter of the target camera to an imaging parameter corresponding to the first three-dimensional coordinate, and outputting a video image after adjusting the imaging parameter.

The first three-dimensional coordinate may be a three-dimensional coordinate of the target video object in a first coordinate system corresponding to the target camera. The target camera may be a navigation camera or an ordinary PTZ camera, and the first coordinate system corresponding to the target camera may be a three-dimensional coordinate system established with the optical center of the target camera as an origin, or established by using any other reference object as an origin. The three-dimensional coordinate system is not limited in the embodiment of the present invention.

The target video object may be any one or more video objects in the shooting scene corresponding to the navigation camera system where the navigation camera is located.

In some embodiments, the acquiring the first three-dimensional coordinates of the target video object comprises:

Obtaining a second three-dimensional coordinate transmitted by the binocular camera connected to the navigation camera, where the second three-dimensional coordinate is a three-dimensional coordinate of the target video object in a second coordinate system corresponding to the binocular camera;

The second three-dimensional coordinates are converted into first three-dimensional coordinates according to a pre-calibrated positional relationship between the binocular camera and the target camera.

In some embodiments, the second three-dimensional coordinates may be two-dimensional coordinates of the binocular camera obtained by respectively acquiring video objects in left and right views of the binocular camera and the binocular camera acquired The internal and external parameters of the data are calculated.

The second three-dimensional coordinate is a three-dimensional coordinate of the target video object in a second coordinate system corresponding to the binocular camera, and the second coordinate system corresponding to the binocular camera may be a binocular camera optical center The three-dimensional coordinate system established by the origin, or the three-dimensional coordinate system established with the origin of any other reference object. The two-dimensional coordinates may specifically be corresponding pixel coordinates of the target video object in the left view and the right view of the binocular camera.

In some embodiments, the determining a target video object that needs to be captured comprises:

Obtaining a captured image transmitted by a binocular camera, the captured image including at least one video object;

Establishing a video object model including the at least one video object, and determining a target video object from the at least one video object;

The target camera for capturing the target video object is filtered out from each camera of the navigation camera system where the navigation camera is located according to a preset navigation policy, including:

Determining the target video object from the captured images acquired by the cameras in the navigation camera system, and acquiring shooting effect parameters of the target video object in each camera;

The camera whose shooting effect parameter satisfies the preset guiding strategy is determined as the target camera for capturing the target video object.

In some embodiments, determining the target video object from the captured image acquired by the camera comprises:

Converting the second three-dimensional coordinates into third three-dimensional coordinates according to a pre-calibrated positional relationship between the binocular camera and the current camera;

Determining whether an area of the target video object in the third three-dimensional coordinate and a region of the video object detected by the current camera in the three-dimensional coordinate of the video object exceed a preset area threshold;

If exceeded, the video object is determined to be the target video object.

The current camera is any camera other than the binocular camera that is in a positional relationship with the navigation camera in the navigation camera system, and the third three-dimensional coordinate is that the target video object corresponds to the current camera. The three-dimensional coordinates in the third coordinate system.

In some embodiments, the shooting effect parameter includes any one or more of an eye-to-eye effect parameter, an occlusion relationship parameter, and a scene object parameter of the shooting area of the target video object in a coordinate system corresponding to the current camera. The current camera is any camera other than the binocular camera in the navigation camera system.

The eye-to-eye effect parameter may include a rotation angle of the target video object relative to a coordinate system corresponding to the current camera, the rotation angle being according to a rotation angle of the target video object in the second coordinate system and the binocular pre-calibrated The positional relationship between the camera and the current camera is determined. The smaller the angle of rotation, the better the eye-to-eye effect.

The occlusion relationship parameter and the scene object parameter may be determined according to a pre-calibrated positional relationship between the binocular camera and the current camera, and the region of the scene object detected by the current camera is re-projected to an imaging plane of the current camera. of. The output image is better when there is no occlusion relationship (the smaller the occlusion relationship parameter is). The smaller the area of the scene object and the smaller the number, the better the output image effect; otherwise, the worse the output image effect.

In a second aspect, an embodiment of the present invention further provides a navigation camera, including: a memory and a processor, wherein the processor is connected to the memory;

The memory is used to store driver software;

The processor reads the driver software from the memory and performs some or all of the steps of the camera parameter adjustment method of the first aspect described above by the driver software.

In a third aspect, the embodiment of the present invention further provides a parameter adjustment apparatus, including an object determining unit, a selecting unit, an acquiring unit, and a parameter adjusting unit, wherein the parameter adjusting device implements a part of the camera parameter adjusting method of the first aspect by using the foregoing unit. Or all steps.

In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, where the computer storage medium stores a program, and the program includes some or all of the steps of the camera parameter adjustment method of the first aspect.

In a fifth aspect, the embodiment of the present invention further provides a navigation camera system, including a first camera and at least one second camera, the first camera includes a navigation camera and a binocular camera, and the navigation camera and the binocular Between the cameras, the first camera and the second camera are connected by a wired interface or a wireless interface;

The guidance camera is configured to determine a target video object that needs to be captured, and select a target camera for capturing the target video object from a camera of the navigation camera system according to a preset navigation strategy;

The binocular camera is configured to acquire a second three-dimensional coordinate of the target video object, and transmit the second three-dimensional coordinate to the navigation camera; wherein the second three-dimensional coordinate is that the target video object is The three-dimensional coordinates of the binocular camera corresponding to the second coordinate system;

The guidance camera is configured to receive the second three-dimensional coordinates transmitted by the binocular camera; convert the second three-dimensional coordinates into a first position according to a pre-calibrated positional relationship between the binocular camera and the target camera a three-dimensional coordinate; adjusting an imaging parameter of the target camera to an imaging parameter corresponding to the first three-dimensional coordinate, and outputting a video image after adjusting the imaging parameter; wherein the first three-dimensional coordinate is the target video object The three-dimensional coordinates in the first coordinate system corresponding to the target camera.

In some embodiments, the second camera may include a navigation camera and a binocular camera, and the target camera may be any of the navigation cameras in the navigation camera system; or the second camera may also be a normal PTZ camera. Then the target camera can be the guide camera or a normal PTZ camera.

In some embodiments, the binocular camera can be placed on a preset guide bracket and connected to the guide camera via the guide bracket.

Embodiments of the present invention have the following beneficial effects:

In the embodiment of the present invention, after determining the target video object that needs to be captured, the target camera that captures the target video object is selected from the cameras of the navigation camera system according to a preset guiding strategy, and is obtained. a three-dimensional coordinate of the target video object in a coordinate system corresponding to the target camera, to control the target camera to perform camera parameter adjustment according to the three-dimensional coordinates of the target video object, and output a video image after adjusting the imaging parameter, so that the guided camera system It can improve the accuracy of video object detection and tracking based on three-dimensional coordinate detection and preset navigation strategy, and improve the efficiency of camera parameter adjustment, and effectively improve the camera's shooting effect.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any creative work.

1 is a schematic diagram of a scene of a video conference;

FIG. 2 is a schematic flowchart of a method for adjusting a camera parameter according to an embodiment of the present invention;

FIG. 3a is a schematic diagram of a camera imaging model according to an embodiment of the present invention; FIG.

FIG. 3b is a schematic diagram of a calibration scenario of a multi-camera according to an embodiment of the present invention; FIG.

FIG. 3c is a schematic diagram of a three-dimensional positioning of a binocular camera according to an embodiment of the present invention; FIG.

FIG. 3 is a schematic diagram of a PTZ camera rotation model according to an embodiment of the present invention; FIG.

4a is a schematic diagram of a video object matching scenario according to an embodiment of the present invention;

Figure 4b is an image view of a set of video objects in Figure 4a;

FIG. 5 is a schematic structural diagram of a parameter adjustment apparatus according to an embodiment of the present invention; FIG.

6 is a schematic structural diagram of a navigation camera system according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a first camera according to an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of networking of a navigation camera system according to an embodiment of the present invention; FIG.

FIG. 9 is a schematic structural diagram of a navigation camera according to an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

It should be understood that the "first", "second", and "third" and the like in the embodiments of the present invention are used to distinguish different objects, and are not used to describe a specific order. Moreover, the term "comprise" and any variants thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that contains a series of steps or units is not limited to the listed Steps or units, but optionally also include steps or units not listed, or alternatively other steps or units inherent to such processes, methods, products or devices.

It should be understood that the navigation camera according to the embodiment of the present invention may be specifically a PTZ camera for performing the technical solution of the embodiment of the present invention, which can be connected to a binocular camera, and the navigation camera can be applied to a scene such as a conference or a training, and The deployment of the position and number of the guided cameras is performed according to different scenarios.

In some embodiments, the binocular camera can be mounted on a guide bracket, that is, the guide camera can be coupled to the binocular camera via a guide bracket (referred to as a "bracket"). The navigation camera is used for guiding shooting and tracking. In addition, a microphone can be mounted on the bracket, which can be used for sound source localization, sound source recognition and the like. The camera and the bracket may be separate or integrated, and a communication interface such as a serial interface may be used for communication between the camera and the bracket.

In some embodiments, the binocular camera can be used for video capture, video pre-processing, motion detection, face detection, humanoid detection, scene object detection, feature detection/matching, binocular camera calibration, multi-camera calibration, etc., microphones are available For audio acquisition, audio pre-processing, video capture, sound source behavior recognition, etc., the camera can be used for audio and video (Audio Video, referred to as "AV") object 3D positioning, AV object modeling, AV object tracking, motion / gesture Identification, guided control and video switching/synthesis, etc. The video capture includes synchronously collecting video streams of the binocular camera and the guide camera; the video pre-processing includes pre-processing the input binocular image, such as performing noise reduction, changing resolution and frame rate, etc.; motion detection includes detecting the scene. The moving object in the moving object and the stationary background are separated to obtain the moving object area; the face detection includes detecting the face target object in the scene, and outputting the detection information of the face, such as the face position, area, direction The information detection includes detecting the human head and shoulder area in the scene and outputting detection information; the scene object detection includes detecting objects other than people in the scene, such as a lamp, a window, a conference table, etc.; feature detection/matching includes Perform feature detection and matching on the detected moving object area, detect characteristic objects (such as feature points) in one image and match in another image, and output matching feature object information; binocular camera calibration includes binocular camera Perform calibration to obtain the internal and external parameters of the binocular camera for calculating the video image. Three-dimensional coordinates of the video object; multi-camera calibration comprises relative positional relationship between the plurality of cameras directed be calibrated to obtain the reference information relative to a plurality of external cameras directed, for positioning a plurality of video objects in the camera coordinate system. Further, the audio collection includes synchronously acquiring the multi-channel audio data of the microphone; the audio pre-processing includes performing 3A processing on the input multi-channel audio data, wherein the 3A processing includes automatic exposure control (AE), auto focus control (AF), and auto white Balance control (AWB); sound source localization includes detecting input multi-channel audio data to find two-dimensional position information of the sounding object; sound source behavior recognition includes detecting and counting the voice behavior of the video object in the scene. Further, the 3D positioning of the AV object includes obtaining the depth information of the object feature in the image according to the parallax information obtained by the internal and external parameters of the binocular camera and the feature detection/matching, and combining the result of the audio positioning to obtain the object feature in a single guided camera coordinate system. The three-dimensional position information can obtain the position information of the feature in other navigation camera coordinate systems according to the position of the feature in a single navigation coordinate system and the relative positional relationship of the plurality of navigation cameras; the AV object modeling includes combining the sound source positioning and the human face. Information such as information, feature objects, and scene objects constructs a model of the AV object; the AV object tracking includes tracking a plurality of AV objects in the scene, and updating state information of the object; and the motion/gesture recognition includes performing actions, gestures, and the like of the AV object. Identifying, for example, identifying a standing posture of a subject, a gesture action, etc.; the navigation control includes determining a navigation strategy in conjunction with the result of the motion/gesture recognition and the sound source behavior recognition, and the navigation camera controls the control instruction corresponding to the output guidance strategy, the video object and the scene feature information, and the video. Output strategy, etc. The camera control command can be used to control the PTZ camera to perform PTZ operation, that is, pan, tilt, zoom operation, etc., and the video object and scene feature information can be used for information sharing between multiple guide cameras. The frequency output strategy can be used to control the output strategy of a single or multiple camera video streams.

The embodiment of the invention provides a camera parameter adjustment method, a guide camera and a system, which can improve the efficiency of camera parameter adjustment and improve the camera shooting effect. The details are explained below.

Further, please refer to FIG. 2. FIG. 2 is a schematic flowchart diagram of a camera parameter adjustment method according to an embodiment of the present invention. Specifically, the method of the embodiment of the present invention may be specifically applied to the above-mentioned navigation camera. As shown in FIG. 2, the camera parameter adjustment method in the embodiment of the present invention may include the following steps:

101. Determine a target video object that needs to be shot.

102. Filter, according to a preset guiding strategy, a target camera for capturing the target video object from each camera of the guiding camera system where the guiding camera is located.

Optionally, the determining the target video object that needs to be captured may be specifically: acquiring a captured image transmitted by the binocular camera, the captured image includes at least one video object; and establishing a video object model including the at least one video object And determining a target video object from the at least one video object. Further, the target camera for capturing the target video object is selected from each camera of the navigation camera system in which the navigation camera is located according to a preset navigation strategy, which may be specifically: separately from the navigation camera system Determining the target video object in the captured image acquired by each camera, and acquiring a shooting effect parameter of the target video object in each camera; determining, by the camera that the shooting effect parameter meets the preset guiding strategy, is used for shooting The target camera of the target video object. One or more navigation cameras can be deployed in the navigation camera system, that is, the navigation camera system can be deployed as a guide camera + a guide camera, and can also be deployed as a guide camera + a normal camera (such as a normal PTZ camera). . Specifically, the video object model may include all video objects in the shooting scene corresponding to the navigation camera system where the guidance camera is located. If the other cameras in the navigation camera system also include the guidance camera, the captured images transmitted by the binocular cameras connected to the other navigation cameras may also be received, and the video object model is updated to obtain the video objects of all the video objects in the captured scene. model. The target video object may be any one or more video objects in the shooting scene.

Optionally, the shooting effect parameter may include any one or more of an eye-to-eye effect parameter, an occlusion relationship parameter, and a scene object parameter of the shooting area of the target video object in a coordinate system corresponding to the current camera. The current camera is any camera other than the binocular camera in the navigation camera system, that is, the current camera may be any of the guidance cameras or ordinary PTZ cameras in the guidance camera system.

The eye-to-eye effect parameter may include a rotation angle of the target video object with respect to a coordinate system corresponding to the current camera, and the rotation angle may be a rotation angle of the target video object in the second coordinate system. And determining a positional relationship between the binocular camera and the current camera that is pre-calibrated. Specifically, the rotation angle of the target video object relative to the coordinate system corresponding to the current camera may refer to an optical axis angle of the face or the humanoid object corresponding to the target video object relative to the current camera (the navigation camera or the ordinary PTZ camera). The smaller the angle, the more the face is rendered in a positive face manner, that is, the better the eye-to-eye effect, the better the output image effect.

The occlusion relationship parameter and the scene object parameter may be that the area of the scene object detected by the current camera is re-injected to the location according to a pre-calibrated positional relationship between the binocular camera and the current camera. The imaging plane of the current camera is determined. Specifically, if the areas of the two video objects overlap, the depth information may be used to determine an occlusion relationship between the two objects, and the video object that is closer to the binocular camera may block the farther video object. The output image is better when there is no occlusion relationship (the smaller the occlusion relationship parameter is). The scene object indicated by the scene object parameter may include a light tube, a window, a table, and the like, and the smaller the area of the scene object is, the smaller the number is, the better the output image effect is; Conversely, the worse the output image is.

103. Acquire a first three-dimensional coordinate of the target video object.

The first three-dimensional coordinate may be a three-dimensional coordinate of the target video object in a first coordinate system corresponding to the target camera. The target camera can be configured as the above-mentioned navigation camera or a normal PTZ camera. The first coordinate system corresponding to the target camera can be a three-dimensional coordinate system established with the target camera's optical center as the origin, or any other reference object. The three-dimensional coordinate system established by the origin is not limited in the embodiment of the present invention.

Optionally, the navigation camera can be connected to a preset binocular camera. The acquiring the first three-dimensional coordinates of the target video object may be specifically: acquiring a second three-dimensional coordinate transmitted by the binocular camera connected to the navigation camera; and according to the pre-calibrated the binocular camera and the target The positional relationship of the camera converts the second three-dimensional coordinates into first three-dimensional coordinates. Further, the second three-dimensional coordinates may be two-dimensional coordinates of the target video object acquired in the left view and the right view of the binocular camera and the binocular acquired by the binocular camera respectively The internal and external parameters of the camera are calculated. The second three-dimensional coordinate is a three-dimensional coordinate of the target video object in a second coordinate system corresponding to the binocular camera, and the second coordinate system corresponding to the binocular camera may be a binocular camera optical center The three-dimensional coordinate system established by the origin, or the three-dimensional coordinate system established with the origin of any other reference object. The two-dimensional coordinates may specifically be corresponding pixel coordinates of the target video object in the left view and the right view of the binocular camera.

In a specific embodiment, the positional relationship between the binocular cameras, the positional relationship between the camera and the binocular camera, and the positional relationship between the cameras of the multi-camera in the camera system can be calibrated in advance. The parameter obtained by the binocular camera system calibration can be used to calculate the three-dimensional coordinates of the video object in the coordinate system corresponding to the binocular camera; the positional relationship calibration between the navigation camera and the binocular camera can be used to calculate the video object in the navigation camera coordinate system. The three-dimensional coordinates of the lower three-dimensional coordinates; and the positional calibration parameters between the cameras of the multi-camera can be used to calculate the three-dimensional coordinates of the video object in the camera coordinate system of each aircraft position in the multi-camera deployment scenario, so as to facilitate coordinate conversion. The deployment mode of the multi-camera camera may be the deployment mode of the above-mentioned guide camera + guide camera, or the deployment mode of the guide camera + ordinary PTZ camera. Each of the navigation cameras can be referred to as a single position. When multiple cameras are used for cooperative shooting, a host position can be determined therefrom, and the rest is a slave position. As a guide camera of the slave position, the camera can be used. Information such as IP is registered to the host bit, so that the host bit can manage multiple slave locations. Specifically, the calibration process is briefly described below. The binocular camera includes a left camera and a right camera. The image acquired by the left camera may be referred to as a left view, and the image acquired by the right camera may be referred to as a right view. Then the imaging (projection) model of a single camera can be described by the following formula:

x=PX=K[R|t]X

As shown in FIG. 3a, x is a pixel coordinate of a certain point in the scene (ie, a video object, specifically a feature point corresponding to the video object) in the image coordinate system, which is a two-dimensional coordinate; X is a certain point in the scene. Position coordinates in the world coordinate system; P is a 3×4 projection matrix. PX means P×X. Where K is a 3×3 camera internal reference matrix, which can be expressed as:

Where f _x , f _y are the equivalent focal lengths in the x and y directions, c _x , c _y are the image coordinates of the optical center, and s is the skew coefficient of the skew (the sensor and the optical axis are not perpendicular, usually small, during the calibration process) Ignorable).

In addition, R and t are camera external parameters, which are represented as a 3×3 rotation matrix and a 3×1 translation vector, respectively, as follows:

R=[r ₁ r ₂ r ₃ ]

t=[t ₁ t ₂ t ₃ ] ^T

Where r ₁ , r ₂ , r ₃ are 3×1 column vectors in the rotation matrix.

Due to the optical characteristics of the camera lens, the manufacture and installation of the image sensor, etc., the actual image taken by the camera is not ideal and there will be distortion, so the image distortion can be modeled to obtain the ideal image. Specifically, the model of camera image distortion can be described according to the following formula:

Where x _p , y _p are the corrected pixel positions, x _d , y _d are the pre-correction pixel positions, k ₁ , k ₂ , k ₃ are radial distortion coefficients, and p ₁ , p ₂ are tangential distortion coefficients.

Based on the imaging model of the monocular camera described above, when the world coordinate system is known to be transformed to the rotation matrix R1 and R2 and the translation vectors t1 and t2 of the left camera coordinate system and the right camera coordinate system, the relative between the binocular cameras can be obtained. External parameters, including the rotation matrix R and the translation vector T:

It should be understood that, in the embodiment of the present invention, the positional relationship between the binocular cameras and the positional relationship between the navigation camera such as the PTZ camera and the binocular camera are fixed, and the two calibrations can be completed before leaving the factory, that is, The data obtained by the two calibrations is fixed as the internal and external data. Optionally, in the embodiment of the present invention, the calibration of the camera may adopt various schemes, such as the plane calibration method of Zhang (also referred to as “Zhang's calibration method”), and the distortion parameter calculation adopts the Brown method, which is not described herein. .

Further, the above-mentioned binocular camera calibration principle shows that the calibration of the positional relationship of a multi-camera camera such as a multi-camera camera is to find a relative external parameter between two adjacent guide cameras, according to the position between adjacent guide cameras. The relative external parameters calculate the external parameters between any two navigation cameras, thereby obtaining the positional relationship between any two navigation cameras. When the multi-guide camera is deployed, a large overlapping area of the camera is required between the two cameras. The multiple positions are similar to the surrounding multi-camera system. The rotation matrix and translation vector of the i-th camera relative to the j-th camera are:

R _i,i-1 R _i-1,i-2 ...R _j+1,j

R _i,i-1 R _i-1,i-2 ...R _j+2,j+1 T _j +R _i,i-1 R _i-1,i-2 ...R _j+3,j+2 T _j+1 +...+R _i,i-1 T _i-2 +T _i-1

Wherein R _i,i-1 R _i-1,i-2 ...R _j+1,j denotes R _i,i-1 ×R _i-1,i-2 ×...×R _j+1,j . Since the position of the camera for positioning on different guide brackets changes according to the actual deployment scenario when the guide camera is deployed, the positional relationship between the multi-guide cameras cannot be pre-calibrated before the device leaves the factory, and can be deployed in the guide camera. Perform on-site calibration.

Further, it is assumed that the cameras in the navigation camera system are all guided cameras, and each of the navigation cameras is connected with a binocular camera. Referring to FIG. 3b, FIG. 3b is a schematic diagram of a calibration scenario of a multi-guide camera according to an embodiment of the present invention. As shown in FIG. 3b, in the calibration, a local area network (LAN) or a wireless Fidelity (Local Area Network, simply referred to as "LAN") or a wireless fidelity can be used between two adjacent navigation cameras or between a binocular camera and a navigation camera. Referred to as "Wi-Fi" network for communication, including transmission of calibration template images and calibration parameters, etc., the transmission protocol can use a variety of network protocols, such as the HyperText Transfer Protocol ("HTTP"). Specifically, a global ID number can be assigned in advance to each of the navigation camera and the camera (binocular camera) connected to the navigation camera. For example, you can select a camera as the starting position. For example, you can select the leftmost or rightmost camera as the starting position. The ID numbers of other cameras are incremented counterclockwise or clockwise. A group of cameras is selected among all cameras to participate in the calibration. The principle of selection may be to ensure that the overlapping area between adjacent cameras is the largest. As shown in FIG. 3b, it is assumed that three positions D1, D2, and D3 are deployed in the current shooting scene, and each of the positions includes a navigation camera (recorded as PTZ0, PTZ1, and PTZ2, respectively) and a binocular camera (the binocular camera). Includes two left and right cameras, denoted as C0, C1, C2, C3, C4, C5). Suppose that cameras with ID numbers C0, C2, and C4 are selected for calibration, and one of the navigation cameras is selected as the calibration computing device, such as the above-mentioned navigation camera for the host position. Further, the external reference calibration between the two cameras can be performed from left to right or from right to left, and the relative external parameters between the two cameras are obtained. Optionally, a camera relative position relationship table can be maintained in each navigation camera, as shown in Table 1 below. Among them, each of the tables will add or update one of the entries, and each entry is uniquely determined by its two camera ID numbers.

Table I

Entry Camera ID1 Camera ID2 Relative external reference 1 C0 C2 R ₀₂ , T ₀₂ 2 C2 C4 R ₂₄ , T ₂₄ ... ...... ... ......

After the calibration is completed, the positional relationship table, which is the calibration computing device, can be sent to all other navigation cameras for storage via the network. Further, according to the position relationship table and the external parameters of the binocular camera (the external reference can be calibrated at the factory), any two or two cameras in the calibration scene can be calculated (including between binocular cameras, binocular) The positional relationship between the camera and the PTZ camera and between the PTZ cameras.

For example, if the navigation camera D3 in FIG. 3b is a calibration computing device, and the navigation cameras D1 and D2 are cameras to be calibrated, the navigation camera D3 can be set as the calibration host position, and the other cameras are set as slaves. The position is set and the calibration is initiated by the camera D3. Before calibration, it is necessary to ensure that the cameras are interconnected through the network. The overlapping areas can be captured between the cameras that need to be calibrated, and there are calibration templates (such as checkerboard templates) in the overlap area. The calibration camera is required to start the calibration process, and the image acquisition command is sent to the navigation camera D1. The acquisition command includes the ID number of the navigation camera D1 (ie, D1) and the ID number of the binocular camera to be acquired (C4 or C5). . After receiving the acquisition command, the navigation camera D1 performs image acquisition of the calibration template, and transmits the collected image data to the navigation camera D3. Similarly, the guide camera D3 obtains the binocular camera on the guide camera D2. The calibration template image taken. If the binocular camera to be calibrated is located on the navigation camera D3, the navigation camera D3 can directly acquire the calibration template image of the binocular camera. After obtaining the calibration template image of the camera to be calibrated, the guide camera D3 can perform checkerboard corner detection on the calibration template images of the two cameras. If the two images can detect all the checkerboard corner points, the collection can be indicated. Successful; otherwise the two images can be discarded and the image reacquired. Further, by changing the position of the calibration template, a plurality of calibration template images of the two cameras that need to be calibrated are cyclically acquired and saved in the navigation camera D3. After the number of calibration template images is met, the camera D3 can be used to perform the camera. Calibration, the internal parameters of each camera have been calibrated at the factory, so it can be used as the initial input value for the calibration. After the calibration is completed, the relative external parameters R and T between the two cameras are obtained, and whether the re-projection (shadow) error is less than a preset threshold is calculated. If the re-injection error is greater than the threshold, the calibration failure may be indicated; otherwise, it may indicate The calibration was successful. After the navigation camera D3 completes the calibration, the positional relationship table can be updated based on the calculated relative position relationship of the camera and transmitted to other navigation cameras.

Further, after realizing the positional relationship between the binocular cameras, the positional relationship between the navigation camera and the binocular camera, and the calibration of the positional relationship between the multi-camera cameras, the video within the shooting range of the navigation camera can be obtained. The object is positioned, and the three-dimensional position information is obtained, so as to determine an appropriate guiding camera position according to the acquired three-dimensional position information, and adjust the parameter of the guiding camera according to the guiding strategy corresponding to the three-dimensional position information, and control the positioning of the guiding camera to be suitable. The location of the video object is taken. The positioning of the video object includes three-dimensional positioning of the binocular camera, single-camera camera such as PTZ camera positioning, and three-dimensional positioning between the cameras of the multi-camera.

Specifically, in the three-dimensional positioning process of the binocular camera, the stereoscopic image captured by the binocular camera can be used to calculate the depth position information of a certain observation point in the camera coordinate system, thereby determining the three-dimensional position information of the observation point. . This method is the same as the principle that the human eye perceives the depth distance, and is called binocular camera ranging. As shown in FIG. 3c, a three-dimensional positioning schematic diagram of a binocular camera is provided. The following describes the ranging principle of the binocular camera system. Among them, P is an observation point in the world coordinate system, and is imaged by two left and right cameras. Wherein, the position of the P point in the physical coordinate system of the left camera is X _L , Y _L , Z _L , and the coordinates of the pixel position of the imaging point in the left view are x _l , y _l ; the position in the physical coordinate system of the right camera is X _R , Y _R , Z _R , the pixel position coordinates of the imaging point in the right view are x _r , y _r , assuming that the relative external parameters of the left and right cameras are R, T; the focal lengths of the left and right cameras are: f _l , f _r . According to the binocular camera model, the relationship between the imaging model of the left and right cameras and the physical coordinate position of the left and right cameras is as follows:

According to the above formula, it can be derived:

Among them, the values of x _l , y _l , x _r , y _r can be obtained by image matching, f _l , f _r , R, T can be obtained by binocular camera calibration, so X _L , Y _L , Z _L can be calculated And the values of X _R , Y _R , Z _R , thereby determining the three-dimensional coordinates of the observation points in the scene in the coordinate system corresponding to the binocular camera.

Further, in the three-dimensional positioning process of the guide camera such as the PTZ camera, the basic purpose of the PTZ camera positioning is to know the physical coordinates of a certain target in the PTZ camera coordinate system, how to position a certain point of the target by rotating the PTZ camera. The specific pixel coordinate position in the image. The physical coordinates of the target in the PTZ camera coordinate system can be calculated by the three-dimensional position of the target in the binocular camera coordinate system and the positional relationship between the binocular camera and the PTZ camera obtained by calibration. FIG. 3 is a schematic diagram of a PTZ camera rotation model according to an embodiment of the present invention. As shown in Fig. 3d, it is assumed that the position coordinates of the desired target point P are x ₀ , y ₀ , the physical coordinate position of the target point P is X, Y, Z, and the pixel coordinate position on the imaging plane is x _c , y _c , then rotate around the X and Y axes respectively, so that the pixel position of the point P coincides with the target position, and the rotation angles of the Pan and Tilt operations can be modeled according to the following formula:

Since the PTZ camera is a zoom camera, it is necessary to obtain a function relationship of the zoom factor Z and the internal parameters such as the focal length and the distortion coefficient. For example, the polynomial can be used to fit the relationship between the zoom factor Z and the focal lengths f _x , f _y to obtain the following relationship:

f _x =a ₀ +a ₁ Z+a ₂ Z ² +...a _n Z ⁿ

f _y =b ₀ +b ₁ Z+b ₂ Z ² +...b _n Z ⁿ

Specifically, under different Z values, the camera internal parameters are obtained, the corresponding f _x , f _y and distortion coefficients are calculated, and the coefficients are fitted using least squares method. Other internal parameters such as distortion coefficients can also be processed in a similar manner. After obtaining the internal parameters of the camera under different Z values, the values of Δp and Δt can be calculated according to the Pan/Tilt model formula.

Further, in the multi-camera camera scene, the captured image sent by the binocular camera connected to other navigation cameras may be acquired, and the video object model is updated after the video object is matched. Determining the target video object from the captured image obtained by the camera, which may be specifically: converting the second three-dimensional coordinate into a third three-dimensional coordinate according to a pre-calibrated positional relationship between the binocular camera and the current camera; Determining whether the area of the target video object in the third three-dimensional coordinate and the area of the video object detected by the current camera in the three-dimensional coordinate of the video object exceeds a preset area threshold; if exceeded, The video object is then determined as the target video object. That is, the video object matches successfully. The third three-dimensional coordinate is a three-dimensional coordinate of the target video object in a third coordinate system corresponding to the current camera, and the current camera is any other than the binocular camera in the navigation camera system. A camera. For example, in a multi-camera camera scene, the current camera can be a binocular camera other than the binocular camera of the host position.

In a specific embodiment, the purpose of the three-dimensional positioning of the multi-camera video object is to calculate the three-dimensional coordinates of the video object in a binocular coordinate system of a navigation camera, and calculate it in other binocular cameras or a PZT camera coordinate system. The three-dimensional coordinates. It is known that a certain observation point (ie, a video object, specifically a certain feature point of the video object) has a coordinate vector X ₁ in the camera D1, and an external parameter R ₂₁ , T _{21 of the} camera D2 relative to the camera D1 (through The binocular camera calibration is obtained), and the coordinate vector X ₂ of the observation point in the camera D2 can be calculated:

X ₂ =R ₂₁ X ₁ +t ₂₁

Specifically, please refer to FIG. 4a, which is a schematic diagram of an object matching scenario according to an embodiment of the present invention. Three-dimensional positioning of multi-camera video objects can be used to determine the correspondence of multiple video objects. As shown in Figure 4a, three-camera cameras D1, D2, and D3 are deployed in the scene. There are 3 participants in O ₁ , O _{2 ,} and O _{3 in the} scene. Further, as shown in Figure 4b, it is a set of video objects in the image of FIG 4a, the participants O ₁ which is a different view of the imaging cameras directed D1, D2 and D3 in. For the participant O ₁ , the binocular camera in the station D1 detects the video object VO ₁₁ by using an algorithm such as face detection, and then uses the binocular camera three-dimensional positioning algorithm to obtain the three-dimensional position of the object in the D1 binocular coordinate system. Similarly, D2 and D3 will also detect the video objects VO ₁₂ and VO ₁₃ and calculate the three-dimensional coordinates of the object in the D2 and D3 binocular coordinate systems. In the three-dimensional positioning of the multi-camera video object, the positional relationship between D1, D2 and D3 can be used to calibrate the three-dimensional position of the video object VO ₁₁ in the D1 coordinate system to the D2 and D3 coordinate systems. And detect the coincident area. If it is detected that the three-dimensional position of the VO ₁₁ and the positional overlap of the VO ₁₂ and the VO ₁₃ exceed a certain area threshold, the VO ₁₁ , VO ₁₂ and VO _{13 are considered to} be the same video object, and the video object is successfully matched. Further, if there are multiple video objects in the image that are close to each other in the image, simply determining the correspondence relationship of the video objects by using the positional coincidence region may result in a matching error. Thereby, the image information of the video object can be further combined and the accuracy of determining the correspondence can be improved by the matching algorithm. The matching algorithm may include a template matching algorithm and the like. For example, a template matching algorithm may be used to take a two-dimensional image of a video object detected by a binocular camera of a certain host position such as a host bit as a known template, and perform a video object detected by a binocular camera of another station. A matching finds the object that matches the video object by algorithm such as squared difference matching and correlation matching, thereby establishing the correspondence of the object.

Further, after determining the three-dimensional positioning of the binocular camera and the three-dimensional positioning of the PTZ camera,

Video object detection tracking and scene modeling. Among them, the purpose of video object detection/tracking is to construct and describe the video objects existing in the scene, and to track and identify these objects. Video objects include participants, as well as scene objects such as lights, windows, conference tables, and more. The system needs to cyclically process the image data of the input binocular camera, including face detection and matching, human shape detection and matching, moving object detection and matching, scene object detection and matching, etc., modeling the video object and updating the model. Parameters to model the entire shooting scene based on the detected object model. The modeled model can be used for subsequent object recognition and guided strategy processing. Among them, face detection can be used to detect video objects with close distances, such as participants with relatively close detection distances. For distant areas, because the face area is small and cannot be detected well, human form or The method of moving object detection. The face detection can obtain various parameters of the face video object, including the two-dimensional coordinates of the circumscribed rectangular area of the face, the coordinates of the center point, the area of the rectangle, and the rotation angle of the face around the coordinate axis (representing the left and right deflection of the face, pitching Parameters such as the degree of rotation and the position of the organs such as the eyes, nose, and mouth in the face.

Further, after the video object is detected in each frame image, the video object needs to be tracked in the video frame sequence to establish a correspondence relationship between the video objects in the time domain. Currently widely used video object tracking algorithms include grayscale based template matching, MeanShift, CamShift, Kalman filtering algorithms and the like. Among them, the video object The matching can be applied to the binocular camera, and the video object region detected in one camera image of the binocular camera is used to find a corresponding video image region in another camera image, so that the feature can be performed in the matching region of the video object. Match and calculation of 3D coordinates. The matching algorithm of the video object is similar to the tracking algorithm, and grayscale-based template matching and algorithms such as MeanShift can be used.

In the embodiment of the present invention, a video object may be represented by its features, and commonly used features include feature points, image textures, histogram information, and the like. The feature detection and matching can be performed in the detected video object region, so that the three-dimensional position information of the video object, that is, the three-dimensional coordinates can be calculated according to the feature point information, and the video object can be tracked according to the texture information and the histogram information. The feature point is the main feature type, and the feature point detection algorithm includes Harris corner detection and SIFT feature point detection. Further, the feature matching is used to establish the correspondence relationship between the features of the same video object of the binocular camera. The feature points can be matched by using a matching algorithm such as FLANN algorithm and KLT optical flow method, and the image texture can be matched by using a gray template matching algorithm and the like. Graphs can be matched using algorithms such as histogram matching. In summary, according to the matching feature information and the above-mentioned binocular camera three-dimensional positioning algorithm, the three-dimensional coordinates of the video object features in the three-dimensional coordinate system of the single guided camera can be calculated, so that a certain video can be located and tracked in the three-dimensional space. Object.

Further, according to the video object detection and matching, the feature detection and matching algorithm, and the result of calculating the three-dimensional position of the video object, a plurality of video object models can be established in a single navigation camera coordinate system, and can be passed through a human face or a human form. And the motion detection tracking algorithm updates the model data. Specifically, each video object model can be assigned a unique ID number, and the data in the model represents the attributes of the video object. For example, for a moving object model, the data in the model may include attributes such as an object ID, a circumscribed rectangle two-dimensional coordinate, a three-dimensional coordinate of the object feature point, a motion region texture data, a histogram data, and the like. When the position of the moving object changes, its properties are refreshed according to the output of the above detection and matching algorithm, but the ID of the object remains unchanged. The creation of human faces and humanoid objects is similar to the moving object model and will not be described here.

It should be understood that in a multi-camera application scenario, video camera model data can be exchanged between multiple guide cameras through network communication. After a single guide camera obtains video object model data of other guide cameras, the above-mentioned multi-guide camera can be used to generate three-dimensional images. The algorithm for locating and matching the video object establishes a correspondence relationship between the video object models, thereby obtaining a guiding strategy for the entire scene. The network communication protocol during communication can adopt a standard protocol such as the HTTP protocol, or a custom protocol, and the data of the video object model is formatted according to a format such as an eXtensible Markup Language ("XML") format. , packaging and transmission. By matching and merging multiple guided camera video object models, a single guided camera can establish a model of the entire shooting scene. The scene model contains models of multiple video objects, reflecting the characteristics of the video objects and their distribution in three dimensions. The camera needs to maintain the scene model, including adding and deleting object models and object model properties. For example, when a new participant appears in the scene, when the binocular camera detects a new face or a humanoid object, the object model is created and added to the object model set; when the participant leaves the scene, the object is deleted. The model; after the participant's position changes, the parameters of the corresponding object model are updated. To develop a navigation strategy based on the latest video object model, select the camera with the best position for shooting.

104. Adjust an imaging parameter of the target camera to an imaging parameter corresponding to the first three-dimensional coordinate, and output a video image after adjusting the imaging parameter.

In a specific embodiment, after the video object model including all the video objects is created (updated) and the target video object is determined, one or more guiding camera positions of the best shooting effect can be selected according to the preset guiding strategy. For example, the camera having the better shooting effect is determined according to the eye-to-eye effect parameter, the occlusion relationship parameter, and the scene object parameter of the shooting area. Specifically, the eye-to-eye effect needs to be determined according to the optical axis angle of the face/human object relative to the PTZ camera. The smaller the angle, the more the face is presented in a positive face manner, and the eye-to-eye effect is better. Specifically, the face/human shape detection algorithm obtains a rotation angle (left and right deflection, pitch and rotation angle) of the three-dimensional coordinate axis centered on the face/human shape with respect to the binocular camera coordinate system, and uses the aforementioned camera. The formula for the conversion of the coordinate system converts the rotation angle of the face/human shape relative to the binocular camera to the rotation angle of the relative PTZ camera. During the conversion process, it is necessary to use the external parameters between the previously-calibrated binocular camera and the PTZ camera, and the external parameters between the binocular cameras of different positions to determine the eye-to-eye effect parameters. A PTZ camera priority queue with an eye-to-eye effect can be further established for each video object, and a camera with better eye-to-eye effect has a higher priority.

Further, when acquiring the occlusion relationship of the video object, the area of the video object detected by a certain camera (such as an circumscribed rectangle) may be known according to the projection equation of the camera, and the single-guided camera binocular camera and the calibration may be utilized. The external parameters between the PTZ cameras and the external parameters between the binocular cameras of different positions re-enter the area onto the imaging plane of each position PTZ camera. If the regions of the two video objects overlap, the depth information can be used to determine the occlusion relationship between the two objects, that is, the video object closer to the binocular camera will block the farther video object. Thus, a PTZ camera priority queue can be established for each video object with an occlusion relationship, and an unoccluded camera has a higher priority.

Further, in addition to human-based video object detection, the system also detects other video objects (scene objects) of interest in the scene, such as lamps, windows, conference tables, and the like. The detection of these objects can employ algorithms based on image color and edge features, and the like. For example, for lamp detection, the Canny operator can be used to extract the edge of the tube to obtain its long straight line feature, and then detect whether there is an overexposed pixel area (lighting area) in the adjacent area. According to these two features, it can be detected. Exit the tube object and get the coordinates of its circumscribed rectangle. The detection of the window is similar to the detection of the tube. The quadrilateral feature can be obtained by edge detection, and then whether the window is determined according to whether there is a certain area of the overexposed pixel area in the quadrilateral. The conference table can also be detected using edge features in the image. When acquiring the scene object parameter, according to the camera projection equation, the area of the scene object detected by a certain guide camera is known, and the external parameters between the single guide camera binocular camera and the PTZ camera are used, and different The external parameter between the binocular cameras of the aircraft re-casts the area onto the imaging plane of each position PTZ camera. For objects such as tubes and windows, there is usually a large area of overexposed area, which causes the camera's automatic exposure effect to deteriorate, the scene to darken, and scene objects such as tables have large areas of red or yellow, resulting in cameras. The automatic white balance color cast, these scene objects should be avoided as much as possible in the image. Thereby, it is possible to establish a PTZ camera priority queue in which a scene object that is not conducive to an image effect is captured, and a camera that has a low probability of capturing a scene object has a higher priority.

Further, the guide camera of the navigation camera, such as the host position, establishes a priority queue for the PTZ cameras of each station according to the acquired image effect parameters and the preset guide strategy, and can determine the camera to be selected. Specifically, one or more video objects that need to be photographed, that is, a target video object, such as a talking video object determined according to the sound source localization result, may be determined to capture a close-up of the video object; or an AutoFrame strategy is required. When all video objects are used as the target video object, Pan/Tilt can be adjusted to include all the video objects in the scene, and Zoom adjusts the object to the appropriate size, and so on. For the target subject, a PTZ camera priority queue using eye-to-eye effect parameters, occlusion relationship parameters, and scene object parameters can determine a comprehensive PTZ camera priority queue according to a certain guiding strategy. The navigation policy may be automatically calculated by the system or preset by the user, which is not limited by the embodiment of the present invention.

For example, suppose that the eye-to-eye effect is the best, the unobstructed PTZ camera, if there are multiple cameras that meet the condition, select the PTZ camera with the best image object parameters and the best image as the target camera. . After the PTZ camera selection is completed, the host bit can adjust the PTZ parameters of the selected PTZ camera according to the three-dimensional coordinates of the target video object to obtain the best image effect as much as possible. For example, during voice tracking, when shooting a close-up of a participant, avoid shooting an object that affects the brightness of the image, such as a lamp or a window; when adjusting the Zoom size, avoid adjusting the white balance effect of the large-area table object on the image. and many more.

Further, the host bit (the host position of the navigation camera) can output the selected PTZ camera video image or ID. Optionally, for a multi-guide camera system that supports video cascading, the host bit can directly output an image of the selected camera; for a multi-guide camera system output through the video matrix, the host bit can pass through a communication interface (such as a serial port or a network port). The ID of the selected PTZ camera is output to the video matrix, and the camera image is switched by the video matrix.

In the embodiment of the present invention, after determining the target video object that needs to be captured, the target camera that captures the target video object is selected from the cameras of the navigation camera system according to a preset guiding strategy, and is obtained. a three-dimensional coordinate of the target video object in a coordinate system corresponding to the target camera, to control the target camera to perform camera parameter adjustment according to the three-dimensional coordinates of the target video object, and output a video image after adjusting the imaging parameter, so that the guided camera system It can improve the accuracy of video object detection and tracking based on three-dimensional coordinate detection and preset navigation strategy, and improve the efficiency of camera parameter adjustment, and effectively improve the camera's shooting effect.

Referring to FIG. 5, FIG. 5 is a schematic structural diagram of a parameter adjustment apparatus according to an embodiment of the present invention. Specifically, the device in the embodiment of the present invention may be specifically configured in the above-mentioned navigation camera. As shown in FIG. 6, the parameter adjustment device in the embodiment of the present invention may include an object determining unit 10, a selecting unit 20, and an acquiring unit. 30 and parameter adjustment unit 40. among them,

The object determining unit 10 is configured to determine a target video object that needs to be photographed.

The selecting unit 20 is configured to select, from a camera of the navigation camera system where the navigation camera is located, a target camera for capturing the target video object according to a preset navigation policy.

Optionally, the shooting effect parameter may include any one or more of an eye-to-eye effect parameter, an occlusion relationship parameter, and a scene object parameter of the shooting area of the target video object in a coordinate system corresponding to the current camera. The current camera is any camera other than the binocular camera in the navigation camera system.

The eye-to-eye effect parameter may include a rotation angle of the target video object with respect to a coordinate system corresponding to the current camera, and the rotation angle may be a rotation angle of the target video object in the second coordinate system. And determining a positional relationship between the binocular camera and the current camera that is pre-calibrated.

The occlusion relationship parameter and the scene object parameter may be that the area of the scene object detected by the current camera is re-injected to the location according to a pre-calibrated positional relationship between the binocular camera and the current camera. The imaging plane of the current camera is determined.

The acquiring unit 30 is configured to acquire first three-dimensional coordinates of the target video object.

The first three-dimensional coordinate may be a three-dimensional coordinate of the target video object in a first coordinate system corresponding to the target camera. The target camera may be the above-mentioned navigation camera or an ordinary PTZ camera. The first coordinate system corresponding to the target camera may refer to a three-dimensional coordinate system established with the optical center of the target camera as the origin, or the origin of any other reference object. The established three-dimensional coordinate system is not limited in the embodiment of the present invention.

The parameter adjustment unit 40 is configured to adjust an imaging parameter of the target camera to an imaging parameter corresponding to the first three-dimensional coordinate, and output a video image after adjusting the imaging parameter.

Optionally, the obtaining unit 30 may be specifically configured to:

Obtaining a second three-dimensional coordinate transmitted by the binocular camera connected to the navigation camera, where the second three-dimensional coordinate is a three-dimensional coordinate of the target video object in a second coordinate system corresponding to the binocular camera;

The second three-dimensional coordinates are converted into first three-dimensional coordinates according to a pre-calibrated positional relationship between the binocular camera and the target camera.

Further, the second three-dimensional coordinates may be two-dimensional coordinates of the target video object acquired in the left view and the right view of the binocular camera and the binocular acquired by the binocular camera respectively The internal and external parameters of the camera are calculated. The second three-dimensional coordinate is a three-dimensional coordinate of the target video object in a second coordinate system corresponding to the binocular camera, and the second coordinate system corresponding to the binocular camera may be a binocular camera optical center The three-dimensional coordinate system established by the origin, or the three-dimensional coordinate system established with the origin of any other reference object. The two-dimensional coordinates may specifically be corresponding pixel coordinates of the target video object in the left view and the right view of the binocular camera.

Optionally, the object determining unit 10 may be specifically configured to:

Obtaining a captured image transmitted by a binocular camera, the captured image including at least one video object;

Establishing a video object model including the at least one video object, and determining a target video object from the at least one video object;

The selecting unit 20 can be specifically configured to:

Determining the target video object from the captured images acquired by the cameras in the navigation camera system, and acquiring shooting effect parameters of the target video object in each camera;

The camera whose shooting effect parameter satisfies the preset guiding strategy is determined as the target camera for capturing the target video object.

Further, the specific manner in which the selecting unit 20 performs the determination of the target video object from the captured image acquired by the camera may be:

Converting the second three-dimensional coordinates into a third three-dimensional coordinate according to a pre-calibrated positional relationship between the binocular camera and the current camera, wherein the current camera is in the navigation camera system except the binocular camera The third three-dimensional coordinate is a three-dimensional coordinate of the target video object in a third coordinate system corresponding to the current camera;

Determining whether an area of the target video object in the third three-dimensional coordinate and a region of the video object detected by the current camera in the three-dimensional coordinate of the video object exceed a preset area threshold;

If exceeded, the video object is determined to be the target video object.

In the embodiment of the present invention, after determining the target video object that needs to be captured, the target camera that captures the target video object is selected from the cameras of the navigation camera system according to a preset guiding strategy, and is obtained. a three-dimensional coordinate of the target video object in a coordinate system corresponding to the target camera, to control the target camera to perform camera parameter adjustment according to the three-dimensional coordinates of the target video object, and output a video image after adjusting the imaging parameter, so that the guided camera system It can improve the accuracy of video object detection and tracking based on three-dimensional coordinate detection and preset navigation strategy, and improve the efficiency of camera parameter adjustment, and effectively improve the camera's shooting effect.

Referring to FIG. 6, FIG. 6 is a schematic structural diagram of a navigation camera system according to an embodiment of the present invention. Specifically, the navigation camera system of the embodiment of the present invention may include a first camera 1 and at least one second camera 2, the first camera 1 includes a navigation camera 11 and a binocular camera 12, and the navigation camera 11 and the camera Between the binocular cameras 12, the first camera 1 and the second camera 2 can be connected through a wired interface or a wireless interface;

The guidance camera 11 is configured to determine a target video object that needs to be captured, and select a target camera for capturing the target video object from a camera of the navigation camera system according to a preset navigation strategy;

The binocular camera 12 is configured to acquire a second three-dimensional coordinate of the target video object, and transmit the second three-dimensional coordinate to the navigation camera 11; wherein the second three-dimensional coordinate is the target video The three-dimensional coordinates of the object in the second coordinate system corresponding to the binocular camera 12;

The navigation camera 11 is configured to receive the second three-dimensional coordinates transmitted by the binocular camera 12; and according to a pre-calibrated positional relationship between the binocular camera 12 and the target camera, the second three-dimensional coordinates Converting to a first three-dimensional coordinate; adjusting an imaging parameter of the target camera to an imaging parameter corresponding to the first three-dimensional coordinate, and outputting a video image after adjusting the imaging parameter; wherein the first three-dimensional coordinate is the The three-dimensional coordinates of the target video object in the first coordinate system corresponding to the target camera.

Optionally, the second camera 2 may also include a navigation camera and a binocular camera, and the target camera may be any of the navigation cameras in the navigation camera system; or the second camera 2 is a normal PTZ camera. , the target camera can be the guide camera or a normal PTZ camera. Further optionally, the binocular camera 12 can be disposed on a preset guide bracket and connected to the guide camera 11 through the guide bracket.

Specifically, as shown in FIG. 7 , it is a schematic structural diagram of a first camera provided by an embodiment of the present invention. The first camera includes a binocular camera and one or more guide cameras. It is assumed that in the embodiment of the present invention, the first camera is equipped with two navigation cameras for guiding shooting and tracking, which can be wired or wirelessly connected to the binocular camera through a guiding bracket (referred to as "cradle"). The binocular camera is mounted on the bracket. In addition, a microphone can be mounted on the bracket, and the installed microphone can be in the form of an array. The microphone in the array can be used for realizing sound source positioning, sound source identification and the like. Includes a horizontal array of microphones and a vertical array of microphones. Further, the guide camera and the bracket may be separated or integrated, and a communication interface such as a serial interface may be used for communication between the guide camera and the bracket. In some embodiments, the above-mentioned navigation camera and the guide bracket (including the binocular camera, the microphone, and the like) may be integrated into one guide device. The connection form of each device in the navigation camera system is not limited in the embodiment of the present invention.

Further, please refer to FIG. 8 , which is a schematic diagram of networking of a guided camera system according to an embodiment of the present invention. As shown in Figure 8, multiple slots can be networked. The multi-camera network includes multiple inter-camera networking with guided cameras, and the camera and guide brackets are installed + multiple common PTZ camera groups. Net, the position of the camera and the guide bracket is installed + the position of the station without PTZ camera (that is, only the guide bracket), and the position without the PTZ camera + the network of multiple ordinary PTZ cameras (ie, no guide bracket). The cameras of each station can be interconnected by LAN or Wi-Fi to transmit control messages, including camera switching messages, audio and video data such as video object model data, and the like. Further optionally, the control message may be transmitted through an Internet Protocol (IP), such as an IP Camera protocol stack. A binning camera is required between the binocular cameras in the two positions. When a certain video camera needs to perform multi-channel video output, it can be connected to the video matrix of the networking system where the navigation camera is located, and is switched and output by the video matrix. Optionally, the switching policy of the video matrix may be controlled by any specified navigation camera in the scenario, such as a navigation camera as a host, or controlled by a third-party device, which is not limited in the embodiment of the present invention. The video image output by the video matrix is encoded by the codec device and transmitted to the far end for video conferencing. Specifically, if the number of cameras in the network is small, the video data can be processed in cascade (the navigation bracket supports video cascading); if the number is large, the video of multiple cameras is output to the video matrix for processing, by the video matrix. Switch or synthesize one or more camera video sources. Further, the bracket can provide external video input / Output interface, LAN/Wi-Fi network port and serial interface. The video input interface is used for external input video of other cameras; the video output interface is used to connect terminals or video matrix devices to output video images; the serial interface provides control and debugging interface for the bracket; LAN/Wi-Fi The network port is used for cascading multiple camera positions, and can transmit audio and video data and control data.

Further, in the multi-guide camera networking scenario, the plurality of navigation cameras have video object detection capability and PTZ camera function, and one of the navigation cameras can be used as a host position, and is responsible for outputting the position selection and PTZ camera control, and other cameras. As the slave position; the position of the guide camera and the guide bracket + multiple common PTZ cameras, only one guide camera has video object detection capability, which is responsible for output position selection and PTZ camera control, and the ordinary camera is only used as a PTZ camera. Since only the navigation camera has the video object detection capability, the data of the video object model of the camera position is not obtained through the network, and the matching process of the multi-camera video object model is performed.

For details, refer to the related descriptions of the corresponding embodiments in the foregoing FIG. 1-6 in the navigation camera and the binocular camera in the embodiment of the present invention, and details are not described herein again.

Referring to FIG. 9, FIG. 9 is a schematic structural diagram of a navigation camera according to an embodiment of the present invention, for performing the above camera parameter adjustment method. Specifically, as shown in FIG. 9, the navigation camera of the embodiment of the present invention includes: a communication interface 300, a memory 200, and a processor 100, and the processor 100 is respectively connected to the communication interface 300 and the memory 200. The memory 200 may be a high speed RAM memory or a non-volatile memory such as at least one disk memory. The communication interface 300, the memory 200, and the processor 100 may be connected to each other through a bus, or may be connected by other means. In the present embodiment, a bus connection will be described. The device structure shown in FIG. 10 does not constitute a limitation on the embodiments of the present invention, and may include more or less components than those illustrated, or a combination of certain components, or different component arrangements. among them:

The processor 100 is a control center of the device, which connects various parts of the entire device using various interfaces and lines, by running or executing programs and/or units stored in the memory 200, and calling driver software stored in the memory 200, Perform various functions and process data of the device. The processor 100 may be composed of an integrated circuit ("IC"), for example, may be composed of a single packaged IC, or may be composed of a plurality of packaged ICs that have the same function or different functions. For example, the processor 100 may include only a central processing unit ("CPU"), or may be a CPU, a digital signal processor ("DSP"), or a graphics processor ( Graphic Processing Unit (referred to as "GPU") and a combination of various control chips. In the embodiment of the present invention, the CPU may be a single operation core, and may also include multiple operation cores.

Communication interface 300 can include a wired interface, a wireless interface, and the like.

The memory 200 can be used to store driver software (or software programs) and units, and the processor 100 and the communication interface 300 perform various functional applications of the devices and implement data processing by calling the driver software and the units stored in the memory 200. The memory 200 mainly includes a program storage area and a data storage area, wherein the program storage area can store driver software and the like required for at least one function; the data storage area can store data according to the parameter adjustment process, such as the above-described three-dimensional coordinate information.

Specifically, the processor 100 reads the driver software from the memory 200 and executes it under the action of the driver software:

Determining the target video object that needs to be captured;

Filtering from each camera of the navigation camera system where the navigation camera is located according to a preset navigation strategy Shooting a target camera of the target video object;

Obtaining a first three-dimensional coordinate of the target video object, where the first three-dimensional coordinate is a three-dimensional coordinate of the target video object in a first coordinate system corresponding to the target camera;

And adjusting an imaging parameter of the target camera to an imaging parameter corresponding to the first three-dimensional coordinate, and outputting a video image after adjusting the imaging parameter.

Optionally, the processor 100 reads the driver software from the memory 200 and performs the acquiring the first three-dimensional coordinates of the target video object by using the driver software, and specifically performing the following steps:

Obtaining, by the communication interface 300, a second three-dimensional coordinate transmitted by the binocular camera connected to the navigation camera, where the second three-dimensional coordinate is that the target video object is in a second coordinate system corresponding to the binocular camera Three-dimensional coordinates

The second three-dimensional coordinates are converted into first three-dimensional coordinates according to a pre-calibrated positional relationship between the binocular camera and the target camera.

Optionally, the processor 100 reads the driver software from the memory 200 and performs the determining of the target video object that needs to be captured by the driver software, and specifically performs the following steps:

Acquiring a captured image transmitted by the binocular camera, the captured image including at least one video object;

Establishing a video object model including the at least one video object, and determining a target video object from the at least one video object;

The processor 100 reads the driver software from the memory 200 and performs the function of the driver software to select from the cameras of the navigation camera system where the camera is located according to a preset navigation policy. For the target camera for capturing the target video object, perform the following steps:

Determining the target video object from the captured images acquired by the cameras in the navigation camera system, and acquiring shooting effect parameters of the target video object in each camera;

The camera whose shooting effect parameter satisfies the preset guiding strategy is determined as the target camera for capturing the target video object.

Optionally, the processor 100 reads the driver software from the memory 200 and performs the determination of the target video object from the captured image acquired by the camera under the action of the driver software, and specifically performs the following steps:

Converting the second three-dimensional coordinates into a third three-dimensional coordinate according to a pre-calibrated positional relationship between the binocular camera and the current camera, wherein the current camera is in the navigation camera system except the binocular camera The third three-dimensional coordinate is a three-dimensional coordinate of the target video object in a third coordinate system corresponding to the current camera;

Determining whether an area of the target video object in the third three-dimensional coordinate and a region of the video object detected by the current camera in the three-dimensional coordinate of the video object exceed a preset area threshold;

If exceeded, the video object is determined to be the target video object.

Optionally, the shooting effect parameter may include any one or more of an eye-to-eye effect parameter, an occlusion relationship parameter, and a scene object parameter of the shooting area of the target video object in a coordinate system corresponding to the current camera. The current camera is any camera other than the binocular camera in the navigation camera system.

The eye-to-eye effect parameter may include a rotation angle of the target video object relative to a coordinate system corresponding to the current camera, the rotation angle being according to a rotation angle of the target video object in the second coordinate system and The pre-calibrated positional relationship between the binocular camera and the current camera is determined.

The occlusion relationship parameter and the scene object parameter may be according to the binocular camera pre-calibrated. And a positional relationship between the current camera, and re-casting an area of the scene object detected by the current camera to an imaging plane of the current camera.

In the above embodiments, the descriptions of the various embodiments are different, and the details that are not detailed in a certain embodiment can be referred to the related descriptions of other embodiments.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separated, and the components displayed as the unit may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

The above-described integrated unit implemented in the form of a software functional unit can be stored in a computer readable storage medium. The above software functional unit is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to perform the methods of the various embodiments of the present invention. Part of the steps. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, simply referred to as "ROM"), a random access memory (RAM), a magnetic disk, or an optical disk, and the like. The medium of the program code.

It will be apparent to those skilled in the art that for the convenience and brevity of the description, only the division of each functional unit described above is exemplified. In practical applications, the above function assignment can be completed by different functional units as needed, that is, the device The internal structure is divided into different functional units to perform all or part of the functions described above. For the specific working process of the device described above, refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, and are not intended to be limiting; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that The technical solutions described in the foregoing embodiments may be modified, or some or all of the technical features may be equivalently replaced; and the modifications or substitutions do not deviate from the technical solutions of the embodiments of the present invention. range.

Claims (15)

A camera parameter adjustment method, the method being applied to a navigation camera, comprising: Determining the target video object that needs to be captured; Screening a target camera for capturing the target video object from each camera of the navigation camera system in which the navigation camera is located according to a preset navigation strategy; Obtaining a first three-dimensional coordinate of the target video object, where the first three-dimensional coordinate is a three-dimensional coordinate of the target video object in a first coordinate system corresponding to the target camera; And adjusting an imaging parameter of the target camera to an imaging parameter corresponding to the first three-dimensional coordinate, and outputting a video image after adjusting the imaging parameter. The method according to claim 1, wherein the acquiring the first three-dimensional coordinates of the target video object comprises: Obtaining a second three-dimensional coordinate transmitted by the binocular camera connected to the navigation camera, where the second three-dimensional coordinate is a three-dimensional coordinate of the target video object in a second coordinate system corresponding to the binocular camera; The second three-dimensional coordinates are converted into first three-dimensional coordinates according to a pre-calibrated positional relationship between the binocular camera and the target camera. The method according to claim 2, wherein the determining a target video object that needs to be captured comprises: Acquiring a captured image transmitted by the binocular camera, the captured image including at least one video object; Establishing a video object model including the at least one video object, and determining a target video object from the at least one video object; The target camera for capturing the target video object is filtered out from each camera of the navigation camera system where the navigation camera is located according to a preset navigation policy, including: Determining the target video object from the captured images acquired by the cameras in the navigation camera system, and acquiring shooting effect parameters of the target video object in each camera; The camera whose shooting effect parameter satisfies the preset guiding strategy is determined as the target camera for capturing the target video object. The method according to claim 3, wherein the determining the target video object from the captured image acquired by the camera comprises: Converting the second three-dimensional coordinates into a third three-dimensional coordinate according to a pre-calibrated positional relationship between the binocular camera and the current camera, wherein the current camera is in the navigation camera system except the binocular camera The third three-dimensional coordinate is a three-dimensional coordinate of the target video object in a third coordinate system corresponding to the current camera; Determining whether an area of the target video object in the third three-dimensional coordinate and a region of the video object detected by the current camera in the three-dimensional coordinate of the video object exceed a preset area threshold; If exceeded, the video object is determined to be the target video object. The method according to claim 3, wherein the shooting effect parameter comprises an eye-to-eye effect parameter, an occlusion relationship parameter, and a scene object parameter of the shooting area of the target video object in a coordinate system corresponding to the current camera. The current camera is any camera other than the binocular camera in the navigation camera system. The method according to claim 5, wherein the eye-to-eye effect parameter comprises a rotation angle of the target video object with respect to a coordinate system corresponding to the current camera, the rotation angle being according to the target video object The rotation angle of the second coordinate system and the positional relationship between the binocular camera and the current camera that are pre-calibrated are determined. The method according to claim 5, wherein the occlusion relationship parameter and the scene object parameter are detected by the current camera according to a pre-calibrated positional relationship between the binocular camera and the current camera. The area of the scene object is re-injected to the imaging plane of the current camera. A navigation camera, comprising: a memory and a processor, wherein the processor is connected to the memory; wherein The memory is used to store driver software; The processor reads the driver software from the memory and executes it under the action of the driver software: Determining the target video object that needs to be captured; Screening a target camera for capturing the target video object from each camera of the navigation camera system in which the navigation camera is located according to a preset navigation strategy; Obtaining a first three-dimensional coordinate of the target video object, where the first three-dimensional coordinate is a three-dimensional coordinate of the target video object in a first coordinate system corresponding to the target camera; And adjusting an imaging parameter of the target camera to an imaging parameter corresponding to the first three-dimensional coordinate, and outputting a video image after adjusting the imaging parameter. The navigation camera of claim 8 wherein said navigation camera further comprises a communication interface, said communication interface being coupled to said processor; said processor reading said driver software from said memory and Performing the acquiring the first three-dimensional coordinates of the target video object by using the driving software, and performing the following steps: Obtaining, by the communication interface, a second three-dimensional coordinate transmitted by the binocular camera connected to the navigation camera, where the second three-dimensional coordinate is a three-dimensional coordinate of the target video object in a second coordinate system corresponding to the binocular camera coordinate; The second three-dimensional coordinates are converted into first three-dimensional coordinates according to a pre-calibrated positional relationship between the binocular camera and the target camera. The navigation camera according to claim 9, wherein the processor reads the driver software from the memory and performs the determining of a target video object that needs to be captured under the action of the driver software, and specifically executes The following steps: Acquiring a captured image transmitted by the binocular camera, the captured image including at least one video object; Establishing a video object model including the at least one video object, and determining a target video object from the at least one video object; The processor reads the driver software from the memory and performs the function of the driver software to filter out the cameras of the camera system in which the camera is located according to a preset navigation policy. To shoot the target camera of the target video object, perform the following steps: Determining the target video object from the captured images acquired by the cameras in the navigation camera system, and acquiring shooting effect parameters of the target video object in each camera; The camera whose shooting effect parameter satisfies the preset guiding strategy is determined as the target camera for capturing the target video object. The navigation camera according to claim 10, wherein said processor reads said driver software from said memory and performs said target from said captured image acquired by the camera under the action of said driver software Video object, perform the following steps: Converting the second three-dimensional coordinates into a third three-dimensional coordinate according to a pre-calibrated positional relationship between the binocular camera and the current camera, wherein the current camera is in the navigation camera system except the binocular camera The third three-dimensional coordinate is a three-dimensional coordinate of the target video object in a third coordinate system corresponding to the current camera; Determining whether an area of the target video object in the third three-dimensional coordinate and a region of the video object detected by the current camera in the three-dimensional coordinate of the video object exceed a preset area threshold; If exceeded, the video object is determined to be the target video object. The navigation camera according to claim 10, wherein the shooting effect parameter comprises an eye-to-eye effect parameter, an occlusion relationship parameter, and a scene object parameter of the shooting area of the target video object in a coordinate system corresponding to the current camera. In any one or more of the following, the current camera is any camera other than the binocular camera in the navigation camera system. The guidance camera according to claim 12, wherein the eye-to-eye effect parameter comprises a rotation angle of the target video object with respect to a coordinate system corresponding to the current camera, the rotation angle being according to the target video object The rotation angle of the second coordinate system and the positional relationship between the binocular camera and the current camera that are pre-calibrated are determined. The navigation camera according to claim 12, wherein the occlusion relationship parameter and the scene object parameter are based on a pre-calibrated positional relationship between the binocular camera and the current camera, and the current camera is detected. The area of the scene object that is reached is re-injected to the imaging plane of the current camera. A navigation camera system, comprising: a first camera and at least one second camera, the first camera comprising a navigation camera and a binocular camera, the navigation camera and the binocular camera, the first a camera and the second camera are connected by a wired interface or a wireless interface; wherein The guidance camera is configured to determine a target video object that needs to be captured, and select a target camera for capturing the target video object from a camera of the navigation camera system according to a preset navigation strategy; The binocular camera is configured to acquire a second three-dimensional coordinate of the target video object, and transmit the second three-dimensional coordinate to the navigation camera; wherein the second three-dimensional coordinate is that the target video object is The binocular camera pair The three-dimensional coordinates of the second coordinate system; The guidance camera is configured to receive the second three-dimensional coordinates transmitted by the binocular camera; convert the second three-dimensional coordinates into a first position according to a pre-calibrated positional relationship between the binocular camera and the target camera a three-dimensional coordinate; adjusting an imaging parameter of the target camera to an imaging parameter corresponding to the first three-dimensional coordinate, and outputting a video image after adjusting the imaging parameter; wherein the first three-dimensional coordinate is the target video object The three-dimensional coordinates in the first coordinate system corresponding to the target camera.

Images