MXPA98001849A - Method and apparatus for automatic electronic replacement of advertisement boards in an imagende vi - Google Patents

Abstract

The present invention relates to an apparatus for the automatic replacement of a bulletin board in a video image that includes a video camera for displaying the bulletin board, characterized in that the bulletin board comprises an image insertion surface and because the apparatus further includes an image insertion unit effective to detect at least one image insertion color of the image image insertion surface of the bulletin board, and wherein the reference color of the insertion unit of The image is adjusted to conform the color of the image insertion surface, thus allowing the image insertion unit to correctly identify the image insertion surface of the bulletin board by means of a virtual advertisement in a video image with correct obstruction of any object of the image. first pla

Description

METHOD AND APPARATUS FOR AUTOMATIC ELECTRONIC REPLACEMENT OF ADVERTISEMENT BOARDS IN A VIDEO IMAGE FIELD OF THE INVENTION The present invention relates to a method and apparatus for automatically replacing advertisement boards in a video image. The present invention has particular use in the electronic replacement of bulletin boards in a stadium and another stage, but can be used to provide accurate data relating to the orientation of the camera for other purposes.

BACKGROUND OF THE INVENTION In previous systems it has been proposed to electronically replace the bulletin boards of a stadium that are seen by an observer on television. The bulletin boards in the stadium are televised by a TV camera and the boards are electronically altered, so that the TV watcher at home sees a different board than the viewer in the stadium or other scenario. The known systems, such as those described in the patent of E.U.A. 5,26, 933, describe an apparatus and method for electronically altering video images. The apparatus and method described in the U.A. patent, and also in U.S. Patent No. 5,353,392, although theoretically it allows the replacement of bulletin boards, does not solve the many practical problems encountered in real media. Most of these problems are related to the recognition and replacement processes. Relying completely on pattern recognition techniques that use only the video signal to identify and locate bulletin boards for replacement introduces significant problems that affect the practical value of that system. Clearly, any pattern recognition scheme, including those described in U.S. Patent Nos. 5,264,933 and 5,353,392, must rely on useful visible characteristic features of the image that can be compared with predefined descriptions. Such characteristic features must be located within the bulletin board or in its vicinity. In real situations, the visibility of these characteristic features can change continuously, or otherwise, practically from 0 to some threshold visibility that allows the pattern recognition scheme to work properly. These changes can occur in the direction of increasing or reducing visibility. These situations include: - Acceleration or deceleration of the movement of the camera that introduces an immense amount of fuzzy strokes.

- Excessive approach or distance from the bulletin board. - Excessive obstruction by the players. - Entry or exit of the visual field of a camera by means of any combination of horizontal rotation, tilt and approach operations. - Any combination of the mechanisms mentioned above. Therefore, in practical situations, a continuous replacement of bulletin boards is not possible. Even if an interrupted replacement is allowed, a delay of at least a few seconds would be required to decide whether the resulting replacement interval is acceptable or not. Usually, this delay is not allowed in live transmissions of sporting events. Arbitrary replacement of bulletin boards introduces additional problems. A replacement without a union line requires identifying the foreground objects that cover the bulletin board to inhibit replacement at obstruction sites. The foreground objects consist mainly of the players, but also the ball or other objects. Consider now a player in a red shirt obstructing a part of a similarly red portion of a bulletin board. The color contrast can not be used strongly to identify the obstruction. In addition, since the player is not a rigid object, movement or form information can not be used accurately enough to guarantee a perfect replacement. Another problem that may arise in practical situations is the identity resolution of the bulletin board. Consider two identical bulletin boards placed in two different places on the sand. Assume that different replacement bulletin boards are assigned for each of these physical bulletin boards; then one must be able to say which is which. This can be extremely difficult, especially if there are no unambiguous visible characteristic features. This invention describes a strong system for bulletin board replacement, based on some or all of the following key elements: - Physical bulletin boards that are appropriately colored to allow efficient detection of obstruction by image insertion techniques. - Variation of color or a pattern within the physical bulletin board to further improve the performance of image processing methods. Detectors of horizontal rotation, inclination, approach and focus, fixed to the camera, that after an appropriate installation procedure, allow to estimate the presence and location of the bulletin boards in any given field of video. Image processing methods and their incorporation, which allow to refine the estimates of the detectors. Therefore, the present invention has as its first objective to provide a physical advertisement board comprising an image insertion surface and also provides means for allowing an electronic replacement bulletin board to be replaced with the image insertion board. The present invention, therefore, provides apparatuses for automatic replacement of a bulletin board in a video image, including a video camera for observing the bulletin board, characterized in that the bulletin board comprises an image insertion surface and the apparatus also includes an image insertion unit in operation to detect at least one color of the image insert in which the color of the image insert is adjustable to adjust the color of the image insertion surface. The present invention also provides a method and apparatus that allow the identification of the location of a bulletin board or other static object in a stadium or other scenario in any weather condition, with any horizontal rotation speed of the camera and any other change in the parameters of the camera. Therefore, the present invention provides an apparatus for automatic electronic replacement of an electronic board in a video image, including an automatic camera orientation measuring apparatus, which includes operational motion measurement means for measuring the visual field (FOV ) of the TV camera in relation to a known reference position. The present invention also provides, preferably, an apparatus for automatic electronic replacement of a bulletin board in a video image, including image processing means for processing video signals generated by the TV camera, said processing means include of calibration to periodically calibrate the movement measurement means automatically; apparatus in which the movement measuring means includes means for measuring horizontal rotation, inclination, approach or focus of the camera, in relation to known reference positions; and apparatuses in which the movement measurement means include means for measuring horizontal rotation, inclination, approach and focus of the camera, in relation to known reference positions. Therefore, the present invention uses dynamic recalibration to correct errors or residual aberrations of the detector in an imperfect model and for variation of the detector over time. In this way, it is possible in accordance with the present invention to use less stable detectors, and the apparatus and method according to the present invention can accommodate movement in the position of the camera. The image correction process for calibration of the detectors eliminates the need to keep the detectors stable by mechanical means by automatic recalibration with reference to the video image. In the initial installation procedure, you can incorporate corrections for the calibration of bulletin boards that, for example, are not in the center of the visual field - for example, a bulletin board that is in the upper left corner of the screen can be adjusted by means of, for example, 3 pixels, to take into account the aberrations in the camera. The problems that arise in the prior art systems are first when the bulletin board is substantially totally obstructed, or second, it is obstructed by an object, such as a player, of the same color as the actual signal on the notice board. As explained before, in the first place, this can lead to a lack of recognition of the bulletin board and second, to make difficult the satisfactory replacement of the bulletin board. In the first case, the actual bulletin board may already have been replaced in the video image, but if the camera zooms in, or if a different camera is used for zooming in, then the shutter may be lost because it is only observe a very small portion of the bulletin board. In the second case, the player may bring a strip that is the same color as the bulletin board. The systems of the prior art propose to distinguish the player's bulletin board based on the movement, if the colors are the same, and to analyze the pixels "of movement" to determine the obstruction. This is reasonable in theory, but fails in practice, since not all players move at the same time. In this way, if a number of players move in front of a bulletin board and another player is left after the others have moved, electronics are not able to distinguish over lands of movement. Since colors are distorted by excessive lighting, shadows, differences in reflectivity and different lighting conditions for a foreground player and a background bulletin board, there will be times in practice when the system fails. In such cases, the original bulletin board may reappear on the video image, or the replacement bulletin board will not be obstructed exactly. Again, it is possible to introduce a delay in the video transmission to allow the electronic processing of the signal to be more accurate, but this does not solve the practical problem when a plurality of players move in different directions to obstruct a bulletin board. The necessary delay is considered unacceptable and in any case it will not solve all the previous problems. In accordance with a preferred embodiment of the present invention, it is proposed to replace the actual bulletin boards with image insertion panels or with delineated areas forming image insertion panels. Image insertion is essentially an obstruction technique that allows, for example, a newspaper reader to remain and move on all sides in front of an image insert board, usually blue or other suitable color. The newspaper reader (foreground) is distinguished from the image insertion board (background) by color differentiation, and thus can be moved in front of the replacement background with normal obstruction of the foreground and background. This technique is well known in television studio systems, and is described in numerous U.A.A. patents, including 2,974,190 and 4,200,980. U.S. Patent No. 5,353,392 discloses an electronic bulletin board replacement system in which a real bulletin board is replaced in a video image by a virtual or electronic bulletin board. Obstacles placed between the video camera and the bulletin board are identified by correlation closure with the color or illumination of the bulletin board. PCT W094 / 05118 describes a virtual studio system using an image insertion panel in the background. The panel comprises a uniform pattern of two colors of image insertion with different tint and / or brightness, which enables the position of a camera in relation to the background image insertion panel, provided that the initial position of the camera is known. In the present invention, the bulletin boards are replaced with image insertion panels or with delineated areas forming image insertion panels. However, in the prior art only one image insertion area is identified and this area is illuminated accurately. In the present invention, several bulletin boards may be placed at various sites within a stadium and the present invention allows the various bulletin boards to be replaced by electronic bulletin boards. Also, in a virtual studio application, typically the camera is 2 to 10 meters away from the foreground, and usually the entire field of view is replaced. In comparison, a bulletin board may be several hundred meters away from the camera, and therefore, a replacement system using detectors is much more susceptible to detector errors: - Due to long focal lengths, the The same accuracy of the detector will result in larger geometric registration errors. Consider a rotary encoder of 81,000 pulses / revolution, then the angular precision is 0.0044 degrees or 75 microradians. The repeatability is twice as bad. Consider a 100-shot scale with a 4-m field of view, so the FOV is 40 milli-radians. The error translates to 768 *? 150/20000 = 2.88 pixels.

- Since the visual field includes many stationary objects (including bulletin boards) that are not replaced, the human observer will be much more sensitive to the error log. Additional errors of lens distortion, rotation axes that do not pass through the focal point, non-zero angle of rotation, etc. may occur. Image insertion is basically a technique for studies in which the lighting is carefully designed and controlled, and the controls of the image insertion are carefully adjusted for the specific arrangement of blue screen color and lighting. In a sporting event, the conditions can be highly non-ideal and require some modification of the image insertion algorithms. In particular, the insert parameters must be adapted to the specific bulletin boards that are replaced, due to changes in lighting through the sand. Therefore, in the present invention it is proposed to use image insertion panels and replace these in the video image by means of the replacement bulletin boards. Since it is necessary, for the perfect obstruction, for players or other obstruction objects to be of a different color than the image insert panels, it is proposed in a further preferred embodiment to provide image insert panels in which the panel color, for example, using a rotating bulletin board structure that is known in the art. For example, one side can be blue and the other green. Green can be preferred in a sports environment, since players tend not to use green because this color would not contrast with the sports surface in the background. In a further preferred embodiment, and in particular when a plurality of bulletin boards require replacement, an image insertion board in the form of patterns is used. The pattern can be in any suitable form, but is preferably selected to suit the size and shape of the bulletin board or series of bulletin boards, and also to the anticipated video conditions. In this way, if it is possible to observe a bulletin board only from a large distance, then a different pattern will be selected for a bulletin board that is observed in approach. The pattern may comprise different colors or may be of different shades of the same color. The pattern may comprise vertical and horizontal lines or may comprise a decorative pattern, a discernible advertisement, the logo of a company, or other expression that may be more aesthetically acceptable. The use of a pattern allows additional discrimination of the position of the camera and can allow the movement of the camera from a fixed position. The orientation data of the camera can be transmitted together with the video signal and will identify the position of the bulletin board in any weather, lighting or obstruction condition. No reference is required for any feature within the sports arena to identify the position of the bulletin board. The camera detectors can be tuned up to a few pixels, or in physical terms, up to about 1 cm on a scale of about 100, thereby allowing for the exact replacement of any bulletin board. The recalibration can be carried out continuously or only periodically, in particular if an initial adjustment of the calibration of the announcement boards that is not in the center of the FOV, is registered in the installation. Through the use of image insertion techniques, it is not required to transmit any obstruction data, since these can easily be inserted into a receiver and the obstruction can be inserted in the normal way. However, when the boards are placed at different locations in the stadium, including the image insertion color areas, which are delineated as image insert boards, referred to below as bulletin boards, will be subjected to several different lighting conditions. The problems that the inventors of the present have noticed, is that an image inserter will be installed to detect a particular image insertion color, and under different lighting conditions, the color of a bulletin board will change (defined for include areas in which the bulletin boards are inserted). In this way, the device will be unable to identify exactly the various bulletin boards located around the stadium. The present invention also provides, therefore, an apparatus comprising a bulletin board map memory to record the position of the plurality of bulletin boards and to register the image insertion color of each bulletin board to be replaced, to thereby provide a color map of image insertion for a stadium. A further problem that the inventors have noticed is that if a plurality of different image insertion colors are transmitted to a remote receiver to provide substituted bulletin boards in a local distribution center, then the image insertion equipment in such a center must also be equipped with selective mapping of image insertion. This substantially complicates the local reception apparatus, and an object of the present invention is to solve this problem. The present invention also provides, therefore, an apparatus for additionally including means for transmitting a perfect key color background signal, for each bulletin board to be replaced.

The present invention also provides apparatuses that include means for transmitting the coordinates of each bulletin board to be replaced, and apparatuses that also include receiving apparatuses for the reception of the coordinates of the bulletin board and the perfect color of insertion of the background image, and includes means for providing a combined video image incorporating obstruction in the coordinate areas identified by the perfect background image insertion color. In a preferred arrangement within a stadium or other sports arena, real bulletin boards with normal advertising material will be placed on one side of the stadium to be observed by a first plurality of cameras, and image insertion bulletin boards will be placed on another side or on the opposite side to be observed by a second plurality of cameras. This, for example, the local nation can observe the normal bulletin boards, and the international TV audience, only observe the replaced boards. The present invention also provides a method for electronically replacing a bulletin board in a video image display generated by a camera, comprising the steps of: a) Identifying the position of a rectangle bulletin board color image insertion in a stage or other scenario, said identification step comprises specifying on the video display the bulletin board to be replaced by identifying its four corners in a first camera position. b) Save the identification information. c) Monitor the movement of the camera in horizontal rotation, inclination and approach. d) Save the motion of the camera on a field on a field basis; and e) Analyze the size and position of the image insertion bulletin board to be replaced from the information recorded in its first known position, and of the saved movements of the camera to provide information regarding the size, perspective and position of the bulletin board. announcements in the present video field. f) Save a replacement advertisement board in a bulletin board replacement memory to be used in the replacement of the bulletin board for image insertion in the stadium. g) Electronically altering the size and perspective of the replacement bulletin board in accordance with camera movement information, to fit the size and perspective of the image insert bulletin board to be replaced in the present video frame; and h) Electronically replace the image insertion bulletin board in the present video frame using the replacement bulletin board.

In a preferred embodiment, the step of analyzing the size and position of the image insertion board to be replaced comprises an additional step of analyzing a plurality of video scan lines to provide fine-tuning information in relation to the size , perspective and exact position of the bulletin board to be replaced. In a further preferred embodiment, the image insertion bulletin board to be replaced may be a blue or green shade, due to the fact that these colors are rarely found on human skin and hair. In another preferred embodiment, the image insertion bulletin board is formed in patterns, with a pattern of suitable shape in order to facilitate the fine-tuning process mentioned above. The step of analyzing the size and position of the bulletin board includes the analysis of the pattern to determine the exact position of the bulletin board. In an additional mode, the correction of the prediction based on the detector by means of the pattern analysis will be controlled by a merit figure (accuracy estimate) for the analysis, which will be calculated automatically. In an additional mode, the step of electronically replacing the bulletin board in the present visual field by replacing the bulletin board, includes the step of superimposing obstructing objects through the use of image insertion techniques. In a further preferred embodiment, the bulletin board to be replaced can be changed to better match the colors and shades of colors on the players' clothes, in order to provide good contrast between the bulletin boards and the players. For example, if these clothes contain shades of blue, then a bulletin board may be selected. The background color can be selected between blue, green and red. In order for the color inserter to calculate all the parameters necessary to effect appropriate image composition, the system requires a sample of the color of the background as a reference. This step can be done automatically by sweeping the image and detecting the purest and brightest color. Advanced color inserters allow the user to manually select the area to be sampled. In a further preferred embodiment, the image insertion apparatus will have a multiplicity of installation conditions, each corresponding to a different region of the stadium. The information of horizontal rotation, inclination and approach of the camera, will allow to load the corresponding conditions of installation. In a further embodiment, the fine-tuning information will be used to compensate for variation errors of the detectors. In a practical situation, the detector error will have a significant portion that is at temporal frequencies that are much lower than the field speed of the video. In this way, these errors induced by the detector can be estimated reliably from a good video field and subtracted from the subsequent measurement. The present invention also provides an apparatus for carrying out the electronic bulletin board replacement method as explained above. The embodiments of the present invention will be described below by means of an example with reference to the accompanying drawings, in which: Figure 1 shows a stage or other scenario illustrating the apparatus in accordance with the present invention. Figure 2 shows the video image of the stadium observed by the camera in a first position. Figure 3 illustrates a stadium with bulletin boards in several different positions. Figure 4 illustrates a camera shot in approach of a bulletin board illustrating the problem with the prior art systems. Figure 5 shows an image insertion bulletin board formed in patterns for use with the present invention. Figure 6 shows the circuit diagrams in block diagram form associated with the camera arrangement of Figure 1, for transmitting video data and camera orientation data. Figure 7 shows the receiver circuits in the form of block diagrams for cooperation with the transmit circuits of Figure 6. Figure 8 shows a flow diagram for the operation of the circuits of Figure 7. Figure 9 shows a layout for a bulletin board installation data memory. Figure 10 shows a layout for the bulletin board installation data. Figure 11 shows a flow chart for perspective transformation computation. Figure 12 shows an arrangement for the memory of intrinsic parameters of the camera. Figure 13 shows equations for dynamic recalibration. Figure 14 shows a flow chart for dynamic recalibration. Figure 15 shows the recalibration process. Figure 16 illustrates the problems in relation to the posting of bulletin boards in different places within a stadium with different lighting conditions. Figure 17 shows a graph of the minimum and maximum levels for U &; V, illustrating the operation of a color inserter. Figure 18 shows a bulletin board with obstructor object illustrating the principle of color adjustment by image insertion for a bulletin board. Figure 19 shows an exemplary remote receiver for reception of coordinate data of bulletin boards and perfect image insertion, with the obstruction being performed by means of image insertion techniques. Figure 20 shows an alternative arrangement for the installed data memory of the bulletin board, illustrating an alternative modality. Figure 21 shows a flow diagram for the dynamic installation procedure for horizontal rotation of the camera for use with the installation data memory of the bulletin board of Figure 20; and Figure 22 shows a flow diagram for a dynamic installation method for tilting the camera for use with the installation data memory of the bulletin board of Figure 20. The principle of the present invention is now explained with reference to Figures 1 to 4. In a stage or other scenario 10, bulletin boards 14, 16, 18 are installed on the side of one end represented by marks 12. These bulletin boards are observable by a camera 20. The bulletin boards ads 15, 17, 19 may be present on the opposite side of the stadium for observation by an additional camera 21. The balconies and stadium seats are shown diagrammatically by the lines 11. The camera 21 may, in a preferred example, be a normal TV video camera. and will transmit its output video signal directly to a first feed that can serve the local population. Although reference is made to camera 20 or 21, it can be clearly understood that there could be a plurality of cameras on each side of the stadium that provide different views. The camera 21, in a preferred embodiment, will televise panels 15, 17, 19, which will be transmitted to the local population without changes. The camera 20 in this preferred embodiment will transmit a feed to an international audience. The camera 20 is equipped with orientation detecting means preferably comprising one or more of the following: horizontal rotation measuring means 24; inclination measurement means 25; approach measurement means 26; and focus measuring means 28. Suitable detectors may comprise the Virtual Reality encoder of RADAMEC EPO, Bridge Road, Chertsey, Surrey KT16, 8IJ, England. Depending on the allowed mobility of the camera, only one, several, or all of them may be required. For example, if the camera 20 is fixed in horizontal rotation and inclination and focus and can only zoom in, as in the case of some non-manual remote control cameras, then it is necessary to measure only the approach parameter. Most cameras in sports stadiums can zoom in, tilt and turn horizontally and it is assumed that these parameters are measured for each camera as explained now. It is assumed that the focus is fixed, but in a similar way, the parameter can be added if required. Figure 2 shows the video image as seen by an observer, in particular by the operator of the equipment. The camera 20 zooms in, horizontally rotates and / or tilts to "center" the bulletin board 14 in a suitable position and at a reasonable size. With reference to Figure 7, each bulletin board is observed in a receiver and its position is preferably marked using a touch screen 700, or keyboard mouse 702, and marking the four corners. The positions are stored in a memory 704. For higher advertisement boards in the stadium, such as 30 (Fig. 3), a correction factor for the camera that depends on the camera's tilting position can be saved. Each position of the bulletin board is stored in the memory 704 together with the parameter information of the camera in the reference position for the camera 20, obtained from the parameter information of the camera which is corrected at the time the camera is saved. bulletin board position. The following procedure is preferably repeated for each of the cameras, and for each of the objective bulletin boards. 1.- Point the camera at the target to obtain a stable unobstructed view of the target. Adjust the approach to obtain a large view of the objective and keeping the whole objective within the visual field. 2.- While the camera is not moving, shoot an acquisition device, to record an image of the objective, as well as the corresponding readings of the detectors. 3.- Mark the corners of the lens on the video image. Preferably, a corner detector is used to locate the corners of the lens with sub-pixel precision. This parameter information of the camera is obtained (figure 6) of the detectors mounted on the camera, and the movement of the camera is referred to a first or fixed reference position for each parameter. The movements of the cameras are detected and the signals are fed to a combiner circuit 24 and then to a volatile transmission memory 36., from which the combined video and position data signals are transmitted. During installation, in the receiver (Fig. 7), the volatile receiving memory 706 receives the signals and feeds these signals to a separator 708. The video signal is stored and delayed in a suitable memory 710 and the parameter data of the camera is extracted and stored in memory 712. In the installation, the VDU 700 is used to mark each bulletin board that may require replacement. The camera 20 is rotated horizontally, etc., to move each bulletin board in a suitable position on the screen and its position is recorded in the memory of the bulletin board 704 together with the camera parameters obtained from the memory 712 by means of the processor 714. A replacement bulletin board memory 716 stores a plurality of replacement bulletin boards and these are selectable to be enabled to replace the original bulletin board. The replacement bulletin board is inserted into operation on the video signal in a combiner 718 to provide a modified output video signal 720. The installation procedure can also identify bulletin board sites and camera parameters for various cameras, saving a camera ID from a source 30 (figure 6). In this way, the position memory of announcement board 704 will keep separate lists of bulletin board data for each camera. The operation of the system will now be described with reference to a single bulletin board and a single camera 20.

With reference to figure 4, it is assumed that the bulletin board 14 enters the field of view in an elongated form on the left side of the screen as the camera 20 rotates horizontally following the approach of the position in figure 3. The data of orientation of the camera are constantly received by the receiver, and the processor 714 will constantly match, on a pixel by pixel basis, the video image with the known bulletin board position stored in memory 704. As soon as the bulletin board announcements appear in the video image, the pixels representing the bulletin board will be identified and the pixels of the replacement bulletin board that are related to those pixels will be replaced in combiner 718. The delay will be minimal, since the identification of the pixels is through a process of correlation of direction that will be virtually instantaneous. After a period, the camera detectors may vary and in this case the replacement bulletin board may not align exactly with the original. This can only be for one or two pixels and may not be discernible to the observer. To correct this, two solutions are possible. First, the position of the bulletin board can be restored manually periodically at an appropriate time, for example, when a camera is not active. This requires the cooperation of the operator.

Secondly, a comparison can be made on a pixel-by-pixel basis of the bulletin board, against an original saved bulletin board, and an adjustment of the reference camera parameters can be made in the bulletin board position memory 704 This process can be done automatically at any setting interval or when the processor 704 has a suitable time position. The essential steps of a preferred recalibration process are to transform perspective between the current video image using the camera data to provide an estimated transformed model. A saved image of the bulletin board is then compared to the transformed model to provide a residual video field. The residual distortion between the transformed model and the residual video field is resolved to provide update information and update the estimated transformation, and to thereby provide a calibration correction factor to recalibrate the position of each bulletin board in memory, according to the camera detection information. The replacement of each bulletin board is carried out using the processor 714 (Figure 7), and the various bulletin board parameters and memories using appropriate software programs as described in greater detail now. Figure 8 describes the complete process for determining the position of each bulletin board in the visual field of the camera, and interprets the corresponding part of the bulletin board in volatile frame memory. Since the interpretation and then the composition of the volatile memory of graphics with the volatile video memory by means of image insertion is known technique, the present invention concentrates on the determination of the position of the bulletin board with reference also to the Figures 6 and 7. At the beginning of each video field, the detectors (24, 25, 26) of horizontal rotation, inclination, approach and focus are read. These values, combined with the bulletin board data in the bulletin board installation data memory 704, and the camera data in the intrinsic parameter memory of the camera 712, enable detection and recognition of all the boards of announcements in the FOV of the camera, independently of the video signal. The processing of Figure 1 consists of a circuit on all bulletin boards (m) 802, 804. For each bulletin board, its installation data is retrieved 806 from the bulletin board installation data memory 704, and is used with intrinsic parameters of the camera 808 to calculate the perspective transformation 810 from the bulletin board m to the current field. Then, the replacement bulletin board information is saved (812) in a volatile frame memory. Figure 9 describes the installation data memory of the message board 900 which consists of a separate register 902 ... 904 for each message board in the arena. Said record consists of a static image 906 recorded under favorable conditions, and of the corresponding static installation data 908. The record also consists of dynamic installation data 910 which is calculated using the image processing means in a process known as dynamic recalibration. , which has been briefly described above and will be described further with reference to Figure 11. An alternative procedure is described which provides static and dynamic calibration with reference to Figures 20, 21 and 22. Figure 10 describes installation data (either static or dynamic) 1000 for a single bulletin board. It consists of the readings of the detectors 1002 in the case of installation, the coordinates of the vertex 1004 of the quadrilateral of the bulletin board and the time code of the installation case 1006. The dynamic recalibration method can be explained as follows: Due to variations in inaccuracies of the detectors, to a final calibration table and other practical reasons, it is impossible to predict the exact location of all visible bulletin boards in a given case. However, in many video fields, the visibility of a bulletin board can be such that an exact geometric position correction can be made. Since that position is both temporally and spatially closer to the subsequent video fields, it is preferable to rely on that "lucky shot" by predicting the position of the bulletin board in relation to the exact quadrilateral detector readings and coordinates. Consider for example a bulletin board that leaves the visual field due to a horizontal rotation of the camera. Having a shot of luck while still highly visible, allows flat tracking of the bulletin board through the detectors only, when its visibility does not allow to apply any means of image processing. Figure 11 depicts flow diagram 1100 for perspective transformation computation. A selection logic 1102 of installation data selects the static installation data 1103 or dynamic data 1105 from the installation data memory 806, as described above. This installation data, together with the intrinsic camera parameter, is used to calculate a prediction, based on the detector, of the perspective transformation 1104, independently of the video signal. A dynamic recalibration 1106 based on image processing means is then applied to the prediction. It uses the video signals 1108 and image insertion 1110, as well as the model image 1112 of the bulletin board of the installation data memory 806 (Fig. 8). Based on a quality factor derived from the image processing means, the transformation based on detectors 1118 or the corrected transformation 1116 is produced. If the estimated quality of the geometric correction is high, then the dynamic installation data 1114 is updated . Figures 12, 13 and 14 describe the prediction of the coordinates of the ad board based on detectors in the video field. This prediction uses the reading of the detectors, as well as the intrinsic parameters of the camera. These parameters are described in figure 12 and they have to be tabulated for a parametric design of the space (horizontal rotation, focus). The significance of these parameters is clear from Figure 13, which is now referenced. Be the set of measurements given by horizontal slew detectors, inclination, approach and focus represented by the vector (P, T, Z, F). It is assumed that the angle of inclination is relative to the horizon. Consider an incident point whose image in some case of installation is in the frame-volatile memory coordinates (x «, and«). Also be the measurement vector of the detector in that case (Pß, T8, Ze, F «). In another case, the prediction case, be the detector measurement vector (PP, TP, ZP, FP). It is required to predict the position of the incident point in the frame-volatile memory coordinates (possibly outside the frame-real volatile memory), (xP, yP). To allow the procedure, the rotation matrix for installation is defined as shown in 600, and the rotation matrix for prediction is defined as shown in 602. Next, the perspective transformation matrix between the coordinate systems in plane of Two images are given as shown in 604 and 1402 (Fig. 14). RBP is a 3 * 3 matrix with row and column indices that vary from 0 to 2. RβpCilCj] denotes the term in row i, column j in the matrix. Thus, given the plane-image installation coordinates of the incident point (uß, v8), the predicted position of the incident point in the plane-image coordinates (uP, VP) is given as shown in 606, 1404. The transformation of the plane-to-frame-volatile memory coordinates is achieved as shown in 608, 1406. The aberration compensation is achieved as shown in 608, 1406 (Fig. 14) to provide predicted coordinates of the bulletin board of volatile memory table and perspective transformation data. An efficient way to derive these parameters for a specific pair (approach, focus) is described in [J. Weng et al., Calibration of stereo cameras using a non-linear distortion model, IEEE lOth Intl. Conf. Pattern Recognition (1990), p. 246-253]. The image processing means for geometric correction, which also allow the recalibration process, are now described in relation to figure 15. The image processing means for geometric correction of the prediction based on detectors ee base in the method differential for the estimation of the movement [C. Cafforio and F. Roca, The differential method for motion esti ation, in T.S. Huang, for example, Image sequence processing and dynamic scene analysis, Spring, Berlin, 1983, pp. 104-124]. Let C be the present video field and M the static image of installation of the bulletin board, transformed into perspective according to the prediction based on detectors. Here, only luminance images are considered. Ideally, M and C are identical within the quadrilateral support of the bulletin board. The real differences may include: Obstruction present in C, but not in M. Geometric errors due to detectors and intrinsic errors of the camera parameters. Changes in luminance. Regardless of any difference that is not due to geometric errors, consider a point (x, y) within the quadrilateral support of the bulletin board.

Let (p, q) be the local geometric error; then, it can be written for the luminance signals of the respective images: M (x + p.y + q) = C (x, y) Under the assumption that the error is small, an expansion of the Tailor series can be written: dM dM M (x + p, y + q) = M (x, y) + p + q + (terms of dx and second order) Regardless of the second order terms and denoting the spatial derivatives dM / dx = H dM / dy = V you get C (x, y) -M (x, y) = pH + qV Denoting also the differences C (x, y) - M (x, y) by D, we obtain D = pH + qV The above equation persists, locally. For a global bulletin board solution, and assuming there is a small error, the perspective model [G. Adiv, Determining Three-Dimensional Motion and Structure from Optical Flow Generated by eveveral moving objects, IEEE Trans. Pattern Analysis and Machine intelligence, 7, pp. 384-401, 1985]. aix + a2y + a3 The coefficients a8 are calculated by minimizing the following expression: f2 / p24 Now, the matrix of the raneformation of the perspective (based on the prediction of the detectors) is multiplied by: You can consider the matrix obtained from the updated prediction from the bulletin board perspective. In a practical environment, the following suggestions can be applied. The obstruction can cause great problems in this formulation, because if the pixels of moving and obstructing objects participate in the minimization of the previous expireion, they could influence the eignificativamente eolución. Preferably, said pixels are discarded from processing using image insert panels. To discard these pixels, a key signal output is preferably used by an image inserter. - Variations in luminance can be reduced to a minimum by previously processing the preend video field, using histogram adaptation techniques. - The prediction-correction process may require 2 to 3 repetitions to converge. The immunity and the convergence of the noise can be increased by previously matching the images.

Thus, the bulletin boards 14, etc., are in accordance with the image insert boards of the present invention, and the obstruction is by color discrimination using standard image insertion techniques. These techniques will improve the obstruction, provided that the players do not wear any color that is the same as that of the board. This may not always be possible, and it is proposed in accordance with a particular embodiment of the present invention, to use boards that can rotate or otherwise change to a second or third color. For example, three colors can be blue, green and red, which can be selected when the color of the stripes of the players is known. Alternatively, it is required to display a bulletin board in an area of the field or a surrounding area, and then said area must be selected to be of a known color which can be recorded after the image inerter as an image insertion color. In a preferred embodiment, the image insertion apparatus may comprise the device formed by ULTIMATTE-7 digital video image from ULTIMATTE Corp., 20554 Plummer St., Chatsworth, CA 91311, USA The color of reinforcement may be selected from blue, green and red. In order for the image inserter to calculate all the necessary parameters to make the proper composition of the image, the system requires a sample of the background color as a reference. This step can be done automatically by scrutinizing the image and detecting the purest and brightest color. The advanced image inserters allow the user to manually select the area to be sampled. In a particular modality, it is proposed to use an image inertion panel dietributed in patronee. Calibration of the camera detectors can easily be achieved by comparing the pattern on a pixel by pixel basis. The pattern on the panel of the announcement board should preferably have lower critical dimensions than the anticipated error of the detector (projected for world coordinates). In summary, the previous seven can operate even in extremely adverse climatic conditions, since the electronic processing circuitry knows exactly where each bulletin board is located, and does not depend on any analysis of the video image to detect the bulletin board. In case the video image is so distorted that the recalibration can not be carried out with reasonable certainty, then the original settings of the camera parameters can continue to be used, since the video image as displayed will be of poor quality and thus the observer will not notice an error in one or two pixels in the position of the replacement bulletin board that will need to be displayed with an equivalent quality that matches the poor quality video image.

In a further preferred embodiment of the present invention, the problem in question is that of having advertisement boards placed in different positions in a stadium, as shown in Figure 16. In said conditions, the illumination of the bulletin boards, 1, 3 and 5 will be different due to the location of lights 7, 9 and 13. Also this lighting can change all the time during the game. If said bulletin boards are image insertion boards, all of the same color, it will then appear that all bulletin boards are slightly different colors due to the different lighting conditions. A fixed adjustment of a global reinforcement color could result in a partial separation of the object's background by the image inserter. In the present invention, it is proposed to provide spatial adaptation of the reinforcement color map, so that the image inserter can correctly recognize each bulletin board. This can be provided by storing in the memory 704 (Figure 7) a spatial map that provides information regarding the color of each image insert board. Thus, the image inserter will compare the color in each video position within a specific color associated with the bulletin board in that position. In a preferred embodiment, the positions of the bulletin boards can be identified by "painting" a slightly elongated box around the bulletin board to identify the position. These boxes are identified as 1 ', 3' and 5"by dotted lines in Figure 16. • The system will track the reinforcement colors over time and, therefore, will be updated continuously, to ensure correct identification after which It has been installed correctly The operation of the system is as follows: First, in relation to Figure 17, minimum and maximum levels are set for U and V. These should be quite large to cover all bulletin boards that are reasonably illuminated. , for each bulletin board, as the lighting conditions change, an adjustment can be made to stored values, as shown in Figure 18, which supposes obstruction of bulletin board 1 by an object 13. An internal frame is defined 1"to ensure that they are only considered pixels within 1. Most of the pixele obetaculizantee can deecartaree, since eetoe eerán a different color. Then, all the pixels (YuV) within the FOV and the quadrilateral 1"of the bulletin board are measured, and an addition is made to the average (which is the average of the UV reinforcement color on the bulletin board) if: Umin <U> V or <V> V The inventors have recognized an additional problem that arises from the use of image insertion boards in a stadium, due to the variable lighting as described above, each The bulletin board will appear on the video image as a slightly different color To transmit correct information about the obstruction, it is necessary to transmit an obstruction map for each bulletin board In accordance with a preferred embodiment of the present invention, it is proposed to transmit , for each bulletin board, a perfect background color and then allow an image inserter in each receiving station to insert the obstructed portions by means of procedures normal images of image insertion. Now consider the layout of the bulletin board as shown in Figure 16. It will appear that each bulletin board 1, 3 and 5, due to its different lighting conditions, is of a different color even though this color may be within the maximum and minimum limits as described in Figure 17. In accordance with this preferred embodiment of the present invention, the transmission apparatus (see Figure 6) will transmit a perfect image insertion color within the area of the bulletin board, and will also transmit the coordinates of the quadrilateral formed by the bulletin board. In this way, the receiving station only has to decode / extract the coordinates of the quadrilateral from the bulletin board, and then inside that quadrilateral, replace those pixels that are of the perfect color of image inertia by the replacement bulletin board. Those pixels that are not of a perfect image insertion color, are not replaced. According to this system, it is not necessary for the image inserter in the remote receiving station to be able to recognize different bulletin boards and to have to store different image insertion values for each bulletin board. It is also not necessary to transmit all information about the object, since the impedance by the image inverter will be relatively simple in each remote position. Referring to Figure 7, the memory of the position of the bulletin board and the color of reinforcement 704 knows the position of each bulletin board, and a 7042 control output of the memory is used, in combination with the video output 7102 to provide inputs for a color reinforcing processor 7044 that can change the color of the bulletin board within the coordinates provided by the memory 704. The output of the processor 7044 is used to control a reinforcing color memory 7046 that changes the color color of the bulletin board within the required coordinates, and may also provide the coordinates for video output 720 for the remote receiver. These can be transmitted by a seventh traffic data video pattern. An example of a remote receiver is shown in Figure 19. The video data are received in the volatile memory of the receiver 1900, and divided and delayed in 1902, 1904. The memory of the coordinates of the bulletin board 1906 stores the transmitted coordinates of the bulletin board and, in combination with the graphics generator 1908 and the image memory of the replacement bulletin board 1910, provides an output signal for an image combiner / inserter 1912 to produce the bulletin board. Admitted obeyed on the screen. Referring now to Figures 20 to 22 in a further embodiment, the installation data stored in memory 704 is modified before any event is televised. The modification comprises the addition of dynamic installation data, as well as the static image data and the static installation data shown in Figure 9. The additional data may be used in place of the installation data by dynamic recalibration 910 shown in FIG. Figure 9, or can be used additionally. In a preferred embodiment, it is assumed that the additional data are used in place of the dynamic recalibration procedure, which is described below. As an introduction, the problems associated with the replacement of advertisement boards with virtual advertisement boards are discussed. The same problem is to identify the position, size and perspective of the original bulletin board, and then replace it with the virtual or replacement bulletin board. In a static camera situation, there is no real problem once the original coordinates have been recorded, provided that the camera detectors do not derive substantially over time. However, the inventors have found that during the horizontal rotation or rapid tilt of the camera, the coordinates 908 of the replacement bulletin board, as recorded in the memory 900, do not coincide with the actual position of the bulletin board in the stadium or stage. This is because the camera detectors exhibit a certain degree of histéreeis. This can be compensated for by the dynamic recalibration procedure already described, but this may not be practical in some circumstances such as during rapid horizontal rotation with substantial obstruction of the objective bulletin board. Possibly the hysteresis can be counteracted by a simple percentage error built in the movement of the camera, but this does not produce very good results because it does not take into account the angle of the camera with respect to each bulletin board, nor does it take into account the variability in the detectors of the parameters of the camera with the angle. Therefore, in the present invention, in an alternative embodiment, in addition to the static installation data of the bulletin board, the installation data memory of bulletin board 900 stores data for each bulletin board and for each camera so less in relation to the left-right horizontal rotation 2002, the right-left horizontal rotation 2004, the up-down inclination 2008, and the downward-upward tilting 2006. The data is obtained and stored as described now following in relation to figures 20 to 22. Figure 20 demotes the memory 900 modified to provide, in addition to the etatic image data for the bulletin boards (B / B) the M and the eetatice inetalation data for the boards of announcements the M, another four data seriese for each bulletin board I to M, and these are multiplied to provide this data for each camera. For each camera, the position of each announcement board is recorded with the camera rotating horizontally from left to right 2002 and from right to left 2004. The horizontal rotation speed can be selected as the normal speed for the event being televised. Thus, for example, for a horse race, it may be very low, but for a race car, it may be longer. The position of each bulletin board was recorded afterwards with the camera tilting upwards 2006 and afterwards downwards through 2008 through each bulletin board. For each measurement, the approach and focus of the camera are preferably set at a known level in which the bulletin board that is being analyzed is in a reasonable display at a reasonable size. The approach and the focus can be, for example, the same as those during the acquisition of the static installation data for each bulletin board, so that a direct comparison can be made with the static installation data. In such a case, it may only be necessary to record an error correction figure. It is preferable not to select a horizontal or tilt speed too high, because at very high speeds, the bulletin boards will be confusing anyway, and therefore, the accuracy of the replacement will not be a problem. This procedure is followed for each bulletin board and for each camera, and the data is used during the event to correct the score of each bulletin board as the camera rotates horizontally or tilts. You can see that each bulletin board will be seen at a low angle for each camera, and also that the output of each detector in each camera can vary, depending on the angle through which the camera should rotate to update the board. of ads. By registering the static data and the data relating to horizontal rotation and tilt in both directions, the replacement bulletin board will be located precisely in the exact position of the original or real bulletin board for static shots and when the camera is in motion. Dynamic recalibration during the event, as described above, will ensure, except during very fast camera movements with great obstruction, that the replacement bulletin board is correctly positioned, but the use of static and dynamic installation data will also ensure this. , unless the camera detectors derive substantially during an event. Thus, as long as the camera detectors are of reasonable quality from the point of view of drift, they can be of variable quality with respect to accuracy during horizontal rotation and inclination. By careful selection of the camera detectors, therefore, extremely accurate detectors are not required, since any variation with respect to camera movement is compensated by dynamic installation data storage. The data is obtained as described in relation to figures 21 and 22 below. After the static data relative to the image of each bulletin board and its static installation data have been obtained (906, 908, Figure 9), the sequence 2100 is started, and the camera is rotated horizontally in 2102 by the operator at a desired speed with respect to the normal horizontal rotational speed. The detectors indicate the direction of the horizontal rotation 2106 and, depending on the direction, the dynamic data are stored in the memory 2002 or 2004 in the steps 2106 or 2108 by selecting that memory. The sequence for left-right and right-left memories 2002 and 2004 is similar, and will be described for left-right memory, but using reference numbers for both memories. As the camera rotates horizontally from left to right, each bulletin board is identified from the data stored in memory 906 of step 2110; 2112. The seventh question asks if the bulletin board has been previously dynamically registered in step 2114, 2116 and, if so, returns to the beginning of the sequence and repeats steps 2104-2110 until you find a bulletin board that does not has been scrutinized dynamically. After a new bulletin board has been found, the position (coordinates) of the bulletin board during horizontal rotation is recorded and compared to the static parameters of the bulletin board previously stored (908) in step 2118, 2120. Any r is calculated (step 2122, 2124), and r is stored in the memory 2126, 2128 of the left-right and right-left horizontal rotation for the announcement board. The system asks whether all the announcement boards registered in the memory 908 have been scrutinized dynamically for the horizontal left-right and right-left turns (steps 2130, 2132). If not, the sequence ee continues that the last bulletin board has been scrutinized dynamically, and then the program is terminated at 2134, 2136. A similar sequence of the program shown in Figure 22 is provided for the tilt of each camera . Obviously, if the camera is not tilted or is unlikely to be tilted to some degree, then this sequence and the data record in the 2006, 2008 reports may not be necessary. The sequence starts at 2200, and each camera tilts in turn at 2202, and the tilt direction is determined at 2204 by the camera detectors. Depending on whether the camera is tilting up or down, the dynamic installation data is stored in the 2006 or 2008 memory in steps 2206, 2208. Both sequences are similar, and only the camera's tilting sequence will be dictated. down in relation to both sequences. Each bulletin board is identified in 2210, 2212 from the static image data, and also from the camera parameters, especially where all the real bulletin boards are the same. The program intgates the datum of the announcement board at step 2214, 2216 to see if the bulletin board has already been intgated.

If so, the program has to start again, but if not, the data of the coordinates of the announcement board during the inclination are compared with the static data of step 2218, 2220. The error, if any, is calculated in step 2122, 2124, and stored in the 2006, 2008 reports (Figure 20), step 2226, 2228. The program interrogates after lae memoriae 2006, 2008 to see if all the bulletin boards have been dynamically interrogated for errors upward (step 2230) and downward (step 2232) and, if so, the program ends in steps 2234, 2236. If not, the program continues beginning at the beginning of the sequence, until all the boards have been interrogated. Normally, the approach and focus of the camera will not require that the same type of dynamic installation data be stored. However, if particular aberrations of the camera are known, they can be compensated by using dynamic installation data eimilaree. The dynamic data stored in the memories 2002-2008 can be used instead of, or in conjunction with, the dynamic recalibration data obtained as described in relation to Figure 9. However, usually, dynamic installation data avoid the need to perform recalibration during most types of events. During use, the seventh eabe, by reading the camera detectors, if the bulletin board is being displayed in a static way, or if it is being rotated horizontally after left-right or right-left, or if it is then tilted up or down. In such cases, the scoreboard poem is taken from the static data memory and then if inclination or horizontal rotation is occurring, the necessary error corrections are applied. After the movement of the camera ceases, the static parameters of the bulletin board are reversed.

Claims (6)

NOVELTY OF THE INVENTION CLAIMS

1. - An apparatus for the automatic replacement of a bulletin board (14) in a video image that includes a video camera (20) for displaying the bulletin board, characterized in that the bulletin board comprises an image insertion surface and because the apparatus further includes an image insertion unit (704,7044,7046) effective to detect at least one image insertion color of the image insertion surface of the announcement board, and wherein the reference color of the image insertion unit is adjusted to conform the color of the image insertion surface, thereby allowing the image insertion unit to correctly identify the surface of the bulletin board and correctly replace the image insertion surface of the bulletin board by a virtual advertisement in a video image with correct obstruction of any foreground object.

2. The apparatus according to claim 1, characterized in that it also comprises a memory of the map of the bulletin board (704) to record the poetry of the plurality of advertisement boards, and to register the color of image inertion of each announcement board (14,16,18) that will replace, to provide a color map of the image insertion for a stadium.

3. The apparatus in accordance with the claim 2, characterized in that it also comprises medium (7044,7046) for automatically adjusting the image insertion color map according to the variations in lighting conditions within the stadium.

4. The apparatus according to any of claims 1 to 3, characterized in that the camera incorporates automatic means of measuring the orientation of the camera (24,25,26,28), including effective measurement means of movement to measure the visual field (FOV) of the camera with respect to a known reference position.

5. The apparatus according to claim 4, including image processing means for processing video signals generated by the camera, characterized in that said image processing means (714) include calibration means to periodically and automatically calibrate the movement measurement means.

6. The apparatus according to any of claims 1 to 5, characterized in that it also includes means (716, 718) for transmitting, for each advertisement board to be replaced, a perfect key signal of background colors. 1. - The apparatus according to claim 6, characterized in that it also includes means (700,702,704) to transmit the coordinates of each bulletin board that will sub-text. 8. The apparatus according to claim 7, characterized in that it also includes a receiving device (1900) to receive the coordinates of the bulletin board and the perfect color of the background image insertion, and because it includes means to provide an image of combined video that incorporates obstruction in the areas of the coordinates identified by the perfect color of the background image insertion. 9. The apparatus in accordance with the claim 2, characterized in that it further comprises a dynamic bulletin board memory (2002) for recording any change in the measured position of the bulletin board as the camera rotates horizontally or tilts. 10. A method for the automatic electronic replacement of bulletin boards in a sports stadium, said bulletin boards having image insertion surfaces in a video image comprising the steps of: i. identifying a first specific area corresponding to a bulletin board in the video image, said specific area having a first defined color of image insertion of the image insertion surface of a first bulletin board; ii. returning the first color of said first specific area in an image insert color memory; iii. identify a first electronic replacement bulletin board for substituting in said first specific area; and iv. replacing, by means of image inertia technique, said first replacement announcement board in said video image, said replacement including the task of obeying those parts of the first specific area that were obetruded in the video image by a object attacking. 11. A method according to claim 10, comprising the steps of: i. identifying other specific areas in the video image, each additional specific area comprising an additional bulletin board having a related image insertion color; ii. returning the color of all the specific areas in an image insertion color memory together with the information relating to the position of each of said specific areas; iii. identify replacement bulletin boards for substitution in each of these specific areas; and iv. replace sub-bulletin boards in the respective correct positions with obstruction by obstructing objects. 12.- A method according to the claim 10, which includes the steps of: updating the registered image insertion colors for each specific area at appropriate time intervals to compensate for changes in lighting within a stadium. 13.- A method in accordance with the claim 11 or claim 12, which also includes the following: i. transmit, together with the video image, information data that identify the specific area comprising the position and size of each of a plurality of advertisement boards; ii. tranemitir, for each specific area identified as an announcement board, a color of perfect image inertion; iii. receiving in a receiver, the data identifying each of a plurality of advertisement boards to be replaced, and information about the perfect image insertion color for said specific areas; iv. comparing the perfect image insertion color with a pre-selected perfect fitting image insertion color in an image insertion apparatus in said receiver; v. substituting a plurality of locally selected announcement boards in said receiver in the specific area identified for each bulletin board; and I saw. Inserting obstructive objects using image insertion techniques for each bulletin board using an image enhancer in which the color of image inertia is adjusted to detect the perfect image insertion color.