TWI709337B - Image-capturing system - Google Patents

Abstract

An image-capturing system is provided to overcome the problem where the image quality is poor during the movement of the image-capturing system. The image-capturing system includes an image-capturing unit configured to generate a fixed image, an audio-receiving unit configured to generate an audio signal, a storage unit configured to store the fixed image and the audio signal, and a processor configured to control a detecting and tracing module to perform an image detection on the fixed image to detect whether there exists a target object in the fixed image. If so, the processor generates a traced area in the fixed image. The traced area contains the target object. The processor controls a synchronization module to perform a post processing on the image and the audio signal contained in the traced area of the fixed image, thereby generating a video document.

Description

Video recording system

The present invention relates to a photographing system, in particular to a recording system capable of automatically tracking and recording the lecture process of a speaker.

In order to pursue better video quality, the conventional video recording method uses several cameras to shoot the panorama of the classroom with different shooting angles. However, in order to ensure that the speaker continues to appear in the screen of the filmed video, even in the middle of the screen, it is necessary to spend extra manpower to control the cameras so that the shooting direction can be positioned on the speaker at any time. After the shooting is completed, it will take time to post-produce the video.

With the advancement of science and technology, various automatic tracking and shooting teaching systems have also been introduced. Please refer to Figure 1. It is a conventional tracking and shooting teaching system 9. The conventional tracking and shooting teaching system 9 has a camera 91 and an automatic Tracking device 92, the camera 91 is a PTZ camera; the automatic tracking device 92 is electrically connected to the camera 91, and based on the video image captured by the camera 91, the face and head and shoulders classifier is used to identify a tracking aims. The automatic tracking device 92 determines the position and size of the tracking target in the video image, so that the tracking target is located at a suitable position in the video image and displayed as a suitable size. The automatic tracking device 92 sends a control command to make The camera 91 performs tracking and shooting of the tracking target. An embodiment of the tracking shooting teaching system 9 similar to the prior art has been disclosed in the patent case of "A single-camera teacher automatic tracking method, device, and recording and broadcasting system" in China Public Notice No. 105718862.

In the above-mentioned conventional tracking shooting teaching system 9, when the tracking target tracked by the automatic tracking device 92 moves, the automatic tracking device 92 must first find the location of the tracking target, and then transmit the location information to the camera The device 91 turns the camera 91 to the direction of the tracking target, and performs shooting of the tracking target. Since the camera device 91 rotates with the movement of the tracking target, the image recorded by the camera device 91 may have a problem of uneven image.

In view of this, the conventional tracking shooting teaching system does still have to be improved.

In order to solve the above-mentioned problems, the purpose of the present invention is to provide a video recording system that can automatically track the speaker and record the speaker's speech.

The second purpose of the present invention is to provide a video recording system that can post-process the written content, conversation content, and teaching material content of the lecturer during the lecture to generate a video file.

The elements and components described in the full text of the present invention use the quantifiers "one" or "one" for convenience and to provide the general meaning of the scope of the present invention; the present invention should be interpreted as including one or at least one, and single The concept of also includes the plural, unless it clearly implies other meanings.

The "fixed screen image" mentioned in the full text of the present invention refers to a situation where a fixed camera is set up and the viewfinder of the fixed camera can always see the same image range and field of view area. A fixed picture produced when shooting a fixed active area of the speaker.

其中，Speed _K 為該基準點與該特寫區域水平距離或垂直距離為K時的追蹤區域之位移速度，K=1,2,3,…,N，N為該特寫區域與該追蹤區域之間的一水平距離及一垂直距離；Speed _min為該追蹤區域的一最小位移速度；Speed _max為該追蹤區域的一最大位移速度，該處理器控制該同步合成模組將該固定畫面影像中，該追蹤區域所涵蓋部分之畫面影像與該音源訊號進行後製，以產生一視頻文件。 The camera recording system of the present invention includes: a camera unit for shooting toward a target area to generate a fixed screen image; a radio unit for performing radio reception on the target area to generate an audio signal; a storage unit To store the fixed screen image and the audio signal; a synchronous synthesis module to synthesize video signals and audio signals synchronously; a detection and tracking module to detect and track a target; and The processor is coupled to the storage unit, the synchronization synthesis module and the detection and tracking module. The processor reads the fixed screen image from the storage unit and controls the detection and tracking module to execute the fixed screen image Detect to detect whether the target exists in the fixed screen image. If the detection result is yes, a tracking area is generated in the fixed screen image. The tracking area has the target, and the processor uses The tracking area continues to track the target, the processor sets a close-up area in the center of the tracking area, the processor sets a reference point for the target in the close-up area, and recognizes whether the target exceeds the For the close-up area, if the identification result is yes, the tracking area is displaced so that the target is located in the close-up area again. The calculation method of a horizontal displacement speed and a vertical displacement speed of the tracking area is shown in the following equation:

Among them, Speed _{K is} the displacement speed of the tracking area when the horizontal or vertical distance between the reference point and the close-up area is K, K=1,2,3,...,N, and N is the distance between the close-up area and the tracking area Speed _min is a minimum displacement speed of the tracking area; Speed _max is a maximum displacement speed of the tracking area. The processor controls the synchronous synthesis module in the fixed frame image. The screen image and the audio signal of the part covered by the tracking area are post-processed to generate a video file.

Accordingly, the camera system of the present invention can use the camera unit to shoot toward the target area to generate the fixed screen image, and use the detection and tracking module to detect and track a target on the fixed screen image; When the detection and tracking module detects the existence of the target in the fixed screen image, the processor can generate a tracking area in the fixed screen image, and combine the part of the screen image covered by the tracking area with the audio source The signal is post-processed to generate the video file. So this The video recording system can track the target object without rotating the camera unit, and has the effect of improving the smoothness of the image of the video file. Moreover, the processor does not need to move the tracking area frequently so that the target is located in the center of the tracking area, and only needs to move the tracking area when the target exceeds the close-up area, and the The displacement speed of the tracking area can be adjusted according to the different moving speeds of the target, which has the effect of improving the smoothness of the moving image of the video file and preventing the target from being lost.

Wherein, the processor reads the fixed screen image from the storage unit, calculates and obtains the number of missing frames of the fixed screen image, and controls the synchronous synthesis module to perform image compensation on the video file with the frame number. In this way, it has the effect of further improving the smoothness of the video file.

Wherein, the detection and tracking module uses a Gaussian mixture model to obtain a foreground object in the fixed screen image, and the detection and tracking module sets a region to be determined for the foreground object, and analyzes the skin tone ratio of the region to be determined Whether it is greater than a skin color threshold, if the analysis result is yes, it means that the target has been detected in the fixed frame image. In this way, it is possible to reduce the influence of the change of light and shadow of the fixed screen image caused by the sunlight in the target area and the change of indoor lighting, and it has the effect of improving the tracking accuracy.

Wherein, the detection and tracking module uses a background sensing correlation filter tracking algorithm, or a deep learning algorithm to track the target. In this way, the foreground object and the background object in the fixed screen image can be effectively distinguished, and the accuracy of identifying the target object can be improved.

Wherein, the detection and tracking module does not detect the target in the tracking area, and the processor calculates that the time that the detection and tracking module does not detect the target is longer than a detection time, then controls the The detection and tracking module re-executes the detection of the target. In this way, when the target is lost, the detection and tracking module can be activated to detect the target again.

The camera recording system of the present invention may further include a recording and playback module, which is coupled to the processor and used to record a computer screen operation screen, and the processor controls the same The step synthesis module post-processes the part of the screen image covered by the tracking area, the audio signal and the computer screen operation screen to generate the video file. In this way, in addition to the screen image and the audio signal, the content of the video file can also have the computer screen operation screen, which has the effect of improving the quality of the video file.

Wherein, the processor controls the recording and playback module to generate and play a picture-in-picture from the video file, and the picture-in-picture is used to display the tracking area and the computer screen operation picture respectively. In this way, it is possible to play the explanation video and the content of the slide file on the computer screen operation screen at the same time, which has the effect of improving the learning quality and viewing experience.

Wherein, the processor determines whether a writing area is covered in the tracking area by establishing a type of neural network model, and if the judgment result is yes, the processor calculates whether the writing area continuously appears in the tracking area for the time It is longer than a time threshold. If the calculation result is yes, the processor sets a to-be-determined area in the tracking area and confirms whether the skin color ratio of the to-be-determined area is not greater than a skin color threshold. If the confirmation result is yes, it will The writing area in the tracking area is zoomed in close-up to generate a writing zoomed-in screen, and the recording and playback module presents the writing zoomed-in screen with the parent screen in the picture-in-picture, and presents the writing zoom screen with the sub-picture in the picture-in-picture Computer screen operation screen. In this way, when the speaker is writing, by zooming in and close-up the writing area, the viewer can clearly see the content and key points written by the speaker in the writing area, which has the effect of improving the quality of learning and viewing experience .

Wherein, the processor determines whether a writing area is covered in the tracking area by establishing a type of neural network model, and if the judgment result is yes, the processor calculates whether the writing area continuously appears in the tracking area for the time It is longer than a time threshold. If the calculation result is yes, the processor sets a to-be-determined area in the tracking area and confirms whether the skin color ratio of the to-be-determined area is not greater than a skin color threshold. If the confirmation result is no, then The mother screen in the picture-in-picture presents the computer screen operation screen, and the sub-picture in the picture-in-picture presents the tracking area. in this way, When the lecturer is based on the content of the slide file on the computer screen operating screen, the viewer can clearly see the content and key points of the slide file on the computer screen operating screen, which can improve the learning quality and viewing experience effect.

Wherein, the camera recording system of the present invention may further include a cloud server for coupling the processor and at least one terminal device, and the processor uploads the video file to the cloud server in real time , For the user of the terminal device to watch the video file online. In this way, the system has the function of providing online live broadcast.

〔this invention〕

1: Camera unit

2: Radio unit

3: storage unit

4: Synchronous synthesis module

5: Detection and tracking module

6: processor

7: Recording and playback module

8: Cloud server

A1: Tracking area

A2: Close-up area

A3: Writing area

E: terminal device

F: Fixed screen image

P: reference point

[Learning]

9: Tracking shooting teaching system

91: camera device

92: Automatic tracking device

[Figure 1] A system block diagram of a conventional tracking shooting teaching system.

[Figure 2] A system block diagram of a preferred embodiment of the present invention.

[Figure 3] A schematic view of the recording area of a preferred embodiment of the present invention.

[Figure 4] A schematic diagram of a fixed screen image of a preferred embodiment of the present invention.

[Figure 5] A schematic diagram of adjusting the displacement speed of the tracking area in a preferred embodiment of the present invention.

In order to make the above and other objects, features and advantages of the present invention more comprehensible, the preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings: Please refer to Figure 2. It is a preferred embodiment of the camera recording system of the present invention, which includes a camera unit 1, a radio unit 2, a storage unit 3, a synchronous synthesis module 4, a detection and tracking module 5, and a processor 6 , The processor 6 is coupled to the storage unit 3, the synchronous synthesis module 4 and the detection and tracking module 5.

Please refer to Figures 3 and 4 together, the camera unit 1 is used to face a target area Perform shooting to generate a fixed frame image F. For example, the camera unit 1 may be a fixed camera with zoom in and zoom out functions.

The radio unit 2 is used to radio the target area to generate an audio source signal. For example, the radio unit 2 may be a radio microphone.

The storage unit 3 is used to store the fixed screen image F and the audio signal. The storage unit 3 can be any storage medium used to store electronic data, for example: a mechanical hard disk (HDD), a solid state hard disk Non-volatile memory such as a hard disk (SSD) or a flash memory (Flash Memory).

The synchronous synthesis module 4 is used for synchronously synthesizing the video signal and the audio signal. For example, the synchronous synthesis module 4 can be a general video and audio merging software tool, and supports video formats such as 3GP, AVI, FLV, MP4, MPG, WMV, or a combination of the above, but not limited to this.

The detection and tracking module 5 is used to detect and track a target. In this embodiment, the target can be a speaker. Specifically, the detection and tracking module 5 uses a Gaussian Mixture Model (GMM) to obtain a foreground object in the fixed frame image F, that is, establishes a background model for the fixed frame image F, and borrows According to the background model, each pixel in the fixed frame image F is presented using a mixture of several Gaussian distributions to obtain the foreground object in the fixed frame image F. Wherein, when there is noise in the fixed frame image F, image processing such as erosion and dilation can be performed on the fixed frame image F, which can be understood by those with ordinary knowledge in the art.

The detection and tracking module 5 can set a to-be-determined area for the foreground object. Preferably, the to-be-determined area can be set as the head area of the target, but it is not limited to this. The detection and tracking module 5 can analyze whether the skin color ratio of the area to be determined is greater than a skin color threshold. In this embodiment, the skin color threshold may be 40%. If the analysis result is yes, it means that the detection and tracking module 5 has been The existence of the target is detected in the fixed screen image F; if the analysis result is negative, it means that the detection and tracking module 5 has not detected the existence of the target in the fixed screen image F. On the other hand, the detection and tracking module 5 uses a tracking algorithm to track the target in the fixed frame image F. In this embodiment, the tracking algorithm may be a background sensing correlation filter (Background -Aware Correlation Filters, BACF) or a deep learning algorithm (Deep Learning), for example: YOLO.

Please refer to Figures 2 and 4, the processor 6 is coupled to the storage unit 3, the synchronization synthesis module 4 and the detection and tracking module 5. The processor 6 can be any computing and signal generation function Electronic devices, such as: programmable logic controller (PLC), digital signal processor (DSP), microcontroller (MCU), or circuit board with the above functions, etc. The processor 6 reads the fixed screen image F from the storage unit 3, and controls the detection and tracking module 5 to perform detection on the fixed screen image F to detect whether there is the target in the fixed screen image F Yes, if the detection result is yes, the detection of the target by the detection and tracking module 5 can be suspended, and a tracking area A1 is generated in the fixed frame image F, and the tracking area A1 has the target , The processor uses the tracking area A1 to continuously track the target object, and the processor controls the synchronization synthesis module 4 to perform tracking between the fixed frame image F and the part of the frame covered by the tracking area A1 with the audio signal Post-production to generate a video file; if the detection result is no, and the processor calculates that the detection and tracking module 5 does not detect the target for longer than a detection time, then the detection and tracking module is controlled Group 5 re-executes the detection of the target. In this embodiment, the processor 6 can set the detection time to 3 seconds.

其中，Speed _K 為該基準點與該特寫區域水平距離或垂直距離為K時的追蹤區域之位移速度，K=1,2,3,…,N，N為該特寫區域與該追蹤區域之間的一水平距離及一垂直距離；Speed _min為該追蹤區域的一最小位移速度；Speed _max為該追蹤區域的一最大位移速度。 Please refer to FIG. 5, the processor 6 can set a close-up area A2 in the center of the tracking area A1, and identify whether the target exceeds the close-up area A2, and if the identification result is yes, it will shift the tracking area A1 , To make the target relocate in the close-up area A2; if the identification result is no, there is no need to shift the tracking area A1. Specifically, the processor 6 may set a reference point P for the target in the close-up area A2, and when the target exceeds the close-up area A2, the processor 6 will determine the difference between the reference point P and the close-up area A2. The horizontal distance and the vertical distance are used to adjust a horizontal displacement speed and a vertical displacement speed of the tracking area A1. The calculation method of the horizontal displacement speed and the vertical displacement speed of the tracking area A1 can be as shown in the following equation (1):

Among them, Speed _{K is} the displacement speed of the tracking area when the horizontal or vertical distance between the reference point and the close-up area is K, K=1,2,3,...,N, and N is the distance between the close-up area and the tracking area Speed _min is a minimum displacement speed of the tracking area; Speed _max is a maximum displacement speed of the tracking area.

The video recording system of the present invention can use the method of performing image processing while recording to generate the video file. However, according to the difference in the degree of use of the processor 6, the writing speed of the processor 6 when writing the fixed image F to the storage unit 3 may be inconsistent, resulting in the fixed image stored in the storage unit 3. The image F has dropped frames. Therefore, after the processor 6 reads the fixed frame image F from the storage unit 3, the processor 6 calculates and obtains the number of missing frames of the fixed frame image F, and controls the synchronization synthesis module 4 to match the frame number Image compensation is performed on the tracking image. In this embodiment, the processor 6 can fill in the number of missing frames in the tracking video file based on how many times each frame of image needs to be filled, or an average of several frames of images need to fill in one frame of image To perform image compensation. In this way, it has the effect of improving the smoothness of the video file when moving.

Please refer to Figure 2, the camera recording system of the present invention may further include a recording and playback module 7, the recording and playback module 7 is coupled to the processor 6, and used to record a computer screen operation screen, the computer screen The operation screen can be stored in the storage unit 3. In this embodiment, the computer screen operation screen may include the screen when operating a slide file. The recording and playback module 7 can be a general computer screen recording software tool. The processor 6 can control the synchronous synthesis module 4 to perform the image processing of the part covered by the tracking area A1, the audio signal, and the computer screen operation screen. Post-production to produce the video file. The processor 6 can control the recording and playback module 7 to generate and play a picture-in-picture using the video file. The picture-in-picture is used to display the tracking area A1 and the computer screen operation picture. In this embodiment, The default playback content of the main screen is the computer screen operation screen, and the default playback content of the sub-screen is the tracking area A1.

The processor 6 can establish a Neural Network (NN) model for identifying a writing area A3 from the tracking area A1. In this embodiment, the writing area A3 can be a whiteboard or A blackboard. Specifically, the fixed screen image F containing the writing area A3 is used as the input of the neural network model, and the neural network training, testing, and analysis are performed to generate the neural network model. In this way, the processor 6 can determine whether the writing area A3 is covered in the tracking area A1 through this type of neural network model. If the judgment result is no, no additional action is required; if the judgment result is yes, the The processor 6 further calculates whether the time for the writing area A3 to appear continuously in the tracking area A1 is longer than a time threshold. In this embodiment, the time threshold may be 2 seconds. If the calculation result is no, it is not necessary Perform additional actions; if the calculation result is yes, the processor 6 can set the area to be determined in the tracking area A1, and confirm whether the skin color ratio of the area to be determined is not greater than the skin color threshold. In this embodiment, The skin color threshold may be 40%. If the confirmation result is yes, the writing area A3 in the tracking area A1 is enlarged and close-up to generate an enlarged writing picture. The recording and playback module 7 presents the magnified writing screen with the parent screen in the picture-in-picture, and presents the computer screen operation screen with the sub-picture in the picture-in-picture; if the confirmation result is no, it uses the picture-in-picture The mother picture in the picture shows the computer screen operation picture, and the sub picture in the picture picture shows the tracking area A1.

Please refer to FIG. 2, the camera recording system of the present invention may further include a cloud server 8 for coupling to the processor 6 and at least one terminal device E. The processor 6 can upload the video file to the cloud server 8 in real time, so that the user of the terminal device E can watch the video file online, or the cloud server 8 can download and watch the video file.

In summary, the camera system of the present invention can use the camera unit to shoot toward the target area to generate the fixed screen image, and use the detection and tracking module to detect and track a target on the fixed screen image; When the detection and tracking module detects the existence of the target in the fixed screen image, the processor can generate a tracking area in the fixed screen image, and combine the part of the screen image covered by the tracking area with The audio signal is post-processed to generate the video file. In this way, the camera recording system of the present invention can track the target without rotating the camera unit, and has the effect of improving the smoothness of the image of the video file.

Although the present invention has been disclosed using the above-mentioned preferred embodiments, it is not intended to limit the present invention. Anyone who is familiar with this technique can perform various changes and modifications relative to the above-mentioned embodiments without departing from the spirit and scope of the present invention. The technical scope of the invention is protected. Therefore, the scope of protection of the invention shall be subject to the scope of the attached patent application.

1: Camera unit

2: Radio unit

3: storage unit

4: Synchronous synthesis module

5: Detection and tracking module

6: processor

7: Recording and playback module

8: Cloud server

E: terminal device

Claims (10)

A video recording system includes: a camera unit for shooting toward a target area to generate a fixed screen image; a radio unit for performing radio reception on the target area to generate an audio signal; a storage unit for To store the fixed screen image and the audio signal; a synchronous synthesis module to synthesize video signals and audio signals synchronously; a detection and tracking module to detect and track a target; and a processor , Coupled to the storage unit, the synchronization synthesis module and the detection and tracking module, the processor reads the fixed screen image from the storage unit, and controls the detection and tracking module to perform detection on the fixed screen image , To detect whether the target exists in the fixed screen image, and if the detection result is yes, a tracking area is generated in the fixed screen image, the tracking area has the target, and the processor uses the tracking The area continuously tracks the target, the processor sets a close-up area in the center of the tracking area, the processor sets a reference point for the target in the close-up area, and recognizes whether the target exceeds the close-up area If the identification result is yes, then the tracking area is displaced so that the target is located in the close-up area again. The calculation method of a horizontal displacement speed and a vertical displacement speed of the tracking area is shown in the following equation:

Among them, Speed _{K is} the displacement speed of the tracking area when the horizontal or vertical distance between the reference point and the close-up area is K, K=1,2,3,...,N, and N is the distance between the close-up area and the tracking area Speed _min is a minimum displacement speed of the tracking area; Speed _max is a maximum displacement speed of the tracking area. The processor controls the synchronous synthesis module in the fixed frame image. The screen image and the audio signal of the part covered by the tracking area are post-processed to generate a video file. For example, the recording system of claim 1, wherein the processor reads the fixed screen image from the storage unit, calculates and obtains the number of missing frames of the fixed screen image, and controls the synchronization synthesis module to match the frame number The video file is subjected to image compensation. For example, the recording system of claim 1, wherein the detection and tracking module uses a Gaussian mixture model to obtain a foreground object in the fixed frame image, and the detection and tracking module sets a region to be determined for the foreground object, And analyze whether the skin color ratio of the area to be determined is greater than a skin color threshold. If the analysis result is yes, it means that the target has been detected in the fixed image. Such as the recording system of claim 1, wherein the detection and tracking module uses a background-sensing correlation filter tracking algorithm, or a deep learning algorithm to track the target. For example, the recording system of claim 1, wherein the detection and tracking module does not detect the target in the tracking area, and the processor calculates the time when the detection and tracking module does not detect the target If it is longer than a detection time, the detection and tracking module is controlled to re-execute the detection of the target. For example, the recording system of any one of claim items 1 to 5 further includes a recording and playback module, the recording and playback module is coupled to the processor and used to record a computer screen operation screen, and the processor controls the synchronous synthesis The module post-processes the part of the screen image covered by the tracking area, the audio signal and the computer screen operation screen to generate the video file. For example, the recording system of claim 6, wherein the processor controls the recording and playback module to generate and play a picture-in-picture with the video file, and the picture-in-picture is used to display the tracking area and the computer screen operation picture respectively . For example, the recording system of claim 7, wherein the processor determines whether a writing area is covered in the tracking area by establishing a neural network model, and if the judgment result is yes, the processor calculates that the writing area is in the Whether the continuous occurrence time in the tracking area is longer than a time threshold, if the calculation result is yes, the processor sets a to-be-determined area in the tracking area, and confirms whether the skin color ratio of the to-be-determined area is not greater than a skin color threshold , If the confirmation result is yes, then The writing area in the tracking area is enlarged and close-up to generate an enlarged writing image, and the recording and playback module presents the enlarged writing image in the parent image of the mother-in-picture image and the sub-image in the mother-in-picture image The computer screen operation screen. For example, the recording system of claim 7, wherein the processor determines whether a writing area is covered in the tracking area by establishing a neural network model, and if the judgment result is yes, the processor calculates that the writing area is in the Whether the continuous occurrence time in the tracking area is longer than a time threshold, if the calculation result is yes, the processor sets a to-be-determined area in the tracking area, and confirms whether the skin color ratio of the to-be-determined area is not greater than a skin color threshold If the confirmation result is no, the computer screen operation screen is presented with the mother screen in the picture-in-picture, and the tracking area is presented with the sub-picture of the picture-in-picture. For example, the recording system of any one of request items 1 to 5 further includes a cloud server for coupling the processor and at least one terminal device, and the processor uploads the video file to The cloud server is provided for users of the terminal device to watch the video file online.

Images