TWI867867B - Panoramic video generation system and method - Google Patents

Abstract

The present invention provides a panoramic video generation system, comprising: a first module, used to obtain first time data of multiple user devices, and provide second time data to integrate the time of multiple user devices into a consistent one, wherein the user devices have a camera unit; a second module, by being associated with a predetermined image or a first image provided by the user device, generating a first data, wherein the first data is a plurality of feature points associated with the target; a third module, based on the location information of the multiple user devices Generate a second data, the second data is related to the position of the user device relative to the target; the fourth module generates an operation data according to the first data and the second data, and provides the operation data to each of the user devices respectively, thereby generating a panoramic video according to the signal sources photographed by multiple user devices, wherein the operation data is an indication related to the focal length and direction of the camera unit for the target. According to the concept of the present invention, a real-time streaming panoramic video generation technology can be realized.

Description

Panoramic video generation system and method

A panoramic video generation system and method, in particular, refers to stitching images using streaming images taken by multiple user devices (such as mobile phones) to generate a panoramic video.

Panoramic video is a video format that can provide viewers with a full range of visual experience. Panoramic videos usually capture a 360-degree horizontal and 180-degree vertical field of view, allowing viewers to freely rotate the viewing angle while playing the video, as if they were in the scene. This video format is often used in virtual reality (VR) and augmented reality (AR) applications to provide a more immersive and interactive experience.

The production of panoramic videos usually involves special photography and editing techniques to capture the visual information of the entire surrounding environment. Photographers usually need to use multiple cameras or 360-degree cameras to capture panoramic videos, and then use professional software to merge these videos into a single panoramic video. When watching panoramic videos, viewers can use tools such as mice, handheld devices, or virtual reality helmets to explore the scenes in the video by changing their perspectives, creating a sense of being there.

The process of making panoramic videos certainly enriches the audience's viewing experience. However, this production method also has a series of disadvantages and challenges, such as (1) high cost: the production of multi-view videos requires the use of multiple cameras, which increases the cost of hardware and equipment. Each camera needs to be configured independently, and the purchase, maintenance and operation of these cameras require expensive budgets. In addition, the post-production of multi-view videos also requires professional software and human resources, which further increases the cost. (2) Complex shooting conditions: Each camera needs to have good shooting conditions, including light, position and angle. The photographer must spend extra time and energy to ensure that each camera is in the best shooting condition, which may increase the difficulty and uncertainty of shooting. (3) Complex camera configuration: Configuring multiple cameras requires careful planning, and the photographer must ensure that they will not interfere with each other and can capture the required images. This may require additional equipment such as brackets and stabilizers to ensure the stability and accuracy of the camera. (4) Long post-production time: During the post-production process, a lot of time is required to edit and synthesize the materials shot by different cameras. This not only increases the time cost of post-production, but may also cause delays in production progress. (5) Professional technology requirements: Post-production of multi-view videos requires professional technology, such as video editing and video synthesis. This may require hiring professionals with relevant skills, or investing time to learn these technologies.

In summary, although the production of panoramic videos can provide a richer viewing experience, it also faces high costs, technical challenges and time costs, as well as challenges and shortcomings in terms of timing differences, system stability, hardware complexity, etc. Therefore, a new type of panoramic video generation technology that can be streamed in real time is currently needed to make this form of video easier to produce.

To achieve the above-mentioned purpose, the present invention provides a panoramic video generation system, comprising: a first module, used to obtain first time data of multiple user devices, and provide a second time data to integrate the time of the multiple user devices into a consistent one, wherein the user device has a camera unit, the first time data is the system time of the user device, and the second time data is the system time of the system; a second module, by being associated with a predetermined image or a first image provided by the user device, generating a first data, the first data is a plurality of related A third module generates a second data based on the location information of the plurality of user devices, the second data being related to the location of the user device relative to the target; a fourth module generates an operation data based on the first data and the second data, and provides the operation data to each of the user devices, thereby generating a panoramic video based on the signal sources captured by the plurality of user devices, wherein the operation data is an indication related to the focal length and direction of the camera unit with respect to the target.

In one embodiment, the first module includes a registration unit that uses Zenoh Auto Discovery technology to enable registration of multiple user devices to the system.

In one embodiment, the first module further includes a time acquisition unit for acquiring a plurality of the first time data.

In one embodiment, the first module further includes a time adjustment unit, which adjusts the time in an application in the user device according to the second time data, thereby determining that the timestamps of multiple user devices are consistent.

In one embodiment, the second module uses Visual SLAM technology to calculate the first data.

In one embodiment, the third module generates the second data using Visual SLAM technology and Zenoh Protocol Peer to Peer technology.

In one embodiment, a fifth module is further included for removing the signal source captured by at least two of the plurality of user devices.

To achieve the above-mentioned purpose, the present invention provides a method for generating a panoramic video, comprising: a time calibration step, obtaining first time data of a plurality of user devices, and providing a second time data to integrate the times of the plurality of user devices into a consistent one, wherein the user device has a camera unit, the first time data is the system time of the user device, and the second time data is the system time of the system; a feature point generation step, generating a first data by associating with a predetermined image or a first image provided by the user device, wherein the first data is a plurality of feature points associated with the target. A feature point, wherein the predetermined image is an image of the target that has been photographed; a relative position confirmation step, generating a second data based on the position information of multiple user devices, wherein the second data is related to the position of the user device relative to the target; a panoramic video shooting step, generating an operation data according to the first data and the second data, and providing the operation data to each user device respectively, wherein the operation data is an indication related to the focal length and direction of the camera unit for the target; a panoramic video generation step, generating a panoramic video according to the signal source shot by multiple user devices.

In one embodiment, the time calibration step further includes a registration step, using Zenoh Auto Discovery technology to implement registration from multiple user devices. In one embodiment, the first module further includes a time acquisition unit for obtaining multiple first time data; a time acquisition step for obtaining multiple first time data; and a time adjustment step for adjusting the time in an application in the user device according to the second time data, thereby determining that the timestamps of multiple user devices are consistent, wherein the second time data includes time data provided by a system time service or a network time service.

In one embodiment, the feature point generation step is implemented using Visual SLAM technology.

In one embodiment, the relative position confirmation step is implemented using Visual SLAM technology and Zenoh Protocol Peer to Peer technology.

In one embodiment, an optimization step is further included between the panoramic video shooting step and the panoramic video generation step, which is used to remove the same signal source shot by at least two of the multiple user devices.

100: System

110: First module

111:Registered unit

112: Time acquisition unit

113: Time adjustment unit

120: Second module

130: The third module

140: Fourth module

150: Fifth module

300: User device

S31: Time calibration step

S311: Registration steps

S312: Time acquisition step

S313: Time adjustment step

S32: Feature point generation step

S33: Relative position confirmation step

S34: Panoramic video shooting steps

S35: Optimization step

S36: Panoramic video generation step

Figure 1 shows the system architecture diagram of the panoramic video generation of the present invention.

Figure 2 shows a flow chart of the panoramic video generation method of the present invention.

Figure 3 shows a flow chart of the time calibration steps of the present invention.

Figure 4 shows a schematic diagram of the Zenoh Protocol Peer to Peer communication technology of the present invention.

Please refer to the system architecture diagram of the panoramic video generation of the present invention shown in Figure 1 to briefly explain the functions of each module. The system 100 of the present invention can generate panoramic videos based on multiple user devices 300. The user device 300 has a camera unit to shoot an image of a target, wherein the user device 300 can be, for example, a mobile phone, tablet or other mobile device with a recording function, but is not limited thereto. The target is defined as the target that the multiple user devices 300 want to shoot, such as a concert venue, a stadium, etc.

The system 100 includes a first module 110, a second module 120, a third module 130, a fourth module 140 and a fifth module 150. The first module 110 is configured to obtain first time data of a plurality of user devices 300 and provide a second time data to integrate the times of the plurality of user devices 300 into a consistent state. Furthermore, the first module 110 further includes a registration unit 111, a time acquisition unit 112 and a time adjustment unit 113. The registration unit 111 is configured to utilize the Zenoh Auto Discovery technology to implement registration of the plurality of user devices to the system. The time acquisition unit 112 is configured to obtain a plurality of first time data from the user device 300, wherein the first time data is defined as the system time of the user device 300. The time adjustment unit 113 is configured to adjust the time in an application in the user device according to the second time data, thereby determining that the timestamps of the plurality of user devices are consistent, thereby determining that the signal sources captured by the plurality of user devices all have the same time, so that different user devices 300 can capture videos of different angles at the same time, thereby allowing the system 100 to perform video stitching, wherein the second time data is defined as the system time of the system 100, such as time data provided by a system time service or a network time service.

In the system 100 of the present invention, the second module 120 is configured to generate a first data by one of a predetermined image associated with a target or a first image provided by the user device 300, wherein the first data is a plurality of feature points associated with the target. The third module 130 is configured to generate a second data based on the position information of the plurality of user devices 300, wherein the second data is associated with the position of the user device relative to the target. The fourth module 140 is configured to generate an operation data according to the first data and the second data, and provide the operation data to each of the user devices respectively, thereby generating a panoramic video according to the signal source captured by the plurality of user devices, wherein the operation data is an indication of the focal length and direction of the camera unit with respect to the target. The fifth module 260 is configured to remove the signal source captured by at least two of the plurality of user devices 300.

The following is a detailed description of the technical means of generating panoramic videos of the present invention. Please refer to the flowchart of the panoramic video generating method of the present invention shown in FIG2 and the flowchart of the time calibration step of the present invention shown in FIG3 and refer to FIG1 at the same time. Since the system time of the user device 300 may be inaccurate due to various reasons, such as the user may not manually adjust the time or due to changes in different time zones. In order to ensure that the time on all user devices 300 is consistent, the system time service of the user device 300 or the time data from the network time service (such as NTP, Network Time Protocol) can be used to ensure that the time of all user devices 300 is consistent, so as to facilitate the subsequent video stitching. Therefore, in the time calibration step S31, the present invention obtains first time data of multiple user devices through the first module 110, and provides a second time data to integrate the times of the multiple user devices into a consistent state. More specifically, the time calibration step S31 includes a registration step S311, a time acquisition step S312, and a time adjustment step S313. In the registration step S311, the registration unit 111 uses Zenoh (Zero Overhead Pub/Sub, Store/Forward, and Query) technology to enable multiple user devices 300 to automatically register their devices, thereby establishing a communication connection with the system 100. The Zenoh is an open source protocol and tool kit for building a real-time data distribution system, which provides automatic discovery and communication capabilities and is used to establish a data channel to share data between different devices. For example, it can enable applications in different user devices 300 to establish a communication connection to ensure that they can communicate with each other and share data. In a specific embodiment, Zenoh includes zenoh client and zenoh Auto Discovery, wherein the zenoh client is mainly used for communication between applications and the zenoh distribution system to realize functions such as data distribution, storage and query, and the zenoh Auto discovery is helpful to build a dynamic and adaptive zenoh system so that nodes can automatically join or leave the system without manual configuration or management. In a preferred embodiment of the present invention, the registration unit 111 further utilizes WASM technology to execute the application (i.e., Zenoh) in a web browser, thereby avoiding the review of the App Store or Google Play Store. Specifically, Zenoh is compiled into a WebAssembly module (the WebAssembly module is a binary file that contains low-level code that can be run in a browser), and the generated module is integrated using the corresponding WebAssembly API to load and run the WebAssembly module, wherein the WebAssembly module can at least support Rust or C to compile Zenoh, but is not limited thereto. In the time acquisition step S312, the time acquisition unit 112 of the present invention obtains the first time data of each user device 300 through the application in the user device 300 (for example, from the system time service or the network time service), thereby confirming which user devices 300 have first time data that are inconsistent with the second time data of the system. Next, in the time adjustment step S313, the time adjustment unit 113 adjusts the time in an application in the user device 300 according to the second time data, thereby determining that the timestamps of multiple user devices 300 are consistent. Through the time calibration step S31, it can be ensured that the timestamps used when stitching the video are consistent, thereby overcoming the problem that the video cannot be stitched due to different times of different user devices 300.

As shown in the feature point generation step S32 of FIG. 2 , the second module 120 generates a first data by using one of a predetermined image associated with a target or a first image provided by the user device 300, wherein the first data is a plurality of feature points associated with the target, and the predetermined image is an image of the target that has been photographed. Specifically, the second module 120 calculates the first data using a Visual Simultaneous Localization and Mapping (VSLM) algorithm, wherein VSLM is a technology for simultaneous positioning and map construction, which can determine the position of the user device 300 and the target in the environment by using a first image captured by a camera unit or sensor of the user device 300. In the preferred embodiment of the present invention, in order to improve the accuracy of the Visual SLAM algorithm, a first data of a feature point map is usually created, and the feature points in the feature point map can be used for subsequent position estimation, wherein the feature points can be landmarks, signs or other easily identifiable points. In the preferred embodiment of the present invention, the second module 120 can calculate the feature point map first by storing multiple images associated with a target such as a database, thereby accelerating the positioning calculation of the user device 300 relative to multiple feature points.

In the relative position confirmation step S33, the third module 130 generates a second data based on the position information of the plurality of user devices, and the second data is related to the position of the user device relative to the target. Specifically, the third module 130 is based on the Zenoh Protocol Peer to Peer (Zenoh P2P) technology combined with Visual SLAM to provide (1) data transmission: Zenoh P2P allows real-time transmission of data between different user devices 300, such as the aforementioned first data. (2) Real-time positioning calculation: Zenoh P2P transmission technology can more quickly and accurately calculate the second data, which is defined as the real-time position of the user device 300 relative to the target. In the present invention, the main function of Zenoh P2P technology is to support data communication and data sharing in a distributed system. Zenoh P2P technology connects various user devices 300 through a network, allowing them to exchange data with each other and be used in combination with Visual SLAM. As shown in the Zenoh Protocol Peer to Peer schematic diagram in FIG4 , Zenoh P2P can share relative position information between different user devices 300 or viewers, thereby determining the different positions of the camera units or sensors of multiple user devices 300, so as to more accurately locate the positions of multiple camera units in three-dimensional space. In addition, Zenoh P2P can also be used to coordinate data sharing between different user devices 300. For example, the camera unit of each user device 300 can transmit image data or sensor measurement data in real time and share these data with the Visual SLAM algorithm so that these data can be effectively transmitted to where they are needed.

Next, in the panoramic video shooting step S34, the fourth module 140 generates an operation data according to the first data and the second data, and provides the operation data to each of the user devices 300, wherein the operation data is an indication related to the focal length and direction of the camera unit with respect to the target. More specifically, the fourth module 140 determines the relative position of each user device 300 in the feature point map according to the first data and the second data, thereby calculating the operation data about the orientation, angle and focal length of the panoramic video shot by each user device 300, and then provides the operation data to the application of the user device 300 to prompt the user how to shoot a panoramic video and generate a plurality of signal sources related to the image. For example, the application in the user device 300 is provided with a user interface. When multiple users open the application in their respective user devices 300, the system 100 of the present invention calculates the operation data and provides it to the application in the user device 300. These users can operate the angle, orientation and focal length of the user device 300 according to the prompts of the user interface to shoot the signal source related to the video. In a preferred embodiment of the present invention, the generation of the operation data is real-time, and the application can be set to be automatic to perform feature point detection and tracking in real time. The user device can automatically adjust the camera unit angle, orientation and focal length based on the real-time operation data, such as adjusting camera parameters, cropping, zooming or other image processing operations, but not limited thereto. In other embodiments of the present invention, if cluster processing is required for multiple videos of related focal lengths, Zenoh P2P can be used to coordinate data sharing and collaboration between processing nodes of different user devices 300, so that each processing node can cooperate with each other to jointly process videos of different focal lengths, thereby optimizing processing performance.

In the panoramic video generation step S36, the fourth module 140 then generates a panoramic video according to the signal sources captured by the plurality of user devices in the panoramic video shooting step S36. In a preferred embodiment of the present invention, the fourth module 140 may be a video streaming server (Video Streaming Server) and has a real-time video stitching function. The fourth module 140 may stitch together the signal sources with the same timestamp in real time through the signal sources captured by the plurality of user devices to generate a panoramic video in real time. In a preferred embodiment of the present invention, the fourth module is configured in an edge computing framework, such as Fog05, to reduce the amount of computing performed in a centralized remote location (such as the "cloud"), thereby minimizing the amount of communication that must occur between remote clients and servers.

In other embodiments of the present invention, an optimization step S35 is further included between the panoramic video shooting step S34 and the panoramic video generation step S36, and the fifth module 150 is configured to remove the same signal source shot by at least two of the multiple user devices. Specifically, since the multiple user devices 300 may have substantially the same angle and orientation, they may have a chance to shoot the same video. Therefore, after the panoramic video shooting step S34, the fifth module 150 of the present invention receives multiple signal sources shot by the user devices 300, and filters the signal sources of duplicate videos through AI or algorithms to optimize the network bandwidth and computing performance, thereby reducing the amount of computing for generating panoramic videos to accelerate the generation of panoramic videos. In a preferred embodiment of the present invention, the fifth module 150 can be configured on a cloud computing server such as GCP (Google Cloud Platform) and AWS (Amazon Web Services), wherein GCP or AWS is a cloud computing platform. The cloud computing server provides a variety of cloud services including computing, storage, database, artificial intelligence, machine learning and network services. GCP also has global data centers to ensure high availability and low latency. Developers and enterprises can use GCP to build and run applications.

In summary, the system and method of the present invention combines Visual SLAM technology with Zenoh Protocol Peer to Peer communication technology to accurately locate the positions of multiple user devices relative to the target to capture multiple images, and uses an edge computing platform to accelerate the generation of real-time panoramic videos, thereby optimizing video quality and viewing experience.

100: System

110: First module

111:Registered unit

112: Time acquisition unit

113: Time adjustment unit

120: Second module

130: The third module

140: Fourth module

150: Fifth module

300: User device

Claims (13)

A panoramic video generation system includes: a first module for obtaining first time data of a plurality of user devices and providing a second time data to integrate the times of the plurality of user devices into a consistent state, wherein the user device has a camera unit, the first time data is the system time of the user device, and the second time data is the system time of the system, wherein the user device has an application; a second module for generating a first data by being associated with a predetermined image or a first image provided by the user device, wherein the first data is a plurality of specific information associated with the target. A feature point, the predetermined image is an image of the target that has been photographed; a third module, based on the location information of multiple user devices, generates a second data, the second data is related to the position of the user device relative to the target; a fourth module, based on the first data and the second data, generates an operation data, and provides the operation data to each application of the user device, so as to allow a user to generate a panoramic video based on the signal source shot by operating the application according to the operation data, wherein the operation data is an indication related to the focal length and direction of the camera unit with respect to the target. The panoramic video generation system as described in claim 1, wherein the first module includes a registration unit, using Zenoh Auto Discovery technology to implement registration from multiple user devices to the system. A panoramic video generation system as described in claim 2, wherein the first module further comprises a time acquisition unit for acquiring a plurality of the first time data. The panoramic video generation system as described in claim 3, wherein the first module further comprises a time adjustment unit, which adjusts the time in an application in the user device according to the second time data, thereby determining that the timestamps of multiple user devices are consistent. The panoramic video generation system as described in claim 1, wherein the second module uses Visual SLAM technology to calculate the first data. The panoramic video generation system as described in claim 1, wherein the third module generates the second data using Visual SLAM technology and Zenoh Protocol Peer to Peer technology. The panoramic video generation system as described in claim 1 further includes a fifth module for removing the signal source captured by at least two of the plurality of user devices. A method for generating a panoramic video includes: a time calibration step, obtaining first time data of a plurality of user devices, and providing a second time data to integrate the times of the plurality of user devices into a consistent time, wherein the user device has a camera unit, the first time data is the system time of the user device, and the second time data is the system time of the system, wherein the user device has an application; a feature point generation step, generating a first data by associating with a predetermined image or a first image provided by the user device, wherein the first data is a plurality of feature points associated with the target, wherein the predetermined image is a photographed image of the target. an image of the target; a relative position confirmation step, generating a second data based on the position information of the plurality of user devices, the second data being related to the position of the user device relative to the target; a panoramic video shooting step, generating an operation data based on the first data and the second data, and providing the operation data to the application of each user device respectively, thereby allowing a user to operate the application based on the operation data to shoot a signal source, wherein the operation data is an indication related to the focal length and direction of the camera unit for the target; a panoramic video generation step, generating a panoramic video based on the signal source shot by the plurality of user devices. The method for generating panoramic videos as described in claim 8, wherein the time calibration step further includes a registration step, using Zenoh Auto Discovery technology to implement registration from multiple user devices. The method for generating a panoramic video as described in claim 9, wherein the time calibration step further includes a time acquisition step for acquiring a plurality of the first time data; and a time adjustment step for adjusting the time in the application of the user device according to the second time data, thereby determining that the timestamps of the plurality of the user devices are consistent, wherein the second time data includes time data provided by a system time service or a network time service. The panoramic video generation method as described in claim 8, wherein the feature point generation step is implemented using Visual SLAM technology. The method for generating panoramic video as described in claim 8, wherein the relative position confirmation step is implemented by using Visual SLAM technology and Zenoh Protocol Peer to Peer technology. The panoramic video generation method as described in claim 8, wherein the panoramic video shooting step and the panoramic video generation step further include an optimization step for removing the same signal source captured by at least two of the multiple user devices.