KR102801836B1 - Method and apparatus for automatically generating enterprise-related videos that reflects references and recommends expression - Google Patents

Abstract

The operating method of the automatic video generation device may include: a step of receiving corporate content; a step of extracting corporate data from the corporate content; a step of receiving a user query and extracting query data; a step of extracting past behavioral data of the user; and a step of generating reference video information based on the query data and the past behavioral data using a video generation model learned with the corporate data as input, and transmitting the generated reference video information to a user device.

Description

METHOD AND APPARATUS FOR AUTOMATICALLY GENERATING ENTERPRISE-RELATED VIDEOS THAT REFLECTS REFERENCES AND RECOMMENDS EXPRESSION

According to various embodiments of the present disclosure, a method and device for automatically generating a corporate video based on corporate information are disclosed. Furthermore, according to various embodiments of the present disclosure, a method and device for generating a corporate video by reflecting a reference provided by a user in video generation and recommending an expression method are disclosed.

In modern society, the way information is consumed is changing due to the spread of smart devices. In particular, as the attention span of target consumers and employees of companies gradually decreases, preference for long text content or long-running video content is decreasing significantly, and preference for short-form video content with short run times is increasing.

In this way, as the way individuals consume information changes, changes are also required in the business activities of companies. In a rapidly changing market/economy/regulation, companies are required to produce and distribute core videos that can effectively communicate or inform individuals of information in order to quickly communicate and make decisions with individuals such as consumers or workers.

In modern society, companies have a vast amount of corporate information, so in rapidly changing market conditions, an artificial intelligence (AI) system that effectively reflects corporate information and automatically generates images that match the characteristics of the company's business is required.

The above information may be provided as related art for the purpose of assisting in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

Republic of Korea Patent Publication No. 10-2560609 (published on July 24, 2023)

AI-based video generation technology has recently made groundbreaking progress, so that if you send a request in natural language without any special format, the video generation foundation model can generate a video by reflecting the request.

However, in practice, rather than directly applying the video created with the foundation model to the business, additional review, editing, and customization processes are often required. This is because the foundation model lacks reflection of business-specific context or expertise, or is difficult to fine-tune to specific business requirements. For example, when creating an advertising video for a specific company, the advertising video must reflect the company's unique brand characteristics, and the company's desired tone, manner, and visual style must be applied, but these cannot all be specified in the foundation model.

Especially in corporate videos, specialized message delivery tailored to each business domain (e.g., insurance, finance, medical, IT, etc.) is required. However, producing unique content that reflects the characteristics of each business domain requires specialized knowledge and skills, and there are limits to doing this manually.

In addition, modern companies generate and collect various types of data, and these data can exist in various forms such as text, images, and videos. However, comprehensively processing and analyzing these various data is technically complex, and can be an even more difficult problem in the process of converting them into business intelligence and business domain-specific video content.

Also, for modern businesses, deciding which presentation method to use in the video content creation process can be a difficult problem. In particular, choosing the appropriate presentation method can be important in order to convey the right message for business intelligence or business domain.

According to various embodiments of the present disclosure, an object is to provide a method and device for automatically generating a corporate video based on corporate information, which supports a corporate to more effectively convey its expertise and uniqueness by reflecting references, recommending expression methods, and automatically generating video content optimized for corporate needs.

According to various embodiments of the present disclosure, an object is to provide a method and device for automatically generating a corporate video based on corporate information, which efficiently generates video content suitable for a corporate entity by simultaneously analyzing and utilizing various forms of data through a multimodality search function.

According to various embodiments of the present disclosure, when a user produces corporate video content based on corporate information, an optimal video expression method that matches the characteristics and goals of the company is recommended, thereby contributing to shortening the video content production time while matching the intended production purpose of the video, thereby providing a method and device for automatically producing a corporate video based on corporate information.

However, the problem to be solved in the present disclosure is not limited to the problem mentioned above, and may be expanded in various ways without departing from the spirit and scope of the present disclosure.

According to various embodiments of the present disclosure, a method of operating an automatic video generation device may include: receiving corporate information; extracting corporate data from the corporate information; receiving a user query and extracting query data; extracting past behavioral data of the user; and generating reference video information based on the query data and the past behavioral data by using a video generation model learned with the corporate data as input, and transmitting the generated reference video information to a user device.

The above reference video information may include information about a video template and information about a video content style.

The above reference video information may further include information about sample video content.

If the similarity between the above query data and the above past behavior data is greater than a specified threshold value, the above reference video information may include reference video information included in the above past behavior data.

The operating method of the above video automatic generation device may include a step of generating a prompt based on the query data; a step of generating a context based on reference video information selected by a user; and a step of generating a video script based on the prompt, the query data, and the context.

The operating method of the automatic video generation device may further include a step of generating video content based on the video script; a step of transmitting the video content to a user device; and a step of receiving completion information or modification information input by the user.

The operating method of the automatic video generation device may further include, when receiving modification information by the user, a step of modifying at least one of object information and background information of the video content based on the modification information to reproduce the video content and transmitting the reproduced video content to the user device.

A video automatic generation device comprises at least one processor; and a memory storing instructions, wherein the instructions, when individually or collectively executed by the at least one processor, cause the processor to: receive business information, extract business data from the business information, receive a user query and extract query data, extract past behavior data of the user, and generate reference video information based on the query data and the past behavior data by using a video generation model learned with the business data as input, and transmit the generated reference video information to a user device.

A computer program may be stored in a computer-readable recording medium to perform the operating method of the automatic video generation device according to any one of claims 1 to 7, in combination with hardware.

The video automatic generation system comprises a user device for inputting a user query; and a video automatic generation device, wherein the video automatic generation device extracts corporate data from the corporate information, receives a user query and extracts query data, extracts past behavior data of the user, and generates reference video information based on the query data and the past behavior data using a video generation model learned with the corporate data as input, and transmits the generated reference video information to the user device.

According to various embodiments of the present disclosure, customized video content that reflects references based on the business intelligence data of the company and matches the strategic goals of the company can be automatically generated, so that the company can deliver messages quickly and consistently and communicate more effectively. In addition, this can increase the marketing efficiency of the company and provide video content that can leave a strong impression on target consumers.

According to various embodiments of the present disclosure, by recommending an expression method optimized for the business domain of the company, the company's expertise and reliability can be improved, and accordingly, the company can easily produce customized content reflecting the characteristics of the business domain and strengthen its position in the market with competitive content.

According to various embodiments of the present disclosure, by supporting a multimodality search function capable of comprehensively analyzing and processing various forms of data such as text, images, audio, or video, a large amount of data can be efficiently utilized, and by generating image content reflecting various aspects of the data, the accuracy and comprehensiveness of information transmission can be maximized.

According to various embodiments of the present disclosure, when a user creates a corporate video content, the efficiency of the content creation process can be improved by recommending an expression method optimized for the corporate business intelligence and business domain. Accordingly, the present disclosure can help the user create optimal video content with less time and effort, and can enable the corporate marketing and communication strategy to be carried out more effectively.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art to which the present disclosure belongs from the description below.

FIG. 1 is a conceptual diagram illustrating a device for automatically generating a corporate-related video based on corporate information according to various embodiments of the present disclosure.
FIG. 2 is a flowchart illustrating a method for automatically generating a corporate-related video based on corporate information according to various embodiments of the present disclosure.
FIG. 3 is a block diagram illustrating configurations of a video generation device according to various embodiments of the present disclosure.
FIG. 4 is a block diagram illustrating configurations of a video generation device according to various embodiments of the present disclosure.
FIG. 5 is a flowchart illustrating a method of generating a content block and a caption for the content block in a service server according to various embodiments of the present disclosure, and extracting a candidate content block for a user query using the caption.
FIGS. 7 to 9 are flowcharts for explaining an operation method of an automatic video generation device according to various embodiments of the present disclosure.
FIG. 12 is a block diagram illustrating a user device according to various embodiments of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components. In addition, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

It should be understood that the various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly dictates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., a memory) readable by a machine or device. For example, a processor of the machine or device may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities, and some of the multiple entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, the multiple components (e.g., a module or a program) may be integrated into one component. In such a case, the integrated component may perform one or more functions of each of the multiple components identically or similarly to those performed by the corresponding component of the multiple components before the integration. According to various embodiments, the operations performed by the module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

The processor in the present disclosure may mean hardware capable of performing functions and operations according to each name described in the present specification, may mean computer program code capable of performing specific functions and operations, or may mean an electronic recording medium loaded with computer program code capable of performing specific functions and operations. According to an embodiment, the operation of the processor may be defined and/or interpreted as the operation of a video generation device, but is not limited thereto. The processor may mean a functional and/or structural combination of hardware for performing the technical idea of the present disclosure and/or software for driving the hardware.

The large language model (LLM) (hereinafter referred to as 'LLM') in the present disclosure may be a language model capable of performing natural language processing (NLP) tasks.

In the present disclosure, the foundation model or video generation model may be a large-scale model that generates a video using artificial intelligence technology. The foundation model may generate a video based on text descriptions, images, or other inputs.

The video automatic generation device in the present disclosure may be defined and/or referred to as a service server, which may mean one physically independent server, but is not limited thereto, and may also be one virtual machine, and may be configured to encompass one module or program or docker operating on one virtual or physical machine.

FIG. 1 is a conceptual diagram illustrating a device for automatically generating a corporate-related video based on corporate information according to various embodiments of the present disclosure.

The embodiment of Fig. 1 can be combined with the embodiments of Figs. 2 to 11.

Hereinafter, a device (100) for automatically generating a corporate-related video based on corporate information may be referred to as an automatic video generation device (100).

Referring to FIG. 1, the video automatic generation device (100) may include a processor (110) and a memory (120).

Although not shown, the video automatic generation device (100) may further include a communication module for being communicatively connected to a user device (200) or a database (300) via wireless or wired communication. The communication module may include communication circuitry.

The automatic video generation device (100) may be implemented as a service server, but is not limited thereto. For example, the automatic video generation device (100) may be provided as an independent service server or an independent physical device connected to a separate service server including a video generation model. Hereinafter, a case in which the automatic video generation device (100) is provided as a service server including a video generation model (111) will be described as an example. According to an embodiment, the automatic video generation device (100) and the user device (200) may be interpreted as being included in the automatic video generation system.

The video automatic generation device (100) can collect enterprise information from a database (300) containing enterprise information. For example, the enterprise information can include, but is not limited to, web pages, documents (e.g., PDF, etc.), images, videos, or audio provided by a user.

Corporate information may include information about the company's brand identity (BI) (hereinafter referred to as 'BI') or information about the company's business domain.

For example, the BI information of a company may include at least one of, but is not limited to, data on a logo symbolizing the company, a unique color combination associated with the brand, preset fonts and typefaces, design elements including graphics, icons, and patterns, and text data on brand personality, core values, mission and vision, positioning, brand story, and communication style. For example, the BI information of a company may include information on a tone and manner set by the company, information on the company's storytelling, information on the company's products and services, and data on brand messages and concepts used in marketing channels.

For example, information about a company's business domain may include, but is not limited to, information about the industry sector to which the company belongs. For example, information about a company's business domain may include data about the regulatory situation (e.g., legal regulations) of the business domain to which the company belongs, data about market trends, data about technical requirements, or data about unique characteristics of the business domain.

The video automatic generation device (100) can collect corporate information including corporate BI information or corporate business domain information from a database (300), and can collect this information at preset intervals or whenever new information (new data) is generated.

The video generation model (111) of the video automatic generation device (100) can implement an artificial intelligence-based learning model by performing learning through corporate BI data (e.g., vector data) or corporate business domain data (e.g., vector data) obtained by preprocessing collected corporate information.

The processor (110) may include a video generation model (111). The processor (110) may generate a business-related video based on a user query input from a user device (200) using the video generation model (111).

The processor (110) may include, but is not limited to, at least one of a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array), for example.

The video generation model (111) may be implemented as part of the processor (110), but is not limited thereto. For example, the video generation model (111), or at least one instruction causing the operation of the processor (110) and/or the automatic video generation device (100), may be stored in the memory (120).

The memory (120) can store instructions (or programs) executable by the processor (110). The memory (120) can include volatile memory or non-volatile memory.

Volatile memory can be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM).

Nonvolatile memory can be implemented as Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, Magnetic RAM (MRAM), Spin-Transfer Torque (STT)-MRAM, Conductive Bridging RAM (CBRAM), Ferroelectric RAM (FeRAM), Phase change RAM (PRAM), Resistive RAM (RRAM), Nanotube RRAM, Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, Molecular Electronic Memory Device, or Insulator Resistance Change Memory.

The user device (200) may be defined as a device used by a user (e.g., a corporate manager or corporate worker), and the user may be communicatively connected to the video generation device (200) through the user device (200). The user device (200) may include various devices such as a smart device (e.g., a smart phone), a personal computer (PC), or a portable laptop computer, which may include an application (210) (or program).

A user can input a user query (e.g., information about video creation, etc.) into a video creation device (100) through an application (210) built into a user device (200) and receive a company-related video created from the video creation device (100).

FIG. 2 is a flowchart illustrating a method for automatically generating a corporate-related video based on corporate information according to various embodiments of the present disclosure.

The embodiment of Fig. 2 can be combined with the embodiment of Fig. 1, or the embodiments of Figs. 3 to 11.

The method for automatically generating a corporate-related video (hereinafter referred to as the “automatic video generation method”) described with reference to FIG. 2 may be defined as a method performed by the video generation model (111), processor (120), or video generation device (100) of FIG. 1, but is not limited thereto.

The method for automatically generating a video may include a step (S110) of receiving corporate content.

The corporate content may include corporate information including corporate BI information or corporate business domain information. For example, the video generation model (111) may collect corporate web pages, corporate documents (e.g., PDF, etc.), corporate-related images, corporate-related videos, or corporate-related audio from the database (300).

The corporate content or corporate information may be defined as various data sources provided by the company, but is not limited thereto. For example, the corporate content or corporate information may be defined as data directly collected by the administrator of the automatic video generation device (100) with permission from the company's users. According to an embodiment, the corporate content or corporate information may include information related to the company's users (for example, including information on similar companies) among data randomly extracted from the Internet by the automatic video generation device (100).

The method for automatically generating a video may include a step of embedding corporate content (S120).

The step of embedding corporate content (S120) may include a step of classifying the collected corporate content or corporate information by format. For example, the step of embedding (S120) may analyze the format of the collected corporate content and classify it into various multi-modalities such as web page modality, text modality, image modality, audio modality, or video modality.

Step (S120) of embedding corporate content can vectorize and process the various types of multi-modality using the VLM (visual-language model) of the video generation model (111). That is, each corporate content can be converted into vector representations of information included in the classified modality. The vector representations can be defined and/or referred to as corporate embedding data for which embedding is complete.

Enterprise embedding data may include vectors containing enterprise BI information (e.g., enterprise BI data) that include vectors for enterprise BI information included in the modality, or vectors containing enterprise business domain information (e.g., enterprise business domain data).

In one embodiment, the step of embedding corporate content (S120) may include a step of converting various formats of multi-modality into vector representations by vectorizing them, and a step of inserting or recording caption information into content blocks of various formats of multi-modality (e.g., a specific part or unit of each content).

The unit content block of each modality may include caption information for each of the various formats of multi-modality. The caption information may be text information including, but not limited to, representative image information, title information, or summary information of the corporate content included in the content block as metadata.

The method for automatically generating a video may include a step (S130) of storing corporate embedding data in a DB.

The step (S130) of storing corporate embedding data in a DB may include a step of adjusting each vector information to be included in the same vector space by reflecting the characteristics of the vector-processed modality information through VLM. For example, each vector expression of different formats of multi-modality such as web pages, text, images, audio, or video may be adjusted and integrated into a single high-dimensional vector (hereinafter referred to as an 'integrated vector').

The step (S130) of storing corporate embedding data in a DB may include a step of storing data integrated into a single high-dimensional vector in a memory (130) or an external server.

The step (S130) of storing corporate embedding data in a DB may include a step of storing modality information including caption information by being caption-blocked in a memory (130) or an external server.

In the step (S130) of storing corporate embedding data in a DB, or in other steps, a video generation model (Foundation model) or an LLM (Large-Scale Language Model) may be trained through the corporate embedding data and trained to generate and output a corporate-related video according to a user's request. For example, the corporate embedding data may be information or data generated by performing a multi-modal analysis using at least one or at least two or more of text data, image data, audio data, and video data extracted based on corporate information.

Although not illustrated in FIG. 2, the method for automatically generating a video may include a step of extracting corporate business domain information from corporate content and generating the same into a knowledge graph.

For example, domain-specific terms can be extracted from corporate content, entities can be defined based on the extracted terms, and relationships between entities can be connected to create a knowledge graph of information specific to the business domain to which the corporate belongs.

These enterprise domain-specific data and knowledge graphs for domain-specific data can be used to train LLMs that generate scenarios. More specifically, LLMs can be fine-tuned to generate scenarios specialized for financial business, insurance business, education business, or steel business by learning enterprise domain-specific data.

The method for automatically generating a video may include a step (S140) of receiving a user query.

The step of receiving a user query may include a step of receiving query information for video generation by a user inputted into an application (210) of a user device (200). For example, the application (210) may include a video generation application. The query information is information inputted by the user, and the query information may include text information, voice information, image information, or other data information. As an example, the user may input text such as "Make a 20-second promotional video for our company's new product," and the text information may be transmitted as query information to the automatic video generation device (100).

The method for automatically generating a video may include a step (S150) of embedding a received user query.

The step of embedding a user query (S150) may include a step of converting a user query into a vector representation and a step of modality-adjusting the user query information converted into the vector representation. The user query information including the modality-adjusted vector representation may be positioned within the same vector space as the integrated vector of multi-modalities.

User query information including modality-adjusted vector representations may be defined and/or referred to as query embedding data.

The method for automatically generating a video may include a step (S160) of searching for corporate embedding data similar to query embedding data.

The step (S160) of searching for similar corporate embedding data to the query embedding data may include a step of searching for similar corporate embedding data from a memory (130) or an external server where corporate embedding data are stored. This may be performed by calculating a similarity score between the query embedding data and the corporate embedding data and extracting a similarity search result. The similarity search result may include information about similarity scores calculated by calculating the similarity scores between the query embedding data and the corporate embedding data.

The method for automatically generating a video may include a step (S170) of extracting a candidate set based on the generated similarity test results.

The step (S170) of extracting a candidate set based on the similarity search result may be operated as a step of extracting, for example, at least one corporate embedding data with high relevance, in order of high similarity score based on the calculated similarity test result.

In one embodiment, the step (S170) of extracting a candidate set may include an operation of selecting the corporate embedding data having the highest weighted similarity score as the final candidate set by applying a weight set by a user or an administrator to a plurality of corporate embedding data having a similarity score greater than or equal to a threshold value. The corporate embedding data selected as the final candidate set may be referenced in the video generation process and utilized to generate a corporate-related video requested by the user.

In another embodiment, the step (S170) of extracting a candidate set may include an operation of selecting all of a plurality of corporate embedding data having a similarity score greater than or equal to a threshold value, or a plurality of corporate embedding data having a weighted similarity score greater than or equal to a threshold value, as a final candidate set. The plurality of corporate embedding data selected as the final candidate set may be referred to in a plurality of video generation processes, respectively, so that a plurality of corporate-related videos requested by the user may be generated. Accordingly, the user may select a video that matches the user's preference from among a plurality of videos provided from an automatic video generation device.

The method for automatically generating a video may include a step (S180) of passing a prompt, query embedding data, and context to an LLM (large-scale language model).

The prompt can be generated based on the query embedding data. For example, if the query embedding data includes information such as "Make a 20-second promotional video about our company's new product," the prompt can include, for example, an instruction to LLM to write a video script that includes information about the product. The video automatic generation device (100) or the video generation model (111) can include a prompt generation module or a prompt generation circuit.

Additionally, the prompt may include information about the corporate embedding data selected for the final candidate set.

Prompts may be created using preset prompt templates, but are not limited thereto.

Query embedding data and context can be passed to the LLM along with the prompt.

The context may include background information or information about the context related to the query embedding data. For example, the context may include information about the context of the query embedding data itself, or information or related data previously provided by the user, to enable the LLM to generate a consistent response. The video automatic generation device (100) or the video generation model (111) may include a context generation module or a context generation circuit.

The method for automatically generating a video may include a step (S190) of transmitting a response generated from an LMM.

The step of transmitting the response generated in the LLM (S190) may include a step of generating a video generation script in the LLM based on the prompt, query embedding data, and context.

The step (S190) of transmitting a response generated in LLM may include a step of transmitting a video generation script to a user device or a step of transmitting it to a video generation model (e.g., the video generation model (520) of FIG. 3).

In one embodiment, when passing a response generated from an LLM to a video generation model, a result generated from the video generation model based on the video generation script (e.g., a company-related video requested by a user) may be transmitted to the user device (200) and/or the application (210).

In another embodiment, when transmitting a response generated from an LLM to a user device, information about a video generation script and information necessary for video generation (e.g., corporate embedding data, etc.) may be received from a video automatic generation device (100), and a video may be generated based on the information.

FIG. 3 is a block diagram illustrating configurations of a video generation device according to various embodiments of the present disclosure.

The embodiment of FIG. 3 can be combined with the embodiments of FIGS. 1 to 2, or the embodiments of FIGS. 4 to 11.

Referring to FIG. 3, the automatic video generation device (100) may include a video generation model (111) (e.g., the video generation model (111) of FIG. 1).

The video generation model (111) can be communicatively and/or operatively connected to a user device (200) including an application (210) or a database (300) storing corporate information.

The video generation model (111) may include a content block generation unit (410), an embedding module (420), an external content index (430), a vector DB (440), a needs prediction unit (450), a content block search unit (460), a script generation unit (470), or a CSS editing unit (480). The video generation module (111) may be communicatively and/or operatively connected to an external content repository (510), a video generation model (520), or an LLM (530) (e.g., a large-scale language model), but is not limited thereto. For example, the external content repository (510), the video generation model (520), and the LLM (530) may be included in the video generation model (111) and may be defined and/or interpreted as some components of the video generation model (111).

The operations performed by the components of the video generation model (111) can be defined as operations performed by the video generation model (111) by an instruction when at least one instruction stored in a memory (e.g., memory (130) of FIG. 1) is executed by a processor (e.g., processor (120) of FIG. 1).

The content block generation unit (410) extracts main objects from corporate information including corporate content, for example, from each frame of an image or video, divides them into semantic units, and, based on the information obtained from the object extraction and semantic unit division, generates captions as metadata using a visual-language model (VLM) or an LLM that performs the function of a VLM.

According to an embodiment of the present invention, the content block generation unit (410) can divide any content into multiple content blocks. The content block generation unit can identify the logical flow of any content, the change points of the storyline, and divide the content based on the change points to generate content blocks. Alternatively, the content can be divided based on the change rate of the scene change or the camera angle, the change points of the background or the setting, the change points of the speaker or the character, the change points of the format or the style, and generate content blocks. As another example, the content block generation unit can divide the content based on the point where the mood or emotional line of the video changes significantly, the point where the structural elements of the video such as the intro, the main text, and the conclusion change, the point where the subtitle or overlay text changes, and the point where the background music or the sound effect changes to generate content blocks.

According to an embodiment of the present invention, the content block generation unit (410) can generate a caption of a content block using VLM or LLM. The content block caption can include text that provides a brief description or summary of each segment or block of the video.

The content block generation unit (410) can extract a representative image of a content block or a representative image feature of a content block and control the LLM to generate a data annotation for the content block.

More specifically, the content block generation unit (410) may apply a first prompt of “generate a question about the content block and generate a response to the question in a step-by-step manner” while transmitting the content block and a representative image of the content block to the LLM trained to understand the image or video.

For example, the content block generation unit (410) may apply the above prompt to the LLM together with the cooking tutorial content block. The LLM may generate a question, “Summarize the above content block,” and generate a first question, “What is the subject of the above content block?” to generate a response to the question, and generate a first response, “It is a cooking recipe.” Then, the LLM may generate a second question, “What is the cooking recipe covered in the above content block?” with reference to the first response, and generate a second response, “It is a carbonara pasta.” Then, the LLM may generate a third question, “What are the ingredients for making a carbonara pasta?” with reference to the second response, and generate a third question, “It is pasta, eggs, and cheese.”

can generate a third response called “What is the cooking order of pasta, eggs, and cheese?” Then, LLM can generate a fourth question with reference to the third response, “What is the cooking order of pasta, eggs, and cheese?” and generate a fourth response to it.

Thereafter, the content block generation unit (410) can apply a second prompt to the LLM: “Generate a caption for the content block using the generated question-answer set.”

In the previous example for the Cooking Tutorial content block, LLM could generate caption sentences for the content blocks such as “The topic is a cooking recipe”, “The cooking recipe is about pasta carbonara”, and “The ingredients for making pasta carbonara are pasta, eggs, and cheese”.

The caption for the content block generated in this way is stored as metadata for the content block and linked to the content block, and can be utilized for searching the content block.

Meanwhile, the content block generation unit (410) can generate one content block by combining the generated caption and the extracted object. The content block can be a result of integrating visual content and text description (e.g., visual content generated as metadata or description of its main object). The content block can be utilized in the video generation task in the video generation model (520) or can be utilized in searching for a video content block in the content block search unit (460).

The extracted main object may be, but is not limited to, a representative image of the video, a title of the video, a summary of the video, or a feature object of the video. The generated caption may include, but is not limited to, a caption for a representative image of the video, a caption for a title of the video, a caption for a summary of the video, or a caption for a feature object of the video. The caption may be generated through metadata of an external content index (430), but is not limited to this.

The generated content block may be stored in an external content storage (510) along with the content or caption, but is not limited thereto.

The generated content block may be processed into a high-dimensional vector by the embedding module (420) and stored in the vector DB (440).

The embedding module (420) can analyze data of various modalities (e.g., web pages, documents, images, videos, or audios, etc.) of corporate content collected from the database (300) through an extractor of a CNN-based model, embed them, and convert them into vectors. The embedding module (420) can convert vectors of each modality into the same vector space through modality adjustment. That is, by adjusting the size or scale of vectors of each modality, the adjusted vectors of each modality can be combined into one integrated vector in the same vector space. Accordingly, a similarity search that comprehensively considers vectors of different modalities can be enabled. The integrated vector can be stored in a vector DB (database) (440). In addition, a video generation model (520) (Foundation model) or an LLM (530) (large-scale language model) can learn the integrated vector.

The embedding module (420) can embed the preprocessed user query and convert it into a high-dimensional vector when the user query entered by the user through the user device (200) and application (210) is preprocessed by the preprocessing module and transmitted.

The content block search unit (460) can analyze a user query vector to identify its intent. For example, if a user inputs a query such as “Make a 20-second long video introducing our company’s new product,” the content block search unit (460) can understand the intent of the query and extract information for searching for content blocks related to the query. The content block search unit (460) can analyze the query to search for content blocks related to the type of content desired by the user. This can be performed by an operation of searching for a content block having a vector most similar to the user’s query. The content block search unit (460) can select content blocks most suitable for the user’s query based on the similarity search results and provide them to the user. The user can check the provided content blocks and, if necessary, create a more detailed user query based on the provided content blocks and re-enter it.

The needs prediction unit (450) can extract past behavior data based on the user's past usage history. The needs prediction unit (450) can analyze the user's query vector and past behavior data to recommend a personalized video template and video content style to the user in the form of selectable information. For example, the video template can provide a framework for how the video content to be generated will be structured. The video content style can include information about the video layout, the video background color, the font of the text included in the video, etc. For example, the needs prediction unit (450) can recommend a video template or video content style preferred by the user based on the query vector and past behavior data. For example, if there is past behavior data that is substantially similar to a query included in the query vector (for example, if the similarity between the two is greater than a specified threshold value), information about the video template included in the past behavior data and information about the video content style can be recommended.

The needs prediction unit (450) can generate sample video content based on personalized video template information and personalized video style information. The generated sample video content can be provided to a user, and the user can check the sample video content provided by the needs prediction unit (450) and, based on the video template information and video content style information, select whether to generate a video or modify the video template or content style.

The sample video content may include video content that reflects reference information related to the company. For example, the reference information related to the company may include BI information of the company or business domain information of the company. Accordingly, the sample video content provided to the user may reflect the BI information or business domain information of the user's company, and may be sample video content that reflects the strategic goals of the company or the specificity of the industry to which the company belongs, or final video content generated according to the user's query. Accordingly, the user may be provided with more customized and relevant video content, which may promote customer participation or effectively support the business goals of the company. Accordingly, the relevant video content may provide an advantage in the company's marketing, internal communication, or competitiveness enhancement.

The script generation unit (470) can transmit a prompt for writing a script, a user query vector, and a context to the LLM (530). The prompt can include information about a guideline that determines the direction and content of the script to be written by the LLM (530). The context can include information about a video template selected by the needs prediction unit (450), information about a video content style, or BI information or business domain information of the company.

LLM (530) can generate a script based on a prompt, a user query vector, and a context. The script can include the basic structure and core message of the video content. LLM (530) is learned through data extracted from the BI information of the company or the business domain information of the company, so that a script generated through LLM (530) can generate a highly relevant script by reflecting the BI characteristics of the company or the business domain information for the video content desired by the user requested through the user query.

The script generation unit (470) can receive a response (e.g., script) generated through LLM (530) and ultimately manipulate it to match the information selected by the needs prediction unit (450) and corporate data to generate a final script.

The video generation model (520) can receive the final script and generate video content requested by the user based on it.

Video content generated through the video generation model (520) can be provided to the user through a user device (200).

In some embodiments, the final script may be transmitted to an application (210) of the user device (200), and the user device (200) may generate video content requested by the user based on the script.

The CSS (Cascading Style Sheets) editing unit (480) can allow the user to edit the generated video content using a CSS-based editor. For example, after checking the provided video content, the user can control the project file of the video content, background rendering (e.g., background image, background color, or specific visual effect), or timeline (e.g., managing the presentation time of each visual element in the video). Accordingly, the video content can be edited to have a quality that meets the user's needs.

FIG. 4 is a block diagram illustrating configurations of a video generation device according to various embodiments of the present disclosure.

The embodiment of FIG. 4 can be combined with the embodiments of FIGS. 1 to 3, or the embodiments of FIGS. 5 to 11.

Referring to FIG. 4, the content block generation module (610) can collect corporate-related content or corporate-related information from the database (300). For example, corporate-related content or corporate-related information can include corporate BI information or corporate business domain information. Corporate-related content or corporate-related information can be information having various multi-modality characteristics such as web pages, documents, images, videos, or audio.

The content block creation module (610) can classify corporate-related content or corporate-related information into corporate BI information, corporate business domain information, or other corporate information, and extract objects through multi-modal analysis for each piece of information to create content blocks.

The generated content block can be converted into a high-dimensional vector with modality adjustment by the embedding module and stored in the vector DB (630).

The embedding module can extract data from corporate content or corporate information, analyze it, convert it into a modality-adjusted high-dimensional vector, and store it in a vector DB (630).

A user may input a user query into the video generation application (210). For example, a user may input a query such as "Make a 20-second promotional video for our company's new product." The input query may be transmitted to the scenario generation module (620) of the automatic video generation device (100).

The scenario generation module (620) of the automatic video generation device (100) can convert an input user query into a vector through an embedding module, perform a similarity search for the vector with respect to corporate data, and generate a plurality of scenarios based on a plurality of corporate data having a similarity score higher than a threshold value. Each of the plurality of scenarios includes information on the configurations of video content (e.g., the flow of the video and objects included in the video) for visually/audioally providing information included in the user query, and this can be provided to the user through the video application (210).

The video application (210) can convert multiple scenarios into videos and output them to the user, or output text information about multiple scenarios to the user.

The user can select a scenario that best suits the user's video production purpose or that the user prefers from among multiple scenarios through the video creation application (210), and information about this can be transmitted to the scenario creation module (620). At this time, the user can edit the configurations of the video content of the selected scenario, and information about the edited configurations of the video content can be transmitted to the scenario creation module (620).

The scenario generation module (620) can transmit information about a user query and a user-selected scenario to the LLM (640) through the vector DB (630).

LLM (640) can generate a script based on information about an input user query and a user-selected scenario. The generated script can be transmitted to a video generation application (210). The video generation application (210) can generate video content based on the script.

FIG. 5 is a flowchart illustrating a method for generating content blocks and captions for content blocks using LLM in a service server, and extracting candidate content blocks for a user query using the captions, according to various embodiments of the present disclosure.

A content creation service server according to an embodiment of the present invention can extract main objects from corporate information including corporate content, for example, from each frame of an image or video, divide them into semantic units, and, based on the information obtained from the object extraction and semantic unit division, create captions as metadata using a visual-language model (VLM) or an LLM that performs the function of the VLM.

Although not illustrated in FIG. 5, the service server may divide arbitrary content into multiple content blocks. The content block generation unit may identify the logical flow of arbitrary content, the change points of the storyline, and divide the original content based on the change points to generate content blocks. Alternatively, the content may be divided based on the rate of change of scene change or camera angle, the change points of background or setting, the change points of speakers or characters, or the change points of format or style to generate content blocks. As another example, the content block generation unit may divide the content based on the point where the mood or emotional line of the video significantly changes, the point where the structural elements of the video such as the intro, main text, and conclusion change, the point where the subtitle or overlay text changes, or the point where the background music or sound effects change to generate content blocks.

The service server can then generate captions for the content blocks using VLM or LLM. Content block captions can include text that provides a brief description or summary of each segment or block of the video.

More specifically, in step 501, the service server can extract a representative image of a content block or a representative image feature of a content block.

In step 502, the service server may generate a first prompt, “Generate a question for the content block and generate a response to the question in a stepwise manner.” In step 505, the service server may apply the first prompt to the LLM, while transmitting the content block and the representative image of the content block to the LLM trained to understand images or videos.

Thereafter, the service server can generate a second prompt “Generate a caption for the content block using the generated query-response set” in step 507, and apply the second prompt to the LLM in step 508.

At step 510, the service server can store the caption for the content block generated by the LLM in a database by linking it with the content block as metadata for the content block.

In step 520 thereafter, when the service server receives a user query, it can extract candidate content blocks based on the similarity between the content block caption and the user query vector.

Although not shown in Figure 5, the service server can use the extracted candidate content blocks to generate other content in response to a user query.

delete

delete

delete

delete

delete

delete

FIGS. 7 to 9 are flowcharts for explaining an operation method of an automatic video generation device according to various embodiments of the present disclosure.

The embodiments of FIGS. 7 to 9 can be combined with the embodiments of FIGS. 1 to 8.

The embodiments illustrated in FIGS. 7 to 9 may be applied to the descriptions of the configurations of embodiments in other drawings, or to the descriptions of the operations of the configurations.

Referring to FIG. 7, the operating method of the automatic video generation device may include a step (S411) of receiving corporate information (e.g., corporate content information including a web page, a document, an image, or a video); a step (S412) of extracting corporate brand identity data (e.g., a vector including corporate BI information) or corporate business domain data from the corporate information; a step (S413) of receiving a user query and extracting query data (e.g., a query vector); and a step (S414) of generating a video script based on the query data and the corporate BI data by using a learned video generation model (e.g., the video generation model (111) of FIGS. 1 and 2 or its components) that uses the corporate BI data as an input.

Referring to FIG. 8, the operating method of the automatic video generation device may include a step of extracting video information from corporate information (e.g., corporate content information including a web page, a document, an image, or a video) (S421); a step of analyzing the video information to generate a content block including object information and caption information (S422); a step of analyzing the content block to generate content block data (e.g., a content block vector) (S423); a step of receiving a user query from a user device (S424); a step of analyzing the user query to generate query data (e.g., a query vector) (S425); a step of searching for a content block with the highest similarity based on the content block data and the query data by using a video generation model learned with the content block data as input (e.g., the video generation model (111) of FIGS. 1 and 2 or its components) (S426); and a step of transmitting information on the searched content block to the user device (S427).

Referring to FIG. 9, the operating method of the automatic video generation device may include a step of receiving corporate information (S431); a step of extracting corporate data (e.g., corporate vector, corporate BI vector, or corporate business domain vector) from the corporate information (S432); a step of receiving a user query and extracting query data (e.g., query vector) (S433); a step of extracting past user behavior data (S434); and a step of generating reference video information based on the query data and the past behavior data by using a video generation model learned with the corporate data as input (e.g., the video generation model (111) of FIGS. 1 and 2 or its components), and transmitting the generated reference video information to a user device (S435).

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

FIG. 12 is a block diagram illustrating a user device according to various embodiments of the present disclosure.

The embodiment of Fig. 12 can be combined with the embodiments of Figs. 1 to 11.

Referring to FIG. 12, the user device (200) may include a system memory (801) including an operating system (802), a processing unit (803), a storage device (804), an input device (805), an output device (806), or a communication device (807).

The user device (200) may include a computing device or a smart device.

The system memory (801) may include a RAM (Random Access Memory) and may provide a temporary storage space where the operating system and program modules are executed. The system memory (801) may temporarily store data and instructions and allow the processing unit (803) to quickly access and process data.

The operating system (802) may include software. The operating system includes memory management, file system management, processor management, and device management, and may control programs running within the device. The operating system (802) may include program modules, and the program modules are software modules for performing specific functions, and may provide or be in charge of a user interface or network, etc.

The processing unit (803) may include, but is not limited to, a CPU. The processing unit (803) may execute instructions stored in the system memory (801), process input data, and generate results or perform computational tasks.

The storage device (804) may include a device that permanently stores data and programs.

The input device (805) is a device that receives data from a user or another system, and may include a keyboard, a mouse, a touch screen, or a soft key implemented on a display panel including a touch panel.

The output device (806) is a device that transmits data processed through the processing unit (803) to a user or another system, and may include a monitor, speaker, or display panel.

The communication device (807) can enable the user device (200) to be communicatively connected to an external network or another device (e.g., an automatic video generation device (100)).

According to various embodiments of the present disclosure, an automatic video generation device may be coupled with a computer (or computing device) as hardware and may include a computer program stored in a computer-readable recording medium to perform the operations described with reference to the drawings described above as examples.

Additionally, the automatic video generation device may be implemented as a computing device including at least one processor that executes instructions of programs loaded into a memory, and a program including instructions described to execute the operations described with reference to the drawings as examples may be loaded into the memory.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, the processing device may include multiple processors, or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal waves, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over network-connected computer systems, and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, flash memories, etc. Examples of the program commands include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

Although the embodiments have been described above by way of limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, appropriate results can be achieved even if the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims are also included within the scope of the claims.

Although the detailed description of this document has described specific embodiments, it will be apparent to those skilled in the art that various modifications may be made without departing from the scope of this document.

delete

delete

Claims (10)

In the operating method of the automatic video generation device,
In the above video automatic generation device, a step of receiving corporate content;
In the above video automatic generation device, a step of extracting domain-specific terms from the corporate content and training a first artificial intelligence model using the domain-specific terms as input;
In the above video automatic generation device, a step of receiving a user query and reference content;
In the above video automatic generation device, a step of generating data on the structure of a sample video based on the reference content using the first artificial intelligence model; and
In the above video automatic generation device, a step of generating a sample video based on data about the structure using a second artificial intelligence model separate from the first artificial intelligence model,
How the video auto-generation device works.
In the first paragraph,
The above reference content is,
Contains information about video templates and information about video content styles.
How the video auto-generation device works.
delete In the first paragraph,
If the similarity between the user query and the user's past behavior data is greater than a specified threshold, the sample video includes sample video information included in the past behavior data.
How the video auto-generation device works.
In the first paragraph,
In the above video automatic generation device, a step of generating a prompt based on the user query;
A step of generating a context based on a sample video selected by a user; and
A step of generating a video script based on the above prompt, the user query, and the context,
How the video auto-generation device works.
In clause 5,
In the above video automatic generation device, a step of generating video content based on the video script;
In the above video automatic generation device, a step of transmitting the video content to a user device; and
In the above video automatic generation device, further comprising a step of receiving completion information or modification information input by a user,
How the video auto-generation device works.
In Article 6,
In the above video automatic generation device, when receiving modification information by the user, the step of reproducing the video content by modifying at least one of the object information and the background information of the video content based on the modification information and transmitting the reproducing video content to the user device is further included.
How the video auto-generation device works.
In the automatic video generation device,
at least one processor; and
Contains memory that stores instructions,
The above instructions, when individually or collectively executed by the at least one processor,
Receive corporate content,
Extract domain-specific terms from the above corporate content, and train the first artificial intelligence model with the domain-specific terms as input.
Receive user queries and reference content,
Using the first artificial intelligence model, data on the structure of the sample video is generated based on the reference content,
An automatic video generation device that operates to generate a sample video based on data about the structure using a second artificial intelligence model separate from the first artificial intelligence model.
A computer-readable recording medium, which is combined with hardware and is stored to perform the operation method of the automatic video generation device of any one of claims 1, 2, 4 to 7,
Computer program.
In the automatic video generation system,
A user device for receiving user queries; and
Including a video auto-generation device,
The above video automatic generation device,
A video automatic generation system which receives corporate content, extracts domain-specific terms from the corporate content, trains a first artificial intelligence model using the domain-specific terms as input, receives a user query and reference content, generates data on the structure of a sample video based on the reference content using the first artificial intelligence model, and generates a sample video based on the data on the structure using a second artificial intelligence model separate from the first artificial intelligence model.