TWI913240B - System of intelligently adjust mixing audio and method of intelligently adjust mixing audio - Google Patents

Abstract

A system of intelligently adjust mixing audio includes a class identification module, an attribute computation module, and an adjusting module. The class identification module is configured to identify an audio class of audio signals. The attribute computation module is coupled to the class identification module so as to determine a primary audio signal of the audio signals by determining whose audio class satisfies a primary audio class. The adjusting module is coupled to the attribute computation module. The attribute computation module adjusts an attribute of the audio signals and the primary audio signal, and then the signals are audio mixed to ouput.

Description

Intelligent audio adjustment and mixing system and intelligent audio adjustment and mixing method

This case relates to an audio adjustment technique, and more particularly to an audio adjustment and mixing technique.

In a multi-source multimedia environment, when multiple audio sources are mixed and output, users often need to highlight specific audio sources so that listeners can clearly receive the theme. The conventional method is to first set a specific audio source channel as the primary audio source. However, this method is quite inflexible, especially when the theme changes, requiring the designated primary audio source channel to be changed. If the audio source channel is not changed, this leads to the amplification of the wrong channel during mixing. Therefore, how to more flexibly determine the primary and secondary audio sources is a technical problem that urgently needs to be solved in this field.

The invention is intended to provide a simplified summary of this disclosure to enable the reader to have a basic understanding of the subject matter. This invention is not a complete overview of this disclosure and is not intended to identify key/critical elements of the embodiments of this case or to define the scope of this case.

According to one embodiment of this case, an intelligent audio adjustment and mixing system is disclosed, suitable for processing a plurality of audio signals. The intelligent audio adjustment and mixing system includes a category identification module, an attribute calculation module, and an adjustment module. The category identification module is configured to identify each audio signal to obtain a corresponding audio category. The attribute calculation module is coupled to the category identification module, wherein the attribute calculation module determines from the audio signals that the audio category matches a primary audio signal of a primary audio category. The adjustment module is coupled to the attribute calculation module, wherein the adjustment module adjusts an audio attribute of each audio signal and/or the primary audio signal respectively before mixing and outputting the audio.

According to another embodiment, an intelligent audio adjustment and mixing method is disclosed, comprising: receiving a plurality of audio signals; identifying each audio signal to obtain a corresponding audio category; determining from the audio signals that the audio category matches a primary audio signal of the primary audio category; and adjusting an audio attribute of the audio signals and/or the primary audio signal respectively, and then mixing and outputting the audio signal.

According to another embodiment, an intelligent audio adjustment and mixing method is disclosed, comprising: receiving at least two audio signals; identifying at least one of the two audio signals to obtain a corresponding audio category; adjusting an audio attribute of at least one of the two audio signals according to the audio category, and then mixing and outputting the audio.

The following disclosure provides numerous different embodiments to implement the various features of this application. Examples of elements and their arrangements are described below to simplify this application. Of course, these embodiments are merely illustrative and not intended to be restrictive. For example, the use of terms such as "first" and "second" to describe elements in this application is solely for distinguishing between identical or similar elements or operations; these terms are not intended to limit the technical elements of this application, nor to limit the order or sequence of operations. Furthermore, element symbols and/or letters may be repeated in various embodiments of this application, and the same technical terms may use the same and/or corresponding element symbols in various embodiments. This repetition is for the purpose of simplicity and clarity and does not in itself indicate a relationship between the various embodiments and/or configurations discussed.

This embodiment discloses an intelligent audio adjustment and mixing method comprising at least the following steps: Step 1: receiving at least two audio signals; Step 2: identifying at least one of the two audio signals to obtain a corresponding audio category; Step 3: adjusting the audio attribute of at least one of the two audio signals according to the audio category, and then outputting the mixed audio.

In the above embodiments, step two can vary depending on actual needs. For example, it can be that one of the two audio signals is identified and its corresponding audio category is obtained, or both audio signals are identified and their corresponding audio categories are obtained. Of course, there can be multiple audio signals, and there is no limitation here.

In addition, step three can also have different variations depending on actual needs. For example, in step two, when the first audio signal is identified as corresponding to a first audio category, step three can adjust at least one audio attribute of the first audio signal according to the first audio category it corresponds to. Of course, in the previous case, it is also possible to adjust at least one audio attribute of other audio signals (for example, the second audio signal) according to the first audio category it corresponds to. For example, in step two, when the first audio signal is identified as corresponding to a first audio category and the second audio signal is identified as corresponding to a second audio category, step three can adjust at least one audio attribute of the first audio signal according to the first audio category corresponding to the first audio signal, and can adjust at least one audio attribute of the second audio signal according to the second audio category corresponding to the second audio signal. The audio attribute adjusted for the first audio signal and the audio attribute adjusted for the second audio signal can be the same or different. Furthermore, after adjusting the audio attributes as described above, mixing can be performed. To illustrate the details of this invention more clearly, some more concrete embodiments are given below with reference to Figures 1 and 2. In this embodiment, assuming both audio signals are adjusted, the two adjusted audio signals are mixed and output. assuming one audio signal is adjusted and the other is not adjusted, the adjusted audio signal is mixed with the unadjusted audio signal and output.

Please refer to Figure 1, which illustrates a schematic diagram of an intelligent audio adjustment and mixing system 100 according to some embodiments of this invention. The intelligent audio adjustment and mixing system 100 may be, for example, a mixer, a device or software equipped with mixing functions, a device or software capable of performing mixing functions connected in series with a mixer, or any electronic device or software capable of performing mixing functions, without limitation herein. The intelligent audio adjustment and mixing system 100 includes audio signal receiving modules 110a-110d, category identification modules 120a-120d, adjustment modules 130a-130d, attribute calculation module 140, and audio output module 150. As shown in Figure 1, audio signal receiving module 110a is coupled to category identification module 120a and adjustment module 130a; audio signal receiving module 110b is coupled to category identification module 120b and adjustment module 130b; audio signal receiving module 110c is coupled to category identification module 120c and adjustment module 130c; and audio signal receiving module 110d is coupled to category identification module 120d and adjustment module 130d. Category identification modules 120a-120d are coupled to attribute calculation module 140. Attribute calculation module 140 is coupled to adjustment modules 130a-130d. Adjustment modules 130a~130d are coupled to audio output module 150.

In some embodiments, audio signal receiving modules 110a-110d are used to receive audio signals from various sources. For example, the audio source can be an audio signal input through an audio connection terminal (not shown), an audio signal captured by an audio capture module (not shown), an audio signal from a video, etc., and is not limited thereto. The audio signal receiving modules 110a-110d can receive audio signals simultaneously, process these audio signals separately, and then transmit them to the audio output module 150 for mixing and output.

In some embodiments, the audio signal receiving module 110a transmits the audio signal to the category identification module 120a. The category identification module 120a identifies the audio signal to obtain the corresponding audio category. For example, the audio category can be human voice, instrumental sound, noise, etc. The category identification module 120a transmits the identified audio category data to the attribute calculation module 140. Similarly, audio signal receiving module 110b transmits the audio signal to category identification module 120b, audio signal receiving module 110c transmits the audio signal to category identification module 120c, and audio signal receiving module 110d transmits the audio signal to category identification module 120d. Category identification modules 120b-120d respectively transmit the identified audio category data to attribute calculation module 140. For example, category identification module 120a identifies the audio signal from audio signal receiving module 110a as an instrument sound or a musical sound. Category identification module 120b identifies the audio signal from audio signal receiving module 110b as a human voice. Category identification module 120c identifies the audio signal from audio signal receiving module 110c as ambient background noise. Category identification module 120d identifies the audio signal from audio signal receiving module 110d as noise. Furthermore, the audio signal identification by category identification modules 120a-120d can be achieved using sound recognition technology.

In some embodiments, the attribute calculation module 140 can pre-set a primary audio category to identify a primary audio signal whose category matches the primary audio category from the audio signals. For example, if the user wants to process human voice audio signals, the attribute calculation module 140 can set audio signals belonging to human voice as the primary audio category. It is worth mentioning that the user can set other categories as the primary audio category according to actual needs. In addition to being preset, the main audio categories can also be generated automatically: For example (1), the main audio categories can be generated automatically in real time by analyzing audio signals. For example, based on the category to which each audio signal belongs, the corresponding scene (such as music performance, singing performance, meeting, etc.) can be found through computational analysis such as built-in database comparison, cloud analysis, and neural network-like computation, and the corresponding main audio category can be generated based on the scene. In addition, regarding the example of automatic generation of main audio categories (2), users can set weights or priorities for each audio category according to their actual needs, and automatically generate the main audio category. For example, set human voice as the first level (highest level), instrumental or musical sounds as the second level (secondary level), ambient background sounds as the third level (secondary level), etc. For example, when it is determined from the audio signal that there is human voice, set this audio signal (not limited to one) as the main audio signal. For another example, when it is determined from the audio signal that there is instrumental or musical sound but no human voice, set this audio signal (not limited to one) as the main audio signal.

In this embodiment, the attribute calculation module 140 determines from the four audio signals that the audio signal of the audio signal receiving module 110b belongs to the category of human voice, and therefore sets the audio signal of the audio signal receiving module 110b as the primary audio signal. Then, the attribute calculation module 140 generates a control signal based on this determination result (hereinafter, for ease of explanation, the control signal is exemplified as an adjustment command, but this is not a limitation), and sends it back to the adjustment module 130b. In another embodiment, the attribute calculation module 140 determines that the other audio signals are secondary audio signals and sends adjustment commands back to the corresponding adjustment modules 130a, 130c~130d. This adjustment command is generated corresponding to the category of each audio signal to perform corresponding audio attribute processing on different audio signal categories. In some embodiments, control signals (adjustment commands) can be returned only to the receiving module of the main audio signal, only to the receiving module of the non-main audio signal, or to the receiving module of all audio signals. In another embodiment, there may be two or more primary audio signals. For example, attribute calculation module 140 determines from these four audio signals that the audio signals of audio signal receiving modules 110a and 110b belong to the category of human voice, and determines that the audio signals of audio signal receiving modules 110c and 110d belong to the categories of ambient background sound and noise, respectively. Therefore, the audio signals of audio signal receiving modules 110a and 110b are set as the primary audio signals. In some embodiments, adjustment modules 130a to 130d receive corresponding adjustment commands and use the corresponding adjustment commands to process the audio attributes of the audio signals and/or the primary audio signals, respectively. In some embodiments, the audio attributes of the primary audio signal may be processed only, the audio attributes of the non-primary audio signal may be processed only, or the audio attributes of all audio signals may be processed. The audio attributes of the audio signal can be volume, loudness, or a combination thereof, and are not limited here. For example, the attribute calculation module 140 determines that the audio signal (voice) of the audio signal receiving module 110b is the primary audio category. Therefore, the adjustment instruction received by the adjustment module 130b can amplify the volume of the voice audio signal so that the output audio signal after subsequent mixing can highlight the voice. For another example, attribute calculation module 140 determines that the audio signal from audio signal receiving module 110d is noise. Therefore, the adjustment instruction received by adjustment module 130d is to reduce the volume of the noise to reduce its impact on the output audio signal after subsequent mixing. For yet another example, different audio attributes of at least one audio signal can be adjusted. For instance, attribute calculation module 140 determines that the audio signal (voice) from audio signal receiving module 110b is the primary audio category, and the audio signal from audio signal receiving module 110d is noise. Therefore, the adjustment command received by the adjustment module 130b can amplify the volume of the human voice audio signal, while the adjustment command received by the adjustment module 130d can reduce the noise level so that the output audio signal after subsequent mixing can highlight the human voice and reduce the impact of noise.

In some embodiments, the audio output module 150 is used to receive audio signals with adjusted audio attributes, mix these audio signals with adjusted audio attributes, and output a mixed signal. In some embodiments, the audio output module 150 can also be used to receive at least one audio signal with adjusted audio attributes and at least one audio signal without adjusted audio attributes, mix the received audio signals, and then output a mixed signal. Therefore, the mixed signal output by the audio output module 150 can be composed of part or all of the audio signals with adjusted audio attributes and part or all of the audio signals without adjusted audio attributes. The actual situation may vary depending on the actual product requirements.

In some embodiments, the category identification modules 120a-120d may be unable to identify the audio category from these audio signals. In such cases, the adjustment modules 130a-130d will directly output the audio signal of the unknown category to the audio output module 150 without processing the audio signal of the unknown category.

In some embodiments, the attribute calculation module 140 pre-sets the priority of each audio attribute, and generates adjustment instructions for each audio attribute's audio based on this priority. For example, the primary audio signal has the highest priority. Therefore, the volume, loudness, and other audio attributes of the primary audio signal are preferentially adjusted to the largest value among all audio signals.

In this way, users can freely set the corresponding main audio category according to their actual needs, and strengthen the audio attributes such as volume and loudness of the audio signals they want to emphasize, while weakening the audio signals that may cause interference, thereby increasing the overall thematic recognition of the output audio source.

Please refer to Figure 2, which illustrates a flowchart of an intelligent audio adjustment and mixing method 200 according to some embodiments of this invention. The intelligent audio adjustment and mixing method 200 can be performed by the intelligent audio adjustment and mixing system 100 of Figure 1.

For example, in step S210, a primary audio category is preset. In some embodiments, the audio category can be multiple different categories (e.g., vocals, instrumental sounds, noise, etc.), and one category is set as the primary audio category from these different categories. Users can set the category to highlight the audio according to the actual application. For example, vocals can be preset as the primary audio category.

For example, in step S220, a plurality of audio signals are received. For instance, the audio signals may be audio signals from an audio capture module or a multimedia file, and this application is not limited to the source of the audio signals.

In step S230, multiple audio categories of the audio signals are identified. For example, category identification module 120a identifies the audio signal of audio signal receiving module 110a as an instrumental sound or musical sound. Category identification module 120b identifies the audio signal of audio signal receiving module 110b as a human voice. Category identification module 120c identifies the audio signal of audio signal receiving module 110c as an ambient background sound. Category identification module 120d identifies the audio signal of audio signal receiving module 110d as noise.

For example, in step S240, the audio signal category is determined to be a primary audio signal belonging to the primary audio category. For instance, in step S230, the audio category of each audio signal has been identified, and therefore it can be determined whether each audio category is a primary audio category. In this embodiment, the audio signal from the audio signal receiving module 110b is human voice, which is a primary audio signal. Other audio signals can be determined as secondary audio signals.

For example, in step S250, it is determined whether the primary audio signal is interfered with by other audio signals. In some embodiments, the attribute calculation module 140 can detect the degree of interference of the primary audio signal by measuring the volume and/or loudness of all received audio signals. For example, the volume and/or loudness of the secondary audio signals may be similar to that of the primary audio signal. After the overall mix output, the primary audio signal may be masked, for example, due to masking effects, making it impossible for the user to clearly hear the content of the primary audio signal. If it is determined that the primary audio signal is interfered with by other audio signals, then step S260 is executed.

In step S260, adjustment commands are used to adjust the audio attributes of each audio signal and the primary audio signal respectively. For example, the volume and/or loudness of the primary audio signal is increased, and/or the volume and/or loudness of the secondary audio signal is decreased.

As in step S270, an audio signal is output to an audio output module. For example, these audio signals with adjusted audio properties are sent to the audio output module for mixing and output.

In some embodiments, if it is determined that the main audio signal is not interfered with by other audio signals, the process returns to step S270. In other words, if the main audio signal is not interfered with by other audio signals, its audio properties do not need to be adjusted, and the mix can be output directly.

In some embodiments, if the audio category of the audio signal cannot be identified in step S230, the audio signal of this unknown category will be directly output to the audio output module 150.

In some embodiments, the intelligent audio adjustment mixing method 200 pre-sets priorities for each audio category, generating adjustment instructions based on these priorities for each audio attribute. For example, the audio category of the primary audio signal has the highest priority. Therefore, the volume, loudness, and other audio attributes of the primary audio signal are prioritized for enhancement, such as being adjusted to the largest value among all audio signals, to stand out from these audio signals.

The intelligent audio adjustment method can be used not only in the field of audio mixing, but also in other places. For example, after intelligent audio adjustment as described in all the above embodiments, all (or part of) the adjusted audio signals and all (or part of) the unadjusted audio signals can be output separately. For example, it can be used in multi-channel amplification or multi-channel output. Of course, it can also be used in applications that connect to other audio outputs. There are no limitations here.

In summary, unlike traditional audio adjustment methods that use specific hardware or channels as the primary audio source, the intelligent audio adjustment and mixing system and method in this application can set corresponding primary audio categories for different scenarios. By judging the category of each audio source, the primary audio source is identified, thus improving its recognizability. In this way, the primary audio source can be dynamically changed according to different primary audio categories, adjusting its audio attributes appropriately based on the corresponding primary audio source, thereby achieving automatic switching between audio sources to highlight the desired audio category.

The foregoing outlines the features of several embodiments to enable those skilled in the art to better understand the nature of this application. Those skilled in the art should understand that, without departing from the spirit and scope of this application, the foregoing can be readily used as the basis for designing or modifying other variations to achieve the same purpose and/or realize the same advantages of the embodiments described herein. The foregoing should be understood as illustrative of this application, and its scope of protection should be determined by the scope of the patent application.

100: Intelligent Audio Adjustment and Mixing System; 110a~110d: Audio Signal Receiving Module; 120a~120d: Category Recognition Module; 130a~130d: Adjustment Module; 140: Attribute Calculation Module; 150: Audio Output Module; 200: Intelligent Audio Adjustment and Mixing Method; S210~S270: Steps

The following detailed description, when read in conjunction with the accompanying diagrams, will facilitate a better understanding of the form of this disclosure. It should be noted that, depending on the practical needs of the illustration, the features in the diagrams are not necessarily drawn to scale. In fact, the dimensions of the features may be arbitrarily increased or decreased for clarity of explanation. Figure 1 illustrates a schematic diagram of an intelligent audio adjustment and mixing system according to some embodiments of this application. Figure 2 illustrates a flowchart of an intelligent audio adjustment and mixing method according to some embodiments of this application.

Domestic storage information (please record in order of storage institution, date, and number): None. International storage information (please record in order of storage country, institution, date, and number): None.

100: Intelligent Audio Adjustment and Mixing System

110a~110d: Audio signal receiving module

120a~120d: Category Recognition Module

130a~130d: Adjustment Module

140: Attribute Calculation Module

150: Audio Output Module

Claims (10)

An intelligent audio adjustment and mixing system is adapted to process a plurality of audio signals, wherein the intelligent audio adjustment and mixing system includes: a category identification module configured to identify each of the audio signals to obtain a corresponding audio category; an attribute calculation module coupled to the category identification module, wherein the attribute calculation module determines from the audio signals that the audio category conforms to a primary audio signal of a primary audio category; and an adjustment module coupled to the attribute calculation module, wherein the adjustment module adjusts an audio attribute of each of the audio signals and/or the primary audio signal respectively before outputting a mixed audio signal. The attribute calculation module is configured to set a priority for the audio category, with the audio category of the main audio signal having the highest priority. The attribute calculation module generates an adjustment command based on the priority of the audio category and sends the adjustment command back to the adjustment module. The adjustment module uses the adjustment command to process the audio attributes of the audio signal respectively for remixing and output. The intelligent audio adjustment and mixing system as described in claim 1, wherein the audio attributes include a volume and/or a loudness of each audio signal, and the audio categories of the audio signals include a voice, an instrument sound, and/or noise. The intelligent audio adjustment and mixing system as described in claim 2, wherein the primary audio category has the highest priority, such that the audio attribute of the primary audio signal is preferentially enhanced. The intelligent audio adjustment and mixing system as described in claim 1, wherein the attribute calculation module is configured to pre-set the primary audio category to determine from the audio signals that the audio category matches the primary audio signal of the primary audio category. A smart audio adjustment and mixing method includes: setting a priority for an audio category; receiving a plurality of audio signals; identifying each audio signal to obtain a corresponding audio category; determining from the audio signals that the audio category matches a primary audio signal of a primary audio category, wherein the audio category of the primary audio signal has the highest priority; adjusting the audio attributes of each audio signal and/or the primary audio signal respectively, and then mixing and outputting the audio signal; generating an adjustment instruction based on the priority of the audio category; and using the adjustment instruction to process the audio attributes of the audio signals respectively, so as to remix and output the audio signal. The intelligent audio adjustment and mixing method described in claim 5 further includes: presetting the main audio category. The intelligent audio adjustment and mixing method as described in claim 5, wherein the audio attributes include a volume and/or a loudness of each audio signal, and the audio categories of the audio signals include a voice, an instrument sound, and/or noise. The intelligent audio adjustment and mixing method as described in claim 5 further includes: generating the adjustment instruction according to the priority of the audio category, wherein the audio category of the primary audio signal has the highest priority. The intelligent audio adjustment and mixing method as described in claim 8, wherein the primary audio category has the highest priority, such that the numerical value of the audio attribute of the primary audio signal is preferentially strengthened. A smart audio adjustment and mixing method includes: setting a priority for an audio category; receiving at least two audio signals; identifying at least one of the at least two audio signals to obtain the corresponding audio category; and generating an adjustment instruction based on the priority of the audio category, and using the adjustment instruction to adjust the audio attributes of at least one of the at least two audio signals, followed by a mixed output.