JP2025124424A - Data processing method and data processing device - Google Patents

Abstract

[Problem] To provide a data processing method that can reproduce optimal video according to the reproduction environment without impeding real-time performance. [Solution] The data processing method generates multi-track audio data that stores audio data of multiple channels including at least a first channel and a second channel, and stores a data string of digital audio signals in the first channel, and stores motion data, which is information on character movement and is associated with the digital audio signals, as a data string of digital audio signals in the second channel. [Selected Figure] Figure 3

Description

One embodiment of the present invention relates to a data processing method for processing multi-track audio data that stores audio data for multiple channels.

Patent document 1 discloses a transmission means that associates the operation information and operation time stored in the second storage means with identification information that identifies the music information output by the first control means and transmits the information to a specified server device.

Patent document 2 describes generating a character movement image from the singer's head movement.

JP 2015-47261 A JP 2012-78526 A

The amount of video data is larger than the amount of audio data. Therefore, if audio data and video data are transmitted separately, the delay in the video data will be large. This makes it difficult to synchronize the audio data with the video data. Furthermore, if audio data and video data are transmitted in synchronization, real-time performance will be reduced (the transmission of the audio data will be delayed in order to synchronize the audio data with the video data).

Furthermore, when video data is distributed, it may not be played in a way that is appropriate for the environment of the playback venue. For example, if video data shot outdoors is played in an indoor playback venue, the user may feel uncomfortable.

Therefore, an object of one embodiment of the present invention is to provide a data processing method that can play optimal video depending on the playback environment without impeding real-time performance.

A data processing method according to one embodiment of the present invention is a data processing method for generating multi-track audio data that stores audio data for multiple channels including at least a first channel and a second channel, in which a data stream of digital audio signals is stored in the first channel, and motion data, which is information about character movement and is associated with the digital audio signals, is stored in the second channel as a data stream of digital audio signals.

According to one embodiment of the present invention, it is possible to play back video that is optimal for the playback environment without impeding real-time performance.

1 is a block diagram showing a configuration of a data processing system 1. 1 is a block diagram showing the main configuration of a data processing device 10. FIG. 4 is a flowchart showing the operation of a CPU 104. 10 is a flowchart showing the operation of a processing unit 154. FIG. 2 is a diagram illustrating an example of the structure of motion data. FIG. 10 is a diagram showing an example of a format when motion data is stored as a data string of a digital audio signal. FIG. 1 is a block diagram showing a configuration of a data processing system 1A in a playback environment. 10 is a flowchart showing the operation of a processing unit 154 of the data processing device 10 during playback.

Figure 1 is a block diagram showing the configuration of a data processing system 1. The data processing system 1 includes a data processing device 10, a mixer 11, and a motion capture device 12.

Each device is connected via a communication standard such as a USB cable, HDMI (registered trademark), Ethernet (registered trademark), or MIDI. Each device is installed in the room of a performer, for example, where a live performance is taking place.

The mixer 11 connects multiple audio devices, such as microphones, musical instruments, or amplifiers. The mixer 11 receives digital or analog audio signals from the multiple audio devices. When the mixer 11 receives an analog audio signal, it converts the analog audio signal into a 24-bit digital audio signal with a sampling frequency of, for example, 48 kHz. The mixer 11 performs signal processing such as mixing, gain adjustment, equalization, or compression on the multiple digital audio signals. The mixer 11 transmits the processed digital audio signals to the data processing device 10.

The motion capture device 12 is a sensor for capturing the motion of the performer, and may be, for example, an optical, inertial, or image sensor. The data processing device 10 receives sensor signals from the motion capture device 12. The data processing device 10 generates motion data based on the sensor signals from the motion capture device 12.

Figure 2 is a block diagram showing the main components of the data processing device 10. The data processing device 10 is composed of a general-purpose personal computer or the like, and includes a display 101, a user interface (I/F) 102, a flash memory 103, a CPU 104, a RAM 105, and a communication interface (I/F) 106.

The display 101 is composed of, for example, an LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode), and displays various information. The user I/F 102 is composed of a switch, keyboard, mouse, trackball, touch panel, or the like, and accepts user operations. If the user I/F 102 is a touch panel, the user I/F 102, together with the display 101, constitutes a GUI (Graphical User Interface).

The communication I/F 106 is connected to the mixer 11 and the motion capture device 12 via a communication line such as a USB cable, HDMI (registered trademark), Ethernet (registered trademark), or MIDI. The communication I/F 106 receives digital audio signals from the mixer 11. The communication I/F 106 also receives sensor signals from the motion capture device 12. The mixer 11 may also transmit digital audio signals to the data processing device 10 via Ethernet (registered trademark) using a protocol such as Dante (registered trademark).

The CPU 104 corresponds to the processing unit of the present invention. The CPU 104 reads a program stored in the flash memory 103, which is a storage medium, into the RAM 105 to implement predetermined functions. For example, the CPU 104 displays an image on the display 101 for accepting user operations, and accepts selection operations and the like on the image via the user I/F 102, thereby implementing a GUI. The CPU 104 receives digital signals from other devices via the communication I/F 106. The CPU 104 generates multi-track audio data based on the received digital signals. The CPU 104 also stores the generated multi-track audio data in the flash memory 103. Alternatively, the CPU 104 distributes the generated multi-track audio data.

Note that the program read by the CPU 104 does not have to be stored in the flash memory 103 of the device itself. For example, the program may be stored in a storage medium of an external device such as a server. In this case, the CPU 104 simply reads the program from the server into the RAM 105 and executes it each time.

Multi-track audio data conforms to the Dante (registered trademark) protocol, for example. For example, Dante (registered trademark) can transmit 64-channel audio signals at 100base-TX. Each channel stores, for example, a 48 kHz sampling frequency, 24-bit digital audio signal (uncompressed digital audio data in WAV format, etc.). However, in the present invention, the number of channels, sampling frequency, and bit depth of the multi-track audio data are not limited to this example. Furthermore, the multi-track audio data does not need to conform to the Dante (registered trademark) protocol.

Figure 3 is a functional block diagram showing the minimum configuration of the present invention. The processing unit 154 shown in Figure 3 is realized by software executed by the CPU 104. Figure 4 is a flowchart showing the operation of the processing unit 154. The processing unit 154 receives a digital audio signal from the mixer 11 (S11) and a sensor signal from the motion capture unit 12 (S12). The processes of S11 and S12 may be performed simultaneously, or either may be performed first.

Then, the processing unit 154 generates motion data based on the sensor signal received from the motion capture device 12 (S13).

Figure 5 shows an example of the structure of motion data. Motion data is information about the movement of a character, and is information used to control the behavior of a 3D model of the character. The motion data in Figure 5 includes description data and position information for multiple bones (bone data). The multiple bones correspond to the movable parts of the character, such as the limbs and fingers. The multiple bones make up a skeleton that corresponds to the character's body.

Description data includes identification information for identifying bones (bone name information), information indicating the connection relationships with other bones, and skeleton identification information. Because description data is static information, it does not need to be included in the motion data. For example, description data may be generated and output separately from the motion data. Also, description data may be included once every few frames or every few dozen frames, rather than every frame.

Bone data includes position coordinate information and rotation coordinate information. Position coordinate information is expressed, for example, as a three-axis Cartesian coordinate system with a certain position as the origin. If 32 bits of information are stored per frame for each of the three axes (X, Y, Z), the position coordinate information will be 96 bits of data. If 32 bits of information are stored per frame for each of the four axes (X, Y, Z, W), the rotation coordinate information will be 128 bits. If one skeleton contains, for example, 30 bones, the motion data will be 6,720 bits of data per frame. If such bone data is driven at, for example, 60 fps, the motion data requires a data capacity of 403,200 bits (50,400 bytes) per second. Note that motion data does not need to be absolute coordinates. Motion data can also be relative coordinates with respect to a reference bone. Relative coordinates do not need to be included every frame; they can be included only in frames where there is a change relative to the reference bone.

After generating the motion data, the processing unit 154 generates multi-track audio data based on the digital audio signal and the motion data (S12). Specifically, the processing unit 154 stores the data string of the digital audio signal received from the mixer 11 in the first channel of the multi-track audio data. When the processing unit 154 receives audio data conforming to the Dante (registered trademark) protocol from the mixer 11, it stores the data string of the digital audio signal for each channel of the received audio data as is in the same channel of the multi-track audio data.

The processing unit 154 also stores the motion data as a data sequence of a digital audio signal in the second channel of the multi-track audio data.

Figure 6 shows an example of a format for storing motion data as a data sequence of digital audio signals. As described above, the digital audio signals of each channel of multi-track audio data are, for example, 24-bit digital audio signals with a sampling frequency of 48 kHz. Figure 5 shows a data sequence of a certain sample in a certain channel of multi-track audio data. The multi-track audio data has a first channel in which the received digital audio signal is stored as a data sequence of digital audio signals as is, and a second channel consisting of a data sequence such as that shown in Figure 6. There can be any number of first channels and any number of second channels. As an example, the processing unit 154 stores the digital audio signals of the audio data received from the mixer 11 in channels 1 to 32 as the first channels, and stores motion data in channels 33 to 64 as the second channels.

As shown in Figure 6, each sample in the second channel consists of 8-bit header information and the motion data itself. The data itself contains a 16-bit data string.

The 8-bit header information includes a start bit, a data size flag, and type data. The start bit is a 1-bit data that indicates whether the data in that sample is the first data. A start bit of 1 indicates that it is the first data. In other words, the samples from one sample whose start bit is 1 to the next sample whose start bit is 1 correspond to one piece of motion data.

The data size flag is 1 bit of data that indicates the data size. For example, if the data size flag is 1, it indicates that one sample contains 8 bits of data, and if the data size flag is 0, it indicates that one sample contains 16 bits of data. For example, if one piece of motion data contains 24 bits of data, the start bit of the first sample will be 1 and the data size flag will be 0. The start bit of the second sample will be 0 and the data size flag will be 1. If one piece of motion data is 8 bits of data, the start bit of each sample will be 0.

Next, type is identification information that indicates the type of data, and is 6-bit data. In this example, type is 6 bits, so it indicates 64 data types. Of course, the number of bits is not limited to 6 bits. The number of bits for type can be set to the number of bits corresponding to the number of types required.

For example, if the type is "01", it indicates motion data. If the type is "02", it indicates other data (e.g., MIDI data). If the type is "00", it indicates idle data (empty data). However, it is preferable not to use "3F", in which all bit data is 1. If the processing unit 154 does not use "3F", the device playing multi-track audio data can determine that some kind of failure has occurred when all bit data is 1. If all bit data is 1, the processing unit 154 does not play the multi-track audio data or output a control signal. This prevents the processing unit 154 from damaging equipment such as lighting when an abnormal signal is output.

When the processing unit 154 sets the type to "00" and all bit data to 0, i.e., idle data (empty data), it is preferable to set the start bit to 1 to prevent all bits from becoming 0. This allows the device that processes multi-track audio data to determine that some kind of error has occurred when all bit data is 0. If all bit data is 0, the processing unit 154 will not play the multi-track audio data. Even in this case, the processing unit 154 will not output an abnormal signal that may cause damage to the device.

The processing unit 154 may also prevent the multi-track audio data from being played back if it contains data that does not match the data format shown in Figure 6. For example, if the data size flag is 1, the lowest 8 bits will all be 0 bits (or all 1 bits). Therefore, even if the data size flag is 1, the processing unit 154 will not play back the multi-track audio data if the lowest 8 bits contain a mixture of 0 and 1 bits. In this case, too, the processing unit 154 will not cause damage to the equipment if an abnormal signal is output.

Note that the processing unit 154 may cause the same channel to contain only data of the same type, or may cause the same channel to contain data of different types. In other words, the processing unit 154 may cause the same channel of the second channels to contain only one piece of identification information, or may cause one channel to contain multiple pieces of identification information.

The processing unit 154 may also store the same type of data across multiple predetermined channels. For example, the processing unit 154 stores data in each of the three channels, 33, 34, and 35, in order. In this case, channels 33, 34, and 35 all contain the same type of bit data. If the data size flags for channels 33, 34, and 35 are all set to 0, the processing unit 154 can store three times as much data (48 bits) in one sample. Alternatively, if the processing unit 154 stores the same type of data in, for example, four channels, it can store four times as much data (64 bits) in one sample. In this way, by storing data across multiple channels, the processing unit 154 can store data of a type that generates a large amount of data per unit time (for example, video data) in one sample.

Furthermore, when storing data across multiple channels, the processing unit 154 may add header information to only one representative channel (e.g., channel 33) and not to the other channels. In this case, the processing unit 154 stores 16 bits of data in the representative channel, and the other channels can store up to 24 bits of data. In other words, the processing unit 154 can store up to 64 bits of data across three channels. In this case, the data size flag for the representative channel may be, for example, 3 bits (bit data indicating 8 types of data sizes) instead of 1 bit.

The data processing device 10 may also receive signal processing parameters indicating the content of signal processing and data related to the basic settings of the mixer 11 from audio equipment such as the mixer 11, and store them as a data stream of a digital audio signal on the second channel.

As mentioned above, in the present invention, the sampling frequency of multi-track audio data is not limited to 48 kHz, and the number of bits is not limited to 24 bits. For example, if the sampling frequency of multi-track audio data is 96 kHz, the processing unit 154 may generate 48 kHz sampling data with invalid data once every two samples. Furthermore, if the number of bits is 32 bits, the processing unit 154 may not use the lowest 8 bits, but may use only the highest 24 bits.

If the multi-track audio data is, for example, 48 kHz and 24 bits, the data string for one sample is, for example, 16 bits. In this case, the multi-track audio data can store a capacity of 16 x 48,000 = 768,000 bits (96,000 bytes) per second.

On the other hand, when bone data is driven at, for example, 60 fps, motion data requires a data capacity of 403,200 bits (50,400 bytes) per second. In other words, multi-track audio data has a bit rate approximately twice that of motion data. Therefore, the data processing device 10 can store motion data at a bit rate lower than that of multi-track audio data. Note that the data processing device 10 may invalidate at least one sample among multiple samples and store data in the samples other than the invalid data. For example, the data processing device 10 may invalidate one sample among two samples and store motion data in the remaining sample. In this case, there is a possibility that the motion data in the second channel may be out of sync by approximately one sample on the playback side. However, the time lag at a sampling frequency of 48 kHz is only 0.0208 msec, and even if there is an lag of several samples, the time lag will be less than 1 msec. Therefore, the viewer will not perceive a lag between the image and sound caused by the motion data.

In this manner, multi-track data recording the live streaming performance is generated. The processing unit 154 outputs the multi-track audio data generated in this manner (S15). The multi-track audio data may be stored in the flash memory 103 of the device itself, or may be distributed to another device via the communication I/F 106.

As described above, the data processing system 1 is installed in the room of a broadcaster who is performing a performance such as a live performance. The data processing device 10 stores the digital audio signal received from the audio equipment during the live performance in a first channel, and stores the motion data in a second channel. The digital audio signal and motion data are generated in accordance with the progress of the same performance such as a live performance. The motion data is stored at the same sampling frequency (e.g., 48 kHz) as the digital audio signal.

The data processing device 10 can therefore synchronize audio signals and motion data without using time codes by generating multi-track audio data in which a digital audio signal with a predetermined sampling frequency (e.g., 48 kHz) is stored in the first channel and motion data with the same frequency (48 kHz) as the first channel is stored in the second channel. This eliminates the need for the data processing system 1 to prepare dedicated recording devices for each protocol used by multiple devices and record each piece of data separately. Furthermore, the data processing system 1 does not require a time code generator, cables, interfaces, etc. to synchronize audio signals and motion data with time codes. Furthermore, the data processing system 1 does not require settings such as matching the time code frame rate for each device.

The second channel shown in this embodiment stores motion data as a data stream of digital audio signals, and therefore conforms to a specific multi-track audio data protocol (e.g., the Dante (registered trademark) protocol). Therefore, the second channel can be distributed and played back as audio data, and can also be edited using an audio data editing application program such as a DAW (Digital Audio Workstation) to perform operations such as copying, cutting, pasting, and timing adjustment. For example, a user can use a DAW to cut and paste audio data from the first and second channels contained in a certain time period to a different time period, thereby moving not only the audio data but also the motion data to a different time period without disrupting synchronization.

Next, Figure 7 is a block diagram showing the configuration of data processing system 1A in a playback environment. Data processing system 1A is installed in a venue, for example, to remotely reproduce a performance such as a live musical performance.

The same components as those in the data processing system 1 shown in Figure 1 are given the same reference numerals and will not be described again. The data processing system 1A is equipped with a display 13. The display 13 may be a panel-type display device such as an LCD, or a video display device such as a projector.

Note that the data processing device 10 at the playback venue does not need to have the same hardware configuration as the venue where the event, such as a live performance, was held.

Figure 8 is a flowchart showing the operation of the processing unit 154 of the data processing device 10 during playback. First, the processing unit 154 accepts multi-track audio data (S21). The multi-track audio data is received from the data processing device 10 at the venue where the live performance is being held, or from a server. Alternatively, the data processing device 10 reads multi-track audio data relating to a past live performance stored in flash memory 103. Alternatively, the data processing device 10 reads multi-track audio data relating to a past live performance stored in another device such as a server.

The processing unit 154 decodes the received multi-track audio data and reproduces the digital audio signal and motion data (S22). For example, the processing unit 154 extracts the digital audio signal of the first channel, channels 1 to 32. The processing unit 154 outputs the extracted digital audio signal to the mixer 11 (S23). The mixer 11 outputs the received digital audio signal to an audio device such as a speaker, and reproduces the singing sound or performance sound.

Furthermore, in the processing of S22, the processing unit 154 reads the 8-bit header information for each sample of channels 33 to 64, which are the second channel, and extracts the 8-bit or 16-bit main body of the motion data. Note that the second channel may also include data related to signal processing parameters and basic settings of the mixer 11. The processing unit may extract this signal processing parameters and data related to basic settings and output it to the mixer 11.

The mixer 11 receives signal processing parameters and setting data from the data processing device 10 and performs various signal processing on the received audio signal based on the signal processing parameters and setting data. As a result, the mixer 11 reproduces singing and performance sounds in the same condition as a live performance.

The processing unit 154 renders an image of the 3D model character based on the extracted motion data (S24). Here, rendering means controlling the 3D model of the character based on the motion data and converting the 3D model into a two-dimensional video signal. As shown in Figure 5, the motion data includes bone data corresponding to the character's movable parts such as the limbs and fingers. The processing unit 154 determines the configuration of the character's 3D model for each frame based on the position coordinate information and rotation coordinate information included in the bone data. The processing unit 154 generates an image of the character as viewed from a specified viewpoint using a predetermined method such as ray tracing or scanline.

The processing unit 154 outputs the generated video signal to the display unit 13 (S25). The display unit 13 displays an image based on the received video signal.

As described above, the data processing device 10 extracts and outputs the digital audio signal of the first channel of each sample at the playback site, and extracts the motion data of the second channel of each sample to render the character image, thereby enabling the digital audio signal and the video signal related to the character image to be played back in sync without using a time code.

Typically, the amount of video data is larger than the amount of audio data. Therefore, if audio data were delayed and synchronized with the video data, the delivery of the audio data would be delayed, resulting in a decrease in real-time performance. However, the motion data shown in this embodiment has a bit rate (403.2 kbps), half the bit rate (768 kbps) of 48 kHz, 24 bits. Therefore, the data processing system of this embodiment can store digital audio signals and motion data within the same frame, without reducing real-time performance. Furthermore, even if motion data is stored across multiple samples, the time lag for one sample at a sampling frequency of 48 kHz is less than 1 msec. Therefore, users of the data processing system of this embodiment can enjoy a new customer experience, perceiving themselves as participating in an event such as a live performance, even when they are in a different venue from the venue where the live performance is taking place.

Furthermore, for example, if the distribution venue is outdoors and video data shot outdoors is played back in an indoor playback venue, the user may feel uncomfortable. However, in order to distribute motion data, the data processing system of this embodiment can, for example, film the inside of the venue with a camera installed at the destination venue and composite video of the character onto the video of the venue. Therefore, the data processing system of this embodiment can play back video that is optimal for the playback environment. This allows users to enjoy a new customer experience in which they can feel an immersive feeling that was previously not possible.

Note that the process of outputting the digital audio signal at S23 and the process of outputting the video signal at S25 shown in Figure 8 may be performed simultaneously, or either may be performed first.

(Variation 1)
The motion data of the first modification includes main motion data, which is data on the movement of the main body part of the character, and sub-motion data, which is data on the movement of the body parts subordinate to the main body part.

The main parts of a character are at least the upper arms and forearms, which are important parts for conveying the performance. Subordinate parts are parts that are not directly necessary for conveying the performance, such as the torso, legs, hands, and fingers. However, the main and subordinate parts differ depending on the type of performance. For example, when playing the guitar, the movement of the forearm of the right arm and the movement of the forearm and fingers of the left arm are important parts for conveying the performance, but when playing the piano, the forearms and movements of both arms are important parts.

The multi-track audio data of variant 1 has a first channel that stores a digital audio signal, a second channel that stores main motion data, and a third channel that stores a sub-motion channel.

The processing unit 154 of the data processing device 10 may determine whether or not to play sub-motion data for the third channel during playback depending on the processing power, the usage rate of the CPU 104, or the available capacity of the RAM 105. As a result, even if the processing power, usage rate of the CPU 104, or the available capacity of the RAM 105 is low, the data processing device 10 will not stop or delay playback, and real-time performance will not be compromised. Furthermore, even if the data processing device 10 does not play sub-motion data, it will play main motion data, which is an important part of conveying the performance, so the user's sense of immersion will not be hindered.

(Variation 2)
The data processing device 10 according to the second modification accepts the designation of a character to be played back, and uses the extracted motion data to render an image of the designated character.

For example, if the first venue to which distribution is to be made is in the Northern Hemisphere and it is winter, the data processing device 10 at the first venue will accept the specification of a character wearing winter clothing. On the other hand, if the second venue to which distribution is to be made is in the Southern Hemisphere and it is summer, the data processing device 10 at the second venue will accept the specification of a character wearing summer clothing.

This allows the data processing device 10 of variant 2 to play back video that is more optimal for the playback environment.

(Variation 3)
The data processing device 10 according to the third modification further corrects the extracted motion data based on the specified character in the second modification, and renders an image of the specified character using the corrected motion data.

The size of the 3D model data may vary depending on the character. Therefore, the data processing device 10 corrects the length of each bone data included in the motion data based on the specified character data (for example, data indicating height).

This allows the data processing device 10 of variant 3 to play more optimal video.

(Variation 4)
The data processing device 10 according to the fourth modification accepts the designation of a background image, and superimposes the rendered image of the character on the designated background image.

As mentioned above, for example, if the distribution venue is outdoors and video data shot outdoors is played back at an indoor playback venue, the user may feel uncomfortable. However, data processing device 10 of variant 4 superimposes (combines) the rendered character video onto the specified background video. For example, if the playback venue is indoors, data processing device 10 of variant 4 may superimpose an indoor background video. Alternatively, data processing device 10 of variant 4 may superimpose a summer background video if the playback venue is summer, or a winter background video if the playback venue is winter.

Therefore, the data processing device 10 of variant 4 can play back video that is optimal for the playback environment. This allows users to have a new customer experience in which they can feel an immersive feeling that was previously not possible.

The description of this embodiment should be considered illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not the above-described embodiments. Furthermore, the scope of the present invention includes equivalents to the claims. For example, in the above embodiment, the processing unit of the present invention is configured by a program read by the CPU 104, but it is also possible to realize the processing unit of the present invention using, for example, an FPGA (Field-Programmable Gate Array).

The data processing system shown in this embodiment can also be applied to systems that require a combination of video and audio, such as theme parks. Alternatively, the data processing system shown in this embodiment can also be applied to a remote session system. In a remote session, it is important to convey not only the sound but also the movements of the performer or singer. However, as mentioned above, the amount of video data is greater than the amount of audio data, so transmitting the audio data and video data separately results in significant delays in the video data. Furthermore, transmitting the audio data and video data synchronously reduces real-time performance. In contrast, the data processing system shown in this embodiment sends and receives motion data, which has a smaller amount of data than video data, and can therefore convey the movements of a remote performer or singer with low delay without reducing real-time performance.

1: Data processing system, 1A: Data processing system, 10: Data processing device, 11: Mixer, 12: Motion capture, 13: Display, 101: Display, 102: User I/F, 103: Flash memory, 104: CPU, 105: RAM, 106: Communication I/F, 154: Processing unit

Claims (9)

A data processing method for generating multi-track audio data storing audio data of multiple channels including at least a first channel and a second channel, comprising:
storing a data string of a digital audio signal in the first channel;
storing motion data, which is information on character movement and is associated with the digital audio signal, as a data string of the digital audio signal in the second channel;
Data processing methods. Distributing the multi-track audio data.
The data processing method according to claim 1 . A data processing method for reproducing multi-track audio data that stores audio data of multiple channels, comprising:
Reproducing the data string stored in the first channel as a digital audio signal;
extracting the data string stored in the second channel as motion data, which is information on character movement associated with the digital audio signal, and rendering an image of the character using the extracted motion data;
Data processing methods. the second channel includes identification information of the motion data;
The data processing method according to any one of claims 1 to 3. the motion data includes main motion data, which is data on the movement of a main body part of the character, and sub-motion data, which is data on the movement of a body part subordinate to the main body part;
the audio data further comprises a third channel in which the sub-motion data is stored;
The data processing method according to any one of claims 1 to 3. Accepts the specification of the character to be played,
Rendering an image of the specified character using the extracted motion data.
The data processing method according to claim 3 . correcting the extracted motion data based on the specified character;
Rendering an image of the designated character using the corrected motion data;
7. The data processing method according to claim 6. Accepts the background video specification,
superimposing the rendered image of the character on the specified background image;
The data processing method according to claim 3 . A data processing method for generating multi-track audio data storing audio data of multiple channels including at least a first channel and a second channel, comprising:
storing a data string of a digital audio signal in the first channel;
storing motion data, which is information on character movement and is associated with the digital audio signal, as a data string of the digital audio signal in the second channel;
A data processing device having a processing unit.