JP2014090251A - Information processing system, information processing program, information processing control method and information processing device - Google Patents

Abstract

To provide an information processing system capable of providing a new user experience with a more realistic viewing effect for music played in a virtual space.
In an information processing system including an information processing unit, a first speaker, and a second speaker, predetermined music data composed of two channels is converted into first channel music data and second channel music data. To separate. Further, a first sound source object associated with one of the separated music data, a second sound source object associated with the other, and a virtual microphone are arranged in the virtual three-dimensional space. Based on the orientation of the virtual microphone and the positional relationship between the first sound source object and the second sound source object, the reproduction volume of the sound reproduced from the first speaker and the sound reproduced from the second speaker is determined. To do.
[Selection] Figure 16

Description

  The present invention relates to an information processing system, an information processing program, an information processing control method, and an information processing apparatus, and more specifically, an information processing system including an information processing unit, a first speaker, and a second speaker, and information The present invention relates to a processing program, an information processing control method, and an information processing apparatus.

  2. Description of the Related Art Conventionally, a game system using a general television device (first video output device) and a controller (second video output device) having a display unit capable of outputting video, which is different from this, is known. ing. In such a game system, for example, a first game video is displayed on the television device, and a second game video different from the first game video is displayed on the display unit of the controller, thereby enabling a new game. Had proposed.

JP 2012-135337 A

  However, the above proposal mainly focuses on what kind of video is displayed and how these are displayed in association with the game processing. For this reason, no particular mention or suggestion was made regarding processing related to voice.

  Therefore, an object of the present invention is to provide an information processing system and the like that can provide a new user experience with a more realistic viewing effect for music reproduced from a sound source in a virtual space.

  The above object is achieved by, for example, the following configuration.

  An example of the configuration is an information processing system including an information processing unit, a first speaker, and a second speaker, and includes a music data separating unit, a sound source object arranging unit, a virtual microphone arranging unit, and an audio output. Control means. The music data separation means separates predetermined music data composed of two channels into first channel music data and second channel music data. The sound source object placement means is a virtual three-dimensional representation of a first sound source object associated with one of the separated music data and a second sound source object associated with the other of the separated music data. Place them at different positions in the space. The virtual microphone arrangement means arranges the virtual microphone at a predetermined position in the virtual three-dimensional space. The audio output control means is a first channel reproduced from the first speaker based on the orientation of the virtual microphone arranged in the virtual three-dimensional space and the positional relationship between the first sound source object and the second sound source object. The reproduction volume of the music data and the second channel music data and the reproduction volume of the first channel music data and the second channel music data reproduced from the second speaker are determined.

  According to the above configuration example, it is possible to provide a user experience with a higher sense of realism regarding the reproduction of BGM emitted from the sound source in the virtual space.

  The information processing system may further include a first output device, posture detection means, and virtual microphone posture change means. The first output device may have a housing provided with a motion sensor, a first speaker, a second speaker, and a first display unit on which a state of the virtual three-dimensional space is displayed. Further, the posture detecting means may detect the posture of the first output device itself in real time or almost real time based on the output of the motion sensor. The virtual microphone posture changing means may change the direction of the virtual microphone according to the posture of the first output device.

  According to the above configuration example, in a game or the like in which the display of the virtual space displayed on the first display unit is changed by the player changing the posture of the first output device itself having two speakers, the sound source object Reproduction with a higher sense of presence is possible for the BGM emitted by.

  The information processing system may further include a speaker positional relationship recognition unit that recognizes the positional relationship between the first speaker and the second speaker based on the attitude of the first output device. Then, the audio output control means further includes the first speaker and the second speaker in addition to the positional relationship between the direction of the virtual microphone arranged in the virtual three-dimensional space and the first sound source object and the second sound source object. Based on the positional relationship, the reproduction volume of the first channel music data and the second channel music data output from the first speaker, and the first channel music data and the second channel output from the second speaker. You may make it determine the reproduction volume of music data.

  According to the above configuration example, it is possible to express the sound expressed by the sound source object in the virtual space reflecting the positional relationship of the speakers in the real space.

  The information processing system may further include a second output device having a third speaker, a fourth speaker, and a second display unit. Then, the audio output control unit is configured based on the positional relationship between the first speaker and the second speaker included in the first output device and the third speaker and the fourth speaker included in the second output device. The playback volume from the speaker may be determined.

  According to the above configuration example, for example, the first set of speakers according to the first output device that can be used as a game controller and the second set of speakers according to the second output device that can be used as a monitor are used. This makes it possible to output audio with a higher sense of presence. For example, the sound output in the vertical direction as viewed from the player is assigned to the speaker related to the first output device, and the sound output related to the left and right direction is assigned to the speaker related to the second output device, thereby expanding the space in the virtual space. It becomes possible for the player to experience the spatial sensation.

  The music data may be music data composed of two stereo channels. Furthermore, the first speaker and the second speaker may be a pair of stereo speakers.

  According to the above configuration example, the expression of music emitted from the sound source object arranged along the left-right direction in the virtual space can be expressed with a higher sense of realism without giving the player a sense of incongruity.

  ADVANTAGE OF THE INVENTION According to this invention, the audio | voice output which raised the realism more can be performed regarding the audio | voice which the sound source object which exists in virtual space utters.

1 is an external view showing an example of a game system 1 according to an embodiment of the present invention. Functional block diagram showing an example of the game apparatus body 5 of FIG. The figure which shows an example of an external appearance structure of the terminal device 6 of FIG. The block diagram which shows an example of an internal structure of the terminal device 6 Figure showing an example of the game screen Figure showing an example of the game screen Figure showing an example of the game screen Figure showing an example of the game screen Figure showing an example of the game screen Figure showing an example of the game screen Figure showing an example of the game screen Memory map of memory 12 Example of configuration of terminal operation data 83 Example of configuration of sound source object data 88 A flowchart showing a flow of game processing based on the game processing program 81 The flowchart which shows the detail of the initial setting process of FIG. The flowchart which shows the detail of the game sound production | generation process of FIG. The flowchart which shows the flow of the control processing of the terminal device 6

  A game system according to an embodiment of the present invention will be described with reference to FIG.

  In FIG. 1, a game system 1 includes a home television receiver (hereinafter referred to as a monitor) 2 which is an example of display means, and a stationary game apparatus 3 connected to the monitor 2 via a connection cord. Composed. The monitor 2 includes speakers 2L and 2R, which are 2ch stereo speakers. The game apparatus 3 includes a game apparatus body 5 and a terminal device 6.

  The monitor 2 displays a game image output from the game apparatus body 5. The monitor 2 has a left speaker 2L and a right speaker 2R. The speakers 2L and 2R output game sound output from the game apparatus body 5. In this example, the monitor 2 has these speakers. However, an external speaker may be separately connected to the monitor 2.

  The game apparatus body 5 executes a game process or the like based on a game program or the like stored on an optical disc that can be read by the game apparatus body 5.

  The terminal device 6 is an input device that can be held by a user. The user can use the terminal device 6 by holding it in his hand or by placing the terminal device 6 in a free position. The terminal device 6 includes an LCD (Liquid Crystal Display: liquid crystal display device) 21 which is a display means, speakers 23L and 23R which are stereo speakers of 1 and 2ch (hereinafter may be collectively referred to as the speaker 23), a headphone terminal, and Input means (analog stick, push button, touch panel, etc.) are provided. The terminal device 6 and the game apparatus body 5 can communicate wirelessly (may be wired). The terminal device 6 receives data of an image (for example, a game image) generated by the game device body 5 from the game device body 5 and displays an image indicated by the data on the LCD 21. Further, the terminal device 6 receives data of sound (for example, game sound effects and BGM) generated by the game apparatus body 5 from the game apparatus body 5, and outputs the sound indicated by the data from the speaker 23. In addition, the terminal device 6 transmits operation data representing the content of the operation performed on the own device to the game apparatus body 5.

  FIG. 2 is a block diagram of the game apparatus body 5. In FIG. 2, the game apparatus body 5 is an example of an information processing apparatus. In the present embodiment, the game apparatus body 5 includes a CPU (control unit) 11 and a memory 12, a system LSI 13, a wireless communication unit 14, an AV-IC (Audio Video-Integrated Circuit) 15, and the like.

  The CPU 11 executes a predetermined information processing program using the memory 12, the system LSI 13, and the like. Thereby, various functions (for example, game processing) in the game apparatus 3 are realized.

  The system LSI 13 includes a GPU (Graphics Processor Unit) 16, a DSP (Digital Signal Processor) 17, an input / output processor 18, and the like.

  The GPU 16 generates an image in accordance with a graphics command (drawing command) from the CPU 11. In the present embodiment, the game apparatus body 5 may generate both a game image to be displayed on the monitor 2 and a game image to be displayed on the terminal device 6. Hereinafter, a game image displayed on the monitor 2 may be referred to as a “monitor game image”, and a game image displayed on the terminal device 6 may be referred to as a “terminal game image”.

  The DSP 17 functions as an audio processor, and generates sound data using sound data and sound waveform (tone color) data stored in the memory 12. In the present embodiment, the game sound is output from the speakers 2L and 2R of the monitor 2 and the speaker 23 of the terminal device 6 (or headphones connected to the terminal device 6) as in the case of the game image. Both of the game sound to be generated may be generated. Hereinafter, the game sound output from the monitor 2 may be referred to as “monitor game sound”, and the game sound output from the terminal device 6 may be referred to as “terminal game sound”.

  The input / output processor 18 transmits / receives data to / from the terminal device 6 via the wireless communication unit 14. In this embodiment, the input / output processor 18 transmits the game image (terminal game image) generated by the GPU 16 and the game sound (terminal game sound) generated by the DSP 17 via the wireless communication unit 14. It transmits to the terminal device 6. At this time, the terminal game image may be compressed and transmitted so that the display image is not delayed. Further, the input / output processor 18 receives operation data or the like transmitted from the terminal device 6 via the wireless communication unit 14 and stores (temporarily stores) it in the buffer area of the memory 12.

  Of the images and sounds generated in the game apparatus body 5, image data and sound data output to the monitor 2 are read by the AV-IC 15. The AV-IC 15 outputs the read image data to the monitor 2 via an AV connector (not shown), and outputs the read audio data to the speaker 2 a built in the monitor 2. As a result, an image is displayed on the monitor 2 and sound is output from the speaker 2a.

  FIG. 3 is a diagram illustrating an example of an external configuration of the terminal device 6. As shown in FIG. 3, the terminal device 6 includes a substantially plate-shaped housing 20. The housing 20 has a size (shape) that the user can hold with both hands or one hand. The terminal device 6 includes an LCD 21 that is an example of a display unit. The terminal game image is displayed on the LCD 21.

  In addition, the terminal device 6 includes a speaker 23. The speaker 23 is a stereo speaker. From the speaker 23, the terminal game sound is output. Further, the terminal device 6 includes a headphone terminal 24 to which predetermined headphones can be attached and detached. Here, the terminal device 6 outputs a sound from the speaker 23 when no headphones are connected to the headphone terminal, but outputs a sound from the speaker 23 when a headphone is connected to the headphone terminal. Not performed. That is, in the present embodiment, the simultaneous output of both is not performed, and the output from the speaker 23 and the output from the headphones are in an exclusive relationship (however, in other embodiments, simultaneous output may be possible. ).

  The terminal device 6 includes a touch panel 22. The touch panel 22 is an example of a position detection unit that detects a position input to a predetermined input surface (screen of the display unit) provided in the housing 20. Furthermore, the terminal device 6 includes an analog stick 25, a cross key 26, a button 27, and the like as an operation unit (the operation unit 31 shown in FIG. 4).

  FIG. 4 is a block diagram showing an electrical configuration of the terminal device 6. As illustrated in FIG. 4, the terminal device 6 includes the above-described LCD 21, touch panel 22, speaker 23, and operation unit 31. A headphone can be connected via the headphone terminal 24. In addition, a motion sensor 32 for detecting the attitude of the terminal device 6 is provided. In the present embodiment, an acceleration sensor and a gyro sensor are provided as the motion sensor 32. The acceleration sensor can detect xyz triaxial acceleration. The gyro sensor can detect the three-axis angular velocity of xyz.

  Further, the terminal device 6 includes a wireless communication unit 34 capable of wireless communication with the game apparatus body 5. In the present embodiment, wireless communication is performed between the terminal device 6 and the game apparatus body 5, but in other embodiments, communication may be performed by wire.

  The terminal device 6 includes a control unit 33 that controls the operation of the terminal device 6. Specifically, the control unit 33 receives output data from each input unit (touch panel 22, operation unit 31, motion sensor 32) and transmits it as operation data to the game apparatus body 5 via the wireless communication unit 34. In addition, when the terminal game image from the game apparatus body 5 is received by the wireless communication unit 34, the control unit 33 performs appropriate processing (for example, decompression processing when image data is compressed). ) To display an image from the game apparatus body 5 on the LCD 21. Furthermore, when the terminal game sound from the game apparatus body 5 is received by the wireless communication unit 34, the control unit 33 outputs the terminal game sound to the speaker 23.

  Next, an overview of processing executed in the system of the present embodiment will be described with reference to FIGS.

  The processing executed in the present embodiment provides a new way of playing music emitted from a sound source object existing in a virtual three-dimensional space (hereinafter simply referred to as a virtual space). The sound source object is an object defined as an object capable of emitting a predetermined sound.

  As an example of the processing of the present embodiment, the following game processing is assumed. The game realized by this game process is a game in which a player character can freely move in a virtual space imitating a theme park. A plurality of “attractions” are set in the virtual space, and different minigames can be played for each attraction.

  FIG. 5 is an example of a game screen displayed on the terminal device 6. In FIG. 5, a player character 101 and an entrance object 102 that is an entrance of a predetermined attraction are displayed. The player character 101 is displayed as seen from the back side. In addition, a left speaker image 103L and a right speaker image 103R are displayed on the entrance object 102 (hereinafter, collectively referred to as a speaker image 103).

  Here, in this game, with respect to display of the game screen, the coordinate system of the real space and the coordinate system of the virtual space are always displayed so as to coincide with each other. In other words, the gravitational direction and the ground in the virtual space always intersect each other vertically. In addition, the terminal device 6 includes the motion sensor 32 as described above. By using this, the attitude of the terminal device 6 itself can be detected. In this game, by tilting the virtual camera simultaneously with the attitude of the terminal device 6, the terminal device 6 can be handled as if it were a “view window” looking into the virtual space. For example, assume a posture in which the LCD 21 of the terminal device 6 is held so as to face the front of the player's face. At this time, it is assumed that a state in the positive direction of the Z axis in the virtual space is displayed on the LCD 21. From this state, it is assumed that the player turns 180 degrees and faces directly behind. Then, the state of the Z axis negative direction in the virtual space is displayed on the LCD 21.

  By the way, there is a case where a sound is played in a game in which the virtual space as described above is expressed. There are roughly two types of sound, “sound effect” and “music (BGM)”. As a method of playing these sounds, conventionally, for sound effects, a virtual microphone is installed in the virtual space, and sound effects are calculated by calculating the sound field (volume and localization) considered to be viewed with this virtual microphone. Was. On the other hand, music, particularly BGM, is generally played back at a constant volume and localization regardless of the position of the virtual camera or virtual microphone. That is, regardless of the state of the virtual space, the BGM is reproduced “outside” the virtual space.

  Here, it is assumed that it is desired to perform an expression as if the BGM is being reproduced in the “inside” virtual space. In particular, it is assumed that this BGM is a stereo sound, that is, a BGM composed of LR stereo 2ch. In this case, a sound source object that emits sound based on the BGM data configured by the 2ch is arranged in the virtual space and reproduced. Then, it is conceivable to perform processing for regarding that sound as viewed with the virtual microphone. For example, in FIG. 5, consider a case where it is desired to perform an expression such that BGM flows from the speaker image 103 of the entrance object 102. In this case, as shown in FIG. 6, a sound source object (this is a transparent object that cannot be seen by the player) is arranged approximately in the middle between the two speaker images 103. Then, a process of emitting (reproducing) BGM composed of 2ch from the sound source object is performed. Further, a process of picking up the emitted voice with a virtual microphone is performed. Here, it is assumed that the virtual microphone is installed at the same position as the player character 101. That is, it is assumed that the sound source object is arranged almost in front of the sound source object. As a result, the sound applied to the L channel of the BGM data is output from the speaker 23L, and the sound applied to the R channel of the BGM data is output from the speaker 23R. In other words, in the state of FIG. 6, the left / right positional (localization) relationship of the LR channel of the sound source object is the same as the left / right positional relationship of the speakers 23 </ b> L and 23 </ b> R of the terminal device 6.

  Next, it is assumed that the player character 101 is rotated 180 degrees from the state shown in FIG. For example, this is a case where the player rotates 180 degrees and turns to the back while holding the terminal device 6 with both hands so that the LCD 21 of the terminal device 6 faces the front of the player's face. FIG. 7 is a diagram showing a screen in a state rotated 180 degrees from the state of FIG. With the rotation of the player (terminal device 6), the orientations of the player character 101, the virtual camera, and the virtual microphone are also rotated by 180 degrees. In this case, the positional relationship in the virtual space is such that the sound source object is positioned almost directly behind the player character 101 (virtual microphone). Further, the left speaker image 103L is positioned to the right rear of the player character 101, and the right speaker image 103R is positioned to the left rear of the player character 101. On the other hand, the localization of the BGM emitted from the sound source object remains maintained. That is, as in the case of FIG. 6 described above, the BGM L channel sound is output from the speaker 23L, and the BGM R channel sound is output from the speaker 23R. From the player's perspective, the LR channel is heard in reverse from the state shown in FIG. Considering the positional relationship when the player character 101 rotates 180 degrees, the sound of the L channel in the state of FIG. 6 is output from the speaker 23R in the state of FIG. 7, and the sound of the R channel is in the state of FIG. The sound output from the speaker 23L is more natural.

  Therefore, in the present embodiment, processing is performed so that the BGM can be heard naturally even when the posture change (positional change) as described above occurs. Specifically, in this embodiment, for the BGM data to be reproduced, first, the L channel and the R channel are separated as different sound data. Then, two sound source objects are arranged at different positions, L channel sound data (hereinafter referred to as L channel data) is assigned to one, and R channel sound data (hereinafter referred to as R channel data) is assigned to the other and played back. . That is, instead of arranging a sound source object for reproducing one BGM data including the LR channel as described above, another sound source object is set and arranged for each LR channel. And the process of adjusting the output sound from the speakers 23L and 23R is performed according to the direction of the virtual microphone that picks up the sound emitted from the sound source object and the positional relationship between the two sound source objects. Specifically, in the local coordinate system based on the virtual microphone (hereinafter referred to as the virtual microphone coordinate system), whether each of the two sound source objects is positioned on the positive x-axis direction or the negative x-axis direction Determine. This makes it possible to determine whether the positions of the two sound source objects are right or left when viewed from the virtual microphone. Based on this positional relationship, the reproduction volume of the L channel data and R channel data in each of the speakers 23L and 23R is determined.

  FIG. 8 is a diagram illustrating an example of the arrangement of the game screen and the sound source object in the present embodiment. In FIG. 8, the sound source object 105L is arranged at a position so as to overlap the left speaker image 103L, and the sound source object 105R is arranged at a position so as to overlap the right speaker image 103R. These sound source objects are transparent objects that are not visible to the player. The sound source object 105L is associated with L channel data of BGM to be reproduced. On the other hand, BGM R channel data to be reproduced is associated with the sound source object 105R. That is, the sound source object 105L is set to reproduce L channel data, and the sound source object 105R is set to reproduce R channel data. Similarly to the above, a virtual microphone is installed at the position of the player character, and the virtual microphone faces the front direction (z-axis positive direction). When the BGM is reproduced in such a state, the sound emitted from the sound source object 105L is mainly heard from the speaker 23L (sounds louder than the speaker 23R). Further, the sound emitted from the sound source object 105R is mainly heard from the speaker 23R (sounds louder than the speaker 23L).

  On the other hand, it is assumed that the player (terminal device 6) rotates 180 degrees as described above from the state of FIG. FIG. 9 is a diagram showing a game screen at this time. In this state, the sound source object 105L is located at the right rear of the player character 101 (virtual microphone), and the sound source object 105R is located at the left rear of the player character 101. Further, as the orientation of the terminal device 6 changes, the front direction of the virtual microphone also faces the negative z-axis direction of the virtual space. That is, when the virtual microphone is used as a reference, the sound source object 105L is on the right side and the sound source object 105R is on the left side. When the BGM is reproduced in such a state, the sound emitted from the sound source object 105L is mainly heard from the speaker 23R, and the sound emitted from the sound source object 105R is mainly heard from the speaker 23L. Become. That is, the BGM is reproduced so that the position and orientation of the player character (virtual microphone) and the positional relationship between the two sound source objects are reflected. Thereby, BGM reproduction from a sound source existing in the virtual space can be expressed more accurately, and BGM reproduction that does not give the player a sense of incongruity can be realized.

  As described above, in the present embodiment, since the reproduction process is performed so as to reflect the position of the sound source object in the virtual space, for example, the following audio reproduction is possible. For example, as shown in FIG. 10, it is assumed that sound source objects 105L and 105R exist in the front direction and upward of the player character 101, and BGM is being reproduced. In FIG. 10, only the lower half of the sound source objects 105L and 105R is in the screen. In this state, when considered in the microphone coordinate system, the sound source object is located above the virtual microphone (in the positive y-axis direction of the space coordinate system of the virtual space). In this case, the sound volume output from the speakers 23L and 23R is substantially the same.

  Next, it is assumed that the terminal device 6 is rotated 90 degrees to the left as shown in FIG. 11 from the state of FIG. 10 (for example, when the player holds the terminal device 6 vertically). In this case, paying attention to the positional relationship between the speakers 23L and 23R, the positional relationship changes from the state in which the speakers 23L and 23R are disposed in the vertical direction to the positional relationship in which they are disposed in the vertical direction. Further, along with the change in the attitude of the terminal device 6, the attitude of the virtual microphone is also rotated 90 degrees to the left (that is, the attitude of the terminal device 6 and the attitude of the virtual microphone are linked). As a result, when considered in the microphone coordinate system, the sound source objects 105L and 105R are located on the right side of the virtual microphone. In such a case, in the present embodiment, the volume from the speaker 23R, that is, the upper speaker in the real space is relatively increased, and the volume from the speaker 23L, that is, the lower speaker in the real space, is relatively increased. Output with a smaller size. In other words, the left / right balance of the volume of the speakers 23L and 23R is adjusted. Thereby, the positional relationship of the sound source object in the vertical direction (y-axis direction) in the virtual space can also be reflected in the audio output. In this example, an example of the vertical direction is given. For example, also in the depth direction (the z-axis direction of the virtual space), the positional relationship in this direction can be reflected in the output volume from the speakers 23L and 23R. For example, when the orientation of the terminal device 6 is such that the LCD 21 faces the upper surface, audio output reflecting the positional relationship in the depth direction is possible.

  Next, the operation of the system 1 for realizing the game process as described above will be described in detail with reference to FIGS.

  FIG. 12 shows an example of various data stored in the memory 12 of the game apparatus body 5 when the game is executed.

  The game processing program 81 is a program for causing the CPU 11 of the game apparatus body 5 to execute a game process for realizing the game. The game processing program 81 is loaded into the memory 12 from an optical disc, for example.

  The processing data 82 is data used in the game process executed by the CPU 11. The processing data 82 includes terminal operation data 83, terminal transmission data 84, game sound data 85, terminal device attitude data 86, virtual microphone attitude data 87, sound source object data 88, and the like.

  The terminal operation data 83 is operation data periodically transmitted from the terminal device 6. FIG. 13 is a diagram illustrating an example of the configuration of the terminal operation data 83. The terminal operation data 83 includes operation button data 91, touch position data 92, motion sensor data 93, and the like. The operation button data 91 is data indicating an input state with respect to the operation unit 31 (the analog stick 25, the cross key 26, and the button 27). Further, the operation button data 91 also includes the input content for the motion sensor 32. The touch position data 92 is data indicating a position (touch position) where an input is performed on the input surface of the touch panel 22. The motion sensor data 93 is data indicating the acceleration and angular velocity detected by the acceleration sensor and the angular velocity sensor included in the motion sensor.

  Returning to FIG. 12, the terminal transmission data 84 is data periodically transmitted to the terminal device 6. The terminal transmission data 84 includes the terminal game image and the terminal game sound.

  The game sound data 85 is data that becomes the basis of the terminal game sound and the monitor game sound. The game sound data 85 includes sound effect sound data and a plurality of BGM data composed of LR2ch as described above.

  The terminal device attitude data 86 is data indicating the attitude of the terminal device 6. The virtual microphone attitude data 87 is data indicating the attitude of the virtual microphone. These posture data are shown as combinations of three-axis vector data, for example. The virtual microphone attitude data 87 includes the attitude data of the terminal virtual microphone and the attitude data of the monitor virtual microphone. However, in the following description, the “virtual microphone attitude data 87” simply indicates the attitude data of the terminal virtual microphone.

  The sound source object data 88 is data related to the sound source object. A plurality of sound source object data 88 is stored in the memory 12. FIG. 14 is a diagram showing an example of the configuration of each sound source object data 88. The sound source object data 88 includes an object ID 881, set target identification data 882, corresponding BGM identification data 883, assigned channel identification data 884, and separated BGM data 885.

  The object ID 881 is an ID for identifying each sound source object. The set target identification data 882 is data indicating a sound source object paired with itself. In the present embodiment, one BGM data is separated into an L channel and an R channel and assigned to two sound source objects. Therefore, these two sound source objects are handled as a set of sound source objects (hereinafter referred to as set type sound source objects). The set target identification data is data indicating the other sound source object paired with itself. For example, the object ID 881 of the partner to be paired is shown as the data. In addition, in the case of a sound source object that does not form such a set (for example, a sound source object that emits only sound effects; hereinafter referred to as a single-type sound source object), information indicating that is set as the set target identification data 882. Is done.

  Corresponding BGM identification data 883 is data indicating BGM data defined as BGM to be reproduced by the sound source object. For example, information indicating the BGM data (BGM data composed of 2ch) included in the game sound data 85 is the main data.

  The assigned channel identification data is data indicating which one of the L channel and the R channel is assigned when the sound source object is a set-type sound source object.

  The post-separation BGM data 885 is one of the data after the BGM data configured by LR2ch is separated into sound data of the L channel and the R channel.

  In addition, although not shown in the figure, the sound source object emits information such as information indicating whether the sound source object is generating sound, information defining the sound volume of the sound generated by the sound source object, the directivity of the sound, or the like. Information about audio is also included as appropriate.

  Next, the flow of game processing executed by the CPU 11 of the game apparatus body 5 based on the game processing program 81 will be described with reference to the flowcharts of FIGS.

  In FIG. 15, when the execution of the game processing program 81 is started, in step S1, the CPU 11 performs an initial setting process. In this initial setting process, initial setting of various objects such as the sound source object, arrangement in the virtual space, and initialization of the attitude of the virtual camera and the virtual microphone (virtual microphone attitude data 87) are executed. FIG. 16 is a flowchart showing details of the initial setting process. In the initial setting process, first, a set type sound source object setting process is performed. That is, in step S41, the CPU 11 extracts the set type sound source object from the sound source objects.

  Next, in step S42, the CPU 11 selects a set of set-type sound source objects to be processed from the extracted set-type sound source objects. Hereinafter, this is referred to as a processing target object.

  Next, in step S <b> 43, the CPU 11 acquires BGM data to be reproduced by the processing target from the game sound data 85 based on the corresponding BGM identification data 883. In subsequent step S44, the CPU 11 generates the L channel data and the R channel data based on the acquired BGM data.

  Next, in step S45, the CPU 11 assigns the generated L channel data and R channel data to the processing target objects (two sound source objects) one by one. That is, first, the CPU 11 refers to the assigned channel identification data 884 of one sound source object among the processing target objects. Then, the L channel data or the R channel data corresponding to the channel indicated therein is stored in the BGM data 885 after separation. Similarly, the remaining L channel data or R channel data is stored in the BGM data 885 after separation of the other sound source object of the processing target objects. As a result, two sound source objects are set for a BGM: a sound source object that reproduces only the sound of the L channel and a sound source object that reproduces only the sound of the R channel.

  Next, in step S46, the CPU 11 places the processing target object at a predetermined position in the virtual space. Note that this arrangement position is such that the two sound source objects are separated from each other to the right and left by a certain amount (that is, a position relationship that provides a stereo effect), such as the sound source objects 105L and 105R in FIG. It is preferable to arrange so that.

  Next, in step S47, the CPU 11 determines whether or not the above processing has been performed for all the set-type sound source objects extracted in step S41. As a result, when an unprocessed set-type sound source object still remains (NO in step S47), the process returns to step S42 and the process is repeated. On the other hand, if all the processes have been completed (YES in step S47), in the subsequent step S48, the CPU 11 executes other initial setting processes. That is, various objects (single sound source objects, player characters, etc.) are arranged in the virtual space, the postures of the virtual camera and the virtual microphone, and other various variables are initialized. Note that the initial values of the virtual camera and the virtual microphone are in a state in which the directions of the respective axes of the microphone coordinate system are made coincident with the directions of the respective axes of the spatial coordinate system in the virtual three-dimensional space. This completes the initial setting process.

  Returning to FIG. 15, next, in step S <b> 2, the CPU 11 acquires the terminal operation data 83.

  Next, in step S <b> 3, the CPU 11 calculates the current posture of the terminal device 6 based on the motion sensor data 93 (acceleration data and angular velocity data). Data indicating the calculated attitude is stored in the memory 12 as terminal apparatus attitude data 86.

  Next, in step S4, the CPU 11 reflects the current posture of the terminal device 6 in the posture of the virtual microphone (terminal virtual microphone). Specifically, the attitude indicated by the terminal apparatus attitude data 86 is reflected in the virtual microphone attitude data 87.

  Next, in step S <b> 5, the CPU 11 executes a predetermined game process based on the operation content indicated by the terminal operation data 83 (mainly the operation content indicated by the operation button data 91 and the touch position data 92). For example, a process of moving various characters such as a player character is performed.

  Next, in step S6, the CPU 11 executes a process for generating a game image in which the result of the game process is reflected. For example, a game image is generated by photographing a virtual game space after the player character has moved based on the operation content with a virtual camera. At this time, the CPU 11 appropriately generates two images, a monitor game image and a terminal game image, according to the game content. For example, each image is generated by using two virtual cameras.

  Next, in step S7, the CPU 11 executes a game sound generation process for generating a monitor game sound and a terminal game sound. FIG. 17 is a flowchart showing details of the game sound generation processing shown in step S7. In FIG. 17, first, in step S21, the CPU 11 selects one sound source object to be processed. This is because when a plurality of sound source objects exist in the virtual space, these are processed one by one in order. Further, the sound source object to be processed is, for example, a sound source object in a state where sound is currently being generated.

  Next, in step S22, the CPU 11 calculates the position and orientation of the processing-target sound source object in the microphone coordinate system. Thereby, in the microphone coordinate system, it is possible to recognize whether the sound source object is located on the right side or the left side of the virtual microphone (based on the front direction).

  Next, in step S23, the CPU 11 calculates a linear distance from the virtual microphone to the sound source object in the microphone coordinate system. In subsequent step S24, the CPU 11 determines the volume values of the speakers 23L and 23R based on the calculated position, orientation, distance, and orientation of the virtual microphone itself in the microphone coordinate system. That is, the left and right volume balance of the speakers 23L and 23R is determined.

  Next, in step S25, the CPU 11 reproduces the game sound data 85 associated with the sound source object. However, when the sound source object is related to the set type sound source object, the reproduced sound is based on the separated BGM data 885. As a result, for the set type sound source object, only the sound of the L channel of the predetermined BGM or only the sound of the R channel is reproduced. Note that the playback volume follows the volume determined in step S24.

  Next, in step S <b> 26, the CPU 11 determines whether or not the above processing has been performed for all sound source objects to be processed. If unprocessed sound source objects still remain (NO in step S26), the CPU 11 returns to step S21 and repeats the process. On the other hand, if all the processes have been completed (YES in step S26), in step S27, the CPU 11 generates terminal game sound including the sound applied to each sound source object subjected to the above process.

  In subsequent step S27, the CPU 11 appropriately generates a monitor game sound corresponding to the result of the game process using the monitor virtual microphone. Here, it is assumed that the monitor game sound is also generated by the same processing as the above-described terminal game sound, with the targets being speakers 2L and 2R. This is the end of the game sound generation process.

  Returning to FIG. 15, next to the game sound generation process, in step S <b> 8, the CPU 11 stores the terminal game image generated in step S <b> 3 and the terminal game sound generated in step S <b> 7 in the terminal transmission data 84. Then, the terminal transmission data 84 is transmitted to the terminal device 6. Here, for convenience of explanation, an example is given in which the transmission cycle of the terminal game sound is matched with the transmission cycle of the terminal game image. However, in other embodiments, the transmission cycle of the terminal game sound is set to the terminal. The game image transmission cycle may be shorter. For example, the terminal game image may be transmitted at a 1/60 second period, and the terminal game sound may be transmitted at a 1/180 second period.

  Next, in step S9, the CPU 11 outputs the monitor game image generated in step S6 to the monitor 2. In subsequent step S10, CPU 11 outputs the monitor game sound generated in step S7 to speakers 2L and 2R.

  Next, in step S11, the CPU 11 determines whether or not a predetermined condition for ending the game process is satisfied. As a result, when the predetermined condition is not satisfied (NO in step S11), the process returns to step S2 and the above-described processing is repeated. When the predetermined condition is satisfied (YES in step S11), the CPU 11 ends the game process.

  Next, the flow of control processing executed by the control unit 33 of the terminal device 6 will be described with reference to the flowchart of FIG. First, in step S <b> 41, the control unit 33 receives terminal transmission data 84 transmitted from the game apparatus body 5.

  Next, in step S <b> 42, the control unit 33 outputs the terminal game image included in the received terminal transmission data 84 to the LCD 21.

  Next, in step S43, the control unit 33 outputs the terminal game sound included in the received terminal transmission data 84 to the speakers 23L and 23R. The volume balance follows the volume determined in step S24.

  Next, in step S <b> 44, the control unit 33 detects an input (operation content) to the operation unit 31, the motion sensor 32, and the touch panel 22, and generates operation button data 91, touch position data 92, and motion sensor data 93. To do.

  Next, in step S <b> 45, the control unit 33 generates terminal operation data 83 including the operation button data 91 and touch position data 92 generated in step S <b> 44 and transmits the terminal operation data 83 to the game apparatus body 5.

  Next, in step S46, the control unit 33 determines whether or not a predetermined condition for ending the control process of the terminal device 6 is satisfied (for example, whether or not a power-off operation has been performed). As a result, when the predetermined condition is not satisfied (NO in step S46), the process returns to step S41, and the above-described processing is repeated. When the predetermined condition is satisfied (YES in step S46), the control unit 33 ends the control process of the terminal device 6.

  As described above, in the present embodiment, the BGM reproduction process reflecting the positional relationship between the sound source object (particularly, the set-type sound source object) and the virtual microphone in the virtual space is performed. Accordingly, it is possible to more accurately represent the acoustic expression when the BGM composed of 2ch is reproduced from the sound source in the virtual space, and it is possible to further enhance the sense of reality in the virtual space. In particular, the above processing is suitable for the game processing of the system that displays the state of the virtual space like a “view window” by changing the attitude of the terminal device 6 itself as described above.

  In the above embodiment, in the initial setting process, a predetermined BGM is separated into LR channel data and assigned to different sound source objects. In addition to such processing, the following processing may be performed. For example, sound data (L channel data and R channel data) obtained by channel-separating the 2ch BGM as described above may be created in advance and included in the game sound data 85 at the game software development stage. . Then, sound source object data associated with L channel data or R channel data may be created in advance as BGM data corresponding to a predetermined sound source object. In this case, the structure of the sound source object data 88 as shown in FIG. 14 may include only the object ID 881 and the corresponding BGM identification data 883. In the initial setting process, it is only necessary to read out the data of the sound source object for reproducing the L channel data or the R channel data together with the data of the other sound source objects and appropriately arrange it in the virtual space. This is advantageous in that the processing time for the initial setting process is shortened.

  In addition, in the game sound generation process in step S7, the BGM channel separation process and the like may be executed in real time. For example, as the configuration of the sound source object data 88, only the post-separation BGM data 885 is removed from the configuration shown in FIG. In the game sound generation process, the set type sound source object is referred to the assigned channel identification data 884 of each sound source object constituting the set. If the channel assigned to the sound source object can be identified, the BGM data may be separated into the sound data of the L channel and the R channel, and only the sound data of the assigned channel may be reproduced.

  In the above embodiment, a game system having two screens and two sets of stereo speakers (four speakers), such as the monitor 2 and the terminal device 6, has been described as an example. For example, the above processing can be applied to an information processing apparatus in which a screen and a stereo speaker are integrated with a housing, such as a portable game device. Further, such an information processing apparatus is preferably an information processing apparatus that has a built-in motion sensor and can detect its own posture. Such an information processing apparatus is suitable when processing using the virtual space display system as described above is performed. In this case, the same processing as described above may be performed with only one virtual camera or virtual microphone.

  Further, for example, the terminal device 6 is used for display of a game screen that is a direct operation target of the player and various operations, but for the sound output, only the speaker of the monitor 2 is used without using the speaker 23 of the terminal device 6. The above processing can also be applied in the case of a configuration to be used. For example, when the player rotates 180 degrees from the posture facing the monitor 2 while holding the terminal device 6 (that is, when the player turns his back to the monitor 2), the set type sound source object is The sound may be output to the speakers 2L and 2R of the monitor 2.

  The same processing can be applied to a case where, for example, a 5.1ch surround system speaker is used as the external speaker. In this case, for example, 5.1ch-compatible BGM data is separated into data for each channel (becomes six sound data), and these are assigned to six sound source objects, and as described above. If the direction of the virtual microphone is changed in accordance with the change in the attitude of the terminal device 6 and the BGM playback processing is performed based on the positional relationship and direction between the sound source object and the virtual microphone in the virtual space, the number of speakers is increased. A sound effect of BGM reproduction with high presence can be provided.

  Further, for example, the above-described processing can be applied when a stereo headphone is connected to the terminal device 6 and the player listens to the game sound using this. In this case, the game apparatus main body 5 is configured to detect the connection state of the headphones to the headphone terminal 24. For example, the terminal operation data 83 is configured to include data indicating a connection state. Then, the above processing may be applied so that the output to the speakers 23L and 23R in the above processing is replaced with the output to the left and right speakers of such headphones. However, as shown in FIG. 11 described above, the process of sound generation in the vertical direction when the terminal device 6 is held vertically may not be applied when headphones are attached. This is because when the player is wearing headphones, the positional relationship of the speakers is fixed to the positional relationship of arrangement in the left-right direction.

  In addition, the above processing may be applied even when the player can operate only the orientation of the virtual camera without changing the orientation of the terminal device 6 itself (for example, the analog camera 25 is used for the virtual camera). Change orientation). From the viewpoint of the player, the virtual space itself is rotated around the user by operating the direction of the virtual camera. In this case, instead of the attitude of the terminal device 6, the orientation of the virtual microphone may be changed in accordance with the attitude of the virtual camera changed by the operation of the player. That is, the attitude of the virtual microphone may be changed so that the front direction (z-axis positive direction) of the virtual microphone matches the front direction (z-axis positive direction) of the virtual camera, and the above-described processing may be performed. .

  In addition, the above processing may be applied using a total of two sets of stereo speakers (a total of four speakers) including the speakers 2L and 2R of the monitor 2 and the speakers 23L and 23R of the terminal device 6. In particular, it is suitable when the terminal device 6 is mainly used in the “vertical orientation”. In this case, for example, the movement of the sound source object in the left-right direction in the virtual space is reflected in the outputs from the speakers 23L and 23R of the monitor 2. The movement of the sound source object in the vertical direction is reflected in the outputs from the speakers 23L and 23R of the terminal device 6. Thereby, for example, a highly realistic sound effect can be obtained without using a 5.1ch speaker set.

  In addition, a game processing program for executing the processing according to the above-described embodiment is an arbitrary computer-readable storage medium (for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic medium). Tape, semiconductor memory card, ROM, RAM, etc.).

  In the above embodiment, the game process is described as an example. However, the contents of the information processing are not limited to the game process, and the above processing is also performed in other information processing using the virtual space display system as described above. The example processing is applicable.

  In the above-described embodiment, a series of processes for performing control such that BGM data composed of 2ch is separated into LR channels and reproduced by separate sound source objects is performed by a single device (game device main body). Although the case where it performed in 5) was demonstrated, in other embodiment, the said series of processes may be performed in the information processing system which consists of a some information processing apparatus. For example, in an information processing system including a game apparatus body 5 and a server-side apparatus that can communicate with the game apparatus body 5 via a network, a part of the series of processes is executed by the server-side apparatus. May be. In the information processing system, the server-side system may be configured by a plurality of information processing apparatuses, and the plurality of information processing apparatuses may share and execute processing to be executed on the server side.

DESCRIPTION OF SYMBOLS 1 ... Game system 2 ... Monitor 2L, 2R ... Speaker 3 ... Game device 5 ... Game device main body 6 ... Terminal device 20 ... Housing 21 ... LCD
23L, 23R ... Speaker

Claims (11)

An information processing system comprising an information processing unit, a first speaker, and a second speaker,
Music data separation means for separating predetermined music data composed of two channels into first channel music data and second channel music data;
The first sound source object associated with one of the separated music data and the second sound source object associated with the other of the separated music data are different from each other in the virtual three-dimensional space. A sound source object placement means for placing a position;
Virtual microphone placement means for placing a virtual microphone at a predetermined position in the virtual three-dimensional space;
The first channel music reproduced from the first speaker based on the positional relationship between the direction of the virtual microphone arranged in the virtual three-dimensional space and the first sound source object and the second sound source object. Audio output control means for determining the reproduction volume of the data and the second channel music data, and the reproduction volume of the first channel music data and the second channel music data reproduced from the second speaker. Processing system. The information processing system includes:
A first output device having a housing provided with a motion sensor, the first speaker, the second speaker, and a first display unit on which a state of the virtual three-dimensional space is displayed;
Posture detection means for detecting the posture of the first output device itself in real time or almost in real time based on the output of the motion sensor;
The information processing system according to claim 1, further comprising: a virtual microphone posture changing unit that changes a direction of the virtual microphone according to a posture of the first output device. The information processing system further includes a speaker positional relationship recognition unit that recognizes a positional relationship between the first speaker and the second speaker based on an attitude of the first output device,
In addition to the positional relationship between the direction of the virtual microphone arranged in the virtual three-dimensional space and the first sound source object and the second sound source object, the audio output control means further includes the first speaker and the Based on the positional relationship of the second speaker, the playback volume of the first channel music data and second channel music data output from the first speaker, and the first channel music output from the second speaker. The information processing system according to claim 2, wherein a reproduction volume of the data and the second channel music data is determined. The information processing system further includes a second output device having a third speaker, a fourth speaker, and a second display unit,
The audio output control unit is based on a positional relationship between the first speaker and the second speaker included in the first output device and the third speaker and the fourth speaker included in the second output device. The information processing system according to claim 2, wherein the reproduction volume from each speaker is determined. The music data is music data composed of two stereo channels,
The information processing system according to claim 1, wherein the first speaker and the second speaker are a pair of stereo speakers. An information processing program to be executed by a computer of an audio output system including an information processing unit, a first speaker, and a second speaker, wherein predetermined music data composed of two channels is converted into first channel music data and second Music data separating means for separating into channel music data;
The first sound source object associated with one of the separated music data and the second sound source object associated with the other of the separated music data are different from each other in the virtual three-dimensional space. A sound source object placement means for placing a position;
Virtual microphone placement means for placing a virtual microphone at a predetermined position in the virtual three-dimensional space;
The first channel music reproduced from the first speaker based on the positional relationship between the direction of the virtual microphone arranged in the virtual three-dimensional space and the first sound source object and the second sound source object. The computer functions as audio output control means for determining the reproduction volume of the data and the second channel music data and the reproduction volume of the first channel music data and the second channel music data reproduced from the second speaker. Information processing program. The information processing system includes a housing provided with a motion sensor, the first speaker, the second speaker, and a first display unit on which a state of the virtual three-dimensional space is displayed. Further comprising a device,
The information processing program causes the computer to
Posture detection means for detecting the posture of the first output device itself in real time or almost in real time based on the output of the motion sensor;
The information processing program according to claim 6, further causing it to function as virtual microphone posture changing means for changing a direction of the virtual microphone according to a posture of the first output device. An information processing control method for controlling an information processing system and an information processing system including a first speaker and a second speaker, wherein predetermined music data composed of two channels is converted into first channel music data and A music data separation step for separating into two-channel music data;
The first sound source object associated with one of the separated music data and the second sound source object associated with the other of the separated music data are different from each other in the virtual three-dimensional space. A sound source object placement step to be placed at a position;
A microphone placement step of placing a virtual microphone at a predetermined position in the virtual three-dimensional space;
The first channel music reproduced from the first speaker based on the positional relationship between the direction of the virtual microphone arranged in the virtual three-dimensional space and the first sound source object and the second sound source object. And an audio output control step for determining a reproduction volume of the data and the second channel music data and a reproduction volume of the first channel music data and the second channel music data reproduced from the second speaker. Control method. The information processing system includes a housing having a motion sensor, the first speaker, the second speaker, and a first display unit on which a state of the virtual three-dimensional space is displayed. Has an output device,
The information processing control method includes:
An attitude detection step for detecting the attitude of the first output device itself in real time or almost in real time based on the output of the motion sensor;
The information processing system according to claim 8, further comprising: a virtual microphone posture changing step of changing a direction of the virtual microphone according to a posture of the first output device. An information processing apparatus comprising a first speaker and a second speaker,
Music data separation means for separating predetermined music data composed of two channels into first channel music data and second channel music data;
The first sound source object associated with one of the separated music data and the second sound source object associated with the other of the separated music data are different from each other in the virtual three-dimensional space. A sound source object placement means for placing a position;
Virtual microphone placement means for placing a virtual microphone at a predetermined position in the virtual three-dimensional space;
The first channel music reproduced from the first speaker based on the positional relationship between the direction of the virtual microphone arranged in the virtual three-dimensional space and the first sound source object and the second sound source object. Audio output control means for determining the reproduction volume of the data and the second channel music data, and the reproduction volume of the first channel music data and the second channel music data reproduced from the second speaker. Processing equipment. The information processing apparatus includes:
A motion sensor;
First display means for displaying the state of the virtual three-dimensional space;
Posture detecting means for detecting the posture of the information processing device itself in real time or almost in real time based on the output of the motion sensor;
The information processing apparatus according to claim 10, further comprising: a virtual microphone posture changing unit that changes a direction of the virtual microphone according to the posture of the information processing.

Images