JP7585761B2 - Program, performance evaluation device, and performance evaluation method - Google Patents

Description

The present invention relates to a technology for evaluating musical instrument performance.

Systems have been developed to evaluate musical instrument performance. Patent Documents 1 and 2 disclose systems that photograph a keyboard instrument performance from above and compare the photographed performance with a model performance to evaluate the performance. The system in Patent Document 1 evaluates the position, inclination, or width of the hands. Meanwhile, the system in Patent Document 2 evaluates whether the fingering used in performance is correct.

JP 2006-091633 A JP 2002-182553 A

Neither Patent Document 1 nor Patent Document 2 mentions the tempo of the performance. The present invention has been made to solve this problem, and its purpose is to provide a technique for evaluating a performance that is performed at a desired tempo.

The program disclosed herein causes a computer to execute a tempo information acquisition process that acquires tempo information capable of identifying a designated tempo, which is the tempo desired by the user for performance, and an evaluation process that generates evaluation information including an evaluation of the performance by the user by comparing, for a piece of music to be evaluated, standard information that represents the evaluation standard for a performance when played at the designated tempo with user performance information that represents the performance of the piece of music by the user.

The performance evaluation device disclosed herein has a tempo information acquisition unit that acquires tempo information capable of identifying a designated tempo, which is the tempo of performance desired by the user, and an evaluation unit that generates evaluation information including an evaluation of the performance by the user by comparing, for a piece of music to be evaluated, standard information that represents a performance evaluation standard when the piece is played at the designated tempo with user performance information that represents the performance of the piece of music by the user.

The performance evaluation method disclosed herein is executed by a computer. The method includes a tempo information acquisition step of acquiring tempo information capable of identifying a designated tempo, which is a performance tempo desired by a user, and an evaluation step of generating evaluation information including an evaluation of the performance by the user by comparing, for a piece of music to be evaluated, standard information representing a performance evaluation standard when the piece is played at the designated tempo with user performance information representing the performance of the piece of music by the user.

The present invention provides a technique for evaluating a performance performed at a desired tempo.

1 is a diagram illustrating an example of an overview of a performance evaluation device according to a first embodiment; 1 is a block diagram illustrating a functional configuration of a performance evaluation device according to a first embodiment. 1 is a block diagram illustrating an example of the hardware configuration of a computer that realizes a performance evaluation device. FIG. 2 is a diagram illustrating an example of the position and orientation of a camera. FIG. 2 is a diagram illustrating an example of joint points and joint lines of a hand. FIG. 13 is a diagram illustrating a method for calculating the height of a finger joint. 13 is a diagram illustrating a method for calculating the size of an angle between joint lines; FIG. FIG. 13 is a diagram illustrating a method for calculating the height of the wrist. 11 is a diagram illustrating a method for calculating the height of the hand upper surface. FIG. 11 is a diagram illustrating a method for calculating a wrist angle. FIG. FIG. 13 is a diagram illustrating a method for calculating the depth of a keystroke. FIG. 1 is a diagram illustrating an example of a usage mode of a performance evaluation device. 4 is a flowchart illustrating a process executed by the performance evaluation device of the first embodiment. FIG. 13 is a diagram illustrating an example of a tempo designation screen. FIG. 11 is a diagram illustrating an example of reference information in a table format. FIG. 13 is a diagram illustrating an example of a designation screen for designating a target song. 11 is a diagram illustrating an example of user performance information in a table format. FIG. 13 is a flowchart illustrating a process of calculating an evaluation score for each item information item. FIG. 11 is a diagram illustrating an example of evaluation information in a table format. 13 is a diagram illustrating an example of evaluation information showing a statistical score for each performance time point. FIG. FIG. 13 is a diagram illustrating an example of evaluation information indicating a statistical score for each evaluation item. 13 is a diagram illustrating an example of evaluation information indicating statistical scores for each partial range and for each evaluation item. FIG. FIG. 13 is a diagram illustrating an example of statistical values of evaluation scores tallied for each group. FIG. 11 is a first diagram illustrating an evaluation result screen. FIG. 2 is a second diagram illustrating an example of the evaluation result screen.

Below, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are given the same reference numerals, and duplicate explanations will be omitted as necessary for clarity of explanation. Furthermore, unless otherwise specified, predetermined values such as predetermined values and threshold values are stored in advance in a storage device accessible from a device that uses the values.

[Embodiment 1]
＜Overview＞
Fig. 1 is a diagram illustrating an example of an outline of a performance evaluation device 2000 according to embodiment 1. Fig. 1 is a diagram conceptually illustrating an example of the operation of the performance evaluation device 2000 to facilitate understanding of the performance evaluation device 2000, and the operation of the performance evaluation device 2000 is not limited to that shown in Fig. 1.

The performance evaluation device 2000 generates evaluation information 30 that represents an evaluation of an instrument performance performed by a user 60. Here, the instrument played by the user 60 is called a target instrument 70. The target instrument 70 is, for example, a keyboard instrument such as a piano, an organ, or a synthesizer. However, the target instrument 70 is not limited to a keyboard instrument.

The performance evaluation device 2000 generates evaluation information 30 by comparing the actual performance given by the user 60 with a predetermined standard. The comparison is performed for one or more evaluation items. Here, the user 60 can specify the tempo of the performance when evaluating the performance of the piece of music to be evaluated. Hereinafter, the piece of music to be evaluated is referred to as the target piece of music. Also, the specified tempo of the performance is referred to as the specified tempo.

To evaluate the performance in the above-mentioned manner, the performance evaluation device 2000 uses tempo information 100, reference information 40, and user performance information 50. The tempo information 100 is information that can identify the specified tempo. The reference information 40 indicates the evaluation standard for the performance when the target song is played at the specified tempo indicated in the tempo information 100. In other words, the reference information 40 indicates what kind of performance should be performed at what timing when the target song is played at the specified tempo indicated in the tempo information 100. For example, the reference information 40 indicates the performance time and the ideal item value for the performance at that performance time for each of multiple evaluation items. The user performance information 50 is information that represents the actual performance of the target song by the user. In other words, the user performance information 50 indicates what kind of performance the user 60 performed at what timing for the target song. For example, the user performance information 50 indicates the performance time and the item value (actual value) for the performance of the user 60 at that performance time for each of multiple evaluation items.

The performance evaluation device 2000 generates evaluation information 30 by comparing the reference information 40 with the user performance information 50. For example, as described below, the performance evaluation device 2000 identifies the degree of match between the reference information 40 and the user performance information 50 by comparing item values indicated by the reference information 40 with item values indicated by the user performance information 50 for each of a number of time points, and generates evaluation information 30 that indicates the degree of match.

Here, FIG. 1 illustrates an example in which the reference information 40 and the user performance information 50 are obtained from outside the performance evaluation device 2000. However, as described below, the reference information 40 and the user performance information 50 may be generated inside the performance evaluation device 2000. The method of obtaining and generating the reference information 40 and the user performance information 50 will be described in detail below.

<Examples of effects>
According to the performance evaluation device 2000 of this embodiment, reference information 40 is acquired that represents an evaluation standard when the target piece of music is performed at a specified tempo. Then, the reference information 40 is compared with user performance information 50 that represents the result of the actual performance by the user 60, thereby evaluating the performance of the target piece of music by the user 60. Thus, according to the performance evaluation device 2000, it is possible to evaluate a performance of the target piece of music performed at a desired tempo. Thus, the user 60 can play the target piece of music at a desired tempo and receive an evaluation of the performance.

For example, user 60 can practice the target song while gradually increasing the tempo of the performance. First, user 60 practices playing the target song at a tempo slower than the standard tempo. Then, when the user 60 is able to receive a sufficiently high evaluation (e.g., an evaluation higher than the evaluation set as a target) in the evaluation by the performance evaluation device 2000 specifying that tempo, user 60 speeds up the tempo a little and practices the target song further. By repeating this practice and evaluation while increasing the tempo, user 60 will be able to perform the target song at the standard tempo or a tempo faster, and receive high evaluation.

The performance evaluation device 2000 of this embodiment is described in more detail below.

<Example of functional configuration>
2 is a block diagram illustrating an example of the functional configuration of a performance evaluation device 2000 of embodiment 1. In this example, the performance evaluation device 2000 has a tempo information acquisition unit 2020 and an evaluation unit 2040. The tempo information acquisition unit 2020 acquires tempo information 100. The evaluation unit 2040 generates evaluation information 30 by comparing reference information 40 corresponding to a designated tempo indicated in the tempo information 100 with user performance information 50.

<Example of hardware configuration>
Each functional component of the performance evaluation device 2000 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further explanation will be given of the case where each functional component of the performance evaluation device 2000 is realized by a combination of hardware and software.

Figure 3 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the performance evaluation device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. In other examples, the computer 1000 is a portable computer such as a smartphone or a tablet terminal. In other examples, the computer 1000 may be a camera 10. In other examples, if the instrument played by the user 60 is an electronic instrument, the computer 1000 may be a computer built into the instrument.

The computer 1000 may be a dedicated computer designed to realize the performance evaluation device 2000, or may be a general-purpose computer. For example, by installing a specific application on the computer 1000, the computer 1000 realizes each function of the performance evaluation device 2000. The application is composed of a program for realizing the functional components of the performance evaluation device 2000.

The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input/output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path through which the processor 1040, the memory 1060, the storage device 1080, the input/output interface 1100, and the network interface 1120 transmit and receive data to each other. However, the method of connecting the processor 1040 and the like to each other is not limited to a bus connection.

The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

The input/output interface 1100 is an interface for connecting the computer 1000 to an input/output device. The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

The storage device 1080 stores a program that realizes each functional component of the performance evaluation device 2000 (a program that realizes the application described above). The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the performance evaluation device 2000.

The performance evaluation device 2000 may be realized by one computer 1000, or may be realized by multiple computers 1000. In the latter case, the configuration of each computer 1000 does not need to be the same, and can be different from each other.

<Evaluation items>
The performance by the user 60 is evaluated based on a number of evaluation items. The evaluation items may be any items that reflect the quality of the performance. Such evaluation items include, for example, items related to sound characteristics (such as sound intensity and duration). Other evaluation items include, for example, the shape of the hands and fingers when playing, fingering, and the shape of the target instrument when played. Here, fingering refers to the correctness of the correspondence between the pressed keys and the fingers. Furthermore, the shape of the target instrument when played refers to, for example, the angle and depth of the keys when pressed with the fingers in the case of a keyboard instrument.

When items related to sound characteristics are included in the evaluation items, for example, a recorder is installed to record the performance of user 60, and the performance is evaluated using the audio data generated by the recorder. However, the audio data may be generated together with the video data by a camera, which will be described later. In other words, in this case, the camera also plays the role of a recorder. For example, when the camera generates a video file containing video data and audio data, the audio data included in the video data can be used to evaluate the sound characteristics (evaluate whether the sound is good or bad).

For example, the target instrument 70 may be provided with a function for generating performance data for a performance given using the target instrument 70. The performance data may be, for example, MIDI (registered trademark) (Musical Instrument Digital Interface) data. When performance data is generated by the target instrument 70 in this way, the performance data can be used to evaluate the characteristics of the sound.

When the evaluation items include items that can be evaluated using visual information, for example, a camera is installed to capture the performance of user 60, and the performance of user 60 is evaluated using the video data generated by the camera. Such evaluation items include, for example, the shape of the hands and fingers when playing, the fingering, and the shape of the target instrument when played. When evaluating the performance of user 60 for both the left and right hands, for example, it is preferable to provide both a camera installed in a position where the left hand of user 60 is in front, and a camera installed in a position where the right hand of user 60 is in front.

Figure 4 is a diagram illustrating the position and orientation of the camera 10 capturing an image of a performance by a user 60. Figure 4(a) is a plan view of the target instrument 70, and Figure 4(b) is a front view of the target instrument 70. In Figure 4, the longitudinal direction of the keys of the target instrument 70 is the x-axis direction, the transverse direction of the keys of the target instrument 70 is the y-axis direction, and the vertical direction is the z-axis direction.

Camera 10-1 is placed so as to capture at least the side of the fingers of the user's 60 left hand from the left side (from a direction in which the left hand is closer to the viewer than the right hand). In FIG. 4, camera 10-1 is placed so as to look down on the target musical instrument 70 in the +x direction.

On the other hand, camera 10-2 is placed so that it can capture at least the side of the fingers of the user's 60 right hand from the right side (from a direction in which the right hand is closer to the viewer than the left hand). In FIG. 4, camera 10-2 is placed so as to look down on the target musical instrument 70 in the -x direction.

The shooting direction of camera 10 is not limited to the example shown in FIG. 4. For example, the horizontal components of the shooting directions of camera 10-1 and camera 10-2 do not need to be completely parallel to the x-axis, and may include a component in the y-axis direction.

The specific installation position and orientation of the camera 10 may be determined arbitrarily by the user or may be determined in advance. In the former case, the camera 10 is installed on or around the instrument in a position and orientation that allows the performance of the user 60 to be captured from the side.

On the other hand, if the installation position and orientation of the camera 10 are predetermined, for example, an apparatus is used that can be installed at a predetermined position on the target instrument and that can install the camera 10 in a predetermined orientation. This apparatus may be installed on the target instrument when the target instrument is provided (for example, sold or leased), or may be installed after the target instrument is provided (for example, provided as a separately sold attachment and installed by the user).

Alternatively, for example, the camera 10 may be installed at a predetermined position and attitude by adjusting the installation position and attitude of the camera 10 while viewing the video data generated by the camera 10. For example, a predetermined reference mark is provided at a predetermined position on the target instrument. Then, the reference mark provided on the target instrument is photographed by the camera 10, and while viewing the video data obtained from the camera 10, the position and attitude of the camera 10 are adjusted so that the reference mark is photographed by the camera 10 at a predetermined position, size, and shape.

In the above example, the cameras 10-1 and 10-2 are set to capture images of different hands, but this is not necessarily required. For example, the camera 10-1 captures images of some of the fingers of the left hand and some of the fingers of the right hand, and the camera 10-2 captures images of the remaining fingers of the left and right hands. In other words, the video data 20 obtained from the camera 10-1 is used to evaluate some of the fingers of the left hand and some of the fingers of the right hand, and the video data 20 obtained from the camera 10-2 is used to evaluate the remaining fingers of the left and right hands. As a more specific example, the video data 20 obtained from the camera 10-1 is used to evaluate the middle finger, ring finger, and little finger of the left hand and the thumb and index finger of the right hand, and the video data 20 obtained from the camera 10-2 is used to evaluate the thumb and index finger of the left hand and the middle finger, ring finger, and little finger of the right hand. In this way, even if one camera 10 is unable to capture all the fingers of one hand because one hand is an obstacle to capturing the image of the other hand, it is possible to evaluate all the fingers on both the left and right hands by using the video data 20 obtained from the two cameras 10.

The number of cameras 10 used to evaluate a performance is not limited to two. For example, if the performance of the user 60 is to be evaluated for only one hand, one camera 10 capable of photographing the hand to be evaluated may be provided. Alternatively, for example, multiple cameras 10 may be provided for each of the right and left hands so that the same hand is photographed at different angles of depression.

Camera 10 is any camera capable of capturing a performance by user 60 and generating video data. Camera 10 may be a dedicated camera for capturing a user's performance of a musical instrument, or may be a general-purpose camera (such as a camera provided on a smartphone).

For example, the camera 10 is connected to a computer 1000 that realizes the performance evaluation device 2000 via the input/output interface 1100 or the network interface 1120. However, the computer 1000 only needs to be able to acquire the video data generated by the camera 10, and does not need to be communicatively connected to the camera 10. The method by which the performance evaluation device 2000 acquires the video data will be described later. The performance evaluation device 2000 may also be realized by the camera 10. That is, the performance evaluation device 2000 is realized by a computer (for example, a semiconductor chip such as a SoC (System on Chip)) provided in the camera 10.

As mentioned above, evaluation items for which video data can be used for evaluation include evaluation items related to the shape of the hand and fingers. The item values of these evaluation items are calculated using, for example, the joint points and joint lines of the hand. A joint point is a point that indicates the position of a joint. A joint line is a straight line that connects adjacent joint points. In addition, a straight line that connects a fingertip and the joint point of the joint closest to the fingertip (i.e., the first joint) is also treated as a joint line. Figure 5 is a diagram illustrating examples of hand joint points and joint lines. In Figure 5, the joint points are represented by circles.

Below are some specific examples of evaluation items whose item values are calculated based on the hand joint points and joint lines.

<<Knuckle height>>
The "height of finger joints" is an evaluation item that indicates the position of the finger joints in the height direction. The joint points used by the performance evaluation device 2000 for evaluation may be all of the joint points or only some of them.

For example, the height of a finger joint is determined by the positional relationship between the finger joint point and a reference plane. For example, the reference plane is determined by the top surface of the keyboard. For example, the top surface of the keyboard when it is normally not being pressed is treated as the reference plane. Alternatively, for example, the top surface of the key when it is pressed all the way down may be treated as the reference plane. Various methods can be used as the specific method for calculating the finger joint height.

The height of the finger joints is preferably calculated for fingers that are typing. However, the height of the finger joints may also be calculated for fingers that are not typing.

Figure 6 is a diagram illustrating a method for calculating the height of a finger joint. For ease of explanation, Figure 6 shows only the image area of the video frame 22 surrounding the finger 110 and the key 120 being pressed by the finger 110.

In FIG. 6, a reference line 130 is used to indicate the position of the reference plane. The reference line 130 is a straight line that overlaps with the long side of the key adjacent to the key 120 on which the finger 110 is placed, which is farther from the camera 10. Note that it is preferable to use the long side of the top surface of the key 120 in an undepressed state as the reference line. Furthermore, the method of indicating the position of the reference plane is not limited to the method using the reference line 130. For example, the long side of the key adjacent to the key 120, which is closer to the camera 10, may be used as the reference line indicating the position of the reference plane.

Figures 6(a) and (b) show different methods for calculating the height of a finger joint. In Figure 6(a), the height of joint point X is represented by the length of line segment XA. Point A is the intersection of a line extending vertically downward from joint point X in video frame 22 and reference line 130. On the other hand, in Figure 6(b), the height of joint point X is represented by the length of line segment XB. Point B is the foot of a perpendicular line drawn from joint point X to reference line 130.

In the example of FIG. 6, the camera 10 is set up so as to capture the side of the hand from diagonally above. In this case, the vertical direction of the video frame 22 and the direction of the perpendicular line drawn to the reference line 130 in FIG. 6(b) are different from the vertical direction in the real world. However, as shown in FIG. 6, the item value of the evaluation item "finger joint height" does not represent the height in the vertical direction in the real world, but only needs to represent the distance from the joint point to the reference plane in a direction that includes a vertical component (i.e., a direction that is not horizontal). The same applies to "wrist height" described below.

However, the performance evaluation device 2000 may treat the height of the finger joint as the height in the vertical direction in the real world. In this case, for example, the performance evaluation device 2000 calculates the position and reference plane of the finger joint in a three-dimensional coordinate system of the real world (hereinafter, world coordinate system) from the position and reference plane of the finger joint in a two-dimensional coordinate system (hereinafter, image coordinate system) on the video frame 22, thereby calculating the height of the finger joint in the world coordinate system.

The conversion from the coordinates in the two-dimensional coordinate system to the coordinates in the three-dimensional coordinate system can be realized, for example, by using stereo vision or a depth camera. In the former case, two cameras 10 are provided to capture the performance of the user 60 from the left and two cameras 10 to capture the performance from the right. Then, to calculate the height of the finger joints on the right hand of the user 60, the height of the finger joints on the world coordinate system is calculated using the video frames 22 obtained from each of the two cameras 10 capturing the image of the user 60 from the right. When a depth camera is used, the distance to the captured object is known for each pixel on the video frame 22. Therefore, the three-dimensional coordinates of the finger joints in the world coordinate system can be determined from the two-dimensional coordinates of the finger joints in one video frame 22.

In order to determine the world coordinate system, it is preferable to provide a predetermined mark (a mark of a certain size and shape) that serves as a reference for the world coordinate system at a predetermined position on the target instrument. Note that existing technology can be used to determine the relationship between the two-dimensional coordinate system on the image obtained from the camera and the world coordinate system by photographing a mark of known size and shape with a camera.

<<The size of the angle between the joint lines>>
The "magnitude of angle between joint lines" is an evaluation item that represents the magnitude of an angle formed by two joint lines having a common end point. FIG. 7 is a diagram illustrating a method for calculating the magnitude of an angle between joint lines. For ease of explanation, FIG. 7 illustrates only an image area around a finger 110 in a video frame 22. Note that it is preferable that the magnitude of the angle between joint lines is calculated for a finger in a typing state. However, the magnitude of the angle between joint lines may also be calculated for a finger in a non-typing state.

In FIG. 7, joint point C and joint point D are adjacent joint points, and the straight line connecting these joint points is joint line CD. Similarly, joint point D and joint point E are adjacent joint points, and the straight line connecting these joint points is joint line DE. The magnitude of the angle formed by joint line CD and joint line DE is α. Therefore, for example, this value α can be used as the item value of the evaluation item of the magnitude of the angle formed by joint line CD and joint line DE. Furthermore, by further using joint line EF connecting joint line E and fingertip point F, an evaluation item of the magnitude of the angle formed by joint line DE and joint line EF may be further set.

For example, the value of the item "angle between two joint lines" may be the size of the angle between these two joint lines in the world coordinate system described above. In this case, the performance evaluation device 2000 identifies the two joint lines in the world coordinate system by converting the coordinates of each finger joint on the video frame 22 into coordinates in the world coordinate system. In this way, it is possible to identify the size of the angle between the two joint lines in the world coordinate system.

<<Wrist height>>
"Wrist height" is an evaluation item that indicates the position of the wrist in the height direction. For example, the wrist height is determined by the positional relationship between the wrist joint point and a reference plane. This reference plane can be the same as the reference plane that determines the height of the finger joints. Note that the wrist height is preferably calculated for a hand in a state where at least one finger is typing. However, the wrist height may also be calculated for a hand in a state where none of the fingers are typing.

Figure 8 illustrates a method for calculating wrist height. For ease of explanation, Figure 8 shows only the image area of the video frame 22 surrounding the hand 140 and the key that the hand 140 is typing.

In FIG. 8, the wrist joint point of the hand 140 is Y. Also, in FIG. 8, a reference line 150 is used to indicate the position of the reference plane. The reference line 150 is a straight line that overlaps with the long side of the key adjacent to the pressed key 160 that is farther from the camera 10. As in the case of calculating the height of the finger joints, it is also possible to use the long side of the top surface of the key when it is not being pressed as the reference line. Also, the key used for the reference line is not limited to the key pressed by the finger closest to the camera 10 or the adjacent key.

Figures 8(a) and (b) show different methods for calculating wrist height. In Figure 8(a), the wrist height is determined by the distance of line segment YG. Point G is the intersection of a line drawn vertically downward from wrist joint point Y on video frame 22 and reference line 150. On the other hand, in Figure 8(b), the wrist height is determined by the distance of line segment YH. Point H is the foot of the perpendicular line drawn from wrist joint point Y to reference line 150.

The performance evaluation device 2000 may also treat the wrist height as the height in the vertical direction in the real world, in the same way as the finger joint height. Specifically, the performance evaluation device 2000 identifies the wrist position and reference plane in the world coordinate system from the wrist position and reference plane in the image coordinate system. Then, the distance from the wrist position to the reference plane in the world coordinate system is treated as the wrist height.

<<Hand top height>>
"Height of the hand top surface" is an evaluation item that indicates the position of the hand top surface in the height direction. For example, the height of the hand top surface is determined by the positional relationship between the third joint point of the middle finger (see FIG. 5) and a reference plane. This reference plane can be the same as the reference plane that determines the height of the finger joints. Note that the height of the hand top surface is preferably calculated for a hand in a state where at least one finger is typing. However, the height of the hand top surface may also be calculated for a hand in a state where none of the fingers are typing.

Figure 9 is a diagram illustrating a method for calculating the height of the hand's upper surface. Here, the hand 140, keyboard, and reference line 150 shown in Figure 9 are the same as those shown in Figure 8. Also, in Figure 9, the position of the upper surface of the hand 140 is represented by point W, which is the third joint point of the middle finger of the hand 140.

Figures 9(a) and (b) show different methods for calculating the height of the upper surface of the hand. In Figure 9(a), the height of the upper surface of the hand is determined by the distance of line segment WK. Point K is the intersection of a straight line drawn vertically downward from the upper surface of the hand W in the video frame 22 and the reference line 150. On the other hand, in Figure 9(b), the height of the upper surface of the hand is determined by the distance of line segment WL. Point L is the foot of the perpendicular line drawn from the upper surface of the hand W to the reference line 150.

The performance evaluation device 2000 may also treat the height of the top surface of the hand as the height in the vertical direction in the real world, just as it does for the height of the finger joints and wrist. Specifically, the performance evaluation device 2000 identifies the position of the top surface of the hand and the reference plane in the world coordinate system from the position of the top surface of the hand and the reference plane in the image coordinate system. Then, the distance from the position of the top surface of the hand in the world coordinate system to the reference plane is treated as the height of the top surface of the hand.

<<Wrist angle>>
"Wrist angle" is an evaluation item that indicates the degree of bending of the wrist. The wrist angle can be determined, for example, by the size of the angle formed by a line representing the posture of the arm and a line representing the posture of the top surface of the hand. Note that it is preferable that the wrist angle is calculated for a hand in a state in which at least one finger is typing. However, the wrist angle may also be calculated for a hand in a state in which none of the fingers is typing.

Figure 10 is a diagram illustrating a method for calculating the wrist angle. For ease of explanation, in Figure 10, only the image area around the hand 140 is shown in the video frame 22. The wrist angle is represented by the magnitude β of the angle formed by lines 170 and 180. Line 170 is a line that represents the posture of the arm. On the other hand, line 180 is a line that represents the posture of the upper surface of the hand.

For example, straight line 170 can be obtained by detecting an edge representing the upper surface of the arm from video frame 22, and connecting any two points on the edge, or performing straight line fitting using three or more points on the edge. Straight line 180 can be obtained similarly by detecting an edge representing the upper surface of the hand from video frame 22, and connecting any two points on the edge, or performing straight line fitting using three or more points on the edge.

Note that angles in the world coordinate system may also be used for the wrist angle. In this case, for example, the performance evaluation device 2000 calculates a line representing the arm posture and a line representing the posture of the top of the hand in the world coordinate system, and calculates the angle between these lines as the wrist angle. Also, in the world coordinate system, the arm posture and the top of the hand posture may be represented by planes rather than lines. In this case, the angle between these two planes can be treated as the wrist angle.

<<Keystroke Depth>>
"Keystroke depth" is an evaluation item that indicates the degree to which a key is pressed when playing. Keystroke depth can be expressed as the relative position of a key when pressed compared to when it is normally pressed.

Figure 11 is a diagram illustrating a method for calculating the depth of a keystroke. For ease of explanation, only the image area of the video frame 22 surrounding the key 190 is shown in Figure 11.

Key 190 is the key that is being pressed. Line segment 200 represents the long side of the top surface of key 190 that is farther from camera 10 in the normal state (when not being pressed). On the other hand, line segment 210 represents the long side of the top surface of key 190 that is farther from camera 10 in the pressed state. Note that line segment 200 may also represent the long side of the top surface of the key next to key 190 that is being pressed and that is farther from camera 10.

Figures 11(a) and (b) show different examples of calculation methods for keystroke depth. In Figure 11(a), the keystroke depth is represented by the size of the angle between line segment 200 and line segment 210. Meanwhile, in Figure 11(b), the keystroke depth is represented by the length of line IJ. Point I is the end point of line segment 200 that is closer to the body of user 60. Point J is the end point of line segment 210 that is closer to the body of user 60.

Note that values in the world coordinate system may also be used for the keystroke depth. For example, the performance evaluation device 2000 converts the line segments 200 and 210 in the image coordinate system into line segments in the world coordinate system, and uses the size of the angle between these line segments and the distance between their endpoints as item values for the keystroke depth. Furthermore, the points used are not limited to the endpoints, and any point on the line segments 200 and 210 may be used based on the similarity relationship.

<Examples of Usage of the Performance Evaluation Device 2000>
To facilitate understanding of the performance evaluation device 2000, an example of how the performance evaluation device 2000 is used will be described. Note that the usage described here is merely an example, and the specific usage of the performance evaluation device 2000 is not limited to the example described here. For example, when dealing with only evaluation items whose item values can be calculated from performance data, the camera 10 does not need to be used.

Fig. 12 is a diagram illustrating an example of how the performance evaluation device 2000 can be used. In Fig. 12(a), the performance evaluation device 2000 is realized by a mobile terminal (user terminal 62) operated by a user 60. For example, it is conceivable that the performance evaluation device 2000 can be realized by the user terminal 62 by installing a program for realizing each of the functional components of the performance evaluation device 2000 described above into the user terminal 62.

The user terminal 62 acquires video data 20 from the camera 10 that captures the user 60 playing the target instrument 70 from the left, and from the camera 10 that captures the user 60 playing the target instrument 70 from the right. The user terminal 62 also acquires performance data 72 from the target instrument 70. As described above, the performance data 72 is MIDI (registered trademark) data or the like. The user terminal 62 then generates user performance information 50 by analyzing the acquired video data 20 and performance data 72, and generates evaluation information 30 by comparing the user performance information 50 with reference information 40.

On the other hand, in FIG. 12(b), the performance evaluation device 2000 is realized by a server machine 80 that operates upon receiving a request from a user terminal 62. In this usage mode, the user terminal 62 is used as an interface for using the performance evaluation device 2000. Specifically, the user terminal 62 acquires video data 20 from the two cameras 10. The user terminal 62 also acquires performance data 72 from the target instrument 70. The video data 20 and performance data 72 are then transmitted from the user terminal 62 to the server machine 80. For example, the video data 20 and performance data 72 are attached to a request for an evaluation of the performance of the user 60. The server machine 80 analyzes the acquired video data 20 and performance data 72 to generate user performance information 50, and generates evaluation information 30 by comparing the user performance information 50 with the reference information 40. The server machine 80 then transmits a response including the evaluation information 30 to the user terminal 62.

<Processing flow>
13 is a flow chart illustrating the flow of processing executed by the performance evaluation device 2000 of the first embodiment. The tempo information acquisition unit 2020 acquires the tempo information 100 (S102). The evaluation unit 2040 acquires the reference information 40 corresponding to the designated tempo indicated in the tempo information 100 (S104). The evaluation unit 2040 acquires the user performance information 50 (S106). The evaluation unit 2040 generates the evaluation information 30 using the reference information 40 and the user performance information 50 (S108).

The process flow executed by the performance evaluation device 2000 is not limited to that shown in FIG. 13. For example, the acquisition of the user performance information 50 (S106) may be executed before the acquisition of the tempo information 100 (S102) or the acquisition of the reference information 40 (S104). Furthermore, these processes may be executed in parallel.

<Acquisition of tempo information 100: S102>
The tempo information acquisition unit 2020 acquires tempo information 100. For example, the tempo information 100 is acquired in response to a user input. That is, the tempo information acquisition unit 2020 acquires tempo information 100 that can identify the tempo specified by the user input.

In this case, for example, the tempo information acquisition unit 2020 causes a screen for inputting the designated tempo (hereinafter, the tempo designation screen) to be displayed on a display device of a terminal that can be operated by the user. For example, this display device is displayed on a touch panel display provided on a computer (such as the user terminal 62 or the target instrument 70) on which an application that realizes the functions of the performance evaluation device 2000 is installed. Note that the user who designates the designated tempo may be any person and does not have to be the user 60 who is the performer. The tempo information acquisition unit 2020 acquires information indicating the result of the user's input on the screen as tempo information 100.

FIG. 14 is a diagram illustrating an example of a tempo designation screen 90. In the example of FIG. 14(a), the user inputs the designated tempo by inputting a specific numerical value. In this case, the tempo information 100 indicates the specific numerical value input by the user (i.e., the specific numerical value representing the designated tempo). Therefore, the tempo information acquisition unit 2020 identifies the value indicated in the tempo information 100 as the designated tempo.

On the other hand, in the example of FIG. 14(b), multiple options for the designated tempo are displayed, and the designated tempo is input by selecting one of them. Specifically, in FIG. 14(b), one of three tempos, "normal", "slightly slow", and "slow", can be selected. In this case, tempo information 100 indicates the option selected by the user as information that can identify the designated tempo. The tempo information acquisition unit 2020 identifies the designated tempo based on the information indicated in the tempo information 100.

For example, the tempo information acquisition unit 2020 identifies a normal tempo corresponding to the target song, and identifies a specified tempo based on the normal tempo and the options shown in the tempo information 100. Note that the normal tempo corresponding to the target song is stored in advance in a storage device accessible from the performance evaluation device 2000, for example, in association with the identification information of the target song. In addition, for relative tempos such as "slightly slow," a ratio to the normal tempo is determined in advance.

For example, in FIG. 14(b), "slightly slow" is 2/3 the normal tempo, and "slow" is 1/2 the normal tempo. Also, assume that the normal tempo of the target song is 120 BPM (Beats Per Minute). In this case, the tempos corresponding to "slightly slow" and "slow" are 80 BPM and 60 BPM, respectively. Therefore, the tempo information acquisition unit 2020 specifies 120 BPM as the designated tempo when "normal" is selected, specifies 80 BPM as the designated tempo when "slightly slow" is selected, and specifies 60 BPM as the designated tempo when "slow" is selected.

The tempo information acquisition unit 2020 may include specific tempo values (e.g., the above-mentioned 120 BPM, 80 BPM, and 60 BPM) corresponding to each option in the tempo designation screen 90 of FIG. 14(b). In this case, the tempo information acquisition unit 2020 identifies the normal tempo of the target song before generating the tempo designation screen 90. The tempo information acquisition unit 2020 also identifies the tempo corresponding to each option based on the normal tempo of the target song.

<Acquisition of Reference Information 40: S104>
The evaluation unit 2040 acquires the reference information 40 (S104). The reference information 40 is information indicating reference item values for each evaluation item for a given piece of music. The reference item values are ideal item values obtained, for example, by analyzing the performance of an advanced player such as a lecturer. Hereinafter, the person who performs to generate the reference information 40 is referred to as a model performer. The reference information 40 is generated in advance for each piece of music that can be evaluated, and is stored in a storage device accessible from the performance evaluation device 2000.

The reference information 40 is time-series data that indicates the item values of each evaluation item in a time series. More specifically, the reference information 40 indicates, for each of a plurality of performance time points, a combination of "performance time point, item value of each evaluation item for the performance at that performance time point." Each performance time point indicated in the reference information 40 is, for example, a time point at which at least one note is played. That is, in this case, the reference information 40 indicates, for each time point at which at least one note is played, the standard for how the performance should be performed at that time point.

The starting point of the performance time (the point when the value is 0) is, for example, the start point of a piece of music. The start point of a piece of music is, for example, the point when the piece of music is first played (such as a keystroke). Hereinafter, unless otherwise specified, the starting point of the performance time indicated in the reference information 40 is assumed to be the start point of the piece of music.

FIG. 15 is a diagram illustrating an example of the criteria information 40 in table format. The criteria information 40 has item information 44 for each performance time point 42. The item information 44 indicates, for each evaluation item, a pair of an evaluation item 46 and an item value 48 for the performance at the corresponding performance time point. The evaluation item 46 indicates identification information (e.g., the name of the evaluation item) that specifies the evaluation item. The item value 48 indicates the item value of the corresponding evaluation item.

The performance time 42 corresponding to each item of information 44 differs depending on the tempo of the performance. For example, a sound played at time t1 with a tempo of 120 BPM will be played at time 2*t1 with a tempo of 60 BPM. Therefore, even for the same piece of music, the reference information 40 to be used will differ for each specified tempo.

Therefore, for example, multiple pieces of reference information 40 corresponding to different tempos are prepared in advance for each target song. In this case, the evaluation unit 2040 acquires the reference information 40 corresponding to the specified tempo from the multiple pieces of reference information 40 prepared for the target song. For example, suppose that there are three tempos that can be designated when evaluating the performance of song M1: 120 BPM, 80 BPM, and 60 BPM. Also, suppose that the designated tempo identified based on the tempo information 100 is 80 BPM. In this case, the evaluation unit 2040 acquires the reference information 40 corresponding to the tempo of 80 BPM from the three pieces of reference information 40 prepared for song M1.

Alternatively, for example, one piece of reference information 40 may be prepared for each target song, and the value of the performance time point 42 may be corrected to match the specified tempo. For example, reference information 40 for when the target song is played at the normal tempo is prepared in advance for each target song. In this case, the reference information 40 further includes information indicating the normal tempo. The evaluation unit 2040 converts the reference information 40 corresponding to the normal tempo into reference information 40 corresponding to the specified tempo by correcting each performance time point 42 indicated by the reference information 40 acquired for the target song based on the ratio between the designated tempo identified using the tempo information 100 and the normal tempo indicated by the reference information 40. In this way, it is possible to acquire the reference information 40 corresponding to the specified tempo.

For example, as mentioned above, a sound played at time t1 at a tempo of 120 BPM will be played at time 2*t1 at a tempo of 60 BPM. In other words, if we generalize this and express the reference tempo as r1 [BPM] and the specified tempo as r2 [BPM], then by multiplying each performance time point indicated by the reference information 40 corresponding to tempo r1 by r1/r2, the reference information 40 corresponding to tempo r1 can be converted to the reference information 40 corresponding to tempo r2.

Note that even when obtaining reference information 40 corresponding to the specified tempo by correcting the reference information 40 in this manner, multiple pieces of reference information 40 with different tempos may be prepared for each target song. In this case, for example, if reference information 40 corresponding to the specified tempo is prepared in advance, the evaluation unit 2040 uses that reference information 40. On the other hand, if reference information 40 corresponding to the specified tempo is not prepared in advance, the evaluation unit 2040 identifies, from the multiple pieces of reference information 40, the reference information 40 corresponding to the tempo closest to the specified tempo, and performs the above-mentioned correction on that reference information 40 to generate reference information 40 corresponding to the specified tempo. In this way, by correcting the reference information 40 corresponding to the tempo closest to the specified tempo, it is possible to minimize errors that may occur due to the correction.

For example, for song M1, two pieces of reference information 40 are prepared in advance: one corresponding to 60 BPM, and the other corresponding to 120 BPM. Also, assume that a tempo of 100 BPM is specified. In this case, the evaluation unit 2040 generates reference information 40 corresponding to 100 BPM by correcting the reference information 40 corresponding to 120 BPM, which is closer to the specified tempo. Note that it is preferable to specify the closeness of the tempos by the ratio of the contrasting tempos. For example, in the above example, the closeness between 60 BPM and 100 BPM is 5/3 (=100/60), and the closeness between 100 BPM and 120 BPM is 6/5 (=120/100).

<<Identifying target songs>>
Here, in order to obtain the reference information 40 about the target piece of music, it is necessary to identify the target piece of music. Furthermore, as described above, if the tempo specified by the user is a relative tempo, it is necessary to know the normal tempo of the target piece of music, and therefore it is necessary to identify the target piece of music. Therefore, the performance evaluation device 2000 identifies the target piece of music.

There are various methods by which the performance evaluation device 2000 identifies the target song. For example, the performance evaluation device 2000 accepts a user input specifying the target song. Note that the user who specifies the target song may be any person, and does not have to be the user 60 who is the performer.

Figure 16 is a diagram illustrating a designation screen (song designation screen 300) for designating a target song. Like the tempo designation screen 90, the song designation screen 300 is displayed on a display device of a terminal that can be operated by the user. The song designation screen 300 includes a search area 310, a song selection area 320, and an evaluation range designation area 330. The search area 310 is an input interface for searching for a target song using search words such as the song title or attributes (genre, etc.).

The search for the target song is not limited to a search using a search word. For example, the evaluation unit 2040 may identify the target song by acquiring audio data of the target song and searching a song database using the audio data.

The search results are displayed in the song selection area 320. The user specifies the target song by selecting one of the songs displayed in the song selection area 320. In the example of FIG. 16, the song "Song F" is selected. Note that before the search is performed, candidate songs may or may not be displayed in the song selection area 320. In the former case, for example, the song selection area 320 displays all songs that can be selected as target songs. Alternatively, for example, the song selection area 320 may display a list of songs that have been selected as target songs (i.e., the history to date).

The evaluation range designation area 330 is used when it is desired to evaluate only a portion of a song. In this example, both the start and end positions can be specified. Hereinafter, the portion of the song selected to be evaluated is referred to as the "evaluation range."

There are various specific methods for specifying the evaluation range. For example, a song is divided into multiple sections (such as an intro and a chorus) in advance, and one or more sections can be specified as the evaluation range by user input. When the evaluation range can be specified on a section-by-section basis in this way, the reference information 40 includes information that can identify which section each piece of item information 44 is included in. For example, the reference information 40 further includes information such as "the name of the section, the start time of the section, and the end time of the section" for each of all sections included in the corresponding song. Then, when the evaluation range is specified on a section-by-section basis, the evaluation unit 2040 uses, for the evaluation, each piece of item information 44 included in the reference information 40 of the target song that is included in the range from the start time of the first section of the evaluation range to the end time of the last section of the evaluation range.

The method of specifying the evaluation range is not limited to the method of specifying a section. For example, the evaluation range may be specified on a measure-by-measure basis. When the evaluation range is specified on a measure-by-measure basis in this way, the reference information 40 includes information that can identify which measure each item information 44 is included in. For example, the reference information 40 further includes information such as "measure number, start time of the measure, and end time of the measure" for each of all measures included in the corresponding song. Then, when the evaluation range is specified on a measure-by-measure basis, the evaluation unit 2040 uses, for the evaluation, each item information 44 included in the reference information 40 of the target song that is included in the range from the start time of the first measure of the evaluation range to the end time of the last measure of the evaluation range.

However, for music in which the measures are of constant length, it is possible to convert the measure number into time if the length of one measure is known. Therefore, when dealing with music in which the measures are of constant length, the length of one measure may be included in the reference information 40, rather than the time range of each measure. In this case, the evaluation unit 2040 uses the length of one measure indicated by the reference information 40 to calculate the start time of the first measure in the evaluation range and the end time of the last measure in the evaluation range. The evaluation unit 2040 then identifies item information 44 whose performance time 42 falls within the range from the start time to the end time as item information 44 included in the evaluation range, and uses this for the evaluation.

In this case where only a portion of the target song is evaluated, the previously prepared reference information 40 is corrected to obtain reference information 40 that corresponds to the specified tempo. In this case, the evaluation unit 2040 does not need to correct the entire reference information 40, but only the portion that corresponds to the evaluation range.

The song selection screen 300 may further include an input interface that allows various conditions related to the evaluation to be specified. For example, an evaluation condition may be the age group of the user 60. The age group of the user 60 may be, for example, either "adult" or "child". In addition, more detailed specifications may be possible, such as "toddler" or "elementary school student". When the age group of the user 60 can be specified in this way, it is preferable to prepare reference information 40 for each age group. This is because the hand size differs depending on the age group of the performer, and this difference affects the evaluation value.

Another example of an evaluation condition is the type of instrument. For example, the types of instruments may be "acoustic piano," "electronic piano," "pipe organ," "electronic organ," or "synthesizer." When the type of instrument can be specified in this way, it is preferable to prepare reference information 40 for each type of instrument. This is because the hardness of the keys (the force required to strike the keys) differs depending on the type of instrument, and this difference affects the evaluation value.

The input interface for specifying the evaluation conditions may be provided as a screen separate from the song selection screen 300.

In addition, the music selection screen 300 and the tempo selection screen 90 may be integrated into a single screen. In other words, the music selection screen 300 may include an area for specifying the tempo.

<<Method of generating reference information 40>>
For example, the reference information 40 can be generated by analyzing the video data 20 in which a performance by a model performer is recorded, or the performance data 72. Here, a device that generates the reference information 40 is called a reference information generating device. The reference information generating device may be realized by the same computer as the performance evaluation device 2000, or may be realized by a different computer.

When performing a model performance to generate reference information 40, the model performer inputs information specifying the tempo of the performance. The reference information generating device includes information indicating the specified tempo in the reference information 40 that it generates.

When generating reference information 40 for evaluation items that can be evaluated using visual information such as the shape of hands and fingers, the reference information generating device generates the reference information 40 using video data 20. Specifically, the reference information generating device acquires video data 20 in which a performance performed by a model performer is recorded, and extracts each of the video frames 22 in which a keystroke is performed from the video frames 22 included in the video data 20. Furthermore, the reference information generating device analyzes the extracted video frames 22 to calculate item values for each evaluation item. Then, for each extracted video frame 22, the reference information generating device generates the reference information 40 by associating the performance time point corresponding to the video frame 22 with the item information obtained from the video frame 22 (multiple pairs of "evaluation item, item value").

To calculate the item value of each evaluation item from the video frame 22, the reference information generating device performs image analysis on the video frame 22. Specifically, the reference information generating device extracts information necessary to calculate the item value of each evaluation item from the video frame 22. The reference information generating device further calculates the item value of each evaluation item using the extracted information. Then, for each video frame 22, the reference information generating device generates user performance information 50 indicating a combination of the performance time point corresponding to that video frame 22 and the item value calculated for each evaluation item using that video frame 22.

The various methods described above with reference to Figures 6 to 11 can be used to calculate the item values of each evaluation item from the video frame 22. Information required to calculate the item values includes, for example, the positions of the knuckles of each finger, the wrist positions of each hand, the various reference lines described above, a line representing the posture of the arm, or a line representing the posture of the top surface of the hand.

For example, when calculating the height of finger joints, the reference information generating device detects each finger joint from the video frame 22 and identifies its position. The reference information generating device also detects the key on which each finger is placed and identifies the aforementioned reference line based on that key. The reference information generating device then calculates the height of the finger joints based on the identified finger joint positions and reference line.

When calculating the size of the angle between joint lines, the reference information generating device detects each finger joint from the video frame 22 and identifies its position. Then, for multiple joint lines that can be determined from the positions of these finger joints, the device calculates the size of the angle between each pair of two joint lines that share a common finger joint.

When calculating the wrist height, the reference information generating device detects the joint that represents the wrist position from the video frame 22 and identifies its position. The reference information generating device also identifies the key on which the hand is placed to determine the reference line, thereby identifying the reference line. The reference information generating device then calculates the wrist height based on the identified wrist position and reference line.

When calculating the height of the upper surface of the hand, the reference information generating device identifies the position of the upper surface of the hand (e.g., the position of the third joint point of the middle finger) from the video frame 22. The reference information generating device also identifies a reference line. The method of identifying the reference line can be the same as in the case of calculating the height of the wrist. The reference information generating device then calculates the height of the upper surface of the hand based on the identified position of the upper surface of the hand and the reference line.

When calculating the wrist angle, the reference information generating device detects the hand and arm from the video frame 22 and identifies a straight line representing the posture of the top surface of the hand and a straight line representing the posture of the arm. The reference information generating device then uses these straight lines to calculate the wrist angle.

When calculating the keystroke depth, the reference information generating device detects the key being pressed from the video frame 22. Then, for each detected key, the reference information generating device calculates a line segment representing the long side of the top surface of the key that is farther from the camera 10 in both the pressed state and the normal state. The reference information generating device then uses these line segments to calculate the keystroke depth of each key. Note that information about the normal state of a key can be obtained from the line segment on the top side of the key in the video frame 22 when the key is not pressed, or from the line segment on the top side of the key next to the key that is farther from the camera 10 in the video frame 22 when the key is pressed.

As described above, the world coordinate system may be used to calculate each item value. In this case, the reference information generating device converts the information detected from the video frame 22 (such as the position of the finger joints) into information in the world coordinate system, as described above, and then calculates the item value.

Fingering is also identified using the video data 20. Specifically, the reference information generating device detects the pressed key and the finger that pressed that key from the video frame 22, and generates reference information 40 based on the detection results. Performance time 42 indicates the performance time corresponding to the video frame 22. Evaluation item 46 indicates the name of the evaluation item "fingering" and the identification information of the note corresponding to the detected key. Item value 48 indicates the identification information of the detected finger.

In order to be able to identify the pressed key from the video frame 22, for example, information is created in advance that represents the correspondence between the identification information of the key and the position of the image area representing that key on the video frame 22. By referring to this information, it is possible to identify the identification information of the pressed key from the position of that key on the video frame 22.

Here, in order to generate reference information 40 using a video frame 22, it is necessary to identify the performance time corresponding to that video frame 22. For example, as described above, the performance time is expressed as a relative time based on the time when the first performance (keystroke) is performed in the target song. In this case, for example, the reference information generating device sets the generation time of the video frame 22 representing the first performance among the video frames 22 included in the acquired video data 20 as the reference time. For the other video frames 22, the reference information generating device converts the generation time of the video frame 22 for which the performance time is to be identified to the performance time based on the difference between the generation time of the video frame 22 for which the performance time is to be identified and the generation time of the video frame 22 representing the reference time, and the frame rate of the camera 10.

The method of identifying the time when the target song is first played is not limited to using the time when the video frame 22 representing the first performance is generated. For example, when the model performer plays the target song, the sound of a metronome can be output so that the timing of the start of the performance and the tempo of the performance can be easily understood. In this case, the reference information generating device can identify the time when the target song is first played based on the sound of the metronome.

For example, if the target piece is in 4/4 time, the model performer will start playing after the metronome has sounded four times. In this case, if the time when the fourth metronome sound is called ta and the time interval between the metronome sounds is called i, the time ta+i can be identified as the time when the first performance will take place.

The metronome sound may be output from the reference information generating device, or from something else (e.g., a mechanical metronome). In the latter case, the reference information generating device uses audio data in which the metronome sound is recorded to determine the time when the metronome sound is generated and the time interval between metronome sounds.

When generating reference information 40 for evaluation items related to sound characteristics, for example, the reference information generating device uses performance data 72 in which a performance by a model performer is recorded. The performance data 72 includes information that can identify the time of performance, sound identification information, and various sound characteristics (strength, duration, etc.) for each sound played. In MIDI (registered trademark) terminology, sound identification information and sound strength are called "note number" and "velocity," respectively.

When generating reference information 40 using performance data 72, for example, the reference information generating device can generate a record of reference information 40 regarding sound intensity by extracting, from the acquired performance data 72, information about each played note, such as the playing time, the note number, and the velocity of that note. Similarly, the reference information generating device can generate a record of reference information 40 regarding sound duration by extracting, from the acquired performance data 72, information about each played note, such as the playing time, the note number, and the duration of that note.

A more specific explanation will be given of the case where MIDI (registered trademark) data is used as performance data. In MIDI (registered trademark) data, the output and stopping of a specific sound are represented by a note-on message, which indicates that the sound is to be played, and a note-off message, which indicates that the sound is to be stopped. The strength of the sound can be specified by the velocity value indicated in the note-on message. Meanwhile, the length of the sound can be specified by the difference between the time when the note-on message is played and the time when the note-off message is played.

For another example, the reference information generating device may obtain video data 20 in which a performance by a model performer is recorded, and use the video data 20 to generate reference information 40 regarding sound characteristics. Here, one of the external characteristics of the performance that affects the sound intensity is the speed of keystrokes. Specifically, the faster the keystroke speed, the stronger the sound intensity. Thus, by analyzing the video data 20, the basic information generating device identifies the "keystroke time (performance time), identification information of the keystroke sound, and keystroke speed" for each keystroke performed by the model performer. This makes it possible to generate reference information 40 regarding the intensity of each sound played. Note that the performance time may be represented as the start or end of the keystroke action.

The speed at which a key is pressed can be expressed, for example, as the reciprocal of the difference between the start time ts of the keystroke and the end time te of the keystroke, 1/(te-ts). The start time of the keystroke can be expressed as the generation time of the video frame 22 that represents the start of the keystroke (the video frame 22 at which the key begins to move down). Similarly, the end time of the keystroke can be expressed as the generation time of the video frame 22 that represents the end of the keystroke (the video frame 22 at which the key has moved all the way down).

Another example of an external characteristic of a performance that affects the strength of a sound is the depth of keystrokes. Specifically, the deeper the keystroke, the louder the sound. In addition, some types of musical instruments do not produce sound unless the key is struck to a certain depth. The basic information generating device analyzes the video data 20 to identify the "keystroke time (play time), identification information of the struck sound, and keystroke depth" for each keystroke performed by the model performer. This makes it possible to generate reference information 40 regarding the strength of each sound played. The play time may be represented as the start or end of the keystroke. In addition, the depth of keystrokes can be represented using the angle of the reference line and the length of the line segment between the reference points, as described in FIG. 11 of the first embodiment.

Note that the reference information 40 regarding sound intensity may indicate only either the keystroke speed or the keystroke depth, or may indicate both.

When the video data 20 is used to determine the length of a sound, it can be determined, for example, as the difference between the time when a key is pressed and the time when the key is released. The reference information generating device analyzes the video data 20 to determine, for each keystroke performed by the model performer, the "time when the key was pressed (time when the performance was performed), the identification information of the pressed sound, and the time when the key was released." This makes it possible to generate reference information 40 regarding the length of each sound played.

Information obtained from the performances of multiple model performers may be used to generate the reference information 40. For example, the reference information generating device generates reference information 40 for each of multiple model performers in the manner described above, and then performs statistical processing on the multiple pieces of generated reference information 40 to generate reference information 40 to be used in evaluating the performances. For example, it is conceivable to use statistics (such as average values) of item values obtained for each of multiple people.

<Acquisition of user performance information 50: S106>
The evaluation unit 2040 acquires the user performance information 50 (S106). The user performance information 50 indicates, for a performance of the target piece of music performed by the user 60, a combination of evaluation items and item values at each of a plurality of performance times, in association with each of the performance times.

Figure 17 is a diagram illustrating an example of user performance information 50 in table format. The configuration of the user performance information 50 shown in Figure 17 is the same as the configuration of the reference information 40 shown in Figure 15. Specifically, the user performance information 50 has item information 54 for each performance time point 52. The item information 54 indicates, for each evaluation item, a pair of an evaluation item 56 and an item value 58 for the performance at the corresponding performance time point. The evaluation item 56 indicates identification information that specifies the evaluation item (e.g., the name of the evaluation item). The item value 58 indicates the item value of the corresponding evaluation item.

The generation of the user performance information 50 may be performed by the performance evaluation device 2000, or may be performed by a device other than the performance evaluation device 2000. In the former case, for example, the performance evaluation device 2000 acquires video data 20 generated by capturing a performance by the user 60 with the camera 10, and performance data 72 representing the performance given by the user 60, and analyzes these to generate the user performance information 50. Here, the method of generating the user performance information 50 using the video data 20 is the same as the method of generating the reference information 40 using the video data 20. Also, the method of generating the user performance information 50 using the performance data 72 is the same as the method of generating the reference information 40 using the performance data 72.

As mentioned above, when generating reference information using the video data 20, the sound of a metronome may be used to make it easier to grasp the timing of the start of performance and the tempo of the performance. In this regard, the sound of a metronome may also be used when the user 60 plays the target song. In this case, the evaluation unit 2040 can use the sound of the metronome to identify the time when the user 60 first performed the song (i.e., time 0 in the user performance information 50).

For example, if the target song is in 4/4 time, user 60 starts playing after the metronome sounds four times. This allows user 60 to easily grasp the correct timing to start playing. In particular, if the evaluation range starts in the middle of the target song, it may be more difficult to start playing compared to playing from the beginning of the song. In this regard, by starting playing after listening to the metronome sound, it becomes possible to start playing more naturally.

The metronome sound may be set to continue to be output while the user 60 is performing. This makes it easier to maintain the correct tempo while performing. However, since there may be cases where the tempo of the performance needs to be evaluated, it may be possible to set the metronome sound not to be output while playing.

The metronome sound may be output from the performance evaluation device 2000, or may be output from something else (e.g., a mechanical metronome). In the former case, the performance evaluation device 2000 sets the tempo of the metronome to the specified tempo indicated in the tempo information 100.

The evaluation unit 2040 can use various methods to acquire the video data 20. For example, when the performance evaluation device 2000 is realized by a computer other than the camera 10, the evaluation unit 2040 acquires the video data 20 by receiving the video data 20 transmitted from the camera 10 or by accessing a storage device in which the video data 20 is stored and reading the video data 20. Note that the storage device in which the video data 20 is stored may be provided either inside or outside the camera 10. When the performance evaluation device 2000 is realized by the camera 10, the evaluation unit 2040 acquires the video data 20 by reading the video data 20 stored inside the performance evaluation device 2000 (e.g., the storage device 1080).

Various methods can be used to acquire the performance data 72. For example, the evaluation unit 2040 acquires the performance data 72 by receiving the performance data 72 transmitted by a device (such as the target musical instrument 70) that generated the performance data 72, or by accessing a storage device in which the performance data 72 is stored and reading out the performance data 72.

When the user performance information 50 is generated by a device other than the performance evaluation device 2000, the evaluation unit 2040 acquires the user performance information 50 generated by the other device. Various methods can be used for this acquisition. For example, the evaluation unit 2040 acquires the user performance information 50 by receiving the user performance information 50 transmitted from the device that generated the user performance information 50, or by accessing a storage device in which the user performance information 50 is stored and reading out the user performance information 50.

<Generation of evaluation information 30: S108>
The evaluation unit 2040 generates the evaluation information 30 using the reference information 40 and the user performance information 50. Specifically, the evaluation unit 2040 associates the item information 44 included in the reference information 40 with the item information 54 included in the user performance information 50, and compares the item information 44 and the item information 54 that correspond to each other. The corresponding item information 44 and the item information 54 are pieces of information that correspond to each other at the same or close performance time points, and should represent the same performance action (keystroke, etc.). More specifically, for example, the evaluation unit 2040 calculates an evaluation score that indicates the degree of agreement of the item values for each evaluation item for the corresponding item information 44 and the item information 54. Then, the evaluation unit 2040 generates the evaluation information 30 using the calculated evaluation score. Note that a method for calculating the evaluation score and a method for generating the evaluation information 30 using the evaluation score will be described later.

<<Process flow>>
FIG. 18 is a flow chart illustrating a process flow for calculating an evaluation score for each item information 44. The loop process L1 is a loop process executed for each item information 44 included in the reference information 40. In S202, the evaluation unit 2040 judges whether or not the loop process L1 has been executed for all the item information 44. If the loop process L1 has already been executed for all the item information 44, the process of FIG. 18 ends. On the other hand, if there is one or more item information 44 that has not yet been the target of the loop process L1, the evaluation unit 2040 selects one item information 44 from among them. For example, the evaluation unit 2040 selects the item information 44 as the target of the loop process L1 in order from the item information 44 whose corresponding performance time 42 is the earliest. The item information 44 selected in S202 is denoted as item information 44-i.

The evaluation unit 2040 identifies the item information 54 that corresponds to the item information 44-i from among the item information 54 included in the user performance information 50 (S204). For example, the evaluation unit 2040 identifies the performance time 52 that is closest to the performance time 42 that corresponds to the item information 44-i from among the performance times 52 indicated in the user performance information 50. The evaluation unit 2040 then identifies the item information 54 that corresponds to the identified performance time 52 as the item information 54 that corresponds to the item information 44-i.

The evaluation unit 2040 calculates an evaluation score for each evaluation item by comparing the identified item information 54 with the item information 44-i (S206). Since S208 is the end of the loop process L1, the process in FIG. 18 proceeds to S202.

Note that if there is no performance time 52 whose difference from the performance time 42 corresponding to the item information 44-i is equal to or less than a predetermined value (in other words, if the difference between the performance time 42 and the performance time 52 closest to the performance time 42 is greater than a predetermined value), the evaluation unit 2040 may determine that there is no item information 54 corresponding to the item information 44-i, and may not execute S206 for that item information 44-i. This may occur, for example, when a sound is missing in the performance by the user 60.

<<How the evaluation score is calculated>>
The evaluation score represents the degree of agreement between item value 58 indicated in item information 54 and item value 48 indicated in item information 44 for a target evaluation item. For example, an evaluation formula for calculating an evaluation score is prepared in advance for each evaluation item. The value indicated by item value 48 in item information 44 and the value indicated by item value 58 in item information 54 for a target evaluation item are input into the evaluation formula. The evaluation formula then outputs an evaluation score by performing a predetermined calculation on the input values.

For example, the evaluation formula outputs a value that represents the difference between item value 48 and item value 58, or the ratio of item value 58 to item value 48. The difference or ratio may be standardized.

For example, the evaluation formula may be configured to output a determination result as to whether or not item value 58 is included in an allowable range determined based on item value 48. For example, the allowable range may be determined to be "the value indicated by item value 48 ±20% (0.8 times or more the value indicated by item value 48, and 1.2 times or less the value indicated by item value 48)." The evaluation formula acquires item value 48 and item value 58, and determines whether or not item value 58 is included in the allowable range based on item value 48. The evaluation formula then outputs a value that represents the determination result (for example, 1 if it is included in the allowable range, 0 if it is not included).

Alternatively, for example, a number of numerical ranges may be defined based on the item value 48, and the performance of the user 60 may be evaluated with a rank. For example, assume that a numerical range A of "±20% of the value indicated by the item value 48," a numerical range B of "±40% of the value indicated by the item value 48, outside numerical range A," and a numerical range C of "outside numerical range B" are defined. In this case, when the item value 58 is included in numerical range A, the evaluation is the highest, when the item value 58 is included in numerical range B, the evaluation is the second highest, and when the item value 58 is included in numerical range C, the evaluation is the lowest. Thus, a rank representing such a high evaluation is assigned to each numerical range. For example, a rank represented by a smaller natural number is assigned to a numerical range with a higher evaluation. In the above example, ranks 1, 2, and 3 are assigned to the numerical ranges A to C, respectively.

When performing an evaluation using ranks in this manner, the evaluation formula acquires item value 48 and item value 58, and identifies the numerical range that includes item value 58 from among the multiple numerical ranges. The evaluation formula then outputs a rank that corresponds to the identified numerical range. For example, in the example described above, if item value 58 is included in numerical range B, a rank of "2" is output.

The evaluation score for fingering represents the result of the determination as to whether the fingering shown in the reference information 40 matches the fingering shown in the user performance information 50. That is, for corresponding item information 44 and item information 54, the evaluation unit 2040 determines whether the finger identification information shown in item value 48 matches the finger identification information shown in item value 58, and generates an evaluation score that represents the result of the determination. For example, the evaluation score indicates "1" if the fingering matches, and indicates "0" if the fingering does not match.

<<Generation of Evaluation Information 30 Using Evaluation Scores>>
The evaluation unit 2040 uses the evaluation scores to generate evaluation information 30. For example, the evaluation unit 2040 generates evaluation information 30 indicating all of the calculated evaluation scores. More specifically, the evaluation unit 2040 generates evaluation information 30 indicating a plurality of pairs of "evaluation item, evaluation score" generated by comparing item information 44 and item information 54 corresponding to each time point (performance time point 42) at which a key is struck in the target song.

Figure 19 is a diagram illustrating an example of evaluation information 30 in table format. The table shown in Figure 19 is denoted as table 400. Table 400 shows a performance time 402 and an evaluation result 404. The evaluation result 404 shows a pair of evaluation item 406 and evaluation score 408. The evaluation score 408 shows the evaluation score for the corresponding evaluation item 406 for the performance of the user 60 at the corresponding performance time 402.

For example, the evaluation unit 2040 may generate the evaluation information 30 by aggregating the evaluation scores in various ways. For example, the evaluation unit 2040 calculates, for each performance time point, a statistical value (such as an average value) of the evaluation scores calculated for the item information 54 corresponding to that performance time point. In this case, a score representing an overall evaluation of the performance at each time point is obtained. Hereinafter, the statistical value of the evaluation score is referred to as the statistical score. When the importance of each evaluation item differs, the statistical score may be a value that takes into account the importance of each evaluation item (for example, a weighted average).

Figure 20 is a diagram illustrating evaluation information 30 showing a statistical score for each performance time point. The table shown in Figure 20 is denoted as table 500. Table 500 shows pairs of performance time points 502 and statistical scores 504. Statistical scores 504 are statistical values of multiple evaluation scores calculated for the corresponding performance time points 502.

The evaluation scores may be tallied for each evaluation item. That is, the evaluation unit 2040 calculates, for each evaluation item, a statistical value of the evaluation scores calculated for each evaluation item at multiple performance points. In this case, an overall evaluation of the user 60's performance is shown for each evaluation item.

FIG. 21 is a diagram illustrating an example of evaluation information 30 indicating a statistical score for each evaluation item. The table shown in FIG. 21 is denoted as table 600. Table 600 indicates a pair of evaluation item 602 and statistical score 604. Statistical score 604 is a statistical value of multiple evaluation scores calculated for the corresponding evaluation item 602.

In the example of FIG. 21, the evaluation scores are tallied for the entire evaluation range. However, the evaluation scores may be tallied for multiple partial ranges included in the evaluation range. For example, the partial ranges are sections or measures. When tallying for each partial range, the reference information 40 includes information that can identify each partial range (for example, information that can identify a section). The evaluation unit 2040 identifies each partial range with reference to the reference information 40, and thereby groups the evaluation scores for each partial range. The evaluation unit 2040 then tallies the evaluation scores for each partial range (for example, calculates a statistical score).

FIG. 22 is a diagram illustrating evaluation information 30 that indicates statistical scores for each subrange and for each evaluation item. The table shown in FIG. 22 is denoted as table 700. Table 700 includes subranges 702 and evaluation results 704. Evaluation results 704 include evaluation items 706 and statistical scores 708. Statistical scores 708 indicate statistical values calculated from the evaluation scores for the corresponding evaluation items 706, out of the evaluation scores calculated for the performance time points included in the corresponding subranges 702.

When evaluation items can be classified into multiple groups, the evaluation scores may be tallied for each group of evaluation items. For example, evaluation items related to the hand can be classified into evaluation items related to the right hand and evaluation items related to the left hand. Furthermore, evaluation scores for evaluation items related to the fingers can be classified into evaluation scores for each of the thumb, index finger, middle finger, ring finger, and little finger.

Figure 23 is a diagram illustrating an example of the statistical values of the evaluation scores tallied for each group. The table shown in Figure 23 is called table 800. Table 800 includes a performance time 802 and an evaluation result 804. The evaluation result 804 includes an item group 806 and an evaluation score 808. The evaluation score 808 indicates the statistical values of the evaluation scores of each evaluation item belonging to the item group 806 for the performance at the corresponding performance time 802.

In this way, even in cases where evaluation scores are tallied for each group, statistical scores may be calculated for the entire evaluation range or a partial range, rather than for each performance time point.

<<Detection of tempo deviation>>
The evaluation unit 2040 may detect a deviation in the tempo of the performance (hereinafter, tempo deviation) by comparing the reference information 40 with the user performance information 50. "A deviation in the tempo of the performance" means that the order of the notes being played is correct, but the playing time is deviated from the ideal time.

If user 60 cannot play the target piece of music at the correct tempo, it can be said that user 60's proficiency has not reached a level where detailed evaluations such as evaluation of the shape of the hands and fingers and evaluation of the characteristics of the sound can be performed. Therefore, before performing these evaluations, it is preferable to practice at a slower tempo to prevent tempo deviations from occurring. Therefore, when the evaluation unit 2040 detects a tempo deviation, it may output a message encouraging the user to practice at a slower tempo.

The method by which the evaluation unit 2040 detects a tempo deviation is arbitrary. For example, the evaluation unit 2040 divides the time axis into a number of blocks, and compares the sounds shown in the reference information 40 and the sounds shown in the user performance information 50 for each block. Here, if the multiple sounds shown in the reference information 40 and the multiple sounds shown in the user performance information 50 match in both the type and order of the sounds in a certain block, but the performance times are different, it can be said that a tempo deviation has occurred. For example, in a certain block, it is assumed that the four sounds "do-re-mi-fa" are played in this order in both the reference information 40 and the user performance information 50. In this situation, if the performance times of any one or more sounds are different between the reference information 40 and the user performance information 50 (for example, the difference in the performance times is a predetermined value or more), it can be said that a tempo deviation has occurred. Therefore, the evaluation unit 2040 detects a tempo deviation by comparing the reference information 40 and the user performance information 50 for each block and examining the above-mentioned situation.

<Output of evaluation information 30>
The performance evaluation device 2000 outputs the evaluation information 30 in an arbitrary manner. For example, the performance evaluation device 2000 stores the evaluation information 30 in a storage device. As another example, the performance evaluation device 2000 displays the evaluation information 30 on a display device. As another example, the performance evaluation device 2000 may transmit the evaluation information 30 to another device. For example, when the performance evaluation device 2000 is realized as a server machine 80 that operates by receiving video data 20 from a user terminal 62 (see FIG. 12(b)), the performance evaluation device 2000 transmits the evaluation information 30 to the user terminal 62. Then, the evaluation information 30 can be viewed in various ways on the user terminal 62.

In addition, instead of outputting the evaluation information 30 as is, the performance evaluation device 2000 may process the evaluation information 30 as appropriate and output it. For example, the performance evaluation device 2000 uses the evaluation information 30 to generate an evaluation result screen and displays the evaluation result screen on a display device.

Figure 24 is a first diagram illustrating an example of an evaluation result screen. The evaluation result screen 900 in Figure 24 shows a graph generated from the evaluation information 30. In this example, a statistical score is calculated for each section and for each evaluation item. Also, in this example, the time progression of the statistical score for each evaluation item is represented by a line graph.

In addition, in FIG. 24, statistical scores that represent particularly poor evaluations are highlighted. Specifically, the threshold value for the statistical scores is indicated by a dashed line, and statistical scores that are smaller than the threshold value are highlighted with a dotted mark. Note that statistical scores that represent particularly good evaluations may be highlighted in a similar manner. Specifically, statistical scores that are larger than a predetermined threshold value are highlighted.

Figure 25 is a second diagram illustrating an example of an evaluation result screen. The evaluation result screen 1300 in Figure 25 shows a graph generated from the evaluation information 30. In this example, a statistical score is calculated for each finger used in playing and for each evaluation item.

In this way, by tallying up the item values of each evaluation item for each finger used in playing, it becomes easier to understand, for example, the habits of each hand or finger. For example, looking at the graph in Figure 25, we can see that for both evaluation items A and B, the scores for performances using the left hand are lower than the scores for performances using the right hand. This shows that the technique of playing with the left hand should be improved.

For example, the graph in Figure 25 shows that for both the right and left hands, the scores for playing using the little finger are lower than the scores for playing using the other fingers. This shows that the technique of playing with the little finger should be improved.

The performance evaluation device 2000 may generate three-dimensional hand shape modeling images for the performance of the user 60 and the performance of the model performer, and output a screen including the images, or store the video data of the images in a storage device. The hand modeling images for the performance of the user 60 can be generated based on information such as the joint points of each finger and wrist obtained from the user performance information 50. Similarly, the hand modeling images for the model performer can be generated based on information such as the joint points of each finger and wrist obtained from the reference information 40.

It is preferable that the performance evaluation device 2000 outputs the evaluation information 30 together with the modeling image. Here, it is preferable that the evaluation result for each evaluation item is shown in association with a portion of the modeling image related to the evaluation item. For example, the evaluation result for the wrist height is displayed near the corresponding wrist. Similarly, the evaluation result for each finger joint point is displayed near the corresponding finger.

By showing the evaluation information 30 along with the hand modeling video in this way, the user 60 can intuitively visually grasp the differences between the model performer's performance and his or her own. This allows the user 60 to easily recognize the points that need to be corrected (practiced).

In the above embodiment, the program can be stored and supplied to the computer using various types of non-transitory computer readable media. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of the transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can supply the program to the computer via wired communication paths such as electric wires and optical fibers, or wireless communication paths.

The present invention has been described above with reference to the embodiment, but the present invention is not limited to the above embodiment. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

10 Camera 20 Video data 30 Evaluation information 40 Reference information 42 Playing time 44 Item information 46 Evaluation item 48 Item value 50 User performance information 52 Playing time 54 Item information 56 Evaluation item 58 Item value 60 User 62 User terminal 70 Target instrument 72 Performance data 80 Server machine 90 Tempo designation screen 100 Tempo information 110 Finger 120 Key 130 Reference line 140 Hand 150 Reference line 160 Key 170 Straight line 180 Straight line 190 Key 200 Line segment 210 Line segment 300 Song designation screen 310 Search area 320 Song selection area 330 Evaluation range designation area 400 Table 402 Playing time 404 Evaluation result 406 Evaluation item 408 Evaluation score 500 Table 502 Playing time 504 Statistical score 600 Table 602 Evaluation item 604 Statistical score 700 Table 702 Partial range 704 Evaluation result 706 Evaluation item 708 Statistical score 800 Table 802 Performance time 804 Evaluation result 806 Item group 808 Evaluation score 900 Evaluation result screen 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input/output interface 1120 Network interface 1300 Evaluation result screen 2000 Performance evaluation device 2020 Tempo information acquisition unit 2040 Evaluation unit

Claims (4)

On the computer,
A tempo information acquisition process for acquiring tempo information capable of identifying a designated tempo, which is a performance tempo desired by a user;
an evaluation process for generating evaluation information including an evaluation of a performance by the user by comparing standard information, which indicates a standard for evaluating a performance when the performance is performed at the specified tempo, with user performance information, which indicates a performance of the musical piece by the user,
the standard information is prepared for each piece of music in association with a plurality of tempos, the standard information indicating a standard for evaluating a performance when the piece of music is performed at each tempo;
The evaluation process is a program for obtaining standard information corresponding to the specified tempo from a plurality of standard information prepared for the piece of music to be evaluated, and generating the evaluation information using the obtained standard information . the standard information is prepared for each piece of music in association with a specific tempo, the standard information indicating a standard for evaluating a performance when the piece of music is performed at that tempo;
The evaluation process includes:
acquiring information indicating the specific tempo and the reference information for the piece of music to be evaluated;
converting the acquired reference information into reference information that represents an evaluation standard when the piece of music to be evaluated is played at the specified tempo based on a difference between the specific tempo and the specified tempo;
The program according to claim 1 , further comprising: generating the evaluation information by utilizing the reference information obtained by the conversion. a tempo information acquisition unit for acquiring tempo information capable of identifying a designated tempo, which is a performance tempo desired by a user;
an evaluation unit that generates evaluation information including an evaluation of the performance by the user by comparing standard information that indicates a standard for evaluating a performance when the musical piece is played at the specified tempo with user performance information that indicates the performance of the musical piece by the user,
the standard information is prepared for each piece of music in association with a plurality of tempos, the standard information indicating a standard for evaluating a performance when the piece of music is performed at each tempo;
The evaluation unit acquires standard information corresponding to the specified tempo from a plurality of standard information prepared for the piece of music to be evaluated, and generates the evaluation information using the acquired standard information . A computer-implemented performance evaluation method, comprising the steps of:
a tempo information acquisition step of acquiring tempo information capable of identifying a designated tempo, which is a performance tempo desired by a user;
an evaluation step of generating evaluation information including an evaluation of the performance by the user by comparing, for a piece of music to be evaluated, standard information that indicates a standard for evaluating a performance when the piece of music is performed at the specified tempo with user performance information that indicates a performance of the piece of music by the user;
the standard information is prepared for each piece of music in correspondence with a plurality of tempos, the standard information indicating a standard for evaluating a performance when the piece of music is performed at each tempo;
The evaluation step is a performance evaluation method in which standard information corresponding to the specified tempo is obtained from a plurality of standard information prepared for the piece of music to be evaluated, and the evaluation information is generated using the obtained standard information .

Images