WO2026018406A1 - Information processing device and moving image display method - Google Patents

Abstract

A moving image display unit 116 reproduces a moving image. A thumbnail display unit 118 displays a thumbnail of moving images related to the displayed image. The thumbnail display unit 118 changes the thumbnail of moving images to be displayed, in accordance with the image displayed by the moving image display unit 116.

Description

Information processing device and video display method

This disclosure relates to technology for allowing users to watch videos.

Videos of games played by various players are uploaded to the video sharing server. When a user enters a string into the search box of a video sharing site, the video sharing server provides the user with game videos that match the entered string, and the user can then play the provided game videos.

While watching a video, thumbnails of other videos, known as related videos, may appear near the video being played. Users can view related videos by selecting the thumbnail of a related video, but the related videos are often not very relevant to the video being played. This disclosure therefore provides a mechanism for presenting videos that are highly relevant to the video being played.

One aspect of the present disclosure is an information processing device with a function for playing videos, which includes a video display unit that plays videos and a thumbnail display unit that displays thumbnails of videos related to the displayed image. The thumbnail display unit changes the thumbnail of the video that it displays depending on the image displayed by the video display unit.

Another aspect of the present invention is a method for displaying a video on an information processing device, comprising the steps of playing the video, displaying a thumbnail of the video associated with the displayed image, and, when the displayed image is changed, displaying a thumbnail of the video associated with the changed image.

In addition, any combination of the above components, and any conversion of the expression of this disclosure into a method, device, system, recording medium, computer program, etc., are also valid aspects of this disclosure.

FIG. 1 is a diagram illustrating an information processing system according to an embodiment. FIG. 1 is a diagram illustrating a hardware configuration of an information processing device. FIG. 2 is a diagram illustrating an example of functional blocks of an information processing device. FIG. 10 is a diagram illustrating an example of a game screen. FIG. 2 is a diagram illustrating an example of functional blocks of a video sharing server. 10A and 10B are diagrams illustrating examples of letters and numbers extracted from a game image. FIG. 10 is a diagram showing an example of dividing a game video. FIG. 10 is a diagram illustrating an example of a home screen of a video sharing site. FIG. 10 is a diagram showing an example of a selection screen for a plurality of candidate videos. FIG. 10 is a diagram showing an example of a playback screen of a game video. FIG. 10 is a diagram showing an example of a playback screen of a game video. FIG. 10 is a diagram showing another example of a playback screen of a game video.

In this embodiment, an information processing device, which is a user terminal device, executes a game program. When an event occurs in the game, the game program outputs event information identifying the event that has occurred to the system software. An event occurs when there is a change in the progress of game play or when there is a change in the behavior of the player character controlled by the player. The game program incorporates multiple play units, each of which has at least a set end condition.

A game program includes activities, which are a type of unit of play and have set start and end conditions. Activities may be made up of stages, quests, missions, tournaments, sessions, etc. Activities are set as appropriate by the game maker, and one quest that appears in the course of the game may constitute one activity, or multiple quests may constitute one activity. When multiple quests make up one activity, completing all of the multiple quests is the end condition for that activity.

When the game program starts an activity, it outputs to the system software an activity identifier (activity ID) that identifies the activity and event information that includes information indicating the start of the activity. When the game program ends the activity, it outputs to the system software event information that includes the activity ID and information indicating the end of the activity. When the activity ends, the game program may include the result of the activity (success or failure) in the event information.

The system software generates event data by adding a user identifier (user account) that identifies the user, a game identifier (game ID) that identifies the game, and time information (timestamp) to the event information output from the game program, and transmits this to an external server device. The server device collects event data transmitted from multiple information processing devices operated by multiple players, and analyzes play trends related to activities. For example, the server device may statistically calculate the average play time of an activity based on the event data transmitted from multiple information processing devices.

In one embodiment, the server device acquires video of a game played by multiple players and analyzes the game video. The server device performs image analysis of the game video to acquire state values in the game. The state values may be values that indicate the state of a scene in the game, or the static and/or dynamic state of a player character controlled by a player. The state value of a player character may be the HP value of the player character or the name of an action that indicates an action performed by the player character. The server device associates the acquired state value with the time that the state value occurred, and sets this as metadata for the game video. The server device may embed metadata related to the state values in the game video, or it may store it in a database.

In one embodiment, the server device provides users with a video sharing service that distributes game videos uploaded by multiple players. When a user enters a string of characters into a search box on a video sharing site, the video sharing server searches for game videos that match the entered string and provides the searched game videos to the user.

FIG. 1 shows an information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 of the embodiment is a system that allows a user to watch game videos, and includes an information processing device 10 operated by the user and a server device 5. An access point (hereinafter referred to as "AP") 8 functions as a wireless access point and a router, and the information processing device 10 connects to the AP 8 wirelessly or via a wired connection to communicate with the server device 5 on the network 3. While FIG. 1 shows one user and one information processing device 10, in the information processing system 1, multiple information processing devices 10 operated by multiple users are connected to the server device 5 via the network 3.

The information processing device 10 is connected wirelessly or via a wire to an input device 6 operated by a user, and the input device 6 outputs information operated by the user to the information processing device 10. When the information processing device 10 receives operation information from the input device 6, it reflects the information in the processing of the system software and game program, and outputs the processing results from the output device 4. In the information processing system 1, the information processing device 10 is a game device (game console) that executes the game, and the input device 6 is a device such as a game controller that supplies user operation information to the information processing device 10. The input device 6 may also be an input interface such as a keyboard or mouse.

The auxiliary storage device 2 is a large-capacity storage device such as an HDD (hard disk drive) or SSD (solid state drive), and may be an internal storage device, or an external storage device connected to the information processing device 10 via a USB (universal serial bus) or the like. The output device 4 may be a television having a display for outputting images and speakers for outputting sound. The output device 4 may be connected to the information processing device 10 via a wired cable, or may be connected wirelessly.

Camera 7, which is an imaging device, is installed near output device 4 and captures images of the space around output device 4. While Figure 1 shows an example in which camera 7 is attached to the top of output device 4, it may also be placed on the side or bottom of output device 4, and in either case, it is placed in a position where it can capture an image of a user positioned in front of output device 4. Camera 7 may also be a stereo camera.

The server device 5 of the embodiment includes a management server 12 and a video sharing server 14. In the following, the management server 12 and the video sharing server 14 are described as separate, distinct servers, but the management server 12 and the video sharing server 14 may also be integrated into one server.

The management server 12 provides network services to users of the information processing system 1. The management server 12 manages user identifiers (user accounts) that identify each user, and each user signs in to the network services provided by the management server 12 using their user account. By signing in to the network service from their information processing device 10, users can register game save data and trophies, which are virtual rewards acquired during game play, on the management server 12. By registering save data and trophies on the management server 12, users can synchronize save data and trophies even if they use an information processing device other than the information processing device 10.

The management server 12 collects event data from multiple information processing devices 10 operated by multiple players (users). The management server 12 records the event data by linking it to user accounts. The management server 12 may perform statistical processing of activity play times from the collected event data to calculate the average play time of an activity, etc.

The video sharing server 14 provides a video sharing service that distributes videos of games played by multiple players. In this embodiment, the video sharing server 14 has the function of analyzing game videos uploaded by players and generating metadata for the game videos.

The video sharing server 14 performs image analysis of the game video to acquire game status values. The video sharing server 14 has an optical character recognition (OCR) function, and may extract the number of combos achieved in battle scenes and the player character's HP (hit points) from the game video to acquire these as game status values. The video sharing server 14 may also have an object detection function, and may detect enemy characters that appear and items that are obtained, and acquire these as game status values. The video sharing server 14 may also have an action recognition function, and may detect the actions of the player character to acquire these as game status values.

The video sharing server 14 associates the acquired status value with the time when the status value occurred and sets it as metadata for the game video. The video sharing server 14 may embed the metadata in the game video, or may store it in a database. Note that analysis of the game video may not be performed by the video sharing server 14, but may instead be performed by the management server 12 or another server.

Figure 2 shows the hardware configuration of the information processing device 10. The information processing device 10 is composed of a main power button 20, a power ON LED 21, a standby LED 22, a system controller 24, a clock 26, a device controller 30, a media drive 32, a USB module 34, a flash memory 36, a wireless communication module 38, a wired communication module 40, a subsystem 50, and a main system 60.

The main system 60 comprises a main CPU (Central Processing Unit), memory and memory controller as main storage devices, a GPU (Graphics Processing Unit), etc. The GPU is primarily used for processing game programs. The main CPU has the function of starting up the system software and executing game programs installed on the auxiliary storage device 2 in the environment provided by the system software. The subsystem 50 comprises a sub-CPU, memory and memory controller as main storage devices, etc., but does not comprise a GPU.

While the main CPU has the function of executing game programs installed on the auxiliary storage device 2, the sub-CPU does not have such a function. However, the sub-CPU does have the function of accessing the auxiliary storage device 2 and the function of sending and receiving data with the server device 5. The sub-CPU is configured with only these limited processing functions, and therefore can operate with less power consumption than the main CPU. These functions of the sub-CPU are executed when the main CPU is in standby mode.

The main power button 20 is an input unit that receives operational input from the user. It is located on the front of the housing of the information processing device 10 and is operated to turn on or off the power supply to the main system 60 of the information processing device 10. The power ON LED 21 lights up when the main power button 20 is turned on, and the standby LED 22 lights up when the main power button 20 is turned off. The system controller 24 detects when the main power button 20 is pressed by the user.

The clock 26 is a real-time clock that generates current date and time information and supplies it to the system controller 24, subsystem 50, and main system 60.

The device controller 30 is configured as an LSI (Large-Scale Integrated Circuit) that, like a southbridge, transfers information between devices. As shown in the figure, devices such as the system controller 24, media drive 32, USB module 34, flash memory 36, wireless communication module 38, wired communication module 40, subsystem 50, and main system 60 are connected to the device controller 30. The device controller 30 absorbs differences in the electrical characteristics and data transfer speeds of each device and controls the timing of data transfer.

The media drive 32 is a drive device that operates by loading a ROM medium 44 that stores application software such as games and license information, and reads programs, data, and the like from the ROM medium 44. The ROM medium 44 is a read-only recording medium such as an optical disc, magneto-optical disc, or Blu-ray disc.

The USB module 34 is a module that connects to external devices via a USB cable. The USB module 34 may also be connected to the auxiliary storage device 2 and camera 7 via USB cables. The flash memory 36 is an auxiliary storage device that constitutes internal storage. The wireless communication module 38 communicates wirelessly with the input device 6 using a communication protocol such as the Bluetooth (registered trademark) protocol or the IEEE802.11 protocol. The wired communication module 40 communicates wired with external devices and connects to the network 3 via the AP 8.

Figure 3 shows an example of functional blocks of the information processing device 10. The information processing device 10 of the embodiment includes a processing unit 100 and a communication unit 102. The processing unit 100 includes a game execution unit 110, a game image generation unit 112, a display processing unit 114, an event information acquisition unit 120, an event data transmission unit 122, a game video acquisition unit 130, a game video transmission unit 132, a reception unit 140, and an acquisition unit 142. The display processing unit 114 includes a video display unit 116 and a thumbnail display unit 118.

The functions of the components in the information processing device 10 may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and/or combinations thereof, configured or programmed to implement the functions described herein. A processor is considered to be a circuitry or processing circuitry that includes transistors and other circuits. A processor may also be a programmed processor that executes a program stored in memory.

In this specification, a circuit, unit, or means is hardware that is programmed to realize or executes the described functions. The hardware may be any hardware disclosed in this specification or any hardware known to be programmed to realize or execute the described functions.

If the hardware is a processor, which is considered to be a type of circuitry, the circuitry, means, or unit may be a combination of hardware and software used to configure the hardware and/or processor.

The communication unit 102 combines the functions of the wireless communication module 38 and the wired communication module 40. While the user is playing a game, the communication unit 102 receives information about the user's operation of the input device 6 (game operation information) and provides this information to the game execution unit 110.

The game execution unit 110 executes the game in accordance with the user's operations. While the user is playing the game, the game execution unit 110 executes a game program based on game operation information and performs calculations to move the player character within the virtual space. The game image generation unit 112 includes a GPU (Graphics Processing Unit), and receives the results of calculations in the virtual space, generating game images from a viewpoint position (virtual camera) within the virtual space and game sounds at the viewpoint position within the virtual space. The video display unit 116 outputs the game images and game sounds generated by the game image generation unit 112 from the output device 4.

Figure 4 shows an example of a game screen displayed on the output device 4. The user plays the game by operating the input device 6 while viewing the game images and game sounds output from the output device 4. Hereinafter, for ease of explanation, the game images and game sounds may also be simply referred to as "game video."

When an activity is started during the game, the game program outputs event information indicating the occurrence of an activity start event, and when the activity is ended, it outputs event information indicating the occurrence of an activity end event. When the event information acquisition unit 120 acquires event information from the game program, it provides the event information to the game video acquisition unit 130. The event information acquisition unit 120 also generates event data by adding the user account, game ID, and time information (timestamp) indicating the time the event occurred to the acquired event information, and provides this to the event data transmission unit 122.

Game developers may incorporate various activities into their games. For example, if a battle activity against an enemy boss is incorporated into the game, the game program will output, at the start of the battle, event information including an activity ID that identifies the battle activity and information indicating the start of the battle activity. When the player wins the battle against the enemy boss, the game program will output event information including an activity ID that identifies the battle activity, information indicating the end of the battle activity, and information indicating the success of the activity.

The event data transmission unit 122 transmits the event data provided by the event information acquisition unit 120 to the management server 12 via the communication unit 102. In the information processing system 1, the event data transmission process is performed by all information processing devices 10 connected to the management server 12, and the management server 12 acquires event data for various games from multiple information processing devices 10. Upon acquiring event information from a game program, the event information acquisition unit 120 preferably immediately generates event data with a user account and timestamp added, and the event data transmission unit 122 transmits the event data to the management server 12.

In this embodiment, the game video acquisition unit 130 has a ring buffer and has the function of automatically recording game videos generated by the game image generation unit 112 into the ring buffer in the background. The ring buffer logically links the end and beginning to allow cyclical use of memory space, and may be generated by setting the start and end addresses of the memory space in the auxiliary storage device 2. The game video acquisition unit 130 records game video data into the ring buffer in real time from the start address in a predetermined address order, and once it has recorded up to the end address, it returns to the start address and overwrites the data, repeating this process. For example, the ring buffer can record up to 60 minutes of game video, and therefore the ring buffer will record game video played by the user over the last 60 minutes along with time information (timestamps).

During game play, the user can generate user-generated content by cutting out at least a portion of the game video recorded in the ring buffer. When the user operates a predetermined button on the input device 6 during game play, a content generation screen is superimposed on the game screen, and the user can select the time period for saving the most recent game play from multiple options (e.g., the last 30 seconds, the last minute, the last 5 minutes, the last 15 minutes, the last 60 minutes, etc.). When the user selects the saving time, the game video acquisition unit 130 cuts out the game video for the time period selected by the user from the ring buffer and saves the cut-out game video in the game video recording unit 154 as user-generated content.

As another method of generating content, the user may specify the start point of the content in real time while playing the game, and then specify the end point of the content in real time. Once the end point is specified, the game video acquisition unit 130 extracts the game video from the ring buffer from the start point to the end point specified by the user, and saves the extracted game video in the game video recording unit 154 as user-generated content.

As described above, the game video acquisition unit 130 is provided with event information from the event information acquisition unit 120. The game video acquisition unit 130 associates the provided event information with time information (timestamp) indicating the time the event occurred and records them in the ring buffer. When the game video acquisition unit 130 cuts out game video from the ring buffer, it adds (embeds) event information and timestamps that occurred between the first time of cutting (start time) and the last time of cutting (end time) to the game video as metadata. Furthermore, the game video acquisition unit 130 adds (embeds) the user account and game ID to the game video as metadata. The game video acquisition unit 130 saves the game video with the embedded metadata in the game video recording unit 154 as user-generated content.

By uploading user-generated content to the video sharing server 14, users can distribute the user-generated content to an unspecified number of users. When a user operates the input device 6 to input an instruction to upload user-generated content into the information processing device 10, the game video transmission unit 132 transmits the user-generated content (game video) stored in the game video recording unit 154 to the video sharing server 14.

Figure 5 shows the functional blocks of the video sharing server 14 of the embodiment. The video sharing server 14 includes a processing unit 200, a communication unit 202, and a recording device 240. The processing unit 200 includes a game video acquisition unit 210, an analysis unit 212, a metadata setting unit 214, a video division unit 216, a vector setting unit 218, a search request reception unit 220, a search unit 222, a selection reception unit 224, a related video identification unit 226, a video provision unit 228, and a thumbnail provision unit 230. The recording device 240 includes a video recording unit 242, a metadata recording unit 244, and a vector recording unit 246.

The functionality of the components in the video sharing server 14 may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (Central Processing Unit), conventional circuitry, and/or combinations thereof, configured or programmed to implement the functionality described herein. A processor is considered to be a circuitry or processing circuitry that includes transistors and other circuits. A processor may also be a programmed processor that executes a program stored in memory.

In this specification, a circuit, unit, or means is hardware that is programmed to realize or executes the described functions. The hardware may be any hardware disclosed in this specification or any hardware known to be programmed to realize or execute the described functions.

If the hardware is a processor, which is considered to be a type of circuitry, the circuitry, means, or unit may be a combination of hardware and software used to configure the hardware and/or processor.

The game video acquisition unit 210 acquires video of games played by multiple players from multiple information processing devices 10 and records it in the video recording unit 242. The video recording unit 242 stores the game videos, linking them to the players' user identifiers (user accounts).

The analysis unit 212 is equipped with AI (artificial intelligence) with image analysis capabilities. The analysis unit 212 performs image analysis of game videos using a trained model prepared for each game title, obtains game state values, and sets them as metadata for the game videos. The analysis unit 212 may have various image analysis functions, as described below.

<Character recognition function>
The analysis unit 212 may have an optical character recognition (OCR) function. The analysis unit 212 recognizes text included in the game video and extracts character data. For example, letters and numbers are displayed in the upper left corner of the game screen shown in Figure 4, and the analysis unit 212 extracts the letters and numbers included in the game screen using the character recognition function.

Figure 6 shows an example of letters and numbers extracted from a game image by the analysis unit 212. The analysis unit 212 sets a coordinate system for the game image with the horizontal axis as the X axis and the vertical axis as the Y axis, and specifies the coordinates of the recognized letters and numbers. In the example shown in Figure 6, the analysis unit 212 extracts the number "150" at position (x1, y3), the letters "HP" at position (x1, y4), the letters "combo" at position (x2, y1), and the number "13" at position (x2, y2). The analysis unit 212 derives the center of gravity of the letter or number as the coordinate of the letter or number.

The analysis unit 212 identifies the object indicated by the number from the relative positions of the letters and numbers. In this example, "HP" is displayed near the number displayed at position (x1, y3), and "combo" is displayed near the number displayed at position (x2, y2). Therefore, the analysis unit 212 determines that the number displayed at position (x1, y3) represents "HP (hit points)," and the number displayed at position (x2, y2) represents "combo number." Therefore, the analysis unit 212 recognizes that the player character's HP at this point is 150, and the combo number is 13.

In this way, the number displayed at the (x1, y3) position is a status value indicating the HP, and the metadata setting unit 214 records the number displayed at the (x1, y3) position together with time information (timestamp) as metadata in the metadata recording unit 244. By recording the number displayed at the (x1, y3) position together with time information, the progression of the HP is recorded in the metadata recording unit 244.

The number displayed at the (x2, y2) position is a status value indicating the combo count, and the metadata setting unit 214 records the number displayed at the (x2, y2) position together with time information in the metadata recording unit 244 as a status value indicating the combo count. By recording the number displayed at the (x2, y2) position together with time information, the progress of the combo count is recorded in the metadata recording unit 244.

<Object detection function>
The analysis unit 212 may have an object detection function and detect objects in game images. For example, the analysis unit 212 detects an item obtained by the player character. When the analysis unit 212 detects an item obtained by the player character, the analysis unit 212 records the name of the detected item and time information indicating the time of detection in the metadata recording unit 244 as a status value indicating the obtained item.

The analysis unit 212 may also detect enemy characters that the player character will face. When the analysis unit 212 detects an enemy character that the player character will face, it records the name of the detected enemy character and time information indicating the time of detection in the metadata recording unit 244 as a status value indicating the enemy character.

<Action recognition function>
The analysis unit 212 may have a behavior recognition function and detect the behavior of the player character in the game video. The analysis unit 212 detects the behavior of the player character in the game video using a trained model prepared for each game title.

The trained model may be an action detection model that has been trained using machine learning on game videos of multiple players. The action detection model performs machine learning using training game videos and the names of actions performed by player characters in the game videos as training data. For machine learning, deep learning techniques such as CNN, RNN, and LSTM may be used. When a game video is input, the trained action detection model is trained to output the names of detected actions and the time at which the actions occurred.

For example, an action detection model trained on game video of the game title "Battle XZ" will detect any of the following actions from the player character's movements: "punch," "kick," "guard," "jump," "run," or "jog." Because the types of player character actions differ depending on the game title, it is preferable to train an action detection model for each game title. Therefore, an action detection model trained on game video of another game title may detect different types of actions from those in the game title "Battle XZ." Training an action detection model for each game title makes it possible to detect actions in game videos with high accuracy. The metadata setting unit 214 records the action name and the time of action occurrence extracted by image analysis of the game video in the metadata recording unit 244 as status values indicating the action.

As described above, the metadata setting unit 214 sets the metadata for the game video. In this embodiment, the video sharing server 14 has the function of dividing a single game video and expressing the characteristics of the resulting multiple videos as vectors.

The video dividing unit 216 divides the game video. The video dividing unit 216 may divide the game video into predetermined unit times. For example, the video dividing unit 216 divides the game video into 20-second units.

Figure 7 shows an example of dividing a game video. The game video shown in Figure 7 is a four-minute video A in which player A plays "Battle XZ," and the video dividing unit 216 divides video A into 20-second segments. Therefore, video A is divided into 12 divided videos A1 to A12.

The vector setting unit 218 has the function of expressing the features of the divided videos as vectors. The vector setting unit 218 may express the features of the divided videos as sentences (text) and vectorize this sentence to express the features of the divided videos as vectors. Specifically, the vector setting unit 218 expresses the features of the divided videos as sentences (text) using the metadata of the divided videos and converts this sentence into a text vector using a known conversion technique. A known method such as doc2vec may be used to vectorize the text. By vectorizing the features of the divided videos, it becomes possible to calculate the similarity between the features of two divided videos, and therefore to evaluate the relevance of the two divided videos.

A method for setting feature vectors of divided moving pictures will be described below.
(Feature vector of divided video A1)
First, the vector setting unit 218 extracts event information and state values associated with time information (time stamps) between 0:00 and 0:20 from the metadata embedded in video A and the metadata recorded in the metadata recording unit 244. The vector setting unit 218 expresses the characteristics of the divided video A1 in text from the extracted event information and state values.

For example, if there is no change in HP between 0:00 and 0:20 and HP remains at the same value, the vector setting unit 218 detects that the player character has not received any damage in divided video A1. Also, if the number of punches is 15, the number of kicks is 0, the number of guards is 0, and the enemy character is zombie Z, the vector setting unit 218 detects that the player character has thrown a punch and attacked zombie Z. Therefore, the vector setting unit 218 converts these metadata into text to represent the characteristics of divided video A1 as follows:
"The player character attacks Zombie Z with 15 punches without taking any damage."

The vector setting unit 218 vectorizes the description of the divided video A1 to generate a feature vector A1. The vector setting unit 218 records the generated feature vector A1 in the vector recording unit 246.

(Feature vector of divided video A2)
Next, the vector setting unit 218 extracts event information and state values associated with time information (timestamps) between 0:20 and 0:40 from the metadata embedded in video A and the metadata recorded in the metadata recording unit 244. The vector setting unit 218 expresses the characteristics of divided video A2 in text from the extracted event information and state values.

For example, if the HP is reduced to half between 0:20 and 0:40, the vector setting unit 218 will detect that the player character has received significant damage in divided video A2. Also, if the number of punches is 0, the number of kicks is 0, the number of guards is 6, and the enemy character is zombie Z, the vector setting unit 218 will detect that the player character has been unilaterally attacked by zombie Z and has been unable to counterattack. Therefore, the vector setting unit 218 will convert the characteristics of divided video A2 from this metadata into text as follows:
"The player character guards against attacks from Zombie Z, but his HP is greatly reduced and he is unable to counterattack."

The vector setting unit 218 vectorizes the description of the divided video A2 to generate a feature vector A2. The vector setting unit 218 records the generated feature vector A2 in the vector recording unit 246.

The vector setting unit 218 also performs the above vectorization process on the remaining divided videos A3 to A12, and records the feature vectors A3 to A12 of the divided videos A3 to A12 in the vector recording unit 246. In this way, the feature vectors A1 to A12 of the divided videos A1 to A12 obtained by dividing video A are recorded in the vector recording unit 246. In this embodiment, the video dividing unit 216 divides all game videos recorded in the video recording unit 242, and the vector setting unit 218 generates feature vectors for the divided videos (divided videos) and records them in the vector recording unit 246.

By generating feature vectors for the split videos in this manner, the relevance of the two split videos can be evaluated based on the similarity between the two feature vectors. The similarity between the two feature vectors may be calculated, for example, using cosine similarity. Cosine similarity may be calculated for all combinations of two feature vectors recorded in the vector recording unit 246, and a data set indicating the correspondence between the combinations of the two split videos and the cosine similarity may be recorded in the vector recording unit 246.

In the above example, the vector setting unit 218 generates a description indicating the characteristics of the divided videos using the metadata embedded in video A and the metadata recorded in the metadata recording unit 244. However, in another example, the vector setting unit 218 may use a VLM (Vision and Language Model) to generate a description indicating the characteristics of the divided videos, and vectorize the description generated by the VLM.

<Provision of video sharing services>
As described above, the video sharing server 14 records multiple game videos uploaded by multiple players in the video recording unit 242, and records the feature vectors of the multiple divided videos that make up the game video in the vector recording unit 246.
A user can operate the information processing device 10 to access the video sharing server 14 and watch game videos uploaded to the video sharing server 14. The following describes the processing procedure when a user watches game videos recorded on the video sharing server 14.

Figure 8 shows an example of the home screen of a video sharing site. A user enters a string of characters indicating the game video they wish to watch into the search box 300. In this example, the user enters "video of playing Battle XZ" into the search box 300, and the reception unit 140 receives the string of characters entered by the user. When the user presses the search button 302, the acquisition unit 142 sends a search request including the search string to the video sharing server 14.

In the video sharing server 14, the search request receiving unit 220 receives a search request from the information processing device 10, obtains the search string included in the search request, and supplies it to the search unit 222. The search unit 222 searches for game videos that match the search string from among multiple game videos recorded in the video recording unit 242. At this time, the search unit 222 references the metadata embedded in the game videos and searches for game videos that include the game ID of the game title "Battle XZ" as metadata.

When the search unit 222 searches for game videos and identifies multiple candidate videos, the video providing unit 228 transmits thumbnail images of the multiple candidate videos to the information processing device 10. The information processing device 10 displays the multiple thumbnail images on the output device 4 in a selectable manner.

Figure 9 shows an example of a selection screen for multiple candidate videos. All of these game videos are videos of playing "Battle XZ." The user selects one game video on the selection screen, and the reception unit 140 accepts the selection of the game video. It is assumed that the user has selected the game video of player A on the selection screen shown in Figure 9. When the user selects the game video of player A, the acquisition unit 142 notifies the video sharing server 14 that the game video of player A has been selected.

In the video sharing server 14, the selection receiving unit 224 receives the selection of the game video of player A. When the selection receiving unit 224 receives the selection of the game video of player A, the video providing unit 228 provides the game video of player A to the information processing device 10. It is assumed that the game video of player A is the four-minute video A shown in Figure 7.

At this time, the related video identification unit 226 identifies the divided video to be played on the information processing device 10, and identifies a game video that includes a divided video that is related to the divided video being played. The first divided video to be played on the information processing device 10 is divided video A1, which is located at the beginning of video A (see Figure 7). Therefore, the related video identification unit 226 identifies a game video that includes a divided video that is related to divided video A1.

The vector recording unit 246 records feature vectors of multiple segmented videos. The related video identification unit 226 calculates the cosine similarity between the feature vector A1 of the segmented video A1 and the feature vectors of the other segmented videos recorded in the vector recording unit 246, and identifies the feature vectors of a predetermined number of other segmented videos with relatively high calculated cosine similarities. For example, the vector recording unit 246 may identify three feature vectors in descending order of cosine similarity with the feature vector A1 of the segmented video A1. Note that if the vector recording unit 246 records a dataset showing the correspondence between all combinations of feature vectors of two segmented videos and cosine similarities, the related video identification unit 226 may identify the top three feature vectors with highest cosine similarities for feature vector A1.

Here, the cosine similarity between feature vector A1 and feature vector H3 of divided video H3 in video H is highest, the cosine similarity between feature vector A1 and feature vector K7 of divided video K7 in video K is second highest, and the cosine similarity between feature vector A1 and feature vector S2 of divided video S2 in video S is third highest. From the cosine similarity calculation results, the related video identification unit 226 recognizes that video H, video K, and video S contain divided videos that are highly related to divided video A1. When the related video identification unit 226 identifies videos H, video K, and video S that contain divided videos that are highly related to divided video A1, the thumbnail providing unit 230 provides thumbnails of videos H, video K, and video S to the information processing device 10. In this way, in the video sharing server 14, when the video providing unit 228 provides video A of player A selected by the user to the information processing device 10, the thumbnail providing unit 230 provides thumbnails of videos H, video K, and video S related to divided video A1 to the information processing device 10.

In the information processing device 10, the acquisition unit 142 acquires video A from the video sharing server 14, as well as thumbnail images of video H, video K, and video S. The video display unit 116 plays video A, and the thumbnail display unit 118 displays thumbnails of video H, video K, and video S.

Figure 10 shows an example of a game video playback screen. The video display unit 116 plays video A in the first display area 310. The video display unit 116 displays a seek bar that represents the overall time length of the game video as the length of a bar and the playback position of the game video as the slider position on the bar. When the acquisition unit 142 acquires video A, the video display unit 116 plays divided video A1 located at the beginning of video A.

The thumbnail display unit 118 displays in the second display area 312 thumbnails of videos related to the image displayed in the first display area 310. In this example, the thumbnail display unit 118 displays thumbnails 314a, 314b, and 314c of videos H, K, and S related to the divided video A1 displayed in the first display area 310. When the user selects, for example, thumbnail 314a, the video display unit 116 may play video H in the first display area 310.

In this embodiment, the acquisition unit 142 notifies the video sharing server 14 of the playback status of video A. Specifically, the acquisition unit 142 notifies time information (playback time) indicating the playback time of video A. In the example shown in FIG. 10, the playback time of video A is 10 seconds, and the acquisition unit 142 notifies this playback time to the video sharing server 14.

In the video sharing server 14, the related video identification unit 226 is notified of the playback time of video A and identifies the segmented video being played in video A. Referring to Figure 7, since segmented video A1 is in the time range of 0:00 to 0:20, if the playback time is 10 seconds, the related video identification unit 226 recognizes that segmented video A1 is currently being played. When the playback time exceeds 20 seconds, the related video identification unit 226 recognizes that playback of segmented video A1 has ended and playback of segmented video A2 has begun.

When a change is made to the segmented video being played on the information processing device 10, the related video identification unit 226 identifies a game video that includes a segmented video related to the newly changed segmented video. When the related video identification unit 226 recognizes that segmented video A2 is being played, it identifies the top three feature vectors with the highest cosine similarity to feature vector A2 of segmented video A2.

Here, the cosine similarity between feature vector A2 and feature vector L4 of divided video L4 in video L is the highest, the cosine similarity between feature vector A2 and feature vector Y5 of divided video Y5 in video Y is the second highest, and the cosine similarity between feature vector A2 and feature vector G2 of divided video G2 in video G is the third highest. From the cosine similarity calculation results, the related video identification unit 226 recognizes that video L, video Y, and video G contain divided videos that are highly related to divided video A2. When the related video identification unit 226 identifies videos L, video Y, and video G that contain divided videos that are highly related to divided video A2, the thumbnail providing unit 230 provides thumbnails of videos L, video Y, and video G to the information processing device 10.

In the information processing device 10, the acquisition unit 142 acquires thumbnail images of the video L, the video Y, and the video G from the video sharing server 14.
11 shows an example of a game video playback screen. The video display unit 116 is playing divided video A2 of video A in the first display area 310. The thumbnail display unit 118 displays thumbnails of videos related to the image displayed in the first display area 310 in the second display area 312. In this example, the thumbnail display unit 118 displays thumbnails 316a, 316b, and 316c of videos L, Y, and G related to divided video A2 displayed in the first display area 310.

In this way, the thumbnail display unit 118 changes the thumbnail of the related video that it displays depending on the image that the video display unit 116 displays. When the video display unit 116 is playing video A, the thumbnail display unit 118 displays thumbnails 316a, 316b, and 316c of videos that are related to the displayed image. This allows the video display unit 116 to play a video that is highly related to the image of video A that was being displayed when the user selects one of the thumbnails 316a, 316b, or 316c.

At least a playback start time may be set for a related video. For example, for video L, the start time of divided video L4 may be set as the playback start time, and when the user selects thumbnail 316a, the video display unit 116 may play video L from the start time of divided video L4. This allows the user to immediately watch divided video L4, which is related to divided video A2. Similarly, for video Y, the start time of divided video Y5 is set as the playback start time, and when the user selects thumbnail 316b, the video display unit 116 plays video Y from the start time of divided video Y5. Furthermore, for video G, the start time of divided video G2 is set as the playback start time, and when the user selects thumbnail 316c, the video display unit 116 plays video G from the start time of divided video G2.

Note that a playback end time may be set for each related video. For video L, the end time of divided video L4 may be set as the playback end time; for video Y, the end time of divided video Y5 may be set as the playback end time; and for video G, the end time of divided video G2 may be set as the playback end time. Once the video display unit 116 has played a related video up to the playback end time, it may resume playback of video A, or it may play another related video from its playback start time to its playback end time.

The above is an example in which the thumbnail display unit 118 changes the thumbnail image displayed in the second display area 312 each time the split video being played switches, but it is also possible not to change the thumbnail image while video A is being played. Instead, the thumbnail display unit 118 may be configured to display a thumbnail of a video related to the image that is being paused and displayed when playback of video A is paused. Furthermore, after the user moves the slider on the seek bar, the thumbnail display unit 118 may be configured to display a thumbnail of a video related to the displayed image when playing a video.

Figure 12 shows another example of a game video playback screen. The video display unit 116 plays video A in the first display area 310. The thumbnail display unit 118 displays thumbnails of multiple split videos of the video A being played in the third display area 318, and displays thumbnails of videos including split videos related to the split videos of video A in the second display area 312. By displaying thumbnails of videos related to the split videos of video A near the thumbnails of those split videos, the user can efficiently watch videos related to the split videos of video A.

The above has described the playback screen of a game video displayed on a stationary output device 4, but the output device 4 may also be a head-mounted display (HMD) output device. Below, it is assumed that the HMD has a function to detect the user's gaze. When the video display unit 116 is displaying video A on the HMD output device and the user looks away from video A for a predetermined period of time, the thumbnail display unit 118 displays thumbnails of multiple videos related to the image that was displayed when the gaze was removed. At this time, the video display unit 116 may display video A in a reduced size. In this state, when the user looks at a thumbnail of a related video, the video display unit 116 starts playing the related video, and the thumbnail display unit 118 hides the thumbnail of the related video. When the user wants to watch a video related to the currently displayed image, they can display thumbnails of the related video by looking away from the displayed image, making it easy to watch the related video.

The video sharing server 14 can analyze user preferences by monitoring the user's viewing of related videos. The video sharing server 14 may identify the segmented videos that the user has viewed among the related videos, thereby more accurately grasping the user's preferences. For example, the video sharing server 14 may refer to the cosine similarity of the related videos viewed by the user and investigate the trend in the cosine similarity of the related videos viewed by the user. For example, if the cosine similarity of the related videos viewed by the user tends to be above a predetermined value, related videos with cosine similarity below the predetermined value may not be presented to the user.

The present disclosure has been described above based on examples. These examples are merely illustrative, and those skilled in the art will understand that various modifications are possible in the combinations of the components and processing processes, and that such modifications are also within the scope of the present disclosure. In the examples, a case has been described in which the information processing device 10 plays game videos, but videos other than game videos may also be played.

In the embodiment, the video dividing unit 216 divides the game video into 20-second units, but it may also divide into other time units, for example, 5-second units. Furthermore, the video dividing unit 216 may divide the game video into units that take into account semantic continuity, rather than dividing into predetermined time intervals. For example, the video dividing unit 216 may divide the game video into scene units such as conversation scenes, battle scenes, etc.

In a modified example, the information processing system 1 may include a cloud game server (not shown) that executes a game program in accordance with user operations. In this modified example, the information processing device 10 transmits information (game operation information) about the user's operation of the input device 6 to the game server. The game server generates game images and game sounds based on the game operation information, and streams the game images and game sounds to the information processing device 10. In this modified example, the information processing device 10 does not need to have the function of executing a game program, and may be a terminal device that outputs game images and game sounds from the output device 4.

The present disclosure may include the following aspects.
[Item 1]
An information processing device having a function of playing back moving images, the information processing device comprising a circuit configured to:
The circuitry comprises:
Play the video,
Displays thumbnails of videos related to the displayed image,
displaying a thumbnail of a video related to the displayed image;
When the displayed image changes, a thumbnail of the video related to the changed image is displayed.
Information processing device.
[Item 2]
The circuit comprises:
When you pause a video, it will display thumbnails of videos related to the image that is currently being displayed.
Item 1. The information processing device according to item 1.
[Item 3]
The circuit comprises:
When playing a video, display thumbnails of videos related to the image being displayed,
Item 1. The information processing device according to item 1.
[Item 4]
The circuit comprises:
Displaying multiple thumbnails of the video to be played and displaying thumbnails of videos related to the multiple thumbnails in the vicinity of the thumbnails;
Item 1. The information processing device according to item 1.
[Item 5]
The circuit comprises:
When a thumbnail of a related video is selected, the related video will be played.
Item 1. The information processing device according to item 1.
[Item 6]
The related videos have a playback start time set,
The circuit comprises:
Play the related video from the set playback start time,
Item 6. The information processing device according to item 5.
[Item 7]
A method for displaying a moving image in an information processing device, comprising:
Play the video,
Displays thumbnails of videos related to the displayed image,
When the displayed image changes, a thumbnail of the video related to the changed image is displayed.
Video display method.
[Item 8]
A recording medium on which a program executed in a computer of an information processing device is recorded, the program being recorded on the computer,
On the computer,
The ability to play videos,
The ability to display thumbnails of videos related to the displayed image,
When the displayed image is changed, a function to display a thumbnail of the video related to the changed image,
Make this a reality.

This disclosure can be used in the technical field of allowing users to watch videos.

1...information processing system, 4...output device, 5...server device, 6...input device, 10...information processing device, 12...management server, 14...video sharing server, 100...processing unit, 102...communication unit, 110...game execution unit, 112...game image generation unit, 114...display processing unit, 116...video display unit, 118...thumbnail display unit, 120...event information acquisition unit, 122...event data transmission unit, 130...game video acquisition unit, 132...game video transmission unit, 140...reception unit, 142 ...acquisition unit, 154...game video recording unit, 200...processing unit, 202...communication unit, 210...game video acquisition unit, 212...analysis unit, 214...metadata setting unit, 216...video division unit, 218...vector setting unit, 220...search request reception unit, 222...search unit, 224...selection reception unit, 226...related video identification unit, 228...video provision unit, 230...thumbnail provision unit, 240...recording device, 242...video recording unit, 244...metadata recording unit, 246...vector recording unit.

Claims (8)

An information processing device having a function of playing video,
a video display unit for playing videos;
a thumbnail display unit that displays thumbnails of videos related to the displayed image,
the thumbnail display unit changes the thumbnail of the moving image to be displayed in accordance with the image displayed by the moving image display unit.
1. An information processing device comprising: When the video display unit pauses playback of the video, the thumbnail display unit displays a thumbnail of a video related to the image that is being displayed while playback is paused.
2. The information processing apparatus according to claim 1, wherein: When the video display unit is playing a video, the thumbnail display unit displays a thumbnail of a video related to the displayed image.
2. The information processing apparatus according to claim 1, wherein: the thumbnail display unit displays a plurality of thumbnails of the moving image to be played, and displays thumbnails of moving images related to the plurality of thumbnails in the vicinity of the thumbnails;
2. The information processing apparatus according to claim 1, wherein: When a thumbnail of a related video is selected, the video display unit plays the related video.
2. The information processing apparatus according to claim 1, wherein: The related videos have a playback start time set,
the video display unit plays the related video from the set playback start time;
6. The information processing apparatus according to claim 5, A method for displaying a moving image in an information processing device, comprising:
playing the video;
displaying a thumbnail of a video related to the displayed image;
when the displayed image is changed, displaying a thumbnail of a video associated with the changed image;
A moving image display method comprising: On the computer,
The ability to play videos,
The ability to display thumbnails of videos related to the displayed image,
When the displayed image is changed, a function to display a thumbnail of the video related to the changed image,
A program to achieve this.