JP2016062184A - Moving image generation system, moving image generation device, moving image generation method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve a feeling of transfer to be provided, while suppressing increase in an amount of data.SOLUTION: A moving image generation system 10 includes: a storage section 20 that stores a plurality of key frame images captured at a plurality of positions and respective imaging positions of the plurality of key frame images in association with each other; an acquisition section 43 for acquiring a user position Px transferring virtually; a selection section 44 that selects a plurality of key frame images A and B, the imaging positions of which are located near the user position Px, from the plurality of key frame images stored in the storage section 20; and an interpolation processing section 46 that acquires an interpolation frame image X at the user position Px at least from one of the selected plurality of key frame images A and B by interpolation processing based on distances dand dbetween the user position Px and imaging positions Pand Pof the plurality of selected key frame images A and B respectively.SELECTED DRAWING: Figure 1

Description

  The present invention relates to moving image generation.

  As shown in Non-Patent Document 1, a service that provides still images taken from various places around the world is known.

Google Street View, https: // www. Google. com / maps / views / streetview? gl = jp & hl = ja Hyperlapse, http: // labs. teenanlax. com / project / hyperlapse

  Here, in order to present a virtual movement feeling to the user, it is required to present the user with a moving image of a landscape image that changes according to the movement. In order to realize this, it is conceivable to present a user with a moving image captured during the movement of the automobile by a camera mounted on the automobile. By presenting the user with a moving image that is captured in advance or in conjunction with the user's action such as driving operation or running on a treadmill, the user can be captured even without actually visiting the captured location. It is possible to obtain a virtual movement sensation as if moving from one place to another.

  In order to provide a virtual movement sensation in various places, moving images taken in advance along all routes connecting those places are required. However, since a moving image has a larger amount of data than a still image, it is not always easy to prepare and provide a moving image that has been captured in advance for all routes.

  There is already a service that provides still images taken from various locations around the world, such as the Street View described above. Hyperlapse (see Non-Patent Document 2) proposes to use a still image provided in Street View as a frame image of a moving image. By continuously presenting still images at nearby points, a moving image is obtained, and a sense of movement can be presented.

  In StreetView, still images in various places are taken at a relatively wide shooting interval, and the shooting interval is said to be approximately 11.5 m. Even a still image shot at a relatively wide shooting interval can be displayed at a very high speed to display a moving image with a smoothly changing landscape. However, when still images taken at wide shooting intervals are switched at high speed, only a very fast moving sensation can be obtained.

  On the other hand, if it is intended to provide a sense of movement at low speeds such as walking and running of people, it becomes a fragmentary moving landscape. In other words, when still images taken at a wide shooting interval are switched at a low speed in accordance with the movement speed of the human walking speed, one still image is displayed for several seconds. For this reason, when the moving speed is low, only a still image is presented to the user in pieces, and the quality of movement sense obtained by the user is deteriorated.

  In order to present a natural sense of movement at low speeds, it is conceivable to sufficiently narrow the shooting interval of still images, which are the frames that make up a moving image. As a result, the number of still images to be photographed increases and the amount of data increases.

  Therefore, it is desired to improve the sense of movement that is presented while suppressing an increase in the amount of data.

  An aspect of the present invention includes a storage unit that stores a plurality of key frame images photographed at a plurality of positions in association with photographing positions of the plurality of key frame images, and a user position that virtually moves An acquisition unit that acquires a plurality of key frame images stored in the storage unit, a selection unit that selects a plurality of key frame images whose shooting position is in the vicinity of the user position, and the user position An interpolation processing unit that obtains an interpolated frame image at the user position from at least one of the selected plurality of key frame images by an interpolation process based on a distance from each of the plurality of selected key frame images and the shooting position. And a moving image generation system. The key frame image is preferably an omnidirectional image.

  In the above aspect, a portion excluding the storage unit in the moving image generation system may constitute a moving image generation device. Another aspect of the present invention is a computer program for causing a computer to function as a moving image generating apparatus. Another aspect of the present invention may be a computer program for causing a computer to function as a moving image generation system. Another aspect of the present invention is a recording medium on which the computer program is recorded.

  According to still another aspect of the present invention, a virtual moving user position is acquired, and a plurality of key frame images whose shooting positions are in the vicinity of the user position are selected from a plurality of key frame images, An interpolation frame image at the user position is obtained from at least one of the selected plurality of key frame images by an interpolation process based on a distance between the user position and each of the plurality of selected key frame images. It is the moving image generation method containing.

  ADVANTAGE OF THE INVENTION According to this invention, the movement feeling shown can be improved, suppressing the increase in data amount.

It is a block diagram of a moving image generation system. It is a flowchart which shows a moving image generation process. It is explanatory drawing of the interpolation frame image production | generation by alpha blending. It is an example of setting α value. It is an example of a key frame image database. It is explanatory drawing which shows the other example of an interpolation frame image production | generation. It is a flowchart which shows a moving image generation process. It is explanatory drawing of the interpolation frame image generation by image deformation | transformation. It is a figure which shows a two-dimensional movement space. It is an example of a key frame image database. It is a figure which shows a three-dimensional movement space. It is an example of a key frame image database. It is a flowchart which shows an image acquisition procedure. It is an example of setting the priority of key frame image acquisition.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[1. Video generation system]
As shown in FIG. 1, the moving image generation system 10 includes a server 20 and a moving image generation device 30. The server 20 (storage unit: first storage device) is provided on a network such as the Internet, and includes a key frame image database 22 that stores key frame images constituting a moving image. In the present embodiment, the server 20 and the moving image generating device 30 are provided as separate devices, but they may be provided as an integrated device. That is, the moving image generation system 10 may be configured by providing the key frame image database 20 in the moving image generation device 30 shown in FIG. The moving image generation system 10 may be configured by providing the server 20 shown in FIG. 1 with the function of the processing device 40. Some functions of the processing device 40 illustrated in FIG. 1 may be provided in the server 20.

  The key frame image is used to generate a spatial video (moving image) for realizing a virtual driving environment. The key frame image of the present embodiment is a still image obtained by photographing each point in the real world with a camera. The still image may be a standard aspect ratio image taken with a normal camera or a panoramic image. The panoramic image is preferably an omnidirectional panoramic image (omnidirectional image). In the following description, it is assumed that the key frame image is an omnidirectional panoramic image.

  The key frame image database 22 stores a plurality of key frame images and shooting positions of the plurality of key frame images in association with each other. When a position (virtual user position) is designated as a search key by the moving image generation device 30, the key frame image database 22 extracts a plurality of key frame images in the vicinity of the designated position. The key frame image database 22 gives the extracted plurality of key frame images and their shooting positions to the moving image generation device 30.

  The shooting interval of the plurality of key frame images can be set to about 80 cm to 20 m, for example. It is preferable that at least a part of the same subject is photographed in a plurality of key frame images at adjacent photographing positions. When the shooting interval is as described above, when only the key frame image is switched and displayed in accordance with the walking or running speed of the person, the video does not appear to change smoothly, but since the shooting interval is relatively wide, the key frame image Thus, the amount of data stored in the server 20 can be suppressed. In addition, since the number of key frame images given from the server 20 to the processing device 40 can be suppressed, an increase in communication traffic between the server 20 and the processing device 40 can be prevented.

  The image generation device 30 acquires a key frame image stored in the server 20, and generates an interpolation frame image corresponding to the user position from the acquired key frame image. The image generation device 30 uses the key frame image and the interpolated frame image as frame images constituting a moving image. The moving image of the present embodiment is an image in which the scenery presented to the user changes according to the virtual user's movement, and gives the user a sense of movement while moving the place without visiting the actual place. Present.

  When the virtual user position matches the shooting position of the key frame image, the key frame image is presented to the user. On the other hand, when the virtual user position is located between the photographing positions of the plurality of key frame images, the interpolation frame image is presented to the user. The interpolated frame image changes according to a change in the positional relationship between the shooting position of each of the plurality of key frame images and the current user position. When the user moves between the shooting positions of each of the plurality of key frame images, an interpolation frame image that changes in accordance with the user position is presented to the user. Thereby, the user can get a natural sense of movement.

  The image generating device 30 is provided in an exercise device such as a treadmill or a fitness bike, for example. In this case, the image generation device 30 displays a moving image in which the landscape changes according to the speed at which the user runs on the treadmill or the speed at which the user runs on the fitness bike. Thereby, a virtual movement sensation can be presented to the user who is exercising, and fun can be given to the exercise.

  The image generation device 30 includes a processing device 40, a display device 50, a speed detector 52, and a direction detector 54. If the image generation device 30 only generates a moving image and does not display a moving image, the display device 50 may be omitted. If unnecessary, at least one of the speed detector 52 and the direction detector 54 may be omitted.

  The processing device 40 is configured by a computer having a processor 41 and a memory 42. The processor 41 performs processing such as user position acquisition processing 43, key frame image selection processing 44, key frame image acquisition processing 45, and interpolation processing 46. The memory 42 (storage unit; second storage device) stores a plurality of key frame image databases 22 acquired from the server 20 and their photographing devices in association with each other.

  The user position acquisition process 43 is a process for causing the processor 41 to function as an acquisition unit for acquiring a user position that moves virtually. The key frame image selection process 44 is a process for causing the processor 41 to function as a selection unit for selecting a key frame image whose shooting position is in the vicinity of the virtual user position. The key frame image acquisition process 45 is a process for causing the processor 41 to function as an image acquisition unit for acquiring the selected key frame image and its shooting position from the server 20 and storing them in the memory 42. The interpolation process 46 is a process for causing the processor 41 to function as an interpolation processing unit that obtains an interpolated frame image at the user position from at least one of a plurality of selected key frame images.

  The processor 41 executes the processes 43, 44, 45, and 46 by executing a computer program included in the processing device 40, and converts the processor 41 (computer) into the acquisition unit, the selection unit, the image acquisition unit, And function as an interpolation processing unit.

  The display device 50 displays the generated moving image. Examples of the display device 50 include a flat display, an immersive display that combines a plurality of displays to cover the entire periphery of the user, a dome-shaped display that covers the periphery of the user, and a head-mounted display that is mounted on the user's head. Consists of. In the case of the display device 50 in which the range that can be displayed at one time is narrower than the entire range of the omnidirectional panoramic image, such as a head-mounted display, the orientation of the user's head or body is detected as a virtual moving direction. There may be. When the display device 50 is a head-mounted display, an image to be displayed on the display device 50 is generated according to the orientation and movement of the head detected by the head-mounted display.

  The speed detector 52 detects the virtual moving speed of the user. When the moving image generating device 30 is provided in an exercise apparatus that performs exercise simulating a user's walking or running, such as a treadmill or a fitness bike, the exercise apparatus has a speed detection function. In this case, the speed detection function of the exercise apparatus can be used as the speed detector 52 of the present embodiment. When it is assumed that the user's moving speed is constant without changing, or when the processing device 40 sets the moving speed, the speed detector 52 is unnecessary. Further, regarding the virtual movement of the user, the user does not have to actually perform an exercise such as walking or running, and the user's position and / or moving speed is simply operated by an operating device operated by the user. It may be a thing. In this case, the operating device operated by the user functions as the speed detector 52 of the present embodiment.

  The direction detector 54 detects the virtual moving direction of the user in the virtual space. The direction detector 54 can be configured, for example, as an operating device that manually operates the moving direction of the user. Further, the direction detector 54 may detect the orientation of the user's head or body for selecting the moving direction, and detect the orientation of the user's head or body as a virtual moving direction. When the display device 50 is a head mounted display, an attitude sensor provided in the head mounted display can be used as the direction detector 54. In the case of an exercise apparatus that assumes one-dimensional movement, such as a treadmill or fitness bike, the direction detector 54 can be omitted.

[2. Video generation processing]
[2.1 First example of moving image generation processing]
FIG. 2 shows a first example of a moving image generation process executed by the processor 41 of the processing device 40. In this first example, a one-dimensional space is assumed as a virtual movement space. In the first example, the moving direction of the user is not considered. As will be described later, the moving space may be a two-dimensional space or a three-dimensional space.

  The processing device 40 acquires the user speed v and the user position Px (step S11). The user speed v is given from the speed detector 52, for example. The processing device 40 performs a user position acquisition process 43 to acquire the user position Px from the user speed v. Specifically, the processing device 40 calculates the movement amount from the initial position based on the virtual initial position (current position) of the user at the start of the moving image generation process and the acquired speed v. The current position (user position) Px after the virtual movement of the user is obtained. The user position Px may be set by the processing device 40 itself.

The processing device 40 executes a key frame image selection process 44 and a key frame image acquisition process 45 (step S12). In other words, the processing device 40 performs the key frame image selection process 44, thereby, based on the speed v and the user position Px, from the plurality of key frame images stored in the server 20, a plurality of key frame images A, Select B. The plurality of key frame images to be acquired are a plurality (two) of key frame images A and B whose shooting positions P _A and P _B are in the vicinity of the user position Px. As shown in FIG. 3, when the moving space is one-dimensional, for example, two key frame images A and B that are closest to the user position Px are selected. The selection of the plurality of key frame images is performed such that the user position Px is positioned between the shooting positions P _A and P _B of the selected images A and B.

  In the present embodiment, even if the user position Px is the same, different key frame images are selected according to the virtual moving speed v of the user. This point will be described later.

The processing device 40 acquires a plurality of selected key frame images A and B and their photographing positions P _A and P _B from the server 20 by performing a key frame image acquisition process 45. The acquired plurality of key frame images A and B and their photographing positions P _A and P _B are stored in the memory 42. The processing device 40 performs subsequent processing based on the plurality of key frame images A and B and their photographing positions P _A and P _B stored in the memory 42.

The processing device 40 calculates distances d ₁ and d ₂ between the user position Px and the captured positions P _A and P _{B of} the acquired key frame images A and B, respectively. The distance d ₁ (first distance d ₁ ) is a distance between the user position Px and the selected first key frame image A, and the distance d ₂ (second distance d ₂ ) is selected as the user position Px. The distance from the second key frame image B. The distances d ₁ and d ₂ may be calculated by the server 20 and provided to the processing device 40.

The processing device 40 performs an interpolation process 46 based on the distances d ₁ and d ₂ , whereby an interpolation frame in which the user position Px becomes a virtual shooting position (viewpoint position) from the plurality of acquired key frame images A and B. An image X is obtained. In the first example, the interpolated frame image X is obtained by alpha blending (first processing) of a plurality of key frame images A and B. In step S13, the processing device 40 sets an α value (transparency) used for alpha blending based on the distances d ₁ and d ₂ . The α value may be set by the server 20 and provided to the processing device 40. Alpha blending may also be performed by the server 20, and the interpolated frame image X obtained by alpha blending may be provided to the processing device 40.

In this embodiment, the alpha ₁ as alpha value to be multiplied to the first key frame image A, and alpha ₂ as alpha value to be multiplied to the second key frame image B, and is set. In this embodiment, as will be apparent from the equation described later, α ₂ = 1−α ₁ , and thus α ₂ can be obtained by calculating α ₁ .

Each of the α values (α ₁ , α ₂ ) multiplied by the images A and B takes a value from 0 to 1, is transparent when 0, and is opaque when 1. alpha _1, the more the distance d ₁ between the imaging position P _A of the user position Px and the first key frame image A is small, is the set to a large value (transparency is set low). α _{2 is} also set to a larger value (transparency is set to be lower) as the distance d ₂ between the user position Px and the shooting position P _{B of} the second key frame image B is smaller.

In the present embodiment, the α value (α ₁ , α ₂ ) is expressed as a function of the distances d ₁ and d ₂ . α ₁ and α ₂ are calculated by the following equations, for example.

FIG. 4A shows α ₁ and α ₂ values (vertical axis) set by the above formula. The horizontal axis is a virtual user position indicates a position range from a position P _A to the position P _B. In FIG. 4 (a), alpha ₁ is the maximum (1) at a position _{P A,} decreases toward the position _{P B,} the minimum (0) at the position _{P B.} α ₂ is the minimum (0) at the position P _A , increases toward the position P _B, and reaches the maximum (1) at the position P _B.

The interpolated frame image X for the user position Px is obtained by multiplying each of the plurality of key frame images A and B by an α value (α ₁ , α ₂ ) as in the following equation.

As shown in FIG. 3, a plurality of key frames image A different photographing positions, to B, and that is reflected is building in the vicinity of the position P _D, both the key frame image A, the shooting position P _A of _{B, and} P _B Because they are different, the size and position of the buildings shown in both key frame images A and B are slightly different. By superimposing such a plurality of key frame images A and B by alpha blending, an image in which the buildings (subjects) in the images A and B appear double is obtained.

In the case where a virtual user position Px is changed from the position _{P A} to the position _{P B,} if user position Px is positioned _{P A, α 1 = 1,} α 2 = 0 , and the interpolation frame image X first key Since it is equal to the frame image A, the user is presented only with the image of the building shown in the image A.

As shown in FIG. 4 (a), the user position Px is, when it is between positions P _B and the position P _A is a position closer to the position P _A, the transparency of the image A is relatively low image B Since the transparency is relatively high (α ₁ > α ₂ ), the building image appears dark in the image A and the building image appears in the image B is thin.

When the user position Px is between the position P _A and the position P _B but closer to the position P _B , the transparency of the image A is relatively high and the transparency of the image B is relatively low (α ₁ <α ₂ ), the building image shown in the image A is thin, and the building image shown in the image B is dark, and the building looks double.

If the user position Px is the position P _B , α ₁ = 0 and α ₂ = 1, and the interpolation frame image X is equal to the second key frame image B. Therefore, the user can see the building shown in the image B. Only the image of is presented.

As shown in FIG. 4A, α ₁ and α ₂ are continuously changed according to the change of the user position Px, so that the interpolated frame image X is also continuously changed according to the movement of the user. While the user is moving between the position P _A and the position P _B , the interpolation frame image X that changes as the user moves is displayed on the display device 50. The interpolated frame image X is an image in which the subject may appear double when viewed as a still image. However, since the subject's duplex state changes according to the movement of the user, it is natural for the human eye to move. It appears as a change of scenery. Therefore, it can be presented to the user as a smooth continuous moving image.

The way of changing α ₁ and α ₂ according to the change of the user position Px is not limited to the example shown in FIG. Instead of α ₁ and α ₂ changing linearly as shown in FIG. 4A, they may change like a curve as shown in FIG. As shown in FIG. 4 (c), the user position Px is, even away from the position _{P A,} if the position _{P A} near, [alpha] 1 = 1, while maintaining the [alpha] 2 = 0, the key frame image A is displayed the so that, similarly, the user position Px is, even away from the position P _B, if the position P _B vicinity, [alpha] 1 = 0, to maintain the [alpha] 2 = 1, as the key frame image B is displayed May be. When image processing (for example, processing shown in the second example and / or the third example) other than alpha blending is performed on the key frame images A and B, α ₁ and α ₂ are set as shown in FIG. It may be changed as follows.

When the α value is set as shown in FIG. 4C, if the user position Px is not the shooting positions P _A and P _B of the key frame images A and _B , but is a position near them, the key frame image A and B are displayed, and the interpolation frame image can be displayed after a certain distance from the shooting positions P _A and P _B. Thereby, it is not necessary to generate the interpolation frame image X when the user position is slightly away from the shooting positions P _A and P _B, and a time margin for generating the interpolation frame image X can be secured.

FIG. 5 is selected by the key frame image selection process 44 when the virtual moving speed v of the user is low (for example, when walking) and high (for example, when running). A key frame image is shown. User position Px is in the case where the position shown in FIG. 3 (a position between the positions P _A and the position P _B), if slow, the key frame images A, B is selected, if fast, keyframes Images A and C are selected.

When the moving speed v is low, a natural moving feeling can be obtained by generating an interpolated frame image from a plurality of key frame images shot at a relatively narrow shooting interval. On the other hand, when the moving speed v is high, a natural moving feeling can be maintained even if a plurality of key frame images photographed at a relatively wide photographing interval are used. For example, when the shooting interval (key frame interval) between adjacent shooting positions in FIG. 3 is 1, when the moving speed v is smaller than a predetermined threshold, the key frame interval of a plurality of selected key frame images is 1. so that the position P _A, key taken with P _B frame images A, B are selected. Further, when the moving speed is larger than a predetermined threshold, the key frame images A and C photographed at the positions P _A and P _c are selected so that the key frame interval of the plurality of selected key frame images is 2. Is done.

[2.2 Second Example of Moving Image Generation]
When the interpolated frame image X is generated, the key frame images A and B themselves are not alpha-blended, but are subjected to image processing (second processing) corresponding to the distances d ₁ and d _2. Alpha blending may be performed on the image (corrected key frame image). The image processing (second processing) according to the distances d ₁ and d ₂ is performed by image processing other than alpha blending, depending on the positional relationship between the user position Px and the shooting positions P _A and P _B. This is a process for emphasizing B or de-emphasizing (such as blurring).

For example, as shown in FIG. 6, the user position Px is, of the two imaging position _P A, _{P B,} close towards imaging position _{P A,} if far from the shooting location _{P B,} processor 40, key A corrected key frame image A1 is generated by performing an emphasis process on the frame image A so as to be strongly presented to the human eye, and a non-enhancement process is performed on the key frame image B to generate the human eye. To generate a post-correction key frame image B1 that is presented weakly. FIG. 6 shows a corrected key frame image A1 that presents a building that is a subject more strongly to the human eye, and a corrected key frame image B1 that presents a building weaker. Since the post-correction key frame images A1 and B1 are alpha-blended, in addition to the difference in transparency set in both the images A1 and B1, the difference between the distances d ₁ and d ₂ is caused by the emphasis / non-emphasis of the subject itself. The user can be presented with a good sense of movement.

  The enhancement process and the non-enhancement process are performed by, for example, one or more processes selected from image brightness adjustment process, contrast adjustment, resolution adjustment, increase / decrease color process, and the like. Increase / decrease color processing is a generic term for color increase processing that increases the color in an image, such as making a black and white image a color image, and color reduction processing that reduces color in the image, such as making a color image a black and white image. is there. The image can be enhanced by the color increase process, and the image can be deemphasized by the color reduction process. The enhancement process can also be performed by edge enhancement process, and the non-enhancement process can also be performed by blurring process.

  The corrected key frame image A1 shown in FIG. 6 is obtained by increasing the brightness, enhancing the contrast, and increasing the resolution with respect to the key frame image A. The corrected key frame image B1 is: The key frame image B is obtained by lowering the brightness, lowering the contrast, and lowering the resolution. Both the enhancement process and the non-enhancement process need not be performed, and only one of them may be performed.

  In the second example, points that are not particularly described are the same as in the first example.

[2.3 Third Example of Moving Image Generation]
The interpolation process for generating the interpolated frame image X is not limited to alpha blending. The interpolated frame image X may be generated by performing image processing (image deformation) on one of a plurality of selected key frame images based on the distances d ₁ and d ₂ .

FIG. 7 shows a moving image generation process according to the third example. In FIG. 7, steps S21 and S22 are the same as steps S11 and S12 of FIG. In step S23, the processing device 40 performs image recognition processing for recognizing individual subjects in the selected key frame images A and B. In step S24, the processing device 40 deforms each subject in the key frame image closest to the user position Px based on the distances d ₁ and d ₂ to generate an interpolation frame image Px.

As shown in FIG. 8, the building in the key frame image A is relatively small, the building in the key frame image B is relatively large, and the user position Px is close to the shooting position PA of the key frame image A. It shall be in position. In this case, the processing device 40 calculates the user position from the building size (and position) in the key frame image A, the building size (and position) in the key frame image B, and the distances d ₁ and d _2. The size (and position) of the building at Px is obtained. This process is performed for all subjects recognized in the key frame image. Based on this, processor 40 obtains the conversion parameters for converting the shot key frame image A in shooting position P _A is closer to the user position Px landscape in user position Px, based on the conversion parameter The interpolated frame image X is obtained by converting the key frame image A.

[2.4 Selection of key frame image in two-dimensional space]
In the description so far, the virtual moving space has been described as being a one-dimensional space, but the virtual moving space may be two-dimensional as shown in FIG. The keyframe image database 22 of the server 20, the key frame image A~H taken at each point _P A to P _H of the two-dimensional space, associated with their shooting position _P A to P _H stores Has been.

In FIG. 9, it is assumed that the user position Px is at a position surrounded by the positions P _A , P _B , P _C , and P _D. In FIG. 9, the shooting interval (key frame interval) between adjacent shooting positions in the vertical direction and the horizontal direction is 1. In the key frame image selection process 44, for example, a plurality of key-frame image _P A is the distance from the user position Px is 1 or _less, P B, _{P C} is selected. The number of key frame images to be selected may be four or more, or may be two in some cases.

When three key frame images A, B, and C are selected, α values (α ₁ , α ₂ , α ₃ ) multiplied by the images A, B, and C are determined from the user position Px to the images A, B, respectively. , shooting position _P a of _C, P B, it is represented by a function of the distance _{d 1,} d 2, _{d 3} to _{P C.} α ₁ , α ₂ , and α ₃ are calculated by the following equations, for example.

The interpolated frame image X for the user position Px is obtained by multiplying each of the plurality of key frame images A, B, and C by α values (α ₁ , α ₂ , α ₃ ), as in the following equation.

Even when the moving space is two-dimensional, when the moving speed v is smaller than a predetermined threshold value, the positions P _A , P _B , P are set so that the key frame interval of the selected key frame images is 1. shot key frame image A with _C, B, _C are selected. Further, when the moving speed is larger than a predetermined threshold, the key frame images photographed at the positions P _A , P _E, P _F , and P _G so that the key frame interval of the plurality of selected key frame images is 2. A, E, F, and G are selected (see FIG. 10).

[2.5 Selection of key frame image in 3D space]
The virtual moving space may be three-dimensional as shown in FIG. The keyframe image database 22 of the server 20, the key frame image A~H taken at each point _P A to P _H of the three-dimensional space, in association with their shooting position _P A to P _H storage Has been.

In FIG. 11, it is assumed that the user position Px is at a position surrounded by the positions P _{A to} P _H. In FIG. 13, the shooting interval (key frame interval) between adjacent shooting positions in each of the three-dimensional directions is 1. In the key frame image selection process 44, for example, a plurality of key frame images P _A , P _B , P _C , and P _F whose distance from the user position Px is 1 or less are selected. The number of key frame images to be selected may be 5 or more, or 2 or 3 depending on circumstances.

When three key frame images A, B, C, and F are selected, the α values (α ₁ , α ₂ , α ₃ , and α ₄ ) multiplied by the images A, B, C, and F are the user positions. each image a from _px, B, _C, shooting position _P a of the _F, P _B, P _C, the distance to _{P F d 1, d 2,} d 3, is represented by a function of the alpha _4. α ₁ , α ₂ , α ₃ , and α ₄ are calculated by the following equations, for example.

The interpolated frame image X for the user position Px multiplies each of the plurality of key frame images A, B, C, and F by α values (α ₁ , α ₂ , α ₃ , α ₄ ) as in the following equation. Can be obtained.

  Even when the moving space is three-dimensional, the selected key frame image can be made different according to the moving speed v.

[3. Image acquisition processing from server]
The processing device 40 needs to generate the interpolated frame image X at high speed according to the movement of the user. For this reason, the processing device 40 does not acquire the key frame image used for generating the interpolated frame image X from the server 20 after the user position Px is obtained, but needs it in the future based on the current user position Px and the like. The key frame image may be acquired in advance.

FIG. 13 shows a processing procedure for preemptive acquisition of a key frame image. Processor 31, along with the start of the moving image processing, acquires the initial position P ₀ of the user (step S31), the position P ₀ around starts acquiring multiple keyframes image is an imaging position (Step S32).

  Thereafter, when the processing device 40 acquires the moving speed, the user position Px, and the moving direction of the moving user (step S33), the priority for acquiring the key frame image is obtained based on the moving speed, the user position Px, and the moving direction. Is set (step S34). The processing device 40 acquires, from the server 20, a plurality of key frame images in which the periphery of the current user position Px is a shooting position in order according to the set priority.

FIG. 14 shows an example of setting priority. 14, the user position Px is between the position _{P C} and the position _{P B,} close better position PB. The moving direction of the user is a direction from the left to the right in the drawing. In this case, the key frame image that the processing device 40 should acquire most wired from the server 20 is the key frame image B at the shooting position P _B closest to the user position Px. When the moving speed of the user is low, the priority decreases as the user moves away from the user position Px in the moving direction of the user. Priority keyframe image A photographing position P _A is the user moving direction in the opposite direction as viewed from the closest photographing position P _B to the user position Px is relatively low.

When the moving speed of the user is high, the priority decreases as the user moves away from the user position Px in the user moving direction, but the shooting positions P _D and P whose key frame interval is a multiple of 2 as viewed from the shooting position PB. _F keyframe image D, the priority of F is the other key frame image C, E, relatively higher than G.

  When determining the priority, at least one of the moving speed and the moving direction may be omitted.

[4. Application example of moving image generation system]
The moving image generation system 10 can be used with a sports equipment such as a treadmill, or for location guidance.

[5. Addendum]
The present invention is not limited to the above embodiment, and various modifications can be made.

10 moving image generation system 20 server (storage unit; first storage device)
30 moving image generating device 40 processing device 41 processor 42 memory (storage unit; second storage device)
43 User position acquisition processing (acquisition part)
44 Key frame image selection processing (selection unit)
45 Key frame image acquisition processing (image acquisition unit)
46 Interpolation processing (interpolation processing unit)
50 Display device 52 Speed detector 54 Direction detector X Interpolated frame image A, B Key frame image Px User position P _A , P _B shooting position

Claims (12)

A storage unit that stores a plurality of key frame images photographed at a plurality of positions in association with a plurality of photographing positions of the key frame images;
An acquisition unit for acquiring a user position to move virtually;
A selection unit that selects a plurality of key frame images in which the shooting position is in the vicinity of the user position from a plurality of key frame images stored in the storage unit;
An interpolation frame image at the user position is obtained from at least one of the selected plurality of key frame images by interpolation processing based on the distance between the user position and each of the selected key frame images. An interpolation processing unit;
A moving image generation system comprising: The interpolation process is a first process of superimposing a plurality of selected key frame images with a transparency set based on a distance between the user position and each of the selected key frame images. The moving image generation system according to claim 1. The moving image generation system according to claim 2, wherein the transparency is set to be lower as the key frame image has a smaller distance between the user position and the shooting position. The interpolation processing includes, for at least one of the selected plurality of key frame images, second processing for image enhancement or image non-emphasis according to the distance between the user position and the shooting position. The moving image generation system according to claim 2 or 3, further comprising: 5. The second process includes one or more processes selected from the group consisting of brightness adjustment process, contrast adjustment process, blurring process, edge enhancement process, resolution adjustment, color reduction process, and color increase process. The moving image generation system described. The said selection part selects the said some key frame image so that the distance between the imaging positions of the said some key frame image selected changes according to a user's virtual moving speed. The moving image generating system according to any one of the above. The moving image generation system according to claim 1, wherein the user position and the shooting position are a position in a one-dimensional space, a position in a two-dimensional space, or a position in a three-dimensional space. The storage unit
A first storage device that stores a plurality of the key frame images and shooting positions of the plurality of key frame images in association with each other;
A plurality of the key frame images and their shooting positions are acquired from the first storage device, and a plurality of the key frame images that are smaller than the number in the first storage device, and a plurality of the key frame images, respectively. A second storage device that stores the shooting position in association with each other;
Including
The selection unit is configured to select a plurality of the key frame images in the vicinity of the user position from a plurality of key frame images stored in the second storage device,
The second storage device acquires a plurality of the key frame images and their shooting positions stored in the first storage device according to a priority determined based on the user position and a virtual moving direction of the user. The moving image generating system according to any one of claims 1 to 7. The storage unit
A first storage device that stores a plurality of the key frame images and shooting positions of the plurality of key frame images in association with each other;
A plurality of the key frame images and their shooting positions are acquired from the first storage device, and a plurality of the key frame images that are smaller than the number in the first storage device, and a plurality of the key frame images, respectively. A second storage device that stores the shooting position in association with each other;
Including
The selection unit is configured to select a plurality of the key frame images in the vicinity of the user position from a plurality of key frame images stored in the second storage device,
The second storage device acquires a plurality of the key frame images and their shooting positions stored in the first storage device according to a priority determined based on the user position and a virtual moving speed of the user. The moving image generating system according to any one of claims 1 to 8. An acquisition unit for acquiring a user position to move virtually;
A selection unit that selects a plurality of key frame images whose shooting positions are in the vicinity of the user position from among a plurality of key frame images;
Interpolation for obtaining an interpolated frame image at the user position from at least one of the selected plurality of key frame images by an interpolation process based on the distance between the user position and the shooting positions of the selected key frame images. A processing unit;
A moving image generating apparatus. Get the user position to move virtually,
From a plurality of key frame images, select a plurality of the key frame images whose shooting position is in the vicinity of the user position,
Obtaining an interpolated frame image at the user position from at least one of the selected plurality of key frame images by an interpolation process based on a distance between the user position and a shooting position of each of the selected key frame images. A method for generating a moving image. An acquisition unit for acquiring a user position to move virtually;
A selection unit that selects a plurality of key frame images whose shooting positions are in the vicinity of the user position from among a plurality of key frame images;
Interpolation for obtaining an interpolated frame image at the user position from at least one of the selected plurality of key frame images by an interpolation process based on the distance between the user position and the shooting positions of the selected key frame images. A processing unit;
A computer program for causing a computer to function as a moving image generating apparatus.