JP2019191528A - Information processing apparatus and program - Google Patents

Abstract

An object of the present invention is to provide an information processing device capable of presenting a region of interest of a omnidirectional image in an easy-to-understand manner while reducing the time and effort required for user operation. According to the present invention, a partial image generation unit that generates a partial image centering on a gazing point designated by a user from a celestial sphere image, and a UI screen that displays the generated partial image. A registration unit for registering the gazing point, a gazing point registration unit for registering the gazing point in response to a user's request via the UI screen, and a gazing point registered most recently is currently designated. An interpolation line calculation unit that calculates an interpolation line that interpolates between the gazing point that is present and an interpolation line that interpolates between the registered gazing point and the gazing point registered immediately before that. An information processing apparatus including a line registration unit, wherein the UI unit superimposes and displays the interpolation line and the registered gazing point on the partial image of the UI screen is provided. [Selection diagram] Fig. 3

Description

  The present invention relates to an information processing apparatus and a program.

  In recent years, with the widespread use of omnidirectional cameras, there are increasing opportunities to view omnidirectional images obtained by photographing all azimuths of photographing points using a dedicated viewer. Here, the omnidirectional image dedicated viewer creates a 3D model by mapping the omnidirectional image onto the surface of a predetermined 3D object, and allows the user to change the display range (pan / tilt / zoom). Accordingly, this is an application for displaying a two-dimensional image obtained by projective transformation of a partial region of the three-dimensional model of the omnidirectional image.

  Here, when using a conventional dedicated viewer to search the attention area in the omnidirectional image, the user performs manual operations (pan / tilt / zoom) while changing the display area. I had to visually check the whole celestial sphere image.

  Regarding this point, Japanese Patent Laid-Open No. 2015-18013 (Patent Document 1) displays a list of attention areas included in the omnidirectional image in advance as a thumbnail, and responds in conjunction with the selection of the thumbnail. Disclosed is a display control device that displays an entire region of interest.

  According to the display control device of Patent Document 1, the effort to search the attention area of the omnidirectional image can be saved, but on the other hand, the trouble of manually selecting the thumbnail remains, and the trouble is that the number of attention areas is the same. It increases as it increases.

  The present invention has been made in view of the above, and an object of the present invention is to provide an information processing apparatus 100 capable of presenting an attention area of an omnidirectional image in an easy-to-understand manner while reducing a user's trouble of operation. To do.

  As a result of earnestly examining the configuration of the information processing apparatus capable of presenting the attention area of the omnidirectional image in an easy-to-understand manner while reducing the user's operation, the present inventors have conceived the following configuration, and It came to.

  That is, according to the present invention, the partial image generation unit that generates a partial image centered on the gazing point designated by the user from the omnidirectional image, and the UI screen that displays the generated partial image A UI unit that accepts registration of a gazing point, a gazing point registration unit that registers the gazing point in response to a user request via the UI screen, and the currently registered gazing point and the currently designated gazing point An interpolation line calculation unit that calculates an interpolation line that interpolates between the gazing point, and an interpolation line registration that registers the interpolation line that interpolates between the registered gazing point and the gazing point registered immediately before the gazing point. The information processing apparatus is provided, wherein the UI unit superimposes and displays the interpolation line and the registered gazing point on the partial image of the UI screen.

  As described above, according to the present invention, it is possible to provide an information processing apparatus capable of presenting an attention area of an omnidirectional image in an easy-to-understand manner while reducing a user's operation.

The figure which shows the smart phone of this embodiment. The conceptual diagram for demonstrating operation which designates a gaze point. The functional block diagram of the smart phone of this embodiment. The flowchart of the process performed at the time of registration of a gaze point. The figure which shows the gaze point registration table of this embodiment. The figure which shows the state transition of UI screen when a gaze point is registered. The flowchart of the process which calculates an interpolation line. The figure for demonstrating the process which calculates an interpolation line. The flowchart of the process performed when a gaze point icon is selected. The figure which shows the state transition of UI screen when a gaze point is changed. The figure which shows the state transition of UI screen when a gaze point is deleted. The flowchart of the process performed when an interpolation line icon is selected. The figure which shows the state transition of UI screen when an interpolation line icon is selected. The figure which shows UI screen when advancing direction is changed. The figure which shows UI screen when a display mode switches. The figure for demonstrating another display mode of UI screen. The figure which shows the network system of another embodiment. The figure which shows the smart phone of another embodiment. The functional block diagram of the smart phone and server of another embodiment. The sequence diagram of the process performed at the time of an animation execution instruction | indication. The flowchart of the process which the server of another embodiment performs. The conceptual diagram for demonstrating the process which a partial image division part performs. The conceptual diagram for demonstrating the process which a partial image division part performs. The conceptual diagram for demonstrating the process which an attention point likelihood distribution calculation part performs. The figure which shows a gaze point registration table. The hardware block diagram of this embodiment.

  Hereinafter, although this invention is demonstrated with embodiment, this invention is not limited to embodiment mentioned later. In the drawings referred to below, the same reference numerals are used for common elements, and the description thereof is omitted as appropriate.

  Hereinafter, an information processing apparatus according to an embodiment of the present invention will be described.

  The information processing apparatus according to the present embodiment is an information processing apparatus that includes a display for displaying an omnidirectional image, and can exemplify various personal computers such as a smartphone, a tablet PC, a notebook PC, and a desktop PC.

  In the information processing apparatus according to the present embodiment, an application program (hereinafter referred to as a dedicated application) for executing an animation for continuously displaying an arbitrary area included in the omnidirectional image is installed. This dedicated application provides a UI screen for registering the gazing point of the omnidirectional image, and the user can display the omnidirectional image on the surface of a predetermined three-dimensional object (for example, a spherical object) via the UI screen. An arbitrary point in the three-dimensional model mapped on the inner surface) is registered as a gazing point. Then, this dedicated application changes the gazing point along a path that interpolates between two or more gazing points registered by the user, and executes an animation in which partial images centered on the changing gazing point are connected. In the present embodiment, the “partial image” means a two-dimensional image obtained by projective transformation of an image region centered on an arbitrary gazing point on the three-dimensional model of the omnidirectional image.

  FIG. 1 shows a smartphone 100 as an example of the information processing apparatus of this embodiment. As shown in FIG. 1, a UI screen (hereinafter referred to as a UI screen) for registering a gazing point of an omnidirectional image is displayed on the touch panel display 18 of the smartphone 100.

  As shown in FIG. 1, the UI screen includes a partial image display area 20 and an icon display area 30, and the partial image display area 20 includes a partial image area (user area) of the omnidirectional image. A partial image obtained by projective transformation of the image region centered on the gazing point designated by the user is displayed, and a button 21 for accepting display mode switching is superimposed on the partial image, an animation playback time, Information 22 indicating the data capacity, a button 23 for accepting registration of the gazing point, and a cross cursor 24 indicating the gazing point of the omnidirectional image are displayed. In the icon display area 30, various icons to be described later are displayed in addition to a button 32 for receiving an animation execution instruction.

  Here, in the present embodiment, by swiping the partial image display area 20 in which the partial image is displayed in an arbitrary direction, an arbitrary point on the omnidirectional image can be designated as a gazing point. In addition, the angle of view on the omnidirectional image can be specified by pinching the partial image.

  FIG. 2 exemplarily shows a partial image panned / tilted by a swipe operation and a partial image zoomed in / zoomed out by a pinch-in / pinch-out operation. In the present embodiment, the cross cursor 24 indicating the gazing point on the omnidirectional image is always displayed at the center of the partial image.

  Next, the functional configuration of the smartphone 100 according to the present embodiment will be described based on the functional block diagram shown in FIG.

  As illustrated in FIG. 3, the smartphone 100 includes a UI unit 102, a partial image generation unit 103, a gazing point registration unit 104, an interpolation line calculation unit 105, an interpolation line registration unit 106, and a gazing point control unit 107. The animation execution unit 108, the moving image data generation unit 109, and the storage area 110 are configured.

  The UI unit 102 is a means for displaying the above-described UI image on the touch panel display 18 and accepting various requests including registration of the gazing point of the omnidirectional image from the user via the UI image.

  The partial image generation unit 103 is a unit that generates a partial image centered on a gazing point designated by the user from the omnidirectional image read from the storage area 110.

  The gaze point registration unit 104 is means for registering the gaze points designated by the user in the order of the requests in response to the user registration request via the UI screen.

  The interpolation line calculation unit 105 is a means for calculating an interpolation line for interpolating between the most recently registered gazing point and the currently designated gazing point.

  The interpolation line registration unit 106 is a means for registering an interpolation line for interpolating between the registered gazing point and the gazing point registered immediately before the gazing point.

  The gazing point control unit 107 is a means for changing the gazing point along a path for interpolating two or more registered gazing points.

  The animation execution unit 108 is a means for executing an animation in which partial images centered on a transition gazing point are connected in the order of transition.

  The moving image data generation unit 109 is means for converting the generated plurality of partial images into moving image data in a general-purpose file format.

  The storage area 110 is a means for holding various data.

  In the present embodiment, a computer mounted on the smartphone 100 functions as each unit described above by executing a dedicated application.

  Although the functional configuration of the smartphone 100 has been described above, the contents of processing executed at the time of registering a gaze point will be described based on the flowchart shown in FIG.

  The smartphone 100 displays a UI screen on the touch panel display 18 and repeatedly executes a series of processes described below.

  First, in step 101, the UI unit 102 displays the partial image generated by the partial image generation unit 103 (that is, the partial image centered on the gazing point designated by the user) in the partial image display area 20 of the UI screen.

  In subsequent step 102, the gazing point registration unit 104 determines whether or not there is a registered gazing point. If no gazing point has been registered before (step 102, No), the process proceeds to step 111. Monitor gaze point registration requests. As a result, when there is no registration request from the user (step 111, No), the processing is ended as it is.

  On the other hand, when there is a registration request (step 111, Yes), the process proceeds to step 112, and the watch point registration unit 104 registers the watch point currently designated by the user in the watch point registration table 500.

  FIG. 5A shows a watch point registration table 500 held in the storage area 110 described above. As shown in FIG. 5A, the gazing point registration table 500 includes a three-dimensional coordinate (x, y, z) of the gazing point, an elevation angle and an azimuth angle that define the gazing point, and color information of an object representing the gazing point ( R, G, B, A) are provided for storing fields respectively. In step 112, the elevation angle and the azimuth corresponding to the gaze point designated when detecting the registration request from the user, The three-dimensional coordinates (x, y, z) on the omnidirectional image obtained from the elevation angle and the azimuth are registered in the gazing point registration table 500. The color information will be described later.

  Thereafter, the UI unit 102 superimposes and displays the object representing the gazing point registered in the gazing point registration table 500 on the partial image displayed in the partial image display area 20 in the following step 113, and in the subsequent step 114, A gazing point icon corresponding to the registered gazing point is displayed in the icon display area 30, and the process ends.

  On the other hand, if one or more gazing points are already registered in the gazing point registration table 500 as a result of the determination in step 102 (step 102, Yes), the process proceeds to step 103. In the subsequent step 103, the interpolation line calculation unit 105 interpolates between the most recently registered gazing point among the already registered gazing points and the gazing point of the partial image currently designated by the user. In the subsequent step 104, the UI unit 102 superimposes and displays the calculated interpolation line on the partial image. The details of the method for calculating the interpolation line described above will be described later.

  In the subsequent step 105, the watch point registration unit 104 monitors a watch point registration request. If there is no registration request from the user (step 105, No), the process is terminated. On the other hand, if there is a registration request from the user (step 105, Yes), in the following step 106, the gazing point registration unit 104 registers the gazing point currently designated by the user in the gazing point registration table 500.

  In the subsequent step 107, the UI unit 102 superimposes and displays the gazing point registered in step 106 on the partial image, and in the subsequent step 108, the gazing point icon corresponding to the registered gazing point is displayed in the icon display area 30. .

  In the subsequent step 109, the interpolation line registration unit 106 calculates an interpolation line that interpolates between the newly registered gazing point and the gazing point registered immediately before (i.e., when the gazing point is newly registered). Interpolation line) is registered in the interpolation line registration table 600 held in the storage area 110. Details of the registration of the interpolation line will be described later.

  Thereafter, in the subsequent step 110, the UI line 102 arranges the interpolation line icons corresponding to the interpolation lines registered in step 109 in the order of registration and displays them in the icon display area 30, and the process ends.

  Here, the state transition of the UI screen when the gazing point is registered will be described with reference to FIG.

  If the user taps the button 23 in a state where no gazing point has been registered before that, the position designated by the user (that is, the position of the cross cursor 24) is registered as the first gazing point. At this time, as shown in FIG. 6A, the object 25 a representing the registered gazing point is superimposed and displayed on the partial image so as to overlap the cross cursor 24. At the same time, a gazing point icon 35 a corresponding to the registered gazing point 25 a is displayed in the icon display area 30. In the present embodiment, the gazing point icon 35a is displayed as a thumbnail of a partial image centered on the registered gazing point 25a (hereinafter the same).

  When the user swipes the partial image display area 20 in the direction of the broken arrow line from the state shown in FIG. 6A, as shown in FIG. 6B, the three-dimensional gaze point to be designated according to the operation Coordinates are changed. At this time, according to the user's swipe operation, the object 25a moves relatively away from the cross cursor 24, and an object 26a representing an interpolation line starting from the cross cursor 24 and ending at the object 25a is displayed on the partial image. It is displayed superimposed.

  In the state shown in FIG. 6B, when the user taps the button 23, the position of the cross cursor 24 at that time is registered as the second gazing point. At this time, as shown in FIG. 6C, the object 25b representing the second registered gazing point is superimposed on the partial image so as to be superimposed on the cross cursor 24 and is registered second. A gazing point icon 35b corresponding to the gazing point icon 25b is displayed on the right side of the gazing point icon 35a.

  At this time, an interpolation line 26a for interpolating between the second registered gazing point 25b and the first registered gazing point 25a is registered, and at the same time, as shown in FIG. An interpolation line icon 36a corresponding to the registered interpolation line 26a is displayed between the gazing point icon 35a and the gazing point icon 35b.

  FIG. 6D shows a UI screen at the time when the third gaze point following the gaze point 25b is registered. As shown in FIG. 6D, on the partial image, an object 25c representing the third registered gazing point is superimposed and displayed on the partial image so as to overlap the cross cursor 24. A gazing point icon 35c corresponding to the third registered gazing point 25c is displayed on the right side of the gazing point icon 35b.

  At the same time, an interpolation line 26b for interpolating between the third registered gazing point 25c and the gazing point 25b registered immediately before is registered, as shown in FIG. 6 (d). An interpolation line icon 36b corresponding to the interpolation line 26b thus displayed is displayed between the gazing point icon 35b and the gazing point icon 35c.

  The UI screen state transition when a gazing point is registered has been described above. Next, a process for calculating an interpolation line for interpolating between two gazing points will be described based on the flowchart shown in FIG. To do. In the following, description will be made with reference to the three-dimensional model of the omnidirectional image shown in FIG.

First, in step 201, by linearly interpolating between the fixation point _P elevation _{E 1 1} and elevation _{E 2} of the focus point _{P 2} at equal intervals, n (n is an integer of 1 or more. Hereinafter the same.) Of An interpolation value e _i (i = 1 to n) of the elevation angle is calculated.

In step 202, between the azimuth angle _{A 2} of the focus point _{P 2} and the azimuth angle _{A 1} of the gaze point _{P 1} by linear interpolation at regular intervals, the interpolation values of the n azimuth a _{i (i} = 1~ n) is calculated.

In step 203, it sets the value of the counter i indicating the order of the interpolation value as viewed from the fixation point P ₁ to the initial value [1].

In the next step 204, three-dimensional coordinates on the three-dimensional model of the omnidirectional image are calculated based on the interpolation value e _{i of the} elevation angle and the interpolation value a _i of the azimuth angle.

In step 205, the gazing point as viewed from the _{P 1} i-th interpolation gazing point _{p i,} 3-dimensional coordinates calculated in step 204 (x, y, z) to be registered in the interpolated line registration table 600. FIG. 5B shows an interpolation line registration table 600 in which the interpolation gazing point p is registered. As shown in FIG. 5B, the interpolation line registration table 600 includes the three-dimensional coordinates (x, y, z) of the interpolation gazing point p and the color information (R, G, B, A) of the object representing the interpolation line. Are provided for storing fields.

  In the next step 206, the value of the counter i is incremented by one.

  Thereafter, the processing of steps 204 to 206 is repeated until the value of counter i exceeds n (step 207, No), and the processing ends when the value of counter i exceeds n (step 207, Yes). To do.

  When the series of processes described above is completed, n interpolation gazing points p shown in FIG. 8B are registered in the interpolation line registration table 600. In this embodiment, one interpolation line registration table 600 is created for each interpolation line, and an object representing the interpolation line is generated based on n interpolation gazing points p registered in the interpolation line registration table 600. The

  The process for calculating the interpolation line has been described above. Next, the process that is executed when the gazing point icon displayed in the icon display area 30 is selected will be described based on the flowchart shown in FIG.

  Upon detecting that the user has selected the gaze point icon, the UI unit 102 determines whether or not the user has requested deletion of the gaze point in step 301. As a result, if the user has not requested deletion of the gazing point (step 301, No), the process proceeds to step 302, where the UI unit 102 selects the partial image display area 20 of the UI screen. Transition to a state of accepting a change of the point of gaze corresponding to the viewpoint icon.

  Thereafter, in step 303, the UI unit 102 waits for the user to change the gaze point for a predetermined period. As a result, when the user times out without changing the point of gaze (No at Step 303), the process is terminated as it is. On the other hand, when the user changes the point of gaze (step 303, Yes), the process proceeds to step 304.

  In the subsequent step 304, the interpolation line calculation unit 105 recalculates an interpolation line for interpolating between the changed gazing point and another gazing point registered immediately before or immediately thereafter. The line registration unit 106 registers the recalculated interpolation line in the interpolation line registration table 600.

  Thereafter, in the subsequent step 306, the UI unit 102 superimposes and displays the changed gazing point and the recalculated interpolation line on the partial image, and ends the process.

  Here, the state transition of the UI screen when the user changes the gaze point will be described with reference to FIG.

  FIG. 10A shows a UI screen in a state where three gazing points (25a, 25b, 25c) are registered. In this state, when the user taps the gaze point icon 35b corresponding to the second registered gaze point 25b, the UI screen accepts a change in the position of the gaze point 25b corresponding to the tapped gaze point icon 35b. , Transition to the state shown in FIG. In response to this, the user changes the position of the second gazing point 25b to an arbitrary position.

  FIG. 10C shows the UI screen after the user changes the position of the second gazing point 25b. At this time, the three-dimensional coordinates of the gazing point 25b in the gazing point registration table 500 are changed with the change of the gazing point 25b. In addition, an interpolation line 26a for interpolating between the gazing point 25b and the gazing point 25a and an interpolation line 26b for interpolating between the gazing point 25b and the gazing point 25c are recalculated, and each interpolation line corresponds to the calculation result. The contents of the interpolation line registration table 600 are changed.

  Returning to FIG. 9 again, the description will be continued.

  As a result of the determination in the previous step 301, when the user requests deletion of the gazing point after selecting the gazing point icon (step 301, Yes), the process proceeds to step 307. In the subsequent step 307, the gazing point registration unit 104 deletes the registration of the gazing point corresponding to the gazing point icon selected by the user from the gazing point registration table 500, and in the subsequent step 308, the UI unit 102 deletes the gazing point. Hide the gaze point icon corresponding to.

  Thereafter, in the subsequent step 309, the interpolation line registration unit 106 deletes the registration of the interpolation line having the deleted gazing point as the start point or end point from the interpolation line registration table 600, and in the subsequent step 310, the UI unit 102 deletes it. Hides the interpolation line icon corresponding to the interpolation line.

  Thereafter, in step 311, the interpolation line calculation unit 105 newly calculates an interpolation line for interpolating between the gaze point registered immediately before and the gaze point registered immediately after that of the deleted gaze point. To do.

  Thereafter, in the following step 312, the interpolation line registration unit 106 registers the newly calculated interpolation line in the interpolation line registration table 600, and in the subsequent step 313, the UI unit 102 adds the newly calculated interpolation line of the partial image. The process is terminated after being superimposed on the top.

  If the gaze point deleted in the previous step 307 is the last gaze point, the process in steps 311 to 313 is performed after the process in steps 308 to 310 is performed. finish.

  Here, the state transition of the UI screen when the gazing point is deleted will be described with reference to FIG.

  FIG. 11A shows a UI screen in a state where three gazing points (25a, 25b, 25c) are registered. In this state, when the trash icon 27 is tapped after the user taps the gazing point icon 35b corresponding to the second gazing point 25b registered by the user, the UI screen changes to the state shown in FIG.

  At this time, in the gazing point registration table 500, registration of the gazing point 25b is deleted, a new interpolation line 26d for interpolating between the gazing point 25a and the gazing point 25c is newly calculated, and the calculation result is the interpolation line. The new interpolation line 26d is registered in the registration table 600 and is superimposed on the partial image.

  The processing executed when the gazing point icon is selected has been described above. Subsequently, the processing executed when the interpolation line icon is selected will be described based on the flowchart shown in FIG.

  In response to the user selecting the interpolation line icon, the UI unit 102 displays a parameter setting screen in step 401.

  Here, FIG. 13 shows the state transition of the UI screen when the user selects the interpolation line icon. As shown in FIG. 13A, when the user taps the interpolation line icon 36b, the UI screen changes to the parameter setting screen shown in FIG. 13B.

  As shown in FIG. 13B, the parameter setting screen includes a parameter setting area 40 and a preview display area 50. Here, in the parameter setting area 40, a button group 42 for selecting a traveling direction (shortest / distance), a slider 43 for setting a speed, a button group 44 for selecting an easing curve, and a stop. A button group 45 for selecting a time is displayed, and any one of the button group 44 is selected by tapping any one of the button groups 42 to select a traveling direction, or operating the slider 43 to select a speed. The easing curve can be selected by tapping, or the stop time can be selected by tapping any of the button group 45.

  Here, the advancing direction means the moving direction of the gazing point when executing the animation, and the speed means the moving speed of the gazing point. Also, the easing curve means a curve indicating the time change of the movement acceleration of the gazing point, and the stop time is the time for stopping the movement of the gazing point at each registered point (that is, the time for stopping the animation). Means.

  Returning to FIG. 12, the description will be continued.

  After the parameter setting screen is displayed, the UI unit 102 determines whether or not the user has changed parameters (speed, easing curve, stop time) other than the traveling direction on the parameter setting screen in the subsequent step 402, and When the parameter other than the traveling direction is changed (step 402, Yes), the process proceeds to step 403.

  In the subsequent step 403, the UI unit 102 includes three types of parameters (speed, easing curve, and stop time) related to the animation executed along the path corresponding to the interpolation line corresponding to the interpolation line icon selected by the user. The parameter value selected by the user is changed to the value selected by the user, and the process is terminated.

  On the other hand, if the result of determination in step 402 is that the user has changed the direction of travel (step 402, No → step 404, Yes), the process proceeds to step 405. If the user times out without requesting any parameter change for a predetermined period (step 402, No → step 404, No), the process is terminated.

  In subsequent step 405, the interpolation line calculation unit 105 recalculates the interpolation line corresponding to the interpolation line icon selected by the user according to the traveling direction changed by the user. For example, when the user operates the button group 42 and changes the traveling direction from “shortest” to “detour”, the path from the start point to the end point of the interpolation line corresponding to the interpolation line icon selected by the user is the longest. Recalculate the correct interpolation line. In subsequent step 406, the interpolation line registration unit 106 registers the recalculated interpolation line in the interpolation line registration table 600, and the process ends.

  Here, FIG. 14A shows a UI screen in which two gazing points 25a and 25b are registered. From this state, when the traveling direction is changed from “shortest” to “detour” in the procedure described above, the UI screen changes to the state shown in FIG.

  In this embodiment, when the user taps the play button 52 displayed in the preview display area 50, the animation execution unit 108 performs animation along the path corresponding to the interpolation line corresponding to the interpolation line icon selected by the user. The UI unit 102 displays the animation in the preview display area 50. By this preview reproduction, the user can confirm whether or not the animation is intended by changing the parameters.

  Here, the function of the button 21 described in FIG. 1 will be described. In the present embodiment, by tapping the button 21, display / non-display of various objects (gaze point 25, interpolation line 26, cross cursor 24) superimposed on the partial image can be switched. FIG. 15A shows the UI screen when the display mode is selected, and FIG. 15B shows the UI screen when the non-display mode is selected.

  As described above, in this embodiment, the user registers an arbitrary gazing point on the omnidirectional image via the UI screen provided by the UI unit 102, and the gazing point control unit 107 is registered. Then, the gazing point is shifted along a path for interpolating two or more gazing points, and the animation execution unit 108 executes an animation in which partial images centered on the changing gazing point are connected in the transition order. Thereby, the attention area of the omnidirectional image is presented in an easy-to-understand manner in the form of animation.

  In addition, according to the present embodiment, on the UI screen for registering the gazing point, the gazing point and the interpolation line as the movement path thereof are visualized on the partial image, so that it is easy to intuitively understand the flow of the entire animation. This makes it easier to edit the animation.

  In the embodiment described above, the registered gaze point is displayed with a circle and the interpolation line is displayed with a solid line. However, in the present embodiment, the object representing the registered gaze point and the interpolation line are represented. The form of the object can be determined arbitrarily.

  In the present embodiment, two or more registered gazing points may be displayed in a distinguishable manner, or two or more registered interpolation lines may be displayed in a distinguishable manner. . In that case, a different color can be assigned to each of the registered gazing points by individually setting the color information (R, G, B, A) of the gazing point registration table 500 (FIG. 5A), Different colors can be assigned to each of the registered interpolation lines by individually setting the color information (R, G, B, A) of the interpolation line registration table 600 (FIG. 5B).

  In addition, in the present embodiment, registered gaze points and gaze point icons can be displayed in a manner in which the correspondence relationship can be grasped, or registered interpolation lines and interpolation line icons can be grasped in the correspondence relationship. It can also be displayed in a manner. In this regard, FIG. 16 is a diagram in which the same number is added to each of the registered gazing point and the corresponding gazing point icon, and each of the registered interpolation line and the corresponding interpolation line icon is different. The UI screen represented by the objects (solid line and broken line) is exemplarily shown.

  As described above, the embodiment of executing the animation based on the gaze point registered by the user has been described as the embodiment of the information processing apparatus that presents the attention area of the omnidirectional image in an easy-to-understand manner. The embodiment will be described. In another embodiment, rather than letting the user register a gaze point, a gaze point that matches the content of the omnidirectional image selected by the user is automatically registered using the learned model.

(Another embodiment)
FIG. 17 shows a network system according to another embodiment. The network system of the present embodiment is configured to include a smartphone 200 and a server 300, and each smartphone 200 and the server 300 are connected via a network 60 so as to communicate with each other. The smartphone 200 is shown as an example of an information processing device that displays an omnidirectional image.

  FIG. 18 shows the smartphone 200. As shown in FIG. 18, the touch panel display 18 of the smartphone 200 displays a UI screen (hereinafter referred to as a UI screen) for accepting execution of an animation for presenting the attention area of the omnidirectional image in an easy-to-understand manner. ing.

  As shown in FIG. 18, the UI screen is composed of an animation display area 70 and a parameter setting area 80 for setting an execution condition for animation. Here, a playback button 72 for instructing execution of the animation is displayed in the animation display area 70, and an input box 82 for designating the number of gazing points and an animation display area are displayed in the parameter setting area 80. An input box 83 for designating the number of loops, a button group 84 for designating the animation stop time at the gazing point, and a button group 85 for designating the generation location of the moving image data are displayed.

  Then, based on the functional block diagram shown in FIG. 19, the functional structure of the smart phone 200 of this embodiment and the server 300 is demonstrated.

  As illustrated in FIG. 19, the smartphone 200 includes a UI unit 202, a gaze point registration request unit 203, a gaze point control unit 204, an animation execution unit 205, a moving image data generation unit 206, and a storage area 207. Consists of.

  The UI unit 202 is means for displaying a UI image for accepting the execution of an animation on the touch panel display 18 and accepting setting of an execution condition of the animation and an instruction for the execution from the user via the UI image.

  The gaze point registration request unit 203 is a means for requesting the server 300 to register a gaze point in response to receiving an animation execution instruction from the user.

  The gazing point control unit 204 is a means for changing the gazing point along a path for interpolating two or more gazing points registered by the server 300.

  The animation execution unit 205 is a means for executing an animation in which partial images centered on a transition point of interest are connected in the order of transition.

  The moving image data generation unit 206 is a means for converting the executed animation into moving image data in a general-purpose file format.

  The storage area 207 is a means for holding various data.

  In the present embodiment, a computer mounted on the smartphone 200 functions as each unit described above by executing a dedicated application.

  On the other hand, as shown in FIG. 19, the server 300 includes a partial image dividing unit 301, a feature amount extracting unit 302, an importance calculating unit 303, an attention point likelihood distribution calculating unit 304, and an attention point calculating unit 305. , A gazing point registration unit 306, a gazing point control unit 307, an animation execution unit 308, and a moving image data generation unit 309.

  The partial image dividing unit 301 converts the omnidirectional image received from the smartphone 200 into an image in the Equirectangular format (equal equirectangular projection), and re-projects the converted image in a plurality of different directions into a plurality of partial images. It is a means to divide.

  The feature amount extraction unit 302 is a means for extracting a feature amount from each partial image.

  The importance calculation unit 303 is a means for calculating the importance for each position of the image to be processed based on a predetermined regression model (learned model) from the extracted feature amount.

  The attention point likelihood distribution calculation unit 304 is a means for calculating the likelihood distribution of the attention point from the calculated importance.

  The attention point calculation unit 305 is means for calculating the attention point based on the calculated likelihood distribution of the attention point.

  The gaze point registration unit 306 is means for registering two or more gaze points based on the two or more attention points calculated by the attention point calculation unit 305.

  The gazing point control unit 307 is a means for changing the gazing point along a path for interpolating two or more gazing points registered by the gazing point registration unit 306.

  The animation execution unit 308 is means for executing an animation in which partial images centered on a transition point of interest are connected in the order of transition.

  The moving image data generation unit 309 is means for converting the executed animation into moving image data in a general-purpose file format.

  In this embodiment, the server 300 functions as each unit described above when a computer constituting the server 300 executes a predetermined program.

  The functional configurations of the smartphone 200 and the server 300 have been described above. Next, the contents of processing executed when the user instructs execution of animation in the present embodiment will be described based on the sequence diagram shown in FIG. To do.

  First, the user selects a desired image from the omnidirectional images stored in the storage area 207, and then performs an animation via the parameter setting area 80 on the UI screen (see FIG. 18) displayed by the UI unit 202. After the execution condition is set, the playback button 72 in the animation display area 70 is tapped to instruct the execution of the animation (S1).

  In response to this, the gaze point registration request unit 203 includes a gaze point that includes the omnidirectional image selected by the user and the execution conditions set by the user (number of gaze points, number of loops, stop time, generation location of moving image data). A registration request is generated and transmitted to the server 300 (S2).

  In response to this, the server 300 creates a gaze point registration table based on the setting of the omnidirectional image and the “number of gaze points” included in the gaze point registration request received from the smartphone 200 (S3).

  Here, the process in which the server 300 creates the watch point registration table in S3 will be described based on the flowchart shown in FIG.

  First, in step 501, the partial image dividing unit 301 converts the omnidirectional image included in the gaze point registration request into an omnidirectional image in an Equirectangular format (equal equirectangular projection), and then changes the shooting direction of the omnidirectional image. Spatial equal division is performed, and the omnidirectional image is reprojected in a plurality of different shooting directions to be divided into a plurality of partial images. Hereinafter, a procedure for dividing the image into partial images will be specifically described.

  As shown in FIG. 22A, from the omnidirectional image in the Equirectangular format, pixel values in an arbitrary three-dimensional direction can be obtained from the coordinate values of longitude and latitude. The omnidirectional image in the Equirectangular format is conceptually Can be regarded as pixel values plotted on a unit sphere. Therefore, in the present embodiment, as shown in FIG. 22B, a predetermined projection plane is defined, and the center of the unit sphere is set as the projection center O, and the pixel of the omnidirectional image in the Equirectangular format according to the following equation (1) A partial image is obtained by performing perspective projection conversion corresponding to the pixel value (x, y) on the projection plane on which the value (θ, φ) is defined. In the following formula (1), P indicates a perspective projection matrix, and the equal sign indicates that it is equal to a scalar multiple other than 0.

  Specifically, after defining a regular polyhedron having a center common to the unit sphere as the projection plane of the omnidirectional image in Equirectangular format, partial projection is performed by performing perspective projection conversion with the normal direction of each plane as the viewing direction. Get. FIG. 23A shows an example in which a regular octahedron is defined as the projection plane of the omnidirectional image, and FIG. 23B shows an example in which a regular dodecahedron is defined as the projection plane of the omnidirectional image.

  In subsequent step 502, the feature amount extraction unit 302 extracts a predetermined feature amount from each partial image obtained in the previous step 501. Note that examples of the feature quantity extracted here include color, edge, saliency, object position / label, LBP, Haar like feature, HOG, SIFT, and the like.

  In the subsequent step 503, the importance calculation unit 303 calculates the importance for each position (for each pixel) of the omnidirectional image from the feature amount extracted from each partial image using a predetermined regression model. This point will be specifically described below.

The vector in which the feature amounts for each position of the i-th partial image of the partial image obtained by dividing the omnidirectional image into N is arranged as l _i, and the vector in which the importance for each position of the omnidirectional image is arranged is set as g. Then, a regression model f shown in the following formula (2) is considered.

  Here, as a specific form of the regression model f, a linear transformation represented by the following formula (3) can be exemplified.

In the above formula (3), W and b represent parameters. In the present embodiment, training data having the feature quantity l _i as input and the importance g as output is prepared in advance, and the parameters W and b are identified by learning the training data (learning). Ready model).

  As a premise thereof, in the present embodiment, importance g, which is output of training data (teacher data), is acquired by an appropriate method. The simplest method for obtaining the importance level g is to let the subject specify a region that the subject thinks important in the target image, and set the importance level of each pixel constituting the region specified by the subject to “1”, otherwise A method of setting the importance of the pixel of “0” to “0” can be mentioned. In addition to this, after acquiring the viewpoint trajectory of the subject viewing the target image with an eye tracker or the like, Gaussian blurring is applied to the acquired viewpoint trajectory (line), and the importance level normalized based on the shading level ( The method of acquiring 0-1) can be illustrated.

  In the subsequent step 504, the attention point likelihood distribution calculation unit 304 pays attention based on the importance distribution calculated in the previous step 503 under the design philosophy that the attention point of the user exists in the higher importance direction. The likelihood distribution of points is calculated. In the present embodiment, as shown in FIG. 24, an area R centering on the shooting direction passing through the viewpoint A is defined on the unit image surface, and the addition value obtained by adding the importance of each position in the area R is added. As the attention point likelihood of the viewpoint A. Further, in the present embodiment, a weight that attenuates as the distance from the viewpoint A is given to the importance of each position in the region R, and a weighted addition value of the importance using the weight is assigned to the viewpoint A. It is also possible to calculate as the attention point likelihood.

  Here, if the three-dimensional vector of the shooting direction is expressed as p and the importance of the shooting direction p is expressed as g (p), the attention point likelihood a (p) can be formulated as the following equation (4). it can.

  In the above equation (4), η represents a monotonically increasing function, w (p, q) represents a weight, the integral is a definite integral, and the integral range is the entire photographing unit spherical surface. In this embodiment, η can be an exponential function, and w (p, q) can be a function represented by the following equation (5).

  The above equation (5) follows the von Mises distribution, and takes the maximum value when the directions p and q match, and takes the minimum value when p and q face in the opposite direction. In the present embodiment, the attenuation rate of the weight can be determined by the parameter α, and the angle of view considering the attention point can be reflected.

In the present embodiment, the weight w (p, q) is represented by the following equation (6), with {α _i } as a parameter, and a polynomial of the inner product of directions p and q as an exponential function argument: You can also.

  Returning to FIG. 21, the description will be continued.

  In subsequent step 505, the attention point calculation unit 305 obtains a local maximum value of the attention point likelihood a (p). Here, when the set value of the “number of gazing points” is M (M is an integer of 2 or more) in the gazing point registration request, the attention point calculation unit 305 performs M for the attention point likelihood a (p). Find the local maximum. Note that the local maximum value of the attention point likelihood a (p) can be obtained by searching by a hill-climbing method from the randomly generated initial value of p. Thereafter, M photographing directions p corresponding to the obtained M local maximum values are obtained, and positions corresponding to the respective directions are acquired as attention points. More specifically, M photographing directions p1, p2,..., PM that maximize the evaluation function shown in the following equation (7) are obtained, and attention points (θ, φ) corresponding to the directions are obtained. To do.

  In the above equation (7), d is a function that expresses the distance between viewpoints. Examples of such a function include the variance of p1, p2,..., PM and the sum of Euclidean distances between viewpoints. Etc. can be illustrated. By using such an evaluation function, M attention points that are separated from each other can be acquired.

  In subsequent step 506, the gaze point registration unit 306 creates a gaze point registration table in the following procedure. Specifically, first, three-dimensional coordinates (x, y, z) on the omnidirectional image corresponding to the M attention points (θ, φ) calculated in the previous step 505 are calculated. Acquired as individual gazing points (x, y, z).

  Next, the acquired M gazing points (x, y, z) are sequentially registered in the table according to a predetermined rule. In the present embodiment, for example, a reference gazing point (home position) is determined in advance on the omnidirectional image, and among the acquired M gazing points, the closest gazing point from the home position is the first. Among the remaining (M-1) gazing points, the second gazing point nearest to the first gazing point is registered secondly, and the remaining (M-2) gazing points, The nearest gaze point is registered third from the second gaze point, and the procedure such as... Is repeated to register M gaze points in the table in order. FIG. 25 shows the gaze point registration table 700 created as described above.

  Returning to FIG. 20, the description will be continued.

  In S3, after the server 300 creates the gazing point registration table 700, the process is divided according to the setting content of “video data generation location” included in the gazing point registration request.

  First, when the setting of “video data generation location” is “client”, the server 300, the gaze point registration table 700 created in S3, the omnidirectional image received in S2 and the execution condition (loop) The number of times and the stop time are transmitted to the smartphone 200 (S4).

  In response to this, the gazing point control unit 204 of the smartphone 200 receives the gazing point registered in the gazing point registration table 700 received in S4 along the path in which the gazing point is registered in the registration order. Then, the gazing point is changed, and the animation execution unit 205 executes the animation in which the partial images centered on the changing gazing point are connected according to the execution condition (the number of loops and the stop time) received in S4 (S5). Thereafter, the moving image data generation unit 206 converts the animation executed in S5 into moving image data in a general-purpose file format and stores it in the storage area 207 (S6).

  Note that when the generation of moving image data is completed on the smartphone 200, a notification that the generation of moving image data has been completed is transmitted from the smartphone 200 to the server 300. Upon receiving this notification, the server 300 may delete the omnidirectional image received from the smartphone 200, the execution condition, information stored in the gaze point registration table, and the like, and may consider personal information.

  On the other hand, when the setting of “video data generation location” is “server”, the gazing point control unit 307 of the server 300 registers the gazing point registered in the gazing point registration table 700 created in S3. The gazing point is shifted on the omnidirectional image received in S2 along the path to be interpolated in order, and the animation execution unit 308 receives an animation in which partial images centered on the changing gazing point are connected in S2. The process is executed according to the executed conditions (loop count, stop time) (S7). Thereafter, the moving image data generation unit 309 converts the animation executed in S7 into moving image data in a general-purpose file format (S8).

  Thereafter, the server 300 transmits the moving image data converted in S8 to the smartphone 200 (S9). In response to this, the smartphone 200 reproduces and displays the received moving image data, and then saves it in the storage area 207 (S10).

  When the smartphone 200 receives the moving image data, the smartphone 200 transmits a notification that the moving image data has been received to the server 300. Upon receiving this notification, the server 300 deletes the omnidirectional image received from the smartphone 200, the execution conditions, the information stored in the gazing point registration table, the moving image data, etc., and considers the personal information. Also good.

  As described above, according to another embodiment described above, the attention area of the omnidirectional image can be presented in an easy-to-understand manner without causing the user to register the gazing point. In the embodiment, the omnidirectional image is described as a suitable content to which the present invention is applied. However, in addition to the omnidirectional image, if the image has an angle of view of 180 degrees or more, this The invention can be applied. In this way, it is possible to increase the appeal of the image to the user by widening the range of expression of the display with respect to an image having a wide angle of view and difficult to display on a normal display screen.

  Finally, the hardware configuration of this embodiment will be described with reference to FIG.

  As shown in FIG. 26A, the computer mounted on the smartphones 100 and 200 of the present embodiment includes a processor 10 that controls the operation of the entire apparatus, a ROM 12 that stores a boot program, a firmware program, and the like, An RAM 13 that provides an execution space, an auxiliary storage device 14 that stores a program for operating the computer as each of the above-described means, an operating system (OS), and the like, and an input for connecting a touch panel display 18 and the like. An output interface 15 and a network interface 16 for connecting to a network 60 are provided.

  As shown in FIG. 26B, the computer mounted on the server 300 of this embodiment includes a processor 310 that controls the operation of the entire apparatus, a ROM 312 that stores a boot program, a firmware program, and the like, A RAM 313 that provides an execution space, an auxiliary storage device 314 for storing a program, an operating system (OS), and the like for causing the computer to function as each unit described above, an input / output interface 315, and a network 60 are connected Network interface 316.

  Note that each function of the above-described embodiment can be realized by a program described in C, C ++, C #, Java (registered trademark), etc., and the program of this embodiment includes a hard disk device, a CD-ROM, an MO, a DVD, and the like. It can be stored in a recording medium such as a flexible disk, EEPROM, EPROM and distributed, and can be transmitted via a network in a format that can be used by other devices.

  As described above, the present invention has been described with the embodiment. However, the present invention is not limited to the above-described embodiment, and as long as the operations and effects of the present invention are exhibited within the scope of embodiments that can be considered by those skilled in the art. It is included in the scope of the present invention.

10 ... Processor 12 ... ROM
13 ... RAM
14 ... auxiliary storage device 15 ... input / output interface 16 ... network interface 18 ... touch panel display 20 ... partial image display area 21 ... button 22 ... information 23 ... button 24 ... cross cursor 25 ... gaze point (object)
26 ... Interpolation line (object)
27 ... Trash can icon 30 ... Icon display area 32 ... Button 35 ... Gaze point icon 36 ... Interpolation line icon 40 ... Parameter setting area 42 ... Button group 43 ... Slider 44 ... Button group 45 ... Button group 50 ... Preview display area 52 ... Playback Button 60 ... Network 70 ... Animation display area 72 ... Playback button 80 ... Parameter setting area 82 ... Input box 83 ... Input box 84 ... Button group 85 ... Button group 100 ... Smartphone (information processing apparatus)
102 ... UI unit 103 ... partial image generation unit 104 ... gaze point registration unit 105 ... interpolation line calculation unit 106 ... interpolation line registration unit 107 ... gaze point control unit 108 ... animation execution unit 109 ... moving image data generation unit 110 ... storage area 200 ... Smartphone (information processing device)
202 ... UI unit 203 ... gaze point registration request unit 204 ... gaze point control unit 205 ... animation execution unit 206 ... moving image data generation unit 207 ... storage area 300 ... server 301 ... partial image division unit 302 ... feature quantity extraction unit 303 ... important Degree calculation unit 304 ... attention point likelihood distribution calculation unit 305 ... attention point calculation unit 306 ... gaze point registration unit 307 ... gaze point control unit 308 ... animation execution unit 309 ... moving image data generation unit 310 ... processor 312 ... ROM
313 ... RAM
314 ... auxiliary storage device 315 ... input / output interface 316 ... network interface 500 ... gaze point registration table 600 ... interpolation line registration table 700 ... gaze point registration table

JP, 2015-18013, A

Here, when the three-dimensional vector of the shooting direction is expressed as p and the importance of the shooting direction q is expressed as g ( q ), the attention point likelihood a ( q ) can be formulated as the following equation (4). it can.

Claims (8)

A partial image generation unit that generates a partial image centered on a gazing point specified by the user from the omnidirectional image;
A UI unit that receives registration of the gazing point via a UI screen that displays the generated partial image;
A gaze point registration unit that registers the gaze point in response to a user request via the UI screen;
An interpolation line calculation unit for calculating an interpolation line for interpolating between the most recently registered gaze point and the currently designated gaze point;
An interpolation line registration unit for registering the interpolation line for interpolating between the registered gazing point and the gazing point registered immediately before,
The UI unit is
Displaying the interpolation line and the registered gazing point on the partial image of the UI screen,
Information processing device. The UI screen is
Including an icon display area for displaying gaze point icons corresponding to the registered gaze points in order of registration;
The UI unit is
In response to selection of the gaze point icon, accepts a corresponding change of the gaze point,
The interpolation line calculation unit
Recalculate an interpolation line for interpolating between the changed gazing point and another gazing point registered immediately before or after the gazing point;
The information processing apparatus according to claim 1. The gaze point icon is
It is a thumbnail of the partial image centered on the registered gazing point,
The information processing apparatus according to claim 2. The UI unit is
In response to selection of the gaze point icon, accepts deletion of the corresponding gaze point,
The interpolation line calculation unit
Calculating an interpolation line for interpolating between the gazing point registered immediately before the deleted gazing point and the gazing point registered immediately after the deleted gazing point;
The information processing apparatus according to claim 2 or 3. The information processing apparatus includes:
A viewpoint control unit that transitions a gazing point along a path for interpolating two or more registered gazing points;
An animation execution unit that executes an animation in which partial images centered on each transition point of interest are connected in the order of transition of the point of interest;
Including
The UI unit is
An interpolation line icon corresponding to the registered interpolation line is displayed in the icon display area,
In response to the selection of the interpolation line icon, accepting parameter settings for the animation executed along the path corresponding to the interpolation line icon;
The information processing apparatus according to any one of claims 2 to 4. The UI unit is
The gaze point and the gaze point icon are displayed in a manner in which the correspondence can be grasped,
Displaying the interpolation line and the interpolation line icon in a manner in which the correspondence can be grasped;
The information processing apparatus according to claim 5. The UI unit is
Displaying the plurality of gazing points or the plurality of interpolation lines in a distinguishable manner.
The information processing apparatus according to any one of claims 1 to 6. Computer
A partial image generating means for generating a partial image centered on a gazing point designated by the user from the omnidirectional image;
UI means for accepting registration of the gazing point via a UI screen that displays the generated partial image;
Gaze point registration means for registering the gaze point in response to a user request via the UI screen;
Interpolation line calculation means for calculating an interpolation line for interpolating between the most recently registered gazing point and the currently designated gazing point;
Interpolation line registration means for registering the interpolation line for interpolating between the registered gazing point and the gazing point registered immediately before the gazing point;
A program that functions as
The UI means includes
Displaying the interpolation line and the registered gazing point on the partial image of the UI screen,
program.