WO2014002539A1

WO2014002539A1 - Information processing apparatus and information processing method

Info

Publication number: WO2014002539A1
Application number: PCT/JP2013/057937
Authority: WO
Inventors: Hiroshi Fujimoto; Tatsuhiro Nishioka; Akinori Komaki; Masaru Ohba; Megumi AIKAWA; Tsugutoyo Osaki
Original assignee: Toshiba Corp
Current assignee: Toshiba Corp
Priority date: 2012-06-27
Filing date: 2013-03-13
Publication date: 2014-01-03
Anticipated expiration: 2014-12-27
Also published as: JP2014011474A

Description

D E S C R I P T I O N

INFORMATION PROCESSING APPARATUS AND INFORMATION PROCESSING METHOD

Cross-Reference to Related Applications This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2012-144206, filed June 27, 2012, the entire contents of which are incorporated herein by reference.

Field

Embodiments described herein relate generally to an information processing apparatus and information processing method.

Background .

Various techniques for displaying three- dimensional (3D) images have been proposed and put into practical applications so far. A television which can display a 3D image (to be referred to as a stereoscopic TV hereinafter) displays a 3D image based on input 3D content. Alternatively, the stereoscopic TV generates depth information based on input two-dimensional (2D) content, converts a 2D image into a 3D image based on this depth information, and outputs the converted 3D image.

When a 3D image is to be output based on the generated depth information, a temporal limitation is set on generation of depth information to assure that processing is carried out in real time to maintain synchronization with content reproduction processing. Also, in terms of the performance of a CPU and a memory size, a resource limitation is set on generation of depth information. Therefore, it is not easy to generate depth information with high precision. Since various limitations are set on generation of depth information, it is not easy to provide a high-quality 3D image satisfactory to the user.

Brief Description of the Drawings A general architecture that implements the various features of the embodiments will now be described with reference to the drawings. The drawings and the associated descriptions are provided to illustrate the embodiments and not to limit the scope of the

invention .

FIG. 1 is a schematic block diagram showing an example of the arrangement of an information processing apparatus according to an embodiment;

FIG. 2 is a flowchart showing an example of generation processing of various depth maps,, recording processing, and reproduction processing of a 3D image based on various depth maps according to the

embodiment ;

FIG. 3 is a block diagram showing an example of the generation processing of various depth maps, the recording processing, and the reproduction processing of a 3D image based on various depth maps according to the embodiment; and

FIG. 4 is a view showing a display example of a content recording list according to the embodiment.

Detailed Description

Various embodiments will be described hereinafter with reference to the accompanying drawings.

In general, according to one embodiment, an information processing apparatus includes a generator and a recorder. The generator is configured to generate first depth information of content before reproduction of the content. The recorder is

configured to record the content and the first depth information .

FIG. 1 is a schematic block diagram showing an example of the arrangement of an information processing apparatus according to an embodiment. An information processing apparatus 100 generates depth information (depth map) based on a resolution of a 2D image

included in content to be recorded or reproduced, various estimation algorithms required to estimate a depth of the 2D image, and the like, converts the 2D image into a 3D image based on the generated depth map, and outputs (displays) the 3D image. The content to be recorded or reproduced may include any of moving image content, still image content, and mixed content of moving and still images. As shown in FIG. 1, the information processing apparatus 100 includes a control unit 1, content acquisition unit 2, content recording unit 3, depth map generation unit 4, depth map recording unit 5, 3D image conversion unit 6, and 3D image display unit 7. Note that the content recording unit 3 and depth map

recording unit 5 need not be independent units.

For example, when the information processing apparatus 100 is a digital TV or digital recorder, the content acquisition unit 2 may include a receiving unit which receives broadcast content. Alternatively, the content acquisition unit 2 may include a communication unit which downloads network distributed content.

Alternatively, the content acquisition unit 2 may include a reading unit which reads package content from various media.

When the information processing apparatus 100 is a digital recorder, the 3D image display unit 7 is not an essential component, and the apparatus 100 includes a 3D image output unit in place of the 3D image display unit 7.

(First Embodiment)

The content acquisition unit 2 of the information processing apparatus 100 receives broadcast content or network distributed content, the depth map generation unit 4 generates first depth information (to be

referred to as a first high-quality depth map hereinafter) of the content in advance (before

reproduction of the content), the content recording unit 3 records the content, and the depth map recording unit 5 records the first high-quality depth map. For example, the depth map generation unit 4 starts

generation of the first depth map during recording of the content. That is, the first depth map is generated parallel to recording processing of the content. Note that the depth map generation unit 4 starts generation of the first depth map during the recording processing of the content. In this case, the depth map generation unit 4 may execute calculation processing of the first depth map so as to end generation of the first depth map simultaneously with the end of recording of the content. Alternatively, the depth map generation unit 4 may execute the calculation processing of the first depth map so that generation of the first depth map is continued for a predetermined period of time after the end of recording of the content.

For example, the first high-quality depth map allows generation of a 3D image having a higher quality than third depth information (to be referred to as a standard depth map hereinafter) , generation of which is started during reproduction of the content. Since generation of the first high-quality depth map is started before reproduction of the content and during recording of the content, the first high-quality depth map hardly suffers temporal and resource limitations in generation of the map compared to the standard depth map, which is generated parallel to reproduction of the content. For this reason, the first high-quality depth map can be generated by calculations with a heavier load and by spending a longer time than the standard depth map. As a result, the first high-quality depth map allows generation of a 3D image with a higher quality than the standard depth map.

As described above, the information processing apparatus 100 generates the first high-quality depth map based on the content while recording the content, and records the generated first high-quality depth map. The content recording unit 3 decodes a 2D image

included in the content, converts the 2D image into a recording format, and records the 2D image converted into the recording format. The depth map generation unit 4 can generate a first high-quality depth map corresponding to the 2D image using results of various kinds of image processing executed in the recording processing of the content. That is, compared to a case in which the recording processing and depth map

generation processing are independently executed, the efficiency of the depth map generation processing can be improved by parallelly executing the recording processing and depth map generation processing. Note that the content recording processing suffers a lighter processing load than the content reproduction

processing. Therefore, when the depth map (first high- quality depth map) is generated while recording the content, the processing load is lighter than a case in which the depth map (standard depth map) is generated while reproducing the content.

Furthermore, after the information processing apparatus 100 records the content and also the first high-quality depth map, it may generate a second high- quality depth map based on the recorded content and may record the generated second high-quality depth map. The information processing apparatus 100 can generate the second high-quality depth map using parameters of higher levels than those (the resolution of the image and estimation algorithms) used at the time of

generation of the first high-quality depth map. For example, the depth map generation unit 4 can generate the second high-quality depth map during a designated time zone (for example, midnight), and the depth map recording unit 5 can record the second high-quality depth map during the designated time zone.

The second high-quality depth map allows

generation of a 3D image having a higher quality than the first high-quality depth map, generation of which is started during recording of the content. Since generation of the second high-quality depth map is started before reproduction of the content and after recording of the content, the second high-quality depth map hardly suffers any resource limitation compared to the first high-quality depth map, generation of which is started during recording of the content. For this reason, the second high-quality depth map can be generated by calculations with a heavier load than the first high-quality depth map. As a result, the second high-quality depth map allows generation of a 3D image having a higher quality than the first depth map.

Note that the second high-quality depth map may be recorded by replacing the first high-quality depth map. The information processing apparatus 100 may replace the first high-quality depth map by the second high- quality depth map without deleting the first high- quality depth map, or may delete the first high-quality depth map and may replace the first high-quality depth map by the second high-quality depth map in terms of management .

An example of the depth map generation processing and recording processing, and reproduction processing of a 3D image based on the depth map will be described below with reference to FIGS. 2 and 3.

As shown in FIG. 2, the content recording unit 3 of the information processing apparatus 100 records content to be recorded. Note that the content

recording unit 3 records the content together with content recording identification information (for example, an ID 1001) . During recording of the content (BLOCKS 11 to 14), the depth map generation unit 4 starts generation of a first high-quality depth map by realtime processing from a 2D image included in the content to be recorded based on first depth estimation parameters (BLOCK 12), and the depth map recording unit 5 records the first high-quality depth map in

association with the ID 1001 (BLOCK 13) . That is, the 2D image and first high-quality depth map are

associated with each other via the ID 1001.

Furthermore, after the content and first high- quality depth map are recorded, the depth map

generation unit 4 starts generation of a second high- quality depth map from the 2D image included in the recorded content based on second depth estimation parameters of higher levels than the first depth estimation parameters (BLOCKS 15 and 16) , and the depth map recording unit 5 records the second high-quality depth map in association with the ID 1001. For

example, the depth map recording unit 5 records the second high-quality depth map in place of the first high-quality depth map (BLOCK 17).

In this way, the information processing apparatus 100 starts generation of the first high-quality depth map corresponding to a 2D image included in the content during recording of the content before the user views the content, and records the generated first high-quality depth map. Furthermore, the information processing apparatus 100 starts generation of the second high-quality depth map corresponding to the 2D image included in the content before the user views the content and after recording of the content (after recording of the first high-quality depth map) , and records the generated second high-quality depth map.

The information processing apparatus 100 receives a reproduction instruction of the content (that of a 3D image) from the user (YES in BLOCK 18), and the control unit 1 searches for a first high-quality depth map associated with the ID 1001 based on the ID 1001 of the content. For example, when the reproduction

instruction is input after recording of the first high- quality depth map and before recording of the second high-quality depth map, the control unit 1 can detect the first high-quality depth map associated with the ID 1001 (cannot find the second high-quality depth map) (YES in BLOCK 19), the depth map generation unit 4 reads the first high-quality depth map (BLOCK 20), the 3D image conversion unit 6 converts a 2D image included in the recorded content into a 3D image based on the first high-quality depth map (BLOCK 21), and the 3D image display unit 7 outputs (displays) the converted 3D image (BLOCK 22) (BLOCK 33 → BLOCK 34 → BLOCK 35 in FIG. 3) .

For example, when the reproduction instruction is input after recording of the second high-quality depth map, the control unit 1 can detect the second high- quality depth map associated with the ID 1001 based on the ID.1001 of the content (YES in BLOCK 19), the depth map generation unit 4 reads the second high-quality depth map (BLOCK 20), the 3D image conversion unit 6 converts a 2D image included in the content into a 3D image based on the second high-quality depth map (BLOCK

21) , and the 3D image display unit 7 outputs (displays) the converted 3D image (BLOCK 22) (BLOCK 36 → BLOCK

37 → BLOCK 38 in FIG. 3) .

Although the information processing apparatus 100 receives a reproduction instruction (that of a 3D image) of the content from the user (YES in BLOCK 18), if neither the first high-quality depth map nor the second high-quality depth map is recorded (NO in BLOCK 19), the depth map generation unit 4 generates a standard depth map (that simpler than the first high- quality depth map) from a 2D image included in the content by realtime processing parallel to the

reproduction processing of the content (BLOCK 23), the 3D image conversion unit 6 converts the 2D image included in the content into a 3D image based on the standard depth map (BLOCK 21), and the 3D image display unit 7 outputs (displays) the converted 3D image (BLOCK

22) (BLOCK 31 → BLOCK 32 → BLOCK 33 in FIG. 3) .

When the information processing apparatus 100 receives a reproduction instruction (that of a 2D image) -of the content from the user (NO in BLOCK 18), it outputs (displays) a 2D image included in the content intact.

When the information processing apparatus 100 records the first and second high-quality depth maps, and receives, for example, a first high-quality reproduction instruction from the user, it may convert a 2D image included in the content into a 3D image based on the first high-quality depth map, and may output the converted 3D image. Alternatively, when the information processing apparatus 100 receives a second high-quality reproduction instruction from the user, it may convert a 2D image included in the content into a 3D image based on the second high-quality depth map, and may output the converted 3D image.

Furthermore, when the information processing apparatus 100 records the standard depth map in

addition to the first and second high-quality depth maps, and receives, for example, a standard

reproduction instruction from the user, it may convert a 2D image included in the content into a 3D image based on the already recorded standard depth map, and may output the converted 3D image.

Also, as shown in FIG. 4, content recording information (to be referred to as a content recording list) may include first service information SI (additional information) and second service information S2 (additional information) , allowing the user to recognize whether or not content is ready to undergo high-quality reproduction or to know the remaining time until content is ready to undergo high-quality

reproduction. The control unit 1 displays the content recording list in response to a display instruction of the content recording list from the user, and the 3D image display unit 7 displays the content recording list in response to this control.

The first service information SI indicates that a second high-quality depth, map corresponding to content A is recorded. That is, this information indicates that content A can be reproduced as a 3D image

processed using the second high-quality depth map.

Thus, the user can recognize that he or she can view content appended with the first service information SI as a high-quality 3D image. For example, when the user inputs a reproduction instruction of content A from the content recording list, content A is reproduced as a 3D image processed using the second high-quality depth map .

The second service information S2 indicates a remaining time required until a second high-quality depth map corresponding to content B is recorded (or a date and time of completion of recording of the second high-quality depth map) . When the depth map generation unit 4 generates a second high-quality depth map corresponding to content B, it can detect a required generation time of the second high-quality depth map from a 2D image included in content B and parameters required for generation of the second high-quality depth map. The control unit 1 calculates, from the detected required generation time, a remaining time until the second high-quality depth map is recorded (or the date and time of completion of recording of the second high-quality depth map) , and performs control to display the content recording list including the second service information S2. In response to this control, the 3D image display unit 7 displays the content recording list. For example, when the user inputs a reproduction reservation instruction of content B from the content recording list, and when content B is ready to be reproduced based on the second high-quality depth map, content B is reproduced as a 3D image processed using the second high-quality depth map.

Note that a plurality of information processing apparatuses 100 may be inhibited from sharing the first or second high-quality depth map. For the sake of simplicity, for example, information processing

apparatuses 100a and 100b will be assumed.

As described above, upon recording content, information processing apparatus 100a records the content together with content recording identification information (for example, an ID lOOal) . Also,

information processing apparatus 100a associates the content and a first high-quality depth map with each other via this ID lOOal. Furthermore, information processing apparatus 100a associates the content and a second high-quality depth map with each other via this ID lOOal.

Likewise, upon recording the same content as above, information processing apparatus 100b records the content together with content recording

identification information (for example, an ID lOObl) . Also, information processing apparatus 100b associates the content and a first high-quality depth map with each other via this ID lOObl. Furthermore, information processing apparatus 100b associates the content and a second high-quality depth map with each other via this ID lOObl.

That is, even when information processing

apparatuses 100a and 100b record the same content, information processing apparatus 100a associates the content and the first and second high-quality depth maps with each other via the ID lOOal, but information processing apparatus 100b associates the content and the first and second high-quality depth maps with each other via the ID lOObl.

In this manner, even when information processing apparatus 100b receives the first or second high-quality depth map from information processing apparatus 100a while it records the content but does not record the first and second high-quality depth maps, since the content recorded in information processing apparatus 100b is managed using the ID lOObl, information processing apparatus 100b cannot use the first or second high-quality depth map managed using the ID lOOal.

(Second Embodiment)

The information processing apparatus 100 can receive broadcast content or network distributed content, can further receive a standard depth map, first high-quality depth map, or second high-quality depth map corresponding to the received content, can convert a 2D image included in the received content into a 3D image based on the received standard depth map, first high-quality depth map, or second high- quality depth map, and can display that 3D image.

Also, the information processing apparatus 100 can record the received standard depth map, first high- quality depth map, or second high-quality depth map.

Furthermore, when a fee is required to be paid for use of a depth map, the information processing

apparatus 100 executes payment processing for use of the depth map, and also receives a standard depth map, first high-quality depth map, or second high-quality depth map. Also, by paying a free higher than that required for temporary use of a depth map, when recording of a depth map is permitted, the information processing apparatus 100 executes payment processing for depth map recording, and receives a standard depth map, first high-quality depth map, or second high- quality depth map.

The information processing apparatus 100 can receive a standard depth map, first high-quality depth map, or second high-quality depth map in advance before broadcast or distribution of content. Alternatively, the information processing apparatus 100 can receive a standard depth map, first high-quality depth map, or second high-quality depth map during broadcast or distribution of content. Alternatively, the

information processing apparatus 100 can receive a standard depth map, first high-quality depth map, or second high-quality depth map after recording of content. In any case, since no depth map generation operation is required, no wait time required for generation of a depth map is generated.

The received depth map includes management

information such as channel information and a broadcast time zone, and the received content similarly includes management information such as channel information and a broadcast time zone. The information processing apparatus 100 associates the received content and received depth map with each other based on the management information of the received content and that of the received depth map, converts a 2D image included in the received content into a 3D image based on the received depth map, and displays the 3D image.

Note that when the received depth map includes a depth parameter 0 in association with a display period of a 2D image corresponding to a commercial, the information processing apparatus 100 practically displays the commercial in the content as a 2D image based on this received depth map.

(Third Embodiment)

The information processing apparatus 100 can read package content from each of various media, can

generate a standard depth map, first high-quality depth map, or second high-quality depth map corresponding to the read content, can convert a 2D image included in the package content into a 3D image based on the generated depth map, and can display the converted 3D image, as described in the first embodiment.

Alternatively, the information processing

apparatus 100 can read package content from each of various media, can receive a standard depth map, first high-quality depth map, or second high-quality depth map corresponding to the read content, can convert a 2D image included in the package content into a 3D image based on the received depth map, and can display the converted 3D image, as described in the second embodiment .

(Fourth Embodiment)

The information processing apparatus 100 can receive content from an image . site (for example, a movie site) , can generate a standard depth map, first high-quality depth map, or second high-quality depth map corresponding to the read content, can convert a 2D image included in the package content into a 3D image based on the generated depth map, and can display, the converted 3D image, as described in the first

embodiment .

Alternatively, the information processing

apparatus 100 can receive content from an image site (for example, a movie site), can receive a standard depth map, first high-quality depth map, or second high-quality depth map corresponding to the read content, can convert a 2D image included in the package content into a 3D image based on the received depth map, and can display the converted 3D image, as

described in the second embodiment.

(Fifth Embodiment)

The information processing apparatus 100 generates a standard depth map, first high-quality depth map, or second high-quality depth map corresponding to a 2D image of content recorded in advance in an external medium, and manages an ID of the external medium and that of the generated depth map in association with each other. Alternatively, the information processing apparatus 100 manages an ID of the content recorded in the external medium and that of the generated depth map in association with each other. Upon reception of a reproduction instruction of the external medium from the user, the information processing apparatus 100 reads the generated depth map based on the ID of the external medium and that of the generated depth map, converts a 2D image of the content recorded in the external medium into a 3D image based on the read depth map, and displays the converted 3D image.

Alternatively, upon reception of a reproduct on

instruction of the external medium from the user, the information processing apparatus 100 reads the

generated depth map based on the ID of the content recorded in the external medium and that of the

generated depth map, converts a 2D image of the content recorded in the external medium into a 3D image based on the read depth map, and displays the converted 3D image.

(Sixth Embodiment)

In response to a depth map generation instruction from the user, the information processing apparatus 100 generates a standard depth map, first high-quality depth map, or second high-quality depth map

corresponding to content designated by the user, and records the generated depth map. Thus, the depth map corresponding to the content designated by the user can be registered.

According to at least one embodiment described above, the information processing apparatus 100 can avoid temporal and resource limitations, and can generate a depth map using a plurality of estimation algorithms at a resolution equivalent to a source image. Thus, the user can view a more appropriate 3D image (with higher definition and closer to an actual image) .

According to at least one embodiment described above, an information processing apparatus and

information processing method, which can avoid or eliminate the influence of limitations associated with generation of depth information, and can generate depth information can be provided.

The various modules of the embodiments described herein can be implemented as software applications, hardware and/or software modules, or components on one or more computers, such as servers. While the various modules are illustrated separately, they may share some or all of the same underlying logic or code.

While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes . in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their

equivalents are intended to cover such forms or

modifications as would fall within the scope and spirit of the inventions.

Claims

C L A I M S

1. An information processing apparatus

comprising :

a generator configured to generate first depth information of content before reproduction of the content; and

a recorder configured to record the content and the first depth information.

2. The apparatus of Claim 1, wherein the

generator is configured to start generation of the first depth information during recording of the

content .

3. The apparatus of Claim 2, wherein the

generator is configured to start generation of second depth information of a higher level than the first depth information after recording of the content, and the recorder is configured to record the second depth information.

4. The apparatus of Claim 3, wherein the recorder is configured to record the second depth information in place of the first depth information.

5. The apparatus of Claim 3, wherein the recorder is configured to associate the content and the first depth information with each other based on recording identification information which is given in

correspondence with recording of the content, and to associate the content and the second depth information with each other based on the recording identification information .

6. The apparatus of Claim 4, wherein the recorder is configured to associate the content and the first depth information with each other based on recording identification information which is given in

correspondence with recording of the content, and to associate the content and the second depth information with each other based on the recording identification information.

7. The apparatus of Claim 1, further comprising: a reproducer configured to reproduce the content based on the first depth information in response to a reproduction instruction of the content after recording of the first depth information.

8. The apparatus of Claim 2, further comprising: a reproducer configured to reproduce the content based on the first depth information in response to a reproduction instruction of the content after recording of the first depth information.

9. The apparatus of Claim 7, wherein the

generator is configured to start generation of third depth information simpler than the first depth

information during reproduction of the content in response to a reproduction information of the content before recording of the first depth information, and the reproducer is configured to reproduce the content based on the third depth information in

response to the reproduction instruction of the content before recording of the first depth information.

10. The apparatus of Claim 8, wherein the generator is configured to start generation of third depth information simpler than the first depth

11. The apparatus of Claim 9, wherein the

reproducer is configured to convert a two-dimensional image of the content into a three-dimensional image based on the first depth information or the third depth information, and reproduces the three-dimensional image .

12. The apparatus of Claim 10, wherein the reproducer is configured to convert a two-dimensional image of the content into a three-dimensional image based on the first depth information or the third depth information, and reproduces the three-dimensional image .

13. The apparatus of Claim 3, further comprising: a reproducer configured to reproduce the content based on the second depth information in response to a reproduction instruction of the content after recording of the second depth information.

14. The apparatus of Claim 4, further comprising: a reproducer configured to reproduce the content based on the second depth information in response to a reproduction instruction of the content after recording of the second depth information.

15. The apparatus of Claim 13, wherein the generator is configured to start generation of third depth information simpler than the first depth

16. The apparatus of Claim 14, wherein the generator is configured to start generation of third depth information simpler than the first depth

17. The apparatus of Claim 15, wherein the reproducer is configured to convert a two-dimensional image of the content into a three-dimensional image based on the second depth information or the third depth information, and reproduces the three-dimensional image .

18. The apparatus of Claim 3, further comprising: an output module configured to output recording information associated with recording of the content after completion of recording of the content and the second depth information,

wherein the output module is configured to output the recording information including additional

information indicating that the content is ready to undergo high-quality reproduction based on the second depth information.

19. The apparatus of Claim 3, further comprising: a detector configured to detect a required

generation time of the second depth information based on the content; and

an output module configured to output, based on the required generation time, time information, required until the content is ready to undergo high-quality reproduction based on the second depth information.

20. An information processing method comprising: generating first depth information of content before reproduction of the content; and recording the content and the first depth information .