[go: up one dir, main page]

WO2021230001A1 - Appareil de traitement d'informations et procédé de traitement d'informations - Google Patents

Appareil de traitement d'informations et procédé de traitement d'informations Download PDF

Info

Publication number
WO2021230001A1
WO2021230001A1 PCT/JP2021/015660 JP2021015660W WO2021230001A1 WO 2021230001 A1 WO2021230001 A1 WO 2021230001A1 JP 2021015660 W JP2021015660 W JP 2021015660W WO 2021230001 A1 WO2021230001 A1 WO 2021230001A1
Authority
WO
WIPO (PCT)
Prior art keywords
resolution
viewer
information
user
information processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/015660
Other languages
English (en)
Japanese (ja)
Inventor
卓己 津留
俊也 浜田
尚尊 小代
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Publication of WO2021230001A1 publication Critical patent/WO2021230001A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities

Definitions

  • This disclosure relates to an information processing device and an information processing method.
  • HD High Definition
  • 4K full HD
  • 8K 8K
  • Non-Patent Document 1 a method of constructing a three-dimensional space with a plurality of three-dimensional objects and distributing data related to these a plurality of objects.
  • the data distributed by this method includes data in which a scene in a three-dimensional space, for example, a scene description, is represented by a graph having a tree hierarchical structure called a scene graph, and the scene graph is represented in a binary format or a text format ().
  • a scene in a three-dimensional space for example, a scene description
  • a scene graph having a tree hierarchical structure called a scene graph
  • the scene graph is represented in a binary format or a text format ().
  • Non-Patent Document 1 Non-Patent Document 1
  • the amount of data to be delivered increases, the load on the communication network increases (that is, the bandwidth of the communication network is insufficient), and the viewer who enjoys the delivered content (that is, the viewer (that is, the bandwidth of the communication network is insufficient)).
  • the amount of money to be paid as communication costs will also increase.
  • an information processing device provided with an acquisition unit for acquiring video content data to be played back for the user in real time according to the visual acuity information of the user is provided.
  • an information processing method including acquiring video content data to be played back for the user in real time according to the visual acuity information of the user is provided.
  • a distribution technology using DASH Dynamic Adaptive Streaming over HTTP, ISO / IEC 23009-1
  • DASH Dynamic Adaptive Streaming over HTTP, ISO / IEC 23009-1
  • the distribution technology a plurality of stream variations of the same content but having different bit rates are prepared in advance in the distribution server, and the streams are switched according to the fluctuation of the communication network bandwidth.
  • the distribution technology for example, when the bandwidth of the communication network is insufficient, the distribution is switched to the distribution of a stream variation having a low bit rate. By doing so, it is possible to reduce the load on the communication network due to the increase in the amount of data to be delivered, but the image quality is deteriorated.
  • object data relating to each 3D object is displayed in order to display all objects (3D objects) existing in the 3D space even if the position of the viewer moves.
  • the object data is a combination of mesh data, which is the shape of a polyhedron of a three-dimensional object, and texture data, which is data to be attached to the surface, or Point Cloud (point cloud), which is a set of multiple points. Consists of data.
  • a scene description for displaying a three-dimensional object according to the viewpoint position of the viewer is also distributed.
  • the scene description not only defines the position and posture of the three-dimensional object in the three-dimensional space, but also provides a plurality of different LODs (Level Of Detail) according to the viewpoint position of the viewer for one three-dimensional object.
  • LOD defines, for example, the number of points (details) representing the same three-dimensional object in the case of a three-dimensional object represented by Point Cloud.
  • a three-dimensional object far from the viewer's viewpoint position makes the display small, and a three-dimensional object close to the viewer's viewpoint position makes the display large.
  • a three-dimensional object having a large display has a high LOD of its display
  • a three-dimensional object having a small display has a low LOD of its display.
  • the higher the LOD the larger the amount of data for the display.
  • a scene is represented by a graph with a tree hierarchical structure called a scene graph, and the scene graph is represented in binary format or text format.
  • the scene graph is spatial display control information based on the viewpoint position of the viewer, and the information regarding the display of the three-dimensional object at the viewpoint position is defined by defining the node as a constituent unit, and a plurality of nodes are hierarchically combined. It consists of.
  • the data of the scene description can conform to MPEG-4 Scene Description (ISO / IEC 14496-11).
  • the MPEG-4 Scene Description data is obtained by binarizing the scene graph in the format of BIFS (Binary Form at for Scenes). It is possible to convert this scene graph to BIFS with a fixed algorithm.
  • BIFS Binary Form at for Scenes
  • ISO base media file format it is possible to specify a scene for each time, and it is possible to express a moving object or the like.
  • the present inventors are diligently studying whether it is possible to avoid an increase in the amount of distribution data and an increase in the load on the communication network due to the high resolution in the distribution of video content. rice field. Then, while proceeding with such a study, the present inventors, in the above-mentioned distribution technology, the data of the video content to be distributed is the amount of data according to the resolution of the display device, the band that can be used in the communication network, and the like. I noticed that it has the resolution of, but the resolution does not correspond to the resolution of the eyes of the viewer who watches the video content.
  • the present inventors have noticed that there are cases where the data of the moving image content having an unnecessarily high resolution with respect to the resolution of the viewer's eyes is delivered. Then, based on the above viewpoint, the present inventors can avoid an increase in the amount of distributed data and an increase in the load on the communication network while avoiding the viewer from feeling the deterioration of the image quality. It came to create the embodiment independently.
  • FIGS. 1 and 2 are explanatory views for explaining an outline of the embodiment of the present disclosure
  • FIG. 1 is a diagram showing a positional relationship between a viewer and a display device as viewed from above
  • FIG. 2 is a diagram. Is a diagram showing the positional relationship between the viewer and the display device as viewed from the side.
  • the display unit information regarding the size of the display device, and the position information indicating the position of the viewer with respect to the display device Calculate the necessary and sufficient resolution (user resolution) for the viewer. Then, according to the embodiment of the present disclosure, by distributing the data of the video content according to the calculated resolution, the amount of the distributed data can be increased and the communication network can be increased while avoiding the viewer from feeling the deterioration of the image quality. It is possible to avoid an increase in load.
  • a method of calculating a resolution necessary and sufficient for the resolution of the viewer's eyes will be described based on the viewer's visual acuity and the distance between the viewer and the display device.
  • a unit called PPD Panels Per Degree
  • the human eye with a visual acuity of 1.0 is used.
  • the resolution is said to be 60 PPD. Since there are various theories about the specific value of the resolution of the human eye with a visual acuity of 1.0, in the following description, the resolution of the human eye with a visual acuity of 1.0 is expressed by a constant P. ..
  • the necessary and sufficient resolution for the human eye with a visual acuity of 1.0 can be satisfied by the presence of P (pixel) in the range of length X on the display device corresponding to one visual field range. Therefore, the width direction resolution (Width) and the height direction resolution (Height) of the display device for the viewers (viewers with "visual acuity") in the situations shown in FIGS. 1 and 2 are determined. , Can be expressed by the following mathematical formula (1).
  • the resolution in the width direction (Width) is 3819.62
  • the resolution in the height direction (Height) is 2148.54.
  • the resolution in the width direction (Width) is 1909.81
  • the resolution in the height direction (Height) is 1074.27.
  • the playback device has a resolution higher than the resolution calculated by using the mathematical formula (3) from a plurality of content data of the same content but different resolutions from the distribution server, and is the most among the content data in the distribution server.
  • Acquire content data with low resolution For example, the distribution server has SD (Standard definition) (720 x 480), HD (1280 x 720), full HD (1920 x 1080), 4K (3840 x 2160), and 8K (7680 x 4320) resolutions. It is assumed that the content data to be stored is stored.
  • 4K (3840 ⁇ 2160) content data may be selected for viewer A, and full HD (1920 ⁇ 1080) content data may be selected for viewer B. Will be.
  • FIG. 3 is an explanatory diagram for explaining an outline of the video content distribution system according to the first embodiment of the present disclosure.
  • the distribution system 1 distributes moving image content composed of two-dimensional video.
  • the video content distribution system 1 includes a content distribution server 10 that distributes content data of video content in response to a request from the playback device 30.
  • the distribution system 1 further includes a reproduction device 30 for reproducing the distributed moving image content, and a display device 20 for displaying the moving image content according to the control from the reproduction device 30.
  • the content distribution server 10 and the playback device 30 are connected to each other by a communication network 40.
  • the reproduction device 30 and the display device 20 may be separate devices as shown in FIG. 3, or may be an integrated device, and are not particularly limited.
  • the communication method used in the communication network 40 any method can be applied regardless of whether it is wired or wireless, but it is desirable to use a communication method capable of maintaining stable operation. The outline of each device included in the distribution system 1 will be described below.
  • the content distribution server 10 distributes the content data of the moving image content having a predetermined resolution to the reproduction device 30 in response to the request from the reproduction device 30. Further, the content distribution server 10 stores a plurality of content data of the same video content having different resolutions (for example, SD (720 ⁇ 480), HD (1280 ⁇ 720), full HD (1920 ⁇ 1080), 4K). (3840 x 2160), 8K (7680 x 4320)).
  • the display device 20 is a two-dimensional display such as a television, a tablet, and a smartphone.
  • the display device 20 incorporates a distance measuring device (for example, a ToF (Time of Flight) sensor) and a sensor (distance measuring unit) such as a camera, and is based on the sensing data of these sensors. It shall be possible to detect the relative position (position and distance) with the viewer.
  • the distance between the viewer and the display device 20 is assumed to be the distance from the viewer to the closest point on the display surface of the display device 20 as seen from the viewer.
  • the display device 20 has a built-in communication unit that wirelessly communicates with a communication device carried by the viewer, and is a relative position (position and distance) with the viewer based on the sensing data by the communication unit. May be detected.
  • the display device 20 is relative to the viewer based on the position information from the distance measuring device carried by the viewer (for example, a GNSS (Global Navigation Satellite System) signal receiver). The position (position and distance) may be detected.
  • GNSS Global Navigation Satellite System
  • the reproduction device 30 acquires the content data of the moving image content in real time from the content distribution server 10 according to the visual acuity information of the viewer, reproduces the acquired content data, and outputs the acquired content data to the display device 20.
  • the details of the reproduction device 30 will be described later.
  • the configuration of the distribution system 1 according to the present embodiment is not limited to such an example. That is, the configuration of the distribution system 1 according to the present embodiment can be flexibly modified according to the specifications and operation.
  • FIG. 4 is a block diagram showing a functional configuration example of the reproduction device 30 according to the present embodiment.
  • the reproduction device 30 mainly includes a display control unit 300, a main control unit 330, a storage unit 340, and a transmission / reception unit 350.
  • the main control unit 330 mainly includes a display control unit 300, a main control unit 330, a storage unit 340, and a transmission / reception unit 350.
  • the display control unit 300 is composed of, for example, a CPU (Central Processing Unit) (not shown), an MPU (Micro Processing Unit) (not shown), or the like, and uses programs and data stored in the storage unit 340 described later. , The process according to this embodiment is executed. Specifically, the display control unit 300 includes a position information acquisition unit 302, a calculation unit 304, and a comparison unit 306 for acquiring content data of video content from the content distribution server 10 according to the visual acuity information of the viewer. , Acquiring unit 308.
  • the display control unit 300 includes a processing unit 322, a decoding unit 324, and a display information generation unit 326 (in the present specification, these functional units are also referred to as a decoding block 320) that decode the acquired content data. Have.
  • the position information acquisition unit 302 acquires distance information (position information) for the viewer's display device 20 from the display device 20, for example, and outputs the distance information (position information) to the calculation unit 304, which will be described later.
  • the calculation unit 304 is output from the viewer's visual information (visual information), the size (width, height) information (display unit information) of the display device 20, and the position information acquisition unit 302, which are acquired in advance. Based on the distance information (distance to the display device 20 of the viewer), the resolution (user resolution) in the height direction and the width direction of the display device 20 for the viewer is calculated. Further, the calculation unit 304 outputs the calculated resolution to the comparison unit 306 described later.
  • the comparison unit 306 compares the resolution calculated by the calculation unit 304 with the resolution of the display device 20 acquired in advance, and outputs the comparison result to the acquisition unit 308 described later.
  • the acquisition unit 308 acquires content data having a resolution necessary and sufficient for the resolution of the viewer's eyes in real time based on the viewer's visual acuity (visual acuity information). Specifically, the acquisition unit 308 acquires content data having a resolution based on the comparison result with respect to the calculated resolution. When there are a plurality of viewers, in the present embodiment, the acquisition unit 308 acquires content data having a resolution based on the comparison result of the resolution comparison for each viewer by the comparison unit 306. Will be done. Then, the acquisition unit 308 outputs the acquired content data to the processing unit 322 of the decoding block 320.
  • the processing unit 322 is a functional unit that performs processing related to playback of video content, performs processing for switching acquisition content data (DASH), analyzes acquired content data, and decodes processing in the decoding unit 324, which will be described later. Can be converted to a file format that can be used. Further, the processing unit 322 outputs the processed content data to the decoding unit 324.
  • DASH acquisition content data
  • the processing unit 322 outputs the processed content data to the decoding unit 324.
  • the decoding unit 324 performs decoding processing (decoding) on the content data output from the processing unit 322, and outputs the decoded content data to the display information generation unit 326 described later.
  • the display information generation unit 326 processes the decoded content data output from the decoding unit 324, generates a display screen for display on the display device 20, and outputs the display screen to the display device 20.
  • the display information generation unit 326 may cut out (render) the image according to the viewpoint of the viewer.
  • the main control unit 330 has a functional configuration that comprehensively controls all the processing performed by the playback device 30, and is composed of, for example, a CPU (not shown), an MPU (not shown), and the like, and is stored in a storage unit 340 described later. Execute the process using the program or data.
  • the storage unit 340 is a functional unit that stores various types of information.
  • the storage unit 340 stores programs, content data, parameters, and the like used by each functional unit of the reproduction device 30.
  • the storage unit 360 is realized by, for example, a magnetic recording medium such as a hard disk (Hard Disk: HD), a non-volatile memory, or the like.
  • the transmission / reception unit 350 performs various communications with the content distribution server 10.
  • the transmission / reception unit 350 is a communication interface having a function of transmitting / receiving data, and is realized by, for example, a communication device (not shown) such as a communication antenna, a transmission / reception circuit, and a port.
  • the reproduction device 30 has an input unit (not shown), and acquires the visual acuity information of the viewer by receiving an input operation of the visual acuity information from the viewer to the input unit. Can be done.
  • the above configuration described with reference to FIG. 4 is merely an example, and the configuration of the reproduction device 30 according to the present embodiment is not limited to such an example. That is, the configuration of the reproduction device 30 according to the present embodiment can be flexibly modified according to the specifications and operation.
  • FIG. 5 is a diagram showing a flowchart of the information processing method according to the present embodiment.
  • the information processing method according to the present embodiment includes steps S101 to S110.
  • steps S101 to S110 are steps S101 to S110.
  • each step of the information processing method according to the present embodiment will be described.
  • the reproduction device 30 acquires information (display unit information) of the size (width, height) of the display device 20 (step S101).
  • the reproduction device 30 acquires the visual acuity information (visual acuity information) of the viewer by the input from the viewer (step S102).
  • the reproduction device 30 acquires information (distance information) of the distance to the display device 20 of the viewer from, for example, the display device 20 (step S103).
  • the distance information is acquired every time the position of the viewer moves, and the subsequent processing is performed.
  • the reproduction device 30 obtains the viewer's sight information, the size (width, height) information of the display device 20, and the distance information of the viewer with respect to the display device 20 acquired in steps S101 to S103. Based on (distance information), the resolution (user resolution) of the viewer in the height direction and the width direction of the display device 20 is calculated (step S105). Since the calculation method has already been described, the description thereof will be omitted here.
  • step S105 the playback device 30 determines whether or not the resolutions of all the viewers have been calculated.
  • the playback device 30 proceeds to step S106 when the resolution has been calculated for all the viewers (step S105: Yes), while the resolution has not been calculated for all the viewers. In that case (step S105: No), the process returns to step S102 described above. That is, in the present embodiment, the processes from step S102 to step S105 are repeated until the resolution is calculated for all the viewers.
  • the playback device 30 compares the viewer resolutions calculated so far for each of the height direction and the width direction of the display device 20, and specifies the highest resolution. In the present embodiment, when there are a plurality of viewers, it is possible to avoid causing all the viewers to feel the deterioration of the image quality by acquiring the content data based on the highest resolution. Further, the reproduction device 30 compares the specified resolution with the resolution of the display device 20 acquired in advance, and selects a lower resolution (step S106).
  • the playback device 30 acquires the content data corresponding to the resolution selected in step S106 described above (step S107). Next, the playback device 30 performs decoding processing on the content data acquired in step S107 described above (step S108). Further, the reproduction device 30 displays the moving image content by outputting the content data decoded in the above-mentioned step S108 to the display device 20 (step S109).
  • the playback device 30 determines whether or not processing has been performed up to the end of the stream (plurality of frames) included in the content data (step S109).
  • the reproduction device 30 ends the information processing according to the present embodiment.
  • the reproduction device 30 returns to the above-mentioned step S102. That is, in the present embodiment, the processing from step S102 to step S110 is repeatedly performed until the processing is completed up to the end of the stream.
  • the present embodiment it is possible to avoid acquiring the content data having a resolution unnecessary for the resolution of the viewer's eyes, and to acquire the content data necessary and sufficient for the resolution of the viewer's eyes. be able to. Therefore, according to the present embodiment, it is possible to avoid an increase in the amount of distributed data and an increase in the load on the communication network while avoiding the viewer from feeling the deterioration of the image quality.
  • FIG. 6 is an explanatory diagram for explaining the outline of the present embodiment.
  • the resolution for the viewer is calculated based on the visual acuity (visual acuity information) of the viewer and the distance (distance information) to the display device 20 of the viewer, which will be described below.
  • the angle information indicating the angle of the viewer with respect to the display device 20 is used to more accurately calculate the resolution for the viewer. can do.
  • the calculation is performed in consideration of the above angle information, it is possible to calculate the resolution suitable for the viewer with high accuracy, so that the viewer feels the image quality deterioration more effectively. While avoiding this, it is possible to avoid an increase in the amount of distributed data and an increase in the load on the communication network.
  • FIG. 6 is an explanatory diagram for explaining the outline of the present embodiment, and the viewer displays the end of the display device 20 (the end portion of the display device 20 closest to the viewer) to the display device 20. It is a figure when the appearance of viewing from an oblique position with respect to the display surface of the viewer is seen from above the viewer.
  • the resolution of the human eye with a visual acuity of 1.0 is expressed by a constant P, and the necessary and sufficient resolution for the human eye with a visual acuity of 1.0 is P (1 degree in the visual field range). resolution) It shall be possible to satisfy by being present. Therefore, assuming a viewer (viewer with "visual acuity") in the situation shown in FIG. 6, P / 2 is set to a length X (m) corresponding to a viewing range of 0.5 degrees. (Pixel) It suffices if it exists. Therefore, the width direction resolution (Width) of the display device 20 which is necessary and sufficient for the viewer in FIG. 6 can be expressed by the following mathematical formula (4).
  • X is the distance L of the viewer to the end of the display device 20 (the end portion of the display device 20 closest to the viewer) and the angle ⁇ with respect to the end of the display device 20 of the viewer.
  • L is the distance L of the viewer to the end of the display device 20 (the end portion of the display device 20 closest to the viewer) and the angle ⁇ with respect to the end of the display device 20 of the viewer.
  • the height resolution (Height) of the display device 20 in the height direction which is necessary and sufficient for the viewer in FIG. 6, can be expressed by the following mathematical formula (7).
  • the angle of the viewer with respect to the end of the display device 20 (the portion of the end of the display device 20 closest to the viewer) in the height direction is defined as ⁇ (degree).
  • P 60PPD
  • the resolution in the width direction (Width) is 1880.94, and the resolution in the height direction (Height) is 1058.03.
  • the content distribution server 10 has SD (720 ⁇ 480), HD (1280 ⁇ 720), full HD (1920 ⁇ 1080), 4K (3840 ⁇ 2160), and 8K (7680 ⁇ 4320) resolutions. It is assumed that the content data with is stored. In such a case, in the present embodiment, the full HD content data having the lowest resolution satisfying the resolution of the viewer A calculated as described above is selected and distributed.
  • the resolution of the display device 20 is 8K (7680 ⁇ 4320), it is necessary and sufficient for the viewer A to have a resolution of full HD (1920 ⁇ 1080). , It is possible to avoid acquiring content data having a resolution unnecessary for the resolution of the viewer's eyes. Therefore, according to the present embodiment, it is possible to avoid an increase in the amount of distributed data and an increase in the load on the communication network while avoiding the viewer from feeling the deterioration of the image quality. Further, in the present embodiment, since the resolution (full HD (1920 ⁇ 1080) in the above example) is calculated in consideration of the viewing angle information of the viewer, the resolution is calculated without considering the viewing angle.
  • the display device 20 incorporates a sensor (angle measuring unit) such as a distance measuring device or a camera, and displays the viewer based on these sensing data. It is assumed that the relative position and viewing angle with respect to the device 20 can also be detected.
  • the viewing angle of the viewer is assumed to be the angle of the viewer with respect to the closest end of the display surface of the display device 20 as seen from the viewer.
  • the position information acquisition unit 302 acquires not only the distance information of the viewer with respect to the display device 20 but also the relative position information and the viewing angle information from the display device 20, and outputs the information to the calculation unit 304 described later.
  • the calculation unit 304 is output from the viewer's visual information (visual information) acquired in advance, the size (width, height) information (display unit information) of the display device 20, and the position information acquisition unit 302. In addition to the distance information, the resolution (user resolution) in the height direction and the width direction of the display device 20 for the viewer is calculated based on the viewing angle with respect to the display device 20 of the viewer.
  • FIG. 7 is a diagram showing a flowchart of the information processing method according to the embodiment.
  • the information processing method according to the present embodiment includes steps S201 to S213. The details of each of these steps according to the present embodiment will be described below. In the following description, only the points different from the above-mentioned first embodiment will be described, and the points common to the first embodiment will be omitted.
  • steps S201 and S202 are the same as steps S101 and S102 of the first embodiment shown in FIG. 5, the description thereof will be omitted here.
  • the playback device 30 acquires information on the relative position of the viewer with respect to the display device 20 as well as information on the distance to the viewer's display device 20 from the display device 20 (step S203). Then, the reproduction device 30 determines whether or not the viewer exists so as to face the frame of the display surface of the display device 20 based on the relative position acquired in the above-mentioned step S203 (step S204). When the viewer is present so as to face the frame of the display surface of the display device 20 (step S204: Yes), the playback device 30 proceeds to step S205, while the viewer is on the display surface of the display device 20. If they do not exist so as to face each other in the frame (step S204: No), the process proceeds to step S206.
  • the playback device 30 sets the viewing angle to 0 degrees (step S205). Further, the playback device 30 acquires information on the viewing angle with respect to the viewer's display device 20 from the display device 20 (step S206).
  • the reproduction device 30 has acquired the viewer's sight information, the size (width, height) information of the display device 20, and the distance information (distance information) with respect to the viewer's display device 20 acquired from the above-mentioned steps S201 to S203. ), And based on the viewing angle set or acquired in steps S205 and S206, the resolution (user resolution) of the viewer in the height direction and the width direction of the display device 20 is calculated (step S207). Since the calculation method has already been described, the description thereof will be omitted here.
  • steps S209 to S213 are the same as steps S106 to S110 of the first embodiment shown in FIG. 5, the description thereof will be omitted here.
  • the resolution is calculated in consideration of the information of the viewing angle of the viewer, the viewing state of the viewer is more as compared with the case where the resolution is calculated without considering the viewing angle. It is possible to obtain a resolution that matches the above.
  • the moving image content delivered in the present embodiment is assumed to be 360-degree virtual space video data (three-dimensional space data) including all-sky video data and 6DoF content (three-dimensional space data).
  • the content data of these contents includes the scene description (three-dimensional space description data) that defines the configuration of the three-dimensional space and the object data that defines the three-dimensional object in the three-dimensional space. It shall be muted.
  • a three-dimensional object that is far from the viewer's viewpoint position makes its display smaller, and a three-dimensional object that is close to the viewer's viewpoint position is its display. Enlarge the display. Further, in the expression, a three-dimensional object having a large display has a high LOD of its display, and a three-dimensional object having a small display has a low LOD of its display. Then, by expressing the 3D object in the 3D space using the LOD defined in this way, the 3D object with a large display is reproduced with high resolution, and the 3D object with a small display is reproduced with low resolution.
  • a plurality of object data of the same 3D object having different LODs are prepared in advance in the content distribution server 10, and the viewpoint of the viewer and the 3D object in the virtual 3D space are prepared in advance.
  • the object data is switched so as to correspond to the LOD defined in the scene description according to the distance between the object and the object.
  • the resolution of the eyes of the viewer is determined based on the visual acuity information of the viewer and the distance information between the viewpoint of the viewer and the three-dimensional object in the virtual three-dimensional space. Calculate the necessary and sufficient LOD selection reference value. Then, the object data for displaying the three-dimensional object is selected based on the calculated LOD selection reference value. By doing so, it is possible to avoid acquiring the data of the three-dimensional object of LOD which is unnecessary for the resolution of the viewer's eyes. Therefore, according to the present embodiment, it is possible to avoid an increase in the amount of distributed data and an increase in the load on the communication network while avoiding the viewer from feeling the deterioration of the image quality.
  • FIGS. 8 to 12. 8 to 12 are explanatory views for explaining the outline of the present embodiment.
  • FIG. 8 shows changes in the size of the displayed 3D object and the required LOD in accordance with the change in the viewpoint position of the viewer in the 3D virtual space in the present embodiment. It is a figure.
  • the display LOD of the displayed object 1 is high (for example, when the object 1 is represented by the Point Cloud, the number of points is large), and the object 1 is reproduced with high resolution.
  • the viewpoint of the viewer is at a medium distance from the object 1, the size of the displayed object 1 is medium. Therefore, the display LOD of the displayed object 1 is medium, and it is reproduced at a medium resolution.
  • the display LOD of the displayed object 1 becomes low, and it is reproduced at a low resolution.
  • a plurality of object data having different LODs are prepared in the content distribution server 10 for the same three-dimensional object.
  • the object to be referred to is linked to the LOD selection reference value, which is the distance (LOD distance) between the viewer's viewpoint and the three-dimensional object in the virtual three-dimensional space.
  • the playback device 30a acquires object data of a three-dimensional object according to a table showing data reference information (for example, a reference URL (Uniform Resource Locator) or the like).
  • the table shown in FIG. 9 is set in consideration of the resolution of the display device 20, and if the resolution of the target display device 20 itself is different, the table will change accordingly.
  • the details required for displaying a three-dimensional object are limited by the resolution (number of pixels) of the display device 20 to be displayed before the displayed size. Therefore, in the present embodiment, the number of points (LOD) associated with the distance (LOD distance) between the viewer's viewpoint and the three-dimensional object in the virtual three-dimensional space is displayed on the display device 20. It is required to satisfy the condition that it is sufficient for the number of pixels in.
  • the resolution of the display device 20 is 4K (3840 ⁇ 2160)
  • the density of points required to display the three-dimensional object is 1/4 of that when the resolution of the display device 20 is 8K (7680 ⁇ 4320). Therefore, when the resolution of the display device 20 is 4K (3840 ⁇ 2160), the density in one dimension along the 20 width direction or the height direction of the display device is 8K (7680 ⁇ 4320) for the display device 20. It is halved compared to the case of. For example, as shown in FIG. 10, if the resolution of the display device 20 is different, the number of points required for display on the 4K display device 20 is 8K even if the three-dimensional objects are displayed in the same size. It is 1/2 of the number of points required for display on the display device 20.
  • FIG. 11 it is assumed that the entire three-dimensional object having a height h is displayed on the 8K display device 20.
  • the viewer is located at a distance I in the virtual space so that the viewer's viewing range can include the entire 3D object, and the number of points in the height direction of the 3D object is determined.
  • the number of pixels in the height direction of the display device 20 is 4320, which is the same as the number of pixels. That is, since the number of points and the number of pixels are the same, it can be said that the display of the three-dimensional object has a sufficient point density for display on the 8K display device 20.
  • the viewer shall move to a position at a distance of I / 2 in the virtual space with respect to the 3D object.
  • half of the range of the three-dimensional object is displayed so as to occupy the height direction of the display surface of the display device 20.
  • the 3D object displayed with the same object data as the initial situation will have 2160 points in the height direction.
  • the number of points is insufficient compared to the number of pixels (4320) in the height direction of the display device. Therefore, under the condition of the distance of I / 2 on the virtual space, the three dimensions are met. The object cannot be properly displayed on the 8K display device 20.
  • the display device 20 has a resolution of 4K (3840 ⁇ 2160), the number of points and the number of pixels are the same. Therefore, under the condition of the distance of I / 2 on the virtual space, the three-dimensional object is 4K. It can be appropriately displayed on the display device 20.
  • the distance (LOD distance) between the viewer's viewpoint and the three-dimensional object in the virtual three-dimensional space is also halved. That is, when the necessary and sufficient resolution of the display device calculated from the viewer's eyesight and distance when viewing on the 8K display device is 4K, the viewer in the virtual three-dimensional space, which is the selection reference value of LOD. It can be seen that the distance (LOD distance) between the viewpoint and the 3D object is half of the value defined in the original scene description.
  • the necessary and sufficient resolution for the viewer is calculated based on the viewer's visual acuity (visual acuity information) and the viewer's distance to the display device 20 (distance information).
  • the resolution can be considered to be the resolution of the display device 20 that is necessary and sufficient under a predetermined viewing situation.
  • the necessary and sufficient resolution for the viewer and the resolution of the display device 20 (display).
  • the relationship between the LOD selection reference value defined in the initial scene description and the LOD selection reference value necessary and sufficient for the actual viewer can be shown by the following formula (8).
  • the formula (8) uses the resolution in the width direction of the display device 20, in the present embodiment, the resolution in the height direction may be used instead of the resolution in the width direction.
  • the ratio of the new LOD selection reference value to the LOD selection reference value defined in the initial scene description between the viewer A and the viewer B with a visual acuity of 0.5 is as follows using the formula (8). Is calculated as follows. In the case of viewer A (visual acuity 1.0), 3819.62 / 7680 ⁇ 1/2, and in the case of viewer B (visual acuity 0.5), 1909.81 / 7680 ⁇ 1/4.
  • the new LOD selection reference values for viewer A and viewer B calculated based on such an idea are shown in a table as shown in FIG.
  • the new LOD selection reference value for the viewer A is shown as the case (1) replacement distance
  • the new LOD selection reference value for the viewer B is shown as the case (2) replacement distance.
  • the H-LOD object which is initially high LOD object data
  • the object data of the M-LOD which is the object data of the medium LOD
  • the object data of the M-LOD may be selected
  • L-LOD object data which is low LOD object data
  • the present embodiment it is possible to avoid an increase in the amount of distributed data and an increase in the load on the communication network while avoiding the viewer from feeling the deterioration of the image quality.
  • the details of the present embodiment will be sequentially described.
  • the display device 20 is not limited to a two-dimensional display such as a television, a tablet, and a smartphone, and is, for example, an AR (Augmented Reality) glass or HMD worn on the head of a viewer. It may be a wearable device such as (Head Mounted Display). These HMDs and the like may include a positioning sensor (not shown) and a motion sensor (not shown), and in this case, the position of the viewer, the direction and inclination of the body, the movement, the moving speed, and the like can be detected.
  • a positioning sensor not shown
  • a motion sensor not shown
  • FIG. 13 is a block diagram showing a functional configuration example of the reproduction device 30a according to the present embodiment. Since the reproduction device 30a according to the present embodiment is common to the first embodiment except for the following points, the description of the common points will be omitted here, and only the differences will be described.
  • the present embodiment is different from the first embodiment in that the display control unit 300 of the reproduction device 30a includes the LOD calculation unit (selection reference value calculation unit) 310.
  • the LOD calculation unit 310 is based on the viewer's visual acuity (visual acuity information) and the viewer's distance to the display device 20 (distance information), and based on the necessary and sufficient resolution (user resolution) for the viewer. Calculate a new LOD selection reference value for. Since the calculation method has been described above, the description thereof will be omitted here.
  • the acquisition unit 308 has object data (three-dimensional object in the moving image content) having a LOD necessary and sufficient for the resolution of the viewer's eyes based on the newly calculated LOD selection reference value. Content data for displaying)).
  • FIG. 14 is a diagram showing a flowchart of the information processing method according to the embodiment.
  • the distribution processing method according to the present embodiment includes steps S301 to S311. The details of each of these steps according to the present embodiment will be described below. In the following description, only the points different from the above-mentioned first embodiment will be described, and the points common to the first embodiment will be omitted.
  • steps S301 to S306 are the same as steps S101 to S106 of the first embodiment shown in FIG. 5, the description thereof will be omitted here.
  • the playback device 30a calculates a new LOD selection reference value for the viewer based on the resolution specified in step S306 described above (step S307). Since the calculation method has been described above, the description thereof will be omitted here. Then, the reproduction device 30a acquires the object data (content data) of the three-dimensional object having the LOD corresponding to the LOD selection reference value selected in step S307 described above (step S308).
  • steps S309 to S311 are the same as steps S108 to S110 of the first embodiment shown in FIG. 5, the description thereof will be omitted here.
  • the present embodiment it is possible to avoid acquiring the object data of the three-dimensional object of LOD which is unnecessary for the resolution of the eyes of the viewer. Therefore, according to the present embodiment, it is possible to avoid an increase in the amount of distributed data and an increase in the load on the communication network while avoiding the viewer from feeling the deterioration of the image quality.
  • the viewing angle calculates the necessary and sufficient resolution (necessary and sufficient resolution).
  • the resolution in the width direction or the resolution in the height direction may be used, but in the present embodiment, the viewing angle ⁇ and the viewing angle are the necessary and sufficient resolutions. If ⁇ is different from each other, the calculated values will also change significantly, so the larger value of the resolution in the height direction and the resolution in the width direction will be used. Therefore, in the present embodiment, the relationship between the LOD selection reference value defined in the initial scene description and the LOD selection reference value necessary and sufficient for the actual viewer can be shown by the following mathematical formula (9).
  • FIG. 15 is a diagram showing a flowchart of the information processing method according to the embodiment.
  • the information processing method according to the present embodiment includes steps S401 to S414. The details of each of these steps according to the present embodiment will be described below. In the following description, only the points different from the above-mentioned second and third embodiments will be described, and the points common to the second and third embodiments will be omitted.
  • steps S401 to S409 are the same as steps S201 to S209 of the second embodiment shown in FIG. 7, the description thereof will be omitted here.
  • steps S410 to S414 are the same as steps S307 to S311 of the third embodiment shown in FIG. 14, the description thereof will be omitted here.
  • the LOD can be performed more accurately.
  • the selection reference value can be calculated. Therefore, according to the present embodiment, since the calculation is performed in consideration of the viewing angle, the LOD selection reference value suitable for the viewer can be calculated with high accuracy, so that the viewer can more effectively calculate. While avoiding the deterioration of image quality, it is possible to avoid an increase in the amount of distributed data and an increase in the load on the communication network.
  • each embodiment of the present disclosure it is possible to avoid acquiring content data having a resolution unnecessary for the resolution of the viewer's eyes, and the content necessary and sufficient for the resolution of the viewer's eyes. You can get the data. Therefore, according to each embodiment of the present disclosure, it is possible to avoid an increase in the amount of distributed data and an increase in the load on the communication network while avoiding the viewer from feeling the deterioration of the image quality. It should be noted that each embodiment of the present disclosure is not limited to application to the provision of entertainment services accompanied by video distribution, but can also be applied to education, medical support, and the like.
  • FIG. 16 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the playback device 30 and the like.
  • the computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each part of the computer 1000 is connected by a bus 1050.
  • the CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400, and controls each part. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200, and executes processing corresponding to various programs.
  • the ROM 1300 stores a boot program such as a BIOS (Basic Input Output System) executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000, and the like.
  • BIOS Basic Input Output System
  • the HDD 1400 is a computer-readable recording medium that non-temporarily records a program executed by the CPU 1100 and data used by such a program.
  • the HDD 1400 is a recording medium for recording an information processing program according to the present disclosure, which is an example of program data 1450.
  • the communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet).
  • the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.
  • the input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000.
  • the CPU 1100 receives data from an input / output device 1650 such as a keyboard, a mouse, and a microphone (microphone) via the input / output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600.
  • the input / output interface 1600 may function as a media interface for reading a program or the like recorded on a predetermined recording medium (media).
  • the media includes, for example, an optical recording medium such as a DVD (Digital Versaille Disc), a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magnet-Optical disc), a tape medium, a magnetic recording medium, a semiconductor memory, or the like.
  • an optical recording medium such as a DVD (Digital Versaille Disc), a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magnet-Optical disc), a tape medium, a magnetic recording medium, a semiconductor memory, or the like.
  • the CPU 1100 of the computer 1000 realizes the functions of the display control unit 310 and the like by executing the program stored in the RAM 1200. Further, the processing program and the like according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program, but as another example, these programs may be acquired from another device via the external network 1550.
  • the information processing device may be applied to a system including a plurality of devices, which is premised on connection to a network (or communication between each device), such as cloud computing. .. That is, the information processing device according to the present embodiment described above can be realized as an information processing system according to the present embodiment by, for example, a plurality of devices.
  • the above is an example of the hardware configuration of the playback device 30 and the like.
  • Each of the above-mentioned components may be configured by using general-purpose members, or may be configured by hardware specialized for the function of each component. Such a configuration may be appropriately modified depending on the technical level at the time of implementation.
  • FIG. 17 is a diagram showing an example of a schematic configuration of an endoscopic surgery system 5000 to which the technique according to the present disclosure can be applied.
  • FIG. 17 shows a surgeon (doctor) 5067 performing surgery on patient 5071 on patient bed 5069 using the endoscopic surgery system 5000.
  • the endoscopic surgery system 5000 includes an endoscope 5001, other surgical tools 5017, a support arm device 5027 for supporting the endoscope 5001, and various devices for endoscopic surgery. It is composed of a cart 5037 and a cart 5037.
  • trocca 5025a to 5025d In endoscopic surgery, instead of cutting and opening the abdominal wall, multiple tubular laparotomy instruments called trocca 5025a to 5025d are punctured into the abdominal wall. Then, from the trocca 5025a to 5025d, the lens barrel 5003 of the endoscope 5001 and other surgical tools 5017 are inserted into the body cavity of the patient 5071.
  • other surgical tools 5017 a pneumoperitoneum tube 5019, an energy treatment tool 5021 and forceps 5023 are inserted into the body cavity of patient 5071.
  • the energy treatment tool 5021 is a treatment tool for incising and peeling a tissue, sealing a blood vessel, or the like by using a high frequency current or ultrasonic vibration.
  • the surgical tool 5017 shown in the illustration is merely an example, and as the surgical tool 5017, various surgical tools generally used in endoscopic surgery such as a sword and a retractor may be used.
  • the image of the surgical site in the body cavity of the patient 5071 taken by the endoscope 5001 is displayed on the display device 5041.
  • the surgeon 5067 performs a procedure such as excising the affected area by using the energy treatment tool 5021 or the forceps 5023 while viewing the image of the surgical site displayed on the display device 5041 in real time.
  • the pneumoperitoneum tube 5019, the energy treatment tool 5021, and the forceps 5023 are supported by the operator 5067, an assistant, or the like during the operation.
  • the support arm device 5027 includes an arm portion 5031 extending from the base portion 5029.
  • the arm portion 5031 is composed of joint portions 5033a, 5033b, 5033c, and links 5035a, 5035b, and is driven by control from the arm control device 5045.
  • the endoscope 5001 is supported by the arm portion 5031, and its position and posture are controlled. Thereby, the stable position fixing of the endoscope 5001 can be realized.
  • the endoscope 5001 is composed of a lens barrel 5003 in which a region having a predetermined length from the tip is inserted into the body cavity of the patient 5071, and a camera head 5005 connected to the base end of the lens barrel 5003.
  • the endoscope 5001 configured as a so-called rigid mirror having a rigid barrel 5003 is illustrated, but the endoscope 5001 is configured as a so-called flexible mirror having a flexible barrel 5003. May be good.
  • An opening in which an objective lens is fitted is provided at the tip of the lens barrel 5003.
  • a light source device 5043 is connected to the endoscope 5001, and the light generated by the light source device 5043 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 5003, and is an objective. It is irradiated toward the observation target in the body cavity of the patient 5071 through the lens.
  • the endoscope 5001 may be a direct endoscope, a perspective mirror, or a side endoscope.
  • An optical system and an image pickup element are provided inside the camera head 5005, and the reflected light (observation light) from the observation target is focused on the image pickup element by the optical system.
  • the observation light is photoelectrically converted by the image pickup device, and an electric signal corresponding to the observation light, that is, an image signal corresponding to the observation image is generated.
  • the image signal is transmitted as RAW data to the camera control unit (CCU: Camera Control Unit) 5039.
  • the camera head 5005 is equipped with a function of adjusting the magnification and the focal length by appropriately driving the optical system thereof.
  • the camera head 5005 may be provided with a plurality of image pickup elements.
  • a plurality of relay optical systems are provided inside the lens barrel 5003 in order to guide the observation light to each of the plurality of image pickup elements.
  • the CCU 5039 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and comprehensively controls the operations of the endoscope 5001 and the display device 5041. Specifically, the CCU 5039 performs various image processing for displaying an image based on the image signal, such as a development process (demosaic process), on the image signal received from the camera head 5005. The CCU 5039 provides the image signal subjected to the image processing to the display device 5041. Further, the CCU 5039 transmits a control signal to the camera head 5005 and controls the driving thereof.
  • the control signal may include information about imaging conditions such as magnification and focal length.
  • the display device 5041 displays an image based on the image signal processed by the CCU 5039 under the control of the CCU 5039.
  • the endoscope 5001 is compatible with high-resolution shooting such as 4K (horizontal pixel number 3840 x vertical pixel number 2160) or 8K (horizontal pixel number 7680 x vertical pixel number 4320), and / or 3D display.
  • the display device 5041 a display device capable of displaying a high resolution and / or a device capable of displaying in 3D can be used.
  • a display device 5041 having a size of 55 inches or more is used for high-resolution shooting such as 4K or 8K, a further immersive feeling can be obtained.
  • a plurality of display devices 5041 having different resolutions and sizes may be provided depending on the application.
  • the light source device 5043 is composed of, for example, a light source such as an LED (Light Emitting Diode), and supplies irradiation light for photographing the surgical site to the endoscope 5001.
  • a light source such as an LED (Light Emitting Diode)
  • LED Light Emitting Diode
  • the arm control device 5045 is configured by a processor such as a CPU, and operates according to a predetermined program to control the drive of the arm portion 5031 of the support arm device 5027 according to a predetermined control method.
  • the input device 5047 is an input interface for the endoscopic surgery system 5000.
  • the user can input various information and input instructions to the endoscopic surgery system 5000 via the input device 5047.
  • the user inputs various information related to the surgery, such as physical information of the patient and information about the surgical procedure, via the input device 5047.
  • the user is instructed to drive the arm portion 5031 via the input device 5047, or is instructed to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) by the endoscope 5001. , Instructions to drive the energy treatment tool 5021, etc. are input.
  • the type of the input device 5047 is not limited, and the input device 5047 may be various known input devices.
  • the input device 5047 for example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5057 and / or a lever and the like can be applied.
  • the touch panel may be provided on the display surface of the display device 5041.
  • the input device 5047 is a device worn by the user, such as a glasses-type wearable device or an HMD (Head Mounted Display), and various inputs are made according to the user's gesture and line of sight detected by these devices. Is done. Further, the input device 5047 includes a camera capable of detecting the movement of the user, and various inputs are performed according to the gesture and the line of sight of the user detected from the image captured by the camera. Further, the input device 5047 includes a microphone capable of picking up the voice of the user, and various inputs are performed by voice via the microphone.
  • a glasses-type wearable device or an HMD Head Mounted Display
  • the input device 5047 is configured to be able to input various information in a non-contact manner, so that a user who belongs to a clean area (for example, an operator 5067) can operate a device belonging to the unclean area in a non-contact manner. Is possible. In addition, the user can operate the device without taking his / her hand off the surgical tool that he / she has, which improves the convenience of the user.
  • a clean area for example, an operator 5067
  • the treatment tool control device 5049 controls the drive of the energy treatment tool 5021 for cauterizing tissue, incising, sealing a blood vessel, or the like.
  • the pneumoperitoneum device 5051 gas in the body cavity of the patient 5071 via the pneumoperitoneum tube 5019 in order to inflate the body cavity of the patient 5071 for the purpose of securing the field of view by the endoscope 5001 and securing the work space of the operator. Is sent.
  • the recorder 5053 is a device capable of recording various information related to surgery.
  • the printer 5055 is a device capable of printing various information related to surgery in various formats such as text, images, and graphs.
  • the support arm device 5027 includes a base portion 5029 which is a base, and an arm portion 5031 extending from the base portion 5029.
  • the arm portion 5031 is composed of a plurality of joint portions 5033a, 5033b, 5033c and a plurality of links 5035a, 5035b connected by the joint portions 5033b, but in FIG. 17, for the sake of simplicity.
  • the configuration of the arm portion 5031 is simplified and illustrated. Actually, the shapes, numbers and arrangements of the joint portions 5033a to 5033c and the links 5035a and 5035b, the direction of the rotation axis of the joint portions 5033a to 5033c, and the like are appropriately set so that the arm portion 5031 has a desired degree of freedom.
  • the arm portion 5031 may be preferably configured to have more than 6 degrees of freedom.
  • the endoscope 5001 can be freely moved within the movable range of the arm portion 5031, so that the lens barrel 5003 of the endoscope 5001 can be inserted into the body cavity of the patient 5071 from a desired direction. It will be possible.
  • Actuators are provided in the joint portions 5033a to 5033c, and the joint portions 5033a to 5033c are configured to be rotatable around a predetermined rotation axis by driving the actuator.
  • the arm control device 5045 By controlling the drive of the actuator by the arm control device 5045, the rotation angles of the joint portions 5033a to 5033c are controlled, and the drive of the arm portion 5031 is controlled. Thereby, control of the position and posture of the endoscope 5001 can be realized.
  • the arm control device 5045 can control the drive of the arm unit 5031 by various known control methods such as force control or position control.
  • the drive of the arm unit 5031 is appropriately controlled by the arm control device 5045 according to the operation input.
  • the position and orientation of the endoscope 5001 may be controlled.
  • the endoscope 5001 at the tip of the arm portion 5031 can be moved from an arbitrary position to an arbitrary position, and then fixedly supported at the position after the movement.
  • the arm portion 5031 may be operated by a so-called master slave method. In this case, the arm portion 5031 can be remotely controlled by the user via an input device 5047 installed at a location away from the operating room.
  • the arm control device 5045 When force control is applied, the arm control device 5045 receives an external force from the user, and the actuators of the joint portions 5033a to 5033c are arranged so that the arm portion 5031 moves smoothly according to the external force. So-called power assist control for driving may be performed.
  • the arm portion 5031 when the user moves the arm portion 5031 while directly touching the arm portion 5031, the arm portion 5031 can be moved with a relatively light force. Therefore, the endoscope 5001 can be moved more intuitively and with a simpler operation, and the convenience of the user can be improved.
  • the endoscope 5001 was supported by a doctor called a scopist.
  • the support arm device 5027 by using the support arm device 5027, the position of the endoscope 5001 can be more reliably fixed without human intervention, so that an image of the surgical site can be stably obtained. , It becomes possible to perform surgery smoothly.
  • the arm control device 5045 does not necessarily have to be provided on the cart 5037. Further, the arm control device 5045 does not necessarily have to be one device. For example, the arm control device 5045 may be provided in each of the joint portions 5033a to 5033c of the arm portion 5031 of the support arm device 5027, and the plurality of arm control devices 5045 cooperate with each other to drive the arm portion 5031. Control may be realized.
  • the light source device 5043 supplies the endoscope 5001 with irradiation light for photographing the surgical site.
  • the light source device 5043 is composed of, for example, an LED, a laser light source, or a white light source composed of a combination thereof.
  • the white light source is configured by the combination of the RGB laser light sources, the output intensity and the output timing of each color (each wavelength) can be controlled with high accuracy, so that the white balance of the captured image in the light source device 5043 can be controlled. Can be adjusted.
  • the laser light from each of the RGB laser light sources is irradiated to the observation target in a time-division manner, and the drive of the image sensor of the camera head 5005 is controlled in synchronization with the irradiation timing to correspond to each of RGB. It is also possible to capture the image in a time-division manner. According to this method, a color image can be obtained without providing a color filter in the image pickup device.
  • the drive of the light source device 5043 may be controlled so as to change the intensity of the output light at predetermined time intervals.
  • the drive of the image sensor of the camera head 5005 in synchronization with the timing of the change of the light intensity to acquire an image in time division and synthesizing the image, so-called high dynamic without blackout and overexposure. Range images can be generated.
  • the light source device 5043 may be configured to be able to supply light in a predetermined wavelength band corresponding to special light observation.
  • special light observation for example, by utilizing the wavelength dependence of light absorption in body tissue, the surface layer of the mucous membrane is irradiated with light in a narrower band than the irradiation light (that is, white light) during normal observation.
  • a so-called narrow band imaging is performed in which a predetermined tissue such as a blood vessel is photographed with high contrast.
  • fluorescence observation may be performed in which an image is obtained by fluorescence generated by irradiating with excitation light.
  • the body tissue is irradiated with excitation light to observe the fluorescence from the body tissue (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue and the body tissue is injected.
  • An excitation light corresponding to the fluorescence wavelength of the reagent may be irradiated to obtain a fluorescence image.
  • the light source device 5043 may be configured to be capable of supplying narrowband light and / or excitation light corresponding to such special light observation.
  • FIG. 18 is a block diagram showing an example of the functional configuration of the camera head 5005 and CCU5039 shown in FIG.
  • the camera head 5005 has a lens unit 5007, an image pickup unit 5009, a drive unit 5011, a communication unit 5013, and a camera head control unit 5015 as its functions.
  • the CCU 5039 has a communication unit 5059, an image processing unit 5061, and a control unit 5063 as its functions.
  • the camera head 5005 and the CCU 5039 are bidirectionally connected by a transmission cable 5065 so as to be communicable.
  • the lens unit 5007 is an optical system provided at a connection portion with the lens barrel 5003.
  • the observation light taken in from the tip of the lens barrel 5003 is guided to the camera head 5005 and incident on the lens unit 5007.
  • the lens unit 5007 is configured by combining a plurality of lenses including a zoom lens and a focus lens.
  • the optical characteristics of the lens unit 5007 are adjusted so as to collect the observation light on the light receiving surface of the image pickup element of the image pickup unit 5009.
  • the zoom lens and the focus lens are configured so that their positions on the optical axis can be moved in order to adjust the magnification and the focus of the captured image.
  • the image pickup unit 5009 is composed of an image pickup element and is arranged after the lens unit 5007.
  • the observation light that has passed through the lens unit 5007 is focused on the light receiving surface of the image pickup device, and an image signal corresponding to the observation image is generated by photoelectric conversion.
  • the image signal generated by the image pickup unit 5009 is provided to the communication unit 5013.
  • CMOS Complementary Metal Oxide Semiconductor
  • image pickup device for example, an image pickup device capable of capturing a high-resolution image of 4K or higher may be used.
  • the image pickup elements constituting the image pickup unit 5009 are configured to have a pair of image pickup elements for acquiring image signals for the right eye and the left eye corresponding to 3D display, respectively.
  • the 3D display enables the surgeon 5067 to more accurately grasp the depth of the living tissue in the surgical site.
  • the image pickup unit 5009 is composed of a multi-plate type, a plurality of lens units 5007 are also provided corresponding to each image pickup element.
  • the image pickup unit 5009 does not necessarily have to be provided on the camera head 5005.
  • the image pickup unit 5009 may be provided inside the lens barrel 5003 immediately after the objective lens.
  • the drive unit 5011 is composed of an actuator, and the zoom lens and the focus lens of the lens unit 5007 are moved by a predetermined distance along the optical axis under the control of the camera head control unit 5015. As a result, the magnification and focus of the image captured by the image pickup unit 5009 can be adjusted as appropriate.
  • the communication unit 5013 is composed of a communication device for transmitting and receiving various information to and from the CCU 5039.
  • the communication unit 5013 transmits the image signal obtained from the image pickup unit 5009 as RAW data to the CCU 5039 via the transmission cable 5065.
  • the image signal is transmitted by optical communication.
  • the surgeon 5067 performs the surgery while observing the condition of the affected area with the captured image, so for safer and more reliable surgery, the moving image of the surgical site is displayed in real time as much as possible. This is because it is required.
  • the communication unit 5013 is provided with a photoelectric conversion module that converts an electric signal into an optical signal.
  • the image signal is converted into an optical signal by the photoelectric conversion module, and then transmitted to the CCU 5039 via the transmission cable 5065.
  • the communication unit 5013 receives a control signal for controlling the drive of the camera head 5005 from the CCU 5039.
  • the control signal includes, for example, information to specify the frame rate of the captured image, information to specify the exposure value at the time of imaging, and / or information to specify the magnification and focus of the captured image. Contains information about the condition.
  • the communication unit 5013 provides the received control signal to the camera head control unit 5015.
  • the control signal from the CCU 5039 may also be transmitted by optical communication.
  • the communication unit 5013 is provided with a photoelectric conversion module that converts an optical signal into an electric signal, and the control signal is converted into an electric signal by the photoelectric conversion module and then provided to the camera head control unit 5015.
  • the image pickup conditions such as the frame rate, the exposure value, the magnification, and the focal point are automatically set by the control unit 5063 of the CCU 5039 based on the acquired image signal. That is, the so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function are mounted on the endoscope 5001.
  • the camera head control unit 5015 controls the drive of the camera head 5005 based on the control signal from the CCU 5039 received via the communication unit 5013. For example, the camera head control unit 5015 controls the drive of the image pickup element of the image pickup unit 5009 based on the information to specify the frame rate of the image pickup image and / or the information to specify the exposure at the time of image pickup. Further, for example, the camera head control unit 5015 appropriately moves the zoom lens and the focus lens of the lens unit 5007 via the drive unit 5011 based on the information that the magnification and the focus of the captured image are specified.
  • the camera head control unit 5015 may further have a function of storing information for identifying the lens barrel 5003 and the camera head 5005.
  • the camera head 5005 can be made resistant to autoclave sterilization.
  • the communication unit 5059 is configured by a communication device for transmitting and receiving various information to and from the camera head 5005.
  • the communication unit 5059 receives an image signal transmitted from the camera head 5005 via the transmission cable 5065.
  • the image signal can be suitably transmitted by optical communication.
  • the communication unit 5059 is provided with a photoelectric conversion module that converts an optical signal into an electric signal.
  • the communication unit 5059 provides the image processing unit 5061 with an image signal converted into an electric signal.
  • the communication unit 5059 transmits a control signal for controlling the drive of the camera head 5005 to the camera head 5005.
  • the control signal may also be transmitted by optical communication.
  • the image processing unit 5061 performs various image processing on the image signal which is the RAW data transmitted from the camera head 5005.
  • the image processing includes, for example, development processing, high image quality processing (band enhancement processing, super-resolution processing, NR (Noise Reduction) processing and / or camera shake correction processing, etc.), and / or enlargement processing (electronic zoom processing). Etc., various known signal processing is included. Further, the image processing unit 5061 performs detection processing on the image signal for performing AE, AF and AWB.
  • the image processing unit 5061 is composed of a processor such as a CPU or GPU, and the processor operates according to a predetermined program, so that the above-mentioned image processing and detection processing can be performed.
  • the image processing unit 5061 is composed of a plurality of GPUs, the image processing unit 5061 appropriately divides the information related to the image signal and performs image processing in parallel by the plurality of GPUs.
  • the control unit 5063 performs various controls regarding the imaging of the surgical site by the endoscope 5001 and the display of the captured image. For example, the control unit 5063 generates a control signal for controlling the drive of the camera head 5005. At this time, when the imaging condition is input by the user, the control unit 5063 generates a control signal based on the input by the user. Alternatively, when the endoscope 5001 is equipped with an AE function, an AF function, and an AWB function, the control unit 5063 has an optimum exposure value, a focal length, and an optimum exposure value according to the result of detection processing by the image processing unit 5061. The white balance is calculated appropriately and a control signal is generated.
  • control unit 5063 causes the display device 5041 to display the image of the surgical unit based on the image signal processed by the image processing unit 5061.
  • the control unit 5063 recognizes various objects in the surgical unit image by using various image recognition techniques.
  • the control unit 5063 detects a surgical tool such as forceps, a specific biological part, bleeding, a mist when using the energy treatment tool 5021, etc. by detecting the shape, color, etc. of the edge of the object included in the surgical site image. Can be recognized.
  • the control unit 5063 uses the recognition result to superimpose and display various surgical support information on the image of the surgical site. By superimposing the surgical support information and presenting it to the surgeon 5067, it becomes possible to proceed with the surgery more safely and surely.
  • the transmission cable 5065 connecting the camera head 5005 and the CCU 5039 is an electric signal cable compatible with electric signal communication, an optical fiber compatible with optical communication, or a composite cable thereof.
  • the communication is performed by wire using the transmission cable 5065, but the communication between the camera head 5005 and the CCU 5039 may be performed wirelessly.
  • the communication between the two is performed wirelessly, it is not necessary to lay the transmission cable 5065 in the operating room, so that the situation where the movement of the medical staff in the operating room is hindered by the transmission cable 5065 can be solved.
  • the above is an example of the endoscopic surgery system 5000 to which the technique according to the present disclosure can be applied.
  • the endoscopic surgery system 5000 has been described here as an example, the system to which the technique according to the present disclosure can be applied is not limited to such an example.
  • the technique according to the present disclosure may be applied to a flexible endoscopic system for examination or a microsurgery system.
  • each step in the information processing method of the embodiment of the present disclosure described above does not necessarily have to be processed in the order described.
  • each step may be processed in an appropriately reordered manner.
  • each step may be partially processed in parallel or individually instead of being processed in chronological order.
  • the processing of each step does not necessarily have to be processed according to the described method, and may be processed by another method, for example, by another functional unit.
  • An information processing device including an acquisition unit that acquires video content data to be played back to the user in real time according to the user's visual acuity information.
  • a calculation unit that calculates the user resolution for the user in real time based on the user's visual acuity information, the display unit information regarding the display unit that displays the moving image content, and the position information of the user with respect to the display unit.
  • the acquisition unit acquires the moving image content data based on the user resolution.
  • the information processing device includes information on the height and width of the display unit.
  • the calculation unit calculates the user resolution in the height direction and the user resolution in the width direction.
  • the information processing device (2) above.
  • a comparison unit for comparing the user resolution and the resolution of the display unit is provided.
  • the acquisition unit acquires the video content data based on the comparison result.
  • the acquisition unit acquires the video content data for reproducing an object in the virtual space in the video content based on the user resolution.
  • the information processing apparatus according to any one of (2) to (6) above.
  • a selection reference value calculation unit for newly calculating a selection reference value for selecting the moving image content data based on the user resolution.
  • the information processing device according to (7) above.
  • the information processing apparatus according to any one of (2) to (8) above, further comprising a position information acquisition unit for acquiring the position information.
  • the acquisition unit acquires the video content data to be reproduced for the plurality of users according to the visual acuity information of the plurality of users.
  • the information processing apparatus according to any one of (1) to (9) above.
  • (11) Further provided with a decoding unit for decoding the moving image content data.
  • the information processing apparatus according to any one of (1) to (9) above.
  • (12) The information processing apparatus according to any one of (2) to (9) above, further comprising the display unit.
  • Distribution system 10 Content distribution server 20 Display device 30, 30a Playback device 40 Communication network 300 Display control unit 302 Location information acquisition unit 304 Calculation unit 306 Comparison unit 308 Acquisition unit 310 LOD calculation unit 320 Decoding block 322 Processing unit 324 Decoding unit 326 Display information generation unit 330 Main control unit 340 Storage unit 350 Transmission / reception unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

L'invention concerne un appareil de traitement d'informations (30) comprenant une unité d'acquisition (308) servant à acquérir, en temps réel, des données de contenu vidéo à lire à l'intention d'un utilisateur en fonction d'informations sur l'acuité visuelle de l'utilisateur.
PCT/JP2021/015660 2020-05-11 2021-04-16 Appareil de traitement d'informations et procédé de traitement d'informations Ceased WO2021230001A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-083493 2020-05-11
JP2020083493 2020-05-11

Publications (1)

Publication Number Publication Date
WO2021230001A1 true WO2021230001A1 (fr) 2021-11-18

Family

ID=78525677

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/015660 Ceased WO2021230001A1 (fr) 2020-05-11 2021-04-16 Appareil de traitement d'informations et procédé de traitement d'informations

Country Status (1)

Country Link
WO (1) WO2021230001A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001076180A (ja) * 1999-09-07 2001-03-23 Nec Corp ３次元グラフィックス表示装置
JP2015510325A (ja) * 2012-01-19 2015-04-02 ヴィド スケール インコーポレイテッド 観視条件への適応をサポートするビデオ配信のための方法およびシステム
US20150178976A1 (en) * 2011-11-28 2015-06-25 Google Inc. View Dependent Level-of-Detail for Tree-Based Replicated Geometry
US20160080448A1 (en) * 2014-09-11 2016-03-17 Microsoft Corporation Dynamic Video Streaming Based on Viewer Activity
US20170085959A1 (en) * 2015-09-23 2017-03-23 International Business Machines Corporation Adaptive multimedia display

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001076180A (ja) * 1999-09-07 2001-03-23 Nec Corp ３次元グラフィックス表示装置
US20150178976A1 (en) * 2011-11-28 2015-06-25 Google Inc. View Dependent Level-of-Detail for Tree-Based Replicated Geometry
JP2015510325A (ja) * 2012-01-19 2015-04-02 ヴィド スケール インコーポレイテッド 観視条件への適応をサポートするビデオ配信のための方法およびシステム
US20160080448A1 (en) * 2014-09-11 2016-03-17 Microsoft Corporation Dynamic Video Streaming Based on Viewer Activity
US20170085959A1 (en) * 2015-09-23 2017-03-23 International Business Machines Corporation Adaptive multimedia display

Similar Documents

Publication Publication Date Title
CN110622500B (zh) 成像设备和成像方法
EP2903551B1 (fr) Système numérique pour la capture et l'affichage d'une vidéo chirurgicale
CN110832842B (zh) 成像装置和图像产生方法
CN110612720B (zh) 信息处理装置、信息处理方法以及可读存储介质
JP7544034B2 (ja) 画像処理装置、画像処理方法およびプログラム
EP3571662B1 (fr) Appareil de traitement de signal vidéo, procédé de traitement de signal vidéo et programme de compression de plage dynamique
JP7095693B2 (ja) 医療用観察システム
JP2021192313A (ja) 情報処理装置および方法、並びにプログラム
WO2019187502A1 (fr) Appareil de traitement d'image, procédé de traitement d'image, et programme
JP2019008766A (ja) 情報処理装置および方法、並びに、情報処理システム
US12200256B2 (en) Delivery apparatus and delivery method
US20220166962A1 (en) Video signal processing apparatus, video signal processing method, and image pickup apparatus
CN110945399A (zh) 信号处理设备、成像设备、信号处理方法和程序
US11671872B2 (en) Communication system and transmission apparatus
WO2020213296A1 (fr) Dispositif de traitement de signal, procédé de traitement de signal, programme, et système de changement de directivité
JP7230923B2 (ja) 情報処理装置、情報処理方法及びプログラム
WO2021230001A1 (fr) Appareil de traitement d'informations et procédé de traitement d'informations
JP2021193762A (ja) 受信装置、および受信方法、並びに、画像処理システム
JP7420141B2 (ja) 画像処理装置、撮像装置、画像処理方法、プログラム
CN110771102B (zh) 通信系统和控制设备
US20220256096A1 (en) Image processing device, image processing method, and program
WO2020241050A1 (fr) Dispositif de traitement audio, procédé de traitement audio et programme

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21804999

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21804999

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP