KR20080034419A - 3D image generation and display system - Google Patents

Abstract

A 3D image generation and display system that facilitates the display of high quality images in a web browser is a means for generating 3D images from a plurality of different images and computer graphics modeling and generating 3D objects from these images with texture and attribute data. ; Means for converting and outputting the 3D object as a 3D description file in a 3D graphics description language; Means for extracting 3D objects and textures from the 3D description file, setting various attribute data, editing and processing the 3D object to introduce animation, and assign various effects; Means for generating various web-based 3D objects from the generated 3D data files compressed to be displayed within a web browser and generating behavioral data for displaying 3D scenes with animation in a web browser; And means for generating an executable file that includes a web page and web-based programs such as scripts, plug-ins, and applets for drawing and displaying 3D scenes in a web browser.

Description

3D image generation and display system {3D IMAGE GENERATION AND DISPLAY SYSTEM}

The invention creates a 3D object for displaying various photographic images and computer graphics models in 3D and edits and processes the 3D objects for drawing and displaying 3D scenes in a web browser. It relates to a 3D image generation and display system.

There are a variety of systems well known in the art for generating 3D objects for use in 3D displays. One such technique that uses a 3D scanner to model and display 3D objects is a light-sectioning method (implemented by projecting a slit of light) that is well known in the art. . This method uses a CCD camera to capture points or lines of light projected on an object by a laser beam or other light source and to perform 3D modeling by measuring the distance from the camera using the principles of triangulation.

FIG. 13A is a schematic diagram illustrating a conventional 3D modeling apparatus using light sectioning.

The CCD camera captures images when the light slit is projected onto the object from the light source. By scanning the entire object to be measured while gradually changing the direction in which the light source projects the light slit, an image as shown in Fig. 13B is obtained. 3D shape data is calculated according to the triangulation method from the known positions of the light source and the camera. However, since the entire outer surface of the object cannot be represented in three dimensions by the light-splitting method, as shown in FIG. 14, by providing a plurality of cameras, the entire area of the object is collected to collect images by rotating the entire outer surface of the object. It must be able to be imaged without this.

In addition, the 3D objects created through these methods can then be used to display various 3D images according to the desired use, as well as various data processes required for three-dimensionally displaying the objects in a web browser. You have to go through effect application and animation processes. For example, the image must be optimized by reducing the file size to match the quality of the communication line.

One type of 3D image display is a liquid crystal panel (LCD) or a display used in game consoles for displaying 3D images that appear to be objects protruding from the screen. This technique uses special glasses, such as polarized glasses, in which the left and right lenses have different polarization directions. In this 3D image display apparatus, the left and right images are captured at the same positions as in the case of the left and right eyes, and polarization is used so that the left image is seen only by the left eye and the right image is seen by the right eye only. Other examples include devices using mirrors or prisms. However, these 3D image displays have complexity, such as requiring a viewer to wear glasses. Accordingly, 3D image display systems have been developed and commercialized using lenticular lenses, parallax barriers, or other devices that allow 3D images to be viewed without glasses. One such apparatus is a "3D video signal generator" disclosed in Reference Patent 1 (Japanese Unexamined Patent Application Publication H10-271533). This device is an improvement on the 3D image display disclosed in US Pat. No. 5,410,345 (April 25, 1995) by allowing 3D images to be displayed on a common LCD system used to display two-dimensional images.

Fig. 15 is a schematic diagram showing this 3D video signal generator. The 3D image signal generator includes a backlight 1 including a light source 12 positioned at sides in a side illumination method; A lenticular lens 15 capable of moving back and forth; A diffuser 5 which slightly diffuses the incident light; And an LCD 6 for displaying an image. As shown in the stereoscopic display image 20 of FIG. 16, the LCD 6 is well known in the art in which pixels P displaying respective colors R, G, and B are arranged in a stripe pattern. Has a structure. A single pixel Pk (k = 0-n) consists of three sub-pixels of horizontally aligned RGB. The color of the pixel is displayed by mixing the three primary colors displayed by each sub-pixel in an additional process.

When displaying a 3D image using the backlight 1 shown in FIG. 15, the lenticular lens 15 has a sub-pixel array on the LCD 6 as seen from the right eye 11. To appear differently from the sub-pixel array seen from To illustrate this phenomenon based on the stereoscopic display image 20 of FIG. 16, the right eye 11 can only see sub-pixels of odd columns 1, 3, 5,. The eye 10 can only see the subpixels of even columns 0, 2, 4,... Thus, to display a 3D image, the 3D image signal generator generates a 3D image signal from the image signals of the left and right images captured at the positions of the left and right eyes and supplies these signals to the LCD 6. .

As shown in FIG. 16, the stereoscopic display image 20 is generated by interleaving the RGB signals of the left image 21 and the right image 22. In this way, the 3D video signal generator forms rgb elements of pixel P0 of the 3D video signal from r and b elements of pixel P0 of the left video signal and g elements of pixel P0 of the right video signal, and the pixels of the left video signal. Rgb elements of the pixel P1 of the 3D image signal (center columns) are formed from the g element of P1 and the r and b elements of the pixel P1 of the right image signal. With this inter-positioning process, the 3D video signal is usually the rgb elements of kth pixels (k is 1, 2, ...) and the r and b elements of the kth pixel of the left video signal and the gth of kth pixels of the right video signal. And rgb elements of the k + 1th image pixel of the 3D image signal are formed from the g element of the k + 1th pixel of the left image signal and the r and b elements of the pixel of the right image signal.

The 3D image signals generated by this method can display a 3D image compressed with the same number of pixels in the original image. As shown in Fig. 18, since the left eye can only see the subpixels of the LCD 6 displayed in even columns, the right eye can see only the subpixels displayed in odd columns, so that the 3D image This can be displayed. In addition, by adjusting the position of the lenticular lens 15, it is possible to switch the display between the 3D and 2D display.

Although the example described in FIG. 15 has a lenticular lens 15 aligned to the back of the LCD 6, the "stereoscopic image display apparatus" disclosed in Reference Patent 2 (Japanese Unexamined Patent Application Publication H11-72745) An example of a lenticular lens positioned in front of the LCD is presented. As shown in FIG. 19, the stereoscopic image display apparatus has a parallax barrier 26 (also a lenticular lens) located in front of the LCD 25. In this device, the pixels Rr, Gr, and Br for the right eye are driven by the image signals for the right eye and the pixels RL, for the left eye are driven by the image signals for the left eye. Pixel groups 27R, 27G, and 27B are formed from pairs of GL and BL. Two parallax signals are generated by aligning the two left and right cameras to photograph the object at left and right viewpoints corresponding to the viewer's left and right eyes. 20 (a) and 20 (b) show the R and L signals generated for the same color. As shown in Fig. 20 (c), compression and combining means of these signals are used to form a single stereoscopic image by rearranging the R and L signals into alternating patterns (R, L, R, L, ...). Since the combined left and right signals must be compressed in half, as shown in Fig. 20 (d), the actual signal forming a single stereoscopic image is composed of pairs of image data of different colors for the left eye and the right eye. In this example, the display can be switched between 2D and 3D by switching slit positions in the parallax barrier.

Reference Patent 1: Japanese Unexamined Patent Application Publication H10-271533

Reference Patent 2: Japanese Unexamined Patent Application Publication H11-72745

However, the 3D scanning method described in Figs. 13 and 14 uses a large volume of data and requires many operations, so that it takes a long time to generate a 3D object. In addition, the device is complex and expensive. The device also requires special expensive software to apply various effects and animations to the 3D object.

Accordingly, one object of the present invention is a method of collecting photographic data with a plurality of cameras having a simple configuration and located around an outer surface of an object to generate accurate 3D objects based on a plurality of different images in a short time. Instead, it provides a 3D image generation and display system using a 3D scanner using a scanning table method of rotating an object. The 3D image generation and display system creates web-specific 3D objects using commercial software that edits and processes the major parts of the 3D object in order to quickly draw and display 3D scenes in a web browser.

In the stereoscopic imaging apparatuses shown in FIGS. 15 to 20, a system for switching between 2D and 3D displays when using the same liquid crystal panel by the movement of the lenticular lens shown in FIG. 15 and the parallax barrier shown in FIG. 19 are fixed. If the types of display devices are different as in the system described above, the types of the left and right parallax signals are different. Likewise, the format of the left and right parallax signals is different for all display devices having different formats, such as various display panels, CRT screens, 3D shutter glasses, and projectors.

The format of the left and right parallax signals also differs when using different video signal formats, such as the VGA method or the method of interlacing video signals.

Also, in the prior art described with reference to FIGS. 15 to 20, left and right parallax signals are generated from two photographic images taken by two digital cameras positioned corresponding to the left and right eyes. However, when the format of the original image data is different, for example, when directly generating left and right parallax data using left and right parallax signals generated by photographing object and character images generated by computer graphics modeling or the like. The method and format are different for generating left and right parallax data.

Accordingly, another object of the present invention is to provide left and right parallax signals that can create a common platform that can match the differences between the various input images and the signal formats of these input images as well as the differences of the various display devices. The present invention provides a 3D image generation and display system for generating 3D images to be synthesized and displaying these 3D images in a web browser.

In order to achieve these objects, the 3D image generation and display system according to claim 1 consists of a computer system for generating 3D objects used for displaying 3D images in a web browser, which 3D image generation and display system comprises: 3D object generating means for generating three-dimensional (3D) images from a plurality of different images and / or computer graphics modeling and generating a 3D object from such images having texture and attribute data; 3D description file output means for converting a format of the 3D object generated by the 3D object generating means and outputting the data as a 3D description file for displaying 3D images according to a 3D graphics description language; Edit and process the 3D object to extract 3D objects from the 3D description file, set various property data, introduce animations, etc., and then convert the resulting data back into a 3D description file or a temporary file for property setting. 3D object processing means for outputting; Texture processing means for extracting textures from the 3D description file, editing and processing the textures to reduce the number of colors, and the like, and outputting the result data again as a 3D description file or texture file; 3D effects that extract 3D objects from the 3D description files, process the 3D objects, assign various effects such as lighting and material properties, and output the result data again as 3D description files or temporary files for effect assignment. Application means; Extracts the various elements necessary for rendering 3D images in a web browser from the 3D description file, texture file, temporary file for setting properties, and temporary file for assigning effects, and compressed textures for display in a web browser. And web 3D object generating means for generating various web-based 3D objects having attribute data; Behavior data generating means for generating behavior data for displaying 3D scenes in a web browser with animation by controlling properties of the 3D objects and assigning effects; And stereoscopic images generated from a plurality of combined images assigned to a predetermined parallax based on the web 3D objects and the behavior data generated, edited and processed by the above described means. And executable file generating means for generating an executable file comprising a web page and one or a plurality of programs including scripts, plug-ins, and applets for drawing and displaying 3D scenes.

In addition, the 3D object generating means according to claim 2 comprises a turntable on which the object is placed on it and rotates horizontally or vertically; A digital camera for capturing images of an object mounted on the turntable to generate digital image files of the images; Turntable control means for rotating the turntable to predetermined positions; Photographing means for photographing an object set at predetermined positions by the turntable control means using the digital camera; Continuous image control means for continuously generating a plurality of image files using the turntable control means and the photographing means; And a 3D object generating 3D images based on the plurality of image files generated by the continuous image control means and generating a 3D object having texture and attribute data from the 3D images to display the images in 3D. Coupling means.

In addition, the 3D object generating means according to claim 3 uses the silhouette data from a plurality of images obtained by a single camera along the entire outer surface of the object when the object rotates on the turntable to form a three-dimensional shape of the object. 3D images are generated according to an estimated silhouette method.

Furthermore, the 3D object generating means according to claim 4 further comprises images obtained by a camera, images made by computer graphics modeling, images scanned by a scanner, hand drawn images, and image data stored in another storage medium. A single 3D image is generated as a composite scene by combining various image data including the back.

Further, the executable file generating means according to claim 5 further comprises left and right parallax for drawing and displaying stereoscopic images according to a rendering function based on right eye images and left eye images to which parallax from a predetermined camera position is assigned. Automatic left and right parallax data generating means for automatically generating data; Disparity data compression means for compressing each of left and right disparity data generated by the automatic left and right disparity data generating means; Disparity data combining means for combining the compressed left and right disparity data; Disparity data expansion means for dividing the combined left and right disparity data into left and right portions and extending the data to be displayed on a stereoscopic image display device; And display data converting means for converting the data to be displayed according to a viewing angle (aspect ratio) of the stereoscopic image display apparatus.

The automatic left and right parallax data generating means according to claim 6 automatically generates left and right parallax data corresponding to the 3D image generated by the 3D object generating means based on the virtual camera set by the rendering function. do.

Further, the parallax data compression means according to claim 7 compresses pixel data of left and right parallax data by omitting pixels.

In addition, the stereoscopic display device according to claim 8 uses at least one of a CRT screen, a liquid crystal panel, a plasma display, an electroluminescence (EL) display, and a projector.

In addition, the stereoscopic display device according to claim 9 displays stereoscopic images that can be seen when the viewer wears the stereoscopic glasses or displays stereoscopic images that can be viewed when the viewer is not wearing the glasses.

The 3D image generation and display system of the present invention may constitute a computer system for generating 3D objects displayed on a 3D display. This 3D image generation and display system employs a scanning table system that collects images along the entire outer surface of an object with a single camera while the turntable rotates and uses a scanning table system to model an object placed on a scanning table. Has In addition, the present 3D image generation and display system takes advantage of commercially available generic software to facilitate the creation of high quality 3D objects.

The 3D image generation and display system may display animation in a web browser by installing a specific plug-in to draw and display 3D scenes in a web browser or by creating applets that effectively display 3D images in a web browser.

The 3D image generation and display system may configure a display program capable of displaying stereoscopic images according to LR parallax image data, types of 3D images that do not "protrude" to viewers, and general 2D images on the same display device. have.

1 is a flowchart showing steps in a process performed by a 3D image generation and display system according to a first embodiment of the present invention.

FIG. 2 is a schematic diagram showing 3D object generation means of the 3D image generation and display system of FIG. 1; FIG.

3 is a flowchart illustrating a process from creation of 3D objects to drawing and displaying 3D scenes in a web browser.

4 is a perspective view of a printer as an example of a 3D object.

5 is a schematic diagram illustrating a 3D image generation and display system according to a second embodiment of the present invention.

6 is a schematic diagram illustrating the 3D image generator of FIG. 5 with 2-n cameras.

7 illustrates a method of setting camera positions of the renderer of FIG.

8 illustrates a process for generating simple stereoscopic images.

9 is a block diagram showing LR data processing circuit in a VGA display.

FIG. 10 is a diagram for explaining operations of enlarging, reducing, moving, and rotating a 3D image. FIG.

Fig. 11 is a block diagram showing LR data processing circuit of a video signal type display.

12 is a schematic diagram illustrating a three-dimensional display system using projectors.

Figure 13 (a) is a schematic diagram of a conventional 3D modeling display device.

Fig. 13B is a diagram explaining the generation of slit images.

14 is a block diagram showing a conventional 3D modeling apparatus using a plurality of cameras.

15 is a schematic diagram of a conventional 3D video signal generator.

16 illustrates LR data of the signal generator of FIG. 15. FIG.

FIG. 17 illustrates a process of compressing LR data of FIG. 16; FIG.

18 illustrates a method of displaying LR data on the display device of FIG.

19 is a schematic diagram of another conventional stereoscopic image display apparatus.

FIG. 20 shows LR data displayed on the display device of FIG. 19; FIG.

Hereinafter, with reference to the accompanying drawings will be described a preferred embodiment of the present invention.

1 is a flowchart showing the steps of a process performed by the 3D image generation and display system according to the first embodiment of the present invention.

In the process of FIG. 1 described below, a 3D scanner described below is used to form a plurality of 3D images. 3D objects are created from 3D images and converted into a standard virtual reality modeling language (VRML) (language for describing 3D graphics). The converted 3D object in the output VRML file goes through various processes to create a web 3D object and a program file that can be executed in a web browser.

First, the 3D scanner of the 3D object generating means using the digital camera captures images of the real object and secures 24 3D images obtained at angles varying by 15 degrees, for example (S101). The 3D object generating means generates a 3D object from these images, and the 3D description file output means temporarily converts the 3D object into a VRML format (S102). 3D ScanWare or a similar program can be used to create 3D images, create 3D objects, and create VRML files.

The 3D object generated by the 3D authoring software (for example, the following software) is extracted from the VRML file and subjected to various editing and processing by the 3D object processing means (S103). A commercial product "3ds max" or other software analyzes the essential areas of 3D objects to extract texture images, set attributes for animation processes, create various 3D objects, and Used to set animation features. After editing and processing, the 3D object is again stored as a 3D description file in VRML format, or temporarily as a temporary file that sets properties in a storage device or memory area. In animation settings, the number or time of frames may be set in a key frame animation that moves an object provided to the 3D scene at a specific number of frame intervals. The animation may be generated using techniques such as path animation and character studio to create a path, such as a Nurbs CV curve along which the object moves. Using texture processing means, a user can extract texture images applied to various objects in a VRML file, edit texture images for color or texture mapping, reduce the number of colors, Modify the location / state and area to be applied, or perform other processes, and store the result data as a texture file (S104). Texture editing and processing can be done using commercial image editing software such as Photoshop.

In order to process 3D objects and apply various effects such as lighting and material properties, the 3D effect application means is used to extract various 3D objects from the VRML file and extract the extracted objects with 3ds max or similar software and various plug-ins. Use in combination. The result data is restored as a 3D description file in VRML format or stored as a temporary file for applying effects (S105). So far, 3D objects have gone through processes for reducing file size as processes for displaying as animations on web pages and as pre-processes such as texture imaging processes. The following steps cover processes for reducing and optimizing object size and file size to actually display objects in a web browser.

The web 3D object generating means extracts 3D objects, texture images, properties, animation data, and other rendering elements from temporary files and VRML created during editing and processing, and displays web 3D images on the web. Create objects (S106). At the same time, the behavior data generating means generates the behavior data as a scenario for displaying the web 3D object as an animation (S107). Finally, the executable file generating means is in the form of a plug-in software for a web browser or a program combined with a Java applet and JavaScript for drawing and displaying images in a web browser based on the data for displaying 3D images. Create an executable file (S108).

By using the VRML format supported by most 3D software programs, it is possible to edit and process 3D images using a multipurpose commercial software program. The system may also optimize the images for use on the web based on the transfer rate of the communication line, or when displaying the images on a web browser of an individual computer, edit and process the images as appropriate for the display environment. Image rendering can then be adjusted to achieve an effective and optimal quality in the display environment.

FIG. 2 is a schematic diagram illustrating the 3D object generating means and the 3D object generating display system described above with reference to FIG. 1.

The web 3D object generating means of FIG. 2 supports the object 33 (corresponding to the "object" in the claims, referred to herein as "object" or "real object") and scanning the object 33. A turntable 31 which rotates 360 degrees; Background panel 32 of a single primary color such as green or blue; A digital camera 34 such as a CCD; Lighting 35; A table rotation controller 36 for rotating the turntable 31 through servo control; Photographing means 37 for adjusting and calibrating the digital camera 34 and illumination 35 to perform gamma correction and other image processing of the image data and to capture images of the object 33. ; And continuous image generating means 38 for controlling the angles of sampling and table rotation and collecting images at predetermined angles. These elements constitute a 3D modeling apparatus that uses a single camera and a scanning table to generate a series of images viewed from a plurality of angles. Here, the images are modified as needed using commercial editing software such as AutoCAD and STL. The 3D object combining means 39 extracts a silhouette from a series of images, generates 3D images by using a silhouette method for estimating 3D shapes, etc. and generates 3D object data.

Next, methods of operating the 3D image generation and display system will be described.

In the silhouette method, the camera is calibrated by, for example, calculating correlations between the world coordinate system, the camera coordinate system, and the image coordinate system. In order to process the images in software, the points in the image coordinate system are converted into points in the standard coordinate system.

After the calibration is completed, the continuous image generating means 38 cooperates with the table rotation controller 36 to adjust the rotation angle of the turntable for the predetermined number of scannings (e.g., every 10 degrees, or 72 for 36 scans). Image scanning every 5 degrees for conference scanning) during which the imaging means 37 captures images of the object 33. By obtaining the background difference, the silhouette data of the object 33 is obtained from the captured images, and the background difference refers to the difference between the background panel 32 images obtained from the previous camera image and the current camera image. The silhouette image of the object is derived from the camera parameters obtained from the background difference and the calibration. 3D modeling is then performed on the silhouette image, for example, by placing a cube with a recursive octal tree structure in three-dimensional space and determining the intersections in the silhouette of the object.

FIG. 3 is a flowchart presenting a more specific / specific example—according to the steps of the 3D image conversion process shown in FIG. 1 so that the steps shown in FIG. 1 may be better / deepened and described. The process of FIG. 3 is implemented by a Java Applet that can display 3D images in a web browser without installing a plug-in for a viewer such as Live 3D, for example. In this example, all the data needed to display interactive 3D scenes is provided on a web server. 3D scenes are displayed when the server is accessed from a web browser running on a client computer. Normally, after 3D objects are created, 3ds max and the like are used to modify the motion, camera, lighting, material properties, etc. in the created 3D objects. However, in this preferred embodiment, the 3D objects or the entire scene are first converted to VRML format (S202).

The resulting VRML file is input to the 3DA system (S203) (where 3DA refers to 3D images displayed as animations on a web browser using a Java applet, and the entire system including authoring software for web-linked editing and processing) Called a 3DA system). The 3D scene is customized according to the user's wishes, and data for rendering an image with the 3DA applet is provided (S205) to draw and display the 3D scene in the web browser. All 3D scene data are compressed at once and stored as compressed 3DA files (S206). The 3DA system generates a toolbar file and an HTML file for interactive tasks, which read the toolbar file into a web browser so that the toolbar file is executed and the 3D scenes are displayed within the web browser. S207).

The new web page (HTML document) contains an applet tag for invoking the 3DA applet. JavaScript code for accessing the 3DA applet may be added to the HTML document in order to improve operations and interactivity (S209). All files necessary to display the 3D scene generated as described above are transmitted to the web server. These files are web pages (HTML documents) that process applet tags for invoking 3DA applets, toolbar files for interactive operations, texture image files, 3DA scene files, and optionally for drawing and displaying 3D scenes. 3DA applet is included (S210).

If the web browser subsequently connects to the web server and requests the 3DA applet, the web browser downloads the 3DA applet from the web server and executes the applet (S211). Once the 3DA applet has been executed, the applet displays a 3D scene from which the user can execute interactive tasks, and the web browser can continuously display the 3D scene independently of the web server (S212).

In the process described so far, after converting 3D objects to web-based VRML, a 3DA Java applet file is created, and the web browser downloads the 3DA file and the 3DA applet. However, rather than creating a 3DA file, it is of course possible to install a plug-in for a viewer such as Live 3D (product name) and process the VRML 3D description file directly. With the 3D image generation and display system of the present preferred embodiment, the company can easily create a web site using three-dimensional, moving displays of products for e-commerce and the like.

As an example of an e-commerce product, the following description deals with the disclosure of a commerce website for printers such as shown in FIG.

First, the company's product, printer 60, as an object 33 is placed and turned on the turntable shown in FIG. 2, during which the imaging means 37 captures images at predetermined sampling angles. The continuous image generating means 38 sets the number of images to be sampled to capture 36 images, assuming that the sampling angle is 10 degrees (360 degrees / 10 degrees = 36). The 3D object combining means 39 calculates the background difference between the background panel 32 and the camera position, which has been previously photographed, and extracts the image data of each of the 36 images of the printer generated by the continuous image generating means 38. The coordinates are converted to standard coordinates by coordinate transformation between world coordinates, camera coordinates, and image positions. A silhouette method for extracting the contours of the object is used to model the printer's appearance and to create the printer's 3D object. This object is temporarily output as a VRML file. At this time, all 3D images to be displayed on the web including the back work screen, the left and right views, the top and bottom views, the front work screen, and the like are generated.

Then, as described in FIG. 1, the 3D object processing means, the texture processing means, and the 3D effect applying means extract the generated 3D image data from the VRML file, analyze corresponding portions of the data, generate 3D objects, and Apply various attributes, perform animation processes, and apply various effects and other processes such as surface shaping and lighting, for example, through color, material, and texture mapping properties. The resulting data is stored as texture files, temporary files for attributes, temporary files for effects, and so on. Then, the behavior data generation means generates data necessary for movement in all 3D description files used at the printer web site. In particular, the behavior data generating means generates a file or the like for animation of the actual operation screen in the setting guide or the like.

By installing a plug-in for a viewer such as Live 3D in a web browser, the generated 3D scene data can be displayed in a web browser. It is also possible to use a method of processing 3D scene data only within a web browser without using a viewer. In this case, as described above, the 3DA file for the Java applet is downloaded to the web browser in order to draw and display the 3D scene data extracted from the VRML file.

When viewing the generated web site displaying a 3D image of a printer, a user can display animated sequences in 3D by clicking items in a setting guide menu displayed in a browser by manipulating a mouse. This animation can describe a series of operations to remove the cover 62 and install the USB connector 66 by turning the button 63 on the cover 62 of the printer 60.

When the user clicks "install cartridge" in the menu, the entire printer will rotate and a 3D animated continuous screen (not shown) showing the front will be played. The top cover 61 of the printer 60 opens, and the cartridge holder in the printer 60 moves to the center position. The black ink cartridge and the color ink cartridge are inserted into the cartridge holder, and the top cover 61 is closed.

Further, when the user clicks on the "maintenance screen", all plastic covers are removed to display a 3D image (not shown) that exposes the internal structures of the printer. In this way, the user can clearly see in three dimensions the spatial relationships between the driver module, the scanning structure, the ink cartridge, etc., to facilitate maintenance operations.

By displaying the operation windows in 3D animation in this manner, the user can browse the products in the same way as when actually operating the printer in a retail store.

Although the above detailed description is a simple example of viewing printer operations, the 3D image generation and display system can be used for other uses, such as wearing clothes. For example, the 3D image generation and display system may allow a user to try on clothes from a female clothing store. The user clicks on the clothes the model is wearing; Change the size and color of clothes; Looking at the modeled clothes from the front, back, and sides; Modify the shape, size, and color of the buttons; You can also order the clothes by e-mail. For example, various items such as sculptures or other art and everyday items at auction may be displayed as more realistic three-dimensional images than two-dimensional images.

Next, a second embodiment of the present invention will be described with reference to the accompanying drawings.

5 is a schematic diagram illustrating a 3D image generation and display system according to a second embodiment of the present invention. The second embodiment further extends the 3D image generation and display system to display 3D images generated and displayed on the web page of the first embodiment as stereoscopic images using other 3D display devices.

The 3D image generation and display system of FIG. 5 includes the same turntable-type 3D object generator 71 as the 3D object generating means of the first embodiment shown in FIG. This 3D object generator 71 creates a 3D image by combining images of an object obtained with a single camera while the object is rotated on the turntable. The 3D image generation and display system of the second embodiment also includes a multi-camera 3D object generator 72. Unlike the turntable-type 3D object generator 71, the 3D object generator 72 is not limited to a specific number of n cameras around a stationary object from two stereoscopic cameras corresponding to the positions of the left and right eyes. 3D objects are generated by arranging a plurality of cameras up to a larger number of cameras. The 3D image generation and display system also includes a computer graphics modeling 3D object generator 73 for generating 3D objects while performing computer graphics modeling through a graphics interface of a program such as 3ds max. The 3D object generator 73 is a computer graphics modeler that combines scenes with computer graphics, photos, and the like.

YAPPA 3D after performing the processes of S103-S107 described in FIG. 1 of the first embodiment to temporarily store 3D objects created by the 3D object generators 71-73 as general-purpose VRML files. 3D scene data is extracted from VRML files using a web authoring tool such as Studio. Edit and process 3D objects and textures; Add animations; Apply, set and process other effects such as cameras and lighting effects; Authoring software is then used to generate web 3D objects and their behavioral data for drawing and displaying interactive 3D images within a web browser. An example of generating web 3D files has been described in S202-S210 of FIG. 3.

The means 75-79 are some of the executable file generating means used in S108 of FIG. 1 which applies left and right parallax data to display stereoscopic images. The renderer 75 applies rendering functions to generate left and right parallax images (LR data) necessary to display stereoscopic images. LR data compression / combining means 76 compresses the LR data generated by the renderer 75, reorders the data in the combining process and stores the data in the display frame buffer. The LR data separation / expansion means 77 separates and expands left and right data when displaying LR data. The data converting means 78 constituted by a down converter or the like is an angle of view (aspect ratio, aspect ratio, etc.) for displaying stereoscopic images so that the LR data can be compatible with various 3D display devices. Adjust it. The stereoscopic display means 79 is based on LR data and displays stereoscopic images using various display devices such as, for example, liquid crystal panels, CRT screens, plasma displays, electroluminescence (EL) displays, or projector shutter type display glasses. Various display formats such as, for example, the conventional VGA format used in personal computer displays and the like and the video formats used in televisions.

Next, methods of operating the 3D image generation and display system according to the second embodiment will be described.

First, the 3D object creation process performed by the 3D object generators 71-73 will be briefly described. The 3D object generator 71 is the same as the 3D object generating means described in FIG. The object 33 to form the 3D image is placed on the turntable 31. Table rotation controller while digital camera 34 and illumination 35 are adjusted to take sample pictures by photographing means 37 as a background, for example against a solid screen (background panel 32) such as a blue screen. 36 adjusts the rotations of the turntable 31. The continuous image generating means 38 then performs a process of combining the sampled images. Based on the resulting composite image, the 3D object combining means 39 extracts the silhouettes (contours) of the object using a silhouette method or the like to estimate the three-dimensional shape of the object and the 3D object. Create This method is performed using the following formula, for example.

(1)

(One)

Coordinate system transformation (calibration) is performed using the standard coordinates Sp of the point P and the camera coordinates Pfp to convert the three-dimensional coordinates at the vertices of the 3D images to the standard coordinate system [x, y, z, r, g, b]. Convert to Model the resulting coordinates using various programs. The 3D data generated from this process is stored in an image database (not shown).

The 3D object generator 72 is a system that captures images of an object by placing a plurality of cameras around the object. For example, as shown in FIG. 6, six cameras (first to sixth cameras) are positioned around the object. The control computer obtains photo data from the cameras via USB hubs and recreates 3D images of the object on the first and second projectors in real time. The present 3D object generator 72 is not limited to six cameras, and may capture images via any number of cameras. The system generates 3D objects in a standard coordinate system from a plurality of superimposed photographs obtained from such cameras and falls under the category of image-based rendering (IBR). Thus, the configuration and process of the present system is significantly more complicated than in the case of the 3D object generator 71. As in the case of the 3D object generator 71, the generated data is stored in a database.

The 3D object generator 73 first of all disposes the "up", "left", "right", "front", "perspective", and "camera" into four views in a split view port window. 3ds max and YAPPA 3D Studio, which assign to each, establish grids that correspond to the vertices of graphics in the display screen, and model images using data stored in various objects, shapes, and other libraries. Focus on computer graphics modeling using the same modeling software. Such modeling programs may combine computer graphics data with image data or photos generated by 3D object generators 71 and 72. This combination can be easily accomplished by adjusting the camera's viewing angle, aspect ratio, for the images rendered within the bitmap of the photo data and the computer graphics data.

The camera (virtual camera) may be created at any point for setting or modifying the perspective of the combined scene. For example, to change the camera position (user's point of view) set to the front as the initial setting to the position moved 30 degrees to the left or the right, the camera angle and position using [X, Y, Z, w] By setting the coordinates of, the composite video scene can be displayed at a position where the scene is moved 30 degrees from the front. In addition, the virtual cameras that can be created include a free camera that can rotate and move freely to any point and a target camera that can rotate around an object. If the user wants to change the perspective of the composite video scene or the like, the user can do so by setting new properties. With lens functions, etc., users can quickly change their perspective by selecting or switching from a group of about ten virtual lenses from WIDE to TELE with the touch of a button. The lighting settings can be changed in the same way with various functions that can be applied to the rendered images. All generated data is stored in the database.

Next, a process of generating left and right parallax images with the renderer and LR data (parallax images) generating means 75 will be described. LR data of parallax signals corresponding to left and right eyes can be easily obtained using the camera positioning function of the modeling software programs. In this case a specific example of calculating camera positions of the left and right eyes is described below with reference to FIG. 7. As shown in Fig. 7 (a), the coordinates of the position of each camera are represented by a vector perpendicular to the modeled object (mobile phone in this example). Here, the coordinate of the position of the camera is O; The focal direction of the camera is to the vector OT; The vector OU is set in a direction perpendicular to the vector OT as the upward direction of the camera. In order to make a stereoscopic display with the positions of the left and right eyes, the positions of the left and right eyes L, R are calculated according to the following equation (2). Is the angle of inclination of the left and right eyes, and d is the distance to the convergence point P for zero parallax between the left and right eyes.

(2)

(2)

Where (0 <d, 0 ≦ θ <180).

The method of calculating the positions described above is not limited to this method, and may be any calculation method capable of achieving the same effects. For example, since the preferred camera position is in front, the coordinates [X, Y, Z, w] can be directly input using the method of considering the camera (virtual camera) position described above.

After setting the positions of the eyes (camera positions) found by the methods described above in the camera function, in order to obtain a left and right parallax image for stereoscopic display, the user can select " Renderer "to convert and render a 3D scene as a two-dimensional image.

LR data may be generated for photo images obtained by the 3D object generators 71 and 72 without being limited to use with composite image scenes. By setting the coordinates [X, Y, Z, w] for camera positions (virtual cameras) corresponding to the positions of the left and right eyes, photographic images can be rendered, for left and right parallax images Image data of the object obtained along the entire outer surface is stored to secure the LR data. It is also possible to generate LR data from image data obtained around the entire outer surface of the stored object in the same way for 3D objects derived from computer graphics images modeled from the 3D object generator 73 and the like. LR data can be easily generated by rendering various composite scenes.

In the actual rendering process, the coordinates of each vertex of the polygons in the world coordinate system are converted to a two-dimensional screen coordinate system. Thus, a 3D / 2D transformation is performed by inversely transforming Equation 1 used to transform camera coordinates into three-dimensional coordinates. In addition to the calculation of camera positions, a calculation of the shadows (brightness) due to the virtual light shining from the light source is required. For example, the light source data Cnr, Cng, and Cnb that cause the colors of the material Mr, Mg, and Mb can be calculated using the conversion matrix equation 3 below.

(3)

(3)

Where Cnr, Cng, Cnb, Pnr, Png, and Pnb represent the nth vertex.

LR data for left and right parallax images obtained through this rendering process is automatically generated by calculating coordinates of shadows and camera positions based on the light source data. Various filtering processes may be performed at the same time but will be omitted in this description. In the display device, an up / down converter or the like converts the image data into bit data and adjusts the aspect ratio before displaying the image.

Next, as another example of the present invention, a method of automatically generating simple LR data will be described. 8 is a diagram illustrating a method of generating simple left and right parallax images. As shown in the example of FIG. 8, LR data of the letter “A” was generated for the left eye. If the object is symmetrical, the parallax image of the right eye can be generated as a mirror image of the LR data of the left eye by simply inverting the LR data of the left eye. This inversion can be calculated using Equation 4 below.

(4)

(4)

Where X is the X coordinate, Y is the Y coordinate, and X 'and Y' are the new coordinates on the mirror image. Rx and Ry are equal to -1. This simple process is practical enough when there is little change in the image data and can greatly reduce memory consumption and processing time.

Next, an example of displaying actual 3D images on various display devices using the LR data obtained in the above process will be described.

For simplicity, the case where LR data is input to the conventional display device shown in FIG. 19 to display 3D images will be dealt with. The display device shown in Fig. 19 is a liquid crystal panel (LCD) used in a personal computer or the like and uses a VGA display system using a sequential display technique. 9 is a block diagram illustrating a parallax video signal processing circuit. When the automatically generated LR data according to the present invention is supplied to this type of display device, the LR data of both the left and right parallax images shown in FIGS. 20 (a) and 20 (b) is compressed by the compressor / combiner 80. ) Is entered. Compressor / combiner 80 reorders the image data with alternating R and L data, as shown in FIG. 20 (c), and omits pixels, as shown in FIG. 20 (d). Compress the image in half. The resulting LR composite signal is input to the separator 81. The separator 81 performs the same process in reverse, as shown in Fig. 20 (c), and separates the R and L rows to rearrange the image data. This data is decompressed and expanded by expanders 82 and 83 and fed to display drivers to adjust the aspect ratio and the like. The drivers display the L signal to be seen by the left eye only and the R signal to be seen by the right eye only to achieve stereoscopic display. Since pixels omitted during compression are lost and cannot be reproduced, the image data is adjusted using interpolation or the like. This data can be used for displays of notebook personal computers, liquid crystal panels, direct-view game consoles, and the like. The signal format of the LR data in these cases is not particularly limited.

Web 3D authoring tools such as YAPPA 23D Studio are configured to convert image data to LR data according to a Java applet process. By attaching a toolbar file to one of the Java applets and downloading the data (3d scene data, Java applets, and HTML files) from the web server to the web browser via the network, the operation buttons as shown in FIG. It can be displayed on the screen of the browser. By selecting the button, the user can operate a stereoscopic image (in this case, a car) displayed on a web browser to zoom in or out, move or rotate the image. The detailed process of operations such as zoom in, zoom out, move, rotate, etc. is represented in the transformation matrix. For example, the movement can be represented by the following equation (5). Other operations can be similarly represented.

(5)

(5)

Where X 'and Y' are new coordinates, X and Y are circular coordinates, and Dx and Dy are distances moved in the horizontal and vertical directions, respectively.

Next, an example of displaying images on an interlaced display such as a television screen will be described. Various converters are commercially available as display means in personal computers and the like for converting image data into conventional TV and video images. This example uses such a converter to display stereoscopic images in a web browser. The construction and operation of the converter itself will not be described.

The following example uses a liquid crystal panel (or CRT screen, etc.) as shown in FIG. 19 to reproduce video signals. A parallax barrier or a lenticular sheet for displaying stereoscopic images is mounted on the front of the display device. The display process will be described using the block diagram of FIG. 11 showing the signal processing circuit of parallax images. The LR data of the left and right disparity images as shown in FIGS. 20A and 20B generated according to the automatic generation method of the present invention are input to the compressors 90 and 91, respectively. Compressors 90 and 91 compress the images by omitting one pixel over one pixel in the video signal. The combiner 92 combines and compresses the left and right LR data as shown in Figs. 20 (c) and 20 (d). The video signal composed of this combined LR data is transmitted to a receiver or recorded and played back on a storage medium such as a DVD. The separator 93 performs the same operation in reverse as shown in Figs. 20 (c) and 20 (d) to separate the combined LR data into left and right signals. The expanders 94 and 95 extend the left and right image data in the original format as shown in Figs. 20A and 20B. Stereoscopic images can be displayed on a display such as that shown in FIG. 19 because the display data is arranged in alternating left video data and right video data in the order of R, G and B over the horizontal scanning lines. For example, the R (red) signal is aligned like "R0 (for left) R0 (for right), R2 (for left) R2 (for right), R4 (for left) R4 (for right) ..." do. The G (green) signal is aligned as "G0 (left) G0 (right), G2 (left) G2 (right), ...". The B (blue) signal is aligned as B0 (left) B0 (right), B2 (left) B2 (right) ... ". Also, the LR data for parallax video signals is classified into even and odd areas. By simultaneously processing the two, three-dimensional display can be performed in the same manner using shutter glasses having liquid crystal shutters or the like as a display device.

Subsequently, explanations will be provided for displaying stereoscopic images on a projector used for presentations, home theater, and the like.

12 shows a projector screen 101 with optical treatment (such as application of a silver metal coating) to its surface; Projectors 106 and 107 positioned in front of the projector screen 101; And polarizing filters 108 and 109 positioned in front of each of the projectors 106 and 107, respectively. Each element of the home theater is controlled by the controller 103. When the projector 106 is provided for the right eye and the projector 107 for the left eye, the filter 109 is of the type that polarizes light vertically, while the filter 108 polarizes light horizontally. It is the type to do. The type of projector is a Meridian lossless packing (MLP) liquid crystal projector using a digital micromirror device (DMD). The home theater is based on the 3D image recorder 104 (of course, the device may generate images through modeling) supporting DVD or other media, and the present invention based on the 3D image data input from the 3D image recorder 104. It may also include a left and right parallax image generator 105 to automatically generate LR data with the display drivers of. The aspect ratio of the LR data generated by the left and right parallax image generator 105 is adjusted by a down converter or the like and supplied to the left and right projectors 106 and 107. Projectors 106 and 107 project images through polarization filters 108 and 109 that polarize the images horizontally and vertically, respectively. The spectator wears polarized glasses 102 with a vertical polarization filter in the right eye and a horizontal polarization filter in the left eye. Therefore, when viewing the image projected on the projector screen 101, the images projected by the projector 106 can only be seen by the right eye and the images projected by the projector 107 can only be seen by the left eye. The viewer can watch stereoscopic images.

By using a web browser for displaying 3D images in this manner, only an electronic device with a browser is required, no special 3D image display device is needed, and 3D images can be supported by various electronic devices. . The invention is also more user in that it does not require the use of different stereoscopic display software, such as a stereo driver, for each type of various hardware such as personal computers, televisions, game consoles, liquid crystal panel displays, shutter glasses, and projectors. -Friendly.

Claims (9)

3D object generating means for generating three-dimensional (3D) images from a plurality of different images and / or computer graphics modeling and generating a 3D object from such images having texture and attribute data; 3D description file output means for converting a format of the 3D object generated by the 3D object generating means and outputting the data as a 3D description file for displaying 3D images according to a 3D graphics description language; Edit and process the 3D object to extract 3D objects from the 3D description file, set various property data, introduce animations, etc., and then convert the resulting data back into a 3D description file or a temporary file for property setting. 3D object processing means for outputting; Texture processing means for extracting textures from the 3D description file, editing and processing the textures to reduce the number of colors, and the like, and outputting the result data again as a 3D description file or texture file; 3D effects that extract 3D objects from the 3D description files, process the 3D objects, assign various effects such as lighting and material properties, and output the result data again as 3D description files or temporary files for effect assignment. Application means; Extracts the various elements needed to render 3D images in a web browser from the 3D description file, texture file, temporary file for setting properties, and temporary file for assigning effects, and compresses textures and properties for display in a web browser. Web 3D object generating means for generating various web-based 3D objects having data; Behavior data generating means for generating behavior data for displaying 3D scenes in a web browser with animation by controlling properties of the 3D objects and assigning effects; And In the web browser with stereoscopic images generated from a plurality of combined images assigned with a predetermined parallax based on the web 3D objects and the behavior data generated, edited and processed by the above described means. Executable file generating means for generating an executable file comprising a web page and one or more programs including scripts, plug-ins, and applets for drawing and displaying 3D scenes 3D image generation and display system comprising a computer system for generating 3D objects used for displaying 3D images in a web browser. The method of claim 1, The 3D object generating means is: A turntable on which the object is mounted and rotated horizontally or vertically; A digital camera for capturing images of an object mounted on the turntable to generate digital image files of the images; Turntable control means for rotating the turntable to predetermined positions; Photographing means for photographing an object set at predetermined positions by the turntable control means using the digital camera; Continuous image control means for continuously generating a plurality of image files using the turntable control means and the photographing means; And 3D object combining means for generating 3D images based on the plurality of image files generated by the continuous image control means and generating a 3D object having texture and attribute data from the images for displaying the images in 3D 3D image generation and display system comprising a. The method of claim 2, The 3D object generating means is a silhouette method for estimating a three-dimensional shape of the object using silhouette data from a plurality of images obtained by a single camera along the entire outer surface of the object when the object is rotated by the turntable ( 3D image generation and display system, characterized in that for generating 3D images according to the silhouette method). The method of claim 1, The 3D object generating means includes various images including images obtained by a camera, images made by computer graphics modeling, images scanned by a scanner, hand drawn images, image data stored in another storage medium, and the like. 3D image generation and display system, comprising combining data to produce a single 3D image as a composite scene. The method of claim 1, The executable file generating means is: Automatic left and right parallax data that automatically generates left and right parallax data for drawing and displaying stereoscopic images according to a rendering function based on right eye images and left eye images to which parallax from a predetermined camera position is assigned Generating means; Disparity data compression means for compressing each of left and right disparity data generated by the automatic left and right disparity data generating means; Disparity data combining means for combining the compressed left and right disparity data; Disparity data expansion means for dividing the combined left and right disparity data into left and right portions and extending the data to be displayed on a stereoscopic image display device; And Display data conversion means for converting the data to be displayed according to the viewing angle (image ratio) of the stereoscopic image display apparatus 3D image generation and display system comprising a. The method of claim 5, The automatic left and right parallax data generating means automatically generates left and right parallax data corresponding to the 3D image generated by the 3D object generating means based on the virtual camera set by the rendering function. Image generation and display system. The method of claim 5, And the disparity data compression means compresses the pixel data of the left and right disparity data by omitting the pixels. The method of claim 5, The stereoscopic display device uses at least one of a CRT screen, a liquid crystal panel, a plasma display, an electroluminescence (EL) display, and a projector. The method of claim 5, The stereoscopic display device is a 3D image generation and display system, characterized in that for displaying the stereoscopic images that can be seen when the viewer wearing the three-dimensional glasses, or the stereoscopic images that can be seen when the viewer wearing glasses.

Images