TWI871926B - Lensless Mixed Reality Interactive System - Google Patents

Abstract

The present invention relates to a lens-less mixed reality interactive system, the main structure of which includes a transparent display that can see through the background scene behind, and a control system electrically connected to the transparent display, the control system includes: an image database, an interactive guidance module, a scene setting module, an image setting module, an image selection module, and an interactive control module. When setting up the control system, the interactive guidance module asks chain questions, and the answers provided by the user can become the setting parameters of the scene setting module, image setting module, and image selection module, respectively generating information such as background interaction conditions, image interaction areas, and dynamic images to be used, so that the interactive control module can form a mixed reality of dynamic images and background scene interactions on the transparent display without the need for lenses, head-mounted devices, or high-end hardware equipment.

Description

Lensless mixed reality interactive system

The present invention provides a lens-less mixed reality interactive system that can realize mixed reality on a transparent display without wearing a head-mounted device, using image capture technology, or being based on high-end hardware.

According to the report, Mixed Reality (MR) is an interactive technology that integrates virtual reality (VR) and augmented reality (AR) elements. Mixed reality provides a comprehensive environment that allows elements of the virtual and real worlds to exist and interact at the same time, creating a new fusion experience.

VR requires a helmet or glasses or other head-mounted devices to present images. In addition, the head-mounted devices are large and heavy, and users need to be completely immersed in the virtual world when using them, which makes the use process inconvenient and easy to cause dizziness and discomfort. As a result, the application areas are often limited to games and movies. AR is different from VR. AR does not completely isolate users from reality, but integrates virtual content into the real world, usually through the device. The camera is equipped to present the real image on the screen, and then after the real image recognition action, the virtual information captured by the camera is combined with the real image on the screen. However, this requires very powerful computing and display capabilities, so high-end hardware and firmware are needed to support a good experience. In particular, the application of AR may involve the collection and processing of data in the real world, which is easy to cause privacy issues, such as the use of personal location data.

As for transparent display, although its principle is similar to that of general display, its selling point lies in its unique appearance and interactive characteristics, making it an attractive technology in various application scenarios. However, for example, the Republic of China invention patent I669633 "Interactive method and system of mixed reality" combines MR with transparent display, but does not solve the above problems.

Therefore, how to solve the above-mentioned problems and deficiencies in usage is the direction that the applicant of this invention and related manufacturers engaged in this industry are eager to study and improve.

Therefore, in view of the above-mentioned deficiencies, the applicant of this invention has collected relevant information, evaluated and considered various aspects, and based on many years of experience in this industry, after continuous trials and modifications, finally designed this kind of lensless mixed reality interactive system that can realize mixed reality on a transparent display without wearing a head-mounted device, without using image capture technology, and without being based on high-end hardware equipment.

The main purpose of this invention is to utilize the design of the interactive guidance module to guide the user's answer content in a question-and-answer manner, and to integrate the above information to form a mixed reality with interactive dynamic images and background scenes on a transparent display.

Another main purpose of the present invention is to directly use the control system to define the background scene behind the transparent display, so as to avoid the use of image capture technology and solve the privacy problem, and to use the fixed background scene as the interactive target of the dynamic image, so as to avoid the need to identify a large number of images and solve the problem of high equipment costs.

To achieve the above-mentioned purpose, the main structure of the present invention includes: a transparent display combined with a fixed medium to allow the user to see through to the background scene behind, and a control system electrically connected to it, the control system includes an image database, an interactive guidance module, a scene setting module, an image setting module, an image selection module, and an interactive control module. The image database is used to store multiple dynamic images, the interactive guidance module displays multiple linked questions in sequence on the transparent display for the user to answer, the scene setting module information is linked to the interactive guidance module to set the background interaction conditions according to the user's answer content, the image setting module information is linked to the interactive guidance module to set the image interaction area according to the user's answer content, the image selection module information is linked to the interactive guidance module and the image database to set the dynamic image displayed on the transparent display according to the user's answer content, and the interactive control module is information linked to the scene setting module, the image setting module and the image selection module to form a mixed reality of the interaction between the dynamic image and the background scene according to the background interaction conditions, the image interaction area and the dynamic image.

When the user wants to generate mixed reality on the transparent display, although the user can see through to the background scene behind, without using the camera, the transparent display does not have any information about the background scene. At this time, the control system uses the interactive guidance module to ask a number of chain questions, and according to the user's answers, it converts the information into background interaction conditions, image interaction areas, and dynamic images to be used. In this way, the control system does not need to identify the background scene, but only needs to use the scene setting module to set the background scene. The group understands the background interaction conditions such as the distance, relative size, and spatial position relationship of the background scene. Through the image setting module, the dynamic image is set to what range and which part of the background scene it wants to interact with (image interaction area), and through the image selection module, it is decided which dynamic image to use for interaction. Finally, the interactive control module integrates the above information to allow the dynamic image to interact with the background scene on the transparent display, so as to realize a low-threshold, lens-free, and high-degree-of-interaction mixed reality.

The above-mentioned technology can overcome the existing problems in mixed reality interaction methods and systems, which still require the use of image capture and image recognition technologies, resulting in higher thresholds for computing and display capabilities, and the inability to solve privacy issues, thereby achieving the practical progress of the above-mentioned advantages.

1: Transparent display

2: Control system

21: Image database

22: Interactive guidance module

23: Scenery setting module

24: Image settings module

241:Sensing unit

25: Image selection module

26: Interactive control module

271: Scenery Database

272: Scenery comparison module

28: Azimuth detection unit

29: Reset unit

291: Linked switch

3: Image import section

31: Port

32: Connector

A: Image interaction area

B: Background scenery

C: Background interaction conditions

I: Moving images

Q: Chain questions

W: Fixed media

The first figure is a structural block diagram of the first preferred embodiment of the present invention.

The second figure is a schematic diagram of the background scene of the first preferred embodiment of the present invention.

The third figure is a schematic diagram of the interactive guidance of the first preferred embodiment of the present invention (I).

Figure 4 is a schematic diagram of the interactive guidance of the first preferred embodiment of the present invention (II).

Figure 5 is a schematic diagram of mixed reality interaction of the first preferred embodiment of the present invention.

Figure 6 is an action block flow chart of the first preferred embodiment of the present invention.

Figure 7 is a structural block diagram of the second preferred embodiment of the present invention.

Figure 8 is a schematic diagram of the interactive guidance of the second preferred embodiment of the present invention.

Figure 9 is a schematic diagram of the implementation of the second preferred embodiment of the present invention.

Figure 10 is a schematic diagram of the implementation of the third preferred embodiment of the present invention.

Figure 11 is a structural block diagram of the fourth preferred embodiment of the present invention.

Figure 12 is a schematic diagram of the fourth preferred embodiment of the present invention.

Figure 13 is a schematic diagram of the fifth preferred embodiment of the present invention.

In order to achieve the above-mentioned purpose and effect, the technical means and structure adopted by the present invention are described in detail in the following figure for the preferred embodiment of the present invention, and its features and functions are explained in detail for a complete understanding.

Please refer to the first and second figures, which are the structural block diagram and background scene schematic diagram of the first preferred embodiment of the present invention. It can be clearly seen from the figure that the present invention includes:

A transparent display 1 is combined with a fixed medium W to allow the user to see through to the background scene B behind;

A control system 2 is electrically connected to the transparent display 1 and includes:

An image database 21 for storing multiple dynamic images I;

An interactive guidance module 22 sequentially displays a plurality of linked questions Q on the transparent display 1 for the user to answer;

A scene setting module 23 is informationally linked to the interactive guidance module 22 to set background interactive conditions C according to the user's answer content;

An image setting module 24 is informationally connected to the interactive guidance module 22 to set the image interactive area A according to the user's answer content. The image setting module 24 has a sensing unit 241 for sensing the user drawing a line outline on the transparent display 1 according to the background scene B, and transmits it to the interactive control module 26 described below as the image interactive area A;

An image selection module 25 is informationally linked to the interactive guidance module 22 and the image database 21 to set the dynamic image I displayed on the transparent display 1 according to the user's answer content; and

An interactive control module 26 is informationally connected to the scene setting module 23, the image setting module 24, and the image selection module 25 to form a mixed reality in which the dynamic image I interacts with the background scene B according to the background interaction condition C, the image interaction area A, and the dynamic image I.

The transparent display 1 refers to a display that uses a transparent substrate, a transparent conductive material, and a simplified pixel structure to achieve a transparent effect; the fixed medium W is a window, a partition, or a billboard; the control system 2 is an application program or control software, and this embodiment takes the control software installed in a processor or an IC as an example; the image database 21 is a memory, a hard disk, or a cloud storage space, etc. The storage medium, this embodiment takes the memory set on the PCB together with the processor or IC as an example; the interactive guidance module 22, the scene setting module 23, the image setting module 24, the image selection module 25 and the interactive control module 26 are plug-in software in the control system 2; the sensing unit 241 is combined with the touch sensing or suspension touch functions of the transparent display 1 to capture its sensing results. However, the corresponding types of the above components are only examples of the preferred embodiments. All types with the same functions are within the scope of the present invention and are not limited to the above examples.

Through the above description, the structure of this technology can be understood. According to the corresponding coordination of this structure, it is possible to achieve the advantages of mixed reality on the transparent display 1 without wearing a head-mounted device, without using image capture technology, and without being based on high-end hardware equipment. The detailed explanation will be described below.

Please refer to the first to sixth figures at the same time, which are the structural block diagrams and action block flow charts of the first preferred embodiment of the present invention. When the above components are assembled, it can be clearly seen from the second figure that the transparent display 1 of this case is combined with a fixed medium W such as a window. The purpose is to eliminate the situation where the transparent display 1 changes its position as much as possible. In this way, the user can see the background scene B from the front of the transparent display 1 to the back, which can become a static picture.

Although the user can see through the background scene B behind, the transparent display 1 will not receive any information about the background scene B without using image capture technology. When the user wants to generate a mixed reality on the transparent display 1, he can simply start from the interactive guidance module 22 of the control system 2. The interactive guidance module 22 will ask a plurality of chain questions Q in sequence and display the questions on the transparent display 1. The chain questions Q are designed multiple-choice questions or tracing questions, which can limit the direction of the user's answer content, so that the answer to each option can generate a corresponding parameter, and can be converted into parameters of information such as background interaction conditions C, image interaction area A, and dynamic image I to be used according to the user's answer content.

For example, the first question is "What is the background you see? (A) Window view; (B) Indoor view", if (A) is selected, the second question is a multiple-choice question "Where is the transparent display 1 installed? (A) Low, within the 5th floor; (B) Medium, 6th to 13th floor; (C) High, 14th to 24th floor; (D) Super high, above the 25th floor", the purpose of this question is to lock the viewing angle, for example, select (B). The third question is a tracing question, "Please continuously draw the outline of the building/scenery opposite you." The purpose of this question is to distinguish between the plane and the elevation, such as describing the roof and wall of an apartment and the wall of a building. The fourth question is a multiple-choice question, "How far away is the building/scenery just drawn from you? (A) Close, next street; (B) Average, two or three buildings away; (C) Far, four blocks away; (D) Very far, a distant building/scenery." The purpose of this question is to determine the distance and size ratio of the background scene B, such as If (B) is selected, the fifth question is a tracing question "Please continuously depict the boundaries of the interactive area of the dynamic image I you want", the purpose of this question is to limit the activity range of the dynamic image I, and the sixth question is a multiple choice question "How far is the interactive area just depicted from you? (A) Close, the next street; (B) Normal, two or three buildings away; (C) Far, four streets away; (D) Very far, a distant building/landscape", the purpose of this question is the distance and size ratio between the interactive area and the building/landscape. The above questions are only examples of preferred embodiments, and the question format, question content, and question quantity are not limited to this.

In this way, the control system 2 does not need to identify the background scene B, does not need to store a large amount of scene data, and does not need to perform complex text analysis. Instead, the control system 2 only needs to understand the background interaction conditions C such as the viewing angle, distance, relative size, and spatial position relationship of the background scene B from the answers in the chain questions Q (such as the first to fourth questions) through the scene setting module 23, and read the answers in the chain questions Q (such as the fifth question and the sixth figure) through the image setting module 24. When reading, the sensing unit 241 detects the user's drawing action to set the image setting module 24 to the image. The parameters of the interactive area A, i.e., the range and part of the background scene B in which the dynamic image I interacts, are set. The action of the image selection module 25 can determine which dynamic image I is to be used for the interactive part, and obtain the corresponding dynamic image I from the image database 21 according to the user's answer. In this embodiment, the dynamic image I includes options such as virtual characters, natural phenomena, or creatures. The selection method can be direct selection or asking the user's preference type through chain questions Q. This embodiment takes the selection of natural phenomena (such as snow) as an example. Finally, the interactive control module 26 integrates the above information so that the dynamic image I can interact with the background scene B on the transparent display 1. In this embodiment, the specific interactive result is that the dynamic image I with snowfall will fall on the entire screen of the transparent display 1, and form snow on the plane (such as the roof) distinguished by the background scene B, and partially float on the vertical surface (such as the outer wall of the building) distinguished by the background scene B. In this way, not only can the use of image capture technology be avoided to solve the privacy problem, but also because the fixed background scene B is used as the interactive target of the dynamic image I, it is possible to avoid identifying a large number of images and solve the problem of high equipment costs. In addition, the interactive mode of the dynamic image I can produce different effects according to different answers given to the interactive guidance module 22, thereby providing a mixed reality with a high degree of freedom experience of DIY interaction.

Please refer to Figures 7 to 9 at the same time, which are structural block diagrams to implementation schematic diagrams of the second preferred embodiment of the present invention. It can be clearly seen from the figures that this embodiment is similar to the above-mentioned embodiment, except that the control system 2 has a scene database 271 for storing static scene type data, and the control system 2 has a scene comparison module 272, which is an information link between the scene database 271 and the interactive guidance module 22 to filter out static scene type data that matches the background scene B according to the user's answer content. Although this case does not require the support of high-end hardware or firmware, if it has the support of high-end hardware or firmware, the functions of the scene database 271 and the scene comparison module 272 can be further used. A large amount of known static scene type data, such as special buildings, various flowers, plants, trees, and various furniture, can be pre-stored in the scene database 271. When the user draws the outline of the building/scene, the scene comparison module 272 is used for identification. When the static scene type data of the drawn background scene B is identified, the interactive guidance module 22 can raise other detailed questions based on the static scene type data, so that the dynamic image I can interact with it in different ways.

For example, the fixed medium W of this embodiment is a floor-to-ceiling window on a lower floor, the background scene B is a park and the grass and slide in the park, and the dynamic image I is a creature such as a cat. When the scene comparison module 272 compares the background scene B with static scene type data that matches the slide, the interactive guidance module 22 can further ask the user to describe the slide as an image interaction area A, so that the user can see the dynamic image I of the cat sliding down the slide of the background scene B from the transparent display 1.

Please refer to FIG. 10 at the same time, which is a schematic diagram of the third preferred embodiment of the present invention. It can be clearly seen from the figure that the present embodiment is similar to the above-mentioned embodiment, except that the control system 2 has an image importing unit 3 for information linking the image database 21. The image importing unit 3 of the present embodiment includes a connection port 31 on the outer side of the transparent display 1, and a control software or program code for processing the storage position and the action mode of the dynamic image I. (The figure takes the data stored in the connector 32 as an example). In this way, if the system manufacturer cooperates with a company, it will design the dynamic image I action mode of the virtual character belonging to the company. The user only needs to purchase or download the virtual character, and can use the image import unit 3 to add the dynamic image I to the image database 21, making the selection of dynamic images I more diverse, the DIY experience of dynamic images I more interesting, and the effect of mixed reality richer.

For example, the fixed medium W of this embodiment is exemplified by the glass of the partition wall, the background scene B is exemplified by the living room and the sofa and coffee table in the living room, and the dynamic image I is exemplified by a virtual character. If the image of the virtual character is lazy, then the scene of lying on the coffee table and sleeping may appear.

Please refer to FIG. 11 and FIG. 12 at the same time, which are the structural block diagram and the implementation schematic diagram of the fourth preferred embodiment of the present invention. It can be clearly seen from the figure that the present embodiment is similar to the above-mentioned embodiment, except that the control system 2 has an orientation detection unit 28, which stops the action of the interactive control module 26 when the transparent display 1 changes its orientation so that the background scene B in the user's line of sight changes. The orientation detection unit 28 of the present embodiment is a gyroscope. For example, the fixed medium W is a window that can be opened and closed freely. Therefore, the background scene B will be obviously different when the window is not closed. In order to avoid abnormal interaction between the dynamic image and the background scene B, when the orientation detection unit 28 determines that the orientation of the transparent display 1 has changed, it will force the subsequent actions of the interactive control module 26 to stop, or make the dynamic image perform special effects of leaving the image interaction area (such as waving goodbye, moving out of the screen, etc.) and end the interaction.

Please refer to FIG. 13, which is a schematic diagram of the fifth preferred embodiment of the present invention. It can be clearly seen from the figure that the present embodiment is similar to the above-mentioned embodiment, except that the control system 2 has a reset unit 29 for the user to operate and restart the interactive guidance module. The reset unit 29 can perform the restart operation through software operation or the operation of the hardware linkage switch 291. The present embodiment takes the hardware linkage switch 291 as an example. In addition, the scope of the fixed medium W is further explained here. The fixed medium W is the charge The control system 2 itself is in a non-mobile state when in use, such as a laptop on a table or a car window. Under this definition, the background scene B in the transparent display 1 may have both a fixed background and a variable background. For example, a laptop on a restaurant table has a fixed background of restaurant decoration and a variable background of restaurant staff. For example, the window of a moving train has a fixed background of adjacent rails, roadside guardrails, or simpler fixed backgrounds such as the sky and seascape, and fleeting variable backgrounds such as street lights and road trees.

In this way, the user only needs to avoid the dynamic background and set the fixed background, for example, set the sky as the image interaction area A, and select the eagle that continuously follows the flight as the dynamic image I, and the mixed reality setting can also be completed. As for the reset unit 29, the user can manually operate it and choose the time to update the mixed reality setting. For example, because the train runs from the sea line into the city, the fixed background changes from the sky to the adjacent railroad. The user can press the linkage switch 291 to drive the reset unit 29. In this way, the application scenario of the present invention is more free and the application venue is more extensive.

However, the above is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Therefore, all simple modifications and equivalent structural changes made by using the contents of the present invention's specification and drawings should be included in the patent scope of the present invention and should be stated.

Therefore, the key technical aspects of the lensless mixed reality interactive system of the present invention that can improve usage are:

First, by combining this case with a transparent display 1, the unique visual effect can enhance the overall sense of technology and consumer appeal.

Second, the design of the interactive guidance module 22 is used to shape the look and feel of the DIY dynamic image I, and the designed question content and restricted answer content can be used to generate effective parameters that can change the interactive mode to simplify the difficulty of DIY.

Third, by using the design of the scene setting module 23, the spatial sense of the background scene B in the picture can be roughly known, so that the system can know the background interaction conditions C such as the viewing angle, distance, relative size, and spatial position relationship of the background scene B, without using image capture technology or image recognition technology to grasp the background scene B, which can speed up the processing speed and avoid privacy issues caused by shooting the user's surrounding environment.

Fourth, by utilizing the design of the image setting module 24, the image interaction area A of the dynamic image I can be controlled, including the range and part of the background scene B to interact with, so that the system can directly set the action mode of the dynamic image I according to the image interaction area A. By performing actions of a customized range on the static background scene B, the hardware/firmware equipment requirements for real-time image interaction can be solved, which can not only reduce the equipment cost requirements, but also reduce the equipment size, making it easier to integrate into the fixed medium W.

Fifth, by utilizing the design of the scene database 271 and the scene comparison module 272, the interactive guidance module 22 can further raise other detailed questions based on the static scene type data, so that the dynamic image I can interact with it in different ways.

Sixth, by utilizing the design of the image import unit 3, the dynamic image I can be selectively added to the image database 21, making the selection of dynamic images I more diverse, the DIY experience of dynamic images I more interesting, and the mixed reality effect richer.

Seventh, the design of the position detection unit 28 can avoid abnormal interaction between the dynamic image I and the background scene B.

Eighth, by utilizing the design of the reset unit 29, the fixed medium W is not limited to buildings, and the image interaction area A is not limited to the completely unchanging background scenery B, thereby making the application scenarios of the present invention more flexible and the application venues more extensive.

In summary, the lensless mixed reality interactive system of this invention can truly achieve its efficacy and purpose when used. Therefore, this invention is truly an invention with excellent practicality. In order to meet the application requirements for invention patents, an application is filed in accordance with the law. I hope that the review committee will approve this invention as soon as possible to protect the applicant's hard work. If the review committee of the Jun Bureau has any doubts, please feel free to write to instruct. The applicant will do his best to cooperate and feel very grateful.

1: Transparent display

2: Control system

21: Image database

22: Interactive guidance module

23: Scenery setting module

24: Image settings module

241:Sensing unit

25: Image selection module

26: Interactive control module

A: Image interaction area

B: Background scenery

C: Background interaction conditions

I: Moving images

Q: Chain questions

W: Fixed media

Claims (9)

A lens-less mixed reality interactive system, which mainly includes: A transparent display is combined with a fixed medium to allow the user to see through to the background scenery behind; A control system is electrically connected to the transparent display and includes: An image database for storing multiple dynamic images; An interactive guidance module displays a plurality of linked questions in sequence on the transparent display for the user to answer; A scene setting module is informationally linked to the interactive guidance module to set background interaction conditions based on the user's answer content; An image setting module is informationally linked to the interactive guidance module to set the image interactive area according to the user's answer content; An image selection module is informationally linked to the interactive guidance module and the image database to set the dynamic image displayed on the transparent display according to the user's answer content; and An interactive control module is an information link to the scene setting module, the image setting module, and the image selection module to form a mixed reality in which the dynamic image interacts with the background scene according to the background interaction condition, the image interaction area, and the dynamic image. A lensless mixed reality interactive system as described in Item 1 of the patent application, wherein the fixed medium is one of a window, a partition, or a billboard. As described in item 1 of the patent application scope, the control system has a scene database for storing static scene type data. As described in item 3 of the patent application scope, the control system has a scene matching module, which is informationally linked to the scene database and the interactive guidance module to filter out static scene type data that matches the background scene according to the user's answer content. As described in the first item of the patent application scope, the lensless mixed reality interactive system, wherein the image setting module has a sensing unit for sensing the user drawing a line outline on the transparent display according to the background scene, and then transmitting it back to the interactive control module as the image interactive area. A lensless mixed reality interactive system as described in item 1 of the patent application, wherein the dynamic image is one of a virtual character, a natural phenomenon, or a living being. As described in item 1 of the patent application scope, the control system has a position detection unit, which stops the action of the interactive control module when the transparent display changes its position, causing the background scene to change. The lensless mixed reality interactive system as described in Item 1 of the patent application scope, wherein the control system has an image import unit that is informationally linked to the image database. The lensless mixed reality interactive system as described in item 1 of the patent application scope, wherein the control system has a reset unit for the user to operate and restart the interactive guidance module.

Images