CH711803B1 - Process of immersive interactions by virtual mirror. - Google Patents

Abstract

The invention relates to a method of immersive interactions and computer vision applied to a virtual mirror being implemented using a computer system comprising, a memory, a plurality of media sensors and an interaction space, comprising at least one sequence with the following steps: a) as a function of a media flow from said media sensors and for one or a plurality of users present in said interaction space, updating a user reference for each of said one or more said users; b) as a function of said media stream and of said user reference (s), synthesizing one or a plurality of mirror reflections necessary for a dialogue by visual contact; c) as a function of said media stream, composing an audio-visual rendering of the software applications executed on said virtual mirror including said one or more mirror reflections; d) as a function of said media stream, creating said dialogue by visual contact by displaying said rendering on said virtual mirror in order to interact with said user or users.

Description

TECHNICAL AREA

The invention relates to a method of immersive interactions and computer vision applied to a virtual mirror capable of creating a dialogue by visual contact with one or a plurality of users.

STATE OF THE ART

The invention of the mirror is attributed to the German chemist Justus von Liebig in the year 1835, the well-known manufacturing process consists of applying a metal film to a transparent glass surface. The principle of reflection of light is governed by the laws of physics and, therefore, physical mirrors have a set of strong limitations resulting from it, such as for example, the fact that the reflected image is inverted with respect to the reality which makes a reflected text difficult to read. Although the use of physical mirrors is still widespread, other so-called virtual mirror systems have emerged by combining a video camera, a computer processing system, and a display on a screen. More recently, new virtual mirror systems have made it possible to add augmented reality elements to offer people the possibility to try on, for example, a new hairstyle or new clothes. More recently still, some virtual mirrors offer functionalities and 3D representations.

[0003] 3D virtual mirror systems already exist in the state of the art.

[0004] Patent US2015120496 (A1) describes a clothing purchasing system which makes it possible to reconstruct a part of the human body in 3D and then to virtually try on a clothing. The system uses a virtual mirror and several shots. In one of the embodiments, the user can select the point of view of one of the shots and even an intermediate angle between several shots.

[0005] Patent US2014232816 describes a tele-presence system composed of several systems interconnected by a communication network. The semi-transparent mirror metaphor is used to allow participants to see each other and distant interlocutors as if they were in the same room, i.e., the image projected on the mirror is a 3D reconstruction of the remote environment. Eye contact between participants can thus be preserved and additional information can also be visualized.

[0006] The patent US2014226000 User interface and authentication for a virtual mirror (and also the patent US2014225977) describe a virtual mirror capable of detecting the presence of a user, of viewing his image in an adequate manner by resizing it with the main objective of recognizing the user among those registered in a database and this while imitating the behavior of a physical mirror.

[0007] Most of these known 3D virtual mirror solutions are therefore based on an extension of the functionalities of a virtual mirror by adding 3D vision, additional information, and in some cases additional elements of augmented reality which faithfully follow the movements of the person.

A major defect of the physical and virtual mirrors proposed and of the interactive systems in general is that they do not offer the possibility of establishing and reacting to visual contact with the interaction surface. This drawback has the consequence that these surfaces do not have the possibility of actively attracting the attention of a potential user or of interacting with him by visual contact. Additionally, when the system is used in spaces where multiple users can potentially interact, other users cannot see who the system is communicating with in particular.

[0009] As regards physical mirrors, these are governed by the laws of physics and are therefore limited by these constraints. They only offer the functionality of a mirror, while occupying an imposing surface.

The physical mirrors and the virtual mirrors described in the state of the art are not intended to establish active visual contact between a user and the mirror itself, therefore, they cannot validly attract the user. visual attention of a particular user, or associate it with immersive interaction when the user is standing nearby.

[0011] Other immersive interactions use headsets and other specific accessories and are therefore not suitable for communication spaces where the interaction is spontaneous and ephemeral. In addition, these systems do not allow each other to be seen or used by a plurality of users.

[0012] It would therefore be advantageous for virtual mirrors to offer immersive interactions and to be able to dialogue by visual contact with one or a plurality of users.

BRIEF SUMMARY OF THE INVENTION

An object of the present invention is to remedy the drawbacks of the prior art which have just been mentioned by carrying out a method of immersive interactions and computer vision applied to a virtual mirror capable of creating a dialogue by contact visual with one or a plurality of users. The invention achieves this object by providing a method according to the appended claim 1.

Computer vision technologies and in particular view synthesis algorithms make it possible to reconstruct in real time a new 3D visualization from a set of media sensors. This synthesis of the mirror reflection and the automatic tracking of the gazes of the users are exploited in the invention to visualize a mirror reflection adapted so as to stimulate visual contact between the virtual mirror and a particular user. This new faculty of communication is used throughout the duration of a user session. The virtual mirror, thanks to its 3D audio-visual rendering and in particular thanks to its interactions, plunges the user into an immersive universe.

[0015] According to an advantageous embodiment of the method of the invention, illustrated in FIG. 2, the mirror can capture the attention of a user by creating visual contact with the latter. Once this is acquired, the virtual mirror interface is personalized for the user and he can then consult his information displayed on the screen while maintaining visual contact with his mirror reflection. In particular, he has the freedom to move in front of the virtual mirror by interacting and keeping visual contact with his image reflected transparently with the applications of the virtual mirror.

The space required to implement the visual communication system is thus reduced compared to the double use of a physical mirror and of a visual communication system.

In addition, in accordance with another advantageous embodiment, illustrated in FIG. 3, the construction of the mirror reflection of a user can be personalized with respect to one or more references which are located in the space of 'interaction. In this case, the calculation of the angle and of the point of view used to construct the mirror reflection is carried out with respect to this reference or these references and with respect to the user himself. Typically, the user will be able, for example, to see himself from every angle by varying the viewing angle interactively from 0 ° to 360 ° by moving around a reference placed at the center of the media sensors.

BRIEF DESCRIPTION OF THE FIGURES

The invention will be explained in more detail with the aid of several exemplary embodiments presented by the following drawings: <tb> • <SEP> FIG. 1 represents a flow diagram illustrating an example of the method of the invention. <tb> • <SEP> Figure 2 illustrates an example of an embodiment of the initial phase of establishing two-way eye contact between the system and a particular user. <tb> • <SEP> FIG. 3 illustrates an example of an embodiment with the use of a reference in the interaction space.

DETAILED DESCRIPTION OF THE INVENTION

The characteristics described in claim 1 make it possible to establish an intuitive immersive communication and interactions with a user but also in the case where several users could potentially interact by visual contact with a virtual mirror.

[0020] In the method of the invention, an "s" should be added to man in the terminology "human-machine interaction" to emphasize that a plurality of users can interact.

[0021] The invention uses the mirror reflection of users to attract their attention and to invite them to interact with the virtual mirror. Throughout the session of use, the user maintains this visual contact by viewing his mirror reflection. A new communication link is thus created between the user and the virtual mirror, but also with nearby spectators. Indeed, the latter can perfectly identify and visualize between whom the communication is established, this thanks to the visual contact established between the user and his mirror reflection.

[0022] The following definitions make it possible to better describe the underlying concepts of the invention.

Definition: by "computer system" is meant a set of interconnected and powered electronic components which have among them, at least one processing unit (processor), inputs and outputs, memory for storing information and software for running applications. The computer systems are preferably distributed, that is to say, they are interconnected by a computer network and their software, which controls them, communicates with each other. Computer systems, sometimes called connected objects, come in a multitude of forms and appearances, such as: a computer, a smartphone, a watch. As on-board computer systems, they can also be integrated into the clothes we wear or into the environment around us, without necessarily being visible.

Definition: by the "geometric characteristics of the mirror reflection", we mean the angles of view (θr, ϕr) of azimuth and elevation and the coordinates of the point of view (xr, yr, zr) and the angle of view αr; in other words, the set of parameters that a virtual pan-tilt-zoom camera should have to capture the mirror reflection image from any point in 3D space.

Definition: by "visual characteristics" of an application is meant the construction of a 3D view of the graphic objects of the application with a shot characterized by the viewing angles (θa, ϕa), of azimuth and elevation and by the coordinates of the point of view (xa, ya, za) with a field angle αa.

Definition: by "graphic aspect of the reflection" mirror is meant all types of graphic visualizations, transformations, augmented reality additions, and digital filtering that can be applied to its graphic representation.

An example of adaptation of the graphic aspect of the mirror reflection is that which makes it possible to simulate the window of a store, when a person is standing and outside the store. The image of the reflection is then often against the light, the person can nevertheless see inside the store. By adapting the graphic aspect of the mirror reflection, the virtual mirror can therefore perfectly give the illusion to a user of being in front of a virtual storefront.

Another example of adaptation of the graphic aspect of the reflection makes it possible to communicate emotions between the reflection of a person and itself, for example when the reflection blushes or by a more complex process of visual transformations, such as that described in the publication (Dynamic 3D Avatar Creation from Hand-held Video Input, by Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly, ACM Transactions on Graphics (Proceedings of SIGGRAPH), 2015).

Definition: by "synthesis of the mirror reflection" is meant a processing chain which makes it possible to construct a new 3D visualization or a 2D view thereof with the desired geometric characteristics and this from the existing views which include, preferably the depth (cf. RGB-D Red, Green, Blue and Depth images). The depth defines the distance of the image voxels from the media sensor.

A multitude of examples of view synthesis algorithms exist, they indeed represent an important line of research in the field of computer vision and an abundant scientific literature exists on the subject. A taxonomy and an overview are described in the reference work: (Computer Vision: Algorithms and Applications by Richard Szeliski, Published by Springer, 2011)

Definition: By "visual contact" is meant the contact established when a user sees a graphic representation of his mirror reflection and when he has the perception that the reflection reacts according to his visual attention.

[0032] In the context of the invention, visual contact between a particular user and his reflection is used to allow several people to interact with the virtual mirror by identifying the pairs: user - reflection.

Definition: by "postponed interaction" is meant that which uses a device connected by a computer network and which communicates with the virtual mirror.

Definition: by "visual contact descriptor" is meant the set of data which contains the identifier of the user, the status of the type of visual contact in progress, and the history of the last visual contacts and of the last interactions with the virtual mirror.

This descriptor can be personalized with respect to particular users or adapted to particular applications; to take into account, for example, an interaction transferred to another device of the portable type or a connected watch. The treatment of visual contact can also be transferred to the device if it has the necessary sensors.

[0036] One or more users can be in visual contact with the virtual mirror. They are then associated with the virtual mirror by one or more descriptors, unlike other spectator users who are not in visual contact.

Definition: by "session of use" is meant the time during which a visual contact descriptor is assigned to a user, a session of use has four phases: the inactive phase, the initialization phase, the phase during session, and phase of termination.

A user session allows intra-session personalization of the virtual mirror as opposed to inter-session personalization which requires the implicit or explicit identification of a user.

Definition: by "interaction space" is meant the space where a user can interact with the virtual mirror; more specifically, where he can make eye contact.

Definition: by "user repository" is meant the set of data extracted from the media stream and which contains information relating to the location, position, and orientation of a user's gaze, such as for example: a identifier, the reference coordinates of the user's head position (xu, yu, zu), the angles of the orientation of the head (θt, ϕt), as well as the angles of his gaze (θu, ϕu ) azimuth and elevation.

Definition: by "media sensors" is meant the equipment which carries out the acquisition of the audio-visual signals necessary for the synthesis of the mirror reflection (s) and for the audio-visual rendering of the virtual mirror. These signals are also exploited in the interpretation of immersive interactions with the virtual mirror.

There is a wide variety of sensors which make it possible to carry out the invention with different qualities and performances. A classification of specific sensors for constructing views is detailed in Chapter 8 of the book (Computer Vision for Visual Effects, By Richard J. Radke, Published by Cambridge University Press) and the latest plenoptic camera technologies are described in the chapter 5, (Digital Representations of the Real World, by Christian Theobalt, Olga Sorkine-Hornung; Oliver Grau; Marcus A. Magnor, Published by CRC Press). The development of plenoptic cameras makes it possible in particular to directly acquire 4D information from a light flux (light-field camera). These cameras offer the advantage of facilitating the calculation of the stereo correspondence necessary for the synthesis of virtual views.

Other panoptic type sensors also make it possible to carry out the invention, such as, for example, that described and filed in the patent (cf. Omnidirectional sensor array system, WO 2012056437 A1).

Definition: by "media stream" is meant all the digital signals originating from the media sensors transported from them to the computing power which performs the vision processing and the interaction processing.

Definition: by "auto-visual content" is meant all of what a software application is likely to generate as output and what the user can perceive, such as for example: text, images, a media, music, a video game, and also sources of augmented reality.

Definition: by “audio-visual context” is meant the scene where the user is placed without including the voxels of the user or of the users.

Definition: by "reference in the interaction space" is meant any zone or point in the interaction space, on the mirror reflection, on an augmented reality representation, or on an image visualized on the virtual mirror.

This reference can also be a fictitious point or a point or an area identified by a marker.

In the context of the invention, the immersive interactions are also determined with respect to these references.

Definition: by "immersive interactions" is meant interactions where the user has the perception of being physically in the audio-visual context or the audio-visual content with which he interacts.

Definition: by "virtual mirror" is meant one or more 2D or 3D auto-stereoscopic type viewing screens. A virtual mirror can also be configured as an immersive room where the entire visual field of the user is recreated.

Definition: by "dialogue by visual contact" is meant an immersive interface which allows visual contact to communicate with a user or a plurality of users.

Definition: by "physical marker" is meant a mark visible to a user, as opposed to a fictitious point which is not visible to him.

[0054] Innumerable technologies and screen formats exist in the state of the art and make it possible to advantageously exploit the invention. Different visual renderings exist, for example auto-stereoscopic visual rendering and dynamic perspective rendering which is achieved by projecting a 3D representation in 2D dynamically according to the orientation of the user's gaze. Although rendering with this projected view does not allow the user to perceive a 3D relief, it still has an interesting advantage compared to a visualization by auto-stereoscopic screen: the visual quality is more constant compared to the displacements. of the user.

The method of immersive interactions and computer vision applied to a virtual mirror will be better understood with the help of the illustration of the flow diagram of the method of FIG. 1.

The method of immersive interactions and computer vision applied to a virtual mirror is broken down into several steps.

[0057] When a user has visual contact with the virtual mirror, this visual contact is characterized by a descriptor. When a user enters the presence of the interaction surface, he does not yet have a descriptor assigned by the surface. Performing the steps captures the user's attention by stimulating eye contact. When the user comes into visual contact with the virtual mirror and enters the media stream i, step a) assigns him a user repository. In step d), the virtual mirror creates the visual contact dialog (s) using, for example, an visual contact descriptor for each user. When a user session is initialized by sub-step d1), the user can then interact at will and in a personalized manner with the surface. On the other hand, when the user no longer wishes to interact with the virtual mirror, he implicitly loses the visual contact descriptor and the user session is terminated.

The user can also end the user session explicitly by a man-machine interaction to allow another user to communicate by visual contact with the virtual mirror.

In some cases, several users can interact in concert with the virtual mirror. For each of them, an eye contact descriptor is then created with an associated user session.

A set of heuristics are used to remove a visual contact descriptor from an inactive user during a user session.

A first heuristic simply detects whether the user leaves the interaction space, in which case, he implicitly gives up visual contact. The second heuristic determines the inactivity of the user, and finally, the third determines the inattention of the user according to the visual contact he maintains with the virtual mirror. The last two heuristics use information from eye contact history and interaction data to determine user inactivity. This inactivity or lack of attention is also communicated to the user by adapting the graphic aspect of the mirror reflection. Typically, if the user is not attentive, the intensity of the mirror reflection gradually fades as time passes. After some time has passed, the system proactively grabs the user's attention. In certain embodiments, the heuristics are advantageously replaced by methods by machine learning where the system learns by itself to better communicate by visual contact with the users.

To carry out the first step a) media sensors preferably arranged around the interaction space are calibrated and modeled, these procedures are illustrated in the following book: (Multiple View Geometry in Computer Vision - Second Edition, by Richard Hartley and Andrew Zisserman, publisher Cambridge University Press, 2004).

In step a), the digital signals of the media stream i are processed to extract the automatic tracking of the user repositories. On the other hand, all the man-machine interactions are dealt with in step d).

When an event is detected in step a), it is sent to the other steps b), c), and d) which reflect the implications in their processing.

The module for the automatic tracking of user repositories is produced by an algorithmic analysis well known in the state of the art of computer vision. A virtual mirror system was published in 1998 (cf. A Virtual Mirror Interface using Real-time Robust Face Tracking, by T. Darrell, G. Gordon, J. Woodfill, M. Harville In Proceedings of the Third International Conférence on Face and Gesture Recognition, April 14-16, 1998, Nara, Japan). This publication describes how to use stereoscopic vision to calculate depth, extracting the contour of the person from the audio-visual context, tracking the position of the head and, finally, the composition of video images with the possibility of '' bring some graphic transformations to the mirror reflection.

Since this publication, many improvements have been made to media sensors and to the view synthesis processing chain. These improvements are typically interrelated; the choice of a type of media sensor partly determines the necessary processing chain, p. ex. when the depth data is made available to the view synthesis algorithm by the use of an active sensor, that is to say, with a sensor which projects light into the scene to be viewed, it The different images must then be aligned with each other. To perform this task, specific algorithms exist, such as p. ex. the different versions of ICP: Iterative Closest Point algorithm (cf. A method for registration of 3-D schapes by PJ Besl and ND McKay In IEEE Transactions on Pattern Analysis and Machine Intelligence, 14 (2): 239-256, 1992. )

In certain cases, some of the specific algorithms are made available by the designers of the media sensors, this is the case in particular for the Kinect ™ media sensor from the Microsoft® company.

Recent approaches to automatic tracking algorithms typically make use of machine learning methods with large amounts of data, such as the Kinect ™ media sensor described in the publication (cf. Decision Forests for Classication, Regression , Density Estimation, Manifold Learning and Semi-Supervised Learning, by A. Criminisi, J. Shotton, and E. Konukoglu, Microsoft Research technical report TR-2011-114).

The object of step d) is also to analyze and process the interactions of users who are in visual contact with the surface. The man-machine interface is preferably multimodal, that is to say, the virtual mirror has several interaction modalities that the user can use separately or by combining several of them. This richness of interaction is advantageously exploited to make the virtual mirror as user-friendly as possible. These interactions typically involve the analysis of gestures, speech, tactile interfaces. By sub-step d2), these interactions can even be transferred to other connected devices, such as a cell phone, a watch, or a gamepad for example.

Step b) consists of the processing chain (cf. pipeline in English terminology) necessary for the synthesis of the mirror reflection from the media stream i. Step b) also applies the processing necessary to the software applications in order to build a virtual reality format at the output. This format is then used to produce an audio-visual composition in step c). Some software, such as video games, directly constructs representations in a suitable format of virtual reality.

The virtual view synthesis processing chain carries out the construction of the mirror reflection of one or more users and adapts the geometric characteristics of the reflection to attract and notify the user's gaze. As illustrated by the example of FIG. 2, the reflection is displayed so as to look at the user when the latter is not yet in visual contact with the surface. Otherwise, the user has a visual contact descriptor and the synthesis of the mirror reflection is then personalized for this user during this session of use.

To achieve this synthesis of views, a multitude of approaches are described in scientific publications, such as for example that published in the following work: (Image Processing, Analysis, and Machine Vision - Fourth Edition, by Milan Sonka, Vaclav Hlavac, Roger Boyle, published by Cengace Learning 2015). The processing chain is titled: 3D reconstruction from an unorganized set of 2D views, and Structure from Motion pipeline. This example describes the whole of a necessary processing chain as well as the type of format generated from a 3D point cloud converted by triangulation then bringing the respective colors to each triangle. The format of the representation is made compatible with virtual reality software.

In step b), the calculation of the synthesis of the mirror reflection, which uses the media stream i and the data from step a), is automatically optimized to use the fewest shots necessary to construct a mirror reflection turned towards the user. These optimizations are possible, because unlike a generic view synthesis application, the system knows a priori the shots to be composed together.

In certain cases of synthesis of views, occlusions appear, these are parts of the image which are hidden by viewers or by the user himself. During a session of use of the virtual mirror, shots are recorded beforehand and possibly make it possible to reconstruct the desired mirror reflection with a better visual rendering, even if, instantly, this operation would have been impossible to achieve due to the occlusion (s). .

In step d), when a visual contact descriptor is assigned to the user, some additional improvements are made possible by personalization and interactions with the user. In particular, the monitoring module of the user reference frame and the view synthesis can take account of an additional reference defined in the interaction space to calculate the point and the view angles of the mirror reflection. These interactions are typically immersive: the user enters in a way into immersion with his mirror reflection or with the visualization of data from a software application. An example of these immersive interactions is illustrated by the embodiment of Figure 3.

Sub-step b2) makes it possible to use the data recorded beforehand in the current session of use or possibly during those recorded previously. To perform backups between sessions (intersessions), the user must be identified explicitly or implicitly.

The result of the mirror reflection synthesis module typically generates a 3D representation which can be viewed and processed by virtual reality software or simply views projected in 2D.

In step c), the 3D audio-visual composition of the sources is carried out. This module essentially uses the following treatments: homographic transformations of the images to better integrate the different visual sources and the subtraction of the background scene which makes it possible to substitute another audio-visual context for that of the space of interaction.

When the virtual mirror has allocated a visual contact descriptor to the user and initiated a user session, the method of the invention has the advantage of offering personalized treatments to the user. These use the data collected during the current session and possibly those of previous sessions.

According to the embodiments, it is advantageous to identify the users in order to even better personalize the interactions and the data of the software applications. This identification of step d1) is carried out in two ways: either explicitly, by asking the user to state his identity or implicitly "on the fly", that is to say, without the user identifies itself, it is the virtual mirror that recognizes the user. In both cases, the objective is twofold, firstly, to improve the quality of mirror visualizations and, secondly, to improve interactions with software applications by taking advantage of the profile, usage statistics, and preferences of the user.

Step c) performs the composition of the audio-visual sources. The sources are from the virtual mirror itself, those from the mirror reflections and those from the applications that are executed on the virtual mirror by the user.

It can sometimes be advantageous to dynamically modify the visual perspective of the audio-visual contents with respect to a user repository in order to maximize the quality of his user experience: p. eg, to facilitate the reading of a document if the user moves around or to place the user in immersion in relation to the information with which he interacts. Step c) therefore performs the necessary transformations. These can be partly described by the metaphor of a fish tank which is used in the scientific literature to define this type of visualization: (cf. Fish tank virtual reality, by C. Ware, K. Arthur, and KS Booth. In INTERCHI'03, 1993).

Another commonly used composition consists in replacing the audio-visual context by another. For audio, the operation is performed by a 3D spatial rendering that achieves the desired objective. To replace the mirror reflection in another context, special algorithms are necessary. These are known in the state of the art (cf. Probabilistic fusion of stereo with color and contrast for bi-layer segmentation, by Kolomogorov et al., In IEEE transactions on Pattern Analysis and Machine Intelligence, 2006).

More complex compositions are defined by the English term environment matting to account for the light interactions between the different visual sources, (cf. Environment matting and compositing, by DE Zonker, DM Werner, B. Curless, and DH Salesin , In ACM SIGGRAPH 1999 Conference Proceedings)

Step b) and in particular sub-step b3) also adapts the graphic appearance of the mirror reflection. This function is advantageously used by the invention to communicate at all times with the user in order to maintain visual contact with him in relation to the interactions he performs. A multitude of possibilities make it possible to visualize the attention that the user pays to his mirror reflection or in relation to an application that the user runs on the virtual mirror. For example, when the user seeks contact with his reflection, it will be fully visualized, while if the user pays his visual attention to the results of a software application, the reflection becomes more transparent with, possibly even, display which is limited to outline only.

In certain cases, the 3D type audio rendering is used to reinforce the impression of immersion and thus to advantageously complete the visual rendering of the virtual mirror. This 3D audio rendering is decisive, if the audio-visual content of a software application is suitable, such as p. ex. for video games, 3D movies, music concerts, or if the creation of a particular user experience is desired. If several virtual mirrors are made available to a set of users and in the same space, a sound shower can be advantageously used to improve the experience of each user individually. A sound shower makes it possible to locate the audio broadcast in relation to a reduced space when this space is shared by several virtual mirror installations.

A person skilled in the art will understand that the scope of the invention is not limited to a type of media sensors or to particular internal data structures, nor even to a particular choice of vision algorithm by computer.

Profitable developments result from the independent claim for application contexts where the following uses can be validly offered: a physical mirror and a visual communication system. Beyond this potential, the invention establishes a new paradigm of man-machine interaction which makes it possible to attract the attention of a user by creating visual contact with his mirror reflection and this for the different phases of a session of use: initialization, the session itself, and termination.

Today, visual communication screens are popping up everywhere, in this context many new applications are made possible by the invention. There are places where the user will naturally find himself in the presence of two uses: that of a physical mirror and that of a communication system. These uses can be found within a private company such as for example in the entrance hall, in a hotel or in a private home.

In these application contexts, the advantages resulting from the invention are numerous: <tb> <SEP> A common advantage of the various embodiments is that they provide an intuitive and natural way to attract the attention of a particular person among others. The English term eye-catching defines this property; to achieve this objective, the virtual mirror visualizes the mirror reflection of the people present, unlike a physical mirror, the reflection follows the person with his gaze and invites him to enter into visual contact.

This mechanism establishes visual communication and makes it possible to capture the attention of a person in order then to establish a visual dialogue between him and the virtual mirror. In particular, after the initialization phase by eye-catching, the user session then makes use of visual contact and the associated interactions. When eye contact with the user is broken, the user session enters the termination phase and, again, eye contact communication is utilized.

Another advantage offered by immersive interactions is the improvement of the user experience; the system makes the best use of the capacities of human perception. Typically, the closer the expressiveness and dimensions of the virtual mirror are to human perceptual capabilities, the better the user experience; in fact, the virtual mirror can thus visualize and make the best use of the exchanges of information with the user. Beyond the dimension of the virtual mirror itself, immersive interactions bring an additional dimension - immersion - which further increases the perceptual space made available by the virtual mirror.

When the interactions are placed in a perceptual space close to human possibilities, the quality of visual communication is optimal. This advantage is illustrated here with the example of a search engine. Indeed, unlike current user interfaces where the results are visualized by a list of texts or by an array of images, the virtual mirror offers an immersive approach where the user “enters” in a way into the virtual space of the data. . The structures of the visualized data can be enriched compared to a simple textual list, the ordering of the relevant results can also fully benefit from the expressiveness offered by immersive interactions.

Another advantage which results from the invention is the saving of space achieved by offering the two uses, of the physical mirror and of a visual communication system on one and the same surface.

Another advantage of the virtual mirror is to offer a man-machine interface where several users can interact, while knowing very exactly who interacts with the virtual mirror thanks to the visualization of the mirror reflection of the interlocutor (s) with whom the virtual mirror communicates. The spectators thus participate in the new communication channel created by the visualization of eye contact.

In other embodiments, a single virtual mirror can be shared between several users. The surface can also be visually distributed over several separate screens.

Depending on the embodiment, the computing power necessary to calculate the mirror reflection can be made available locally or remotely in the cloud, typically. The same architecture can be envisaged for storing the data, which can be available locally or remotely, connected by a computer network.

Performing the calculations and storing data remotely by pooling computer resources makes it possible, for certain embodiments, to significantly reduce costs and makes the invention all the more commercially advantageous.

EXAMPLES OF EMBODIMENTS OF THE INVENTION

The first embodiment is where the users are located in places where they spend a few moments either to wait or to get information, this ephemeral mode of use is illustrated in FIG. 2. This mode illustrates the initiation of a dialogue by eye contact. A second embodiment is illustrated by FIG. 3, where the interactions are described and this in the context of a tattoo shop.

[0100] The first embodiment of FIG. 2 illustrates the initialization phase of a use session. The application context is that of a hotel lobby. In these places, the two uses of a physical mirror and that of an information system are frequently found. Today and in the absence of having an interactive virtual mirror, information on tourist and cultural activities is often communicated with paper leaflets.

[0101] The virtual mirror (210) comprises the surface itself (212), typically an auto-stereoscopic screen with an audio amplification system and speakers, a computer with local computing power (214 ) and a database (213) which contains the recorded data of hotel guests. The virtual mirror displays the reflection (211) of the user (201). Media sensors are installed (220) to acquire the media stream and to construct the 3D image of the user's mirror reflection so that the person can make visual contact with their mirror reflection (211).

[0102] At first, the user (201) walks in a given direction (202) without paying attention to the virtual mirror (210). When the user enters the interaction space (200), the user's reflection is visualized in step d) of the method of the invention, this reflection is constructed to look towards the user (201) to get his attention and therefore to make eye contact. The user (201) then looks at his reflection (211) by turning his head in the direction of the reflection. From this moment, the visual behavior of the surface changes and allows the user to interact in a personalized way. In step d) of the method, a visual contact descriptor is assigned to the user and a user session is initiated. With the approach of 3D mirror reflection reconstruction and unlike other types of mirrors, the user does not need to be positioned in front of the virtual mirror to see themselves from the front or to interact with it.

[0103] During a session of use with the virtual mirror, the user could typically request by voice interactions an Italian restaurant proposal for the evening. The audio-visual content of the virtual mirror is then enriched with different restaurant proposals, while displaying the user's mirror reflection (211) transparently and with a front view. The graphic aspect of the reflection is now adapted to allow easy reading of the visual content relating to restaurants, the outline of the user's face is displayed only. The transformation and filtering of the mirror reflection are carried out in step b) of the method of FIG. 1. Finally, when the user has read the information displayed and is about to leave the interaction space , the virtual mirror detects, in step d), the termination of the use session, the visual contact descriptor is then deleted. The method of the invention continues while waiting for another user to come into visual contact with the virtual mirror.

If the user so desires, during step d1), he can identify himself and register on the virtual mirror via his laptop by using an application on his laptop provided for this purpose. He can also benefit from a richer personalized interaction by sharing his personal data from his laptop with the virtual mirror.

[0105] In this embodiment, several people could potentially attend the interaction with the virtual mirror without necessarily taking part in it.

[0106] A second exemplary embodiment is that of a tattoo artist's shop, therefore the virtual mirror and its interaction space are illustrated in FIG. 3.

The immersive interactions of the user make it possible in particular to control the point of view and the parameters of the pan-tilt-zoom of the reflection of the user with the additional advantage of being able to frame the vision of the mirror reflection on a part. accurate of the human body.

The idea is to use the invention in an application context of augmented reality to make it possible to try on a tattoo before having it tattooed. A software application is also available on the virtual mirror to facilitate the search and discovery of a tattoo that corresponds to the wishes. The user can even, if necessary, recreate, modify, or personalize a watermark with the use of this software application.

[0109] The state of the art mentions the possibility of trying on clothes before buying them or hairstyles, undoubtedly the most sensitive application is that which concerns tattoos, it is difficult to go back one step. once the tattoo is tattooed. It will therefore be particularly advantageous to use the invention to visualize a tattoo. The invention makes it possible to visualize a tattoo on all parts of the body, such as, for example, on the lower back without having to twist. With augmented reality visualization, all geometric parameters of the tattoo can be adjusted, e.g. ex. size, orientation, position. The invention makes it possible to see certain parts of the body which are difficult and sometimes impossible to look at with another type of mirror.

[0110] The virtual mirror (310) is composed with the following elements: the surface which is typically an auto-stereoscopic screen of a human size with integrated speakers, a connection to a computer communication network (313). The purpose of this connection is to perform the majority of calculations and data storage in a shared IT infrastructure. The interaction space (300) is surrounded by media sensors (314), these are necessary to have a realistic reconstruction of the mirror reflection from all viewing angles. High visual performance systems typically include a sphere with a hundred media sensors.

The method of man-machine interactions and of computer vision applied to a virtual mirror is shown in the example of FIG. 1. The illustration of FIG. 3 shows the various elements which make up the use. of the virtual mirror and the interaction space. When visual contact is already established with the user (301), that is to say, the initialization phase of the user session has already been carried out, the customer of the shop now has control of the interactions immersive, unlike the tattoo artist (302) who is simply a spectator. Only the person (301) has visual contact with their reflection (311) and an associated visual contact descriptor by the virtual mirror.

[0112] The immersive interactions are determined with respect to a reference (320) located in the interaction space. The mirror reflection is constructed in step b) of the method of FIG. 1. The particularity of this embodiment of FIG. 3 and the sub-step b4), is that the geometric characteristics of the mirror reflection are determined by the reference frame user and with respect to a first reference (320) which is located in the interaction space. The mirror reflection is constructed relative to this reference as if the user were virtually replicated at the precise location of this reference. By moving around this reference, the user can see themselves from all possible angles while looking at their reflection on the virtual mirror. The user can therefore control the geometric characteristics of the mirror reflection by moving around this first reference. It can also control the elevation angle by looking down or up on the virtual mirror. This construction of the mirror reflection is illustrated in FIG. 3. It allows the user (301) to see himself in profile on the virtual mirror, while looking at his mirror reflection from the front. This characteristic of the invention is particularly advantageous because a physical mirror does not allow one to see oneself correctly in profile.

In the embodiment illustrated by FIG. 3, the user (301) therefore has the possibility of moving relative to the reference (320) and in the different directions of the arrows (303). These movements are reflected in real time on the virtual mirror by the construction of the mirror reflection.

[0114] In this case, the user (301) can move around the reference point to control the angle of view of the mirror reflection and thus he can vary the angle of view between 0 ° and 360 °. It can also move according to the arrows which go from left to right (303), by approaching the reference point, it enlarges the details of the image by zooming in, by moving away from the reference point, it zooms out. The geometric characteristics of the mirror reflection are therefore modified in real time by the movements of the user in the interaction space and by the orientation of the gaze he brings to his reflection.

[0115] The two concentric circles (321) and (322) define the limits within which the user can influence the zoom in and zoom out functions.

[0116] In this embodiment, visual contact between the user and the surface is therefore permanent. Immersive interactions are made up of movements and the orientation of the user's gaze.

In the context of a tattooist's shop, tattoos are preferably placed on the body and more rarely on the face. The user can personalize the eye contact with a second referral. This will be advantageously placed where the tattoo will be tattooed. In this way, the visual contact is now transferred from the face to the place where the tattoo will be placed, that is to say, on the second reference which is on the user himself.

In the embodiment of a tattooist's stall, this second reference is typically identified by a physical marker applied to the user's body. The mirror reflection is then constructed and personalized with respect to the two references simultaneously; the first being placed in the interaction space (320) to control, by the movements of the user, the point of view and the pan-tilt-zoom parameters of the mirror reflection and the second reference, that of the physical marker, allows to automatically postpone the cropping of the mirror reflection and, at the same time, the visual contact on the future tattoo.

[0119] Subsequently, this physical marker allows the tattoo artist to pinpoint the precise point where the tattoo will be positioned, first by augmented reality, then by the actual production of the tattoo.

[0120] With the two references, the user can now look at the tattoo placed on the lower back from all angles while moving around the first reference (320), with the mirror reflection which is framed on the height of the marker. physical.

In the example of this embodiment several problems arise. The first is that of the occlusions created by the presence of the tattoo artist (302) who is also present in the interaction space. The solution proposed by the invention is to record shots beforehand and use them to recompose the missing visual parts due to the occlusions in sub-step b2).

[0122] Depending on the situation and the data recorded, a complete infographic model of a person can thus be recreated, the person can view his model on the virtual mirror with the interactions captured from the interaction space. This model can also be viewed later by software capable of displaying files with a format specific to virtual reality, the model can then be published and viewed on a website.

[0123] In another use case, the person might wish to view the tattoo on someone other than themselves. This interaction is then carried out in sub-step b1) using the recordings of the media stream of another person, an infographic model is then created for this other person. The user present in the interaction space can view in real time the tattoo added by augmented reality on the infographic model of the other person while interacting in the interaction space, as illustrated in figure 3.

[0124] In some cases, the person preferred to view himself in an audio-visual context different from that of the tattooist's shop. These other contexts can either be stored beforehand or be created on the fly in order to then be composed with the voxels of the person, this visual composition corresponds to step c) of the method of the invention.

[0125] The visual composition will typically be adapted as a function of the light conditions in order to obtain a quality rendering.

[0126] In other cases, when the use of the mirror is used at the same time as the tattoo design software application. The composition of visual sources and the user's mirror reflection are viewed from different perspectives. This situation is the one used by default, but it is not always desired. To harmonize the geometric characteristics of the transformations that are necessary, the first typically consists of placing the image from a software application in the same 3D space as that of the mirror reflection with the coordinates that define the position and orientation as well as the size and the perspective of the embedded image. The objective is to harmonize the visual composition of the two uses: that of the mirror and that of the application.

[0127] In certain cases of use, the two views, that of the mirror reflection and that of the tattoo design application, are made integral in 3D space. The coordinates of the mirror reflection are then moved p. ex. on the left of the surface to create space on the right to view the tattoos offered by the design application. The 3D mirror reflection and the other 3D images are then grouped together in the same 3D graphic object. The immersive interactions of the virtual mirror then directly control the entire composition of these two visual sources.

In the example of the embodiment of a tattoo artist's stall, it is advantageous to apply a 3D transformation to the tattoo so that it perfectly matches the surface of the skin. The 3D infographic model of the person precisely makes it possible to perform this transformation with the help of the physical marker which identifies the precise place where the tattoo will be tattooed. The application then makes it possible to better compare the different tattoos with adequate transformations that make the tattoos realistic.

[0129] The illustrated embodiments show only a subset of the scope of the advantageous developments of the method of the invention.

Claims (10)

1. A method of immersive interactions and computer vision applied to a virtual mirror being implemented using a computer system comprising, a memory, a plurality of media sensors and an interaction space, comprising at least a sequence with the following steps: a) as a function of a media flow from said media sensors and for one or a plurality of users present in said interaction space, updating a user repository for each of said user or users; b) as a function of said media stream and of said user reference (s), synthesizing one or a plurality of mirror reflections necessary for a dialogue by visual contact; c) as a function of said media stream, composing an audio-visual rendering of the software applications executed on said virtual mirror including said one or more mirror reflections; d) as a function of said media stream, creating said dialogue by visual contact by displaying said rendering on said virtual mirror in order to interact with said user or users. 2. The method of immersive interactions according to claim 1, wherein step c) composes said rendering with an audio-visual context different from that of said interaction space. 3. A method of immersive interactions according to claim 2, wherein said audio-visual context is adapted as a function of said one or more user repositories. 4. Method of immersive interactions according to one of the preceding claims, in which step b) comprises a sub-step b1) which uses stored data of said media stream of a person different from said users to construct a said mirror reflection. . 5. Method of immersive interactions according to one of claims 1 to 3, in which step b) comprises a sub-step b2) which synthesizes said or said mirror reflections by using a plurality of shots of said stored media stream. in said memory. 6. Method of immersive interactions according to one of the preceding claims, in which step b) comprises a sub-step b3) which modifies a graphic aspect of said mirror reflection as a function of an application context of a. of said software applications executed on said virtual mirror. 7. Method of immersive interactions according to one of the preceding claims, in which step b) comprises a sub-step b4) which constructs one or a plurality of geometric characteristics of a said mirror reflection with respect to a said reference frame. user and with respect to at least one reference in said interaction space. 8. The method of immersive interactions according to claim 7, wherein said one or more geometric characteristics of a said mirror reflection are determined as a function of a physical marker. 9. Method of immersive interactions according to one of the preceding claims, in which step d) comprises a sub-step d1) which creates a user session personalized by an identification of said user. 10. Method of immersive interactions according to one of the preceding claims, in which step d) comprises a sub-step d2) which allows another device to connect and interact with said virtual mirror.