CN116601921A - Session Privacy for Third-Party Applications - Google Patents

Abstract

The present disclosure relates to techniques for cooperating in a multi-user communication session, comprising: at a first device, receiving a request to launch a first application in a multi-user communication session with a second device; in the multi-user communication session Presenting the environment of the first application program; receiving data from the first user through one or more processes of the multi-user communication session, the data including input data and first communication data; sending data to the first user through the one or more processes An application program provides the input data; provides the first communication data to the second device through the one or more processes; receives second communication data from the second device through the one or more processes; and through the one or more processes Processes present the second communication data in the environment of the first application, wherein the first application is restricted from accessing the first communication data and the second communication data.

Description

Session Privacy for Third-Party Applications

Background technique

This disclosure generally relates to multi-user environments. More specifically, but not by way of limitation, the present disclosure relates to techniques and systems for session privacy for third-party applications executing in the context of a multi-user communication environment, such as an extended reality (XR) environment (XRE).

Some devices are capable of generating and rendering XRE. XREs may include fully or partially simulated environments that humans perceive and/or interact with via electronic systems. In XRE, a subset or representation of the person's physical motion is tracked, and in response, one or more properties of one or more virtual objects simulated in XRE are adjusted in a manner consistent with at least one law of physics. Some XREs allow multiple users to interact with each other within the XRE. What is needed, however, is a technology to host applications in XRE while managing users' private information.

Description of drawings

Figure 1 shows, in block diagram form, a simplified system diagram according to one or more embodiments.

Figure 2 illustrates a diagram of an exemplary operating environment, according to one or more implementations.

3A and 3B are line drawings illustrating interfaces for bringing an application experience to a multi-user communication extended reality environment in accordance with aspects of the present disclosure.

4 is a diagram illustrating an application experience according to aspects of the present disclosure.

5 is a flow diagram illustrating techniques for cooperating in a multi-user communication session in accordance with aspects of the present disclosure.

6A-6B illustrate example systems for use in various augmented reality technologies.

Detailed ways

This disclosure relates to techniques for users to collaborate within an extended reality environment (XRE) to share application experiences. Certain applications can be experienced within XRE by multiple users. In some cases, it is desirable for certain applications to present an application environment in place of XRE. For example, gaming applications or multi-user painting applications can present their own environments in place of XRE, while retaining certain elements of XRE, such as avatars and voice communication between participants. In some cases, there may be privacy concerns about allowing an application to potentially have access to voice communications, visual representations of participants, or identifiable information about participants experiencing an application experience. Accordingly, it would be desirable to allow applications to present application environments in place of XRE, while preventing applications from accessing potentially sensitive information.

A person can interact with and/or perceive the physical environment or the physical world without the aid of electronic devices. The physical environment may include physical features, such as physical objects or surfaces. An example of a physical environment is a physical forest including physical plants and animals. Humans can directly perceive and/or interact with the physical environment through various means such as hearing, vision, taste, touch and smell. In contrast, a human can use an electronic device to interact with and/or perceive a fully or partially simulated extended reality (XR) environment. The XR environment may include mixed reality (MR) content, augmented reality (AR) content, virtual reality (VR) content, and the like. Using an XR system, a person's physical motion, or some representation thereof, can be tracked, and in response, characteristics of a virtual object simulated in an XR environment can be adjusted in a manner consistent with at least one law of physics. For example, the XR system can detect the movement of the user's head and adjust the graphical and auditory content presented to the user (similar to how such views and sounds change in the physical environment). As another example, the XR system may detect movement of an electronic device (e.g., mobile phone, tablet, laptop, etc.) presenting an XR environment and adjust the graphical and auditory content presented to the user (similar to such visual and How sound changes in the physical environment). In some cases, the XR system may adjust characteristics of the graphical content in response to other inputs such as indications of physical motion (eg, voice commands).

Many different types of electronic systems may enable a user to interact with and/or perceive an XR environment. A non-exclusive list of examples includes heads-up displays (HUDs), head-mounted systems, projection-based systems, windows or vehicle windshields with integrated display capabilities, displays formed as lenses placed over the user's eyes (e.g., contact lenses ), headsets/earphones, input systems with or without haptic feedback (eg, wearable or handheld controllers), speaker arrays, smartphones, tablet computers, and desktop/laptop computers. A head mounted system may have an opaque display and one or more speakers. Other head-mounted systems may be configured to accept opaque external displays (eg, smartphones). A head-mounted system may include one or more image sensors for capturing images or video of the physical environment, and/or one or more microphones for capturing audio of the physical environment. A head mounted system may have a transparent or translucent display instead of an opaque display. Transparent or translucent displays may have a medium through which light is directed to the user's eyes. Displays can utilize various display technologies such as uLEDs, OLEDs, LEDs, liquid crystal on silicon, laser scanning light sources, digital light projection, or combinations thereof. Optical waveguides, optical reflectors, holographic media, optical combiners and combinations thereof, or other similar technologies can be used for the media. In some implementations, transparent or translucent displays can be selectively controlled to become opaque. Projection-based systems may utilize retinal projection technology that projects graphic images onto the user's retina. Projection systems can also project virtual objects into the physical environment (eg, as holograms or onto physical surfaces).

For purposes of this disclosure, an XR communication (XRC) session refers to an XR multi-user communication session, such as an XRE in which two or more devices are participating.

For the purposes of this disclosure, a local XRC device refers to the current device in the XRC session described or controlled by the described user.

For the purposes of this disclosure, a co-located XRC device refers to two devices that share a physical environment and an XRC session such that users of the co-located devices experience the same physical objects and XR objects.

For the purposes of this disclosure, a remote XRC device refers to an auxiliary device located in a separate physical environment from the current local XRC device. In one or more implementations, the remote XRC device may be a participant in an XRC session.

For the purposes of this disclosure, a shared virtual element refers to an XR object that is visible to or otherwise experienced by participants in an XR session.

For purposes of this disclosure, an XRC Computing Environment (XRCE) refers to a computing environment or container capable of hosting an XRC session of an application. XRCE enables applications to run within an XRC session. In some cases, XRCE may enable users of an XRC session to interact with hosted applications within the XRC session.

For purposes of this disclosure, an XRCE instance refers to the XRCE of the current device described or controlled by the described user. An XRCE instance may allow users to participate in XRC sessions and run applications in XRC sessions.

For the purposes of this disclosure, a second XRCE instance refers to an XRCE of an auxiliary device in an XRC session other than the local XRCE instance or an XRCE controlled by a second user. The second XRCE instance can be remote or co-located.

For the purposes of this disclosure, an XRCE application refers to an application capable of executing within the context of XRCE.

For the purposes of this disclosure, a second XRCE application refers to an XRCE application of an auxiliary device in an XRC session other than the native XRCE application or an XRCE application controlled by a second user. The second XRCE application can be remote or co-located.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the concepts disclosed. As part of this description, some of the drawings of the present disclosure represent structures and devices in block diagram form in order to avoid obscuring the novel aspects of the disclosed concepts. In the interest of clarity, not all features of an actual implementation may be described. Additionally, some of the drawings of this disclosure may be provided in the form of flowcharts as part of this specification. The blocks in any particular flowchart may be presented in a particular order. It should be understood, however, that the specific order of any given flowchart is for illustrative purposes only of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be deleted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowchart. In addition, the language used in this disclosure has been chosen primarily for readability and instructional purposes, and may not have been chosen to delineate or delimit the inventive subject matter, so that recourse to the claims is necessary to determine such subject of invention. Reference in this disclosure to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to " One embodiment" or "an embodiment" should not be read as necessarily all referring to the same embodiment.

It should be appreciated that in the development of any actual implementation (as in any software and/or hardware development project), many decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints ), and these goals may vary between implementations. It should also be appreciated that such development efforts might be complex and time-consuming, but nonetheless constitute a daily task for those of ordinary skill in designing and implementing graphical modeling systems having the benefit of this disclosure.

Referring to FIG. 1 , a simplified block diagram of an electronic device 100 communicatively connected to an additional electronic device 110 and a network storage device 115 via a network 105 is shown, in accordance with one or more embodiments of the present disclosure. The electronic device 100 may be a multifunction device such as a mobile phone, tablet computer, personal digital assistant, portable music/video player, wearable device, head mounted system, projection based system, base station, laptop computer, desktop computer, part of a network device or any other electronic system such as those described herein. Electronic device 100, additional electronic device 110, and/or network storage 115 may additionally or alternatively include one or more additional devices (such as server devices, base stations, accessory devices, etc.) in which Various functions can be accommodated or distributed across the devices. Exemplary networks such as network 105 include, but are not limited to, local networks such as a Universal Serial Bus (USB) network, an organization's local area network, and wide area networks such as the Internet. According to one or more embodiments, the electronic device 100 is configured to participate in an XRC session. It should be appreciated that various components and functions within electronic device 100, additional electronic device 110, and network storage 115 may be distributed across the devices differently or may be distributed across additional devices.

Electronic device 100 may include one or more processors 125, such as central processing units (CPUs). Processor 125 may include a system-on-chip, such as found in mobile devices, and may include one or more dedicated graphics processing units (GPUs). Additionally, the processor 125 may include multiple processors of the same or different types. The electronic device 100 may also include a memory 135 . Memory 135 may include one or more different types of memory that may be used in conjunction with processor 125 to perform device functions. For example, memory 135 may include cache, ROM, RAM, or any kind of transitory or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 135 may store various programming modules for execution by processor 125 , including XR module 165 , XRCE module 170 and other various application programs 175 . The electronic device 100 may further include a storage device 130 . Storage device 130 may include one or more non-transitory computer-readable storage media including, for example, magnetic disks (fixed, floppy, and removable) and magnetic tape, optical media such as CD-ROM and digital video compact disk (DVD)), and semiconductor storage devices such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM). According to one or more embodiments, storage device 130 may be configured to store content item 160 .

The electronic device 100 may also include one or more cameras 140 or other sensors 145, such as a depth sensor from which a depth of a scene may be determined. In one or more implementations, each of the one or more cameras 140 may be a conventional RGB camera or a depth camera. Additionally, cameras 140 may include stereo cameras or other multi-camera systems, time-of-flight camera systems, and the like. The electronic device 100 may also include a display 155 . Display device 155 may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The media can be optical waveguides, holographic media, optical combiners, optical reflectors, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems may employ retinal projection techniques that project graphic images onto a person's retina. The projection system may also be configured to project virtual objects into a physical environment, for example as a hologram or onto a physical surface.

According to one or more embodiments, the memory 135 may include one or more modules including computer readable code executable by the processor 125 to perform functions. This memory may include, for example, an XR module 165 that may be used to provide XRE to a local XRC device. The XRCE module 170 can generate an XRE-enabled environment. For example, the XRCE module 170 may provide services and/or application programming interfaces that may be used by the XRE and/or hosted applications to interface with the operating system (OS) and/or hardware of the electronic device 100 . An XRC session may be a computing environment that supports a shared experience of the electronic device 100 and the additional electronic device 110 .

The memory 135 may also include an OS module 180 for supporting basic functions of the electronic device 100 and managing hardware of the electronic device. The OS module 180 provides an environment in which the application program 175 can execute. In some cases, XRCE module 170 and XR module 165 may run within the context of OS module 180 . In other cases, the XRCE module 170 and the XR module 165 may process basic functions of the electronic device 100 and manage hardware of the electronic device in parallel with the OS module 180 or instead of the OS module. XRCE module 170 also provides an environment in which application programs 175 can execute. Applications 175 may include, for example, computer applications that may be experienced by multiple devices, such as electronic device 100 and additional electronic device 110, in an XRC session.

Although electronic device 100 is described as including the numerous components described above, in one or more implementations, various components may be distributed across multiple devices. Thus, although certain calls and transfers are described herein with respect to the particular system depicted, in one or more implementations, various calls and transfers may be directed differently based on different distributions of functionality. Additionally, additional components may be used, and some combination of the functionality of any of the components may be combined.

Figure 2 illustrates a diagram of an exemplary operating environment, according to one or more implementations. While relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other feature. To this end, operating environment 240 includes, by way of non-limiting example, a first physical environment and operating environment 250 includes a second physical environment.

Figure 2 illustrates a diagram of an exemplary operating environment, according to one or more implementations. While relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other feature. To this end, operating environment 240 includes, by way of non-limiting example, a first physical environment and operating environment 250 includes a second physical environment.

As shown in FIG. 2 , the first environment 240 includes a first user 220 who is using the first electronic device 200 , and the second environment 250 includes a second user 232 who is using the second electronic device 210 . In one or more embodiments, the first electronic device 200 and the second electronic device 210 include mobile devices, such as handheld devices, wearable devices, and the like.

In one or more implementations, the first electronic device 200 and the second electronic device 210 communicate with each other via the network 205 . Examples of network 205 may include, for example, the Internet, a wide area network (WAN), a local area network (LAN), and the like. In one or more implementations, the first electronic device 200 and the second electronic device 210 may participate in an XRC session.

Although the electronic device 200 and the second electronic device 210 may participate in a common XRC session, the XRE may be rendered differently on each device. As shown, electronic device 200 may depict physical objects of environment 240 . As shown, physical table 222 may be depicted on display 242 as virtual table 224 . In one or more implementations, display 242 may be a see-through display, and virtual table 224 may simply be a view of physical table 222 through display 242 .

The display 242 of the electronic device 200 may also include an avatar 226 corresponding to the user 232 in the physical environment 250 . For purposes of this disclosure, an avatar may include a virtual representation of a user. The avatar may depict real-time actions of the corresponding user 232, including movement, updated location, and/or interaction with various physical and/or virtual components within the XRC session. The avatar may or may not simulate the user's physical characteristics, and may or may not simulate the user's facial expressions.

According to one or more embodiments, XRCE may be an XRC session that supports one or more XRCE applications or other modules across all participating devices such as the electronic device 200 and the second electronic device 210 provides a depiction of the object. As shown in display 242, presentation panel 230A is an example of a virtual object that is visible to all participating devices.

As an example, returning to environment 250, second electronic device 210 includes display 252 on which presentation panel virtual object 230B is depicted. It should be understood that, in one or more embodiments, although the same virtual object may be visible across all participating devices, depending on the position of the electronic device, the orientation of the electronic device, or other physical or virtual characteristics associated with the electronic devices 200 and 210 And/or the XRCE depicted within displays 242 and 252 may render the virtual object differently.

Returning to environment 250, another characteristic of XRC sessions is that while virtual objects may be shared across participating devices, the physical world may appear different. Accordingly, physical chair 234 is depicted as virtual chair 236 . As noted above, and as one or more embodiments, display 252 may be a see-through display, and virtual chair 236 may be a view of physical chair 234 through see-through display 252 . Additionally, the second electronic device 210 depicts an avatar 238 corresponding to the user 220 within the physical environment 240 .

According to one or more embodiments, virtual objects such as presentation panel 230 may be rendered in conjunction with an XRCE application. In one or more implementations, multiple XRCE applications can be executed within XRCE. For example, XRCE may host XRCE applications as virtual objects within XRCE's XRE. Although hosted within XRCE, an XRCE application may execute as a process separate from one or more processes of XRCE. As discussed further below, applications can be configured to run inside or outside of XRCE. For example, an application may include multiple modes of operation, such as an XRCE mode and a non-XRCE mode, based on the operating environment or context in which the application is executing. The application can be determined to be running within the context of XRCE and operate in XRCE mode, such as through an application programming interface (API). When running outside of XRCE, applications can run in non-XRCE mode. For example, an application runs in a non-XRCE mode when running in the context of the OS of the electronic device but not in XRCE. As used herein, an XRCE application or hosted application can also refer to an application running in XRCE mode as well as an application configured to run only within the context of XRCE. XRCE can provide a container or computing environment in which XRCE applications can run, in a manner similar to an operating system. These containers can be presented to the user as virtual objects, making it appear that the XRCE application is executing in a window, on a surface, or on another virtual object.

In some cases, XRCE applications execute in the context of a specific XRCE instance. For example, a first user associated with a first device may participate in an XRC session with a second user associated with a second device. The XRCE application in this example executes on both the first device and the second device. Based on input from the first user, the first device may execute a first XRCE application hosted in a local XRCE instance. The first XRCE application may share data with a second XRCE application executing on the second device hosted in the second XRCE instance. Execution of applications hosted within the context of a specific XRCE instance helps provide privacy measures for native users within XRCE. In some cases, XRCE instances can be executed locally on the device. In other cases, an XRCE instance may execute at least partially on a network or hosted device.

In some cases, a user of an XRC session may desire to share the application experience of the XRCE application with other users of the XRC session. For example, a group of users may want to experience an XRCE application together, such as a game, group drawing application, productivity application, etc., as a shared XRCE application experience within an XRC session. In some cases, applications can be configured to run inside or outside of XRCE. For example, an application may determine that it is running within the context of XRCE, such as through an API, and operate in XRCE mode. Allowing XRCE applications to access data from XRCE can raise privacy concerns. For example, it may not be desirable to allow an XRCE application to access voice data, avatars, or other communication data between users as in a shared XRCE application experience. To help maintain privacy, XRCE applications may be limited in the types of data that XRCE applications may receive.

3A is a line drawing illustrating an XRCE interface 300 for bringing the XRCE application experience to XRCE in accordance with aspects of the present disclosure. In some cases, XRCE 302 may display views into a shared XRC session. XRCE 302 may include interfaces such as menus, gestures, voice commands, or other interfaces that a user may utilize to select applications to be brought into XRCE 302 . In this example, XRCE 302 includes a menu 304 for bringing the application experience to XRCE 302 . Menu 304 may be subdivided, for example, to categorize available XRCE applications. In this example, the games category 306 is selected, which displays a list 308 of available applications that can be brought into the XRCE 302 .

In some cases, XRCE may run, for example, as a background application. For example, a user may switch out of an active XRCE and switch to another application to browse and/or select an application to bring into the XRC session. 3B is a line drawing illustrating another interface for bringing application experience to an XRC session 350 in accordance with aspects of the present disclosure. In this example, the interface for sharing content executes outside the context of XRCE. As an example, a user may browse an application or content on their device and access an interface for bringing the application experience into XRCE, eg, using a button, long press, selecting a content item, and the like. In this example, a user may select application 352 and access context menu 354 associated with application 352 . Context menu 354 includes menu item 356 that brings the application experience into XRCE. In some cases, users can access XRCE applications from within XRCE, while XRCE applications are not visible to other users of XRCE. In some cases where XRCE is not running, an instance of XRCE can be started and the application then brought into XRCE.

In some cases, XRCE may include functionality that applications may be able to take advantage of. As an example, a first user may be in an XRC session with a second user and a third user, and the first user may start an application experience with the second user's XRCE application. The XRCE application may be invoked or launched with an indication that the XRCE application is executing within the context of an XRC session, such as using a flag or switch. In some cases, initiating an XRCE application experience among multiple users in an XRC session may launch an instance of the XRCE application on each user's local device. For example, an XRC session may invoke an XRCE application on each local device of each user participating in the application experience. The application can access one or more APIs that allow the application to take advantage of the features provided by the XRCE session. For example, an XRCE application may be able to call an API provided by XRCE to access functionality provided by XRCE. In some cases, XRCE may provide data transfer, networking, audio, video, and avatar processing functionality that an XRCE application may use when the XRCE application executes within the context of XRCE. Continuing with the example, where a participant in the application experience, such as a second user, does not have an XRCE application, an XRCE application, such as an XRCE application executing on the first user's local device, can take advantage of XRCE's networking capabilities and transmit Or cause the second device to otherwise obtain a version of the XRCE application for use during the application experience. In some cases, a version of an XRCE application may provide reduced or different functionality compared to a full XRCE application. The instance of the XRCE application associated with the first user can then use the networking functionality provided by XRCE to transfer data to other instances of the XRCE application.

According to aspects of the present disclosure, graphical data from an XRCE application may be incorporated into a video stream of an XRC session along with visual representation data. Visual representation data is graphical data representing the user, such as an avatar and/or video data from the user. Visual representation data is eg collected by an XRCE instance and distributed among other devices participating in the XRC session. In some cases, the visual representation data can be processed by another XRCE application. This other XRCE application may be integrated into XRCE, for example, as a first party logical component of a separate XRCE, eg to allow individual updates to be sent, or as a first or third party trusted application.

Users can be associated with an avatar within XRE. In XRE, an avatar is generated, packaged into a video stream, and distributed to other participants of the XRC session. Animation of the avatar and/or effects applied to the avatar, as well as video (if any) between participants in an XRC session can also be handled by the XRCE instance and can be based on information in the video stream.

When an XRCE application is running in the context of an XRCE instance, such as during an application experience, graphics provided by the XRCE application may be passed to the XRCE instance, eg, via an API, and sent as part of the video stream of the XRC session. In some cases, the XRCE application's graphical environment may be displayed as part of the XRC session's graphical environment. In other cases, the graphical environment of the XRCE instance may be replaced by the graphical environment of the XRCE application during the application experience. FIG. 4 is a diagram illustrating an application experience 400 according to aspects of the present disclosure. In this example application experience 400, instead of the graphical environment of an XRCE instance, a graphical environment of an XRCE application 402 is shown. In some cases, graphics data generated by an XRCE application may be sent to the XRCE instance, eg, via an API, and integrated into the video stream of the XRC session, along with visual representation data such as an avatar. According to aspects of the present disclosure, the XRCE instance proceeds to draw avatars representing the first user 404 and the second user 406 into the graphical environment of the XRCE application 402 . For example, where two users are participating in an XRCE experience, the XRCE application may provide location information where to display two avatars representing the two users. Avatar animations, effects, etc. may be rendered by the XRCE instance in the graphical environment of the XRCE application 402 in a manner similar to those of the avatar animations, effects, etc. rendered in the graphical environment of XRCE. For example, XRCE may have access to the local user's avatar information, and XRCE may render the avatar representing the local user at a location corresponding to the location information provided by the XRCE application. Similarly, the native XRCE may receive avatar information from a second XRCE application. The avatar information may include information for rendering and/or animating a second avatar representing the second user. The XRCE application may render the second avatar at a location corresponding to the location information provided by the XRCE application. In some cases, the second XRCE application may also provide location information for one or more avatars. In some cases, an animation associated with the user himself or a graphical representation of the user himself may be adjusted or changed based on the engaging application experience.

When a video stream is received, the input video stream is processed by the local XRCE instance, and the graphics data of the XRCE application can be directly rendered by the XRCE instance, or the graphics data of the XRCE application can be extracted from the received video stream and passed to the XRCE application for processing and rendering. Since the XRCE application only has access to graphical data generated or used by the XRCE application, the XRCE application does not have access to the user's visual representation.

In some cases, an XRCE application may be able to control aspects of the visual representation of the data. For example, an XRCE application may request that the XRCE instance not display a graphical representation of the user. In such cases, the user's avatar and/or other graphical representation may be hidden. As another example, an XRCE application may provide an XRCE instance with position information and/or size information for a graphical representation. This helps to allow the XRCE application to dictate where to position eg a user avatar within the XRCE application environment to help optimize the application experience. In this example, a first user 404 is positioned opposite a second user 406 across a game board generated by the XRCE application. In some cases, the XRCE application may change or update the position and/or size of the user's graphical representation. In some cases, XRCE applications may provide location information on a per-participant basis. For example, an XRCE application may be able to access non-personally identifiable information about a user, such as a user identifier, via an XRCE instance. Position and/or size information for the graphical representation of the user may then be provided separately for each user identifier. This non-personally identifiable information may change or be randomized between instances of the application experience. In some cases, an XRCE application may include login or account functionality within the XRCE application. In such cases, the participant may log in or create an account, and the XRCE application may be able to associate non-personally identifiable information provided by the XRCE instance with the account or login.

According to certain aspects, audio content associated with or representing a user may also be processed by an XRCE instance of an XRCE application. For example, audio data such as music, sound effects, interactive sounds, etc. generated by an XRCE application can be delivered by a native XRCE application to a particular device's XRCE instance. In some cases, the XRCE application can also pass metadata about the audio data, such as timing information, direction information, etc., to the XRCE instance to help control the playback of the audio data. The XRCE instance may then transmit a representation or indication of the audio to other devices participating in the XRC session experienced by the application for reproduction by the other devices. Speech audio data between participants in an application experience can be handled by an XRCE instance. For example, during an XRC session, speech audio data is received from participants of the XRC session to generate an audio stream that is suitably passed between and played back by the devices participating in the XRC session. During the application experience, the XRCE instance continues to process speech audio data. Audio data generated by an XRCE application can eg be sent by an API to an XRCE instance and integrated into an audio stream. For example, an XRCE application can provide audio data that can be incorporated into the audio stream of the XRCE instance and sent to other participants for playback. The received audio stream can be processed by an XRCE instance, and the XRCE application's audio data can either be rendered directly by the XRCE instance, or the audio data can be extracted from the received audio stream and passed to the XRCE application for playback, without XRCE The application accesses other audio data such as speech audio data in the remainder of the audio stream. Since an XRCE application only has access to audio data generated or consumed by the XRCE application, the XRCE application does not have access to speech audio data between participants in the application experience. In some cases, the XRCE application may be able to control aspects of the speech audio data, such as by providing directional, mute, or other information to the XRCE. This information can be provided on a participant-by-participant basis or for all participants in the application experience. In some cases, speech audio data may be processed and/or transmitted by another XRCE application. This other XRCE application may be integrated into XRCE, for example as a first part of XRCE's logic, or as a first or third party trusted application.

5 is a flow diagram illustrating techniques for cooperating in a multi-user communication session 500 in accordance with aspects of the present disclosure. At block 502, a request to launch a first application in a multi-user communication session with a second device is received. For example, multiple users can collaborate in an XRC session, and a first user can begin an application experience using an XRCE application. The first user may launch the XRCE application, for example, via an interface of XRCE, such as a menu. In other cases, the first user may deactivate the XRCE instance on the first device and launch the XRCE application and begin the application experience, eg, via the OS interface or application program interface. At block 504, an environment of a first application is presented in a multi-user communication session. For example, an XRCE application can be presented within an XRC session. In some cases, the environment of the XRCE application may be presented in place of the XRE of the XRC session. At block 506, one or more processes of the multi-user communication session receive data including input data and first communication data. For example, a first user may provide control inputs as well as voice information to the XRCE application. This input can be received by an XRCE instance. In some cases, a multi-user communication session may be performed using one or more processes. For example, a multi-user communication session may include a process that renders a graphical interface, another process that handles communications, and so on. In some cases, a multi-user communication session may include a process for interfacing with one or more XRCE applications. The process for interfacing with one or more XRCE applications may be separate from other processes of the multi-user communication session. For example, a first process may provide communication data to and/or receive communication data from other devices of the multi-user communication session, and another separate process may receive information from and/or the environment of the XRCE application in the multi-user communication session render the environment. In some cases, the communication data may be handled by one or more processes integrated into XRCE, for example, as a first part logic component of XRCE. At block 508, the one or more processes provide input data to the first application. For example, control input may be provided by an XRCE instance to an XRCE application. At block 510, the one or more processes provide first communication data to a second device. For example, an XRCE instance may transmit received speech input data to another device participating in an application experience in an XRC session. Voice input data may be received by the first device, for example by a microphone of the first device. Speech input may be handled by an XRCE instance and provided to an XRC session but not to an XRCE application. At block 512, the one or more processes receive second communication data from the second device. For example, an XRCE instance may receive voice data from another device participating in an application experience in an XRC session. At block 514, the one or more processes present the second communication data in the environment of the first application program, wherein the first application program is restricted from accessing the first communication data and the second communication data. For example, the XRCE instance presents the received speech data to the first user without providing the received speech data to the XRCE application. The received speech data may be rendered, for example, by one or more speakers of the first device.

6A and 6B depict an example system 600 for various extended reality technologies.

In some examples, as shown in Figure 6A, system 600 includes device 600a. Device 600a includes various components such as processor 602 , RF circuitry 604 , memory 606 , image sensor 608 , orientation sensor 610 , microphone 612 , position sensor 616 , speaker 618 , display 620 , and touch-sensitive surface 622 . These components optionally communicate over communication bus 650 of device 600a.

In some examples, elements of system 600 are implemented in a base station device (e.g., a computing device such as a remote server, mobile device, or laptop) and other elements of system 600 are implemented in a second device (e.g., a head-mounted device) ) is implemented. In some examples, device 600A is implemented in a base station device or a second device.

As shown in FIG. 6B , in some examples, system 600 includes two (or more) devices that communicate, such as through wired or wireless connections. The first device 600B (eg, base station device) includes a processor 602 , an RF circuit 604 and a memory 606 . These components optionally communicate over communication bus 650 of device 600B. The second device 600C (e.g., a head-mounted device) includes various components such as a processor 602, RF circuitry 604, memory 606, image sensor 608, orientation sensor 610, microphone 612, position sensor 616, speaker 618, display 620, and Touch-sensitive surface 622 . These components optionally communicate over communication bus 650 of device 600C.

System 600 includes processor 602 and memory 606 . Processor 602 includes one or more general purpose processors, one or more graphics processors, and/or one or more digital signal processors. In some examples, memory 606 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 602 Execute to perform the technique described below.

System 600 includes RF circuitry 604 . RF circuitry 604 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, intranets), and/or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 604 optionally includes components for use with near-field communications and/or short-range communications such as ) circuits for communication.

System 600 includes display 620 . Display 620 may have an opaque display. Display 620 may have a transparent or translucent display that may incorporate a substrate through which light representing an image is directed to the individual's eye. Display 620 may incorporate LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The light-transmitting substrate can be an optical waveguide, an optical combiner, an optical reflector, a holographic substrate, or any combination of these substrates. In one example, a transparent or translucent display is selectively switchable between an opaque state and a transparent or translucent state. Other examples of display 620 include a head-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, glasses capable of displaying graphics, a tablet computer, a smartphone, and a desktop or laptop computer. Alternatively, system 600 may be designed to receive an external display (eg, a smartphone). In some examples, system 600 is a projection-based system that uses retinal projection to project images onto an individual's retina or to project virtual objects into a physical set (eg, onto a physical surface or as a hologram).

In some examples, system 600 includes touch-sensitive surface 622 for receiving user input, such as tap input and swipe input. In some examples, display 620 and touch-sensitive surface 622 form a touch-sensitive display.

System 600 includes image sensor 608 . Image sensors 608 optionally include one or more visible light image sensors, such as charge-coupled device (CCD) sensors and/or complementary metal-oxide-semiconductor (CMOS) sensors, operable to acquire images of physical elements from the physical scene. The one or more image sensors also optionally include one or more infrared (IR) sensors, such as passive IR sensors or active IR sensors, for detecting infrared light from the physical set. For example, an active IR sensor includes an IR emitter, such as an IR spot emitter, for emitting infrared light into a physical set. Image sensor 608 also optionally includes one or more event cameras configured to capture movement of physical elements in the physical set. Image sensor 608 also optionally includes one or more depth sensors configured to detect the distance of a physical element from system 600 . In some examples, system 600 uses a combination of CCD sensors, event cameras, and depth sensors to detect the physical scenery surrounding system 600 . In some examples, image sensor 608 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical elements in the physical set from two different viewing angles. In some examples, system 600 uses image sensor 608 to receive user input, such as gestures. In some examples, system 600 uses image sensor 608 to detect the position and orientation of system 600 and/or display 620 within a physical set. For example, system 600 uses image sensor 608 to track the position and orientation of display 620 relative to one or more fixed elements in the physical set.

In some examples, system 600 includes microphone 612 . System 600 uses microphone 612 to detect sounds from the user and/or the user's physical setting. In some examples, microphone 612 includes a microphone array (including a plurality of microphones), optionally operating in concert, to identify ambient noise or localize sound sources in the space of a physical set.

System 600 includes orientation sensor 610 for detecting orientation and/or movement of system 600 and/or display 620 . For example, system 600 uses orientation sensor 610 to track changes in system 600 and/or display 620 , such as changes in position and/or orientation relative to physical elements in a physical set. Orientation sensor 610 optionally includes one or more gyroscopes and/or one or more accelerometers.

The technologies defined herein consider options for obtaining and utilizing users' personal information. For example, such personal information may be utilized to provide an improved XRE experience and application experience on electronic devices. However, insofar as such personal information is collected, such information should be obtained with the informed consent of users so that users know and control the use of their personal information.

Parties with access to personal information will use that information only for lawful and reasonable purposes and will abide by privacy policies and practices that at least comply with applicable laws and regulations. Additionally, such policies should be comprehensive, user-accessible, and considered to meet or exceed government/industry standards. In addition, personal information will not be distributed, sold or otherwise shared outside of any reasonable and lawful purpose.

However, users may limit the extent to which personal information is available to parties. The processes and devices described herein may allow changes to settings or other preferences that allow users to control access to their personal information. Furthermore, while some of the features defined herein are described in the context of the use of personal information, aspects of these features can be implemented without the use of such information. As an example, a user's personal information may be obfuscated or otherwise generalized such that the information does not identify the specific user from whom it was obtained.

It should be understood that the above description is intended to be illustrative and not restrictive. Material has been presented to enable any person skilled in the art to make and use the claimed disclosed subject matter and is presented in the context of specific embodiments in which variations will be apparent to those skilled in the art (eg, some of the disclosed embodiments may be used in conjunction with each other). Thus, the specific arrangement of steps or actions shown in FIG. 5 , or the arrangement of elements shown in FIGS. 1-4 and 6 should not be construed as limiting the scope of the disclosed subject matter. The scope of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "comprising" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein".

Claims (38)

1. A method for collaboration in a multi-user communication session, comprising:

at a first device:

receiving a request to launch a first application in a multi-user communication session with a second device;

presenting an environment of the first application in the multi-user communication session;

receiving data through one or more processes of the multi-user communication session, the data comprising input data and first communication data;

providing, by the one or more processes, the input data to the first application;

providing the first communication data to the second device through the one or more processes;

receiving, by the one or more processes, second communication data from the second device; and

presenting, by the one or more processes, the second communication data in the environment of the first application, wherein the first application is restricted from accessing the first communication data and the second communication data.

2. The method of claim 1, further comprising:

receiving, by the one or more processes and from the first application, first head portrait location information for a first head portrait; and

Presenting, by the one or more processes, the first avatar in the environment of the first application based on the first avatar location information.

3. The method of claim 2, further comprising:

receiving, by the one or more processes and from the first application, second avatar location information;

receiving, by the one or more processes, second avatar information from the second device; and

and presenting a second head portrait based on the second head portrait position information and the second head portrait information.

4. A method according to any of claims 1-3, wherein the one or more processes comprise a system application or trusted application of the multi-user communication session.

5. The method of any of claims 1-4, further comprising updating the first application based on the input data.

6. The method of any of claims 1-5, wherein the input data is provided to the first application using an application programming interface.

7. The method of any of claims 1-6, wherein the first communication data is provided for presentation by an instance of the first application running on the second device in the multi-user communication session.

8. The method of any of claims 1-7, wherein the environment comprises an immersive environment of the first application.

9. The method of any of claims 1-8, wherein the input data includes data for controlling operation of the first application.

10. The method of any of claims 1-9, wherein the first communication data comprises audio data received by the first device, and wherein second communication data comprises audio data received from the second device.

11. The method of any of claims 1-10, wherein the first application comprises a game program.

12. A non-transitory program storage device comprising instructions stored thereon that cause one or more processors to:

receiving a request to launch a first application in a multi-user communication session with a second device;

presenting an environment of the first application in the multi-user communication session;

receiving data through one or more processes of the multi-user communication session, the data comprising input data and first communication data;

providing, by the one or more processes, the input data to the first application; and

Providing the first communication data to the second device through the one or more processes;

receiving, by the one or more processes, second communication data from the second device; and

presenting, by the one or more processes, the second communication data in the environment of the first application, wherein the first application is restricted from accessing the first communication data and the second communication data.

13. The non-transitory program storage device of claim 12, wherein the instructions further cause the one or more processors to:

receiving, by the one or more processes and from the first application, first head portrait location information for a first head portrait; and

presenting, by the one or more processes, the first avatar in the environment of the first application based on the first avatar location information.

14. The non-transitory program storage device of claim 13, wherein the instructions further cause the one or more processors to:

receiving, by the one or more processes and from the first application, second avatar location information;

receiving, by the one or more processes, second avatar information from the second device; and

And presenting a second head portrait based on the second head portrait position information and the second head portrait information.

15. The non-transitory program storage device of any one of claims 12-14, wherein the one or more processes include a system application or trusted application of the multi-user communication session.

16. The non-transitory program storage device of any one of claims 12-15, wherein the instructions further cause the one or more processors to update the first application based on the input data.

17. The non-transitory program storage device of any one of claims 12-16, wherein the input data is provided to the first application using an application programming interface.

18. The non-transitory program storage device of any one of claims 12-17, wherein the first communication data is provided for presentation by an instance of the first application running on the second device in the multi-user communication session.

19. The non-transitory program storage device of any one of claims 12-18, wherein the environment comprises an immersive environment of the first application.

20. The non-transitory program storage device of any one of claims 12-19, wherein the input data includes data for controlling operation of the first application.

21. The non-transitory program storage device of any one of claims 12-20, wherein the first communication data comprises audio data received by the first device, and wherein second communication data comprises audio data received from the second device.

22. The non-transitory program storage device of any one of claims 12-21, wherein the first application comprises a game program.

23. An electronic device, comprising:

a memory;

a display;

networking device

An input device; and

one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions that cause the one or more processors to:

receiving a request to launch a first application in a multi-user communication session with a second device;

presenting, by the display, an environment of the first application in the multi-user communication session;

receiving data from the input device and through one or more processes of the multi-user communication session, the data comprising input data and first communication data;

providing, by the one or more processes, the input data to the first application; and

Providing, by the one or more processes, the first communication data to the second device via the networking device;

receiving, by the one or more processes, second communication data from the second device via the networking device; and

presenting, by the one or more processes and via the display, the second communication data in the environment of the first application, wherein the first application is restricted from accessing the first communication data and the second communication data.

24. The device of claim 23, wherein the one or more processors are configured to execute instructions that further cause the one or more processors to:

receiving, by the one or more processes and from the first application, first head portrait location information for a first head portrait; and

the first avatar is presented in the environment of the first application by the one or more processes and via the display based on the first avatar location information.

25. The device of claim 24, wherein the one or more processors are configured to execute instructions that further cause the one or more processors to:

Receiving, by the one or more processes and from the first application, second avatar location information;

receiving, by the one or more processes, second avatar information from the second device via the networking device; and

and presenting a second avatar via the display based on the second avatar position information and the second avatar information.

26. The apparatus of any of claims 23-25, wherein the one or more processes comprise a system application or trusted application of the multi-user communication session.

27. The apparatus of any of claims 23-26, wherein the instructions further cause the one or more processors to update the first application based on the input data.

28. The apparatus of any of claims 23-27, wherein the input data is provided to the first application using an application programming interface.

29. The device of any of claims 23-28, wherein the first communication data is provided for presentation by an instance of the first application running on the second device in the multi-user communication session.

30. The apparatus of any of claims 23-29, wherein the environment comprises an immersive environment of the first application.

31. The apparatus of any of claims 23-30, wherein the input data comprises data for controlling operation of the first application.

32. The device of any of claims 23-31, wherein the first communication data comprises first audio data received by the first device, and wherein second communication data comprises second audio data received from the second device via the networking device.

33. The device of claim 32, further comprising a microphone, and wherein the first audio data is received through the microphone.

34. The device of claim 32, further comprising one or more speakers, and wherein the second audio data is presented through the one or more speakers.

35. The apparatus of any of claims 23-34, wherein the first application comprises a game program.

36. The method of any of claims 1-11, wherein a first process of the one or more processes of the multi-user communication session provides the first communication data to the second device, and wherein a second process of the one or more processes of the multi-user communication session presents the environment of the first application in the multi-user communication session.

37. The non-transitory program storage device of any one of claims 12-22, wherein a first process of the one or more processes of the multi-user communication session provides the first communication data to the second device, and wherein a second process of the one or more processes of the multi-user communication session presents the environment of the first application in the multi-user communication session.

38. The device of any of claims 23-35, wherein a first process of the one or more processes of the multi-user communication session provides the first communication data to the second device, and wherein a second process of the one or more processes of the multi-user communication session presents the environment of the first application in the multi-user communication session.