CN111367603A - Multi-participant real-time communication user interface - Google Patents

Abstract

The present invention is entitled "Multi-Participant Real-Time Communication User Interface". The present invention provides a method, the method comprising: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including a message area including communication between three or more participants Multiple messages in a message conversation. First data is received indicating that a real-time communication session available to the three or more participants is active. In response to receiving the first data, a notification is displayed indicating that the real-time communication session is active. Upon displaying the notification, second data is received indicating that the real-time communication session between the three or more participants is no longer active. In response to receiving the second data, ceasing to display or updating the notification indicating that the real-time communication session is active to indicate that the real-time communication session is no longer active.

Description

Multi-Participant Real-Time Communication User Interface

This application is a divisional application for an invention patent application with an application date of September 28, 2018, an application number of 201811136766.7, and the title of "Multi-Participant Real-Time Communication User Interface".

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is related to US Provisional Patent Application Serial No. 62/668,229, filed May 7, 2018, entitled "MULTI-PARTICIPANT LIVECOMMUNICATION USER INTERFACE," the contents of which are hereby incorporated by reference in their entirety.

technical field

The present disclosure relates generally to computer user interfaces, and more particularly, to techniques for real-time communications involving multiple participants.

Background technique

For personal and business reasons, users are increasingly using electronic devices to communicate virtually with others. Most electronic devices are capable of providing real-time communication between a user and multiple participants. As an example, some electronic devices provide an interactive interface for displaying and controlling real-time communications.

SUMMARY OF THE INVENTION

However, some techniques for real-time communication involving multiple participants using electronic devices are often cumbersome and inefficient. For example, some prior art uses complex and time-consuming user interfaces that may include multiple keystrokes or keystrokes. Existing techniques require more time than necessary, resulting in wasted user time and device energy. This latter consideration is particularly important in battery-operated devices.

Accordingly, the present technology provides faster, more efficient methods and interfaces for electronic devices for real-time communications involving multiple participants. Such methods and interfaces optionally supplement or replace other methods for real-time communication involving multiple participants. Such methods and interfaces reduce the cognitive load imposed on the user and result in a more efficient human-machine interface. For battery-operated computing devices, such methods and interfaces conserve power and increase the time interval between battery charges.

In some implementations, a method is performed at an electronic device having a display. The method may include: displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including a message area including a plurality of messages in a messaging conversation between three or more participants; receiving an indication first data indicating that a real-time communication session available to the three or more participants is active; in response to receiving the first data indicating that the real-time communication session between the three or more participants is active, displaying an indication a notification that the real-time communication session is active; while displaying the notification indicating that the real-time communication session is active, receiving second data indicating that the real-time communication session between the three or more participants is no longer active; and In response to receiving the second data indicating that the real-time communication session between the three or more participants is no longer active, the notification is stopped from being displayed or the notification is updated to indicate that the real-time communication session is no longer active.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for displaying an instant messaging user interface of the instant messaging application, the instant messaging user interface including a message area including communication between three or more participants. a plurality of messages in a messaging conversation; receiving first data indicating that a real-time communication session available to three or more participants is active; in response to receiving the first data indicating a real-time communication session between the three or more participants first data in an active state, displaying a notification indicating that the real-time communication session is active; receiving a notification indicating that the real-time communication session between the three or more participants is not active while displaying the notification indicating that the real-time communication session is active; second data that is re-active; and in response to receiving the second data indicating that the real-time communication session between the three or more participants is no longer active, ceasing to display the notification or updating the notification to indicate the real-time communication session is no longer active.

In some embodiments, a transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for displaying an instant messaging user interface of the instant messaging application, the instant messaging user interface including a message area including communication between three or more participants. Multiple messages in a messaging conversation; receiving first data indicating that a real-time communication session available to three or more participants is active. In response to receiving the first data indicating that the real-time communication session between the three or more participants is active, displaying a notification indicating that the real-time communication session is active; after displaying the notification indicating that the real-time communication session is active At the same time, receiving second data indicating that the real-time communication session between the three or more participants is no longer active; and in response to receiving second data indicating that the real-time communication session between the three or more participants is no longer active Active second data, stop displaying the notification or update the notification to indicate that the real-time communication session is no longer active.

In some embodiments, an electronic device includes a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for displaying an instant messaging user interface of the instant messaging application, the instant messaging user interface including a message area including communication between three or more participants. Multiple messages in a messaging conversation; receiving first data indicating that a real-time communication session available to three or more participants is active. In response to receiving the first data indicating that the real-time communication session between the three or more participants is active, displaying a notification indicating that the real-time communication session is active; after displaying the notification indicating that the real-time communication session is active At the same time, receiving second data indicating that the real-time communication session between the three or more participants is no longer active; and in response to receiving second data indicating that the real-time communication session between the three or more participants is no longer active Active second data, stop displaying the notification or update the notification to indicate that the real-time communication session is no longer active.

In some embodiments, the electronic device includes: a display; means for displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including a message area that includes communication between three or more participants a plurality of messages in a message conversation; means for receiving first data indicating that a real-time communication session available to three or more participants is active; means for: displaying a notification indicating that the real-time communication session is active; means for performing the following operations while displaying the notification indicating that the real-time communication session is active : receiving second data indicating that the real-time communication session between the three or more participants is no longer active; and means for performing the following operations: in response to receiving the indication that one of the three or more participants is stop displaying the notification or update the notification to indicate that the real-time communication session is no longer active.

In some implementations, a method is performed at an electronic device having a display. The method may include: receiving a request to display a user interface for a real-time communication session between the two or more participants; and in response to receiving a display for the real-time communication between the two or more participants A request for a user interface for a session, displaying a real-time communication user interface, includes simultaneously displaying representations of multiple participants in a real-time communication session, wherein displaying representations of multiple participants in a real-time communication session includes: determining the participants in the real-time communication session according to The number includes the number of participants greater than the threshold, while displaying representations of the first plurality of participants other than the user of the electronic device in one or more sizes greater than the corresponding threshold size, while displaying the second plurality in the overflow area representations of participants, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the representations of one or more other participants displayed in the overflow area are less than the corresponding A size display of a threshold size; and based on determining that the number of participants in the real-time communication session includes a number of participants less than or equal to the threshold, displaying the size of the participants other than the user of the electronic device in one or more sizes greater than the corresponding threshold size representations without displaying representations of participants other than the user of the electronic device in one or more dimensions less than or equal to the respective threshold dimensions.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: receiving a request to display a user interface for a real-time communication session between two or more participants; and in response to receiving a request to display a user interface for the two or more participants A request for a user interface for a real-time communication session between more participants, displaying a real-time communication user interface, including simultaneous display of representations of multiple participants in a real-time communication session, wherein displaying representations of multiple participants in a real-time communication session includes : based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold, while displaying representations of the first plurality of participants other than the user of the electronic device in one or more sizes greater than the corresponding threshold size, while the representation of the second plurality of participants is displayed in the overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the one or more participants displayed in the overflow area The representations of the plurality of other participants are displayed in a size smaller than the corresponding threshold size; and based on determining that the number of participants in the real-time communication session includes the number of participants less than or equal to the threshold, displayed in one or more sizes larger than the corresponding threshold size except representations of participants other than the user of the electronic device without displaying representations of the participants other than the user of the electronic device in one or more dimensions less than or equal to the respective threshold dimensions.

In some embodiments, a transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for: receiving a request to display a user interface for a real-time communication session between two or more participants; and in response to receiving a request to display a user interface for the two or more participants A request for a user interface for a real-time communication session between more participants, displaying a real-time communication user interface, including simultaneous display of representations of multiple participants in a real-time communication session, wherein displaying representations of multiple participants in a real-time communication session includes : based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold, while displaying representations of the first plurality of participants other than the user of the electronic device in one or more sizes greater than the corresponding threshold size, while the representation of the second plurality of participants is displayed in the overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the one or more participants displayed in the overflow area The representations of the plurality of other participants are displayed in a size smaller than the corresponding threshold size; and based on determining that the number of participants in the real-time communication session includes the number of participants less than or equal to the threshold, displayed in one or more sizes larger than the corresponding threshold size except representations of participants other than the user of the electronic device without displaying representations of the participants other than the user of the electronic device in one or more dimensions less than or equal to the respective threshold dimensions.

In some embodiments, an electronic device includes a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for: receiving a request to display a user interface for a real-time communication session between two or more participants; and in response to receiving a request to display a user interface for the two or more participants A request for a user interface for a real-time communication session between more participants, displaying a real-time communication user interface, including simultaneous display of representations of multiple participants in a real-time communication session, wherein displaying representations of multiple participants in a real-time communication session includes : based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold, while displaying representations of the first plurality of participants other than the user of the electronic device in one or more sizes greater than the corresponding threshold size, while the representation of the second plurality of participants is displayed in the overflow area, wherein the second plurality of participants includes one or more participants not included in the first plurality of participants, and the one or more participants displayed in the overflow area The representations of the plurality of other participants are displayed in a size smaller than the corresponding threshold size; and based on determining that the number of participants in the real-time communication session includes the number of participants less than or equal to the threshold, displayed in one or more sizes larger than the corresponding threshold size except representations of participants other than the user of the electronic device without displaying representations of the participants other than the user of the electronic device in one or more dimensions less than or equal to the respective threshold dimensions.

In some embodiments, the electronic device includes: a display; means for receiving a request to display a user interface for a real-time communication session between two or more participants; and responsive to receiving the display for the two Means for displaying a real-time communication user interface, including simultaneous display of representations of multiple participants in a real-time communication session, at the request of a user interface for a real-time communication session between or among more participants, wherein the real-time communication session is displayed The plurality of participants in the representation includes: based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold, means for: simultaneously displaying the elimination electrons in one or more sizes greater than the corresponding threshold size A representation of a first plurality of participants other than the user of the device, while a representation of a second plurality of participants is displayed in the overflow area, wherein the second plurality of participants includes one not included in the first plurality of participants or more participants, and representations of one or more other participants displayed in the overflow area are displayed at a size less than a corresponding threshold size; and means for: based on determining the number of participants in the real-time communication session including Number of participants less than or equal to a threshold, representations of participants other than the user of the electronic device displayed in one or more sizes greater than the corresponding threshold size and not displayed in one or more sizes less than or equal to the corresponding threshold size A representation of a participant other than the user of the electronic device.

In some implementations, a method is performed at an electronic device having a display. The method may include displaying a real-time video communication user interface including simultaneously displaying a representation of a first participant in the real-time video communication session, a representation of a second participant in the real-time video communication session, and a third participant in the real-time video communication session A representation of three participants; while displaying a real-time video communication user interface, receiving data indicating that the activity level of a second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session; and in response to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session: changing the first participant's activity level in the real-time video communication user interface; the size and/or position of the representation; change the size and/or position of the representation of the second participant in the real-time video communication user interface; and change the size and/or position of the representation of the third participant in the real-time video communication user interface.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for displaying a real-time video communication user interface including concurrently displaying a representation of a first participant in the real-time video communication session, and a representation of a second participant in the real-time video communication session. a representation, and a representation of a third participant in the real-time video communication session; while displaying the real-time video communication user interface, receiving an indication that the activity level of the second participant in the real-time video communication session has increased above that in the real-time video communication session and in response to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session: changing the real-time the size and/or position of the representation of the first participant in the video communication user interface; changing the size and/or position of the representation of the second participant in the real-time video communication user interface; and changing the third participant in the real-time video communication user interface The size and/or location of the representation.

In some embodiments, a transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for displaying a real-time video communication user interface including concurrently displaying a representation of a first participant in the real-time video communication session, and a representation of a second participant in the real-time video communication session. a representation, and a representation of a third participant in the real-time video communication session; while displaying the real-time video communication user interface, receiving an indication that the activity level of the second participant in the real-time video communication session has increased above that in the real-time video communication session and in response to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session: changing the real-time the size and/or position of the representation of the first participant in the video communication user interface; changing the size and/or position of the representation of the second participant in the real-time video communication user interface; and changing the third participant in the real-time video communication user interface The size and/or location of the representation.

In some embodiments, an electronic device includes a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for displaying a real-time video communication user interface including concurrently displaying a representation of a first participant in the real-time video communication session, and a representation of a second participant in the real-time video communication session. a representation, and a representation of a third participant in the real-time video communication session; while displaying the real-time video communication user interface, receiving an indication that the activity level of the second participant in the real-time video communication session has increased above that in the real-time video communication session and in response to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session: changing the real-time the size and/or position of the representation of the first participant in the video communication user interface; changing the size and/or position of the representation of the second participant in the real-time video communication user interface; and changing the third participant in the real-time video communication user interface The size and/or location of the representation.

In some embodiments, the electronic device includes: a display; means for displaying a real-time video communication user interface, displaying the real-time video communication user interface including concurrently displaying a representation of a first participant in the real-time video communication session, in the real-time video communication session a representation of a second participant of a the activity level of the participant has increased above the activity level of the first participant in the real-time video communication session; and in response to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the real-time video communication session means for changing the size and/or position of the representation of the first participant in the real-time video communication user interface; changing the second participant in the real-time video communication user interface and changing the size and/or position of the representation of the third participant in the real-time video communication user interface.

In some implementations, a method is performed at an electronic device having a display. The method may include displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including a message area including multiple messages between participants in a messaging conversation including three or more participants and an affordance separate from a message area used to initiate a real-time video communication session in the real-time video communication application; detecting an input corresponding to the selection of the affordance; and in response to detecting the selection of the affordance A corresponding input initiates a process in the real-time video communication application for starting a real-time video communication session, the real-time video communication session including the participants of the messaging conversation.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for displaying an instant messaging user interface of the instant messaging application, the instant messaging user interface including: a message area included in a system that includes three or more participants; a plurality of messages between participants in a messaging conversation; and an affordance separate from the message area used to initiate a real-time video communication session in the real-time video communication application; detecting an input corresponding to the selection of the affordance; and In response to detecting the input corresponding to the selection of the affordance, a process is initiated in the real-time video communication application for initiating a real-time video communication session, the real-time video communication session including the participants of the messaging conversation.

In some embodiments, a transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for displaying an instant messaging user interface of the instant messaging application, the instant messaging user interface including: a message area included in a system that includes three or more participants; a plurality of messages between participants in a messaging conversation; and an affordance separate from the message area used to initiate a real-time video communication session in the real-time video communication application; detecting an input corresponding to the selection of the affordance; and In response to detecting the input corresponding to the selection of the affordance, a process is initiated in the real-time video communication application for initiating a real-time video communication session, the real-time video communication session including the participants of the messaging conversation.

In some embodiments, an electronic device includes a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for displaying an instant messaging user interface of the instant messaging application, the instant messaging user interface including: a message area included in a system that includes three or more participants; a plurality of messages between participants in a messaging conversation; and an affordance separate from the message area used to initiate a real-time video communication session in the real-time video communication application; detecting an input corresponding to the selection of the affordance; and In response to detecting the input corresponding to the selection of the affordance, a process is initiated in the real-time video communication application for initiating a real-time video communication session, the real-time video communication session including the participants of the messaging conversation.

In some embodiments, the electronic device includes: a display; means for displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface including: a message area including a message area including three or more participants multiple messages between participants in a message conversation of a participant; and an affordance separate from a message area used to start a real-time video communication session in a real-time video communication application; used to detect an affordance corresponding to the selection of an affordance means for input; and means for initiating, in a real-time video communication application, a process for initiating a real-time video communication session, the real-time video communication session comprising, in response to detecting an input corresponding to selection of an affordance A participant in a message conversation.

In some implementations, a method is performed at an electronic device having a display. The method may include displaying a real-time video communication user interface of the real-time video communication application, the real-time video communication user interface comprising: a plurality of participant affordances corresponding to a corresponding one or more participants, the plurality of participants The participant affordances include a first participant affordance corresponding to a group of three or more participants; and a new session affordance for starting a new real-time video communication session; detecting a input, i.e. initiating a process for starting a new real-time video communication session; and in response to detecting the input: initiating a process for starting a new real-time video communication session according to the input corresponding to the selection of the first participant's affordance, the new The real-time video communication session includes a group of three or more participants; and according to an input corresponding to the selection of the new session affordance, an initiation for selecting two or more participants and starting a new real-time video The process of a communication session that includes participants associated with the device and two or more selected participants.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for displaying a real-time video communication user interface of the real-time video communication application, the real-time video communication user interface including: a plurality of participations corresponding to the corresponding one or more participants a participant affordance, the plurality of participant affordances including a first participant affordance corresponding to a group of three or more participants; and a new session for starting a new real-time video communication session an affordance; detecting an input corresponding to a request to initiate a process to start a new real-time video communication session; and in response to detecting the input: initiating a new start according to an input corresponding to the selection of the first participant's affordance the process of a real-time video communication session that includes a group of three or more participants; and initiates a method for selecting two or more participants according to an input corresponding to the selection of the new session affordance A plurality of participants and the process of starting a new real-time video communication session including the participants associated with the device and two or more selected participants.

In some embodiments, a transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device having a display. The one or more programs include instructions for displaying a real-time video communication user interface of the real-time video communication application, the real-time video communication user interface including: a plurality of participations corresponding to the corresponding one or more participants a participant affordance, the plurality of participant affordances including a first participant affordance corresponding to a group of three or more participants; and a new session for starting a new real-time video communication session an affordance; detecting an input corresponding to a request to initiate a process to start a new real-time video communication session; and in response to detecting the input: initiating a new start according to an input corresponding to the selection of the first participant's affordance the process of a real-time video communication session that includes a group of three or more participants; and initiates a method for selecting two or more participants according to an input corresponding to the selection of the new session affordance A plurality of participants and the process of starting a new real-time video communication session including the participants associated with the device and two or more selected participants.

In some embodiments, an electronic device includes a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for displaying a real-time video communication user interface of the real-time video communication application, the real-time video communication user interface including: a plurality of participations corresponding to the corresponding one or more participants a participant affordance, the plurality of participant affordances including a first participant affordance corresponding to a group of three or more participants; and a new session for starting a new real-time video communication session an affordance; detecting an input corresponding to a request to initiate a process to start a new real-time video communication session; and in response to detecting the input: initiating a new start according to an input corresponding to the selection of the first participant's affordance the process of a real-time video communication session that includes a group of three or more participants; and initiates a method for selecting two or more participants according to an input corresponding to the selection of the new session affordance A plurality of participants and the process of starting a new real-time video communication session including the participants associated with the device and two or more selected participants.

In some embodiments, the electronic device includes: a display; means for displaying a real-time video communication user interface of the real-time video communication application, the real-time video communication user interface including: a plurality of corresponding one or more participants participant affordances, the plurality of participant affordances including a first participant affordance corresponding to a group of three or more participants; and a new participant affordance for starting a new real-time video communication session a session affordance; means for detecting an input corresponding to a request to initiate a process for starting a new real-time video communication session; and, in response to detecting the input, means for: according to a communication with the first participant an input corresponding to the selection of the affordance, initiates the process of starting a new real-time video communication session, the new real-time video communication session including a group of three or more participants; and according to the selection of the affordance associated with the new session A corresponding input initiates a process for selecting two or more participants and starting a new real-time video communication session that includes the participants associated with the device and the two or more selected participants.

Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. Executable instructions for performing these functions are optionally included in a transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

Accordingly, faster and more efficient methods and interfaces are provided for devices for real-time communication involving multiple participants, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may complement or replace other methods for real-time communication involving multiple participants.

Description of drawings

For a better understanding of the various described embodiments, reference should be made to the following Detailed Description in conjunction with the following drawings, wherein like reference numerals indicate corresponding parts throughout.

1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.

FIG. 1B is a block diagram illustrating exemplary components for event processing, according to some embodiments.

2 illustrates a portable multifunction device with a touch screen in accordance with some embodiments.

3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, according to some embodiments.

4A illustrates an exemplary user interface for a menu of an application on a portable multifunction device, according to some embodiments.

4B illustrates an exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display, according to some embodiments.

Figure 5A shows a personal electronic device according to some embodiments.

5B is a block diagram illustrating a personal electronic device according to some embodiments.

5C-5D illustrate exemplary components of a personal electronic device having a touch-sensitive display and an intensity sensor according to some embodiments.

5E-5H illustrate exemplary components and user interfaces of a personal electronic device according to some embodiments.

6A-6AF illustrate exemplary user interfaces according to some embodiments.

7A-7F illustrate exemplary methods according to some embodiments.

8A-8BK illustrate exemplary user interfaces in accordance with some embodiments.

9A-9K illustrate exemplary methods according to some embodiments.

10A-10S illustrate exemplary user interfaces in accordance with some embodiments.

11A-11F illustrate exemplary methods according to some embodiments.

12A-12N illustrate exemplary user interfaces according to some embodiments.

13A-13D illustrate exemplary methods according to some embodiments.

14A-14I illustrate exemplary user interfaces in accordance with some embodiments.

15A-15C illustrate exemplary methods according to some embodiments.

Detailed ways

The following description sets forth exemplary methods, parameters, and the like. It should be appreciated, however, that such descriptions are not intended to limit the scope of the disclosure, but are provided as descriptions of exemplary embodiments.

Electronic devices need to provide efficient methods and interfaces for real-time communication involving multiple participants. Such techniques may reduce the cognitive burden on users engaged in real-time communications involving multiple participants, thereby increasing productivity. Additionally, such techniques can reduce processor power and battery power that would otherwise be wasted on redundant user inputs.

Below, FIGS. 1A-1B, 2, 3, 4A-4B, and 5A-5H provide descriptions of exemplary devices for performing techniques for real-time communication sessions between multiple participants . 6A-6AF illustrate exemplary user interfaces for managing event notifications. 7A-7F are flowcharts illustrating a method of conducting a real-time communication session between multiple participants, according to some embodiments. The user interfaces in Figures 6A-6AF are used to illustrate the processes described below including the processes in Figures 7A-7F.

8A-8BK illustrate exemplary user interfaces for real-time communication sessions between multiple participants. 9A-9K are flowcharts illustrating a method of conducting a real-time communication session between multiple participants, according to some embodiments. The user interfaces in FIGS. 8A-8BK are used to illustrate the processes described below including the processes in FIGS. 9A-9K.

10A-10S illustrate exemplary user interfaces for a real-time communication session between multiple participants. 11A-11F are flowcharts illustrating a method of conducting a real-time communication session between multiple participants, according to some embodiments. The user interfaces in FIGS. 10A-10S are used to illustrate the processes described below including the processes in FIGS. 11A-11F .

12A-12N illustrate exemplary user interfaces for a real-time communication session between multiple participants. 13A-13D are flowcharts illustrating a method of conducting a real-time communication session between multiple participants, according to some embodiments. The user interfaces in FIGS. 12A-12N are used to illustrate the processes described below including the processes in FIGS. 13A-13D.

14A-14I illustrate exemplary user interfaces for real-time communication sessions between multiple participants. 15A-15C are flowcharts illustrating a method of conducting a real-time communication session between multiple participants, according to some embodiments. The user interfaces in Figures 14A-14I are used to illustrate the processes described below including the processes in Figures 15A-15C.

Although the following description uses the terms "first," "second," etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch could be named a second touch and similarly a second touch could be named a first touch without departing from the scope of the various described implementations. Both the first touch and the second touch are touches, but they are not the same touch.

The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms "a" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "includes", "including", "comprises" and/or "comprising" when used in this specification are intended to designate the presence of stated features, integers, steps, operations, elements and/or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groupings thereof.

The term "if" is optionally construed to mean "when," "at," or "in response to determining" or "in response to detecting," depending on the context. Similarly, depending on the context, the phrases "if it is determined that..." or "if a [statement or event] is detected" are optionally interpreted to mean "in determining..." or "in response to determining..." or "on detection of [recited condition or event]" or "in response to detection of [recited condition or event]".

设备、iPod

设备、和

设备。任选地使用其他便携式电子设备，诸如具有触敏表面(例如，触摸屏显示器和/或触摸板)的膝上型电脑或平板电脑。还应当理解的是，在一些实施方案中，该设备并非便携式通信设备，而是具有触敏表面(例如，触摸屏显示器和/或触摸板)的台式计算机。Embodiments of electronic devices, user interfaces for such devices, and related processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also incorporates other functions such as PDA and/or music player functions. Exemplary embodiments of a portable multifunction device include, but are not limited to, a portable multifunction device from Apple Inc. (Cupertino, California).

Devices, iPods

equipment, and

equipment. Other portable electronic devices such as laptops or tablets with touch-sensitive surfaces (eg, touch screen displays and/or touch pads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (eg, a touch screen display and/or a touch pad).

In the following discussion, an electronic device is described that includes a display and a touch-sensitive surface. It should be understood, however, that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and/or joystick.

The device typically supports various applications, such as one or more of the following: drawing applications, rendering applications, word processing applications, website creation applications, disk editing applications, spreadsheet applications, gaming applications , phone application, video conferencing application, email application, instant messaging application, fitness support application, photo management application, digital camera application, digital video camera application, web browsing application, digital music player application and/or digital video player application.

Various applications executing on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and/or changed for different applications and/or within respective applications. In this way, the common physical architecture of the device, such as a touch-sensitive surface, optionally supports various applications with a user interface that is intuitive and clear to the user.

Attention is now turned to an embodiment of a portable device with a touch-sensitive display. FIG. 1A is a block diagram illustrating a portable multifunction device 100 having a touch-sensitive display system 112 in accordance with some embodiments. The touch-sensitive display 112 is sometimes referred to for convenience as a "touch screen" and is sometimes referred to or referred to as a "touch-sensitive display system." Device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), memory controller 122 , one or more processing units (CPUs) 120 , peripherals interface 118 , RF circuitry 108 , audio circuitry 110 , speaker 111 , microphone 113 , input/output (I/O) subsystem 106 , other input control devices 116 and external ports 124 . Device 100 optionally includes one or more optical sensors 164 . Device 100 optionally includes one or more contact intensity sensors 165 for detecting the intensity of a contact on device 100 (eg, a touch-sensitive surface, such as touch-sensitive display system 112 of device 100 ). Device 100 optionally includes one or more haptic output generators 167 for generating haptic output on device 100 (eg, on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or trackpad 355 of device 300 ). generate haptic output). These components optionally communicate via one or more communication buses or signal lines 103 .

As used in this specification and in the claims, the term "strength" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (eg, finger contact) on a touch-sensitive surface, Or refers to a surrogate (surrogate) for the force or pressure of contact on a touch-sensitive surface. The intensity of the contact has a range of values that includes at least four distinct values and more typically hundreds of distinct values (eg, at least 256). The strength of the contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (eg, weighted average) to determine an estimated contact force. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and/or variation of the contact area detected on the touch-sensitive surface, the capacitance of the touch-sensitive surface in the vicinity of the contact and/or its variation, and/or the resistance of the touch-sensitive surface in the vicinity of the contact and/or its variation Variation is optionally used as a surrogate for the force or pressure of contact on the touch-sensitive surface. In some implementations, the surrogate measurement of contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (eg, the intensity threshold is described in units corresponding to the surrogate measurement). In some implementations, a surrogate measurement of contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (eg, the intensity threshold is measured in units of pressure) pressure threshold). Use the strength of the contact as an attribute of the user input, thereby allowing the user to access additional device functionality that would otherwise be inaccessible to the user on a smaller sized device with limited physical area for (e.g., on a touch-sensitive display) ) display an affordance and/or receive user input (eg, via a touch-sensitive display, touch-sensitive surface, or physical/mechanical controls, such as knobs or buttons).

As used in this specification and in the claims, the term "haptic output" refers to a physical displacement of a device relative to a previous position of the device, a component of the device (eg, a touch-sensitive surface) that is detected by a user using the user's sense of touch. Physical displacement of another component of the device (eg, the housing), or displacement of a component relative to the center of mass of the device. For example, where a device or part of a device is in contact with a user's touch-sensitive surface (eg, a finger, palm, or other part of the user's hand), the haptic output generated by physical displacement will be interpreted by the user as a haptic sensation corresponding to Perceived changes in physical characteristics of equipment or components of equipment. For example, movement of a touch-sensitive surface (eg, a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press-click" or "release-click" of a physically actuated button. In some cases, the user will feel a tactile sensation, such as a "press down click" or "release click", even on a touch-sensitive surface associated with a touch-sensitive surface that is physically pressed (eg, displaced) by the user's movement. When the physical actuation button is not moving. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface is optionally interpreted or sensed by the user as the "roughness" of the touch-sensitive surface. While such interpretations of touch by a user will be limited by the user's individual sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a haptic output is described as corresponding to a user's particular sensory perception (eg, "click down," "click release," "roughness"), unless otherwise stated, the generated haptic output corresponds to a device or the physical displacement of its components that would generate said sensory perception of a typical (or average) user.

It should be understood that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or any There are optionally different configurations or arrangements of these components. The various components shown in FIG. 1A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and/or application specific integrated circuits.

Memory 102 optionally includes high speed random access memory, and also optionally includes nonvolatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other nonvolatile solid state memory devices. Memory controller 122 optionally controls other components of device 100 to access memory 102 .

Peripherals interface 118 may be used to couple the device's input peripherals and output peripherals to CPU 120 and memory 102 . The one or more processors 120 run or execute various software programs and/or sets of instructions stored in the memory 102 to perform various functions of the device 100 and process data. In some embodiments, peripheral interface 118 , CPU 120 and memory controller 122 are optionally implemented on a single chip such as chip 104 . In some other embodiments, they are optionally implemented on separate chips.

RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts/converts electrical signals to/from electrical signals and communicates with communications networks and other communications devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, encoders Decoding chipsets, Subscriber Identity Module (SIM) cards, memory, and more. RF circuitry 108 optionally communicates via wireless communications with networks such as the Internet (also known as the World Wide Web (WWW)), intranets, and/or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and other devices ) and/or Metropolitan Area Network (MAN)). RF circuitry 108 optionally includes well-known circuitry for detecting near field communication (NFC) fields, such as through a short range communication radio. Wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA) ), High Speed Uplink Packet Access (HSUPA), Evolution, Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Broadband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (for example, IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n and/or IEEE802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols such as Internet Message Access Protocol (IMAP) and/or Post Office Protocol (POP) )), instant messaging (eg, Extensible Message Processing and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilization Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Messages Services (SMS), or any other appropriate communication protocol including a communication protocol that has not been developed at the date of this document.

Audio circuitry 110 , speaker 111 and microphone 113 provide an audio interface between the user and device 100 . The audio circuit 110 receives audio data from the peripheral device interface 118 , converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111 . The speaker 111 converts electrical signals into sound waves audible to human ears. The audio circuit 110 also receives the electrical signal converted from the sound wave by the microphone 113 . Audio circuitry 110 converts the electrical signals to audio data and transmits the audio data to peripherals interface 118 for processing. Audio data is optionally retrieved from and/or transferred to memory 102 and/or RF circuitry 108 by peripherals interface 118 . In some embodiments, the audio circuit 110 also includes a headphone jack (eg, 212 in Figure 2). The headset jack provides an interface between the audio circuitry 110 and a removable audio input/output peripheral, such as an output-only headset or one that has an output (eg, a mono or binaural headset) and an input (eg, microphone) for both headsets.

I/O subsystem 106 couples input/output peripherals such as touch screen 112 and other input control devices 116 on device 100 to peripherals interface 118 . I/O subsystem 106 optionally includes display controller 156, optical sensor controller 158, depth camera controller 169, intensity sensor controller 159, haptic feedback controller 161, and one or more for other input or control devices A plurality of input controllers 160 . The one or more input controllers 160 receive/send electrical signals from/to other input control devices 116 . Other input control devices 116 optionally include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, and the like. In some alternative implementations, the one or more input controllers 160 are optionally coupled to any of the following (or not to any of the following): keyboard, infrared port , USB ports, and pointing devices such as a mouse. The one or more buttons (eg, 208 in FIG. 2 ) optionally include up/down buttons for volume control of speaker 111 and/or microphone 113 . The one or more buttons optionally include a push button (eg, 206 in Figure 2).

A quick press of the push button optionally unlocks the touchscreen 112 or optionally initiates the process of unlocking the device using gestures on the touchscreen, as filed on December 23, 2005 under the title "Unlocking a Device by Performing Gestures" on an Unlock Image" in US Patent Application 11/322,549 (ie US Patent No. 7,657,849), which is hereby incorporated by reference in its entirety. A long press on the push button (eg, 206 ) optionally turns the device 100 on or off. The function of one or more buttons is optionally user-customizable. The touch screen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

The touch-sensitive display 112 provides an input and output interface between the device and the user. Display controller 156 receives electrical signals from and/or sends electrical signals to touch screen 112 . The touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively, "graphics"). In some embodiments, some or all of the visual output optionally corresponds to a user interface object.

The touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on tactile and/or tactile contact. Touch screen 112 and display controller 156 (together with any associated modules and/or sets of instructions in memory 102 ) detect contact on touch screen 112 (and any movement or interruption of the contact) and translate the detected contact For interaction with user interface objects (eg, one or more soft keys, icons, web pages, or images) displayed on the touch screen 112 . In one exemplary embodiment, the point of contact between the touch screen 112 and the user corresponds to the user's finger.

和iPod

中使用的技术。The touch screen 112 optionally uses LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology, or LED (Light Emitting Diode) technology, although other display technologies are used in other embodiments. The touch screen 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed as well as other proximity sensor arrays or for determining the point or points of contact with the touch screen 112 The various touch sensing technologies include, but are not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In one exemplary embodiment, projected mutual capacitance sensing technology is used, such as in the

and iPod

technology used in.

The touch-sensitive display in some embodiments of the touch screen 112 is optionally similar to the multi-touch-sensitive touchpads described in the following US Patents: 6,323,846 (Westerman et al.), 6,570,557 (Westerman et al.), and/or 6,677,932 (Westerman et al. Humans) and/or US Patent Publication 2002/0015024 Al, each of which is hereby incorporated by reference in its entirety. However, touch screen 112 displays visual output from device 100, while touch-sensitive trackpads do not provide visual output.

In some embodiments, the touch-sensitive display of touch screen 112 is as described in: (1) US Patent Application No. 11/381,313, filed May 2, 2006, entitled "Multipoint Touch Surface Controller"; (2) US Patent Application No. 10/840,862, filed on May 6, 2004, entitled "Multipoint Touchscreen"; (3) entitled "Gestures For Touch Sensitive InputDevices", filed on July 30, 2004 ( US Patent Application No. 10/903,964; (4) US Patent Application No. 11/048,264, "Gestures For Touch Sensitive Input Devices", filed January 31, 2005 ; (5) US Patent Application No. 11/038,590, filed January 18, 2005, entitled "Mode-Based Graphical UserInterfaces For Touch Sensitive Input Devices" ; (6) US Patent Application No. 11/228,758, filed on September 16, 2005, entitled "VirtualInput Device Placement On A Touch Screen User Interface"; (7) US Patent Application No. 11/228,700, filed Sep. 16, 2005, entitled "Operation Of A Computer With A Touch Screen Interface"; (8) filed Sep. 16, 2005 U.S. Patent Application No. 11/228,737 entitled "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard"; and (9) filed March 3, 2006 entitled "Multi- US Patent Application No. 11/367,749 to FunctionalHand-Held Device". All of these applications are incorporated herein by reference in their entirety.

The touch screen 112 optionally has a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of about 160 dpi. The user optionally makes contact with touch screen 112 using any suitable object or appendage such as a stylus, finger, or the like. In some embodiments, the user interface is designed to work primarily through finger-based contact and gestures, which may not be as accurate as stylus-based input due to the larger contact area of a finger on a touch screen. In some embodiments, the device translates the rough finger-based input into precise pointer/cursor positions or commands for performing the actions desired by the user.

In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad for activating or deactivating certain functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike a touchscreen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112, or an extension of the touch-sensitive surface formed by the touchscreen.

Device 100 also includes a power system 162 for powering various components. Power system 162 optionally includes a power management system, one or more power sources (eg, batteries, alternating current (AC)), recharging systems, power failure detection circuitry, power converters or inverters, power status indicators (eg, , Light Emitting Diodes (LEDs), and any other components associated with the generation, management and distribution of power in portable devices.

Device 100 optionally also includes one or more optical sensors 164 . FIG. 1A shows an optical sensor coupled to optical sensor controller 158 in I/O subsystem 106 . Optical sensor 164 optionally includes a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) phototransistor. Optical sensor 164 receives light projected through one or more lenses from the environment and converts the light into data representing an image. In conjunction with imaging module 143 (also referred to as a camera module), optical sensor 164 optionally captures still images or video. In some embodiments, the optical sensor is located on the rear of the device 100, opposite the touchscreen display 112 on the front of the device, enabling the touchscreen display to be used as a viewfinder for still and/or video image capture. In some embodiments, an optical sensor is located on the front of the device so that an image of the user is optionally captured for use in the videoconference while the user is viewing other videoconference participants on the touch screen display. In some embodiments, the position of the optical sensor 164 can be changed by the user (eg, by rotating the lens and sensor in the device housing), such that a single optical sensor 164 is used with a touch screen display for videoconferencing and still images and/or video Image acquisition for both.

Device 100 optionally also includes one or more depth camera sensors 175 . FIG. 1A shows a depth camera sensor coupled to depth camera controller 169 in I/O subsystem 106 . The depth camera sensor 175 receives data from the environment to create three-dimensional models of objects (eg, faces) within the scene from a viewpoint (eg, a depth camera sensor). In some embodiments, in conjunction with imaging module 143 (also referred to as a camera module), depth camera sensor 175 is optionally used to determine depth maps for different portions of the image captured by imaging module 143 . In some embodiments, a depth camera sensor is located on the front of the device 100 so that an image of the user with depth information is optionally acquired for the videoconference while the user is viewing other videoconference participants on the touch screen display, and captures with Selfie of depth map data. In some embodiments, the depth camera sensor 175 is located on the rear of the device, or both the rear and front of the device 100 . In some embodiments, the position of the depth camera sensor 175 can be changed by the user (eg, by rotating the lens and sensor in the device housing) such that the depth camera sensor 175 is used with a touch screen display for videoconferencing and still images and/or Video images were acquired for both.

In some implementations, a depth map (eg, a depth map image) contains information (eg, values) related to the distances of objects in a scene from a viewpoint (eg, camera, optical sensor, depth camera sensor). In one embodiment of the depth map, each depth pixel defines where in the Z-axis of the viewpoint its corresponding two-dimensional pixel is located. In some implementations, the depth map consists of pixels, where each pixel is defined by a value (eg, 0 to 255). For example, a value of "0" represents the pixel located in the "3D" scene furthest from the viewpoint (eg, camera, optical sensor, depth camera sensor), and a value of "255" represents the pixel located in the "3D" scene closest to the viewpoint . In other embodiments, the depth map represents the distance between objects in the scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depths of various features of the object of interest in the depth camera's field of view (eg, the relative depths of the eyes, nose, mouth, ears of the user's face). In some embodiments, the depth map includes information that enables the device to determine the contour of the object of interest in the z-direction.

Device 100 optionally also includes one or more contact intensity sensors 165 . FIG. 1A shows a contact intensity sensor coupled to intensity sensor controller 159 in I/O subsystem 106 . Contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (eg, for measuring touch force (or pressure) sensor) of a contact on a sensitive surface. Contact strength sensor 165 receives contact strength information (eg, pressure information or a surrogate for pressure information) from the environment. In some implementations, at least one contact intensity sensor is juxtaposed or adjacent to the touch-sensitive surface (eg, touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the rear of the device 100 opposite the touch screen display 112 located on the front of the device 100 .

Device 100 optionally also includes one or more proximity sensors 166 . FIG. 1A shows proximity sensor 166 coupled to peripherals interface 118 . Alternatively, proximity sensor 166 is optionally coupled to input controller 160 in I/O subsystem 106 . Proximity sensor 166 optionally performs as described in the following US patent applications: 11/241,839 entitled "Proximity Detector In Handheld Device"; No. 11/240,788 entitled "Proximity Detector In Handheld Device" Proximity Detector); No. 11/620,702, entitled "Using Ambient Light Sensor To Augment Proximity Sensor Output"; No. 11/586,862, entitled "Automated Response To AndSensing Of" User Activity In Portable Devices" (Automatically Responding and Sensing User Activity in Portable Devices); and No. 11/638,251, entitled "Methods And Systems For Automatic Configuration Of Peripherals" (Methods and Systems for Automatic Configuration of Peripherals ), these US patent applications are hereby incorporated by reference in their entirety. In some embodiments, when the multifunction device is placed near the user's ear (eg, when the user is on a phone call), the proximity sensor turns off and the touch screen 112 is disabled.

Device 100 optionally also includes one or more haptic output generators 167 . FIG. 1A shows a haptic output generator coupled to haptic feedback controller 161 in I/O subsystem 106 . Haptic output generator 167 optionally includes one or more electroacoustic devices, such as speakers or other audio components; and/or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymerizers, pressure Electrical actuators, electrostatic actuators, or other haptic output generating components (eg, components that convert electrical signals into haptic outputs on the device). Contact strength sensor 165 receives haptic feedback generation instructions from haptic feedback module 133 and generates haptic output on device 100 that can be felt by a user of device 100 . In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (eg, touch-sensitive display system 112 ), and optionally by vertically (eg, in/out of the surface of device 100 ) ) or move the touch-sensitive surface laterally (eg, back and forth in the same plane as the surface of device 100 ) to generate the haptic output. In some embodiments, at least one tactile output generator sensor is located on the rear of the device 100 , opposite the touch screen display 112 located on the front of the device 100 .

Device 100 optionally also includes one or more accelerometers 168 . FIG. 1A shows accelerometer 168 coupled to peripherals interface 118 . Alternatively, accelerometer 168 is optionally coupled to input controller 160 in I/O subsystem 106 . The accelerometer 168 optionally performs as described in the following US Patent Publications: US Patent Publication 20050190059 entitled "Acceleration-based Theft Detection System for Portable Electronic Devices" and US Patent Publication 20060017692 entitled "Methods And Apparatuses ForOperating A" Portable Device Based On An Accelerometer", both US patent publications are hereby incorporated by reference in their entirety. In some embodiments, information is displayed on a touch screen display in portrait or landscape view based on analysis of data received from one or more accelerometers. Device 100 optionally includes a magnetometer and a GPS (or GLONASS or other global navigation system) receiver in addition to accelerometer 168 for obtaining information about the position and orientation (eg, portrait or landscape) of device 100 .

In some embodiments, the software components stored in memory 102 include operating system 126, communication module (or instruction set) 128, contact/motion module (or instruction set) 130, graphics module (or instruction set) 132, text input A module (or set of instructions) 134 , a global positioning system (GPS) module (or set of instructions) 135 , and an application (or set of instructions) 136 . Additionally, in some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) stores device/global internal state 157, as shown in FIGS. 1A and 3 . Device/global internal state 157 includes one or more of the following: active application state, indicating which applications (if any) are currently active; display state, indicating what application, view, or other information occupies the touch screen Various areas of the display 112; sensor status, including information obtained from the device's various sensors and input control device 116; and location information about the device's position and/or attitude.

Operating system 126 (eg, Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes functions for controlling and managing general system tasks (eg, memory management, storage device control, power management, etc. ) of various software components and/or drivers, and facilitates communication between the various hardware and software components.

(Apple Inc.的商标)设备上所使用的30针连接器相同的或类似的和/或与其兼容的多针(例如，30针)连接器。Communication module 128 facilitates communication with other devices through one or more external ports 124 , and also includes various software components for processing data received by RF circuitry 108 and/or external ports 124 . External port 124 (eg, Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly through a network (eg, the Internet, wireless LAN, etc.). In some embodiments, the external port is connected to

(trademark of Apple Inc.) a multi-pin (eg, 30-pin) connector identical to or similar to and/or compatible with the 30-pin connector used on the device.

Contact/motion module 130 optionally detects contact with touch screen 112 (in conjunction with display controller 156) and other touch-sensitive devices (eg, a trackpad or physical click wheel). The contact/motion module 130 includes various software components for performing various operations related to the detection of contacts, such as determining whether a contact has occurred (eg, detecting a finger press event), determining the strength of the contact (eg, the force of the contact) or pressure, or a surrogate for force or pressure of the contact), determine if there is movement of the contact and track movement on the touch-sensitive surface (e.g., detect one or more finger drag events), and determine if the contact has ceased ( For example, detecting a finger lift event or contact break). The contact/motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the velocity (magnitude), velocity (magnitude and direction) and/or acceleration (change in magnitude and/or direction) of the contact point, the movement of the contact point being determined by a series of Contact data representation. These operations are optionally applied to single-point contact (eg, single-finger contact) or multiple simultaneous contacts (eg, "multi-touch"/multi-finger contact). In some implementations, contact/motion module 130 and display controller 156 detect contact on the trackpad.

In some embodiments, the contact/motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by the user (eg, determining whether the user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds are determined from software parameters (eg, the intensity thresholds are not determined by the activation thresholds of a particular physical actuator and can be determined without changing the physical hardware of the device 100 ). adjusted). For example, the mouse "click" threshold for a trackpad or touchscreen can be set to any of a large range of predefined thresholds without changing the trackpad or touchscreen display hardware. Additionally, in some implementations, a user of the device is provided for adjusting one or more of a set of intensity thresholds (eg, by adjusting individual intensity thresholds and/or by utilizing a system-level click on the "intensity" parameter software settings to adjust multiple intensity thresholds at once).

The contact/motion module 130 optionally detects a user's gesture input. Different gestures on a touch-sensitive surface have different contact patterns (eg, different movements, timings, and/or intensities of detected contacts). Thus, gestures are optionally detected by detecting specific contact patterns. For example, detecting a finger tap gesture includes detecting a finger down event and then detecting a finger up (lift off) at the same location (or substantially the same location) as the finger down event (eg, at the location of the icon) event. As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger down event, then detecting one or more finger drag events, and then detecting a finger up (lift off) event.

Graphics module 132 includes various known software components for rendering and displaying graphics on touch screen 112 or other display, including for changing the visual impact (eg, brightness, transparency, saturation, contrast, or other) of the displayed graphics visual properties). As used herein, the term "graphics" includes any object that can be displayed to a user including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, and the like.

In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. Graphics module 132 receives one or more codes from an application program or the like specifying graphics to be displayed, along with coordinate data and other graphics attribute data if necessary, and then generates screen image data for output to display controller 156 .

Haptic feedback module 133 includes various software components for generating instructions used by haptic output generator 167 to generate haptic output at one or more locations on device 100 in response to user interaction with device 100 .

A text entry module 134, optionally a component of the graphics module 132, is provided for use in various applications (eg, contacts 137, email 140, IM 141, browser 147, and any other application requiring text entry) Soft keyboard for entering text.

GPS module 135 determines the location of the device and provides this information for use in various applications (eg, to phone 138 for use in location-based dialing; to camera 143 as picture/video metadata; and Provided to applications that provide location-based services, such as the Weather applet, the Local Yellow Pages applet, and the Maps/Navigation applet).

Application 136 optionally includes the following modules (or sets of instructions), or subsets or supersets thereof:

a contacts module 137 (sometimes referred to as an address book or contact list);

· Telephone module 138;

· Video conferencing module 139;

· Email client module 140;

· Instant Messaging (IM) module 141;

fitness support module 142;

a camera module 143 for still and/or video images;

· Image management module 144;

· Video player module;

·Music player module;

Browser module 147;

· Calendar module 148;

- a desktop applet module 149, which optionally includes one or more of the following: a weather desktop applet 149-1, a stock market desktop applet 149-2, a calculator desktop applet 149-3, an alarm clock desktop applet 149-4, dictionary desktop applet 149-5, and other desktop applets acquired by the user, and desktop applet 149-6 created by the user;

a widget creator module 150 for forming user created widgets 149-6;

·Search module 151;

a video and music player module 152, which incorporates a video player module and a music player module;

· Notepad module 153;

the map module 154; and/or

• Online video module 155.

Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA enabled applications, encryption, digital rights management, speech recognition and voice copying.

In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, contacts module 137 is optionally used to manage an address book or contact list (eg, stored in memory 102 or Contacts module 137 in application internal state 192 in 370), including: adding one or more names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names associating images with names; categorizing and categorizing names; providing phone numbers or email addresses to initiate and/or facilitate communication via phone 138, video conferencing module 139, email 140 or instant message 141;

In conjunction with RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, telephone module 138 is optionally used to input and telephone numbers Corresponding character sequences, accessing one or more phone numbers in the contacts module 137, modifying an entered phone number, dialing the corresponding phone number, having a conversation, and disconnecting or hanging up when the conversation is complete. As mentioned above, wireless communication optionally uses any of a variety of communication standards, protocols, and techniques.

Incorporates RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact/motion module 130, graphics module 132, text input module 134, contacts Module 137 and telephony module 138, videoconferencing module 139 includes executable instructions for initiating, conducting and terminating a videoconference between a user and one or more other participants in accordance with user instructions.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, email client module 140 includes functions for creating, sending, receiving, and managing electronic mail in response to user instructions Executable instructions for mail. In conjunction with the image management module 144 , the email client module 140 makes it very easy to create and send emails with still or video images captured by the camera module 143 .

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for entering a sequence of characters corresponding to an instant message , modify previously entered characters, transmit the corresponding instant message (eg, using the Short Message Service (SMS) or Multimedia Messaging Service (MMS) protocols for telephony-based instant messaging, or XMPP, SIMPLE, or IMPS for Internet instant messaging), receive instant messages, and view received instant messages. In some embodiments, the transmitted and/or received instant messages optionally include graphics, photos, audio files, video files, and/or other attachments supported in MMS and/or Enhanced Messaging Services (EMS). As used herein, "instant messaging" refers to both phone-based messages (eg, messages sent using SMS or MMS) and Internet-based messages (eg, messages sent using XMPP, SIMPLE, or IMPS).

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, fitness support module 142 includes creating a fitness (eg, , with time, distance, and/or calorie burn goals); communicate with fitness sensors (sports equipment); receive fitness sensor data; calibrate sensors used to monitor fitness; select and play music for fitness; and display, Store and transfer fitness data.

In conjunction with touch screen 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact/motion module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for the following operations : capture still images or videos (including video streams) and store them in memory 102 , modify characteristics of still images or videos, or delete still images or videos from memory 102 .

In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (eg, editing) or otherwise manipulate, tag, delete, present (eg, in a digital slideshow or photo album), and store still and/or video images.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, browser module 147 includes a browser module for browsing the Internet (including searching, linking to, receiving and Executable instructions to display a web page or parts thereof, as well as attachments and other files linked to the web page.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes a Executable instructions to create, display, modify, and store a calendar and data associated with the calendar (eg, calendar entries, to-dos, etc.).

In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact/motion module 130, graphics module 132, text input module 134, and browser module 147, the desktop applet module 149 is a miniature program that is optionally downloaded and used by the user Applications (eg, Weather applet 149-1, Stocks applet 149-2, Calculator applet 149-3, Alarm clock applet 149-4, and Dictionary applet 149-5) or created by the user A mini-application (eg, a user-created desktop applet 149-6). In some embodiments, the desktop applet includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, a desktop applet includes an XML (Extensible Markup Language) file and a JavaScript file (eg, a Yahoo! desktop applet).

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, and browser module 147, desktop applet creator module 150 is optionally used by a user to create a desktop applet Programs (eg, turn user-specified parts of web pages into desktop applets).

In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, search module 151 includes a search module 151 for searching memory 102 for matching one or more search criteria (eg, one or more according to user instructions) Executable instructions for text, music, sound, images, video, and/or other files for multiple user-specified search terms).

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, audio circuitry 110, speakers 111, RF circuitry 108, and browser module 147, video and music player module 152 includes a video and music player module that allows the user to download and playback to Executable instructions for recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and for displaying, rendering, or otherwise playing back video (e.g., on touch screen 112 or on Executable instructions on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, notepad module 153 includes executable instructions to create and manage notes, to-dos, etc., according to user instructions.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 is optionally configured to receive, Display, modify, and store maps and data associated with maps (eg, driving directions, data related to stores and other points of interest at or near a particular location, and other location-based data).

Incorporating touch screen 112, display controller 156, touch/motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140 and browser module 147, online video module 155 includes instructions for allowing the user to access, browse, receive (eg, by streaming and/or download), playback (eg, on a touch screen or on an external display connected via external port 124), Send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send links to specific online videos. Additional descriptions of online video applications can be found in U.S. Provisional Patent Application 60/936,562, filed June 20, 2007, and entitled "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos" and filed on December 31, 2007 Found in US patent application Ser. No. 11/968,067, entitled "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," the contents of which are hereby incorporated by reference in their entirety.

Each of the modules and applications described above corresponds to executable instructions for performing one or more of the functions described above and the methods described in this patent application (eg, the computer-implemented methods and other information processing methods described herein) set. These modules (eg, sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. For example, a video player module is optionally combined with a music player module into a single module (eg, video and music player module 152 in FIG. 1A ). In some embodiments, memory 102 optionally stores a subset of the modules and data structures described above. In addition, memory 102 optionally stores additional modules and data structures not described above.

In some embodiments, device 100 is the only device on which operations of a predefined set of functions are performed through a touch screen and/or trackpad. By using a touch screen and/or trackpad as the primary input control device for operating device 100, the number of physical input control devices (eg, push buttons, dials, etc.) on device 100 is optionally reduced.

A predefined set of functions performed exclusively through a touch screen and/or trackpad optionally includes navigation between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to the main, home or root menu. In such embodiments, a "menu button" is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device rather than a touchpad.

FIG. 1B is a block diagram illustrating exemplary components for event processing, according to some embodiments. In some embodiments, memory 102 (FIG. 1A) or memory 370 (FIG. 3) includes event classifier 170 (eg, in operating system 126) and corresponding application 136-1 (eg, applications 137-151 previously described) , 155, any of 380 to 390 applications).

The event classifier 170 receives the event information and determines the application 136-1 to which the event information is to be delivered and the application view 191 of the application 136-1. Event classifier 170 includes event monitor 171 and event dispatcher module 174 . In some embodiments, application 136-1 includes application internal state 192 that indicates one or more current applications displayed on touch-sensitive display 112 when the application is active or executing view. In some embodiments, device/global internal state 157 is used by event classifier 170 to determine which application(s) are currently active, and application internal state 192 is used by event classifier 170 to determine which event information to deliver to the application view 191.

In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resume information to be used when the application 136-1 resumes execution, an indication that the application 136-1 is being executed -1 displayed information or user interface state information ready for information displayed by the application, a state queue for enabling the user to return to a previous state or view of the application 136-1, and previous Action's repeat/undo queue.

Event monitor 171 receives event information from peripherals interface 118 . Event information includes information about sub-events (eg, user touches on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I/O subsystem 106 or sensors such as proximity sensor 166, accelerometer 168, and/or microphone 113 (through audio circuitry 110). The information that peripherals interface 118 receives from I/O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, peripheral interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (eg, receiving input above a predetermined noise threshold and/or receiving input for longer than a predetermined duration).

In some embodiments, the event classifier 170 also includes a hit view determination module 172 and/or an active event identifier determination module 173 .

When touch-sensitive display 112 displays more than one view, hit view determination module 172 provides a software process for determining where a sub-event has occurred within one or more views. Views are made up of controls and other elements that the user can see on the display.

Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which the touch is detected optionally corresponds to a programmatic level within the application's programmatic or view hierarchy. For example, the lowest horizontal view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct input is optionally determined based at least in part on the hit view of the initial touch that begins based on touch gesture.

The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should process sub-events. In most cases, the hit view is the lowest level view in which the initiating sub-event (eg, the first sub-event in a sequence of sub-events forming an event or potential event) occurs. Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

The active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is completely localized to the area associated with a particular view, views higher in the hierarchy will remain the actively participating views.

Event dispatcher module 174 dispatches event information to event recognizers (eg, event recognizer 180). In embodiments that include active event recognizer determination module 173 , event dispatcher module 174 delivers event information to event recognizers determined by active event recognizer determination module 173 . In some embodiments, the event dispatcher module 174 stores event information in an event queue, which is retrieved by the corresponding event receiver 182 .

In some embodiments, operating system 126 includes event classifier 170 . Alternatively, application 136 - 1 includes event classifier 170 . In other embodiments, event classifier 170 is a stand-alone module, or part of another module (such as contact/motion module 130 ) that is stored in memory 102 .

In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, wherein each application view includes an application for processing that occurs within a corresponding view of the application's user interface. Instructions for touch events. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the respective application view 191 includes a plurality of event recognizers 180 . In other embodiments, one or more of the event recognizers 180 are part of a separate module, a higher level such as the user interface toolkit or the application 136-1 from which it inherits methods and other properties Object. In some embodiments, respective event handlers 190 include one or more of: data updater 176 , object updater 177 , GUI updater 178 , and/or event data 179 received from event classifier 170 . Event handler 190 optionally utilizes or invokes data updater 176 , object updater 177 or GUI updater 178 to update application internal state 192 . Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190 . Additionally, in some embodiments, one or more of data updater 176 , object updater 177 , and GUI updater 178 are included in respective application views 191 .

A corresponding event recognizer 180 receives event information (eg, event data 179 ) from the event classifier 170 and identifies the event from the event information. Event recognizer 180 includes event receiver 182 and event comparator 184 . In some embodiments, event recognizer 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

Event receiver 182 receives event information from event classifier 170 . The event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (eg, from a portrait orientation to a landscape orientation, or vice versa), and the event information includes a current orientation (also referred to as a device pose) about the device ) corresponding information.

The event comparator 184 compares the event information to predefined event or sub-event definitions and determines an event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, event comparator 184 includes event definition 186 . Event definitions 186 contain definitions of events (eg, predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, sub-events in event (187) include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double click on the displayed object. For example, a double tap includes a first touch of a predetermined duration on the displayed object (touch start), a first lift off of a predetermined duration (touch end), a second touch of a predetermined duration on the displayed object ( touch start) and a second lift-off (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on the displayed object. For example, dragging includes a predetermined duration of touch (or contact) on the displayed object, movement of the touch on the touch-sensitive display 112, and lift-off (end of touch) of the touch. In some embodiments, the event also includes information for one or more associated event handlers 190 .

In some embodiments, event definitions 187 include definitions of events for corresponding user interface objects. In some embodiments, the event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user interface objects A user interface object is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the results of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

In some embodiments, the definition of the corresponding event (187) also includes a delay action that delays delivery of event information until it has been determined whether the sequence of sub-events does or does not correspond to the event type of the event recognizer.

When the corresponding event recognizer 180 determines that the sequence of sub-events does not match any of the events in the event definition 186, the corresponding event recognizer 180 enters the event impossible, event failed, or event end state, after which subsequent touch-based gestures are ignored sub-event. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

In some embodiments, respective event recognizers 180 include metadata 183 with configurable attributes, flags, and/or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable attributes, flags, and/or lists that indicate whether sub-events are delivered to different levels in the view or programmatic hierarchy.

In some embodiments, when one or more specific sub-events of an event are identified, the respective event recognizer 180 activates the event handler 190 associated with the event. In some embodiments, the respective event recognizer 180 delivers event information associated with the event to the event handler 190 . Activating the event handler 190 differs from sending (and deferred sending) sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag receives the flag and executes a predefined procedure.

In some embodiments, the event delivery instructions 188 include sub-event delivery instructions that deliver event information about the sub-event without activating the event handler. Instead, sub-event delivery instructions deliver event information to event handlers associated with the sub-event sequence or to actively participating views. Event handlers associated with sub-event sequences or with actively participating views receive event information and execute predetermined procedures.

In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, the data updater 176 updates the phone numbers used in the contacts module 137, or stores video files used in the video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares and sends the display information to graphics module 132 for display on a touch-sensitive display.

In some embodiments, event handler 190 includes, or has access to, data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176 , object updater 177 , and GUI updater 178 are included in a single module of respective application 136 - 1 or application view 191 . In other embodiments, they are included in two or more software modules.

It should be understood that the above discussion regarding event handling of user touches on touch-sensitive displays also applies to other forms of user input utilizing input devices to operate multifunction device 100, not all of which are initiated on a touch screen. For example, mouse movements and mouse button presses optionally in conjunction with single or multiple keyboard presses or holds; contact movements on a trackpad, such as taps, drags, scrolls, etc.; stylus input; devices movements of; verbal commands; detected eye movements; biometric inputs; and/or any combination thereof are optionally used as inputs corresponding to sub-events that define the event to be identified.

FIG. 2 shows a portable multifunction device 100 with a touch screen 112 in accordance with some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200 . In this embodiment, as well as in other embodiments described below, the user can use one or more fingers 202 (not drawn to scale in the drawings) or one or more styluses 203 (not drawn to scale in the drawings), for example. (not drawn to scale) make a gesture on the graphics to select one or more of these graphics. In some embodiments, selection of one or more graphics will occur when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (left to right, right to left, up and/or down) and/or has occurred with device 100 Scrolling of the touching finger (right to left, left to right, up and/or down). In some implementations or in some cases, inadvertent contact with a graphic does not select the graphic. For example, when the gesture corresponding to the selection is a tap, a swipe gesture to swipe over an application icon optionally does not select the corresponding application.

Device 100 optionally also includes one or more physical buttons, such as a "home" or menu button 204 . As previously mentioned, menu button 204 is optionally used to navigate to any application 136 in a set of applications optionally executed on device 100 . Alternatively, in some embodiments, the menu buttons are implemented as soft keys displayed in a GUI on the touch screen 112 .

In some embodiments, the device 100 includes a touch screen 112, a menu button 204, a push button 206 for turning the device on/off and for locking the device, one or more volume adjustment buttons 208, a Subscriber Identity Module (SIM) card Slot 210 , headset jack 212 and docking/charging external port 124 . A push button 206 is optionally used to turn the device on/off by depressing the button and keeping the button depressed for a predefined time interval; Release the button before the past to lock the device; and/or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts verbal input through the microphone 113 for activating or deactivating some functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contacts on touch screen 112 and/or one or more haptic output generators 167 for generating haptic output for a user of device 100 .

3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia player device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (eg, a home controller or industrial using the controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects and controls communications between system components. Device 300 includes an input/output (I/O) interface 330 having a display 340, which is typically a touch screen display. I/O interface 330 also optionally includes keyboard and/or mouse (or other pointing device) 350 and trackpad 355, haptic output generator 357 for generating haptic output on device 300 (eg, similar to the references above tactile output generator 167 described in FIG. 1A ), sensors 359 (eg, optical sensors, acceleration sensors, proximity sensors, touch sensitive sensors, and/or contact intensity sensors (similar to touch intensity sensor 165 described above with reference to FIG. 1A ) ). Memory 370 includes high speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and optionally nonvolatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory memory devices or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310 . In some embodiments, memory 370 stores programs, modules and data structures similar to, or subsets of, those stored in memory 102 of portable multifunction device 100 (FIG. 1A). In addition, memory 370 optionally stores additional programs, modules and data structures that are not present in memory 102 of portable multifunction device 100 . For example, memory 370 of device 300 optionally stores drawing module 380, presentation module 382, word processing module 384, website creation module 386, disk editing module 388, and/or spreadsheet module 390, while portable multifunction device 100 (FIG. 1A) ) memory 102 optionally does not store these modules.

Each of the above-described elements in FIG. 3 are optionally stored in one or more of the previously mentioned memory devices. Each of the aforementioned modules corresponds to a set of instructions for performing the aforementioned functions. The above-described modules or programs (eg, sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores subsets of the modules and data structures described above. In addition, memory 370 optionally stores additional modules and data structures not described above.

Attention is now turned to an implementation of a user interface optionally implemented on, for example, the portable multifunction device 100 .

4A illustrates an exemplary user interface for a menu of applications on portable multifunction device 100, according to some embodiments. A similar user interface is optionally implemented on device 300 . In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

one or more signal strength indicators 402 for one or more wireless communications, such as cellular and Wi-Fi signals;

·time 404;

·Bluetooth indicator 405;

battery status indicator 406;

- A tray 408 with icons of frequently used applications such as:

o an icon 416 of the phone module 138 labeled "Phone," which optionally includes an indicator 414 of the number of missed calls or voice messages;

o an icon 418 of the email client module 140 labeled "Mail" which optionally includes an indicator 410 of the number of unread emails;

o icon 420 of browser module 147 labeled "Browser"; and

o an icon 422 labeled "iPod" for the video and music player module 152 (also known as the iPod (trademark of Apple Inc.) module 152); and

· Icons for other applications, such as:

o the icon 424 of the IM module 141 labeled "Messages";

o icon 426 of calendar module 148 labeled "Calendar";

o icon 428 of the image management module 144 labeled "Photos";

o the icon 430 of the camera module 143 labeled "Camera";

o an icon 432 of the online video module 155 labeled "Online Video";

ο the icon 434 of the stock market desktop applet 149-2 marked as "stock market";

o the icon 436 of the map module 154 labeled "Map";

ο the icon 438 of the weather desktop applet 149-1 labeled "weather";

o the icon 440 of the Alarm Clock applet 149-4 labeled "Clock";

o icon 442 of fitness support module 142 labeled "fitness support";

o icon 444 of notepad module 153 labeled "notepad"; and

o Icon 446 labeled "Settings" for a settings application or module, which provides access to settings for device 100 and its various applications 136.

It should be noted that the icon labels shown in FIG. 4A are exemplary only. For example, the icon 422 of the video and music player module 152 is labeled "music" or "music player." Other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

4B shows an example on a device (eg, device 300 of FIG. 3 ) having a touch-sensitive surface 451 (eg, tablet or trackpad 355 of FIG. 3 ) separate from display 450 (eg, touchscreen display 112 ) Sexual user interface. Device 300 also optionally includes one or more contact intensity sensors (eg, one or more of sensors 359 ) for detecting the intensity of a contact on touch-sensitive surface 451 and/or for a user of device 300 One or more haptic output generators 357 that generate haptic output.

Although some of the following examples will be given with reference to input on touchscreen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, as shown in FIG. shown in 4B. In some implementations, the touch-sensitive surface (eg, 451 in FIG. 4B ) has a major axis (eg, 452 in FIG. 4B ) that corresponds to the major axis (eg, 453 in FIG. 4B ) on the display (eg, 450 ) . According to these embodiments, the device detects contact with touch-sensitive surface 451 (eg, in 460 and 462). In this way, when the touch-sensitive surface (eg, 451 in FIG. 4B ) is separated from the display of the multifunction device (450 in FIG. 4B ), user input detected by the device on the touch-sensitive surface (eg, contacts 460 and 462 ) and their movements) are used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

Additionally, while the following examples are given primarily with reference to finger inputs (eg, finger touch, single finger tap gestures, finger swipe gestures), it should be understood that in some embodiments one of these finger inputs The or multiple finger input is replaced by input from another input device (eg, mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (eg, rather than a contact), followed by movement of the cursor (eg, rather than a contact) along the swipe's path. As another example, a tap gesture is optionally replaced by a mouse click (eg, instead of detecting a contact, followed by ceasing to detect the contact) while the cursor is over the location of the tap gesture. Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mouse and finger contacts are optionally used simultaneously.

FIG. 5A shows an example personal electronic device 500 . Device 500 includes body 502 . In some embodiments, device 500 may include some or all of the features described with respect to devices 100 and 300 (eg, FIGS. 1A-4B ). In some implementations, the device 500 has a touch-sensitive display screen 504 hereinafter referred to as a touch screen 504 . Alternatively or in addition to touch screen 504, device 500 also has a display and a touch-sensitive surface. As with devices 100 and 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors for detecting the intensity of contact (eg, touch) applied. One or more intensity sensors of touch screen 504 (or touch-sensitive surface) may provide output data indicative of the intensity of the touch. The user interface of device 500 can respond to touches based on its intensity, which means that touches of different intensities can invoke different user interface operations on device 500 .

Exemplary techniques for detecting and processing touch intensity can be found in, for example, the following related patent applications: International patent entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application" filed May 8, 2013 Application Serial No. PCT/US2013/040061, published as WIPO Patent Publication No. WO/2013/169849; and titled "Device, Method, and Graphical User Interface for Transitioning BetweenTouch Input to Display Output Relationships" filed on November 11, 2013 International Patent Application Serial No. PCT/US2013/069483, published as WIPO Patent Publication No. WO/2014/105276, each of which is hereby incorporated by reference in its entirety.

In some embodiments, device 500 has one or more input mechanisms 506 and 508 . Input mechanisms 506 and 508 (if included) may be in physical form. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, may allow device 500 to be attached to, for example, hats, glasses, earrings, necklaces, shirts, jackets, bracelets, watch straps, bracelets, pants, belts, shoes, wallets, backpacks, and the like. These attachment mechanisms allow the user to wear the device 500 .

FIG. 5B shows an example personal electronic device 500 . In some embodiments, apparatus 500 may include some or all of the components described with respect to FIGS. 1A , 1B and 3 . Device 500 has a bus 512 that operatively couples I/O section 514 with one or more computer processors 516 and memory 518 . The I/O portion 514 may be connected to a display 504, which may have a touch sensitive component 522 and optionally an intensity sensor 524 (eg, a touch intensity sensor). Additionally, the I/O section 514 may be connected to the communication unit 530 for receiving application programs and operating system data using Wi-Fi, Bluetooth, Near Field Communication (NFC), cellular, and/or other wireless communication technologies. Device 500 may include input mechanisms 506 and/or 508 . For example, the input mechanism 506 is optionally a rotatable input device or a depressible input device as well as a rotatable input device. In some examples, input mechanism 508 is optionally a button.

In some examples, input mechanism 508 is optionally a microphone. Personal electronic device 500 optionally includes various sensors, such as GPS sensor 532, accelerometer 534, orientation sensor 540 (eg, a compass), gyroscope 536, motion sensor 538, and/or combinations thereof, all of which are operable Connected to the I/O section 514.

The memory 518 of the personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which, when executed by the one or more computer processors 516 , for example, may cause a computer The processor performs the following techniques, including processes 700, 900, 1100, 1300, and 1500 (FIGS. 7A-7F, 9A-9K, 11A-11F, 13A-13D, and 15A-15C). A computer-readable storage medium can be any medium that can tangibly contain or store computer-executable instructions for use by or in connection with the instruction execution systems, apparatus, and devices. In some examples, the storage medium is a transitory computer-readable storage medium. In some examples, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and/or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD or Blu-ray technology, and persistent solid state memory such as flash memory, solid state drives, and the like. Personal electronic device 500 is not limited to the components and configuration of FIG. 5B, but may include other components or additional components in a variety of configurations.

As used herein, the term "enabling representation" refers to a user-interactive graphical user interface object optionally displayed on the display screen of devices 100, 300, and/or 500 (FIGS. 1A, 3, and 5A-5B). . For example, images (eg, icons), buttons, and text (eg, hyperlinks) optionally each constitute an affordance.

As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" so that when the cursor is over a particular user interface element (eg, a button, window, slider, or other user interface element) the touch-sensitive surface is In the event that an input (eg, a press input) is detected on the touchpad 355 in FIG. 3 or the touch-sensitive surface 451 in FIG. 4B , that particular user interface element is adjusted according to the detected input. In some implementations including a touchscreen display (eg, touch-sensitive display system 112 in FIG. 1A or touchscreen 112 in FIG. 4A ) that enables direct interaction with user interface elements on the touchscreen display, detected The contact acts as a "focus selector" so that when an input (eg, a press input by a contact) is detected at the location of a particular user interface element (eg, a button, window, slider, or other user interface element) on the touchscreen display ), the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of the user interface to another area of the user interface without corresponding movement of the cursor or movement of a contact on the touchscreen display (eg, by using the tab or arrow keys to change focus from one button to another); in these implementations, the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically controlled by the user in order to deliver the user's intended interaction with the user interface (eg, by indicating to the device elements with which the user of the user interface expects to interact) Interface elements (or contacts on touchscreen displays). For example, when a press input is detected on a touch-sensitive surface (eg, a touchpad or touchscreen), the position of the focus selector (eg, cursor, touch, or selection box) over the corresponding button will indicate that the user expects the corresponding button to be activated (and not other user interface elements shown on the device display).

As used in the specification and claims, the term "characteristic strength" of a contact refers to a characteristic of a contact based on one or more strengths of the contact. In some embodiments, the characteristic intensity is based on a plurality of intensity samples. Feature intensities are optionally based on relative to predefined events (e.g., after a contact is detected, before a contact lift-off is detected, before or after a contact starts to move, before a contact ends, before a contact is detected. before or after an increase in intensity and/or before or after a decrease in intensity of the contact is detected) for a predetermined time period (eg, 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, A predefined number of intensity samples or a set of intensity samples collected during 5 seconds, 10 seconds). The characteristic strength of the contact is optionally based on one or more of the following: the maximum value of the strength of the contact, the mean value of the strength of the contact, the mean value of the strength of the contact, the value at the top 10% of the strength of the contact, The half maximum value of the contact strength, the 90% maximum value of the contact strength, etc. In some embodiments, the duration of the contact is used in determining the characteristic intensity (eg, when the characteristic intensity is an average of the intensity of the contact over time). In some embodiments, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an action. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, a contact whose characteristic intensity does not exceed a first threshold results in a first operation, a contact whose characteristic intensity exceeds a first intensity threshold but does not exceed a second intensity threshold results in a second operation, and a touch whose characteristic intensity exceeds the second threshold results in a second operation. Third operation. In some embodiments, a comparison between feature strengths and one or more thresholds is used to determine whether to perform one or more operations (eg, perform the corresponding operation or abort performing the corresponding operation) rather than for determining whether to perform the first The operation is still the second operation.

5C illustrates detection of multiple contacts 552A-552E on touch-sensitive display screen 504 using multiple intensity sensors 524A-524D. FIG. 5C also includes an intensity graph showing current intensity measurements of intensity sensors 524A-524D against intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are each 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are each 7 intensity units. In some implementations, the cumulative intensity is the sum of the intensity measurements of the plurality of intensity sensors 524A-524D, in this example 32 intensity units. In some embodiments, each contact is assigned a corresponding intensity, ie, a portion of the cumulative intensity. 5D shows the assignment of cumulative intensities to contacts 552A-552E based on their distance from the center of force 554. In this example, each of contacts 552A, 552B, and 552E is assigned an intensity of 8 units of cumulative intensity for the contact, and each of contacts 552C and 552D is assigned an intensity of 4 of the cumulative intensity. The strength of the contact in units of strength. More generally, in some implementations, each contact j is assigned a corresponding intensity Ij according to a predefined mathematical function Ij=A·(Dj/ΣDi), which is part of the cumulative intensity A, where Dj is the corresponding contact j is the distance from the force center, and ΣDi is the sum of the distances from the force center for all corresponding contacts (eg, i=1 to last). The operations described with reference to FIGS. 5C-5D may be performed using an electronic device similar to or equivalent to device 100, 300 or 500. In some embodiments, the characteristic strength of a contact is based on one or more strengths of the contact. In some embodiments, an intensity sensor is used to determine a single characteristic intensity (eg, a single characteristic intensity of a single contact). It should be noted that the intensity maps are not part of the displayed user interface, but are included in Figures 5C-5D to assist the reader.

In some implementations, a portion of the gesture is identified for use in determining feature strength. For example, the touch-sensitive surface optionally receives successive swipe contacts that transition from a starting position to an ending position where the contact strength increases. In this example, the characteristic strength of the contact at the end location is optionally based on only a portion of the continuous swipe contact, rather than the entire swipe contact (eg, only the portion of the swipe contact at the end location). In some embodiments, a smoothing algorithm is optionally applied to the strength of the swipe contact prior to determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and/or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow spikes or dips in the intensity of the swipe contact for the purpose of determining the intensity of a feature.

The contact intensity on the touch-sensitive surface is optionally characterized relative to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and/or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device would perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep-press intensity threshold corresponds to the intensity at which the device will perform a different operation than that normally associated with clicking a button of a physical mouse or trackpad. In some embodiments, when a contact is detected with a characteristic intensity below a light press intensity threshold (eg, and above a nominal contact detection intensity threshold, contacts below the nominal touch detection intensity threshold are no longer detected), The device will move the focus selector based on the movement of the contact on the touch-sensitive surface without performing the operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across sets of user interface drawings.

The increase in contact feature strength from a strength below the light press strength threshold to a strength between the light press strength threshold and the deep press strength threshold is sometimes referred to as a "light press" input. The increase in contact feature strength from a strength below the deep press strength threshold to a strength above the deep press strength threshold is sometimes referred to as a "deep press" input. The increase in contact feature strength from a strength below the contact detection strength threshold to an strength between the contact detection strength threshold and the light press strength threshold is sometimes referred to as detecting a contact on the touch surface. The decrease in contact feature intensity from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold is sometimes referred to as detecting contact lift-off from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

In some embodiments described herein, one or more operations are performed in response to detecting a gesture comprising a corresponding press input or in response to detecting a corresponding press input performed with a corresponding contact (or contacts), wherein at least A corresponding press input is detected based in part on detecting an increase in the intensity of the contact (or contacts) above a press input intensity threshold. In some implementations, the corresponding operation is performed in response to detecting that the intensity of the corresponding contact increases above a press input intensity threshold (eg, a "downstroke" of the corresponding press input). In some embodiments, the press input includes an increase in the intensity of the corresponding contact above a press input intensity threshold and a subsequent decrease in intensity of the contact below the press input intensity threshold, and a subsequent decrease in intensity of the corresponding contact in response to detection To below the press input threshold (eg, the "up stroke" of the corresponding press input), the corresponding operation is performed.

Figures 5E-5H illustrate the detection of a gesture comprising an increase in the intensity of contact 562 from an intensity below the light press intensity threshold (eg, "IT _L ") in Figure 5E to above that in Figure 5H The intensity of the deep press intensity threshold (eg, "IT _D ") corresponds to the press input. On the displayed user interface 570 including the application icons 572A-572D displayed in the predefined area 574, while the cursor 576 is displayed over the application icon 572B corresponding to application 2, the detection is performed on the touch-sensitive surface 560 using the contact 562 gesture. In some implementations, gestures are detected on touch-sensitive display 504 . The intensity sensor detects the intensity of the contact on the touch-sensitive surface 560 . The device determines that the intensity of contact 562 peaks above a deep-press intensity threshold (eg, "IT _D "). Contact 562 is maintained on touch sensitive surface 560 . In response to detecting the gesture, and based on contacts 562 whose intensity rose above a deep-press intensity threshold (eg, "IT _D ") during the gesture, a reduced-scale representation 578A-578C ( For example, thumbnails), as shown in Figures 5F-5H. In some embodiments, the intensity is a characteristic intensity of the contact compared to one or more intensity thresholds. It should be noted that the intensity map for contact 562 is not part of the displayed user interface, but is included in Figures 5E-5H to assist the reader.

In some embodiments, the display of representations 578A-578C includes animation. For example, representation 578A is initially displayed adjacent to application icon 572B, as shown in Figure 5F. As the animation progresses, representation 578A moves up and displays representation 578B adjacent to application icon 572B, as shown in Figure 5G. Then, representation 578A moves up, 578B moves up toward representation 578A, and representation 578C is displayed near application icon 572B, as shown in Figure 5H. Representations 578A-578C form an array above icon 572B. In some embodiments, the animation progresses according to the strength of the contact 562, as shown in Figures 5F-5G, where representations 578A-578C appear and press the strength threshold (eg, "IT _D ") deeper with the strength of the contact 562 increase and move up. In some embodiments, the strength by which the animation progresses is the characteristic strength of the contact. The operations described with reference to FIGS. 5E-5H may be performed using an electronic device similar to or equivalent to device 100, 300, or 500.

In some embodiments, the device employs an intensity hysteresis to avoid accidental input sometimes referred to as "jitter", wherein the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (eg, the hysteresis intensity threshold is greater than the press input intensity threshold). The intensity threshold is X intensity units lower, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, the press input includes an increase in the strength of the corresponding contact above a press input strength threshold and a subsequent decrease in strength of the contact below a hysteresis strength threshold corresponding to the press input strength threshold, and in response to detecting The intensity to the corresponding contact then decreases below a hysteresis intensity threshold (eg, an "upstroke" of the corresponding press input) to perform the corresponding operation. Similarly, in some embodiments, the device detects an increase in contact intensity from an intensity equal to or below the hysteresis intensity threshold to an intensity equal to or above the press input intensity threshold and optionally a subsequent decrease in contact intensity to equal to or equal to the hysteresis intensity threshold. The press input is detected only at a strength lower than the hysteresis strength, and a corresponding operation is performed in response to the detection of the press input (eg, the contact strength increases or the contact strength decreases depending on the environment).

For ease of explanation, optionally, an operation performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input is triggered in response to detecting any of the following conditions: Description: Contact strength increases above a press input strength threshold, contact strength increases from a strength below a hysteresis strength threshold to a strength above a press input strength threshold, contact strength decreases below a press input strength threshold, and/or Or the contact strength decreases below a hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in examples where the operations are described as being performed in response to detecting a decrease in the intensity of the contact below a press input intensity threshold, optionally in response to detecting a decrease in the intensity of the contact below corresponding to and less than the press input Intensity Threshold Hysteresis Intensity Threshold to perform the operation.

As used herein, an "installed application" refers to a software application that has been downloaded onto an electronic device (eg, devices 100, 300, and/or 500) and is ready to be launched (eg, becomes open) on the device. In some embodiments, a downloaded application becomes an installed application using an installer that extracts program parts from the downloaded software package and integrates the extracted parts with the computer system's operating system.

As used herein, the term "open application" or "executing application" refers to a software application that has maintained state information (eg, as part of device/global internal state 157 and/or application internal state 192 ) . An open or executing application is optionally any of the following types of applications:

Active application, which is currently displayed on the display of the device on which the application is being used;

a background application (or background process) that is not currently displayed but one or more processes of that application are being handled by one or more processors; and

• A suspended or hibernated application that is not running but has state information that is stored in memory (volatile and non-volatile, respectively) and that can be used to resume execution of the application.

As used herein, the term "closed application" refers to a software application that does not have maintained state information (eg, the closed application's state information is not stored in the device's memory). Thus, closing an application includes stopping and/or removing the application's application process and removing the application's state information from the device's memory. Generally speaking, when in the first application, opening the second application does not close the first application. When the second application is displayed and the first application stops being displayed, the first application becomes a background application.

Attention is now turned to an implementation of a user interface ("UI") and associated processes implemented on an electronic device such as portable multifunction device 100, device 300, or device 500.

6A-6AF illustrate exemplary user interfaces for initiating and participating in real-time communication sessions (eg, from an instant messaging application) in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below including the processes in Figures 7A-7F.

FIG. 6A shows a device 600 with a touch-sensitive display 602 . In some embodiments, device 600 includes one or more features of devices 100 , 300 , and 500 . In some embodiments, the device 600 also includes a camera 603 that includes at least an image sensor capable of capturing data representing a portion of the spectrum (eg, visible, infrared, or ultraviolet light). In some embodiments, camera 603 includes multiple image sensors and/or other types of sensors. In addition to capturing data representing sensed light, in some embodiments, camera 603 is capable of capturing other types of data such as depth data. For example, in some embodiments, camera 603 also captures depth data using blob, time of flight, parallax, or focus based techniques. Image data captured by device 600 using camera 603 includes data corresponding to a portion of the spectrum of the scene within the camera's field of view. Additionally, in some embodiments, the captured image data also includes depth data for light data. In some other embodiments, the captured image data includes sufficient data to determine or generate depth data for the data of the portion of the spectrum.

In some examples, electronic device 600 includes a depth camera (eg, as part of camera 603 ), such as an infrared camera, a thermal imaging camera, or a combination thereof. In some examples, the device also includes a light emitting device (eg, a light projector), such as an IR floodlight, a structured light projector, or a combination thereof. Optionally, the light emitting device is used to illuminate the object during image capture by the visible light camera and the depth camera (eg, the IR camera), and the information from the depth camera and the visible light camera is used to determine different parts of the object captured by the visible light camera depth map. In some implementations, a depth map (eg, a depth map image) includes information (eg, values) related to the distances of objects in the scene from a viewpoint (eg, a camera). In one embodiment of the depth map, each depth pixel defines where in the Z-axis of the viewpoint its corresponding two-dimensional pixel is located. In some examples, the depth map consists of pixels, where each pixel is defined by a value (eg, 0 to 255). For example, a "0" value represents the pixel located furthest from the viewpoint (eg, the camera) in the "3D" scene, and a "255" value represents the pixel located closest to the viewpoint in the "3D" scene. In other examples, the depth map represents the distance between objects in the scene and the plane of the viewpoint. In some embodiments, the depth map includes information about the relative depths of various features of the object of interest in the depth camera's field of view (eg, the relative depths of the eyes, nose, mouth, ears of the user's face). In some embodiments, the depth map includes information that enables the device to determine the contour of the object of interest in the z-direction. In some embodiments, the lighting effects described herein are displayed for backward images using disparity information from two cameras (eg, two visible cameras), and depth information from the depth cameras is used in conjunction with the images from the visible cameras data for forward-facing images (eg, selfie images). In some embodiments, the same user interface is used when depth information is determined using two visible light cameras and when depth information is determined using the depth camera, so that even when using distinct techniques to determine the information used to generate the lighting effect, Provide users with a consistent experience. In some embodiments, when displaying a camera user interface with one of the lighting effects applied, the device detects the selection of a camera switch affordance and switches from a forward-facing camera (eg, a depth camera and a visible light camera) to a rear-facing camera ( For example, two visible light cameras spaced apart from each other) (or vice versa) while maintaining the display of user interface controls to apply lighting effects and replace the display of the front-facing camera's field of view with the rear-facing camera's field of view (or vice versa) ).

As shown in FIG. 6A, device 600 displays an instant messaging user interface 604 of an instant messaging application. Instant messaging user interface 604 includes a message area 606 that includes messages 608A through 608D in a messaging conversation between participant groups ("dream teams") that include three or more participants.

While displaying the instant messaging user interface 604, the device 600 receives data indicating that a real-time communication session is active (eg, an invitation to join the real-time communication session), where the real-time communication session is available to participants in the "dream team" group. In some embodiments, the real-time communication session is an audio communication session or an audio/video communication session.

In response to receiving data indicating that the real-time communication session is active, device 600 displays a notification indicating that the real-time communication session is active. As shown in Figure 6B, the notification includes notification 614 as a message from participant Pablo in message area 606 indicating that Pablo initiated a real-time communication session. Notification 614 indicates the type of real-time communication session with the text "Video" and a video camera icon. Notification 614 also indicates the number of participants currently active in the real-time communication session ("1 person is active"). In some embodiments, notification 614 indicates the names of one or more participants currently active in the real-time communication session. Notification 614 also includes a real-time communication join affordance 614A, which, when selected, causes device 600 to initiate a process for joining the real-time communication session represented by notification 614 . Techniques for joining a real-time communication session are described in more detail below.

Optionally, in response to receiving data indicating that the real-time communication session is active, device 600 generates audio output indicating that the real-time communication session is active. In some embodiments, the audio output is different from the audio output associated with the instant messaging application (eg, notifications of received messages) and the audio output associated with the telephony application (eg, ringtones for incoming calls). In some embodiments, the audio output is a mix of audio associated with the message notification and audio associated with the incoming call notification. In some embodiments, the audio output shares audio characteristics (eg, common pitch, note, rhythm) with both the audio output for phone calls and message notifications.

Also in response to receiving data indicating that the real-time communication session is active, device 600 displays real-time communication affordance 610 . In some embodiments, the real-time communication affordance 610 is displayed before the real-time communication session is initiated (eg, in a grayed-out state), and is visually emphasized (eg, highlighted) when the real-time communication session is initiated (eg, highlighted) or is modified (eg, becomes non-grayed). In some embodiments, selection of real-time communication affordance 610 or selection affordance 612 initiates a process for joining a real-time communication session. In some embodiments, notification 614 and/or real-time communication affordance 610 is displayed without audio and/or haptic output (eg, phone ringing output) (eg, because it is being displayed in an active message conversation that the user may have focused on) .

As shown in FIG. 6C , notification 614 moves up to a different location in message area 606 as additional messages are added to the message conversation in message area 606 . In some embodiments, device 600 moves notification 614 to a location outside of the message area (eg, notification 614 transitions from the message to a banner at the top of display 602 outside of message area 606).

In some embodiments, device 600 moves notification 614 in response to meeting notification relocation criteria. In some embodiments, the notification relocation criteria include one or more criteria selected from the group consisting of: a predetermined time elapses (eg, time elapses after the notification 614 is displayed), the instant message user interface 604 ceases to be displayed, an instant The messaging application is closed, and/or all or a portion of notification 614 is hidden in message area 606 (eg, due to new incoming messages filling message area 606). In some implementations, notification 614 is moved to the bottom, right, or left of display 602 . In some embodiments, notification 614 is a banner notification displayed near the top of display 612 directly in response to receiving data indicating that a real-time communication session is available.

In some embodiments, device 600 receives reminder data indicating a reminder to join a real-time communication session (eg, a reminder sent by a participant in a conversation or after a predetermined period of time). As shown in FIG. 6D , in response to receiving data indicating a reminder to join the real-time communication session, device 600 displays a full screen notification 616 and outputs audio output 618 . In some embodiments, audio output 618 is different from audio output associated with instant messaging applications (eg, notifications of received messages) and audio output associated with telephony applications (eg, ringtones for incoming calls). In some embodiments, audio output 618 is a mix of audio associated with message notifications and audio associated with incoming call notifications. In some embodiments, the audio output 618 is shorter than the audio output associated with the telephony application (eg, a ringtone for incoming calls), and is shorter than the audio output associated with the instant messaging application (eg, notification of received messages) )long. In some embodiments, the ringtone for an incoming call includes multiple repetitions of the corresponding audio output, and the audio output 618 is a single repetition of the corresponding audio output. In some embodiments, the audio output 618 shares audio characteristics (eg, common pitch, note, rhythm) with both the audio output for phone calls and message notifications. In some embodiments, device 600 outputs haptic output. As shown in Figure 6D, full screen notification 616 includes the name of the group ("Dream Team"), the names of the participants currently active in the real-time communication session ("PABLO" and "STEPHEN"), and the initial initiating real-time communication The name of the session or participant who initiated the reminder ("PABLO"). In some embodiments, the full screen notification 616 includes the names of participants who have been invited to the real-time communication session (eg, according to the order in which they were added to the group). The full screen notification 616 also includes a reply affordance 620 for joining the real-time communication session (eg, for immediately joining the real-time communication session or displaying a menu with options for joining the real-time communication session), for dismissing the full screen notification 616 " Not now" affordance 622, and a message affordance 624 for returning to the instant messaging user interface 604 or sending a message to the "Dream Team" group. The full screen notification also includes an image (eg, video of the user of device 600 from camera 603) (eg, as a preview of the user's image, which, once joined, will be seen by other participants in the real-time communication session).

As shown in FIG. 6D , device 600 receives (eg, detects) user input 650A (eg, a tap) corresponding to the selection of “not now” affordance 622 . As shown in FIG. 6E, in response to user input 650A, device 600 stops displaying full screen notification 616 and returns to instant messaging user interface 604.

As shown in FIG. 6E , device 600 receives (eg, detects) user input 650B (eg, a tap) corresponding to the selection of option affordance 612 . In response to receiving user input 650B, device 600 expands the header of instant messaging user interface 604 to include audio affordance 626A, video join affordance 626B, and group detail affordance 626C.

As shown in FIG. 6F, device 600 receives (eg, detects) user input 650C (eg, taps) corresponding to the selection of video join affordance 626B. In some embodiments, device 600 joins the real-time communication session directly (eg, without additional input) in response to receiving user input 650B. In some embodiments, device 600 joins a real-time communication session and transmits video and audio simultaneously. In some embodiments, device 600 joins a real-time communication session and transmits only audio (eg, no video). In some embodiments, in response to selection of the audio affordance 626A, the device 600 joins the real-time communication session with audio only or initiates a separate teleconference call for the group of participants.

As shown in Figure 6G, in response to receiving user input 650C, device 600 displays a real-time communication interface 628 of the real-time communication application. The real-time communication interface 628 includes an indication of the group ("dream team"), images (eg, video of the user of the device 600 from a camera (eg, camera 603 ) on the front of the device 600 ) currently active in the real-time communication session Indication of the participant of the status ("PABLO" and "STEPHEN"), indication of the participant initiating the real-time communication session ("Video call from PABLO"), and the call control menu 630. Call control menu 630 includes processing affordances 631 , effects affordances 632 , call affordances 634 , and menu affordances 636 .

As shown in FIG. 6G , device 600 receives (eg, detects) user input 650D on menu affordance 636 . As shown in Figure 6H, in response to user input 650D, device 600 expands call control menu 630 to display additional information and controls related to the real-time communication session. In some embodiments, device 600 expands call control menu 630 in response to an up swipe initiated near processing affordance 631 . When expanded, call control menu 630 also includes audio on/off affordance 638, audio source menu affordance 640, video on/off affordance 642, group name 644 (with an indication of the number of participants in the group), A group message affordance 646 , a list 648 including the affordances 648A to 648E of the participants corresponding to the group, and an add participant affordance 652 . In some embodiments, in response to receiving selection of group message affordance 646, device 600 initiates and/or displays instant messaging user interface 604 (eg, FIG. 6F ) for sending messages to the participants of the group.

As shown in FIG. 6H, the affordances of the participants in the list 648 include an indication of the communication status of the corresponding participant regarding the real-time communication session. Representation 648A indicates that participant Pablo is connected to a real-time communication session and is providing video data (eg, a real-time media stream including video and audio data). Representation 648B indicates that participant Stephen is connected and provides audio data (eg, no video). Representations 648C and 648D indicate that participants Marcel and Allison, respectively, are not connected to the real-time communication session (eg, video and audio data are not provided). Unconnected participants either never joined the real-time communication session, or have previously joined and then left the real-time communication session. Exemplary participant communication status types include audio only (eg, participant is communicating using audio only); video (eg, participant is talking using both video and audio); video pause (eg, participant's video is paused); video Undecoded (e.g., the participant video stream cannot be processed due to latency issues, video stream format issues, etc.); Away (e.g., the participant has left the live chat session); and waiting to join (e.g., the participant is invited to the live chat session but has not joined a live communication session).

In some embodiments, the participants of the real-time video communication include participants of the group that are currently connected to the communication session and participants of the group that are not currently connected to the real-time video communication session. That is, all participants in the group are referred to as participants in a real-time communication session, regardless of the connection status. While the real-time communication session is active, participants can join (connect) or leave (disconnect) the real-time communication session.

As shown in FIG. 6I , device 600 receives (eg, detects) user input 650E (eg, a tap) corresponding to the selection of video on/off affordance 642 . As shown in Figure 6J, in response to receiving user input 650E, device 600 modifies video on/off affordance 642 to indicate a video off state, and by changing the representation of the video camera in Figure 6I to that of the phone in Figure 6J Representation to modify the call affordance 634 to indicate an audio-only state. In the audio-only state, device 600 joins the real-time communication session with audio only (eg, device 600 does not provide video data).

As shown in FIG. 6J , device 600 receives (eg, detects) user input 650F (eg, taps) corresponding to selection of call affordance 634 . As shown in FIG. 6K, in response to receiving user input 650F, device 600 initiates a connection to join the real-time communication session in audio-only mode. As shown in FIG. 6K , device 600 stops displaying expanded call control menu 630 (eg, device 600 reduces call control menu 630 ) and displays connect screen 654 . In some embodiments, device 600 initiates a connection to join a real-time communication session and displays connection screen 654 in response to user input 650C on video join affordance 626B. In some embodiments, notification 614 includes an affordance (eg, a "join" icon on notification 614 or the entire notification 614 itself is selectable). In some embodiments, device 600, in response to selection of notification 614, initiates a connection to join the real-time communication session and displays connection screen 654.

As shown in FIG. 6L, after device 600 joins the real-time communication session, device 600 displays real-time communication user interface 656 that includes a representation 658 of the user of device 600, participant Pablo (e.g., participation in initiating the real-time communication session) participant) representation 670A and participant Stephen's representation 670B. According to the communication status indicated in list 648, participant Pablo's representation 670A includes a live video stream. Since participant Stephen provides only audio data, participant Pablo's representation 670B includes an avatar. In some embodiments, the layout of the representation on the real-time communication user interface 656 is based on the number of participants currently connected to the real-time communication session (eg, two participants and the user of the device 600). The layout shown in FIG. 6L is referred to as a 2-to-1 layout, where the first position corresponds to the position of representation 670A and the second position corresponds to the position of representation 670B.

In some embodiments, when device 600 joins a real-time communication session, device 600 determines whether the number of participants (eg, active or total (active and invited)) exceeds a predetermined threshold number. Upon determining that the number of participants exceeds a threshold, the device 600 displays an overflow area with an overflow representation (eg, a thumbnail), as described in more detail below. Alternatively, upon determining that the number of participants does not exceed the threshold, the device 600 does not display the overflow area with the overflow representation.

When device 600 has joined the real-time communication session, device 600 streams real-time media to other participants (eg, participants who have joined the real-time communication session). When a participant associated with another device has joined the real-time communication session and device 600 has joined, device 600 receives the real-time media stream corresponding to that participant.

As shown in FIG. 6L, device 600 receives (eg, detects) user input 650G (eg, a tap) on display 602 when real-time communication user interface 656 is displayed. As shown in FIG. 6M, in response to receiving user input 650G, device 600 displays call control menu 630. Representations 670A and 670B are reduced in size, and representations 670B and 658 are moved up on display 602 so that call control menu 630 does not overlap the representations. In contrast to the configuration shown in FIG. 6L in which representations 670A and 670B are vertically aligned and do not overlap, when call control menu 630 is displayed in FIG. 6M, representations 670A and 670B are offset and overlap each other.

As shown in FIG. 6N, device 600 receives (eg, detects) user input 650H (eg, a tap) on menu affordance 636 in call control menu 630. As shown in FIG. 60, in response to receiving user input 650H, device 600 expands call control menu 630 (eg, as discussed with respect to FIG. 6H). In some embodiments, the device 600 displays a reminder enablement representation in the call control menu 630 at representations of participants not connected to the real-time communication session. As shown in FIG. 60, list 648 indicates that participants Marcel and Allison are not connected to the real-time communication session. Since participants Marcel and Allison (eg, devices associated with participants Marcel and Allison) are not connected to the real-time communication session, extended call control menu 630 includes representations 648C and 648D of participants Marcel and Allison, respectively, in list 648 The ringtones at 662 and 664 can be represented. In response to selection of the ringtone affordance 662 or 664, device 600 causes a notification (eg, an audio output such as a call or ringtone) associated with the participant corresponding to the selected ringtone affordance to be provided at the device. In some embodiments, the notification is a new notification different from the previous notification sent to the participant. In some embodiments, the new notification is more intrusive than the previous notification (eg, an audio ringtone, haptic output, or full screen notification compared to a banner notification without audio or haptic output). In this manner, the user may alert participants who have not yet joined the real-time communication session that the real-time communication session is active and available for participants to join (eg, similar to the manner in which the user of device 600 is alerted by a full-screen notification as shown in FIG. 6D ) .

As shown in FIG. 60 , device 600 receives (eg, detects) user input 650I (eg, a tap) corresponding to the selection of video on/off affordance 642 . In response to receiving user input 650I, device 600 provides a real-time video stream (eg, from a camera on device 600) to the real-time communication session and modifies video on/off affordance 642 to indicate video on, as shown in Figure 6P. In FIG. 6Q , device 600 receives (eg, detects) user input 650J (eg, swipe down) on display 602 to shrink call control menu 630 . As shown in FIG. 6R , when the call control menu 630 is zoomed out, the representation 658 corresponding to the user of the device 600 includes video from the camera 603 of the device 600 .

As shown in FIG. 6S , device 600 receives (eg, detects) user input 650K (eg, taps) corresponding to the selection of call affordance 634 . In response to user input 650K, device 600 is disconnected from the real-time communication session. 6T illustrates an embodiment of an instant messaging user interface 604 of an instant messaging application displayed after device 600 is disconnected from the real-time communication session. Although device 600 has been disconnected from the real-time communication session, the real-time communication session remains active. In some embodiments, as long as at least one of the participants is connected, the real-time communication session remains active even if the participant who initiated the real-time video communication session leaves the session. As shown in FIG. 6T, device 600 maintains display of notification 614 and real-time communication affordance 610, indicating that the real-time communication session is active. Notification 614 is updated to indicate the duration of the real-time communication session (eg, three minutes and forty-two seconds, corresponding to the amount of time since the real-time communication session was initiated).

While displaying notification 614 indicating that the real-time communication session is active, device 600 receives data indicating that the real-time communication session is no longer active (eg, has ended). In some embodiments, the real-time communication session ends when no participants are connected to the real-time communication session (eg, all participants that have initiated or joined the real-time communication session are no longer connected). In response to receiving data indicating that the real-time communication session is no longer active, device 600 stops displaying notification 614 and/or updates notification 614 to indicate that the real-time communication session is no longer active. 6U illustrates an embodiment of an instant messaging user interface 604 of an instant messaging application displayed after the real-time communication session ends (eg, in response to receiving data indicating that the real-time communication session is no longer active). As shown in FIG. 6U, notification 614 is updated to include an indication that the real-time communication session is no longer active and removes the indication of real-time communication affordance 610 (eg, the text "call ended"). In some embodiments, the real-time communication affordance 610 and/or the notification 614 are grayed out to indicate that the real-time communication session is no longer active. In some embodiments, notification 614 is no longer optional after the real-time communication session ends. In some embodiments, after the real-time communication session ends, notification 614 remains selectable and can be selected to initiate a new real-time communication session with the group. In some embodiments, if device 600 has not joined the real-time communication session, device 600 updates notification 614 to indicate that the user has missed the real-time communication session (eg, "missed call").

As shown in FIG. 6U , device 600 receives (eg, detects) user input 650L (eg, taps) corresponding to the selection of return affordance 611 . As shown in FIG. 6V, in response to receiving user input 650L, device 600 displays a user interface 671 of the instant messaging application, which includes a list 673 of affordances 673A-673G for accessing corresponding messaging conversations. The affordance 673D corresponds to a message conversation between three participants in a group (Matthew, Danielle, and the user of device 600). The affordance 673D includes a real-time communication indicator 675 that indicates that there is an active real-time communication session for the group. Conversely, the affordance 673A corresponding to the "Dream Team" does not include the real-time communication indicator 675 because the "Dream Team" real-time communication session is not active.

Turning now to FIG. 6W , device 600 is shown displaying an interface other than instant messaging user interface 604 . In Figure 6W, device 600 displays a lock screen 668, indicating that device 600 is in a locked state. Lock screen 668 includes notifications 677A and 677B. Notification 677A indicates that a message from user John Appleseed was received five minutes ago. Notification 677B indicates a real-time communication session that is no longer available to join (eg, a missed video call). Missed video call notifications include invited participants (John, Amy, and Rodrigo), the status of the real-time communication session ("Missed"), and an indication of when the real-time communication session was initiated or ended ("1 hour ago"). In the locked state, device 600 does not display active applications.

When in the locked state, device 600 receives data indicating that a real-time communication session is available for joining. In some embodiments, the response of device 600 to receiving data indicating that a real-time communication session is available for joining depends on the context of device 600, available real-time communication sessions, or other factors. In some embodiments, device 600 displays a first type of notification based on an available real-time communication session (eg, a one-to-one communication session) between the two participants. As shown in FIG. 6X , the first type of notification includes full screen notification 672 and audio output 674 . In some embodiments, the audio output 674 is different from the audio output associated with phone call notifications and the audio output associated with message notifications. In some embodiments, the audio output 674 shares audio characteristics (eg, common pitch, note, rhythm) with both the audio output for phone calls and message notifications. Alternatively, device 600 displays a second type of notification based on a real-time communication session (eg, a group real-time communication session) available between the three or more participants. As shown in Figure 6Y, a second type of notification includes a banner notification 676 (eg, not full screen) and a haptic output 678 (eg, but no audio output). In some embodiments, device 600 initially displays a full screen notification (eg, similar to the full screen shown in FIG. 6E ) based on a real-time communication session (eg, a group real-time communication session) available between three or more participants notification 616), and then display (eg, transition to) a banner notification 676. In some embodiments, notification 676 includes an affordance that, when selected, initiates a process for joining the corresponding real-time communication session (eg, as described above with reference to FIGS. 6F-6L).

In contrast to notification 677B, which corresponds to a missed video call, notification 676 corresponds to an active real-time communication session. Notification 676 includes the group ("Dream Team"), the participant (Pablo) who performed the action that triggered the presentation of the notification, and an indication of when the real-time communication session was initiated ("now"). In some embodiments, the notification 676 is updated to accurately reflect the time at which the real-time communication session was initiated and the status of the real-time communication session. In some embodiments, when the real-time communication session represented by notification 676 ends, the text of notification 676 is updated from "Join this video call from PABLO" to "Missed the video call from PABLO."

In some embodiments, upon determining that the active application is not displayed (as shown in FIG. 6W), the device 600 displays a full screen notification in response to receiving data indicating that the real-time communication session is available for joining. In some embodiments, upon determining that an active application (eg, an instant messaging application) is being displayed, device 600 displays a non-full screen notification (eg, a banner notification).

In some embodiments, upon determining to alert the user of device 600, device 600 outputs an alert (eg, a notification or audio or tactile output); and upon determining not to alert the user of device 600, device 600 refrains from outputting the alert. In some embodiments, the determination to alert the user is based on interference criteria (eg, whether the device is locked or unlocked). In some embodiments, the device outputs an alert (eg, haptic output 678 ) based on the device 600 being in a locked state (eg, as shown in FIG. 6Y ); and based on the device 600 being in an unlocked state (eg, as shown in FIG. 6Z ), Device 600 displays notifications (eg, banner notification 676) and avoids providing alerts (eg, no haptic output).

In some embodiments, the interference criterion indicates whether the user is looking at device 600 (eg, based on data from sensors on the front of device 600). FIG. 6AA shows user 684 holding device 600 . The user's gaze 680 is not directed at device 600 . Device 600 determines whether user 684 is not looking at device 600, and based on the determination that user 684 is not looking at device 600, device 600 provides a non-visual notification 682 (eg, a haptic or audio output). As shown in FIG. 6AB , the user's gaze 680 is directed towards the device 600 . Based on the determination that user 684 is looking at device 600, device 600 refrains from providing non-visual notifications (eg, device 600 displays a banner notification without haptic or audio output). In some embodiments, device 600 determines whether the user is looking at device 600 using an image sensor configured to recognize the user's face.

6AC-6AF, the device 600A includes a display 602A (eg, a touch-sensitive display), a rotatable input mechanism 601A, and a mechanical button 601B. In some embodiments, device 600A includes features of device 100 , 300 , 500 or 600 . In some embodiments, device 600A communicates with device 600 (eg, via two-way wireless communication). In some embodiments, device 600A and device 600 are in a paired relationship.

In some embodiments, device 600A (eg, from device 600 ) receives data indicating that a real-time communication session available to three or more participants is active. 6AC, in response to receiving data indicating that the real-time communication session is active, the device 600A displays a user interface 604A including a notification 605, a real-time communication join affordance 607, and a dismissal affordance 609. In some embodiments, notification 605 includes the characteristics of notification 614 described above (eg, participant name, number of participants, participant name that triggered the notification). In some embodiments, device 600A generates audio and/or haptic output in response to receiving data indicating that the real-time communication session is active. In some embodiments, device 600A generates audio and/or haptic output that is different from the audio and/or haptic output generated by device 600 in response to receiving data indicating that the real-time communication session is active. In some embodiments, in response to receiving initial data indicating that the real-time communication session is active, a notification (eg, display, audio, and/or) that the real-time communication session is active is provided at device 600 (eg, not at device 600A). or haptic output), and a notification is provided at both device 600 and device 600A in response to receiving reminder data indicating a reminder that the real-time communication session is active. In some embodiments, in response to receiving initial data indicating that the real-time communication session is active, a notification (eg, display, audio, and/or haptic output) that the real-time communication session is active is provided at both device 600 and device 600A , and a notification is provided at device 600 (eg, not at device 600A) in response to receiving reminder data indicating a reminder that the real-time communication session is active.

In some embodiments, device 600A receives user input corresponding to the selection of real-time communication join affordance 607 (eg, a tap on real-time communication affordance 617). In response to receiving the selection of the real-time communication join affordance 607, the device 600A causes the device 600 and/or the device 600A to join the real-time communication session or initiates the process of joining the real-time communication session. In some embodiments, in response to receiving the selection of real-time communication join affordance 607, device 600A causes device 600 to display real-time communication user interface 628 (FIG. 6G), 654 (FIG. 6K), or 656 (FIG. 6L).

Figure 6AD shows user interface 604A displayed on device 600A after joining a real-time communication session. In some embodiments, device 600A includes a microphone for a user to provide audio input to the real-time communication session via device 600A. In Figure 6AD, user interface 604A includes volume controls 611, indications 613 of participants ("PABLO" and "2 others") and duration of the real-time communication session ("9:45"), end call affordance 615 , audio on/off affordance 617 , and menu affordance 619 .

In Figure 6AE, device 600A receives (eg, detects) user input 650M (eg, an up swipe gesture beginning at or near menu enablement 619). As shown in Figure 6AF, in response to receiving user input 650M, device 600A modifies indication 613 ("number of participants 4") and displays keyboard affordance 621, audio source affordance 623 (eg, using device 600 to participate in a real-time communication session ), and a list 625 of the participants of the real-time communication session (eg, joined and/or invited participants). In some embodiments, device 604A displays list 625 in response to receiving user input 650M on user interface 604 in Figures 6AC (eg, prior to joining a real-time communication session). In some embodiments, list 625 includes an indication of the participant's communication status. In some embodiments, list 625 includes reminder affordances that may be selected to send reminder notifications to participants who have not yet joined the real-time communication session.

7A-7F are flowcharts illustrating methods according to some embodiments. Method 700 is performed at a device having a display (eg, 100, 300, 500, 600). Some operations in method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

As described below, method 700 provides, among other things, an intuitive way to provide interactive notifications (eg, 614, 672, 676) for real-time communication sessions between multiple participants. The approach reduces the cognitive burden on users to provide notifications for real-time communication sessions between multiple participants using innovative techniques, thereby creating a more efficient human-machine interface. For battery powered computing devices, enabling users to quickly interact with notifications to make real-time communication sessions between multiple participants faster and more efficient, thus saving power and increasing time between battery charges.

At block 702, the device displays an instant messaging user interface of the instant messaging application (eg, 604). The instant messaging user interface (eg, 604) includes a message area (eg, 606). The message area includes multiple messages (eg, 608A-608D) in a message conversation between three or more participants.

At block 704, the device receives first data indicating that a real-time communication session (eg, an audio communication session or an audio/video communication session) available to three or more participants is active. In some embodiments, participants in a real-time communication session have a status of "currently joined" (eg, providing real-time media streams to other participants; also referred to as "connected" or "active") or "currently not joined" (eg, , a participant who has never joined a session or has previously joined a session and then left; also known as "not connected" or "invited"). In some embodiments, the participants in the real-time communication session include participants who are currently in the real-time communication session and participants who are not currently in the real-time communication session (eg, 648A-648D). When a real-time communication session is active, a participant joins (connects) or leaves (disconnects) the real-time communication session. When a participant associated with another device has joined the real-time communication session, the device receives the real-time media stream corresponding to the participant. When the user of the device has joined the real-time communication session, the device streams the real-time media to other participants (eg, participants who have joined the real-time communication session). In some embodiments, the real-time communication session remains active as long as at least one of the participants is connected (eg, even if the participant who initiated the real-time communication session leaves the session).

At block 706, in response to receiving the first data indicating that the real-time communication session between the three or more participants is active, the device displays a notification indicating that the real-time communication session is active (eg, 614). In some embodiments, the notification indicating that the real-time communication session is active includes the real-time communication joining affordance (eg, 614A). In some embodiments, the notification includes an indication of the number of participants in the real-time communication session (eg, the number of participants active in the session; the number of participants invited to the session). In some embodiments, the notification includes an indication of the participant (eg, name, initials, photo, or avatar) that performs the action that triggers the presentation of the notification on the display (eg, the participant who initiated the real-time communication session, or selects "again" "Ring" button, etc. to remind the user of the device to join the participants in the real-time communication session). In some embodiments, in response to receiving the first data indicating that the real-time communication session between the three or more participants is active, the device generates via one or more audio output devices of the device indicating that the real-time communication session is in an active state Audio output of an active state (eg, 618 ), wherein the audio output indicating that the real-time communication session is active is different from the audio output associated with the phone call notification and the audio output associated with the message notification. In some embodiments, the audio output indicating that the real-time communication session is active shares audio characteristics (eg, common pitch, note, rhythm) with both the audio output of the phone call and message notification. Displaying a notification (eg, 614 ) indicating that the real-time communication session is active and/or generating an audio output (eg, 618 ) indicating that the real-time communication session is active provides feedback to the user regarding the current state of the real-time communication session and to the user Visual feedback is provided indicating that if the user activates the notification (eg, when the notification includes a join affordance), certain actions associated with the notification and/or the real-time communication session will be performed. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at block 704, the device displays a notification (eg, 606) indicating that the real-time communication session is active at the first location in the message area. In some embodiments, the notification (eg, 614) indicating that the real-time communication session is active includes the real-time communication joining affordance (eg, 614A).

At block 710, while displaying a notification (eg, 614) indicating that the real-time communication session is active, the device receives second data indicating that the real-time communication session between the three or more participants is no longer active .

Optionally, at block 712, the device moves a notification indicating that the real-time communication session is active from a first position (eg, the bottom of the message area 606) to a second position (eg, the middle or top of the message area 606, or top side, bottom side, right side and/or left side of display (602). In some embodiments, a mobile notification occurs (eg, 614) in response to meeting notification relocation criteria. In some embodiments, notification relocation criteria include one or more criteria, such as a predetermined amount of time elapses (eg, time elapses after the notification is displayed), instant messaging user interface (eg, 604 ) ceases to display, instant messaging The application is closed, the entire notification (eg, 614 ) or a portion of the notification (eg, 614 ) is hidden in the message area (eg, 606 ) due to a new incoming message filling the message area (eg, 606 ), etc. Moving a notification indicating that the real-time communication session is active provides feedback to the user about the current state of the real-time communication session, providing the user with visual feedback indicating that certain actions associated with the notification and/or the real-time communication session will be performed ( if selected), and reduce the number of inputs required to perform the operation. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently. In addition, reducing the number of inputs required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (eg, by providing the user with more inputs available and reducing the number of inputs), which in turn makes the user The ability to use the device more quickly and efficiently reduces power usage and improves the battery life of the device.

Optionally, at blocks 714 and 716, the device receives a first user input (eg, 650C) corresponding to a selection of a real-time communication join affordance (eg, 614A or 626B). In response to receiving the first user input corresponding to the selection of the real-time communication join affordance, the device joins the real-time communication session (eg, joins the real-time communication session and simultaneously transmits video and audio from the electronic device and/or joins the real-time communication session and only Stream audio (not video) from electronic devices. Joining the real-time communication in response to receiving the selection of the real-time communication to join the affordance included in the notification reduces the amount of input required to perform the operation of joining the real-time communication session. Reducing the amount of input required to perform an operation also enhances the operability of the device and makes the user-device interface more efficient (eg, by providing more input available to the user and reducing the amount of input required to join a real-time communication session), which in turn Power usage is reduced and battery life of the device is improved by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 718 and 720, the device receives reminder data (eg, FIG. 6D) indicating a reminder to join the real-time communication session (eg, a reminder sent by a participant in the conversation or after a predetermined period of time). In response to receiving reminder data indicating a reminder to join the real-time communication (e.g., the reminder was sent by a participant in the conversation), the device generates audio output via one or more audio output devices of the device indicating that the real-time communication session is active (e.g., , 618) (eg, phone ringing output). In some embodiments, audio output (eg, 618) is generated in conjunction with the redisplay notification (eg, 616). In some embodiments, the display of the notification is in progress when the reminder data is received. In some embodiments, the audio output indicating that the real-time communication session is active is different from the audio output associated with the phone call notification and the audio output associated with the message notification. In some embodiments, the audio output indicating that the real-time communication session is active shares audio characteristics (eg, common pitch, note, rhythm) with both the audio output of the phone call and message notification. In some embodiments, the initial audio output indicating that the real-time communication session is active is different from the reminder audio output indicating that the real-time communication session is active (eg, audio output generated based on a participant selecting the "ring" affordance) ( For example, shorter, quieter, or lower intensity). In some embodiments, the initial audio output indicating that the real-time communication session is active is a single repetition of the repeated audio output for the phone call and the reminder audio output indicating that the real-time communication session is active is the repeated audio for the phone call Multiple repetitions of the output, or the same audio output used for a phone call. In some embodiments, in response to receiving reminder data indicating the reminder to join the real-time communication, the device displays an indication (eg, name, initials, photo, or avatar) of the participant in the real-time communication session that initiated the reminder. Generating a reminder that includes an audio output when specified conditions are met allows the user to quickly identify that a real-time communication session is available when they miss a visual notification (eg, 614). Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 722 and 724, in embodiments where the notification includes a menu access affordance (eg, 626B), the device receives a second user input (eg, 650C) corresponding to the selection of the menu access affordance. The device, in response to receiving a second user input corresponding to a selection of a menu access affordance, displays a display including a video join option (eg, 634 ) (eg, an affordance, a drop-down menu, a checkbox) and an audio-only option ( For example, 642) a menu (eg, 630) (eg, affordance, drop down, checkbox). Displaying a menu including different options for joining a real-time communication session in response to receiving selection of a menu access affordance provides additional control options without cluttering the UI with additionally displayed controls. Providing additional control options without cluttering the UI with additionally displayed controls enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the amount of time when operating/interacting with the device) user error), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 726 and 728, the device receives a third user input while displaying the menu (eg, 630). According to the third user input (eg, 650F) corresponding to the selection of the video join option (eg, 634), the device joins the real-time communication session using the audio and video transmitted from the electronic device. In some embodiments, the device prompts the user to check if they want to join as audio only and/or video only (eg, "Do you want to join as video?"). Joining a real-time communication session using audio and video transmitted from an electronic device allows a user to join a real-time communication session using audio and video without requiring the user to select or modify the device's Additional inputs and outputs (eg, turn on the camera and/or turn on the microphone). Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at block 730, based on a third user input corresponding to selection of an audio-only join option (eg, 642), the device uses audio transmitted from the electronic device (eg, audio recorded by one or more microphones of the device). ) to join a real-time communication session without transmitting video data from the electronic device. In some embodiments, the device displays an indicator (eg, 658 in FIG. 6L ) that the user has joined the real-time communication using audio only. This makes the user comfortable because the video is not shared). When specified conditions are met, joining a real-time communication session using audio transmitted from an electronic device allows a user to join a real-time communication session using audio only, without requiring the user to select or modify the device's additional inputs and outputs (eg, turn the camera off and/or turn on microphone). Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

At block 732, in response to receiving the second data indicating that the real-time communication session between the three or more participants is no longer active, the device stops displaying the notification (eg, 614) or updates the notification to indicate the real-time communication session is no longer active. In some embodiments, when the real-time communication session is active, the notification is updated to indicate the current duration of the real-time communication session. In some embodiments, when the real-time communication session is no longer active, the notification ceases to display the duration of the session and displays an indication that the session has ended (eg, "call ended"). In response to receiving data indicating that the real-time communication session is no longer active, the display of the notification is stopped, providing visual feedback to the user indicating that certain operations associated with the notification and/or the real-time communication session are no longer available. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently. Additionally, ceasing to display the notification in response to receiving data indicating that the real-time communication session is no longer active, automatically performs actions when a set of conditions is met without further user input. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (eg, by reducing unnecessary elements on the user interface and helping the user Provide appropriate input and reduce user error when operating/interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 736, 738 and 740, the device receives third data indicating that the second real-time communication session is available between the two or more participants. Upon the availability of the second real-time communication session between the two participants, the device displays a notification (eg, 672 ) of the first type (eg, a full-screen notification) indicating that the second real-time communication is active. In some embodiments, the first type of notification (eg, 672 ) includes a menu (eg, 630 ) with options for video (eg, 634 ), audio only (eg, 643 ), and/or rejection (eg, 622 ) )Options. Based on the availability of the second real-time communication session among the three or more participants, the device displays a second type of notification (eg, 676 ) that is different from the first type of notification indicating that the second real-time communication is active. In some embodiments, the second type of notification is a banner notification (eg, 676), which is not full screen or is not continuously full screen. Optionally, at block 742, the device initially displays a second type of notification in a first state (eg, full screen state 672). Optionally, at step 744, the device transitions the notification of the second type (eg, 672) to a second state (eg, 676) (eg, a persistent non-fullscreen state (eg, a persistent banner at the top edge of the display) ). Displays certain types of notifications that a real-time communication session is active based on when specified conditions are met, allowing the user to quickly identify that real-time communication is available based on specified conditions while providing the user with more and/or less reduced interference (eg, when two A full-screen notification is displayed when a live chat session is available between participants, and a banner is displayed when a live chat session is available between two participants). Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 746, 748, 750 and 752, the device receives fourth data indicating that the third real-time communication session is available. In response to receiving the fourth data indicating that the third real-time communication session is available, an active application (eg, an application (instead of, eg, a lock screen (eg, 668) or a home screen (eg, 1400) of the device) is determined according to the determination , such as an instant messaging application or a third-party application provided by an application developer other than the developer of the operating system), the device displays a third type (eg, a non-full-screen state) indicating that a third real-time communication session is available ( For example, a banner at the top edge of the display)) notification (eg, 614 or 676). In response to receiving the fourth data indicating that the third real-time communication session is available, upon determining that the active application is not displayed (eg, FIG. 6W ), the device provides a fourth type of notification (eg, 678 or Fullscreen notification (eg, 616)). Displays specific types of notifications for real-time communication sessions based on when specified conditions are met, allowing the user to quickly identify that real-time communication is available based on specified conditions while providing the user with more and/or less reduced distraction (e.g., displayed when the device displays an application) Full screen notifications and a banner when the device is not showing the app). Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 754, 756, 758 and 760, the device receives fifth data indicating that the fourth real-time communication session is available. In response to receiving the fifth data indicating that the fourth real-time communication session is available, determining the current context of the electronic device (eg, the current operating mode of the device (eg, whether the device is in a Do Not Disturb (DND) mode; certain applications or hardware is operationally valid; whether the device is locked or unlocked; whether one or more of the device's sensors detect that the user is looking at the display) meets the alerting criteria, the device displays a notification that a fourth real-time communication session is available (e.g., 676), while A non-visual notification (eg, 678) (eg, an audio or tactile output) is provided that the fourth real-time communication session is available. In some embodiments, the output or alert corresponds to the notification (eg, 614). In response to receiving an indication that the fourth Fifth data available for the real-time communication session, upon determining that the current context of the electronic device does not meet the alerting criteria, the device displays a notification (eg, 614) that the fourth real-time communication session is available without providing a non-visual notification that the fourth real-time communication session is available Notification (eg, 678). In some embodiments, when the first type of alert is not issued, a second type of alert (eg, a visual alert, such as a banner notification (eg, 676)) is issued. In some embodiments , the relevant context of the device is whether the device is locked or unlocked. In some embodiments, when the device is locked, a first type of alarm (eg, audio or haptic output) is issued; when the device is unlocked (eg, when When the user is actively operating/interacting with the device), no alerts are issued (e.g., suppressed). In some embodiments, the relevant context of the device is whether the device detects (e.g., via one or more sensors (e.g., camera, infrared) sensor))) whether the user is looking at the display or not. In some embodiments, a first type of alert (eg, audio or haptic output) is issued when the device is locked. In some embodiments, when the device detects that the user is not When looking at the display (eg, Figure 6AA), a first type of alert (eg, 682) is generated. In some embodiments, when the user is actively operating/interacting with the device (eg, Figure 6AB), no alert (eg, Figure 6AB) is generated (eg, 682). , suppress) the first type of alarm.

In some embodiments, the alerting criteria include a requirement that the device be locked in order to meet the alerting criteria. In some embodiments, the alerting criteria include a requirement that the device not detect the user's attention to the device in order to meet the alerting criteria. (eg, based on detected user input with an input device of the electronic device, based on user gaze detected via gaze tracking, face tracking, etc. (eg, 680)). Displaying a notification (eg, 614 ) based on when specified conditions are met and determining whether to provide a non-visual notification allows the user to quickly identify that real-time communication is available based on the specified conditions while providing the user with more and/or less distraction. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Note that the details of the processes described above with respect to method 700 (eg, FIGS. 7A-7F ) also apply in a similar manner to the methods described below. For example, method 700 optionally includes one or more features of the various methods described below with reference to methods 900 , 1100 , 1300 and 1500 . For example, method 700 may use methods 1300 and 1500 to initiate a real-time communication session, and starting and running a real-time communication session may include methods 900 and 1100 . For the sake of brevity, these details are not repeated below.

8A-8BK illustrate exemplary user interfaces for real-time communication sessions, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below including the processes in Figures 9A-9K.

8A shows a device 600 displaying an instant messaging user interface 604 for messaging conversations between groups "dream teams." The real-time communication session between the participants of the group is not active (eg, as indicated by the lack of real-time communication affordance 610). As shown in FIG. 8A, the header of the instant messaging user interface 604 is expanded to display the video join affordance 626B (described above). As shown in FIG. 8A, device 600 receives user input 850A corresponding to a request to display a user interface for a real-time communication session between participants of the participant group "Dream Team" (eg, on a video join affordance 626B). tap). As shown in Figure 8A, the request includes a request to start a new real-time video communication session. In some embodiments, the request includes a request to join an existing real-time communication session (eg, as described above with respect to Figure 6F).

In response to user input 650A, device 600 displays message 800 in message area 606 along with an indication 802 that a real-time communication session is being initiated. In some embodiments, message 800 includes some of the features of notification 614 described above. Once a real-time communication session is initiated, device 600 displays a real-time communication user interface 804 that simultaneously includes multiple representations of the participants in the real-time communication session.

Based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold (eg, more than five participants, including the user of device 600 ), device 600 takes one or more dimensions greater than a corresponding threshold size (eg, three different sizes) simultaneously display representations of a first plurality of participants other than the user of device 600, while displaying representations of a second plurality of participants in the overflow area, wherein the second plurality of participants includes not included in the The representations of one or more of the first plurality of participants and one or more of the other participants displayed in the overflow area are displayed at a size that is less than the respective threshold size.

In the embodiment shown in Figures 8A-8BK, the threshold number of participants is five participants (including the user of device 600). Since the group "dream team" includes fifteen participants, the device 600 determines that the number of participants in the real-time communication session includes a number of participants that is greater than a threshold. As shown in FIG. 8C , according to this determination, the real-time communication user interface 804 includes four primary representations 810A through 810D (collectively 810 ) of the four respective participants in addition to the user of the device 600 displayed in the canvas area 811 . User representation 806 for 1 layout, camera view of device 600 (corresponding to the user of device 600 ), and with participants corresponding to real-time communication sessions other than the user of device 600 and participants represented in canvas area 811 The overflow represents the overflow region 820 of 822A-822J (collectively 822), a subset thereof (eg, 822A-822D) displayed at any time. In some embodiments, the layout is one of a number of predefined layouts in which representations of participants are anchored at various locations in the user interface 804 . FIG. 8AW shows another embodiment of a user interface 804 . In the embodiment shown in FIG. 8AW, overflow area 820 includes overflow representations of all participants of the real-time communication session other than the user of device 600, such that the participants represented in canvas area 811 are also represented in overflow area 820 .

If the number of participants in the group is less than or equal to five, then upon determining that the number of participants in the real-time communication session is less than or equal to five, the device 600 displays the primary representation of the participants and the representation of the user of the device 600, while in the overflow area The overflow representation is not displayed.

Each of the primary representations 810 is displayed in one of three predetermined representation sizes (small, medium, and large). Main representation 810A is displayed in small size, main representation 810B is displayed in large size, and main representation 810C and main representation 810D are displayed in medium size. In Figure 8C, each of the primary representations 810 overlaps with at least one other primary representation. The overflow representations 822 are all displayed at the same size, which is smaller than the small representation size associated with the primary representation 810 . In the illustrated embodiment, the respective threshold sizes mentioned above are larger than the size of the overflow representation 822 and smaller than the small representation size of the primary representation 810 . User representation 806 is the same size as overflow representation 822 and is aligned with or displayed within overflow area 820.

As shown in Figure 8C, the real-time communication session user interface 804 displays three full overflow representations and one partial overflow representation at a time. The remaining overflow representations are hidden and can be displayed by scrolling the overflow area 820 (eg, in response to user input or detected events, such as active participation by a participant by talking or moving). In FIG. 8C, overflow representation 820D is partially shaded to indicate that there are more overflow representations and/or that overflow area 820 can be scrolled. In some embodiments, the order of the overflow representations 822 in the overflow area 820 depends on the order in which the participants joined the real-time communication session. In some embodiments, the order of the overflow representations 822 in the overflow area 820 changes over time (eg, as a participant joins or leaves the real-time communication due to, for example, user selection of the overflow representation or active participation of participants represented in the overflow area. session or when participants move in or out of canvas area 811, as described in more detail below). In some implementations, the order of the overflow representations in the overflow area 820 is fixed (eg, the order in which the overflow representations are displayed does not change over time). In some embodiments, the order of the overflow representations is based on the order of the list associated with the group (eg, the order in which participants were added to the group when the group was created). In some embodiments, the display of the main representation 810 (eg, a real-time video feed) is updated at a higher rate than the display of the overflow representation 822 .

8C shows the real-time communication session interface 804 shortly after the real-time communication session is initiated and before any participants (other than those associated with device 600) join the real-time communication session. As shown in FIG. 8C, main representation 810 and overflow representation 822 include placeholders for participants not connected to the real-time communication session. As shown in Figure 8C, the placeholder includes a representation with the participant's name (eg, first name or first and last initials) and an indication of the colored object. In some embodiments, the color and/or pattern of each colored object is different or selected from a predefined set of colors and/or patterns. Optionally, the placeholder includes a status indicator indicating that a participant is waiting to connect to the real-time communication session, and a reminder alert ("bell") for sending a reminder to the corresponding participant that the real-time communication session is available. In some embodiments, the reminder enabler is not initially included on the placeholder, and is displayed based on a determination that the corresponding participant has not joined the real-time communication session after a predetermined period of time.

As shown in FIG. 8C, device 600 receives (eg, detects) user input 850A (eg, a tap) corresponding to the selection of primary representation 810C. As shown in Figure 8D, in response to receiving user input 850A, device 600 modifies primary representation 810C to include additional identifying information (eg, participant's name, Marcel) and a reminder to the corresponding participant that a real-time communication session is available The reminder can indicate 808A.

FIG. 8D also shows a participant corresponding to primary representation 810B joining the real-time communication session. In some embodiments, device 600 receives data indicating that a participant corresponding to primary representation 822B has joined the real-time communication session. As shown in Figure 8D, in response to a participant corresponding to primary representation 822B joining the real-time communication session, device 600 replaces the placeholder for primary representation 822B with real-time video data from the corresponding participant. In some embodiments (eg, Figure 8AX), device 600 replaces overflow representation 822B corresponding to the participant with real-time video data from the participant. As used herein, the phrase "real-time video data from a participant" (or its equivalent) includes information generated by a device associated with a participant (eg, a smartphone, laptop, desktop computer, or tablet) or by a participant associated with the participant. real-time video data transmitted by devices associated with a user's associated account (eg, an email account or a cloud storage account).

As shown in FIG. 8D , device 600 receives (eg, detects) user input 850B (eg, a tap) corresponding to the selection of reminder affordance 808 . In response to input 850B, device 600 sends instructions to cause a new notification (eg, call, ringtone, etc.) indicating that a real-time communication session is available at the device associated with the corresponding participant, wherein the new notification is different from the one previously sent to the participant. Notice. In some embodiments, the new notification is more intrusive than the previous notification (eg, an audio ringtone, haptic output, or full screen notification compared to a banner notification without audio or haptic output). In FIG. 8D, device 600 also displays reminder enablement representation 808B on primary representation 810D to indicate that the participant corresponding to primary representation 810D is currently being alerted that a real-time communication session is available (eg, due to another person currently connected to the real-time communication session). A reminder sent by a participant, such as the participant represented by primary representation 810B or a reminder previously sent by the user of device 600). The device 600 also modifies the reminder affordance 808A to indicate that the corresponding participant is being notified (eg, by replacing "ringing" with "ringing" as shown in Figure 8E).

As shown in FIG. 8E , device 600 stops displaying overflow representation 822 and increases the size of user representation 806 upon determining that a predetermined period of time has elapsed since the start of the real-time communication session. In some embodiments, when the overflow representation 822 is removed, the main representation 810 is resized and/or moved. In the illustrated embodiment, the main representation 810D is moved down. In some embodiments, device 600 stops displaying overflow representation 822 based on determining that the number of participants currently participating in the real-time communication session is less than or equal to a threshold number of participants (eg, five participants, including the user of device 600).

At the time corresponding to Figure 8E, two participants joined the real-time communication session. In some embodiments, after a further amount of time, device 600 stops displaying one of primary representations 810 and changes the layout of real-time communication user interface 804 to a 3-to-1 layout. In some embodiments, the 4-to-1 layout and the 3-to-1 layout are in a set of predetermined layouts with similar relative placement of the primary representation 810 . In some embodiments, relatively similar placement with a primary representation includes when there is a transition between layouts within the group (eg, a representation's position in one layout overlaps the representation's position in another layout of the same group) , primarily represents a placement that remains in a similar position on the user's display 602 .

In some embodiments, the device 600 determines which primary representation to remove based on whether the corresponding participant has joined the real-time communication session (eg, the representation is not removed if the corresponding participant has joined), whether the participant has been alerted (e.g., the reminder of the participant's representation will not be removed until a predetermined amount of time after the corresponding reminder representation is selected), the participant's placement in the participant list (e.g., earlier in the list Representations of participants are removed before representations of participants later in the list) and placement of that representation on real-time communication user interface 804 (eg, representations closer to the top of display 602 are moved ahead of representations lower on display 602 remove). As shown in FIG. 8F, device 600 stops displaying primary representation 810A and moves primary representation 810B up (eg, to reduce overlap between representations 810B and 810C), while maintaining the relative vertical order of the remaining representations (eg, representation 810B larger than and taller) in representation 810C, and representation 810C is higher than representation 810D and has the same dimensions). Figure 8F also shows that the reminder affordance 808A has been modified to "ring again," which indicates that the corresponding participant has been reminded at least once and is not currently reminded. Reminder affordances 808B are also modified (eg, from "Ring" to "Ring" and a different color) to indicate that the corresponding participant has been previously alerted, but not by the user of device 600 .

At the time corresponding to Figure 8F, still only two participants have joined the real-time communication session. In some embodiments, after a further amount of time, device 600 stops displaying another primary representation 810 and changes the layout of real-time communication user interface 804 to a 2-to-1 layout (eg, as shown in FIGS. 8E-8F ). 4-to-1 layout and 3-to-1 layout in the same set of pre-determined layouts). As shown in FIG. 8G, device 600 stops displaying primary representation 810D and zooms in on primary representations 810B and 810C. 8G shows a three-participant (eg, 2-to-1) layout in which the primary representations of the two participants, other than the user of device 600, are of equal size and aligned vertically one on top of the other. The vertical order of primary representations 810B and 810C is maintained, with primary representation 810B above primary representation 810C. In the layout of Figure 8F, there is no overlap between primary representations, and user representation 806 overlaps primary representation 810C.

At the time corresponding to Figure 8G, still only two participants have joined the real-time communication session. In some embodiments, after a further amount of time, device 600 stops displaying another primary representation and changes the layout of real-time communication user interface 804 to a 1-to-1 layout. As shown in Figure 8H, device 600 stops displaying primary representation 810C and enlarges primary representation 810B to full screen size. In some embodiments, the device 600 changes from the layout of FIG. 8G to the layout of FIG. 8H at a predetermined amount of time after the reminder affordance 808A is selected. FIG. 8H shows a two participant (eg, 1-to-1) layout with the remaining primary representations 810B displayed in full screen and overlaid by user representations 806 . As shown in Figure 8H, the remaining primary representations are compared to the square shapes of primary representation 810 and user representation 806 in the 4-to-1 layout, 3-to-1 layout, 2-to-1 layout shown in Figures 8E-8G, respectively 810B and user representation 806 are vertically elongated rectangles (eg, oriented along the outline).

Turning to Figure 8I, a third participant, Stephen, has joined the real-time communication session. As shown in FIG. 8I , in response to participant Stephen joining the real-time communication session, device 600 transitions from the display of the 1-to-1 layout of FIG. 8H to a 2-to-1 layout, with the primary representation 810E of the most recently joined participant in the existing participant On top of representation 810B (other than the user of device 600). In some embodiments, device 600 displays notification 812 and/or generates audio or haptic output indicating that the participant has joined the real-time communication session, as shown in FIG. 8I. In some embodiments, device 600 refrains from displaying notification 812 or generating audio or haptic output since a new primary representation is displayed when participant Stephen joins the real-time communication session because primary representation 810E serves as the participant's joined real-time communication Session instructions.

Participant Stephen joins the real-time communication session in audio-only mode and provides real-time audio data without video. The audio-only communication status of participant Stephen is indicated by primary representation 810E and notification 812 . Primary representation 810E indicates audio-only communication status by displaying the participant's name and avatar associated with the participant. Notification 812 indicates the communication status in text ("Join this call with audio"). As shown in FIG. 8J, device 600 stops displaying notification 812 (eg, after a predetermined amount of time after participant Stephen joins the real-time communication session).

In some embodiments, in response to receiving data indicating that a participant joining via audio only is actively participating (eg, speaking), device 600 displays a visual indication on the representation of the speaking participant. In Figure 8J, a representative image (eg, an avatar) within primary representation 810E is animated (eg, with a "voice circle") to indicate that participant Stephen is speaking.

Turning to Figure 8K, a fourth participant, Marcel, has joined the real-time communication session. As shown in FIG. 8K, in response to a fourth participant joining the real-time communication session, the device 600 transitions from a 2-to-1 layout to a 3-to-1 layout (eg, as shown in FIG. 8F), with the primary representation of the most recently joined participant 810F On top of primary representations 810B and 810E of existing participants (other than the user of device 600).

Participant Marcel joins the real-time communication session in video mode and provides real-time video and audio data. The video communication status of participant Marcel is indicated by primary representation 810F. The primary representation 810F indicates video communication status by displaying a live video stream. As shown in Figure 8K, device 600 does not immediately display a notification indicating that participant Marcel has joined the real-time communication session.

Turning to Figure 8L, a fifth participant, Hoan, has joined the real-time communication session. As shown in Figure 8L, in response to participant Hoan joining the real-time communication session, device 600 transitions from a 3-to-1 layout to a 4-to-1 layout with the most recently joined participant's primary representation 810G as the smallest primary representation and positioned at primary representation 810F above and to the right. In some embodiments, upon determining that participants Marcel and Hoan have both joined the real-time communication session within a predetermined time window, device 600 displays a combined notification 816 indicating that participants Marcel and Hoan have joined the real-time communication session, as shown in FIG. 8L Show. In some embodiments, the combined notification 816 is displayed based on determining that the participants joined in the same communication state (eg, video).

As indicated by the live video stream displayed in main representation 810G and the representation 816A of the video camera in notification 816, participant Hoan joins the real-time communication session in video mode such that device 600 receives real-time video and audio associated with participant Hoan data.

As shown in FIG. 8L, device 600 receives (eg, detects) user input 850C (eg, a double tap) at the location of primary representation 810F on display 602. In some embodiments, as shown in FIG. 8M, in response to receiving user input 850C, device 600 enlarges primary representation 810F to a maximum representation size and shrinks primary representation 810E to a medium size (while maintaining a 4-to-1 layout). As shown in FIG. 8M, device 600 stops displaying notification 816 (eg, after a predetermined amount of time after participant Hoan joins the real-time communication session).

In some embodiments, the configuration of the layout is dynamically changed in response to receiving data indicating that a participant is actively participating in a real-time communication session. In Figure 8M, device 600 receives data indicating that participant Hoan corresponding to primary representation 810G is actively participating. As shown in FIG. 8M, in response to receiving data indicating that the participant corresponding to primary representation 810G is actively participating, device 600 upscales primary representation 810G from the small representation size to the medium representation size and downscales primary representation 810B from the medium representation size to Small indicates size. In this way, the 4-to-1 layout maintains one small-size representation, two medium-size representations, and one large-size representation.

As shown in FIG. 8M, device 600 receives (eg, detects) user input 850D (eg, double tap) at the location of primary representation 810G on display 602. In some embodiments, as shown in FIG. 8N, in response to receiving user input 850D, device 600 zooms in on primary representation 810G and brings primary representation 810G to the foreground in front of other displayed primary representations. Except for the user representation 806, the background after the enlarged primary representation 810G is grayed out.

As shown in FIG. 80 , device 600 receives (eg, detects) user input 850E (eg, taps) on cancellation affordance 818 . As shown in Figure 8P, in response to receiving user input 850E, device 600 reduces the size of primary representation 810G and returns the display to the 4-to-1 layout displayed before user input 850D, except primary representation 810G is instead reduced to The medium representation size primary representation 801F occupies the large size primary representation.

Turning to Figure 8Q, a sixth participant joins the real-time communication session. As shown in FIG. 8Q, in response to the sixth participant joining the real-time communication session, and based on the number of connected participants exceeding a predetermined threshold of five participants, device 600 displays an overflow representation of the sixth participant in overflow area 820 822A and move the user representation 806 to the overflow area 820 , and the overflow representation 822A moves to the left of the user representation 806 . As shown in FIG. 8Q, overflow representation 822A and user representation 806 are center-aligned in overflow area 820. In some embodiments, device 600 replaces the smallest primary representation (eg, 810B) with the representation of the newly added participant (eg, sixth participant), and displays the replaced participant's overflow representation in overflow area 820 (eg, representation 810B is reduced in size and moved to the location of overflow representation 822A in overflow area 820). In some embodiments, in response to a sixth participant joining the real-time communication session, and depending on the number of connected participants exceeding a predetermined threshold of five participants, the device 600 displays the currently joining participant or all participants in the overflow area An overflow representation of participants (currently joined and invited), as shown, for example, in Figure 8AY.

As shown in Figure 8Q, overflow region 820 and main representation 810 do not overlap. In some embodiments, to make room for the overflow area, device 600 reduces size and/or vertically translates one or more of the primary representation. In some embodiments, device 600 reduces the size of the medium-sized primary representation and maintains the size of the small and large primary representations. In some embodiments, apparatus 600 reduces vertical spacing and/or increases vertical overlap between primary representations.

As shown in Figure 8R, a seventh participant joins the real-time communication session. In response to the seventh participant joining the real-time communication session, device 600 adds the seventh participant's overflow representation 822B in overflow area 820 to the left of overflow representation 822A. As shown in FIG. 8R , overflow representation 822A and user representation 806 are shifted to the right (compared to FIG. 8Q ) such that overflow representations 822A and 822B and user representation 806 are center-aligned in overflow area 820 . In some embodiments, between the overflow representation 822A and the user representation 806, the seventh participant's overflow representation 822B is added to the right of the overflow representation 822A. In the embodiment shown in Figures 8AZ, in response to the seventh participant joining the real-time communication session, device 600 replaces the smallest primary representation (eg, 810H in Figure 8AY) with primary representation 810I, and will correspond to the overflow of participant Pablo Indicates that the 822F changed from a placeholder to a live video stream.

As shown in Figure 8S, an eighth participant joins the real-time communication session. In response to the eighth participant joining the real-time communication session, device 600 adds overflow representation 822C of the eighth participant in overflow area 820 to the left of overflow representation 822B. As shown in FIG. 8S , overflow representations 822A and 822B and user representation 806 are shifted to the right (compared to FIG. 8Q ) such that overflow representations 822A, 822B and 822C and user representation 806 are center-aligned in overflow area 820 .

As shown in Figure 8T, a ninth participant joins the real-time communication session. In response to the ninth participant joining the real-time communication session, device 600 adds overflow representation 822D of the ninth participant in overflow area 820 to the left of overflow representation 822C. As shown in Figure 8T, overflow representations 822D, 822C, and 822B and user representation 806 are fully displayed. The overflow representation 822A is partially displayed with the right portion overlapping the user representation 806 . The overflow representation 822A is shaded (eg, with a shadow effect), where the overflow representation 822A satisfies the user representation 806 . In the illustrated embodiment, the size of the overflow representation 822 and the user representation 806 remains the same as the seventh, eighth, and ninth participants joining the real-time communication session. 8T shows device 600 displaying combined notification 824 indicating that participants Mary, Mick, John, and Tom have joined the real-time communication (represented by overflow representations 822A, 822B, 822C, and 822D, respectively). In some embodiments, when additional participants (eg, beyond the ninth participant) join the real-time communication session, the overflow representation of the newly joined participant is added to the left of the existing overflow representation, which is in the overflow representation The right shift in area 820 makes room for the new overflow representation, and the user representation 806 remains fully displayed to the right of the overflow area 820 (eg, the existing overflow representation is scrolled to the right "below" the user representation 806 ). In some embodiments, the newly added participant's overflow representation is added to the right end of the existing overflow representation, with the user representation 806 remaining fully displayed to the right of the overflow area 820 . In some embodiments, if the overflow area is fully occupied, representations of newly joined participants added to the right end of an existing overflow representation are not displayed immediately (eg, a new overflow representation is displayed in response to scrolling the overflow representation to the left) .

Turning to Figure 8U, device 600 receives data indicating that the participant corresponding to overflow representation 822B is actively participating in the real-time communication session. In some embodiments, device 600 replaces the primary representation with a representation of the actively engaged participant in response to determining that the activity level of the actively engaged participant has increased above the activity level of the participant represented by the primary representation. In the illustrated embodiment, in response to receiving data indicating that the participant corresponding to the overflow representation 822B is actively participating in the real-time communication session, the device 600 replaces the minimal primary representation with the representation of the actively participating participant and displays the primary representation with the replaced primary representation The overflow representation of the corresponding participant. As shown in FIG. 8U, primary representation 810B (the smallest primary representation) is shaded or begins to fade out; overflow representation 822B of the active participant is shaded or begins to fade out and decreases in size; and when compared to primary representation 810B (by The overflow representations 822C and 822D are shifted to the right as the participant's corresponding new overflow representation 822E enters the overflow area 820 from the left edge of the display 602 to the left of the overflow representation 822D. In some embodiments, the main representation 810B is slightly enlarged (eg, to emphasize that it is being replaced). As shown in Figures 8V-8W, primary representation 810B is replaced by primary representation 810H of the active participant; overflow representation 822B continues to shrink until it is removed; and overflow representation 822C and 822D continue to shift right until overflow representation 822E is fully displayed, and overflow representation 822C occupies the position in overflow area 820 previously occupied by the removed overflow representation 822B. In Figures 8U-8W, overflow representation 822A and user representation 806 remain in the same position.

In some embodiments, the device 600 visually differentiates the user representation 806 in response to the active participation of the user of the device 600 . In the embodiment shown in Figures 8AZ, device 600 displays a widened border around user representation 806 and highlights the initials bar at the bottom of user representation 806.

Turning to Figure 8X, the device 600 is rotated from a portrait (vertical) orientation to a landscape (horizontal) orientation. As shown in FIG. 8X, in response to device 600 being rotated, device 600 maintains representations in the corresponding areas of display 602 that were occupied before device 600 was rotated (eg, the user representation and each of the main and overflow representations occupied prior to device 600 being rotated) and the same area on display 602 afterwards). Rotate the image within the representation so that the participant appears upright relative to the landscape orientation. In some embodiments, device 600 displays an animated rotation of the primary representation or an image within the primary representation (eg, 90 degrees opposite the direction of rotation of device 600). In some embodiments where the overflow representation is not a square, the participant's video data is cropped and/or translated to include (eg, center) the user's face in the corresponding overflow representation as the device 600 is rotated. In some embodiments where the primary representation is not a square, the participant's video data is cropped and/or panned to center the user's face in the corresponding primary representation as the device 600 is rotated. In some embodiments, the participant's video data is cropped to include the user's face (eg, regardless of whether the device 600 is rotated, such as when the received video data has a different aspect ratio or format than the representation in which it is displayed).

As shown in Figure 8X, the participant corresponding to the primary representation 810H begins to actively participate. In response, device 600 enlarges primary representation 810H and reduces the size of primary representation 810E, as shown in FIG. 8Y. In the embodiment shown in FIGS. 8BA-8BB, device 600 visually distinguishes overflow representations 822F (eg, emphasized initials bars) corresponding to active participants.

In some embodiments, in response to a participant joining the real-time communication session, device 600 replaces the existing primary representation with the primary representation of the newly joined participant and moves the replaced participant to overflow area 820 . In Figure 8Z, participant Pablo joins a real-time communication session. In response, the device 600 replaces the smallest primary representation 810E (Stephen) with the primary representation 810I of the newly added participant Pablo, and displays the overflow representation 822F (Stephen) in the overflow area 820 . To make room for overflow representation 822F, overflow representations 822A, 822C, 822D, and 822E are scrolled (eg, upward according to the orientation of device 600 shown in FIG. 8Z). As shown in FIG. 8Z , overflow representation 822A is no longer visible, and overflow representation 822C is partially hidden by user representation 806 .

As shown in Figure 8AA, the device 600 is rotated from a landscape (horizontal) orientation to a portrait (vertical) orientation. In response, device 600 rotates the images of the participants in the representations while maintaining the area occupied by each representation on the display (eg, device 600 reverses the rotation performed in response to the device being rotated from a portrait orientation to a landscape orientation).

In Figure 8AB, the participant corresponding to primary representation 810G leaves (disconnects) the real-time communication session. Optionally, as shown in the embodiment shown in Figure 8BC, in response, the device 600 displays a notification 828 indicating that the participant has left the real-time communication session. As the number of participants remains greater than five, device 600 maintains a 4-to-1 layout with the display of overflow area 820 . As shown in FIG. 8AC, main representation 810G is replaced by main representation 810J of participant Stephen with the next highest activity level, and overflow representation 822F corresponding to participant Stephen is removed from overflow area 820. When the overflow representation 822F is removed, the remaining overflow representations 822A, 822C, 822D and 822E are scrolled to the left. In the embodiment shown in Figure 8BD, the participant Hoan's overflow representation 822D is updated to display a placeholder indicating that the participant Hoan is not connected and a reminder affordance requesting the participant Hoan to rejoin.

In some embodiments, if the number of participants in the real-time communication decreases from six to five when the participant leaves the real-time communication session, the device 600 stops displaying the overflow area 820 and, optionally, increases one or more of the main representations size and/or vertical spacing.

In Figure 8AC, device 600 receives (eg, detects) user input 850F (eg, double-click) corresponding to participant Allison's selection of overflow representation 822E. As shown in FIG. 8AD, in response to receiving user input 850F, device 600 displays a magnified representation 830 of participant Allison in front of the 4-to-1 layout of primary representation 810 that is currently displayed. The display of magnified representation 830 is similar to that of magnified representation 810G in Figure 8N, except that magnified representation 830 is not presented from the primary representation in a 4-to-1 layout.

As shown in FIG. 8AD, device 600 receives (eg, detects) user input 850G (eg, a tap) corresponding to the selection to turn off affordance 818 . As shown in FIGS. 8AE-8AF, in response to receiving user input 850F, device 600 displays participant Allison's primary representation 810K in place of participant Stephen's primary representation 810J (eg, the participant with the lowest activity status replaces the participant with the current primary representation 810J). participants). The main representation 810K includes (at least initially) a band (or sash) 842A along the bottom of the edge that includes the participant's name and an affordance for the magnified representation (eg, as shown in Figure 8AD). 8AE-8AF illustrate exemplary animations in which primary representation 810J is replaced with primary representation 810K. The main representation 810K is initially displayed at a small representation size (similar to the main representation 810J in FIG. 8AC) and then enlarged to a medium representation size, while the main representation 810H is reduced from medium to small. The overflow representation 822E corresponding to participant Allison is reduced in size and/or fades out, while the replaced participant's overflow representation 822F is translated into the overflow area 820 . In some embodiments, stopping the display of the primary representation 810J and displaying the primary representation 810K includes a cross-fading from the primary representation 810J to the primary representation 810K and/or the tint used to create the "glitter" effect. 8BE-8BH illustrate an embodiment in which the primary representation 810J is removed with a flash effect.

In some embodiments, overflow area 820 is scrollable. 8BH-8BI, the device 600 scrolls the overflow representation 822 in the overflow area 820 in response to a participant's active participation, while the corresponding overflow representation for that participant is not currently displayed or fully displayed. As shown in Figure 8BH, the participant's overflow representation 822F (corresponding to the primary representation 810I) is partially hidden in the overflow region 820. In Figure 8BH, a participant corresponding to primary representation 810I is actively participating in a real-time communication session. As shown in FIG. 8BI, in response to receiving data indicating that the participant corresponding to primary representation 810I is actively participating, device 600 zooms in primary representation 810I and scrolls overflow representation 822 such that overflow representation 822F is fully displayed (eg, in the overflow area) 820) and visually indicate overflow representation 822F by highlighting the initials bar at the bottom of overflow representation 822F.

Turning to FIG. 8AG, device 600 detects contact 850H, which moves horizontally from right to left over spill area 820 while maintaining contact with display 602. As shown in FIG. 8AH, in response to receiving contact 850H, device 600 scrolls overflow representation 822 to the left on display 602 such that overflow representation 822F moves partially off the left edge of display 602 and overflow representation 822A no longer overlaps user representation 806 . ISE, touch 850H and overflow means 822 moves the same horizontal amount.

As the overflow area 820 is scrolled, the primary representation 810 remains unchanged. Also, the user representation 806 remains in the overflow area 820 .

In the embodiment shown in FIGS. 8BJ-8BK, the contact 850H starts on the left side of the overflow area 820 and moves horizontally to the right. In response, device 600 scrolls overflow representation 822 on display 602 to the right. In FIGS. 8BJ-8BK, when device 600 reaches user representation 806, the device removes (eg, no longer displays) the representation from display 602.

8AH, device 600 receives (eg, detects) user input 850I (eg, a tap) at a location on display 602 where primary representation 810 is not displayed. As shown in FIG. 8AI, in response to receiving user input 850I, device 600 displays call control menu 832 including call affordance 834 (eg, to leave the real-time communication session), effect affordance 836, menu affordance 838 and handle the affordances 840 .

As shown in FIG. 8AI, the call control menu 832 does not overlap the overflow area 820 and the main presentation 810. To make room for call control menu 832 , device 600 reduces size and/or vertically pans one or more of primary representations 810 (eg, reduces the size of canvas area 811 ) and moves overflow area 820 up on display 602 . In some embodiments, the device 600 reduces the size of the medium-sized primary representation and maintains the size of the small and large primary representations. In some embodiments, apparatus 600 reduces vertical spacing and/or increases vertical overlap between primary representations.

In response to receiving user input 850I, device 600 also displays bands 842A through 842D on each primary representation. Bands 842A-842D include affordances (eg, as shown in FIG. 8O ) corresponding to the participant's name and magnified representation. In some embodiments, the vertical height of the bands 842A-842D is the same as the amount of vertical overlap between the primary representations.

As shown in FIG. 8AJ , device 600 receives (eg, detects) user input 850J (eg, a tap) corresponding to selection of menu affordance 838 . In some embodiments, user input 850J includes a swipe up from call control menu 832 (eg, on processing affordance 840). As shown in Figure 8AK, in response to user input 850J, device 600 expands call control menu 832 to display additional information and controls related to the real-time communication session, similar to those described with respect to call control menu 630 in Figure 6H .

As shown in FIG. 8AL, device 600 receives (eg, detects) user input 850K (eg, taps) corresponding to selection of effect affordance 836 . As shown in FIG. 8AM, in response to receiving user input 850K, device 600 displays a zoomed-in image 843 of the user of device 600 from camera 603 and modifies call control menu 832. In the modified call control menu 832, the effect affordances 836 are highlighted, and the effect option affordances 844A through 844E are displayed. In FIG. 8AN, device 600 receives (eg, detects) user input 850L (eg, a tap) corresponding to the selection of effect option affordance 844A. As shown in FIG. 8AO, in response to receiving user input 850L, device 600 modifies call control menu 832 to include avatar option affordances 846A-846C. In Figure 8AO, device 600 receives (eg, detects) user input 850M (eg, taps) corresponding to selection of avatar affordance 846C. As shown in Figure 8AP, in response to receiving user input 850M, device 600 displays effect 843A on the user's face in image 843 and scrolls call control menu 832 to center affordance 846C of the currently selected avatar on the call control menu 832. In FIG. 8AP, device 600 receives (eg, detects) user input 850N (eg, a tap) on cancel affordance 848. As shown in FIG. 8AQ, in response to receiving user input 850N, device 600 returns call control menu 832 to the configuration of FIG. 8AN and maintains display of effect 843A on magnified image 843. In FIG. 8AQ, device 600 receives (eg, detects) user input 8500 (eg, a tap) on display 602 outside of magnified image 843 and call control menu 832. As shown in FIG. 8AR, in response to receiving user input 850O, device 600 stops displaying magnified image 843 and displays effect 843A on the image of the user in user representation 806. When effect 843A is activated, as in FIG. 8AR, the real-time video stream of the user of device 600 includes effect 843A.

As shown in FIG. 8AR, call control menu 832 includes media affordance 844E. Device 600 receives (eg, detects) user input 850P (eg, taps) corresponding to selection of media affordance 844E. As shown in FIG. 8AS, in response to receiving user input 850P, device 600 modifies call control menu 832 to include media option affordances 852A-852D. In Figure 8AS, device 600 receives (eg, detects) user input 850Q (eg, taps) corresponding to the selection of media option affordance 852A (sketch). In response to receiving user input 850Q, device 600 displays in canvas area 811 a representation of the media content associated with media option affordance 852A.

As shown in FIG. 8AT, in response to receiving user input 850Q, device 600 replaces primary representation 810K (eg, a minimal primary representation) with representation 810L of the selected media item. In FIG. 8AT, device 600 receives (eg, detects) user input 850R (eg, swipe down on call control menu 832) corresponding to a request to dismiss call control menu 832. As shown in FIG. 8AU, in response to receiving user input 850R, device 600 stops displaying call control menu 832.

In some embodiments, selecting the media option affordance 852A causes the associated media content to be shared with other participants of the real-time communication session (eg, displayed in a canvas area on the display of another participant's device). In some embodiments, the participants of the real-time communication session may interact with the shared media content (eg, via a sequence of one or more inputs in the user interface of the real-time communication application). Figure 8AV shows an embodiment in which an additional feature 810L-1 has been added to the media content representing 810L. In some embodiments, in response to a participant's interaction with the media content associated with representation 801L, device 600 magnifies representation 801L (eg, similar to the manner in which the primary representation is magnified and then selected or the participant corresponding to the representation actively participates The way). As shown in FIG. 8AV, the representation 810L is enlarged (compare to FIG. 8AT) in response to the participant's interaction with the associated media content. Figure 8AV also shows a representation 810M of other content (eg, a game between two participants) shared in the real-time communication session.

9A-9K are flowcharts illustrating methods according to some embodiments. Method 900 is performed at a device having a display (eg, 100, 300, 500, 600). Some operations in method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

As described below, method 900 provides, among other things, an intuitive way to provide an adaptive and intelligent real-time communication user interface. Furthermore, method 900 provides, among other things, an intuitive way for a user to interact with a real-time communication user interface. The method reduces the cognitive load on the user by providing an adaptive and intelligent real-time communication interface based on the number of participants in the real-time communication, thereby creating a more efficient human-machine interface. For battery-operated computing devices, enabling a user to view a real-time communication user interface more quickly and efficiently based on a predetermined number of participants in the real-time communication saves power and increases the interval between battery charges.

At block 901, the device receives a request (eg, 850A) to display a user interface for a real-time communication session between two or more participants.

At block 902, in response to receiving a request to display a user interface for a real-time communication session between two or more participants, the device displays a real-time communication user interface (eg, 804), including concurrently displaying the real-time communication session Multiple representations of the participants in (eg, 810A-810D), wherein displaying the multiple representations of the participants in the real-time communication session includes blocks 902 and 903 . In some embodiments, the real-time communication session user interface includes a placeholder (eg, 810A of FIG. 8C ) associated with the first participant (eg, a graphical placeholder (eg, an image or virtual avatar), a textual placeholder sign (e.g., name or initials)), the first participant has been invited to join the real-time communication session but has not yet joined the real-time communication session (e.g., a participant other than the user of the electronic device; yet to join the communication session and has the participant who was sent an invitation to join the communication session). In some embodiments, if the participant is sharing a live media stream that includes a live video stream, the representation includes an image of the live video stream (eg, 810B of Figure 8D). In some embodiments, if the participant is sharing the real-time media stream with audio only, the representation includes the avatar of the corresponding participant (eg, 810E of FIG. 8I ). In some embodiments, the plurality of representations of the participants in the real-time communication session includes selected from an audio-only state (eg, the participant communicates using audio only), a video state (eg, the participant uses video and audio to talk), Video paused state (e.g. participant's video paused), video not decoded state (e.g., participant video stream cannot be processed due to latency issues, video stream format issues, etc.), away state (e.g. participant has left live communication Session), and user communication status pending to join (eg, a participant is invited to a live communication session but has not yet joined the live communication).

At block 903, based on determining that the number of participants in the real-time communication session includes a number of participants greater than a threshold, the device simultaneously displays the first plurality of participants other than the user of the electronic device in one or more sizes greater than the corresponding threshold size representations of participants (eg, 810A-810D), while representations of a second plurality of participants (eg, 822 ) are displayed in the overflow area (eg, 820 ), wherein the second plurality of participants includes participants not included in the first plurality of participants (eg, 822 ). One or more of the participants, and representations of one or more of the other participants displayed in the overflow area are displayed at a size smaller than the corresponding threshold size. In some embodiments, currently active participants (eg, 810E of FIG. 8J ) (eg, the most active participants, those who speak or move in a manner that meets the criteria set of active participants) are displayed as the first most active participant. part of each participant. In some embodiments, the currently active participants are displayed in a main area (eg, 811 ) of the user interface (eg, the area where the first plurality of participants are displayed). In some embodiments, if a participant that is not currently displayed in the main area (eg, the participant corresponding to 822B of FIG. 8U ) becomes the currently active participant, the representation of that participant is displayed in the main area (eg, the participant corresponding to 822B of FIG. 8U ) becomes the currently active participant , 810H of Figure 8V). In some embodiments, the representations (eg, 810) of the first plurality of participants are displayed in two or more different sizes (eg, where the more active participants or the more recent active participants are displayed in two or more different sizes than the less active participants Active participants or older active participants are displayed at a larger size), and representations of the second plurality of participants (eg, 822) in the overflow area are displayed at the same size.

In some embodiments, the overflow area (eg, 820 ) also includes representations (eg, 810A-810D) of the first plurality of participants (eg, representations of participants displayed in the main area of the user interface are also in the overflow area) shown in) (eg, Figure 8AX). In some embodiments, the overflow area (eg, 820) also includes placeholder representations (eg, 822A in FIG. 8AW) (eg, avatars, names, initials). Displaying representations of participants differently based on whether specified conditions are met (eg, based on whether the real-time communication session includes greater than the number of participants) allows a user to more easily or more efficiently view representations of participants in a real-time communication session. Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 904 and 905, the device updates the display (eg, arrangement, location, size, orientation, visual indication, visual indication, presentation) of the representations (eg, 810A-810D) of the first plurality of participants at the first rate. visual content and/or video feeds) (e.g., representations of the first plurality of participants are live video feeds updated at frame rates equal to or greater than typical video refresh rates, such as 24, 30, or 60 frames per second), and The second rate updates the display (eg, arrangement, location, size, positioning, visual indication, visual content of the representation, and/or video feed of the representation) of the representation (eg, 822 ) of the second plurality of participants in the overflow area, wherein The second rate is lower than the first rate. (For example, the thumbnails in the overflow area are updated over time, but at a rate that is less than the video refresh rate of the representations of the multiple participants in the canvas area 811, such as 1, 5, or 10 frames per second). Updating the display of representation groups at different rates allows the device to provide feedback to the user about the current state of representations in real-time communications, while reserving processing power to provide one set of representations at a faster rate than another, thereby proving that the user can Higher rates view higher priority presentation groups. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at block 906, the device, while displaying representations (eg, 822) of the second plurality of participants in the overflow area (eg, 820), displays a view of the electronic device's camera in the overflow area (eg, 822). self-viewing camera view) representation (eg, 806). Displaying a representation of the view of the camera of the electronic device provides visual feedback to the user regarding the current view state of the user's camera (eg, what the user is displaying to other participants in the real-time communication session). Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 907, 908, 909, and 910, the device detects an input (eg, 850H) corresponding to the overflow area (eg, a horizontal finger drag). In response to detecting the input corresponding to the overflow area, the device scrolls the overflow area (eg, scrolls the representation of the second plurality of participants in the first direction according to the input) to stop displaying (eg, hide) the second plurality of participants' representations A subset of representations (eg, 822G and 822H in Figures 8BJ-8BK). While scrolling the overflow area, the device maintains the display of a representation (eg, 806 ) of the electronic device's camera's view in the overflow area (eg, FIGS. 8BJ-8BK ). In some embodiments, scrolling the overflow area includes an animation displaying one or more representations of the second plurality of participants that appear to scroll below the representation of the view of the device's camera. Maintaining the display of the view representation of the electronic device's camera while scrolling the overflow area provides continuous feedback to the user about the current view state of the electronic device's camera (eg, what the user is displaying to other participants in the real-time communication session) while also Allows the user to be able to scroll through representations of participants in the overflow area. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 911, 912, and 913, an input (eg, 850F) corresponding to a selection of a representation (eg, 822E of Figure 8AC) in the plurality of representations of the second plurality of participants in the overflow area is detected (eg, , corresponding to a tap gesture of the representation in overflow area 820 or a gesture with a feature intensity exceeding the intensity threshold corresponding to the representation in overflow area 820). In response to detecting an input corresponding to a selection represented in the plurality of representations of the second plurality of participants in the overflow area, the device uses the representation of the participant corresponding to the selected representation in the overflow area (eg, 810K of Figure 8AE) The display of the representation (eg, 810J in Figure 8AC) in the multiple representations of the first plurality of participants is replaced (eg, the most recently active participant in the primary display is replaced). In some embodiments, replacing the display of the representation in the plurality of representations of the first plurality of participants with the representation of the participant corresponding to the selected representation in the overflow area includes ceasing to display the representation of the second plurality of participants in the overflow area The selected representation in the representation. In some embodiments, ceasing to display the selected representation in the overflow area includes an animation of the selected representation shrinking. In some implementations, other representations in the overflow area are translated to fill the gaps in which the selected representation is displayed. In some embodiments, replacing the display of the representation in the plurality of representations of the first plurality of participants with the representation of the participant corresponding to the selected representation in the overflow area includes displaying the participant corresponding to the selected representation in a first size The representation of the participant is then enlarged to a second size larger than the first size for the representation of the participant corresponding to the selected representation. In some embodiments, the representation of the selected participant replaces the smallest representation in the canvas area (corresponding to the participant with the lowest activity level in the canvas area), and then automatically increases in size (eg, to a medium representation size), While the other representation (corresponding to the participant with the second lowest activity level in the canvas area) is reduced in size (eg, to a small representation size).

Replacing the display of the representation corresponding to the selected representation in the overflow area and/or stopping the display of the selected representation in the overflow area provides additional control options without cluttering the UI, and provides the user with more control over the device and through help The user quickly replaces the representation with the user's selected presentation to display the representation. Providing additional control over the device without cluttering the UI with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing manipulation/interaction with the device) user error while interacting), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at block 914, the device displays a first animation effect (eg, FIGS. 8AA-8AC ) (eg, representing getting bigger, then smaller, optionally with a crossfade and a Hue), thereby affecting the representation in the representations of the first plurality of participants. Optionally, at block 915, the device displays a second animation effect affecting a selected representation of the representations of the second plurality of participants, the second animation effect sharing animation characteristics with the first animation effect (e.g., FIG. 8BF). to FIG. 8BH) (eg, the rate of fading/shrinking from the main display may correspond to the rate of blinking in the overflow display). Displaying a second animation corresponding to the first animation provides visual feedback to quickly identify which representations are being switched and/or replaced. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at block 916, further in response to detecting the input corresponding to the overflow area (eg, 850H), the device displays a new (eg, previously hidden) representation of the second plurality of participants in the overflow area (eg, 822A in Figures 8AG-8AH). Displaying the participant's new representation in the overflow area in response to the input allows the user to view the representation more easily and efficiently by increasing the accessibility of the displayed representation. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at block 917, in response to receiving data indicating that a new participant has joined or has been added to the real-time communication session, the device displays at the end of the series of representations of the second plurality of participants in the overflow area Representation of the new participant in the overflow area (eg, 822B in Figure 8R).

Optionally, at blocks 918, 919, 920, and 921, the device receives second data related to the level of activity (eg, speaking, moving) of the first participant of the second plurality of participants. In response to receiving the second data, the device emphasizes (eg, increases or initiates) a visual characteristic (eg, size, highlighting a representation (eg, a highlighted representation) of the representation based on an increase in the activity level of the first participant in the second plurality of participants. display border), animation (eg, pulsing avatar), bring image to the front) that corresponds to the representation of the first participant in the multiple representations of the second plurality of participants in the overflow area (eg, 822B of Figure 8U) . In response to receiving the second data, the device decreases, de-emphasizes (eg, increases or initiates) a visual characteristic (eg, size, highlighting a representation (eg, an increase) of the representation based on the activity level of the first participant in the second plurality of participants. , showing border), animation (eg, pulsing avatar), bringing image to the front), the representation corresponds to the first participant in the representation of the second plurality of participants in the overflow area. Emphasizing and/or de-emphasizing visual indications when specified conditions are met allows the user to quickly identify the activity level of participants, including the most active and/or least active participants. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, emphasis and/or de-emphasis is provided to provide the user with visual feedback indicating the activity level of each participant when specified conditions are met. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 922 and 923, while displaying the real-time communication user interface, the device detects that a second participant in the second plurality of participants becomes the currently active participant (e.g., the most active participant to satisfy the Active participants speak or move in a standard group manner). The device scrolls the overflow in response to detecting that the second participant of the second plurality of participants becomes the currently active participant and based on determining that the representation of the second participant of the second plurality of participants is not currently displayed in the overflow area area to display a representation of a second participant of the second plurality of participants in an overflow area (eg, 822F of FIGS. 8BH-8BI). Dynamically scrolling the overflow area to display active participants in the overflow area allows viewing representations of active participants in the overflow area without having to select any additional inputs. Reducing the amount of input required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (for example, by helping the user provide appropriate input and reducing user errors in operating/interacting with the device), This in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

At block 924, based on determining that the number of participants in the real-time communication session includes a number of participants that is less than or equal to the threshold, the device displays representations of participants other than the user of the electronic device in one or more sizes greater than the corresponding threshold size , without displaying representations of participants other than the user of the electronic device at one or more dimensions less than or equal to the corresponding threshold dimensions (eg, FIG. 8P ). In some embodiments, the electronic device receives (eg, prior to displaying a user interface for the real-time communication session) multiple real-time media streams associated with respective ones of the multiple participants of the real-time video communication session. In some embodiments, the number of real-time media streams received corresponds to the number of participants currently participating in the real-time video communication session, excluding participants using the device. Displaying representations of participants differently based on whether specified conditions are met (eg, based on whether the live communication session includes less than or equal to the number of participant display representations that have no overflow area) allows the user to more easily or more efficiently view the participants in the live communication session representation of the person. Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

In some embodiments, none of the multiple representations of the remote participants (eg, participants other than the user) overlap each other ( For example, Figure 8J). In some embodiments, at least some of the plurality of representations of remote participants (eg, participants other than the user) are based on determining that the number of participants in the real-time communication session includes a number of participants greater than or equal to the second threshold overlapping each other (eg, Figure 8K).

Optionally, at blocks 925 and 926, based on determining that the number of participants in the real-time communication session includes a number of participants that is less than a second threshold (e.g., two participants, including a user of the electronic device (e.g., one other participant) )), the device displays at least one of the representations of participants other than the user of the device (eg, 810B of FIG. 8H ) in a first aspect ratio (eg, a rectangular aspect ratio). Based on determining that the number of participants in the real-time communication session includes a number of participants greater than a second threshold (eg, three or more participants), the device operates at a second aspect ratio (eg, a square aspect ratio) that is different from the first aspect ratio ratio) displays at least one of the representations of participants other than the user of the device (eg, 810B of FIG. 8I ). Displaying representations of participants at different aspect ratios based on the number of participants in the real-time communication session based on meeting specified conditions, allows the device to provide an intelligent real-time communication user interface by maximizing the size of the representation and/or the organization of the representation to provide a user interface based on the number of participants a better user experience, allowing users to more easily and efficiently view representations on the device. Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 927 and 928, based on determining that the number of participants in the real-time communication session includes a number of participants that is less than a second threshold (eg, two ratio) displays a representation of the device's camera's view (eg, a self-view from the device's camera) (eg, 806 of Figure 8H). Based on determining that the number of participants in the real-time communication session includes a number of participants greater than or equal to the second threshold (eg, three or more participants), the device operates at a fourth aspect ratio that is different from the third aspect ratio (eg, square aspect ratio) displays a representation of the device's camera's view (eg, from the device's camera's self-view) (eg, 806 of Figure 8J). Displaying the view of the device's camera at different aspect ratios based on the number of participants in the real-time communication session based on satisfying specified conditions, allows the device to provide the user with an intelligent real-time communication user interface by maximizing the size of the representation of the camera's view and allows the user to more easily and more efficient viewing of on-device representations. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 929 and 930, based on the number of participants in the real-time communication session being the first number of participants, the device displays representations of the participants in a first layout (eg, arrangement or pattern) (eg, FIG. 8J ) . In some embodiments, the first layout is one of a plurality of predefined layouts in which the user's representation is anchored at different locations in the user interface (eg, a randomly selected layout). In some embodiments, the layout (eg, arrangement) of the representations of the first plurality of participants (eg, other than the user of the device) is determined based on the number of participants in the real-time communication session. Based on the number of participants in the real-time communication session being a second number of participants, the device displays representations of the participants in a second layout different from the first layout (eg, Figure 8K) (eg, layout with participants joining/leaving real-time communication session). In some embodiments, the first layout and the second layout are in a series of predetermined layouts with similar relative placement of the user's representation. (For example, a series of predetermined layouts include layouts that work well together). In some embodiments, a series of predetermined layouts includes layout members for multiple possible numbers of participants (eg, a first member for a non-device user participant (eg, 1+ layout), for A second member of two non-device user participants (eg, 2+ layouts), etc. In some embodiments, the electronic device includes a plurality of predetermined layout series, where members of each series share common characteristics). Displaying representations with similar relative placement of user representations in different layouts based on the number of users allows users to view the representations more easily and more efficiently with increased visibility. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 931, 932, 923 and 934, while displaying representations of participants other than the user of the device, the device receives data indicating that the number of participants in the real-time communication session has increased. In response to receiving data indicating that the number of participants in the real-time communication session has increased, the device reduces the size of one or more representations of participants other than the user of the device (eg, Figures 8J-8K). In response to receiving data indicating that the number of participants in the real-time communication session has increased, the device displays one or more representations of new participants in the real-time communication session (eg, participants who recently joined the group just prior to the original display) ( For example, Figures 8J to 8K).

Reducing the size of one or more representations and displaying the new representation provides feedback to the user about the current state of the real-time communication session and provides visual feedback to the user indicating that a new participant has joined the real-time communication. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 935 and 936, the device receives input (eg, 850B) corresponding to the selection of the first reminder affordance (eg, 808). In response to receiving the input corresponding to the selection of the first reminder affordance, the device sends a new notification (eg, a call, a ringtone, etc.) to the first participant that is different from a previous notification sent to the first participant (eg, the first notification). The second notice is more intrusive than the first notice). Sending a notification to the first participant in response to receiving an input corresponding to a selection that is different from a reminder affordance than previously notified to the participant reduces the amount of input required to perform the operation. Reducing the amount of input required to perform an action (e.g., the number of options required by the user to alert participants) enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user error when operating/interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. Furthermore, automatically performing actions, such as sending a different notification than the first notification without further user input, enhances the operability of the device and makes the user-device interface more efficient (eg, by helping the user provide appropriate input and reduce user error when operating/interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

In embodiments where the placeholders and representations of the first plurality of participants are arranged in a first layout, optionally, at blocks 937, 938 and 939, the device is based on determining that the first participant has not joined the real-time communication session for a predetermined period of time. For a determined amount of time (eg, no real-time communication has been engaged after a predetermined period of time), the display of the placeholder is stopped (eg, Figures 8E and 8F). Further, upon determining that the first participant has not joined the real-time communication session for the predetermined amount of time, the device changes the arrangement of the representations of the first plurality of participants to a second layout (eg, including zooming in and/or) different from the first layout Or move one or more representations of participants (eg, display a layout where the first participant is not included in the layout). In some embodiments, the device receives data from the server or another device to determine that the first participant has not joined the real-time communication session for a predetermined amount of time. In some embodiments, the device may determine that the first participant has not joined the real-time communication session for a predetermined amount of time because it has not displayed any representation of the user. When specified conditions are met, stopping the display of placeholders and changing the placement of the representation allows users to quickly identify when other users are joining or not joining the call, while maximizing the visibility of the displayed participants and/or placeholders. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, ceasing to display the placeholders and changing the arrangement of the display provides feedback to the user about the current state of the real-time communication session and provides visual feedback to the user indicating that certain operations associated with the notification and/or real-time communication session will be performed. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at block 940, upon determining that the first participant has joined the real-time communication session, the device modifies the display of the first placeholder (eg, with a representation of the first participant (eg, 810B of FIGS. 8C-8D ) , placeholders can have participant names). In some embodiments, the device may receive data from a server or another device to make this determination. In some embodiments, the device may make this determination because it does not display any representation of the user. In some embodiments, an animation occurs. Modifying the display of the first placeholder with representations of participants allows the user to quickly identify which users have joined the call, provides feedback to the user on the current state of the real-time communication session, and provides visual feedback to the user indicating that actions will be performed with notifications and/or or certain operations associated with a real-time communication session. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 941 and 942, when multiple representations of the participants in the real-time communication session are displayed, the device detects a change in the number of participants in the real-time communication session. In response to detecting a change in the number of participants in the real-time communication session, a notification is displayed (eg, 812) indicating that the number of participants in the real-time communication session has changed. In some embodiments, when a participant leaves the real-time communication session, the notification is that the representation of the away participant is replaced with a placeholder graphic (eg, including the participant's animation of names or initials). In some embodiments, when a new participant joins a real-time communication session, the notification is subsequently (eg, after a predetermined time) by the new participant's representation (eg, based on a representation of video data transmitted from the new participant's device) ) to replace the display of a placeholder (eg, a graphic including the participant's name or initials) for the new participant. In some embodiments, based on determining that the number of participants in the real-time communication session exceeds a threshold number of participants (eg, when the overflow area includes representations greater than a predefined maximum number (eg, 3 other than the user of the electronic device) or 4 indications)), display a notification indicating that the number of participants in the live communication session has changed. In some embodiments, the device discards the notification showing that the number of participants in the real-time communication session has changed (eg, when there is another visible indication that a participant has joined or left the real-time communication session, such as showing a representation of a newly joined participant or remove the display of representations of participants who have left the session, no notification is displayed). In some embodiments, (eg, when there is no visible indication that participants have joined or left the real-time communication session), the device provides audio output indicating that the number of participants in the real-time communication session has changed without displaying a notification. In some embodiments, the notification includes an indication (eg, name) of the participants who have joined or left the real-time communication session. In some embodiments, if multiple participants join and/or leave the real-time communication session within a threshold amount of time, the device generates a single notification indicating the participants who joined and/or left the real-time communication session within the threshold amount of time quantity. In some embodiments, participants joining the real-time communication session and participants leaving the real-time communication session are each combined into a single notification. In some embodiments, when a participant leaves or joins a real-time communication session, the device delays displaying the notification until a threshold amount of time has elapsed in order to determine whether the notification should include information about other participants joining and/or leaving the real-time communication session . Displaying a notification indicating that the number of participants in the real-time communication session has changed provides feedback to the user about the current status of the number of participants in the real-time communication session and provides visual feedback to the user indicating that the number of participants in the real-time communication session has changed. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 943, 944, 945, and 946, the device displays a representation (eg, 806 of FIG. 8AH) of the device's camera's view (eg, self-view) at the first size as the participant's in the real-time communication session. part of multiple representations. While displaying the multiple representations of the participants in the real-time communication session, the device receives a first input (eg, 850I) corresponding to a selection for displaying a first menu (eg, 832) (eg, based on a menu on the background). next tap or swipe up). In response to receiving the input corresponding to the selection for displaying the first menu, the device displays the first menu (eg, call control menu 832) in the real-time communication user interface (eg, 804), wherein the first menu includes a An option (eg, 836) to display a visual effect (eg, a creative camera) on the representation of the participant in the real-time communication session. In some embodiments, the menu includes other options, such as switching cameras for ending a call (eg, front camera to rear camera). In response to receiving the second input (eg, 850K) corresponding to the selection of the option to display the visual effect (eg, 842 ), the device displays a representation (eg, 842 ) of the view of the device's camera in the second size (eg, 842 ). , overlaying the participant's representation), where the second dimension is larger than the first dimension.

Optionally, at blocks 947, 948, and 949, the device receives a third input corresponding to a selection of a first representation (eg, 810F of Figure 8L) for zooming in on the plurality of representations of the participants in the real-time communication session (eg, 850C) (eg, a double click on the representation in the main area or an input with a characteristic intensity exceeding an intensity threshold). In response to receiving a third input corresponding to a selection for zooming in on a first representation of the plurality of representations of the participants in the real-time communication session, the device zooms in on the first representation (eg, 810F of Figure 8M). The device reduces the size of at least one representation of a plurality of representations of a participant in a real-time communication user interface that is different from the first representation (eg, 810E of Figures 8L-8M). Dynamically reducing the size of one or more representations in a set of displayed representations and enlarging the selected representation provides improved feedback that allows the user to focus and easily view the enlarged representation while minimizing the reduced representation. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 950, 951, 952, and 953, the device receives data indicative of a change in orientation of the electronic device (eg, in a first direction (eg, a change in a 90-degree clock)). In response to receiving data indicative of a change in orientation of the electronic device, the device rotates the display (eg, FIGS. 8W-8X ) of the multiple representations of the participants in the real-time communication session (eg, in a second direction (eg, with the first direction). The direction is reversed to keep the image upright )). The device maintains the relative attributes of previous displays of representations of participants in the real-time communication session (eg, size of another display, location, arrangement, etc.) of user interface objects (eg, representations) while rotating the display. Automatically rotating the device when certain conditions are met and maintaining the relative properties of the previously displayed representation (eg, data indicating that the user is rotating the device) allows the user to view the rotated representation in the same previously displayed location without changing with minimal visual distraction The arrangement, size or position of the representation. Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at block 954, in response to receiving first data indicative of a change in orientation of the electronic device (eg, from a portrait orientation to a landscape orientation), the device is based on the first representation of the midface (eg, the 822C) changing the position of the display of the first representation of the participant's representation in the real-time communication session (e.g., if the representation is not square, adjust the display so as to move from the portrait representation (where the participant's face may be biased toward the top of the representation) to the The participant's face is in the center of the representation when represented laterally).

Optionally, at blocks 955 and 956, while displaying a second representation of the plurality of representations of the participant, the device receives a request to change the aspect ratio of at least one representation of the plurality of representations of the participant in the real-time communication session. In response to receiving a request to change the aspect ratio of at least one representation of the plurality of representations of the participant in the real-time communication session, the device changes the aspect ratio of the second representation from a sixth aspect ratio to a seventh aspect ratio (eg, a square or circle). aspect ratio), where the second representation also includes, in a seventh aspect ratio, a second portion (eg, a portion smaller than the first portion, a cropped portion) of the real-time media stream received by the electronic device, the second portion being selected to be included in the Faces detected in the live media stream (eg, 822C of Figures 8AZ-8BA). In some embodiments, the electronic device changes the aspect ratio of the representation based on the real-time video feed, and crops a portion of the real-time video feed by selecting a portion of the real-time video feed that includes the detected face such that the detected face is not removed from the representation Cut out. Maintaining a view of the face in the representation of the participant provides feedback to the user, allowing the user to view the faces of other participants when changing the aspect ratio. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 957, 958, 959, 960, and 961, when displaying multiple representations of participants in the real-time communication session, the device receives a fourth corresponding to the request to display the menu (eg, 832). Input (eg, 850I) (eg, triggered based on a single tap or swipe up on the background). In response to receiving the fourth input, the device displays a menu (eg, 832) in the user interface for the real-time communication session. The device moves at least a third representation (eg, 810H of Figure 8AI) of the plurality of representations of the participants in the real-time communication session (eg, to accommodate the menu; to avoid overlapping with the menu). The device changes the size (eg, the same representation as the third representation; a different representation from the third representation) (eg, 810F of FIG. 8AI ) of at least a fourth representation (eg, the same representation as the third representation; a different representation from the third representation) (eg, 810F of FIG. 8AI ) of the plurality of representations of the participants in the real-time communication session. , to accommodate the menu; to avoid overlapping with the menu). In some embodiments, when the menu is displayed at the bottom of the screen, the representation is moved up and resized. In some embodiments, at least one representation of the plurality of representations is not resized when the menu is displayed. In some embodiments, the representation is a representation with three predefined image sizes (eg, 1 "small", 2 "medium", and 1 "large"), and only one size representation is adjusted (eg, "medium" image), while keeping "small" and "large" the same size. When the menu is displayed, moving one or more representations and changing the size of one or more representations in the real-time communication session provides additional control options while maintaining the visibility of the representations in the real-time communication user interface. Providing additional control options without cluttering the real-time communication user interface with additional displayed controls enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing operating/interacting with the device) user error while interacting), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

In some embodiments, the device receives an indication that the activity level of the first participant of the second plurality of participants represented in the overflow area increased above that corresponding to the first representation of the plurality of representations of the first plurality of participants Data on the activity level of the participant (e.g. the participant represented in the overflow area starts talking such that his activity level increases above that of one of the participants represented in the canvas area), and in response: use the The representation of the first participant of the two plurality of participants replaces the display of the first representation of the plurality of representations of the first plurality of participants (eg, the participant with the lowest activity level in the canvas area is replaced with the representation of the speaking participant ); and stop showing the representation of the first participant in the overflow area. In some embodiments, ceasing to display the representation of the first participant in the overflow area includes an animation representing a contraction. In some implementations, other representations in the overflow area are translated to fill gaps in the displayed representation. In some embodiments, replacing the display of the first representation of the plurality of representations of the first plurality of participants includes displaying the representation of the participant corresponding to the replaced first representation in the overflow area.

In some embodiments, when displaying multiple representations of participants in the real-time communication session, the device receives an input corresponding to a request to display a menu. In response to receiving an input corresponding to a request to display a menu, the device displays a menu in the real-time communication user interface, where the menu includes options for selecting a content item (eg, game, video, sketch). In some embodiments, the device receives input corresponding to the selection of the content item, and in response to receiving the input corresponding to the selection of the content item, the device displays a representation of the content item with a size greater than the corresponding threshold size (eg, the content item is displayed on A location in the canvas area) replaces representations of participants displayed at a size greater than the corresponding threshold size (eg, representations of participants in the canvas area).

In some embodiments, displaying a representation of the second plurality of participants in the overflow area includes displaying in the overflow area the first a group representation (eg, two representations, including a representation of the least active participant and a representation of the view of the device's camera); and based on determining a second number of participants different from the first number of participants (eg, a total of 7 participants) are participating in a live communication session, showing a second set of representations (e.g., three representations including the representations of the two least active participants and the device's camera's) in the overflow area different from the first set of representations representations of views), where the second set of representations includes a different number of representations than the first set of representations.

Note that the details of the processes described above with respect to method 900 (eg, FIGS. 9A-9K ) also apply in a similar manner to the methods described below/above. For example, method 900 optionally includes one or more features of the various methods described above with reference to methods 700 , 1100 , 1300 and 1500 .

10A-10S illustrate exemplary techniques for, among other things, dynamically adjusting a user interface for a real-time communication session, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIGS. 11A-11F .

FIG. 10A shows a device 600 displaying a user interface 1000 of an active real-time communication session. User interface 1000 includes representations of the five participants in the real-time communication session displayed in canvas area 1001 . Representation 1010A includes placeholders for participants not connected to the real-time communication session. Representation 1010B includes representative images of participants providing audio data only. Representations 1010C and 1010D include real-time video feeds from two respective participants. Representation 1002 includes a view from camera 603 of device 600 .

Each participant has an activity level that determines the layout of the user interface 1000 . In some embodiments, the activity level of the participant is based on audio and/or video feeds received from the participant's device (eg, motion or sound in the video feed). In some implementations, the activity level of the participant is based on filtered audio of the audio feed received from the participant's device (eg, audio filtered from background noise or audio filtered to isolate/amplify speech audio). In some embodiments, the participant's activity level is based on movement identified in the video feed received from the participant's device that satisfies a set of movement criteria (eg, a certain type of movement (eg, hand movement, head movement) /face movement, movement that is not background movement (eg, non-participant movement)).

In some embodiments, the participant's activity level is based on data indicative of activity detected at the first participant's device (eg, data indicative of the first participant performing a user interface action at the participant's device (eg, Choice of Attention Aids or Visual Effects Aids)). In some embodiments, the activity level is determined at an external electronic device (eg, a server) and then transmitted to device 600 .

As shown in FIG. 10A, the activity levels of the participants in the real-time communication session are in the following order of representation, from highest to lowest: representation 1010C, representation 1010D, representation 1010B, and representation 1010A. The tables shown on the right side of Figures 10A-10O list the participants in descending order of activity level from high to low, with participants at the top of list (1) having the highest activity levels and those at the bottom of list (4) having The participant's minimum activity level represented in canvas area 1001. In FIGS. 10A-10O , the representation corresponding to the participant with the highest activity level is the largest representation in canvas area 1001 , and the representation corresponding to the participant with the lowest activity level of the participant represented in canvas area 1001 is the smallest. In FIGS. 10A-10O , the representations corresponding to the participant with the second highest activity level and the third highest activity level have the size of the representation of the participant with the highest activity level and the participation represented in the canvas area 1001 The dimensions between the dimensions of the representation of the participant's lowest activity level.

In some embodiments, a participant's activity level changes relative to other participants. While user interface 1000 is displayed, device 600 receives data indicating that the activity level corresponding to participant Stephen representing representation 1010B has increased above the activity level of participant Nicholas corresponding to representation 1010D. As shown in FIG. 10B, the activity level of representation 1010B increases above that of representation 1010D, but does not change size because it has the second to highest activity level. In Figure 10B, participant Stephen continues to speak and becomes the participant with the highest activity level. In response, the size of representation 1010B is increased, the size of representation 1010C is decreased, and representation 1010A is shifted to the right (eg, to avoid excessive overlap with representation 1010B due to the increased size of representation 1002B), as shown in FIG. 10C . In some embodiments, device 600 changes the size and/or position of three or more representations on user interface 1000 .

As shown in FIG. 10C, representations 1010A to 1010D overlap each other. In some embodiments, one or more of the representations 1010A to 1010D decrease in size, and/or move as they increase in size, to avoid occluding a participant's face and/or to avoid overlapping that exceeds a predefined maximum overlap. In some embodiments, the context of representations 1010A-1010D (also referred to as "z-order") does not change as representation size increases (eg, when a participant actively participates). In some embodiments, the representation 1010 changes size (eg, due to active participation of the corresponding participant or user input at the device 600 that selects the representation (eg, in response to the user input 850C described above in FIG. 8M )). Change the order of one or more representations (eg, move from back to front or from front to back). For example, in some embodiments, representation 1010B moves before representation 1010C and/or representation 1010A when increased in size in FIGS. 10B-10C.

In some embodiments, the representation of the participant remains in the same general area on the display 602 when changing size and/or position. As shown in Figures 10A-10C, after the representations 1010A-1010D have changed size and/or position, each representation occupies at least a portion of the area it occupied before the change in size and/or position.

Turning to Fig. 10D, the participant Stephen, representing 1010B, begins to stop talking, and as shown in Fig. 10E, the participant in the real-time video feed representing 1010A begins to move physically. Detect participant movements in a live video feed. In some embodiments, the participant is determined to be actively participating in the real-time communication session based on motion detected in the participant's real-time video feed (eg, motion indicative of the participant's physical movement). In some embodiments, the movement of the participant results in an increase in the activity level of the participant. In some embodiments, a participant is determined to be actively engaged based on the action satisfying one or more criteria (eg, duration, size, direction, or predetermined gesture).

As shown in Figure 10F, in response to receiving data indicating that the participant representing 1010A is actively participating, device 600 increases the size of representation 1010A and decreases the size of representation 1010D. Indicates that 1010B stays the same size and moves down. In this manner, representation 1010A transitions from the smallest representation to the medium-sized representation, and representation 1010D transitions to become the smallest representation.

In some embodiments, in response to receiving data indicating that a participant representing 1010A (eg, a participant with the lowest activity level) is actively participating, device 600 increases the size of representation 1010A from the smallest representation to the largest representation, and decreases Indicates the size of 1010B. In some embodiments, representation 1010A increases from a minimum representation to a maximum representation based on the active participation of the participant corresponding to representation 1010A as the most recent participant. For example, in some embodiments, device 600 transitions from the layout and relative representation size shown in FIG. 10E to the layout and relative representation size shown in FIG. 10G (below discuss).

In some embodiments, the participant actively participates in the real-time communication session by applying visual effects to the real-time video feed provided to the real-time communication session. In some embodiments, the participant applies the visual effects through the user interface of the participant's device. In some embodiments, the participant is determined to be actively participating in the real-time communication session based on visual effects included in the participant's real-time media feed. As shown in Figure 10F, visual effects (robot faces) are applied to the real-time video feed representing 1010A. As a result, it is determined that the participant representing 1010A is actively participating and increasing the participant's activity level. As shown in FIGS. 10F-10H , upon determining that the participant of representation 1010A is actively participating, device 600 increases the size of representation 1010A to the maximum representation size and reduces the size of representation 1010B to the medium representation size. The relative z-orders of representations 1010A and 101B also change because representation 1010A moves ahead of representation 1010B due to increased activity levels of the participants associated with representation 1010A.

In Figure 10H, device 600 receives (eg, detects) user input 1050A (eg, a tap) on representation 1010C. As shown in Figures 10I-10J, in response to user input 1050A, device 600 increases the size of representation 1010C to the maximum representation size. Additionally, the participant corresponding to representation 1010D begins to actively participate (eg, talk), increasing representation 1010D to a medium size, and decreasing representation 1010B to a small size.

In Figure 10J, device 600 receives (eg, detects) user input 1050B (eg, taps) at a location on display 602 where no representation of the participant is displayed. 10K, in response to user input 1050B, device 600 displays call control menu 1004, which is similar to call control menu 630 described above.

As shown in FIG. 10K , when the call control menu 1004 is displayed, the size of the canvas area 1001 is reduced so that the call control menu 1004 does not overlap the representation 1010 . To make room for call control menu 1004, device 600 modifies the layout of representation 1010 by reducing the size and/or vertically panning one or more representations up. In some embodiments, the device 600 reduces the size of the medium-sized representations 1010A and 1010D, and maintains the size of the small and large primary representations 1010B and 1010C, respectively. In some embodiments, device 600 reduces the vertical spacing between representations (eg, between representations 1010A and 1010C).

In response to receiving user input 1050B, device 600 displays bands 1012A through 1012D on each representation in addition to representation 1002 of the user of device 600 . Each band includes an energy representation of the name and magnified representation of the corresponding participant.

Turning to Figure 10L, device 600 is shown in a landscape (horizontal) orientation. The relative activity levels of the participants were the same as in Figure 10J. Compared to the portrait (vertical) orientation in Figure 10J, the representation occupies the same corresponding area of the display 602 as the portrait (vertical) orientation, and the images within the representation are rotated so that the participant appears to be upright. In some embodiments, in response to device 600 being rotated from the portrait orientation of FIG. 10J to the landscape orientation of FIG. 10L, device 600 displays an animated rotation of the representation or an image within the representation (eg, 90 degrees opposite the direction of rotation of device 600) .

Turning to Figure 10M, starting from the layout and activity levels shown in Figure 10J, a sixth participant, Pablo, joins a real-time communication session. As shown in Figure 10M, in response to the sixth participant joining the real-time communication session, and upon determining that the number of participants connected to the real-time communication session is greater than the predetermined threshold of five connected participants, the device 600 replaces the image of the newly connected participant with an image The image in 1010B that corresponds to the participant with the lowest activity level is represented. Device 600 optionally provides notification that the newly connected participant has joined the real-time communication session, and displays in overflow area 1020 an overflow representation 1022A of the newly connected participant. As shown in FIGS. 10M to 10O , when the overflow area 1020 is displayed, the size of the canvas area 1001 is reduced. At least some of the representations 1010A to 1010D are reduced in size and/or moved on the display 602 .

As shown in FIG. 10N, overflow representation 1010A corresponds to a participant not represented by a primary representation (eg, the representation above overflow area 1020 in FIG. 10N). In Figure 10N, device 600 receives (eg, detects) user input 1050C (eg, a tap) on overflow representation 1022A. As shown in FIG. 100, in response to user input 1050C, device 600 replaces the image in representation 1010A with the image corresponding to the participant of the selected overflow representation 1022A and displays the overflow representation corresponding to the replaced participant (Hoan). 1022B.

In some embodiments, in response to active participation of the participant corresponding to overflow representation 1022A (eg, instead of responding to user input 1050C), device 600 replaces the image in representation 1010A of FIG. 10N with an image of the participant corresponding to overflow representation 1022A Image. In some embodiments, in response to determining that the activity level of the participant corresponding to the overflow representation 1022A has increased above the activity level of the participant corresponding to the representation 1010A of FIG. 10N, the device 600 uses the participant corresponding to the overflow representation 1022A replaces the image in representation 1010A of Figure 10N.

Turning to Figure 10P, a device 1008 with a display 1014 is shown. In some implementations, display 1014 has different characteristics (eg, size and/or aspect ratio) than display 602 of device 600 . Device 1008 displays user interface 1016, which is divided into two parts 1016A (eg, canvas area) and 1016B. Portion 1016A has approximately the same aspect ratio as display 602 of device 600 . According to the aspect ratio of portion 1016A, portion 1016A includes representations 1030A through 1030D, corresponding to the participants in representations 1010A through 1010D in FIG. 100 . As shown in FIG. 10P, the layout (eg, relative size, position, spacing, and overlap) of representations 1030A-1030D is similar to the layout of representations 1010A-1010D in FIG. 10O.

As shown in Figure 10P, device 1008 receives (eg, detects) user input 1050D (eg, a swipe to the right) to move divider 1018 to the right. In response, device 1008 reduces the size of portion 1016B and increases the size of portion 1016A. As shown in Figures 10P-10S, representations 1030 are enlarged and the horizontal spacing between representations 1030 is increased. As shown in Figure 10S, device 1008 displays menu 1024 with effect affordance 1026, call affordance 1028, and menu affordance 1034 (similar to effect affordance 632, call affordance 632, call affordance 1034 described above 634 and menu affordances 636).

11A-11F are flowcharts illustrating methods according to some embodiments. Method 1100 is performed at a device having a display (eg, 100, 300, 500, 600). Some operations in method 1100 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

As described below, method 1100 provides an intuitive way to provide an adaptive and intelligent real-time communication user interface that emphasizes one or more active participants in a real-time communication session. The method relieves the user's cognitive load when viewing active participants in a real-time communication session, thereby creating a more efficient human-machine interface. For battery-operated computing devices, enabling users to view active participants more quickly and efficiently, saving power and increasing the interval between battery charges.

At block 1102, the device displays a real-time video communication user interface (eg, 1000) that includes concurrently displaying a representation of a first participant in the real-time video communication session (eg, 1010C), a second participant in the real-time video communication session of a representation (eg, 1010B) and a representation of a third participant in the real-time video communication session (eg, 1010D). In some embodiments, if the participant is sharing a live media stream that includes a live video stream, the representation includes an image of the live video stream (eg, 1010C). In some embodiments, if the participant is sharing the real-time media stream with audio only, the representation includes the corresponding participant's avatar (eg, 1010B). In some embodiments, the representation includes a placeholder image (eg, an avatar) (eg, 1010A) if the participant is not currently participating in the live video communication session. In some embodiments, the real-time communication interface includes a canvas area (eg, 1001) (eg, the canvas area (eg, the main area) is different from the overflow area). In some embodiments, the canvas area is immediately adjacent to the overflow area. In some embodiments, a representation of the first participant in the real-time communication session, the representation of the second participant in the real-time communication session, and the representation of the third participant in the real-time communication session are displayed in the canvas area (eg, FIG. 10A).

At block 1104, while displaying the real-time video communication user interface, the device receives an indication that the activity level of the second participant (eg, 1010B) in the real-time video communication session has increased above that of the first participant (eg, 1010B) in the real-time video communication session. For example, 1010C) activity level data. In some embodiments, data is received from an external source (eg, a server) indicating that the first participant is actively participating. In some embodiments, the device determines that the first participant is actively participating based on the received data (eg, real-time video or real-time audio from a real-time media feed associated with the first participant). In some embodiments, prior to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session, the first participant, the One or more representations of the second participant and the third participant overlap with at least one other representation of the participant (eg, 1010B, 1010C, and 1010D of Figure 10A).

In some embodiments, the activity level of at least the first participant is based on (eg, based on) audio and/or video feeds (eg, motion or sound in the video feed) and/or instructions received from the first participant's device Data of activity detected at the first participant's device (eg, data instructing the first participant to perform a user interface action (eg, selecting an attention affordance) at the first participant's device). In some embodiments, the activity level is determined at an external electronic device (eg, a server) and then transmitted to the electronic device. In some implementations, the activity level of at least the first participant is based on filtered audio (eg, audio filtered to isolate/amplify speech audio) of the audio feed received from the first participant's device. In some embodiments, the activity level of at least the first participant is based on movement identified in the video feed received from the first participant's device (eg, identified at the server; identified at the electronic device) that satisfies a Group movement criteria (eg, 1010A of FIG. 10E ) (eg, some type of movement (eg, hand movement, head/face movement, movement that is not background movement (eg, non-user movement)).

At block 1106, in response to receiving an indication that the activity level of the second participant in the real-time video communication session (eg, 1010B of FIG. 10A ) has increased above the activity level of the first participant in the real-time video communication session (eg, 10A, 1010C), the device changes the size and/or position of the representation of the first participant in the real-time video communication user interface (at block 1108), changes the size of the representation of the second participant in the real-time video communication user interface and/or position (at block 1110), and change the size and/or position (at block 1112) of the representation (eg, 1010D) of the third participant in the real-time video communication user interface. Changing the size and/or position of the representation when specified conditions are met allows the user to quickly identify the activity level of participants, including the most active and/or least active participants. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Furthermore, changing the size and/or position of the representation only when specified conditions are met reduces the amount of input required by the user to determine or focus on an active participant. Reducing the number of inputs also enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate inputs and reducing user errors in operating/interacting with the device), which in turn makes the The user is able to use the device more quickly and efficiently while reducing power usage and extending the battery life of the device.

In some embodiments, in response to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session, the first participant in the real-time communication user interface The size of the representation of the participant (eg, 1010C of FIG. 10A ) changes, the size of the representation of the second participant in the real-time communication user interface (eg, 1010B of FIG. 10A ) changes, and the size of the third participant in the real-time communication user interface changes The size of the representation (eg, 1010D of FIG. 10A ) changes. In some embodiments, if the least active participant becomes the most active participant (eg, based on an activity score), the device increases the size of the representation of the most active participant and decreases the size of all other representations (eg, based on an activity score). , Figure 10E to Figure 10F). In some embodiments, if the most active participant becomes the least active participant (eg, based on an activity score), the device reduces the size of the representation of the least active participant and increases the size of all other representations. In some embodiments, in response to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session: the first participant in the real-time communication user interface The location of the representation of the participant changes; the location of the representation of the second participant in the real-time communication user interface changes; and the location of the representation of the third participant in the real-time communications user interface changes. In some embodiments, if the least active participant becomes the most active participant (eg, based on an activity score), all positions are swapped (eg, the representations of the most active participants move up or away, while the other representations move down or move away). In some embodiments, if the most active participant becomes the least active participant (eg, based on an activity score), the representation of the least active participant moves down and/or away, while other representations move up Move and/or move away.

In some embodiments, a portion of the representation (eg, 1010C of FIG. 10A ) of the participant having (eg, currently having) the highest activity level for the first, second, and third participants is associated with the first in the real-time communication user interface. , a portion of the representation of the other of the second and third participants overlaps (eg, a portion of the most active representation is below the other representations). In some embodiments, the representation of the participant with the lowest activity level is displayed without overlapping the representation of any other participant (eg, because the representation of the participant with the lowest activity level is the smallest representation of the participant's representation, and So stay on top of the representations of other participants to avoid overshadowing the representation of the participant with the lowest activity level). In some embodiments, the representation of the participant with the lowest activity level (eg, 1010B of FIG. 10I ) is displayed overlapping at least a portion of the representation of the participant with the highest activity level (eg, 1010A of FIG. 10I ). In some embodiments, a portion of the representation (eg, 1010D of FIG. 10H ) of the participant having (eg, currently having) the lowest activity level of the first, second, and third participants is associated with the first in the real-time communication user interface. , a portion of the representation of the other of the second and third participants (eg, 1010C of FIG. 10H ) overlaps a portion (eg, a portion of the least active representation is below the other representations).

Optionally, at block 1114, the device alters the first participant's facial recognition data based on facial recognition data obtained (eg, obtained through analysis) from the second participant's video feed (and/or the third participant's video feed). The size and/or position of the representation, wherein changing the size and/or position of the representation of the first participant does not result in obscuring the face of the second participant (and/or obscuring the face of the third participant). Changing the size and/or position of a participant's representation in a real-time communication session without obscuring another participant's face provides feedback to the user about the participant's current activity level without hiding other participants in the real-time communication session Participants' faces, allowing the user to view the faces of other participants when the size or position of a participant whose activity level has changed changes. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at block 1116, the device changing the size and/or position of the representation of the first participant (eg, 1010B of FIGS. 10A-10C ) includes gradually increasing the size of the representation from the first size (eg, over a period of time) gradually increase over time) to the second size. Changing the size and/or position of the representation provides feedback to the user about the participant's current activity level while minimizing user interface changes. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently. Gradually increasing the size of the representation provides visual feedback to the user that the participant's activity level is changing as their activity level changes, while minimizing disruption to the user caused by sudden changes in size. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at block 1118, the device detects a canvas size change event (eg, 1050B; Figures 10J-10K) (eg, rotation of the device; other elements (eg, for one or more calls to contact participants) affordance; adding representations of participants displayed outside the canvas area) into the real-time video communication user interface; user input corresponding to a request to change the size of the canvas area). Optionally, at block 1120, in response to detecting a canvas size change event, the device changes the size of the canvas area (eg, 1001) (optionally, at block 1124). In response to detecting the canvas size change event, the device modifies the layout of the representations of the first, second, and third participants (eg, 1010B, 1010C, and 1010D of Figure 10K). In some embodiments, the layout of the representation is modified by shifting the position of the representation and/or changing the size of the representation (eg, to reduce the spacing between representations as the size of the canvas area decreases). Intelligently modifying the layout based on changes in the size of the canvas area allows the user to view the user's representation with minimal changes to the displayed representation when the canvas size changes, thereby reducing distraction to the user while allowing the user to view the displayed presentation. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at block 1126, the device also changes the size of one or more representations of the first, second, and third participants (eg, 1010B, 1010C, and 1010D of FIG. 10K ) in response to detecting the canvas size change event. size. In some embodiments, before receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session, the first participant in the real-time communication user interface , representations of the second and third participants (eg, 1010B, 1010C, and 1010D of FIG. 10K ) are displayed according to z-order (eg, "z-order" refers to the representation of participants relative to an imaginary axis perpendicular to the face of the display (eg, axes coming out of the screen) (eg, when the positions of two representations overlap, the z-order of the representations determines which object is displayed before the other). At block 1128, the device maintains the first and second and the z-order of the representation of the third participant (eg, 1010B, 1010C, and 1010D of Figure 10K).

In some embodiments, after changing the size and/or position of the representation of the first participant (eg, 1010B of FIG. 10B ) in the real-time communication user interface, the changed representation of the first participant occupies at least a portion of the area (eg, 1010B of FIG. 10B ). , general area, location, pixels on the display, etc.) that was occupied by the representation of the first participant in the real-time communication user interface prior to changing at least one of the size or location of the representation of the first participant. In some embodiments, after changing the size, the participant is displayed in the same area of the display as the participant was previously in (eg, at least a portion (eg, portion, pixel, location, etc.) of the representation) continues to touch it prior to the change position on the previously touched display). In some embodiments, representations maintain the placement/general position of representations while emphasizing active participants (eg, representations shift as they increase/decrease in size but maintain position relative to other images). In some embodiments, after changing the size and/or position of the representation of the second participant (eg, 1010C of FIG. 10B ) in the real-time communication user interface, the altered representation of the second participant occupies at least a portion of the area (eg, 1010C of FIG. 10B ). , general area, location, pixels on the display, etc.) that was occupied by the representation of the second participant in the real-time communication user interface prior to changing at least one of the size or location of the representation of the second participant. In some embodiments, after changing the size, the participant is displayed in the same area of the display as the participant was previously in (eg, at least a portion (eg, portion, pixel, location, etc.) of the representation) continues to touch it prior to the change position on the previously touched display). In some embodiments, representations maintain the placement/general position of representations while emphasizing active participants (eg, representations shift as they increase/decrease in size but maintain position relative to other images). In some embodiments, after changing the size and/or position of the representation of the third participant (eg, 1010D of FIG. 10B ) in the real-time communication user interface, the altered representation of the third participant occupies at least a portion of the area (eg, 1010D of FIG. 10B ). , general area, location, pixels on the display, etc.) that was occupied by the representation of the third participant in the real-time communication user interface prior to changing at least one of the size or location of the representation of the third participant. In some embodiments, after changing the size, the participant is displayed in the same area of the display as the participant was previously in (eg, at least a portion (eg, portion, pixel, location, etc.) of the representation) continues to touch it prior to the change position on the previously touched display). In some embodiments, representations maintain their arrangement and/or general position while emphasizing active participants (eg, representations shift as they increase/decrease in size but maintain position relative to other images).

Displaying representations occupying at least a portion of the area they previously occupied after changing the size and/or position of the representations allows the user to identify the activity level of the participants with minimal changes to the user interface, thereby increasing the user's focus on participating in a real-time communication session the ability to express. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at block 1130, the device displays a representation of the fourth participant in the real-time communication user interface (eg, 1010D of Figure 10E). Optionally, in response to receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session, the device changes a fourth in the real-time communication user interface. The size and/or location of the participant's representation (at block 1132). In some embodiments, the fourth representation changes size and/or position when all other representations change size or position. In some embodiments, the fourth representation changes (eg, in response to receiving first data indicating that the activity level of the second participant in the real-time communication session has increased above the activity level of the first participant in the real-time communication session) . In some embodiments, some representations, such as the fourth representation, may maintain their size and position even though other representations are changing.

Optionally, at blocks 1134, 1136, 1138, 1140, and 1142, after changing the size and/or position of the representation of the first participant in the real-time video communication user interface, the device receives an indication of the first participant in the real-time video communication session. second data that the activity level of the participant has increased above the activity level of the second participant in the real-time video communication session. In response to receiving second data indicating that the activity level of the first participant in the real-time video communication session has increased above the activity level of the second participant in the real-time video communication session, the device changes the first participant in the real-time video communication user interface The size and/or location of the person's representation. The device changes the size and/or position of the representation of the second participant in the real-time video communication user interface. The device maintains the size and position of the representation of the third participant in the real-time communication session. In some embodiments, representations of some representations displayed in the canvas (eg, the main area) are maintained in response to changes in the activity level of the participant, even if other representations change size or position (eg, two representations may be based on determining other representations) means changing its size and position to swap positions). In some embodiments, the maintained representation corresponds to a participant having a lower activity level than the participant's activity level corresponding to the altered representation. Maintaining the size and/or position of some representations while changing the size and/or position of other representations when specified conditions are met allows users to quickly identify participant activity levels with minimal changes to the user interface, thereby increasing user focus on real-time communications The capabilities of the representation of the participants in the session. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 1144 and 1146, in response to receiving the first data indicating that the activity level of the second participant in the real-time communication session has increased above the activity level of the first participant in the real-time communication session, the device emphasizes (e.g., adding or initiating) a visual indication (e.g., highlighting the representation (e.g., showing a border), animation (e.g., when the second participant is currently more active) of a representation of the second participant (e.g., when the second participant is currently more active) in the real-time communication user interface , pulse avatar), make image front, increase size). In response to receiving the first data indicating that the activity level of the second participant in the real-time communication session has increased above the activity level of the first participant in the real-time communication session, the device de-emphasizes (eg, reduces or stops) the real-time communication user Visual indication of representation of the first participant in the interface (e.g., when the first participant is not currently active) (e.g., highlighting the representation (e.g., showing a border), animation (e.g., pulsing avatar), putting an image in the back , reduce the size). Emphasizing and/or de-emphasizing visual indications when specified conditions are met allows the user to quickly identify the activity level of participants, including the most active and/or least active participants. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, emphasis and/or de-emphasis is provided to provide the user with visual feedback indicating the activity level of each participant when specified conditions are met. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

In some embodiments, before (eg, before) the device receives data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session: the first One participant's representation is displayed at a first size (eg, the middle size of a set of 3 or more predefined sizes; the largest size in the set), and the second participant's representation is displayed at a first size smaller than the first size. A second size (eg, the smallest size in a group of 3 or more predefined sizes; the middle size in the group) is displayed (eg, a small representation). Optionally, at block 1148, the device increases the size of the representation of the second participant from the second size to the first size. Increasing the size and/or position of the representation from the smaller size to the larger size provides visual feedback to the user indicating that the participant's activity level is changing as their activity level changes. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 1150 and 1152, after receiving data indicating that the activity level of the second participant in the real-time video communication session has increased above the activity level of the first participant in the real-time video communication session, the device detects An input (eg, 850C) corresponding to the representation of the first participant (eg, a tap on the representation). In response to detecting an input corresponding to the representation of the first participant (eg, the user may change the size of any representation displayed on the real-time communication user interface (eg, in the overflow area, canvas area, etc.), the device increases the first participant the activity level of the person (eg, 810F of FIGS. 8L-8M) (eg, increase the value of the activity level).

At blocks 1154 and 1156, the device receives data indicating that the fourth participant in the real-time communication session is more active than the participant with the lowest activity level of the first, second, and third participants. In response to receiving data indicating that the fourth participant in the real-time communication session is more active than the participant with the lowest activity level of the first, second, and third participants, a representation of the fourth participant (e.g., FIG. 100 1010A) of FIG. 10N replaces the display of the representation of the participant with the lowest activity level of the first, second, and third participants (eg, 1010A of FIG. 10N). In some embodiments, at block 1158, the device displays a representation of the fourth participant in a size that is larger than the size of the representation of the participant with the lowest activity levels of the first, second, and third participants. Replacing the display of representations of participants with the lowest activity level with participants not shown in the first set of representations, when specified conditions are met, allows the user to focus on representations of the most active participants in the real-time communication session, while When the user is more active than the user shown in the first set of representations, reduce the user's attention to the least active participant in the real-time communication session. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Increasing the size of the newly displayed participant provides the user with feedback on the current activity level of the other displayed participants (eg, in the first subset of the representation) in relation to the new participant, while minimizing user interface changes. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at block 1160, the device displays a representation (eg, 1002) of a view (eg, self-view) of the camera of the electronic device in the real-time communication session user interface. Optionally, at block 1162, the device is also responsive to receiving first data indicating that the activity level of the second participant in the real-time communication session has increased above the activity level of the first participant in the real-time communication session: optional At block 1164, the dimensions of the representation (eg, 1002) of the view of the camera of the electronic device are maintained in the real-time communication session user interface. In some embodiments, the self-view does not change based on changes in the activity levels of other participants. The dimensions of the representations of the view of the device's camera are maintained, while other representations may change when a set of conditions have been met, allowing users to view their representations while continuously minimizing changes to the layout. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Note that the details of the processes described above with respect to method 1100 (eg, FIGS. 11A-11F ) also apply in a similar manner to the methods described below/above. For example, method 1100 optionally includes one or more features of the various methods described above with reference to methods 700 , 900 , 1300 or 1500 .

12A-12N illustrate exemplary user interfaces for initiating a real-time communication session, particularly from an instant messaging application, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below including the processes in Figures 13A-13D.

12A shows a device 600 displaying an instant messaging user interface 1200 of an instant messaging application (as described above). The instant messaging user interface 1200 includes messages in a message area 1202 between participants in a group of 15 participants ("dream team") participating in a message conversation. The instant messaging user interface 1200 includes a call full affordance 1206 for initiating a real-time communication session with all participants in a messaging conversation (eg, all participants in a "Dream Team" group) and for accessing and communicating with the "Dream Team" group. The group contact information affordance 1208 for additional information related to the "Team" group.

As shown in FIG. 12A , device 600 receives (eg, detects) user input 1250A (eg, a tap) corresponding to the selection of call full affordance 1206 . In response to receiving user input 1250A, device 600 initiates a real-time communication session with all participants in the message conversation that are included as participants in the real-time communication session. In some embodiments, in response to receiving user input 1250A, device 600 sends instructions that cause devices associated with members of the group to display a notification that the real-time communication session is active. In this manner, the call full affordance 1206 immediately begins a real-time communication session with all participants in the messaging conversation group.

Figure 12B shows the instant messaging user interface 1200 of the instant messaging application when a real-time communication session is initiated. The instant messaging user interface 1200 includes a message 1212 in the message area 1202 and a visual indicator 1214 at the top of the display 602, which indicates that the instant communication session is active.

In some embodiments, in response to receiving user input 1250A, device 600 launches a real-time communication application. As shown in FIG. 12C, after the real-time communication session begins, the device 600 displays the real-time communication user interface 1216 of the real-time communication application. The real-time communication user interface 1216 corresponds to a real-time communication session initiated with a "dream team" group.

As shown in FIG. 12C, the real-time communication user interface 1216 includes four primary representations 1210A through 1210D (collectively 1210) of corresponding participants (eg, the first four participants in the participant list of the "Dream Team" group) and User representation 1218 of the user of device 600 . The real-time communication user interface 1216 also includes an overflow area 1220 having an overflow representation 1222 corresponding to the participants of the real-time communication session (eg, each member of the "dream team" group). Primary representation 1210B includes live video from participant Allison, indicating that Allison has joined a live communication session with video data.

Primary representations 1201A, 1210C, and 1210D include placeholders indicating that the corresponding participant has not joined (eg, is not connected to) a real-time communication session. The placeholder includes a textual representation of the participant's name (eg, first name or first and last initials). Optionally, the placeholder includes a status indicator indicating that the participant is waiting to connect to the real-time communication session, and an alerting indicator (e.g., "bell" 808A) for sending a reminder to the corresponding participant that the real-time communication session is available. ). In some embodiments, the reminder enabler is not initially included on the placeholder and is displayed based on a determination that the corresponding participant has not joined the real-time communication session after a predetermined period of time.

In some embodiments, the device 600 detects an input corresponding to the selection of the reminder affordance (eg, a tap on the reminder affordance), and in response, sends an instruction to cause a new notification (eg, a call, a ringtone, etc.) ), which indicates that a real-time communication session is available at the device associated with the corresponding participant, wherein the new notification is different from the notification previously sent to the participant. In some embodiments, the new notification is more intrusive than the previous notification (eg, an audio ringtone, haptic output, or full screen notification compared to a banner notification without audio or haptic output). In some implementations, the reminder enabler is displayed in the overflow representation 1222 (eg, according to the same criteria used to display the reminder enabler in the primary representation). In some embodiments, selecting a reminder energy on the overflow representation results in the same response as described above with respect to reminder energy entries displayed on the primary representation (eg, a new, updated message to the corresponding participant). intrusive notices).

In some embodiments, upon determining that one of the participants represented by the placeholder has joined the real-time communication session, the device 600 modifies the representation associated with the user (eg, replaces the placeholder with real-time video; see, eg, Figures 8C- Figure 8D). In some embodiments, modifying the representation includes animation. In some embodiments, device 600 receives data from a server or other device to determine whether a participant has joined the session. In some embodiments, device 600 determines that a participant has joined the session based on receiving real-time media data associated with the participant.

In some embodiments, the device 600 changes the layout of the representation displayed in the real-time communication session user interface 1216 upon determining that one of the participants represented by the placeholder has not joined the real-time communication session for a predetermined amount of time (eg, the device 600 Stop displaying representations corresponding to participants who have not joined the real-time communication session for a predetermined amount of time, and display a layout with fewer primary representations or replace that participant's representation with another participant's representation; see, e.g., FIG. 8E to 8F). In some embodiments, device 600 receives data from a server or other device to determine whether a participant has not joined the real-time communication session for a predetermined amount of time. In some embodiments, device 600 determines whether a participant has not joined a real-time communication session for a predetermined amount of time based on not receiving real-time media data associated with the participant.

12D shows the real-time communication user interface 1216 at a later time during the real-time communication session with the "Dream Team" group. As shown in Figure 12D, the real-time communication user interface 1216 includes a call control menu 1224, which is similar to the call control menu 630 described above. As described above, in some embodiments, call control menu 1224 is displayed in response to user input (eg, a tap or swipe up on display 602). As shown in FIG. 12D, device 600 receives (eg, detects) user input 1250B (eg, swipe up) corresponding to a request to expand call control menu 1224. As shown in FIG. 12E, in response to user input 1250B, device 600 expands call control menu 1224. The extended call control menu 1224 is similar to the extended call control menu 630 described with respect to FIGS. 6I-6K. As shown in Figure 12E, the call control menu 1224 includes a reminder affordance 1226 associated with the participant Hoan. In some embodiments, device 600 detects an input corresponding to selection of reminder affordance 1226 (eg, a tap on reminder affordance 1226 ), and in response, sends an instruction to cause a new notification (eg, call, Ringtone, etc.), which indicates that a real-time communication session is available at the device associated with participant Hoan, wherein the new notification is different from the notification previously sent to participant Hoan. In some embodiments, the new notification is more intrusive than the previous notification (eg, an audio ringtone, haptic output, or full screen notification compared to a banner notification without audio or haptic output).

Figure 12F again shows the real-time communication user interface 1216 prior to initiating a real-time communication session with the message group "Dream Team". As shown in FIG. 12F , device 600 receives (eg, detects) user input 1250C (eg, taps) corresponding to selection of group contact affordance 1208 . As shown in Figure 12G, in response to receiving user input 1250C, device 600 displays group contacts user interface 1228 that displays additional information related to the "Dream Team" group, including real-time communication session representations of participants in the group Representation 1230 and List 1232. The group contacts user interface 1228 also includes an add contact affordance 1234 . In some embodiments, in response to detecting an input (eg, a tap) corresponding to the selection of the add contact affordance 1234, the device 600 initiates a process for adding a new contact to the real-time communication session (eg, by Displaying a user interface with text entry fields for entering participant names using the keyboard and/or adding participant affordances for selecting participants from a menu such as a contact list (eg, as described below with reference to FIG. 14F ) )).

As shown in FIG. 12H , device 600 receives (eg, detects) user input 1250D (eg, taps) corresponding to selection of real-time communication session affordance 1230 . In response to receiving user input 1250D, device 600 launches a real-time communication application and begins a real-time communication session with participants in the "Dream Team" group (eg, as shown in Figure 12C).

The user may also select representations of participants in the group on the group contacts user interface 1228 to access additional information and options related to the corresponding participant. As shown in Figure 12I, device 600 receives (eg, detects) user input 1250E (eg, taps) corresponding to participant Allison's selection of list item 1232B. In response to user input 1250E, device 600 displays individual contacts user interface 1236 with additional information associated with the selected participant and options 1238A-1238E for individually communicating with the participant through various communication modes. As shown in Figure 12J, the individual contact user interface 1236 includes, inter alia, affordances 1238A, 1238B, and 1238C for initiating individual messaging conversations, individual phone calls, and individual real-time communication sessions (eg, video communications), respectively, with participant Allison .

In some embodiments, the device 600 determines whether the feature strength of the user input 1250E exceeds a threshold strength. As shown in Figure 12K, upon determining that the characteristic strength of user input 1250E exceeds a threshold strength, device 600 displays menu 1240 over group contact user interface 1228, where options 1242A-1242E correspond to various individual communications with participant Allison communication mode.

12L-12N illustrate exemplary 4-to-1, 3-to-1, and 2-to-1 layouts, respectively. In some embodiments, device 600 displays one of the initial layouts shown in Figure 12L based on a real-time communication session that includes five participants (including the user of device 600). In some embodiments, device 600 displays one of the initial layouts shown in Figure 12M based on a real-time communication session that includes four participants (including the user of device 600). In some embodiments, device 600 displays one of the initial layouts shown in Figure 12N based on a real-time communication session that includes three participants (including the user of device 600). In some embodiments, device 600 displays representations of participants in one of the layouts shown in FIGS. 12L-12N as the initial layout when initiating a real-time communication session based on the number of invited and/or connected participants ( For example, instead of the layout shown in FIG. 12C ). For example, in some embodiments, based on a real-time communication session that includes a total of four participants, device 600 displays representations of participants according to one of the layouts shown in Figure 12M.

13A-13D are flowcharts illustrating methods according to some embodiments. Method 1300 is performed at a device having a display (eg, 100, 300, 500, 600). Some operations in method 1300 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

As described below, method 1300 provides an intuitive way to initiate and interact with a real-time communication session between multiple participants. The method reduces the cognitive load on the user by providing intelligent initiation and interaction of real-time communication sessions among multiple participants, thereby creating a more efficient human-machine interface. For battery-operated computing devices, enabling users to interact and initiate real-time communication sessions more quickly and efficiently saves power and increases the interval between battery charges.

At block 1302, the device displays an instant messaging user interface (eg, 1200) of the instant messaging application. The instant messaging user interface includes a message area (eg, 1202) that includes multiple messages between participants (including three or more participants) in a messaging conversation. The instant messaging user interface includes an affordance (e.g., 1206) separate from the message area (e.g., 1202) for starting a real-time video communication session in the real-time video communication application (e.g., a message detail affordance or having real-time video an able representation of a graphical indication of a communication application). In some embodiments, the affordance is a call-to-all affordance (eg, 1206), which will immediately begin a real-time communication session with all participants in the instant messaging application. In some embodiments, the affordance is a group contact affordance (eg, 1208 ) for displaying a group contact user interface to invite participants to a real-time communication session.

At block 1340, the device detects an input (eg, 1250A) corresponding to a selection of an affordance (eg, 1206). At block 1306, in response to detecting the input corresponding to the selection of the affordance, the device initiates a process in the real-time video communication application for initiating a real-time video communication session, the real-time video communication session including the participants of the messaging conversation. In some embodiments, the real-time video communication application (eg, 1216) is launched, and a new real-time video communication session is started directly in response to selection of an affordance. In some embodiments, a menu of the instant messaging application (eg, 1228) is displayed in response to selection of an affordance, and a menu option (eg, 1230) is selected to launch the real-time video communication application and start a new real-time video communication session. When specified conditions are met (eg, by detecting an input), automatically initiating a process for starting a real-time video communication session in a real-time video communication application allows a user to start a real-time communication session. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, providing additional controls (eg, affordances) to initiate a process for initiating real-time communications provides additional control options (eg, affordances) without rendering the UI due to additional displayed controls (eg, group chat UIs). clutter, which enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn by enabling the user to The device is used more quickly and efficiently while reducing power usage and extending the battery life of the device.

Optionally, at block 1308, the device transitions (eg, immediately in response to detecting user input) from the instant messaging user interface (eg, 1200) to the instant messaging application's user interface (eg, 1216). Optionally, at block 1310, the device begins a real-time communication session. Optionally, at block 1312, the device displays a group contacts user interface (eg, 1228), the group contacts user interface (eg, the interface displayed while continuing to display at least a portion of the instant messaging user interface or replacing the instant messaging user The discrete user interface of the interface) includes a group contact affordance (eg, 1230) for initiating a real-time communication session. When specified conditions are met (eg, by detecting an input), automatically transitioning from the instant messaging user interface and starting the real-time video communication session in the real-time video communication application allows the user to immediately start the real-time communication session. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, transitioning from an instant messaging user interface and starting a real-time video communication session in a real-time video communication application reduces the amount of input required to perform an operation when specified conditions are met. Reducing the amount of input required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors in operating/interacting with the device), This in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

In some embodiments, the group contacts user interface further includes a plurality of participant affordances (eg, 1232 ) that include a first participant of the participants (eg, a list of individual participants) in a conversation with the message The associated first participant affordance (eg, 1232B)). In some embodiments, the group contacts user interface also includes a new contact affordance (eg, 1234). Optionally, at block 1314, the device starts a real-time communication session in a real-time communication application. Displaying the group contact UI provides additional control options (eg, affordances) without cluttering the UI with additional displayed controls when additional controls are required, which enhances device operability and makes the user-device interface more efficient (eg, by helping the user provide appropriate input and reducing user error when operating/interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at block 1316, the device displays a real-time communication session user interface (eg, 1216) (eg, an interface for a real-time communication placeholder) that includes (eg, initially includes) a placeholder (eg, a graph 1210A of 12C) (e.g., a graphical placeholder (e.g., an image or avatar), a first participant (e.g., a participant other than the user of the electronic device) of the real-time communication session participants, yet to join the communication A text placeholder (eg, name or initials) associated with a participant in the session and who has been sent an invitation to join the communication session). In some embodiments, the real-time communication session user interface further includes a second placeholder associated with the second participant. In some embodiments, the first placeholder includes a first reminder affordance (eg, a bell button) for a first participant of the real-time communication session participant (eg, FIG. 12C ). In some embodiments, the real-time communication session user interface also includes a second placeholder (eg, 1210C) associated with the second participant. In some implementations, the first placeholder and the second placeholder are arranged according to the first layout. Displaying the real-time communication session user interface when the real-time communication begins provides feedback to the user about the current state of the real-time communication session and provides visual feedback to the user indicating that operations associated with the real-time communication interface can be performed. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at block 1318, upon determining that the first participant has joined the real-time communication session, the device replaces the display of the first placeholder (eg, FIG. 12C ) with a representation of the first participant (eg, 1210A of FIG. 12D ). 1210A) (eg, placeholders may have participant names). In some implementations, an animation occurs (eg, Figures 8T-8W). In some embodiments, the device receives data from a server or another device to make this determination. In some embodiments, the device makes this determination because it does not display any representation of the user. Modifying the display of the first placeholder with representations of participants allows the user to quickly identify which users have joined the call, provides feedback to the user on the current state of the real-time communication session, and provides the user with an indication that notification and/or real-time communication will be performed Visual feedback for certain actions associated with the session. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at block 1320, the device stops (optionally, the device stops) based on determining that the first participant has not joined the real-time communication session for a predetermined amount of time (eg, has not joined the real-time communication after a predetermined period of time). At block 1322) the first placeholder is displayed, and optionally at block 1324, the device changes the arrangement of the second placeholder to a second layout different from the first layout (eg, a display layout in which the first Participants are not included in this layout). In some embodiments, the device may receive data from a server or another device to make this determination. In some embodiments, the device may make this determination because it does not display any representation of the user. When specified conditions are met, stopping the display of placeholders and changing the placement of the representation allows users to quickly identify when other users are joining or not joining the call, while maximizing the visibility of the displayed participants and/or placeholders. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, ceasing to display the placeholders and changing the arrangement of the display provides feedback to the user about the current state of the real-time communication session and provides visual feedback to the user indicating that certain operations associated with the notification and/or real-time communication session will be performed. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 1326 and 1328, the device receives input (eg, 850B) corresponding to the selection of the first reminder affordance (eg, 808). In response to receiving the input corresponding to the selection of the first reminder affordance, the device sends a new notification (eg, a call, a ringtone, etc.) to the first participant that is different from a previous notification sent to the first participant (eg, the first notification). The second notice is more intrusive than the first notice). Sending a notification to the first participant in response to receiving an input corresponding to a selection that is different from a reminder affordance than previously notified to the participant reduces the amount of input required to perform the operation. Reducing the amount of input required to perform an action (e.g., the number of options a user needs to call a participant) enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user error when operating/interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. Furthermore, automatically performing actions, such as sending a different notification than the first notification without further user input, enhances the operability of the device and makes the user-device interface more efficient (eg, by helping the user provide appropriate input and reduce user error when operating/interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 1330 and 1332, the device detects an input (eg, 1250C) corresponding to the selection of a group contact affordance (eg, 1208) for starting the real-time communication session. In response to detecting the input corresponding to the selection of the group contact affordance for starting the real-time communication session, the device initiates the real-time communication session in the real-time communication application. (eg, transitioning to the user interface of a real-time communication application). In some embodiments, the group contacts user interface includes a list (eg, 1232) of participants for the real-time communication session. In some embodiments, a list of participants is provided without options and/or affordances for individually joining or starting a video or audio conference with a given participant. In some embodiments, the instant messaging user interface includes two affordances for starting a real-time video communication session in the real-time video communication application: a first affordance that when selected results in display of the group contacts user interface, and a second affordance that, when selected, results in transitioning to the user interface of the real-time communication application and starting (eg, immediately starting) a real-time communication session. Beginning a real-time communication session in a real-time communication application in response to selection of a group contact affordance provides additional control options (eg, affordances) without cluttering the UI with additional displayed controls when additional controls are required, which enhances The operability of the device, and making the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors in operating/interacting with the device), which in turn by enabling the user to use the device more quickly and efficiently The device reduces power usage and extends the battery life of the device. Furthermore, starting a real-time communication session in a real-time communication application in response to selection of a group contact affordance reduces the amount of input required to perform an operation. Reducing the amount of input required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors in operating/interacting with the device), This in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 1334, 1336, 1338, and 1340, the device, after initiating communication: displays a reminder associated with the first participant in response to determining that the first participant has not joined the real-time communication session for a predetermined period of time options. In some embodiments, the reminder option is displayed on or near the first participant affordance in the group contacts user interface. Displaying a reminder option associated with a participant when specified conditions are met allows the user to quickly identify users who have not joined the call and call those specific users. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Upon displaying the reminder option associated with the first participant, the device receives input corresponding to the selection of the reminder option. In response to receiving the input corresponding to the selection of the reminder option, the device sends a notification indicating that the real-time communication session is available for the first participant (eg, a notification that causes the first participant's device to output (eg, audible, visual, tactile) notification ). Sending a notification to the first participant in response to receiving an input corresponding to a selection of a reminder option associated with the participant reduces the amount of input required to perform the operation. Reducing the amount of input required to perform an action (e.g., the number of options a user needs to call a participant) enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user error when operating/interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 1342 and 1344, the device detects an input (eg, 1250E) corresponding to the selection of a first participant affordance (eg, 1232B) that represents the first part of the message conversation. associated with a participant. In some embodiments, these participants are participants in a real-time communication session (eg, participants in a live communication session may include one or more different users than users from the messaging conversation, and may also not include participation from the messaging conversation By). In response to detecting the input corresponding to the selection of the first participant affordance associated with the first participant of the message conversation, the device displays an individual participant user interface (eg, 1236) that includes characteristics of the first participant (eg, 1236) ( For example, an individual participant user interface has detailed information about an individual user (eg, user identifiers such as name, initials, etc.; user contact information; communication mode to contact the user)). Displaying an individual participant user interface in response to detecting an input (eg, participant affordances for each individual participant on the group contacts UI) and not rendering the UI due to additional displayed controls when additional controls are required (eg, , group contacts UI) clutter, which enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which This in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 1346 and 1348, the device detects an input (eg, an input with a characteristic intensity exceeding an intensity threshold) corresponding to the selection of an option for selecting a communication mode (eg, 1238A-1238C). In response to detecting the input corresponding to the selection of the communication mode option, the device initiates a process for selecting the reversing mode of the first participant. In some embodiments, initiating the process includes displaying a plurality of communication option affordances (eg, mail affordances, call affordances, instant message affordances).

Optionally, at blocks 1350 and 1352, the device detects an input corresponding to the selection of a new contact affordance (eg, 1234). In response to detecting the input corresponding to the selection of the new contact affordance, the device initiates a process for adding the new contact to the real-time communication session (eg, displaying a user interface for selecting the new contact to add). Initiating a process for adding a new contact to a live communication session allows the user to quickly identify which users to add to the call, provides feedback to the user on the current state of the real-time communication session, and provides the user with an indication that execution will be performed with notifications and/or real-time Visual feedback for certain actions associated with the communication session. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

Note that the details of the processes described above with respect to method 1300 (eg, FIGS. 13A-13D ) also apply in a similar manner to the methods described below/above. For example, method 1300 optionally includes one or more features of the various methods described with reference to methods 700 , 900 , 1100 or 1500 .

14A-14I illustrate exemplary user interfaces for initiating a real-time communication session, particularly from a real-time communication application, in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below including the processes in Figures 15A-15C.

14A shows a device 600 (as described above) displaying a user interface 1400 (eg, a home screen or springboard) with an affordance for launching an application. As shown in FIG. 14A , device 600 receives (eg, detects) user input 1450A (eg, taps) corresponding to selection of real-time communication application affordance 1402 .

As shown in Figure 14B, in response to receiving user input 1450A, device 600 launches the real-time video communication application and displays the real-time video communication user interface 1404 of the real-time video communication application. The real-time video communication user interface includes a list 1408 of participant affordances 1408A-1408K corresponding to respective participants or groups of participants. In some embodiments, based on previous (eg, recent) real-time communication with respective participants or groups of participants in the real-time communication application (eg, two or more participants other than the user of device 600 ) Communication sessions, participant affordances are included in list 1408. In some embodiments, previous real-time communication sessions include communications that were successfully connected and attempted by the user of device 600 or another participant. The real-time video communication user interface 1404 includes a new session affordance 1406 for selecting participants for the new real-time communication session.

In some embodiments, based on communication with another application (eg, a messaging application (eg, an affordance 1408A; eg, a list of group chat participants), a telephony application (eg, an affordance 1408C; eg, a participant in a conference call) , a group of participants associated with a calendar application (an affordance 1408B; eg, the attendees of a calendar event) or an email application (an affordance 1408D; eg, a list of recipients for an email), the participant affordance Representation is included in list 1408. For example, participant affordances 1408A correspond to participant groups in previous or existing group messaging conversations (eg, text or instant messages) and are displayed as suggestions in list 1408 (eg, the user does not have to manually create participant groups) . In some embodiments, the participant affordances correspond to a group of participants in a conference call associated with the telephony application. Participant affordances 1408A-1408K include an indication of the participant or group of participants, status indicators, detail affordances, and optionally related dates. As shown in Figure 14B, the status indicator identifies the source of the participant or group (eg, "from message", "from phone", "from calendar", "from mail") or the communication status of the associated communication (eg, video, audio, unavailable, etc.). In some embodiments, the source of a participant or group is indicated by a graphical or textual indication, a color associated with a particular application, or an application-related logo (eg, a phone icon, video camera icon, or message bubble icon).

In some embodiments, the device 600 visually distinguishes the participant affordances associated with the group in which there is an active real-time communication session. In Figure 14B, participant affordance 1408E is associated with the group for which the real-time communication session is currently active. The real-time communication session active in participant affordance 1408E is represented by participant name, video camera Icon 1408E-1 and bold text indication of the "Tap to Join" message.

In some embodiments, the presence of participant affordances associated with a previous real-time communication session is based on whether a user of the device has joined the corresponding real-time communication session. In FIG. 14B, participant affordance 1408H is associated with a previous real-time communication session in which the user of device 600 did not join. As a result, participant affordance 1408H includes italicized text, as compared to non-italicized text of other participant affordances associated with previous real-time communication sessions that the user of device 600 (eg, 1408F) joined.

As shown in FIG. 14C, device 600 receives (eg, detects) user input 1450B (eg, taps) corresponding to the selection of participant affordance 1408A of the participant group "Dream Team" of 15 participants described above. .

In response to receiving user input 1450B, device 600 initiates the process of starting a new real-time video communication session with the participant corresponding to the selected participant affordance 1408A ("dream team"). In some embodiments, a new real-time video communication session is initiated directly in response to the selection of the participant affordance 1408A. For example, as shown in Figure 14D, in response to receiving user input 1450B, device 600 displays real-time communication user interface 1410 and initiates a real-time communication session with the "dream team" group without further user input. The real-time communication user interface 1410 includes a representation of the user of the device 600 (eg, video from a camera on the front of the device 600) and a call control menu 1412, similar to the call control menu 630 discussed.

In some embodiments, a menu of the real-time video communication application is displayed in response to selection of the participant affordance 1408A, and a menu option is selected to start a new real-time video communication session. For example, in some embodiments, device 600 displays real-time communication user interface 1410 (as shown in FIG. 14D ) without initiating a real-time communication session. Conversely, while the real-time communication user interface 1410 is displayed, the device 600 receives (eg, detects) user input (eg, taps) corresponding to the selection of the call affordance 1414 on the call control menu 1412, and initiates in response to the real-time communication Selection of the affordance 1414 initiates a real-time communication session with the "Dream Team" group.

FIG. 14E also shows a real-time communication user interface 1404. Instead of receiving selections of participant affordances in list 1408 (as shown in FIG. 14C ), device 600 receives (eg, detects) user input 1450C (eg, taps) corresponding to selections of new session affordances 1406 . In response to user input 1450C, and in accordance with user input 1450C corresponding to selecting a new session affordance 1406, device 600 initiates a process for selecting participants and starting a new real-time video communication session between the selected participant and the user of device 600 the process of. As shown in FIG. 14F, the real-time communication user interface 1404 is replaced by a user interface 1416 that includes a text entry field 1418 for entering participant names using the keyboard 1420, for selecting participation from a menu (eg, a contact list) Add participant affordances 1422 , audio-only affordances 1424 , and video affordances 1426 of the participant.

In Figure 14F, in response to entering the text "JOE" into the text input field 1418, the device 600 displays two communication options 1417A and 1417B for the contact "JOE SMITH". In some embodiments, device 600 visually distinguishes communication options associated with devices capable of joining a real-time communication session. In Figure 14F, a communication mode (eg, email) represented by communication option 1417B is associated with a device capable of joining a real-time communication session, as shown in bold text. Instead, the communication mode (eg, home phone) represented by communication option 1417A is displayed in normal (non-bold) text to indicate that the communication mode is not associated with a device capable of joining the real-time communication session.

As shown in Figure 14G, participants named Joe, Ashley, and Sam were selected. Device 600 receives (eg, detects) user input 1450D (eg, taps) corresponding to the selection of video affordance 1426 .

In response to receiving user input 1450D, device 600 initiates a new real-time communication session with the selected participant and provides a real-time media stream including video and audio data (as shown in Figure 14H). In some embodiments, in response to receiving user input corresponding to the selection of audio-only affordances 1424, device 600 initiates a new real-time communication session with the selected participant and provides real-time media including audio data and no video data flow.

In some embodiments, after a new real-time communication session begins, device 600 displays a representation of the new real-time communication session in an application other than the real-time communication application. After initiating a real-time communication session with participants Joe, Ashley, and Sam, the user of device 600 navigates to an instant messaging application. As shown in Figure 14I, after starting the real-time communication session, the device 600 displays the instant messaging user interface 1428 of the instant messaging application. The instant messaging user interface 1428 displays the group messaging session including the participants of the real-time communication session, and displays a message 1430 in the message area 1432 indicating that the real-time communication session has started. In some embodiments, after starting the real-time communication session, the device 600 displays a group including the participants of the real-time communication session in the instant messaging application's message conversation list. In some embodiments, device 600 displays instant messaging user interface 1428 (described above) in response to selection of a group in the message conversation list.

15A-15C are flowcharts illustrating methods according to some embodiments. Method 1500 is performed at a device having a display (eg, 100, 300, 500, 600). Some operations in method 1500 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

As described below, method 1500 provides an intuitive way to initiate and interact with a real-time communication session between multiple participants. The method reduces the cognitive load on the user by providing intelligent initiation and interaction of real-time communication sessions among multiple participants, thereby creating a more efficient human-machine interface. For battery-operated computing devices, enabling users to interact and initiate real-time communication sessions more quickly and efficiently saves power and increases the interval between battery charges.

At block 1502, the device displays a real-time video communication user interface of the real-time video communication application (eg, 1404). At block 1504, the real-time video communication user interface includes a plurality of participant affordances (eg, 1408A-1408K) corresponding to the respective one or more participants, the plurality of participant affordances including corresponding to three A first participant affordance of a group of one or more participants (eg, 1408A). At block 1506, the real-time video communication user interface includes a new session affordance for starting a new real-time video communication session (eg, 1406). In some embodiments, the participant affordance corresponds to a previous (eg, recent) video communication (eg, successfully connected or attempted (by the participant using the device or by the participant associated with the external device)) and includes Indication to the participant of the video communication (eg, 1408F). In some embodiments, the participant affordance represents a group associated with another communication application (eg, an instant messaging application or a telephony application (eg, a recent conference call)). In some embodiments, the real-time communication user interface includes a list of recent real-time communication sessions (eg, the list of recent video chats may be individual chats and/or group chats). Displaying a real-time communication user interface including a list of recent real-time communication sessions allows the user to quickly access and enable the most recent real-time communication sessions in which the user or device has participated (e.g., invited, joined, communicated from, etc.), providing the user with information about the real-time communication session Feedback of the current status and provide visual feedback to the user indicating that certain actions associated with the notification and/or real-time communication session will be performed. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

In some embodiments, the plurality of participant affordances include a second participant affordance corresponding to a suggested participant group (eg, 1408B) (eg, a suggested participant group not manually selected/created by the user), the The proposed participant group is determined based on data (eg, usage data) from a second application (eg, telephony application, instant messaging application) different from the real-time video communication application.

In some embodiments, the suggested participant group corresponds to a participant (eg, 1408A) in a group conversation (eg, a group instant messaging session, a group conference call) conducted in the second application or in the second application Participants in a conference call (eg, 1408C). Displaying a real-time communication user interface that includes an affordance corresponding to a suggested group of participants allows the user to quickly access and initiate real-time communication with the suggested group, and reduce execution associated with establishing a real-time communication session that includes each member of the selected group The number of inputs required for the operation. Reducing the amount of input required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors in operating/interacting with the device), This in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. Furthermore, displaying an affordance corresponding to the group of suggestions provides additional control operations without cluttering the UI with additionally displayed controls. Providing additional control operations without cluttering the UI with additional displayed controls enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the amount of time when operating/interacting with the device) user error), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

In some embodiments, the second participant affordance includes an indication (eg, a graphical or textual indication (eg, "suggested by message"), colors associated with an application or a logo associated with that application). The display also includes a real-time communication user interface that includes an affordance corresponding to a suggested group of participants corresponding to participants in a group conversation in an application different from the real-time communication application, allowing the user to quickly access And initiate real-time communication with the suggested group and reduce the amount of input required to perform operations associated with establishing a real-time communication session that includes each member of the selected group (eg, such as navigating different applications). Reducing the amount of input required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors in operating/interacting with the device), This in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the plurality of participant affordances include a third participant affordance corresponding to a second suggested participant group (eg, a suggested participant group not manually selected/created by the user), the second suggested The participant base is based on data from a third application (eg, phone application, instant messaging application, email application, calendar application) that is distinct from the real-time video communication application and distinct from the second application (eg, Use data) to determine (eg, groups from recent calendar events or groups from recent mail events). Displaying a second participant affordance that includes an indication of the application from which the suggested group came from provides feedback to the user about the current state of the real-time communication session and provides visual feedback to the user indicating certain operations associated with the second application . Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently. In some embodiments, the real-time communication user interface further includes a participant selection affordance (eg, "+").

At block 1508, the device detects an input (eg, 1450B) corresponding to a request to initiate the process of starting a new real-time video communication session. At blocks 1510, 1512, and 1514, the device, in response to detecting the input: initiates a process for starting a new real-time video communication session according to the input corresponding to the selection of the first participant affordance (eg, 1408A), the new A real-time video communication session includes a group of three or more participants. In some embodiments, a new real-time video communication session is initiated directly in response to selection of the first participant affordance (eg, FIG. 14D ). In some embodiments, in response to selection of the first participant affordance, a menu of the real-time video communication application is displayed (eg, 1416), and a menu option is selected to start a new real-time video communication session. In response to detecting the input, the device initiates a process for selecting two or more participants and starting a new real-time video communication session based on the input corresponding to the selection of the new session affordance, the new real-time video communication The session includes participants associated with the device and two or more selected participants (eg, Figure 14F). In some embodiments, the user interface is replaced with another user interface that includes a first affordance for selecting a participant (eg, 1422 ) and for starting a real-time video communication session (eg, a response The selection of the second affordance directly begins the second affordance (eg, 1424 and 1426) of the real-time video communication session). In some embodiments, participants are selected by entering contact information into a text field (eg, 1418) or selecting a contact from a contact list. In some embodiments, a new real-time video communication session is started directly in response to selection of an affordance (eg, 1408A). Initiating a process for starting a new real-time communication when specified conditions are met allows the device to provide the user with different real-time video communication sessions based on the number of selected participants. Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. Furthermore, initiating a different process for starting a new real-time communication when specified conditions are met reduces the amount of input required to perform the operation. Reducing the amount of input required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors in operating/interacting with the device), This in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 1516, 1518, 1520, 1522, 1524, and 1526, the device receives a second input corresponding to the participant's selection of an affordance. In response to receiving the second user input corresponding to the participant's selection of the affordance, the device displays a group selection user interface (eg, 1416 ) that includes a text input field (eg, 1416 ) for identifying one or more participants. , 1418) (eg, to: field); keyboard (eg, 1420); audio-only affordances (eg, 1424); and video affordances (eg, 1426). While displaying the group selection user interface, the device receives a third user input selection. In response to receiving the third user input, the device starts a new session using audio transmitted from the electronic device (eg, audio recorded by one or more microphones of the device) according to the third user input corresponding to the selection of the audio-only affordance. real-time communication sessions without transmitting video data from electronic devices. When specified conditions are met (eg, in response to an audio-only affordance being selected), starting a new real-time communication session using audio transmitted from the electronic device allows the user to join the real-time communication session using audio without requiring the user to select or modify the device's Additional inputs and outputs (eg, camera off and/or microphone on). Performing optimization operations when a set of conditions have been satisfied without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently. In some embodiments, the system displays an indicator that describes the user joining the real-time communication using audio only. In some embodiments, the device prompts the user to display a phrase such as "Do you want to connect with video" in order to determine whether the user wants to use audio only or video and audio. This can make the user comfortable since the video is not shared). In response to receiving the third user input, according to the third user input corresponding to the selection of the video affordance, the device initiates a new real-time communication session using the audio and video transmitted from the electronic device. When specified conditions are met (eg, in response to the video affordance option being selected), starting a real-time communication session using audio and video transmitted from the electronic device allows the user to join the real-time communication session using audio and video without user selection or modification Additional inputs and outputs to the device (eg, turn on the camera and/or turn on the microphone). Performing optimization operations when a set of conditions have been met without further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing the need to manipulate the device/device) user error when interacting with the device), which in turn reduces power usage and increases the battery life of the device by enabling the user to use the device more quickly and efficiently.

Optionally, at blocks 1528 and 1530, the device starts a new real-time communication session. After starting a new real-time communication session, the device displays a representation of the new real-time communication session in a fourth application (eg, a messaging or telephony application) that is different from the real-time communication session. In some embodiments, a real-time communication participant is displayed as a group participant in a group message in a message conversation list in an instant messaging application. In some embodiments, the group message in the message conversation list includes an indication of whether there is an active real-time communication session that includes group participants (eg, 1408E). In some embodiments, the real-time communication participant is displayed as a group participant in a group call in the phone application or the recent calls list in the calling application. In some embodiments, real-time communication participants are displayed as invitees to calendar events in a calendar application. Displaying a representation of a real-time communication session in a different application than the real-time communication session provides feedback to the user about the current state of the real-time communication session in multiple applications, in some embodiments, if the user activates the representation only once (eg, , later start another new real-time communication session in the real-time communication application from another application with the same participants as the previous real-time communication session), then provide an indication to the user that the execution will be performed in relation to the notification and/or real-time communication session Visual feedback for certain actions linked. Providing the user with improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating/interacting with the device), which in turn Power usage is reduced and battery life of the device is extended by enabling the user to use the device more quickly and efficiently.

In some embodiments, the plurality of participant affordances (eg, 1408) include a fourth participant affordance (eg, 1408A, 1408E) corresponding to a group of three or more participants. In some embodiments, upon determining that the fourth participant affordance corresponds to the active real-time communication session (eg, 1408E), the fourth participant affordance is displayed in the first visual appearance (eg, the first visual appearance is participating the font color or font style of the participant affordance or an icon or glyph displayed at or near the participant affordance); and a fourth participant affordance is determined to correspond to an inactive real-time communication session (e.g., 1408A ), the fourth participant empowerment is displayed in a second visual appearance different from the first visual appearance (e.g., the second visual appearance is a different font color or font style than the font color or font style of the first visual appearance, or The second visual appearance does not include icons or glyphs displayed near the representation of the potential participant or includes icons or glyphs that differ from the first visual appearance). In some embodiments, the appearance of the participant affordance associated with the previous real-time communication session is based on whether the user of the device joined the corresponding real-time communication session (eg, the participant affordance of the missed real-time communication session (eg, 1408H) ) includes font colors, font styles, icons, glyphs, etc. that are different or not included in the participant affordances (eg, 1408F) of the real-time communication session that the user of the device 600 joined; if the user of the device joins the session, then Participant affordances include text in one color, and text in a different color if the user of the device did not join the session before it ended). In some embodiments, when the session is no longer active (eg, from bold text and a video camera icon to non-bold text without a video camera icon), participant affordances associated with an active real-time communication session appearance changes.

In some embodiments, initiating a process for selecting two or more participants includes displaying representations (eg, 1417A, 1417B) of potential participants (eg, contacts in a contact list). In some embodiments, upon determining that the potential participant is associated with a device capable of joining the real-time communication session, the representation of the potential participant is displayed in a first visual appearance (eg, the first visual appearance is the font color of the representation of the potential participant or font style or an icon or glyph displayed near the representation of the potential participant); and based on a determination that the participant is not associated with a device capable of joining the real-time communication session, the potential participation is displayed in a second visual appearance different from the first visual appearance representation of the participant (e.g., the second visual appearance is a different font color or font style than that of the first visual appearance, or the second visual appearance does not include an icon or glyph displayed near the representation of the potential participant or include icons or glyphs that differ from the first visual appearance).

Note that the details of the processes described above with reference to method 1500 (eg, FIGS. 15A-15C ) also apply in a similar manner to the methods described above. For example, method 1500 optionally includes one or more features of the various methods described above with reference to methods 700 , 900 , 1100 or 1300 .

For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the techniques and their practical applications. Others skilled in the art can thus best utilize these techniques and various embodiments with various modifications as are suited to the particular use contemplated.

Although the present disclosure and examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of the present disclosure and examples as defined by the appended claims.

As mentioned above, one aspect of the present technology resides in the collection and use of data from various sources to improve multimedia communications. This disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or may be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter accounts, home addresses, data or records related to the user's health or fitness level (eg, vital sign measurements, medication information, exercise information), date of birth or any other identifying or personal information.

This disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data may be used to facilitate multimedia communications between users. Therefore, the use of such personal information data enables users to improve such communications. In addition, this disclosure also contemplates other uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into a user's overall health, or can be used as positive feedback for individuals using technology to pursue health goals.

This disclosure envisages that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and/or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies are easily accessible to users and should be updated as data is collected and/or used. Users' personal information should be collected for the entity's lawful and fair use and should not be shared or sold outside of these lawful uses. Additionally, such capture/sharing should take place after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to safeguard and secure access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures. Additionally, such entities may subject themselves to third-party assessments to demonstrate compliance with generally accepted privacy policies and practices. In addition, policies and practices should be adjusted for the specific types of personal information data collected and/or accessed, and for applicable laws and standards including jurisdiction-specific considerations. For example, in the United States, the collection or access of certain health data may be governed by federal and/or state laws such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be governed by other regulations and policies, so should be dealt with accordingly. Therefore, different privacy practices for different types of personal data should be maintained in each country.

Regardless of the foregoing, the present disclosure also contemplates implementations in which users selectively block the use or access to personal information data. That is, the present disclosure contemplates that hardware elements and/or software elements may be provided to prevent or prevent access to such personal information data. For example, with respect to multimedia communications, the techniques of the present invention may be configured to allow a user to opt-in or opt-out of participation in the collection of personal information data during registration for a service or at any time thereafter. As another example, the user may choose not to provide location data and/or availability. In yet another example, a user may choose to limit the length of time a record of a communication session is maintained or to prohibit retention of such data entirely. In addition to providing "opt-in" and "opt-out" options, this disclosure contemplates providing notices related to access or use of personal information. For example, the user can notify the user when the application is downloaded that their personal information data will be accessed, and then remind the user again before the personal information data is accessed by the application.

Furthermore, the purpose of this disclosure is that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risk can be minimized by limiting data collection and deleting data once it is no longer needed. Additionally, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. Where appropriate, this can be achieved by removing specific identifiers (eg, date of birth, etc.), controlling the amount or characteristics of data stored (eg, collecting location data at the city level instead of the address level), controlling how the data is stored ( For example, aggregating data among users) and/or other methods to facilitate de-identification.

Thus, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, this disclosure also contemplates that the various embodiments may also be used without access to such personal information data. accomplish. That is, various implementations of the present technology will not fail due to lack of all or a portion of such personal information data. For example, a communication session may be established based on non-personal information data or absolute minimum personal information (eg, content requested by a device associated with the user, other non-personal information available to the communication service provider, or publicly available information).

Claims (30)

1. An electronic device comprising: monitor; one or more processors; and a memory that stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: Displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface comprising: a message area that includes a plurality of messages between participants in a message conversation that includes three or more participants; and an energy representation separate from the message area, the energy representation being used to initiate a real-time video communication session in a real-time video communication application; detecting an input corresponding to the selection of the affordance; and in response to detecting the input corresponding to the selection of the affordance, initiating a process for starting the real-time video communication session in the real-time video communication application, the real-time video communication session including the message the participant in a conversation, wherein initiating a process for starting the real-time video communication session in the real-time video communication application includes starting the real-time video communication session in the real-time video communication application; and After starting the live video communication session: In response to determining that a first participant has not joined the real-time video communication session within a predetermined period of time, displaying a reminder option associated with the first participant in the real-time video communication application: receiving an input corresponding to the selection of the reminder option when the reminder option associated with the first participant is displayed; and In response to receiving the input corresponding to the selection of the reminder option, a notification is sent indicating that the real-time video communication session is available for the first participant. 2. The electronic device of claim 1, wherein initiating the process for starting the real-time video communication session in the real-time communication application comprises: Transition from the instant messaging user interface to the user interface of the real-time video communication application. 3. The electronic device of claim 1, wherein initiating the process for starting the real-time video communication session in the real-time video communication application comprises displaying a group contact user interface, the group contact The user interface includes a group contact affordance for initiating the real-time video communication session, and wherein the one or more programs further include instructions for: Before starting the live video communication session: detecting an input corresponding to a selection of the group contact affordance for starting the real-time video communication session; and The real-time video communication session is initiated in the real-time video communication application in response to detecting the input corresponding to the selection of the group contact affordance for initiating the real-time video communication session. 4. The electronic device of claim 3, wherein the group contacts user interface further comprises: a plurality of participant affordances, the plurality of participant affordances comprising the participation with the message conversation The one or more programs further include instructions for performing the following operations: detecting an input corresponding to a selection of the first participant affordance associated with the first participant of the message conversation; and In response to detecting the input corresponding to the selection of the first participant affordance, a separate participant user interface is displayed that includes characteristics of the first participant, the first participant affordance being associated with the first participant associated with the first participant of the message conversation. 5. The electronic device of claim 4, wherein the first participant affordance includes an option for selecting a communication mode for contacting the first participant of the message conversation, the one or the Several programs also include instructions for doing the following: detecting an input corresponding to selection of the option for selecting the communication mode; and In response to detecting the input corresponding to the selection of the option for selecting the communication mode, a process for selecting the communication mode to contact the first participant is initiated. 6. The electronic device of claim 3, wherein the group contact user interface further comprises a new contact affordance, the one or more programs further comprising instructions for: detecting an input corresponding to the selection of the new contact affordance; and In response to detecting the input corresponding to the selection of the new contact affordance, a process for adding a new contact to the real-time video communication session is initiated. 7. The electronic device of claim 1, wherein initiating the process for starting the real-time video communication session comprises: After starting the real-time video communication session in the real-time video communication application, a real-time video communication session user interface is displayed, the real-time video communication session user interface includes: A first placeholder associated with a first participant of the real-time video communication session participants. 8. The electronic device of claim 7, wherein the first placeholder comprises a first reminder affordance for the first of the real-time video communication session participants, the one or more programs also include instructions for doing the following: receiving input corresponding to a selection that the first reminder can represent; and In response to receiving the input corresponding to the selection of the first reminder affordance, a new notification is sent to the first participant that is different from a previous notification sent to the first participant. 9. The electronic device of claim 7, wherein the real-time video communication session user interface further comprises a second placeholder associated with a second participant, wherein the first placeholder and the second The placeholders are arranged according to the first layout, and wherein the one or more programs further include instructions for: Upon determining that the first participant has not joined the real-time video communication session within a predetermined amount of time: stop displaying the first placeholder; and The arrangement of the second placeholders is changed to a second layout different from the first layout. 10. The electronic device of claim 7, the one or more programs further comprising instructions for: The display of the first placeholder is modified with the representation of the first participant based on a determination that the first participant has joined the real-time video communication session. 11. A non-transitory computer-readable storage medium storing one or more processors configured to be executed by one or more processors of an electronic device having a display and a touch-sensitive surface Programs, the one or more programs including instructions for: Displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface comprising: a message area that includes a plurality of messages between participants in a message conversation that includes three or more participants; and an energy representation separate from the message area, the energy representation being used to initiate a real-time video communication session in a real-time video communication application; detecting an input corresponding to the selection of the affordance; and in response to detecting the input corresponding to the selection of the affordance, initiating a process for starting the real-time video communication session in the real-time video communication application, the real-time video communication session including the message the participant in a conversation, wherein initiating a process for starting the real-time video communication session in the real-time video communication application includes starting the real-time video communication session in the real-time video communication application; and After starting the live video communication session: In response to determining that a first participant has not joined the real-time video communication session within a predetermined period of time, displaying a reminder option associated with the first participant in the real-time video communication application: receiving an input corresponding to the selection of the reminder option when the reminder option associated with the first participant is displayed; and In response to receiving the input corresponding to the selection of the reminder option, a notification is sent indicating that the real-time video communication session is available for the first participant. 12. The non-transitory computer-readable storage medium of claim 11, wherein initiating the process for starting the real-time video communication session in the real-time communication application comprises: Transition from the instant messaging user interface to the user interface of the real-time video communication application. 13. The non-transitory computer-readable storage medium of claim 11, wherein initiating the process for starting the real-time video communication session in the real-time video communication application comprises displaying a group contact user interface , the group contact user interface includes a group contact affordance for initiating the real-time video communication session, and wherein the one or more programs further include instructions for: Before starting the live video communication session: detecting an input corresponding to a selection of the group contact affordance for starting the real-time video communication session; and The real-time video communication session is initiated in the real-time video communication application in response to detecting the input corresponding to the selection of the group contact affordance for initiating the real-time video communication session. 14. The non-transitory computer-readable storage medium of claim 13, wherein the group contact user interface further comprises: a plurality of participant affordances, the plurality of participant affordances including a first participant affordance associated with a first one of the participants of the message conversation, the one or more programs further comprising instructions for: detecting an input corresponding to a selection of the first participant affordance associated with the first participant of the message conversation; and In response to detecting the input corresponding to the selection of the first participant affordance, a separate participant user interface is displayed that includes characteristics of the first participant, the first participant affordance being associated with the first participant associated with the first participant of the message conversation. 15. The non-transitory computer-readable storage medium of claim 14, wherein the first participant affordance includes a communication mode for selecting a communication mode for contacting the first participant of the messaging conversation option, the one or more programs further include instructions for: detecting an input corresponding to selection of the option for selecting the communication mode; and In response to detecting the input corresponding to the selection of the option for selecting the communication mode, a process for selecting the communication mode to contact the first participant is initiated. 16. The non-transitory computer-readable storage medium of claim 13, wherein the group contact user interface further comprises a new contact affordance, the one or more programs further comprising a instruction: detecting an input corresponding to the selection of the new contact affordance; and In response to detecting the input corresponding to the selection of the new contact affordance, a process for adding a new contact to the real-time video communication session is initiated. 17. The non-transitory computer-readable storage medium of claim 11, wherein initiating the process for initiating the real-time video communication session comprises: After starting the real-time video communication session in the real-time video communication application, a real-time video communication session user interface is displayed, the real-time video communication session user interface includes: A first placeholder associated with a first participant of the real-time video communication session participants. 18. The non-transitory computer-readable storage medium of claim 17, wherein the first placeholder includes a first reminder for the first participant of the real-time video communication session participants It can be represented that the one or more programs further include instructions for performing the following operations: receiving input corresponding to a selection that the first reminder can represent; and In response to receiving the input corresponding to the selection of the first reminder affordance, a new notification is sent to the first participant that is different from a previous notification sent to the first participant. 19. The non-transitory computer-readable storage medium of claim 17, wherein the real-time video communication session user interface further comprises a second placeholder associated with a second participant, wherein the first placeholder and the second placeholder are arranged according to the first layout, and wherein the one or more programs further include instructions for: Upon determining that the first participant has not joined the real-time video communication session within a predetermined amount of time: stop displaying the first placeholder; and The arrangement of the second placeholders is changed to a second layout different from the first layout. 20. The non-transitory computer-readable storage medium of claim 17, the one or more programs further comprising instructions for: The display of the first placeholder is modified with the representation of the first participant based on a determination that the first participant has joined the real-time video communication session. 21. A method comprising: At an electronic device with a display: Displaying an instant messaging user interface of an instant messaging application, the instant messaging user interface comprising: a message area that includes a plurality of messages between participants in a message conversation that includes three or more participants; and an energy representation separate from the message area, the energy representation being used to initiate a real-time video communication session in a real-time video communication application; detecting an input corresponding to the selection of the affordance; and in response to detecting the input corresponding to the selection of the affordance, initiating a process for starting the real-time video communication session in the real-time video communication application, the real-time video communication session including the message the participant in a conversation, wherein initiating a process for starting the real-time video communication session in the real-time video communication application includes starting the real-time video communication session in the real-time video communication application; and After starting the live video communication session: In response to determining that a first participant has not joined the real-time video communication session within a predetermined period of time, displaying a reminder option associated with the first participant in the real-time video communication application: receiving an input corresponding to the selection of the reminder option when the reminder option associated with the first participant is displayed; and In response to receiving the input corresponding to the selection of the reminder option, a notification is sent indicating that the real-time video communication session is available for the first participant. 22. The method of claim 21, wherein initiating the process for starting the real-time video communication session in the real-time video communication application comprises: Transition from the instant messaging user interface to the user interface of the real-time video communication application. 23. The method of claim 21, wherein initiating the process for starting the real-time video communication session in the real-time video communication application comprises displaying a group contact user interface, the group contact user The interface includes a group contact affordance for starting the real-time video communication session, and the method further includes: Before starting the live video communication session: detecting an input corresponding to a selection of the group contact affordance for starting the real-time video communication session; and The real-time video communication session is initiated in the real-time video communication application in response to detecting the input corresponding to the selection of the group contact affordance for initiating the real-time video communication session. 24. The method of claim 23, wherein the group contacts user interface further comprises: a plurality of participant affordances, the plurality of participant affordances comprising the participants in a conversation with the message A first participant affordance associated with a first participant in the method, the method further comprising: detecting an input corresponding to a selection of the first participant affordance associated with the first participant of the message conversation; and In response to detecting the input corresponding to the selection of the first participant affordance, a separate participant user interface is displayed that includes characteristics of the first participant, the first participant affordance being associated with the first participant associated with the first participant of the message conversation. 25. The method of claim 24, wherein the first participant affordance includes an option for selecting a communication mode for contacting the first participant of the messaging conversation, the method further comprising : detecting an input corresponding to selection of the option for selecting the communication mode; and In response to detecting the input corresponding to the selection of the option for selecting the communication mode, a process for contacting the communication mode of the first participant is initiated. 26. The method of claim 23, wherein the group contact user interface further comprises a new contact affordance, the method further comprising: detecting an input corresponding to the selection of the new contact affordance; and In response to detecting the input corresponding to the selection of the new contact affordance, a process for adding a new contact to the real-time video communication session is initiated. 27. The method of claim 21, wherein initiating the process for starting the real-time video communication session comprises: After starting the real-time video communication session in the real-time video communication application, a real-time video communication session user interface is displayed, and the real-time video communication session user interface includes: A first placeholder associated with a first participant of the real-time video communication session participants. 28. The method of claim 27, wherein the first placeholder includes a first reminder affordance for the first participant of the real-time video communication session participants, the method further include: receiving input corresponding to a selection that the first reminder can represent; and In response to receiving the input corresponding to the selection of the first reminder affordance, a new notification is sent to the first participant that is different from a previous notification sent to the first participant. 29. The method of claim 27, wherein the real-time video communication session user interface further comprises a second placeholder associated with a second participant, wherein the first placeholder and the second placeholder. The placeholders are arranged according to the first layout, and wherein the method further comprises: Upon determining that the first participant has not joined the real-time video communication session within a predetermined amount of time: stop displaying the first placeholder; and The arrangement of the second placeholders is changed to a second layout different from the first layout. 30. The method of claim 27, further comprising: The display of the first placeholder is modified with the representation of the first participant based on a determination that the first participant has joined the real-time video communication session.

Images