JP2023120280A - Multi-participant live communication user interface - Google Patents

Abstract

To provide a more quick and efficient method and interface for live communication in which a plurality of participants are involved.SOLUTION: A method includes: displaying a messaging user interface of a messaging application, the messaging user interface including a message region that includes a plurality of messages in a message conversation between three or more participants; receiving first data indicating that a live communication session that is available to the three or more participants is active; in response to receiving the first data, displaying a notification indicating that the live communication session is active; while displaying the notification, receiving second data indicating that the live communication session between the three or more participants is no longer active; and, in response to receiving the second data, ceasing displaying the notification indicating that the live communication session is active or updating information indicating that the live communication session is no longer active.SELECTED DRAWING: Figure 7A

Description

This application is related to U.S. Provisional Patent Application No. 62/668,229, filed May 7, 2018, entitled "MULTI-PARTICIPANT LIVE COMMUNICATION USER INTERFACE," which is hereby incorporated by reference in its entirety. .

TECHNICAL FIELD This disclosure relates generally to computer user interfaces, and more specifically to techniques for live communications involving multiple participants.

Users are increasingly using electronic devices to virtually communicate with others for personal and business reasons. Most electronic devices are capable of providing live communication between users and between multiple participants. By way of example, some electronic devices provide interactive interfaces for viewing and controlling live communications.

However, some techniques for live communication involving multiple participants using electronic devices are generally cumbersome and inefficient. For example, some existing technologies use complex and time-consuming user interfaces that may involve multiple key presses or keystrokes. Existing techniques take more time than necessary, wasting user time and device power. This latter problem is particularly acute in battery operated devices.

Accordingly, the present technology provides electronic devices with faster, more efficient methods and interfaces for live communications involving multiple participants. Such methods and interfaces optionally complement or replace other methods for live communication involving multiple participants. Such methods and interfaces reduce the cognitive burden on users and enable more efficient human-machine interfaces. For battery-operated computing devices, such methods and interfaces conserve power and extend battery charging intervals.

In some embodiments, the method is performed on an electronic device with a display. The method is displaying a messaging user interface of a messaging application, the messaging user interface including a message area containing a plurality of messages in a message conversation between the three or more participants; Receiving first data indicating that a live communication session available to participants is active; and receiving first data indicating that a live communication session between three or more participants is active. display a notification that a live communication session is active, and live communication between three or more participants while displaying a notification that a live communication session is active, depending on receiving second data indicating that the session is no longer active; and in response to receiving the second data indicating that the live communication session between the three or more participants is no longer active; Ceasing the display or updating the notification to indicate that the live communication session is no longer active.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. The one or more programs display a messaging user interface of a messaging application including a message area containing multiple messages in a message conversation between the three or more participants and live communication available to the three or more participants. receiving first data indicating that a session is active; in response to receiving first data indicating that a live communication session between three or more participants is active, the live communication session is active; and a second indicating that the live communication session between the three or more participants is no longer active, while displaying a notification indicating that the live communication session is active. Ceasing display of the notification or the live communication session is no longer active in response to receiving the data and receiving second data indicating that the live communication session between the three or more participants is no longer active. contains instructions for updating the notification to indicate that

In some embodiments, a temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. The one or more programs display a messaging user interface of a messaging application including a message area containing multiple messages in a message conversation between the three or more participants and live communication available to the three or more participants. Includes instructions for receiving first data indicating that the session is active. Displaying a notification indicating that a live communication session is active in response to receiving first data indicating that a live communication session between three or more participants is active, and the live communication session is active receiving second data indicating that the live communication session between the three or more participants is no longer active while displaying the notification indicating that the live communication session between the three or more participants is no longer active; responsive to receiving second data indicating that the communication session is no longer active, ceasing to display the notification or updating the notification to indicate that the live communication session is no longer active; can contain.

In some embodiments, an electronic device comprises a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by one or more processors. The one or more programs display a messaging user interface of a messaging application including a message area containing multiple messages in a message conversation between the three or more participants and live communication available to the three or more participants. Includes instructions for receiving first data indicating that the session is active. Displaying a notification indicating that a live communication session is active in response to receiving first data indicating that a live communication session between three or more participants is active, and the live communication session is active receiving second data indicating that the live communication session between the three or more participants is no longer active while displaying the notification indicating that the live communication session between the three or more participants is no longer active; responsive to receiving second data indicating that the communication session is no longer active, ceasing to display the notification or updating the notification to indicate that the live communication session is no longer active; can contain.

In some embodiments, the electronic device comprises a display and means for displaying a messaging user interface of a messaging application comprising a message area containing a plurality of messages in a message conversation between three or more participants; and means for receiving first data indicating that a live communication session is active between the three or more participants; means for, in response to receiving, displaying a notification indicating that a live communication session is active; means for receiving second data indicating that the communication session is no longer active; and notification in response to receiving the second data indicating that the live communication session between the three or more participants is no longer active. or for updating the notification to indicate that the live communication session is no longer active.

In some embodiments, the method is performed on an electronic device with a display. The method includes receiving a request to display a user interface for a live communication session between two or more participants; and displaying a user interface for the live communication session between the two or more participants. displaying a live communication user interface, including simultaneously displaying representations of the multiple participants in the live communication session, in response to receiving a request from the multiple participants in the live communication session; Displaying the representations includes displaying representations of the first plurality of participants other than the user of the electronic device in response to determining that the number of participants in the live communication session is greater than the threshold number of participants. Simultaneously displaying at one or more sizes greater than the threshold size while displaying representations of a second plurality of participants in the overflow region, including one or more participants not included in the first plurality of participants. , representations of one or more other participants displayed in the overflow area are displayed in sizes smaller than their respective threshold sizes, and the number of participants in the live communication session is less than or equal to the threshold number of participants. without displaying the representation of the participants other than the user of the electronic device at one or more sizes equal to or smaller than the respective threshold size, and displaying at one or more sizes greater than respective threshold sizes.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. One or more programs receive a request to display a user interface for a live communication session between two or more participants and display a user interface for a live communication session between the two or more participants. displaying a live communication user interface including simultaneously displaying representations of multiple participants in the live communication session in response to receiving a request for multiple participants in the live communication session; displaying representations of a first plurality of participants other than the user of the electronic device in response to determining that the number of participants in the live communication session is greater than the threshold number of participants; Simultaneously display at one or more sizes greater than the respective threshold size, while representing a second plurality of participants in the overflow region, including one or more participants not included in the first plurality of participants. representations of the one or more other participants displayed and displayed in the overflow region are displayed in sizes smaller than their respective threshold sizes; Representation of participants other than the user of the electronic device without displaying the representation of the participant other than the user of the electronic device at one or more sizes less than or equal to the respective threshold size following a determination that the at one or more sizes greater than the respective threshold size.

In some embodiments, a temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. One or more programs receive a request to display a user interface for a live communication session between two or more participants and display a user interface for a live communication session between the two or more participants. a plurality of participants in a live communication session, including instructions for displaying a live communication user interface including simultaneously displaying representations of the plurality of participants in the live communication session in response to receiving a request for displaying representations of a first plurality of participants other than the user of the electronic device upon determination that the number of participants in the live communication session is greater than the threshold number of participants, respectively while simultaneously displaying representations of a second plurality of participants in the overflow region, including one or more participants not included in the first plurality of participants. and representations of one or more other participants displayed in the overflow region are displayed in sizes smaller than their respective threshold sizes, and that the number of participants in the live communication session is equal to the threshold number of participants. Representation of participants other than the user of the electronic device without displaying the representation of the participant other than the user of the electronic device at one or more sizes less than or equal to the respective threshold size, following a determination that: , and displaying at one or more sizes greater than the respective threshold size.

In some embodiments, an electronic device comprises a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by one or more processors. One or more programs receive a request to display a user interface for a live communication session between two or more participants and display a user interface for a live communication session between the two or more participants. displaying a live communication user interface including simultaneously displaying representations of multiple participants in the live communication session in response to receiving a request for multiple participants in the live communication session; displaying representations of a first plurality of participants other than the user of the electronic device in response to determining that the number of participants in the live communication session is greater than the threshold number of participants; Simultaneously display at one or more sizes greater than the respective threshold size, while representing a second plurality of participants in the overflow region, including one or more participants not included in the first plurality of participants. representations of the one or more other participants displayed and displayed in the overflow region are displayed in sizes smaller than their respective threshold sizes; Representation of participants other than the user of the electronic device without displaying the representation of the participant other than the user of the electronic device at one or more sizes less than or equal to the respective threshold size following a determination that the at one or more sizes greater than the respective threshold size.

In some embodiments, the electronic device comprises a display and means for receiving a request to display a user interface for a live communication session between two or more participants; means for displaying a live communication user interface, including simultaneously displaying representations of multiple participants in the live communication session, in response to receiving a request to display the user interface for the communication session. , displaying representations of the plurality of participants in the live communication session is performed by a first person other than the user of the electronic device in response to a determination that the number of participants in the live communication session is greater than the threshold number of participants; Simultaneously displaying representations of the plurality of participants at one or more sizes greater than their respective threshold sizes, while displaying a second plurality of participants including one or more participants not included in the first plurality of participants. Participating in a live communication session, means for displaying a representation of a participant in an overflow area, wherein representations of one or more other participants displayed in the overflow area are displayed at sizes smaller than their respective threshold sizes; Upon determining that the number of participants is equal to or less than the threshold number of participants, the electronic device without displaying representations of participants other than the user of the electronic device at one or more sizes equal to or less than the respective threshold sizes. and means for displaying representations of participants other than the user of the event at one or more sizes greater than their respective threshold sizes.

In some embodiments, the method is performed on an electronic device with a display. A method simultaneously displays a representation of a first participant in a live video communication session, a representation of a second participant in the live video communication session, and a representation of a third participant in the live video communication session. displaying the communication user interface; and, while displaying the live video communication user interface, the activity level of the second participant in the live video communication session is equal to the activity level of the first participant in the live video communication session. and that the activity level of the second participant in the live video communication session increased above the activity level of the first participant in the live video communication session. changing the size and/or position of the representation of the first participant in the live video communications user interface and the size and/or the representation of the second participant in the live video communications user interface in response to receiving the indicating data; Or changing the position and changing the size and/or position of the representation of the third participant in the live video communication user interface.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. The one or more programs concurrently represent a first participant in the live video communication session, a second participant in the live video communication session, and a third participant in the live video communication session. displaying a live video communication user interface to display, wherein, while displaying the live video communication user interface, the activity level of the second participant in the live video communication session is equal to that of the first participant in the live video communication session; Receiving data indicating an increase above the activity level indicating an increase in the activity level of the second participant in the live video communication session above the activity level of the first participant in the live video communication session Responsive to receiving data, changing the size and/or position of the representation of the first participant in the live video communications user interface and changing the size and/or position of the representation of the second participant in the live video communications user interface. includes instructions for modifying and changing the size and/or position of the representation of the third participant in the live video communication user interface.

In some embodiments, a temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. The one or more programs concurrently represent a first participant in the live video communication session, a second participant in the live video communication session, and a third participant in the live video communication session. displaying a live video communication user interface to display, wherein, while displaying the live video communication user interface, the activity level of the second participant in the live video communication session is equal to that of the first participant in the live video communication session; receiving data indicating an increase above the activity level, wherein the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; changing the size and/or position of the representation of the first participant in the live video communications user interface and the size and/or position of the representation of the second participant in the live video communications user interface in response to receiving the data indicating the to change the size and/or position of the representation of the third participant in the live video communication user interface.

In some embodiments, an electronic device comprises a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by one or more processors. The one or more programs concurrently represent a first participant in the live video communication session, a second participant in the live video communication session, and a third participant in the live video communication session. displaying a live video communication user interface to display, wherein, while displaying the live video communication user interface, the activity level of the second participant in the live video communication session is equal to that of the first participant in the live video communication session; receiving data indicating an increase above the activity level, wherein the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; changing the size and/or position of the representation of the first participant in the live video communications user interface and the size and/or position of the representation of the second participant in the live video communications user interface in response to receiving the data indicating the to change the size and/or position of the representation of the third participant in the live video communication user interface.

In some embodiments, the electronic device includes a display, a representation of a first participant in the live video communication session, a representation of a second participant in the live video communication session, and a third participant in the live video communication session. means for displaying a live video communication user interface simultaneously displaying representations of the participants; means for receiving data indicating that an activity level of a first participant in the communication session has risen above an activity level of a second participant in the live video communication session; changing the size and/or position of the representation of the first participant in the live video communications user interface and the representation of the second participant in the live video communications user interface in response to receiving data indicating that the activity level of the participant has risen above the participant's activity level; and means for resizing and/or repositioning a representation of a third participant in the live video communication user interface.

In some embodiments, the method is performed on an electronic device with a display. The method initiates a live video communication session in a messaging user interface of a messaging application, a message area containing a plurality of messages between participants in a message conversation containing three or more participants, and a live video communication application. displaying a messaging user interface comprising: an affordance separate from the message region for; detecting input corresponding to the affordance selection; and, in response to detecting the input corresponding to the affordance selection, initiating a process of initiating a live video communication session in a live video communication application, the live video communication session including participants in a message conversation.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. The one or more programs are a messaging user interface of a messaging application, a message area containing multiple messages between participants in a message conversation containing three or more participants, and a live video communication session in a live video communication application. displaying a messaging user interface comprising: an affordance separate from the message region for initiating a live In a video communication application, instructions are included for initiating a process to initiate a live video communication session including participants in a message conversation.

In some embodiments, a temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. The one or more programs are a messaging user interface of a messaging application, a message area containing multiple messages between participants in a message conversation containing three or more participants, and a live video communication session in a live video communication application. displaying, displaying, detecting input corresponding to selecting the affordance, detecting input corresponding to selecting the affordance, and responding to detecting input corresponding to selecting the affordance; and instructions for initiating a process to initiate a live video communication session including message conversation participants in a live video communication application.

In some embodiments, an electronic device comprises a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by one or more processors. The one or more programs are a messaging user interface of a messaging application, a message area containing multiple messages between participants in a message conversation containing three or more participants, and a live video communication session in a live video communication application. displaying a messaging user interface comprising: an affordance separate from the message region for initiating a live In a video communication application, instructions are included for initiating a process to initiate a live video communication session including participants in a message conversation.

In some embodiments, the electronic device is a display and a means for displaying a messaging user interface of a messaging application, wherein the messaging user interface provides multiple communication between participants in a message conversation including three or more participants. means including a message area containing a message and an affordance separate from the message area for initiating a live video communication session in a live video communication application; means for detecting input corresponding to selection of the affordance; and means for, in response to detecting input corresponding to the selection of the affordance, initiating a process in the live video communication application to initiate a live video communication session including participants in the message conversation.

In some embodiments, the method is performed on an electronic device with a display. The method includes a user interface for a live video communication application, a plurality of participant affordances each corresponding to one or more participants, including a first participant affordance corresponding to a group of three or more participants. , a new session affordance for starting a new live video communication session; and detecting input corresponding to a request to initiate a process to start a new live video communication session. and, in response to detecting the input, initiating a process to initiate a new live video communication session including a group of three or more participants in response to the input corresponding to the selection of the first participant affordance. and in response to the input corresponding to the selection of the new session affordance, initiate a process of selecting two or more participants, the participant associated with the device and the two or more selected participants and initiating a new live video communication session.

In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. The one or more programs are a live video communications user interface of a live video communications application each corresponding to one or more participants including a first participant affordance corresponding to a group of three or more participants for displaying a live video communication user interface and initiating a process to start a new live video communication session, including a multiple participant affordance and a new session affordance for starting a new live video communication session. Detecting input corresponding to the request, and responsive to detecting the input, in response to the input corresponding to the selection of the first participant affordance, starting a new live video communication session including a group of three or more participants. Initiate a process to initiate and, in response to the input corresponding to the selection of a new session affordance, initiate a process to select two or more participants, the participant associated with the device and the two or more selected participants. including instructions for starting a new live video communication session with the participants.

In some embodiments, a temporary computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device with a display. The one or more programs are a live video communications user interface of a live video communications application each corresponding to one or more participants including a first participant affordance corresponding to a group of three or more participants for displaying a live video communication user interface and initiating a process to start a new live video communication session, including a multiple participant affordance and a new session affordance for starting a new live video communication session. Detecting input corresponding to the request, and responsive to detecting the input, in response to the input corresponding to the selection of the first participant affordance, starting a new live video communication session including a group of three or more participants. Initiate a process to initiate and, in response to the input corresponding to the selection of a new session affordance, initiate a process to select two or more participants, the participant associated with the device and the two or more selected participants. including instructions for starting a new live video communication session with the participants.

In some embodiments, an electronic device comprises a display, one or more processors, and memory. The memory stores one or more programs configured to be executed by one or more processors. The one or more programs are a live video communications user interface of a live video communications application each corresponding to one or more participants including a first participant affordance corresponding to a group of three or more participants for displaying a live video communication user interface and initiating a process to start a new live video communication session, including a multiple participant affordance and a new session affordance for starting a new live video communication session. Detecting input corresponding to the request, and responsive to detecting the input, in response to the input corresponding to the selection of the first participant affordance, starting a new live video communication session including a group of three or more participants. Initiate a process to initiate and, in response to the input corresponding to the selection of a new session affordance, initiate a process to select two or more participants, the participant associated with the device and the two or more selected participants. including instructions for starting a new live video communication session with the participants.

In some embodiments, the electronic device is a display and means for displaying a live video communications user interface of a live video communications application, the live video communications user interface corresponding to a group of three or more participants. a plurality of participant affordances each corresponding to one or more participants, including one participant affordance; and a new session affordance for starting a new live video communication session; means for detecting input corresponding to a request to initiate a process to initiate a video communication session; Initiate a process to start a new live video communication session that includes more than one group of participants, input corresponding to the selection of the new session affordance, to start a process to select two or more participants, and to the device: means for initiating a new live video communication session including the relevant participant and the two or more selected participants.

Executable instructions for performing these functions are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. Executable instructions for performing these functions are optionally contained on a temporary computer-readable storage medium or other computer program product configured to be executed by one or more processors.

As such, electronic devices can provide faster, more efficient methods and interfaces for live communications involving multiple participants, thereby increasing the effectiveness, efficiency, and user friendliness of such devices. Increased satisfaction. Such methods and interfaces can complement or replace other methods for live communication involving multiple participants.

For a better understanding of the various described embodiments, reference should be made to the following detailed description in conjunction with the following drawings. Here, like reference numerals refer to corresponding parts throughout all of the figures.

FIG. 3 is a block diagram illustrating a portable multifunction device with touch sensitive display according to some embodiments; FIG. 4 is a block diagram illustrating exemplary components for event processing, according to some embodiments; FIG. 10 illustrates a portable multifunction device having a touch screen according to some embodiments; 1 is a block diagram of an exemplary multifunction device with display and touch-sensitive surface, according to some embodiments; FIG. FIG. 10 illustrates an exemplary user interface for a menu of applications on the portable multifunction device, according to some embodiments; FIG. 10 illustrates an exemplary user interface for a multifunction device with a touch-sensitive surface separate from the display, according to some embodiments; FIG. 2 illustrates a personal electronic device, according to some embodiments; 1 is a block diagram illustrating a personal electronic device, according to some embodiments; FIG. FIG. 10 illustrates exemplary components of a personal electronic device having a touch-sensitive display and an intensity sensor, according to some embodiments; FIG. 10 illustrates exemplary components of a personal electronic device having a touch-sensitive display and an intensity sensor, according to some embodiments; FIG. 4 illustrates exemplary components and user interface of a personal electronic device, according to some embodiments; FIG. 4 illustrates exemplary components and user interface of a personal electronic device, according to some embodiments; FIG. 4 illustrates exemplary components and user interface of a personal electronic device, according to some embodiments; FIG. 4 illustrates exemplary components and user interface of a personal electronic device, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 4 illustrates an exemplary user interface, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments; FIG. 3 illustrates an exemplary method, according to some embodiments;

The following description describes exemplary methods, parameters, and the like. It should be understood, however, that the purpose of such description is not to limit the scope of the disclosure, but to provide descriptions of exemplary embodiments.

There is a need for electronic devices that provide efficient methods and interfaces for live communication involving multiple participants. Such techniques can reduce the cognitive burden of users participating in live communications involving multiple participants, which can increase productivity. In addition, such techniques can reduce processor and battery power that would otherwise be wasted due to redundant user input.

Figures 1A-1B, 2, 3, 4A-4B, and 5A-5H below provide descriptions of exemplary devices for conducting live communication session techniques between multiple participants. . 6A-6AF illustrate exemplary user interfaces for managing event notifications. 7A-7F are flow diagrams illustrating methods of live communication sessions between multiple participants, according to some embodiments. The user interfaces of Figures 6A-6AF are used to illustrate the processes described below, including the processes of Figures 7A-7F.

8A-8BK illustrate exemplary user interfaces for a live communication session between multiple participants. 9A-9K are flow diagrams illustrating methods of conducting live communication sessions between multiple participants, according to some embodiments. The user interfaces of Figures 8A-8BK are used to illustrate the processes described below, including the processes of Figures 9A-9K.

10A-10S illustrate exemplary user interfaces for a live communication session between multiple participants. 11A-11F are flow diagrams illustrating methods of conducting live communication sessions between multiple participants, according to some embodiments. The user interfaces of Figures 10A-10S are used to illustrate the processes described below, including the processes of Figures 11A-11F.

12A-12N illustrate exemplary user interfaces for a live communication session between multiple participants. Figures 13A-13D are flow diagrams illustrating methods of conducting a live communication session between multiple participants, according to some embodiments. The user interfaces of Figures 12A-12N are used to illustrate the processes described below, including the processes of Figures 13A-13D.

14A-14I illustrate exemplary user interfaces for a live communication session between multiple participants. 15A-15C are flow diagrams illustrating methods of conducting live communication sessions between multiple participants, according to some embodiments. The user interfaces of FIGS. 14A-14I are used to illustrate the processes described below, including the processes of FIGS. 15A-15C.

Although the following description uses the terms "first", "second", etc. to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first touch could be termed a second touch, and, similarly, a second touch could be termed a first touch, without departing from the scope of various described embodiments. . The first touch and the second touch are both touches, but not the same touch.

The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a," "an," and "the" do not clearly indicate to the contrary the context To the extent possible, it is also intended to include plural forms. As used herein, the term "and/or" refers to and includes any and all possible combinations of one or more of the associated listed items. It should also be understood. The terms "includes," "including," "comprises," and/or "comprising," as used herein, refer to the features, integers, steps, or , acts, elements and/or components, but does not exclude the presence or addition of one or more other features, integers, steps, acts, elements, components and/or groups thereof will be further understood.

Optionally, the term "if" is changed to "when", "upon", "in response to determining" depending on the context. )”, or “in response to detecting”. Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" are optional , depending on the context, "upon determining", "in response to determining", "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]".

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device such as a mobile phone that also includes PDA functionality and/or other functionality such as music player functionality. Exemplary embodiments of portable multifunction devices include Apple Inc. of Cupertino, Calif.; iPhone(R), iPod Touch(R) and iPad(R) devices. Other portable electronic devices such as laptop or tablet computers with touch-sensitive surfaces (eg, touch screen displays and/or touch pads) are optionally used. It should also be appreciated that in some embodiments, the device is a desktop computer with a touch-sensitive surface (eg, touchscreen display and/or touchpad) rather than a portable communication device.

In the discussion that follows, electronic devices that include a display and a touch-sensitive surface are described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices such as physical keyboards, mice and/or joysticks.

The device is commonly used in drawing applications, presentation applications, word processing applications, website creation applications, disc authoring applications, spreadsheet applications, gaming applications, telephony applications, video conferencing applications, e-mail applications, instant messaging applications, training It supports various applications such as one or more of a support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.

Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and/or changed from application to application and/or within each application. In this way, the common physical architecture (such as the touch-sensitive surface) of the device optionally supports a variety of applications with user interfaces that are intuitive and transparent to the user.

Attention is now directed to embodiments of portable devices with touch-sensitive displays. FIG. 1A is a block diagram illustrating portable multifunction device 100 with touch-sensitive display system 112, according to some embodiments. The touch-sensitive display 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes known or referred to as a "touch-sensitive display system." Device 100 includes memory 102 (optionally including one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110, , speaker 111 , microphone 113 , input/output (I/O) subsystem 106 , other input control devices 116 and external ports 124 . Device 100 optionally includes one or more photosensors 164 . Device 100 optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on device 100 (eg, a touch-sensitive surface such as touch-sensitive display system 112 of device 100). device 100 to generate tactile output on device 100 (e.g., on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300); One or more tactile output generators 167 are optionally included. These components optionally communicate via one or more communication buses or signal lines 103 .

As used herein and in the claims, the term "strength" of contact on the touch-sensitive surface refers to the force or pressure (force per unit area) of contact (e.g., finger contact) on the touch-sensitive surface. ), or alternatives to the force or pressure of contact on the touch-sensitive surface. The intensity of contact has a value range that includes at least four different values, and more typically includes hundreds (eg, at least 256) different values. The strength of contact is optionally determined (or measured) using various techniques and various sensors or combinations of sensors. For example, one or more force sensors underlying or adjacent to the touch-sensitive surface are optionally used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (eg, weighted average) to determine a contact force estimate. Similarly, the stylus' pressure-sensitive tip is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and/or change in the contact area detected on the touch-sensitive surface, the capacitance and/or change in the touch-sensitive surface proximate to the contact, and/or the resistance of the touch-sensitive surface proximate to the contact. and/or variations thereof are optionally used as a proxy for contact force or pressure on the touch-sensitive surface. In some implementations, the alternative measure for contact force or pressure is used directly to determine if an intensity threshold has been exceeded (e.g., the intensity threshold is in units corresponding to the alternative measure). explained). In some implementations, the alternative measurement for contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine if the intensity threshold is exceeded. (eg, the intensity threshold is the pressure threshold measured in units of pressure). for displaying affordances (e.g., on touch-sensitive displays) and/or for receiving user input (e.g., touch-sensitive displays, touch-sensitive surfaces) by using the strength of contact as an attribute of user input. , or via physical/mechanical controls such as knobs or buttons) that may not otherwise be accessible by the user on the reduced-size device. The function is made accessible to the user.

As used herein and in the claims, the term "tactile output" refers to the physical displacement of the device relative to its previous position, the configuration of the device, to be detected by the user in the user's sense of touch. Refers to the physical displacement of an element (eg, touch-sensitive surface) relative to another component of a device (eg, housing), or the displacement of a component relative to the device's center of mass. Tactile output generated by physical displacement, for example, in situations where the device or device component is in contact with a touch-sensitive surface of the user (e.g., the user's finger, palm, or other part of the hand) is interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or device component. For example, movement of a touch-sensitive surface (eg, a touch-sensitive display or trackpad) is optionally interpreted by the user as a "down-click" or "up-click" of a physical actuator button. In some cases, the user may "down-click" or "click" even when there is no physical actuator button movement associated with the touch-sensitive surface physically pressed (e.g., displaced) by the user's motion. You will feel a tactile sensation such as 'click up'. As another example, movement of the touch-sensitive surface is optionally interpreted or perceived by the user as "roughness" of the touch-sensitive surface, even if there is no change in the smoothness of the touch-sensitive surface. be. Such a user's interpretation of touch depends on the user's personal sensory perception, but there are many sensory perceptions of touch that are common to the majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "up-click", "down-click", "roughness"), the generated Tactile output corresponds to physical displacement of the device, or components of the device, that produce the described sensory perception of a typical (or average) user.

Device 100 is only one example of a portable multifunction device and device 100 optionally has more or fewer components than those shown, optionally two or more components. It should be understood that any combination of elements may optionally be combined or have different configurations or arrangements of components. The various components shown in FIG. 1A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and/or application specific integrated circuits.

Memory 102 optionally includes high speed random access memory and optionally also non-volatile memory such as one or more magnetic disk storage devices, flash memory devices or other non-volatile solid state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100 .

Peripheral interface 118 may be used to couple input and output peripherals of the device to CPU 120 and memory 102 . One or more processors 120 operate or execute various software programs and/or instruction sets stored in memory 102 to perform various functions for device 100 and to process data. . In some embodiments, peripheral interface 118 , CPU 120 and memory controller 122 are optionally implemented on a single chip, such as chip 104 . In some other embodiments, they are optionally implemented on separate chips.

RF (radio frequency) circuitry 108 transmits and receives RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to electromagnetic signals or vice versa to communicate with communication networks and other communication devices via electromagnetic signals. RF circuitry 108 optionally includes well-known circuits for performing their functions, including an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC. Chipsets, Participant Identity Module (SIM) cards, memory, etc. include, but are not limited to. The RF circuit 108 is optionally connected to the Internet, also referred to as the World Wide Web (WWW), intranets, and/or wireless networks such as cellular telephone networks, wireless local area networks (LAN) and/or metropolitan area networks (MAN). such as networks, and other devices via wireless communication. RF circuitry 108 optionally includes well-known circuitry for detecting a near field communication (NFC) field, such as by a near field communication radio. Wireless communication optionally uses any of a plurality of communication standards, protocols and technologies, including the Global System for Mobile Communications ( Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSDPA) speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (Dual-Cell HSPA, DC-HSPDA), Long term evolution (LTE), Near near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access , TDMA), Bluetooth®, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi)® (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n , and/or IEEE 802.11ac), voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or not yet developed as of the filing date of this document. Any other suitable communication protocol including, but not limited to, non-compliant communication protocols.

Audio circuitry 110 , speaker 111 , and microphone 113 provide an audio interface between the user and device 100 . Audio circuitry 110 receives audio data from peripherals interface 118 , converts the audio data to electrical signals, and transmits the electrical signals to speaker 111 . The speaker 111 converts the electrical signal into human audible sound waves. Audio circuitry 110 also receives electrical signals converted from sound waves by microphone 113 . Audio circuitry 110 converts the electrical signals to audio data and transmits the audio data to peripherals interface 118 for processing. Audio data is optionally retrieved from and/or transmitted to memory 102 and/or RF circuitry 108 by peripherals interface 118 . In some embodiments, audio circuitry 110 further comprises a headset jack (eg, 212 in FIG. 2). The headset jack connects audio circuitry 110 and detachable audio input/output peripherals, such as output-only headphones or headsets that have both an output (e.g., monaural or binaural headphones) and an input (e.g., a microphone). provides an interface between

I/O subsystem 106 couples input/output peripherals of device 100 , such as touch screen 112 and other input control devices 116 , to peripherals interface 118 . I/O subsystem 106 optionally includes one or more for display controller 156, light sensor controller 158, depth camera controller 169, intensity sensor controller 159, haptic feedback controller 161, and other input or control devices. of input controllers 160 . One or more input controllers 160 receive/send electrical signals from/to other input control devices 116 . Other input control devices 116 optionally include physical buttons (eg, push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and the like. In some alternative embodiments, input controller(s) 160 is optionally coupled to any of a keyboard, an infrared port, a USB port, and a pointer device such as a mouse (or not bound to either). One or more buttons (eg, 208 in FIG. 2) optionally include up/down buttons for volume control of speaker 111 and/or microphone 113 . The one or more buttons optionally include push buttons (eg, 206 in FIG. 2).

U.S. patent application Ser. No. 11/322,549, "Unlocking a Device by Performing Gestures on an Unlock Image," filed Dec. 23, 2005, U.S. Pat. No. 7,657,849; , which is incorporated herein by reference in its entirety), optionally a quick press of the push button unlocks the touch screen 112 or touches to unlock the device. Processing is optionally initiated using on-screen gestures. A longer press on the push button (eg, 206) optionally powers the device 100 on and off. The functionality of one or more of the plurality of buttons is optionally user customizable. Touch screen 112 is used to implement virtual buttons or soft buttons and one or more soft keyboards.

Touch-sensitive display 112 provides an input and output interface between the device and the user. Display controller 156 receives and/or sends electrical signals to touchscreen 112 . Touch screen 112 displays visual output to the user. Visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, optionally some or all of the visual output corresponds to user interface objects.

Touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on tactile and/or tactile contact. Touchscreen 112 and display controller 156 (together with any associated modules and/or instruction sets in memory 102) detect contact (and any movement or cessation of contact) on touchscreen 112 and into interactions with user interface objects (eg, one or more softkeys, icons, web pages, or images) displayed on touch screen 112 . In an exemplary embodiment, the contact point between touch screen 112 and the user corresponds to the user's finger.

The touch screen 112 optionally uses LCD (Liquid Crystal Display) technology, LPD (Light Emitting Polymer Display) technology or LED (Light Emitting Diode) technology, although in other embodiments other display technologies are used. Touchscreen 112 and display controller 156 may use capacitive, resistive, infrared, and surface ultrasonic technologies, as well as other proximity sensor arrays for determining one or more points of contact with touchscreen 112. or other elements, optionally detecting contact and any movement or cessation thereof using any of a number of now known or later developed touch sensing technologies. to detect. In an exemplary embodiment, Apple Inc. of Cupertino, Calif. Projected mutual-capacitance sensing technology is used, as seen in the iPhone(R) and iPod Touch(R) by Co., Ltd.

The touch-sensitive display in some embodiments of the touch screen 112 optionally includes US Pat. No. 6,677,932 (Westerman) and/or similar to the multi-touch sensing touchpads described in US Patent Application Publication No. 2002/0015024A1. These documents are incorporated herein by reference in their entireties. However, the touchscreen 112 displays visual output from the device 100, whereas the touch-sensitive touchpad does not provide visual output.

Touch sensitive displays in some embodiments of touch screen 112 are described in the following applications. (1) U.S. Patent Application Serial No. 11/381,313, filed May 2, 2006, entitled "Multipoint Touch Surface Controller"; Multipoint Touchscreen," (3) 10/903,964, filed Jul. 30, 2004; "Gestures For Touch Sensitive Input Devices," (4) 11/048, filed Jan. 31, 2005, 264, "Gestures For Touch Sensitive Input Devices," (5) Serial No. 11/038,590, filed Jan. 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input De vices”, (6) 2005 11/228,758, filed September 16, 2005, "Virtual Input Device Placement On A Touch Screen User Interface," (7) 11/228,700, filed September 16, 2005, " Operation Of A Computer With A Touch Screen Interface," (8) Serial No. 11/228,737, filed September 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard." , and (9) 2006 Serial No. 11/367,749, filed March 3, "Multi-Functional Hand-Held Device." All of these applications are incorporated herein by reference in their entirety.

Touch screen 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. A user optionally contacts touchscreen 112 using any suitable object or accessory, such as a stylus, finger, or the like. In some embodiments, user interfaces are designed to work primarily with finger-based contacts and gestures, and the large finger contact area on touchscreens can be less accurate than stylus-based input. be. In some embodiments, the device converts coarse finger-based input into precise pointer/cursor positions or commands to perform the user's desired action.

In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad for activating or deactivating certain functions. In some embodiments, a touchpad is a touch-sensitive area of a device that displays no visual output, unlike a touchscreen. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112 or an extension of the touch-sensitive surface formed by the touchscreen.

Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power failure detection circuitry, a power converter or inverter, power status indicators (e.g., light emitting diodes). (LEDs)), and any other components associated with the generation, management, and distribution of power within a portable device.

Device 100 also optionally includes one or more photosensors 164 . FIG. 1A shows a photosensor coupled to photosensor controller 158 in I/O subsystem 106 . Photosensor 164 optionally includes a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) phototransistor. Optical sensor 164 receives light from the environment projected through one or more lenses and converts the light into data representing an image. In conjunction with imaging module 143 (also referred to as a camera module), optical sensor 164 optionally captures still images or video. In some embodiments, the light sensor is opposite the touchscreen display 112 on the front of the device to enable the touchscreen display as a viewfinder for still and/or video image capture. , is located on the back side of the device 100 . In some embodiments, a light sensor is placed on the front of the device so that the user can optionally obtain an image of the other videoconference participant for videoconferencing while viewing the other videoconference participant on the touch screen display. be. In some embodiments, the location of optical sensor 164 is such that a single optical sensor 164 is used with a touch screen display for both videoconferencing and still and/or video image capture. , can be changed by the user (eg, by rotating the lens and sensor within the device housing).

Device 100 also optionally includes one or more depth camera sensors 175 . FIG. 1A shows a depth camera sensor coupled to depth camera controller 169 within I/O subsystem 106 . Depth camera sensor 175 receives data from the environment to create a three-dimensional model of an object (eg, face) in the scene from a viewpoint (eg, depth camera sensor). In some embodiments, in conjunction with imaging module 143 (also referred to as a camera module), depth camera sensor 175 optionally determines depth maps of different portions of the image captured by imaging module 143. used for In some embodiments, the user can optionally obtain an image of the user with depth information for videoconferencing while viewing other videoconference participants on a touch screen display, and depth map data. A depth camera sensor is placed on the front of the device 100 to capture an accompanying selfie image. In some embodiments, depth camera sensor 175 is placed on the back of the device, or on the back and front of device 100 . In some embodiments, depth camera sensors 175 are configured such that a single depth camera sensor 175 is used with a touch screen display for both videoconferencing and still and/or video image acquisition. The position can be changed by the user (eg, by rotating lenses and sensors within the device housing).

In some embodiments, a depth map (eg, depth map image) includes information (eg, values) regarding the distance of objects in a scene from a viewpoint (eg, camera, light sensor, depth camera sensor). In one embodiment of the depth map, each depth pixel defines where the corresponding two-dimensional pixel is located in the Z-axis of the viewpoint. In some embodiments, the depth map is made up of pixels where each pixel is defined by a value (eg, 0-255). For example, a '0' value represents the farthest located pixel in a '3D' scene, and a '255' value represents a viewpoint in a '3D' scene (e.g., camera, light sensor, depth camera sensor). In other embodiments, the depth map represents the distance between objects in the scene and the plane of the viewpoint. In some embodiments, the depth map contains information about the relative depths of various features of the object of interest in the view from the depth camera (e.g., the relative depths of the eyes, nose, mouth, ears on the user's face). . In some embodiments, the depth map includes information that allows the device to determine the z-direction contour of the object of interest.

Device 100 also optionally includes one or more contact intensity sensors 165 . FIG. 1A shows a contact intensity sensor coupled to intensity sensor controller 159 within I/O subsystem 106 . Contact intensity sensor 165 may include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., on the touch-sensitive surface). optionally a sensor used to measure the force (or pressure) of the contact. Contact intensity sensor 165 receives contact intensity information (eg, pressure information or a proxy for pressure information) from the surrounding environment. In some embodiments, at least one contact intensity sensor is located on or in close proximity to the touch-sensitive surface (eg, touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back of device 100 as opposed to touch screen display 112 located on the front of device 100 .

Device 100 also optionally includes one or more proximity sensors 166 . FIG. 1A shows proximity sensor 166 coupled with peripherals interface 118 . Alternatively, proximity sensor 166 is optionally coupled to input controller 160 within I/O subsystem 106 . Proximity sensor 166 is described in U.S. patent application Ser. No. 11/620,702, "Using Ambient Light Sensor To Augment Proximity Sensor Output," No. 11/586,862, "Automated Response To And Sensing Of User Activity In Portable Devices", and ibid. 11/638 , 251, "Methods And Systems For Automatic Configuration Of Peripherals". In some embodiments, the proximity sensor turns off and disables the touchscreen 112 when the multifunction device is placed near the user's ear (eg, when the user is talking).

Device 100 also optionally includes one or more tactile output generators 167 . FIG. 1A shows a tactile output generator coupled to haptic feedback controller 161 within I/O subsystem 106 . Tactile output generator 167 may be one or more electroacoustic devices such as speakers or other audio components and/or motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators or other tactile output generating configurations. Optionally includes an electromechanical device that converts energy into linear motion, such as an element (eg, a component that converts an electrical signal to a tactile output on the device). Contact intensity sensor 165 receives tactile feedback generation instructions from tactile feedback module 133 and generates a tactile output on device 100 that can be sensed by a user of device 100 . In some embodiments, at least one tactile output generator is located on or proximate to a touch-sensitive surface (eg, touch-sensitive display system 112) and optionally , vertically (eg, into/out of the surface of device 100) or laterally (eg, back and forth in the same plane as the surface of device 100) to generate tactile output. In some embodiments, at least one tactile output generator sensor is located on the back side of device 100 , opposite touch screen display 112 located on the front side of device 100 .

Device 100 also optionally includes one or more accelerometers 168 . FIG. 1A shows accelerometer 168 coupled to peripherals interface 118 . Alternatively, accelerometer 168 is optionally coupled to input controller 160 within I/O subsystem 106 . Accelerometer 168 is optionally configured according to U.S. Patent Application Publication Nos. 20050190059, "Acceleration-based Theft Detection System for Portable Electronic Devices," and 20060017692, " Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer". In some embodiments, information is displayed on the touch screen display in portrait or landscape orientation based on analysis of data received from one or more accelerometers. In addition to accelerometer(s) 168, device 100 includes a magnetometer and a GPS (or GLONASS or other global navigation system) for obtaining information about the position and orientation of device 100 (e.g., portrait or landscape orientation). system) optionally includes a receiver.

In some embodiments, the software components stored in memory 102 include operating system 126, communication module (or instruction set) 128, touch/motion module (or instruction set) 130, graphics module (or instruction set) 132, , a text input module (or instruction set) 134 , a Global Positioning System (GPS) module (or instruction set) 135 and an application (or instruction set) 136 . Additionally, in some embodiments, memory 102 (FIG. 1A) or memory 370 (FIG. 3) stores device/global internal state 157, as shown in FIGS. 1A and 3 . The device/global internal state 157 indicates which, if any, applications are currently active; occupancy; sensor states, including information obtained from the device's various sensors and input control device 116; and location information regarding the location and/or orientation of the device.

operating system 126 (e.g., Darwin®, RTXC®, LINUX®, UNIX®, OS X®, iOS®, WINDOWS®, or Embedded operating systems such as VxWorks®) contain various software components and/or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.). , facilitates communication between various hardware and software components.

Communication module 128 also includes various software components for facilitating communication with other devices via one or more external ports 124 and for processing data received by RF circuitry 108 and/or external ports 124 . . External port 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) connects to other devices either directly or indirectly through a network (e.g., Internet, wireless LAN, etc.). adapted to join. In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector the same as or similar to the 30-pin connector used on the iPod® (trademark of Apple Inc.) device, and/or Or a compatible multi-pin (eg, 30-pin) connector.

Contact/motion module 130 (in conjunction with display controller 156) optionally detects contact with touchscreen 112 and other touch-sensitive devices (eg, touchpads or physical click wheels). The contact/motion module 130 determines whether contact occurs (eg, detecting a finger-down event), the strength of the contact (eg, force or pressure of contact, or alternatives to force or pressure of contact). determining whether there is movement of the contact, tracking movement across the touch-sensitive surface (e.g., detecting one or more events of dragging a finger), and determining whether the contact ceases. It includes various software components for performing various operations related to contact detection, such as detecting a finger-up event or cessation of contact, for example. Contact/motion module 130 receives contact data from the touch-sensitive surface. Determining movement of the contact point represented by the set of contact data optionally includes speed (magnitude), velocity (magnitude and direction), and/or acceleration (magnitude and/or or change in direction). These operations are optionally applied to single contacts (eg, single-finger contacts) or multiple simultaneous contacts (eg, “multi-touch”/multiple-finger contacts). In some embodiments, contact/motion module 130 and display controller 156 detect contact on the touchpad.

In some embodiments, contact/motion module 130 includes one or more motion indicators for determining if an action has been taken by the user (e.g., for determining if the user "clicked" on an icon). Use a set of intensity thresholds. In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by a particular physical actuator activation threshold, but are determined by the physical hardware of device 100). can be adjusted without changing ). For example, the mouse "click" threshold of a trackpad or touchscreen display can be set to any of a wide range of predetermined thresholds without modifying the hardware of the trackpad or touchscreen display. Additionally, in some implementations, the user of the device is provided with software settings for adjusting one or more of the sets of intensity thresholds (e.g., by adjusting individual intensity thresholds, and/or by adjusting multiple intensity thresholds at once with the system-level click "intensity" parameter).

Contact/motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (eg, different movements, timing, and/or strength of detected contact). Thus, gestures are optionally detected by detection of specific contact patterns. For example, detecting a finger tap gesture involves detecting a finger down event (e.g., at an icon location) followed by a finger down event at the same (or substantially the same) location as the finger down event. including detecting lift-off events. As another example, detecting a finger swipe gesture on a touch-sensitive surface includes detecting a finger down event, followed by detecting one or more finger dragging events, followed by Including detecting a finger up (lift-off) event.

The graphics module 132 includes components for changing the visual effects (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of displayed graphics on the touch screen 112 or other display. It includes various known software components for rendering and displaying graphics. As used herein, the term “graphics” includes, but is not limited to, text, web pages, icons (such as user interface objects including softkeys), digital images, video, animations, and the like. Contains any object that can be displayed against.

In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. Graphics module 132 receives, from an application or the like, one or more codes specifying graphics to be displayed, along with coordinate data and other graphics characteristic data as appropriate, for subsequent output to display controller 156 . Generates image data for the screen to be displayed.

Haptic feedback module 133 provides instructions used by tactile output generator(s) 167 to generate tactile outputs at one or more locations on device 100 in response to user interaction with device 100. contains various software components for generating

Text input module 134 is optionally a component of graphics module 132 and is used for various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input). application) provides a soft keyboard for entering text.

GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to phone 138 for use in location-based calling, as photo/video metadata). to the camera 143 and to applications that provide location-based services such as weather widgets, local yellow pages widgets and maps/navigation widgets).

Application 136 optionally includes the following modules (or instruction sets), or subsets or supersets thereof.
- the contacts module 137 (sometimes also called an address book or contact list);
a telephone module 138;
a teleconferencing module 139,
- email client module 140;
an instant messaging (IM) module 141;
● training support module 142,
- a camera module 143 for still and/or video images;
an image management module 144;
● video player module,
● music player module,
the browser module 147,
a calendar module 148;
Of weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5 and other widgets obtained by the user, and user-created widgets 149-6 Widget module 149, optionally including one or more of
- a widget creation module 150 for creating user-created widgets 149-6;
a search module 151,
- a video and music player module 152, which integrates a video player module and a music player module;
● memo module 153,
• a map module 154 and/or • an online video module 155;

Examples of other applications 136 that are optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA enabled applications, encryption, digital Rights management, speech recognition, and speech reproduction are included.

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, contact module 137 (e.g., application internal state 192 of contact module 137 in memory 102 or memory 370 is optionally used to manage an address book or contact list, including adding name(s) to the address book, deleting name(s) from the associating phone number(s), email address(es), address(es), or other information with names; associating images with names; sorting and sorting names; providing a phone number or email address to initiate and/or facilitate communication by phone 138, videoconferencing module 139, email 140, or IM 141;

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, telephone module 138 optionally includes , enter a series of characters corresponding to a phone number, access one or more phone numbers in the contacts module 137, modify the entered phone number, dial the corresponding phone number, and have a conversation. , and is used to disconnect or hang up when the conversation is complete. As previously mentioned, wireless communication optionally uses any of a number of communication standards, protocols, and technologies.

RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch screen 112, display controller 156, light sensor 164, light sensor controller 158, touch/motion module 130, graphics module 132, text input module 134, contact module 137. , and telephony module 138, the videoconferencing module 139 executes, according to the user's instructions, for starting, conducting, and terminating a videoconference between the user and one or more other participants. Contains possible instructions.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, email client module 140 composes emails in response to user instructions. contains executable instructions for transmitting, receiving, and managing In conjunction with image management module 144 , email client module 140 greatly facilitates creating and sending emails with still or video images captured by camera module 143 .

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, instant messaging module 141 can input text strings corresponding to instant messages, modify already entered characters (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or XMPP for Internet-based instant messaging) , SIMPLE, or IMPS), receive instant messages, and view received instant messages. In some embodiments, instant messages sent and/or received are optionally graphics, photos, audio files, as supported by MMS and/or Enhanced Messaging Service (EMS). , video files, and/or other attachments. As used herein, "instant messaging" refers to telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., using XMPP, SIMPLE, or IMPS). (messages sent as

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154 and music player module, training support module 142 ( Communicate with training sensors (sports devices); receive training sensor data; calibrate sensors used to monitor training; Contains executable instructions for selecting and playing music for training, and for displaying, storing, and transmitting training data.

In conjunction with touch screen 112, display controller 156, light sensor(s) 164, light sensor controller 158, touch/motion module 130, graphics module 132, and image management module 144, camera module 143 captures still images or It contains executable instructions for capturing videos (including video streams) and storing them in memory 102 , modifying characteristics of still images or videos, or deleting still images or videos from memory 102 .

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 arranges and modifies still and/or video images. Contains executable instructions to (eg, edit) or otherwise manipulate, label, delete, present (eg, in a digital slideshow or album), and store.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, browser module 147 provides web pages or portions thereof and links to web pages. It contains executable instructions for browsing the Internet according to the user's instructions, including retrieving, linking, receiving, and displaying attached attachments and other files.

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 can and executable instructions for creating, displaying, modifying, and storing calendars and data associated with calendars (eg, calendar items, to-do lists, etc.) in accordance with the .

In conjunction with RF circuit 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, and browser module 147, widget module 149 is optionally downloaded by the user. Mini-applications used (eg, weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or mini-applications created by users ( For example, a user created widget 149-6). In some embodiments, widgets include HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files and JavaScript files. In some embodiments, widgets include XML (extensible markup language) files and JavaScript files (eg, Yahoo! widgets).

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, and browser module 147, widget creation module 150 optionally creates widgets. (eg, turn user-specified portions of web pages into widgets).

In conjunction with touch screen 112 , display controller 156 , touch/motion module 130 , graphics module 132 , and text input module 134 , search module 151 follows user instructions and provides one or more search criteria (e.g., one or more user-specified search terms) to search for text, music, sounds, images, videos, and/or other files in memory 102 that match the search term.

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108 and browser module 147, video and music player module 152 can display MP3 or AAC files, etc. and video (on touch screen 112 or via external port 124). contains executable instructions for displaying, presenting or otherwise reproducing (on an external display to which it is connected); In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, and text input module 134, notes module 153 is used to create and manage notes, to-do lists, etc., according to user instructions. contains executable instructions for

In conjunction with RF circuitry 108, touch screen 112, display controller 156, touch/motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 optionally: receive, display, and modify maps and data associated with maps (e.g., driving directions, store and other points of interest data at or near a particular location, and other location-based data) at the direction of a user; used to store and store

In conjunction with touch screen 112, display controller 156, touch/motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140 and browser module 147, online video Module 155 instructs the user to use the H.264. 264, etc., to access, view, receive (eg, by streaming and/or downloading), and receive (eg, on a touch screen or externally connected via the external port 124) (on your display), send emails containing links to specific online videos, and otherwise manage them. In some embodiments, instant messaging module 141 is used rather than email client module 140 to send links to specific online videos. Additional description of online video applications can be found in U.S. Provisional Patent Application No. 60/936,562, "Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos," filed Jun. 20, 2007, and 2007 U.S. patent application Ser. incorporated herein.

Each of the above-identified modules and applications comprise a set of executable instructions and methods described herein (e.g., computer-implemented methods and methods described herein) for performing one or more of the functions described above. other information processing methods described). These modules (eg, instruction sets) need not be implemented as separate software program procedures or modules, and thus in various embodiments various subsets of these modules are optionally combined. or otherwise reorganized. For example, a video player module is optionally combined with a music player module into a single module (eg, video and music player module 152 of FIG. 1A). In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. Additionally, memory 102 optionally stores additional modules and data structures not previously described.

In some embodiments, device 100 is a device in which operation of a predetermined set of functions on the device is performed exclusively through a touchscreen and/or touchpad. By using a touchscreen and/or touchpad as the primary input control device for operation of device 100, the number of physical input control devices (pushbuttons, dials, etc.) on device 100 is optionally reduced.

A predetermined set of functions performed exclusively through the touchscreen and/or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad navigates device 100 to the main menu, home menu, or root menu from any user interface displayed on device 100 when touched by a user. In such embodiments, the "menu button" is implemented using a touchpad. In some other embodiments, the menu buttons are physical push buttons or other physical input control devices instead of touchpads.

FIG. 1B is a block diagram illustrating exemplary components for event processing, according to some embodiments. In some embodiments, memory 102 (FIG. 1A) or memory 370 (FIG. 3) includes event sorter 170 (eg, within operating system 126) and corresponding applications 136-1 (eg, applications 137-151, 155, any of 380-390).

Event sorter 170 receives event information and determines application 136-1 and application view 191 for application 136-1 to which the event information will be delivered. Event sorter 170 includes event monitor 171 and event dispatcher module 174 . In some embodiments, application 136-1 has application internal state 192 that indicates the current application view(s) displayed on touch-sensitive display 112 when the application is active or running. included. In some embodiments, device/global internal state 157 is used by event sorter 170 to determine which applications are currently active, and application internal state 192 is the application view to which event information is delivered. 191 by event sorter 170 .

In some embodiments, application internal state 192 indicates resume information used when application 136-1 resumes execution, information being displayed or prepared for display by application 136-1. one of user interface state information, a state queue for allowing the user to return to a previous state or view of the application 136-1, and a redo/undo queue of previous actions taken by the user. Include additional information such as:

Event monitor 171 receives event information from peripherals interface 118 . Event information includes information of sub-events (eg, a user touch on touch-sensitive display 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I/O subsystem 106 or sensors such as proximity sensor 166, accelerometer(s) 168 and/or microphone 113 (through audio circuitry 110). Information that peripherals interface 118 receives from I/O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (eg, receiving input above a predetermined noise threshold and/or for longer than a predetermined period of time).

In some embodiments, event sorter 170 also includes hit view determination module 172 and/or active event recognizer determination module 173 .

Hit view determination module 172 provides software procedures for determining where within one or more views a sub-event occurred when touch-sensitive display 112 displayed one or more views. Views are made up of controls and other elements that the user can see on the display.

Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the respective application) in which the touch is detected optionally corresponds to a program level within the application's program hierarchy or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally called the hit view, and the set of events recognized as appropriate input is optionally the initial view that initiates a touch-based gesture. is determined based at least in part on the hit view of the touch.

The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should process the sub-event. In most situations, the hit view is the lowest level view where the first sub-event occurs (eg, the first sub-event in a sequence of sub-events forming an event or potential event). Once a hit view is identified by hit view determination module 172, the hit view typically receives all sub-events associated with the same touch or input source for which it was identified as the hit view.

Active event recognizer determination module 173 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only hit views should receive a particular sequence of sub-events. In another embodiment, the active event recognizer determination module 173 determines that all views containing the physical location of the sub-event are actively participating views, and thus all actively participating views. A viewing view determines that it should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is completely confined to the region associated with one particular view, the view higher up the hierarchy remains the actively participating view.

Event dispatcher module 174 dispatches event information to an event recognizer (eg, event recognizer 180). In embodiments that include active event recognizer determination module 173 , event dispatcher module 174 distributes event information to event recognizers determined by active event recognizer determination module 173 . In some embodiments, the event dispatcher module 174 stores event information obtained by the corresponding event receiver 182 in an event queue.

In some embodiments, operating system 126 includes event sorter 170 . Alternatively, application 136 - 1 includes event sorter 170 . In yet other embodiments, event sorter 170 is a standalone module or part of another module stored in memory 102 , such as contact/motion module 130 .

In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each containing instructions for handling touch events that occur within a corresponding view of the application's user interface. including. Each application view 191 of application 136 - 1 includes one or more event recognizers 180 . Typically, the corresponding application view 191 includes multiple event recognizers 180 . In other embodiments, one or more of event recognizers 180 are part of separate modules, such as user interface kits or higher-level objects, from which application 136-1 inherits methods and other properties. be. In some embodiments, the corresponding event handler 190 processes one or more of the data updater 176, the object updater 177, the GUI updater 178 and/or the event data 179 received from the event sorter 170. include. Event handler 190 optionally utilizes or calls data updater 176 , object updater 177 , or GUI updater 178 to update application internal state 192 . Alternatively, one or more of application views 191 include one or more corresponding event handlers 190 . Also, in some embodiments, one or more of data updater 176 , object updater 177 and GUI updater 178 are included in the corresponding application view 191 .

A corresponding event recognizer 180 receives event information (eg, event data 179) from the event sorter 170 and identifies events from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184 . In some embodiments, event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (optionally including sub-event delivery instructions).

The event receiver 182 receives event information from the event sorter 170 . The event information includes information about sub-events, such as touches or touch movements. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event involves touch movement, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes rotation of the device from one orientation to another (eg, portrait to landscape or vice versa), and the event information is the current orientation of the device. contains corresponding information about (also called device pose).

The event comparator 184 compares the event information with predetermined event or sub-event definitions and determines the event or sub-event or determines or updates the state of the event or sub-event based on the comparison. In some embodiments, event comparator 184 includes event definitions 186 . Event definitions 186 include definitions of events (eg, a predetermined sequence of sub-events), eg, event 1 (187-1), event 2 (187-2), and so on. In some embodiments, sub-events in event (187) include, for example, beginning of touch, end of touch, movement of touch, stop of touch, and multiple touches. In one example, the definition of Event 1 (187-1) is a double tap on a displayed object. A double-tap is, for example, a first touch on a displayed object at a given stage (start of touch), a first lift-off at a given stage (end of touch), a , and a second lift-off at a predetermined stage (end of touch). In another example, the definition of event 2 (187-2) is a drag operation on a displayed object. This drag operation includes, for example, touching (or contacting) on a display object at a predetermined stage, moving the touch across the touch-sensitive display 112, and lifting off the touch (finishing the touch). In some embodiments, the event also includes information about one or more associated event handlers 190 .

In some embodiments, event definitions 187 include event definitions for respective user interface objects. In some embodiments, event comparator 184 performs hit testing to determine which user interface objects are associated with sub-events. For example, in an application view in which three user interface objects are displayed on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 determines which of the three user interface objects has been touched. Hit test to determine if it is associated with (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the results of the hit test to determine which event handler 190 should be activated. . For example, the event comparator 184 selects the event handler associated with the object and the sub-event that triggers the hit test.

In some embodiments, the definition of the corresponding event (187) also includes a delay that delays the delivery of event information until after it has been determined whether a sequence of sub-events corresponds to the event type of the event recognizer. Including action.

When the corresponding event recognizer 180 determines that the sequence of sub-events does not match any of the events in the event definition 186, the corresponding event recognizer 180 enters an event-unable, event-failed, or event-completed state; After that, ignore subsequent touch-based gesture sub-events. In this situation, if there is another event recognizer that remains active for the hit view, that event recognizer will continue to track and process the ongoing touch-based gesture sub-events.

In some embodiments, the corresponding event recognizer 180 has configurable properties, flags and/or properties that indicate to actively participating event recognizers how the event delivery system should handle sub-event delivery. Contains metadata 183 with a list. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate how event recognizers interact or are allowed to interact with each other. include. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate whether sub-events are delivered to various levels in the view hierarchy or program hierarchy.

In some embodiments, each event recognizer 180 activates the event processor 190 associated with the event when one or more specific sub-events of the event are recognized. In some embodiments, each event recognizer 180 delivers event information associated with the event to the event handler 190 . Activating the event handler 190 is separate from sending (and deferring sending) sub-events to each hit view. In some embodiments, event recognizer 180 populates a flag associated with a recognized event, and event processor 190 associated with the flag captures the flag and performs predetermined processing.

In some embodiments, event delivery instructions 188 include sub-event delivery instructions to deliver event information about the sub-event without activating the event handler. Instead, sub-event delivery instructions deliver event information to event handlers associated with a sequence of sub-events or to actively participating views. An event handler associated with a sequence of sub-events or actively participating views receives event information and performs predetermined processing.

In some embodiments, data updater 176 creates and updates data used by application 136-1. For example, the data updater 176 updates phone numbers used by the contacts module 137 or stores video files used by the video player module. In some embodiments, object updater 177 creates and updates objects used by application 136-1. For example, the object updater 177 creates new user interface objects or updates the positions of user interface objects. The GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

In some embodiments, event handler(s) 190 includes or has access to data updater 176 , object updater 177 and GUI updater 178 . In some embodiments, data updater 176 , object updater 177 , and GUI updater 178 are included in a single module of each application 136 - 1 or application view 191 . In other embodiments, they are contained in two or more software modules.

The preceding discussion regarding event handling of a user's touch on a touch-sensitive display also applies to other forms of user input for operating multifunction device 100 with input devices, all of which are initiated on the touch screen. It should be understood that For example, mouse movement and mouse button presses, optionally adapted to pressing or holding single or multiple keyboards, contact movements such as tapping, dragging, scrolling on a touchpad, pen stylus input, device Movements, verbal commands, detected eye movements, biometric inputs, and/or any combination thereof are optionally used as inputs corresponding to sub-events defining the event to be recognized. used.

FIG. 2 illustrates portable multifunction device 100 having touch screen 112, according to some embodiments. The touch screen optionally displays one or more graphics within a user interface (UI) 200 . In this embodiment, as well as other embodiments described below, the user may, for example, use one or more fingers 202 (not shown to scale) or one or more styluses 203 (not shown to scale). , not drawn to scale), one or more of the graphics can be selected by gesturing on the graphics. In some embodiments, selection of one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture is one or more taps, one or more swipes (left to right, right to left, up and/or down), and/or contact with device 100 Finger rolling (right to left, left to right, up and/or down) is optionally included. In some implementations or situations, accidental contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

Device 100 also optionally includes one or more physical buttons, such as “home” or menu button 204 . As mentioned above, menu button 204 is optionally used to navigate to any application 136 in the set of applications that are optionally running on device 100 . Alternatively, in some embodiments, menu buttons are implemented as softkeys in a GUI displayed on touch screen 112 .

In some embodiments, the device 100 includes a touch screen 112, a menu button 204, push buttons 206 for turning power to the device on/off and locking the device, volume control button(s) 208, It includes a Subscriber Identity Module (SIM) card slot 210, a headset jack 212, and an external port 124 for docking/charging. Push button 206 optionally turns on/off power on the device by depressing the button and holding the button down for a predetermined time, depressing the button for a predetermined time. Used to lock the device by previously releasing the button and/or unlock the device or initiate the unlock process. In an alternative embodiment, device 100 also accepts verbal input through microphone 113 to activate or deactivate some functions. Device 100 also optionally includes one or more contact strength sensors 165 for detecting the strength of contact on touch screen 112 and/or one for generating tactile output to the user of device 100 . One or more tactile output generators 167 are also included.

FIG. 3 is a block diagram of an exemplary multifunction device with a display and touch-sensitive surface, according to some embodiments; Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, desktop computer, tablet computer, multimedia module device, navigation device, educational device (such as a children's learning toy), gaming system, or control device (e.g., home controller for commercial use or business use). Device 300 generally includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. include Communication bus 320 optionally includes circuitry (sometimes called a chipset) that interconnects and controls communications between system components. Device 300 includes input/output (I/O) interface 330 including display 340, which is typically a touch screen display. I/O interface 330 also includes a keyboard and/or mouse (or other pointing device) 350 and touchpad 355 for generating tactile output on device 300 (e.g., as described above with reference to FIG. 1A). tactile output generator 357 (similar to tactile output generator(s) 167), sensor 359 (e.g., light sensor, acceleration sensor, proximity sensor, touch sensitive sensor, and/or as described above with reference to FIG. 1A); optional contact intensity sensor(s) similar to 165). Memory 370 includes high speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and also one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile memory devices. Optionally includes non-volatile memory such as a non-volatile solid state storage device. Memory 370 optionally includes one or more storage devices located remotely from CPU(s) 310 . In some embodiments, memory 370 includes programs, modules and data structures similar to, or a subset thereof, stored in memory 102 of portable multifunction device 100 (FIG. 1A). memorize Additionally, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100 . For example, memory 370 of device 300 optionally stores drawing module 380, presentation module 382, word processing module 384, website creation module 386, disc authoring module 388 and/or spreadsheet module 390, although portable multiple Memory 102 of functional device 100 (FIG. 1A) optionally does not store these modules.

Each of the elements of FIG. 3 identified above are optionally stored in one or more of the aforementioned memory devices. Each of the modules identified above corresponds to a set of instructions that perform the functions previously described. The modules or programs (e.g., instruction set) identified above need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules are combined in various embodiments. or optionally reorganized in other ways. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Additionally, memory 370 optionally stores additional modules and data structures not previously described.

Attention is now directed to user interface embodiments optionally implemented on, for example, portable multifunction device 100 .

FIG. 4A shows an exemplary user interface for a menu of applications on portable multifunction device 100, according to some embodiments. A similar user interface is optionally implemented on device 300 . In some embodiments, user interface 400 includes the following elements, or subsets or supersets thereof.
- signal strength indicator(s) 402 for wireless communication(s) such as cellular and Wi-Fi signals;
● Time 404,
- Bluetooth indicator 405,
- battery status indicator 406;
• a tray 408 containing icons for frequently used applications, such as;
o an icon 416 for the phone module 138 labeled "phone", optionally including an indicator 414 of the number of missed calls or audiomail messages;
o an icon 418 for the email client module 140, labeled "Mail", optionally including an indicator 410 of the number of unread emails;
o icon 420 for browser module 147, labeled "Browser"; and o for video and music player module 152, also referred to as iPod (trademark of Apple Inc.) module 152, labeled "iPod". icon 422, and icons for other applications such as
o icon 424 for IM module 141, labeled "message";
o an icon 426 for the calendar module 148, labeled "Calendar";
o an icon 428 for the image management module 144, labeled "Photos";
o icon 430 for camera module 143, labeled "Camera";
o icon 432 for online video module 155, labeled "online video";
o icon 434 for stock price widget 149-2, labeled "stock price";
o icon 436 for map module 154, labeled "Map";
o icon 438 for weather widget 149-1, labeled "weather";
o icon 440 for alarm clock widget 149-4, labeled "Clock";
o an icon 442 for the Training Support module 142, labeled "Training Support";
o Icon 444 for Notes module 153, labeled "Notes"; and o Settings application or module, labeled "Settings", which provides access to settings for device 100 and its various applications 136. icon 446 for.

Note that the icon labels shown in FIG. 4A are merely exemplary. For example, icon 422 for video and music player module 152 is labeled "music" or "music player." Other labels are optionally used for various application icons. In some embodiments, the label of each application icon includes the name of the application corresponding to each application icon. In some embodiments, the label of a particular application icon is different than the name of the application corresponding to the particular application icon.

FIG. 4B is an illustration on a device (eg, device 300 of FIG. 3) with a touch-sensitive surface 451 (eg, tablet or touchpad 355 of FIG. 3) separate from display 450 (eg, touchscreen display 112). a typical user interface. Device 300 may also include one or more contact intensity sensors (eg, one or more of sensors 359 ) for detecting the intensity of contact on touch-sensitive surface 451 and/or a tactile sensor for a user of device 300 . Optionally includes one or more tactile output generators 357 for generating output.

Although some of the examples below are described with reference to input on the touch screen display 112 (where the touch sensitive surface and display are combined), in some embodiments the device Input is detected on a touch-sensitive surface separate from the display, as shown in FIG. 4B. In some embodiments, this touch-sensitive surface (eg, 451 in FIG. 4B) has a major axis (eg, 452 in FIG. 4B) that corresponds to a major axis (eg, 453 in FIG. 4B) on the display (eg, 450). have According to these embodiments, the device detects contact with touch-sensitive surface 451 (e.g., 460 and 462 in FIG. 4B) at locations corresponding to their respective locations on the display (e.g., in FIG. 4B). , 460 corresponds to 468 and 462 corresponds to 470). Thus, if the touch-sensitive surface is separate from the display, user input (e.g., contacts 460 and 462 and their movement) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4B) ) is used by the device to operate the user interface on the multifunction device's display (eg, 450 in FIG. 4B). It should be appreciated that similar methods are optionally used for other user interfaces described herein.

Additionally, although the examples below are described primarily with reference to finger inputs (e.g., finger contact, finger tap gestures, finger swipe gestures), in some embodiments these finger inputs It should be appreciated that one or more of the may be replaced with input from another input device (eg, mouse-based input or stylus input). For example, a swipe gesture is optionally replaced with a mouse click (eg, instead of contact) followed by cursor movement along the path of the swipe (eg, instead of contact movement). As another example, a tap gesture is optionally replaced with a mouse click while the cursor is positioned over the location of the tap gesture (e.g. instead of detecting contact and subsequently stopping detecting contact). ). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mouse and finger contact are optionally used simultaneously.

FIG. 5A shows an exemplary personal electronic device 500. As shown in FIG. Device 500 includes body 502 . In some embodiments, device 500 can include some or all of the features described with respect to devices 100 and 300 (eg, FIGS. 1A-4B). In some embodiments, device 500 has a touch-sensitive display screen 504 (hereinafter touch screen 504). Alternatively, or in addition to touchscreen 504, device 500 has a display and a touch-sensitive surface. As with device 100 and device 300, in some embodiments, touch screen 504 (or touch-sensitive surface) optionally includes a 1D sensor for detecting the intensity of contact (eg, touch) being applied. have one or more intensity sensors. One or more intensity sensors of touch screen 504 (or touch-sensitive surface) can provide output data representing the intensity of the touch. The user interface of device 500 can respond to touches based on the intensity of the touch, meaning that different intensity touches can invoke different user interface operations on device 500 .

Exemplary techniques for detecting and processing touch strength are described, for example, in related application entitled "Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application," filed May 8, 2013. International Patent Application No. PCT/US2013/040061 (published as WO 2013/169849), filed on November 11, 2013, and "Device, Method, and Graphical User Interface for Transitioning Between Touch Input to D Display Output No. PCT/US2013/069483 (published as WO 2014/105276) entitled “Relationships”, each of which is incorporated herein by reference in its entirety.

In some embodiments, device 500 has one or more input mechanisms 506 and 508 . Input mechanisms 506 and 508, if included, may be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, may, for example, attach device 500 to hats, eyewear, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, pants, belts, shoes, purses, backpacks, etc. allow to install. These attachment mechanisms allow device 500 to be worn by a user.

FIG. 5B shows an exemplary personal electronic device 500. As shown in FIG. In some embodiments, device 500 can include some or all of the components described with respect to FIGS. 1A, 1B and 3. FIG. Device 500 has a bus 512 operably coupling I/O section 514 to one or more computer processors 516 and memory 518 . The I/O section 514 can be connected to a display 504, which can have a touch sensing component 522 and, optionally, an intensity sensor 524 (eg, a contact intensity sensor). Additionally, the I/O section 514 includes a communication unit 530 for receiving application and operating system data using Wi-Fi, Bluetooth, Near Field Communication (NFC), cellular and/or other wireless communication technologies. can connect to Device 500 can include input mechanisms 506 and/or 508 . Input mechanism 506 is optionally, for example, a rotatable input device or a depressible and rotatable input device. Input mechanism 508 is optionally a button in some examples.

Input mechanism 508 is optionally a microphone in some examples. Personal electronic device 500 optionally includes various sensors such as GPS sensor 532, accelerometer 534, orientation sensor 540 (eg, compass), gyroscope 536, motion sensor 538, and/or combinations thereof. , all of which are operably connectable to I/O section 514 .

Memory 518 of personal electronic device 500 may include one or more non-transitory computer-readable storage media for storing computer-executable instructions, which are executed by one or more computer processors 516. 7A-7F, 9A-9K, 11A-11F, 13A-13D, and 15A-15C, to a computer processor, for example. ), including the techniques described below. A computer-readable storage medium can be any medium capable of tangibly containing or storing computer-executable instructions for use by or in connection with an instruction-execution system, apparatus, or device. In some examples, the storage medium is temporary computer-readable storage medium. In some examples, the storage medium is non-transitory computer-readable storage medium. Non-transitory computer-readable storage media can include, but are not limited to, magnetic storage devices, optical storage devices, and/or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD or Blu-ray technology, and persistent solid state memories such as flash, solid state drives. Personal electronic device 500 is not limited to the components and configuration of FIG. 5B, but may include other or additional components in multiple configurations.

As used herein, the term “affordance” refers to a user-interactive refers to a graphical user interface object in For example, images (eg, icons), buttons and text (eg, hyperlinks) each optionally constitute affordances.

As used herein, the term "focus selector" refers to an input element that indicates the current portion of the user interface with which the user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector", where the cursor is positioned over a particular user interface element (e.g., button, window, slider, or other user interface element). A particular user interface element is detected when an input (e.g., press input) is detected on a touch-sensitive surface (e.g., touch pad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B) when the adjusted according to the input. Some implementations include a touchscreen display (eg, touch-sensitive display system 112 in FIG. 1A or touchscreen 112 in FIG. 4A) that allows direct interaction with user interface elements on the touchscreen display. In , the touch detected on the touch screen acts as a "focus selector", allowing input (e.g., press input from touch) to be directed to a specific user interface element (e.g., button, window, etc.) on the touch screen display. , slider, or other user interface element), that particular user interface element is adjusted according to the detected input. In some implementations, the user can move the focus from one button to another without corresponding cursor movement or contact movement on the touch screen display (e.g., by using tab or arrow keys to move focus from one button to another). Focus is moved from one area of the interface to another area of the user interface, and in those implementations the focus selector moves according to the movement of focus between different areas of the user interface. Regardless of the specific form taken by the focus selector, it is generally used to communicate the user's intended interaction with the user interface (e.g., the user interface with which the user intends to interact). A user interface element (or a touch on a touch screen display) that is controlled by the user (by presenting the element of the interface to the device). For example, the position of a focus selector (e.g. cursor, touch or selection box) over the corresponding button when a press input is detected on a touch-sensitive surface (e.g. a touchpad or touchscreen) indicates that the user is activating the corresponding button (rather than other user interface elements shown on the display of the ).

As used herein and in the claims, the term "characteristic strength" of contact refers to a characteristic of contact based on one or more strengths of contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity may be a predetermined number of intensity samples, or for a predetermined event (e.g., after detecting contact, before detecting lift-off of contact, before or after detecting start of contact movement, detecting end of contact). for a predetermined time (e.g., 0.05, 0.1, 0.2, 0.05, 0.1, 0.2, optionally based on a set of intensity samples collected at 0.5, 1, 2, 5, 10 seconds). The characteristic strength of contact is optionally a maximum strength of contact, a mean value of strength of contact, an average value of strength of contact, a value in the top 10% of strength of contact, Based on one or more of a half maximum contact strength value, a 90% maximum contact strength value, and the like. In some embodiments, the duration of contact is used to determine the characteristic strength (eg, when the characteristic strength is the average of the strength of the contact over time). In some embodiments, the characteristic strength is compared to a set of one or more strength thresholds to determine if the manipulation was performed by the user. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, the first action is performed as a result of a contact having a characteristic intensity that does not exceed the first threshold, and the contact that has a characteristic intensity that exceeds the first intensity threshold but does not exceed the second intensity threshold results in the first action. A second action is performed as , and a third action is performed as a result of a contact having a characteristic strength exceeding a second threshold. In some embodiments, rather than the comparison between the characteristic strength and one or more thresholds being used to determine whether to take the first action or the second action, one It is used to determine whether to perform the above operations (for example, to perform each operation or to refrain from performing each operation).

FIG. 5C shows detection of multiple contacts 552A-552E on touch-sensitive display screen 504 by multiple intensity sensors 524A-524D. FIG. 5C additionally includes an intensity diagram showing current intensity measurements of intensity sensors 524A-524D versus intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are each 9 intensity units, and the intensity measurements of intensity sensors 524B and 524C are each 7 intensity units. In some implementations, the total intensity is the sum of intensity measurements of multiple intensity sensors 524A-524D, which in this example is 32 intensity units. In some embodiments, each contact is assigned a respective intensity that is a fraction of the total intensity. FIG. 5D shows the assignment of total strength to contacts 552A-552E based on the distance of contacts 552A-552E from center of force 554. FIG. In this example, contacts 552A, 552B, and 552E are each assigned a strength of contact that is 8 strength units of the total strength, and contacts 552C and 552D are each assigned a strength of contact that is 4 strength units of the total strength. is assigned. More generally, in some implementations, each contact j is assigned a respective intensity Ij, which is a fraction of the total intensity A, according to a predetermined mathematical function Ij=A·(Dj/ΣDi); where Dj is the distance from the center of force to each contact j, and ΣDi is the sum of the distances from the center of force to all respective contacts (eg, from i=1 to the last). The operations described with reference to FIGS. 5C-5D may be performed using an electronic device similar or identical to device 100, 300 or 500. FIG. In some embodiments, the characteristic strength of contact is based on one or more strengths of contact. In some embodiments, an intensity sensor is used to determine a single characteristic strength (eg, a single characteristic strength of a single contact). Note that intensity maps are not part of the displayed user interface, but are included in FIGS. 5C-5D for the benefit of the reader.

In some embodiments, a portion of the gesture is identified for purposes of determining characteristic strength. For example, the touch-sensitive surface optionally receives successive swipe contacts transitioning from a starting position to an ending position with increasing strength of the contact. In this example, the characteristic strength of the contact at the end position is optionally based on only a portion of that swipe contact (eg, only the portion of the swipe contact at the end position) rather than the entire continuous swipe contact. In some embodiments, a smoothing algorithm is optionally applied to the strength of the swipe contact prior to determining the characteristic strength of the contact. For example, the smoothing algorithms optionally include one or more of an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and/or an exponential smoothing algorithm. In some situations, these smoothing algorithms remove small increases or decreases in strength of swipe contacts for purposes of determining characteristic strength.

The intensity of contact on the touch-sensitive surface is optionally scaled to one or more intensity thresholds, such as a touch detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and/or one or more other intensity thresholds. characterize against. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device performs actions commonly associated with clicking a physical mouse button or trackpad. In some embodiments, the deep pressure intensity threshold corresponds to the intensity at which the device performs an action that differs from actions typically associated with clicking a physical mouse button or trackpad. In some embodiments, if touch is detected with a characteristic intensity below a light pressure intensity threshold (and above a slight touch detection intensity threshold below which no contact is detected), the device detects a light pressure. A focus selector is moved according to the movement of the contact on the touch-sensitive surface without taking actions associated with the pressure intensity threshold or the deep pressure intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent across different sets of user interface diagrams.

Increasing the characteristic strength of a contact from an intensity below the light pressure threshold to an intensity between the light pressure threshold and the deep pressure threshold may be referred to as a "light pressure" input. Increasing the characteristic strength of a contact from an intensity below the deep pressure threshold to an intensity above the deep pressure threshold may be referred to as a "deep pressure" input. Increasing the characteristic strength of a touch from an intensity below the touch detection intensity threshold to an intensity between the touch detection intensity threshold and the light pressure intensity threshold may be referred to as detecting contact on the touch surface. A decrease in the characteristic strength of the contact from an intensity above the contact detection intensity threshold to an intensity below the contact detection intensity threshold may be referred to as detecting lift-off of the contact from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection strength threshold is greater than zero.

In some embodiments described herein, one or more operations are performed in response to detecting a gesture, including a respective pressure input, or on a respective contact (or multiple contacts). in response to detecting each pressure input, each pressure input being detected based at least in part on detecting an increase in intensity of the contact (or contacts) above a threshold pressure input intensity. be done. In some embodiments, the respective action is performed in response to detecting an increase in the strength of the respective contact (eg, a "downstroke" of the respective pressure input) above a pressure input strength threshold. In some embodiments, the pressure input includes increasing the strength of each contact above the pressure input strength threshold and subsequently decreasing the strength of the contact below the pressure input strength threshold, wherein each action is This is done in response to detecting a decrease in the strength of the respective touch below the input threshold (eg, an "upstroke" of the respective pressure input).

FIGS. 5E-5H illustrate the intensity of contact 562 from an intensity below the light pressure intensity threshold (eg, “IT _L ”) in FIG. 5E to an intensity above the deep pressure intensity threshold (eg, “IT _D ”) in FIG. 5H. Fig. 4 shows detection of gestures including press inputs corresponding to increasing strength; The gesture made by contact 562 causes cursor 576 to be displayed over application icon 572B corresponding to app 2 on displayed user interface 570, including application icons 572A-572D displayed in predetermined area 574. is detected on touch-sensitive surface 560 when In some embodiments, gestures are detected on touch-sensitive display 504 . The intensity sensor detects the intensity of contact on touch-sensitive surface 560 . The device determines that the intensity of contact 562 has peaked above a deep pressure intensity threshold (eg, “IT _D ”). Contact 562 is maintained on touch sensitive surface 560 . In response to detection of the gesture, and in response to contact 562 having an intensity above a deep pressure intensity threshold (eg, "IT _D ") during the gesture, the recently opened app 2, as shown in FIGS. A reduced representation 578A-578C (eg, thumbnail) of the document is displayed. In some embodiments, the strength compared to one or more strength thresholds is the characteristic strength of the contact. Note that the intensity diagram of contact 562 is not part of the displayed user interface, but is included in FIGS. 5E-5H for the benefit of the reader.

In some embodiments, the display of representations 578A-578C includes animation. For example, representation 578A is initially displayed near application icon 572B, as shown in FIG. 5F. As the animation progresses, representation 578A moves upward and representation 578B appears near application icon 572B, as shown in FIG. 5G. Representation 578A then moves upward, representation 578B moves upward toward representation 578A, and representation 578C is displayed near application icon 572B, as shown in FIG. 5H. Representations 578A-578C form an array above icon 572B. In some embodiments, the animation progresses according to the intensity of the contact 562, as shown in FIGS. 5F-5G, where the intensity of the contact 562 reaches the deep pressure intensity threshold (eg, “IT _D ”). representations 578A-578C appear and move upward. In some embodiments, the strength on which the animation progression is based is the characteristic strength of the contact. The operations described with reference to FIGS. 5E-5H may be performed using an electronic device similar or identical to device 100, 300 or 500. FIG.

In some embodiments, the device employs strength hysteresis to avoid unintended inputs, sometimes referred to as "jitter," wherein the device has a hysteresis strength with a predetermined relationship to the pressure input strength threshold. Define or select a threshold (e.g., the hysteresis intensity threshold is X intensity units below the pressure input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable ratio of the pressure input intensity threshold ). Thus, in some embodiments, the pressure input includes a corresponding increase in strength of the contact above the pressure input strength threshold followed by a subsequent decrease in strength of the contact below the hysteresis strength threshold corresponding to the pressure input strength threshold. , a corresponding action is taken in response to detecting a subsequent decrease in the intensity of the corresponding touch below its hysteresis intensity threshold (eg, an “upstroke” of the corresponding pressure input). Similarly, in some embodiments, a pressure input causes the device to increase the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the pressure input intensity threshold, and optionally a subsequent step below the hysteresis intensity. is detected only if it detects a decrease in the strength of the touch to the strength of the touch, and the corresponding action is to detect that pressure input (e.g., increase the strength of the contact, or decrease the strength of the contact, depending on the situation). will be done accordingly.

For ease of explanation, descriptions of actions taken in response to a pressure input associated with a pressure input intensity threshold, or in response to gestures involving that pressure input, are defined as increasing the strength of the contact above the pressure input intensity threshold. , increasing the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the pressing input intensity threshold, decreasing the intensity of the contact below the pressing input intensity threshold, and/or falling below the hysteresis intensity threshold corresponding to the pressing input intensity threshold. Optionally triggered in response to detecting any decrease in the intensity of the contact. Additionally, in the examples described as taking action in response to detecting a decrease in the strength of the contact below the pressure input strength threshold, the action corresponds to the pressure input strength threshold and has a lower hysteresis. Optionally in response to detecting a decrease in the intensity of the touch below an intensity threshold.

As used herein, an "installed application" is one that has been downloaded to an electronic device (eg, device 100, 300, and/or 500) and launched on that device (eg, refers to a software application that is ready In some embodiments, the downloaded application becomes an installed application by means of an installation program that extracts program portions from the downloaded package and integrates the extracted portions with the operating system of the computer system.

As used herein, the terms "open application" or "running application" refer to the state information maintained (e.g., as part of device/global internal state 157 and/or application internal state 192). Refers to a software application that is The open or running application is optionally one of the following types of applications.
● the active application currently displayed on the display screen of the device in which the application is being used,
A background application (or background process) that is not currently visible, but one or more processes for that application are being processed by one or more processors; , volatile and/or non-volatile) that has state information that can be used to resume execution of that application.

As used herein, the term "closed application" refers to a software application for which state information is not maintained (eg, no state information about closed applications is stored in the device's memory). Thus, closing an application includes stopping and/or removing application processing for that application, and removing state information for that application from the device's memory. Generally, opening a second application during a first application does not close the first application. When the second application is displayed and the first application ceases to be displayed, the first application becomes the background application.

Attention is now directed to embodiments of the user interface (“UI”) and associated processing implemented on an electronic device such as portable multifunction device 100 , device 300 or device 500 .

6A-6AF illustrate exemplary user interfaces for initiating and joining live communication sessions (eg, from a messaging application) according to some embodiments. The user interfaces of these figures are used to illustrate the processes described below, including the processes of Figures 7A-7F.

FIG. 6A shows device 600 with touch sensitive display 602 . In some embodiments, device 600 includes one or more features of devices 100 , 300 and 500 . In some embodiments, device 600 also includes camera 603, which includes an image sensor capable of capturing data representing at least a portion of the light spectrum (eg, visible, infrared, or ultraviolet light). In some embodiments, camera 603 includes multiple image sensors and/or other types of sensors. In addition to capturing data representing sensed light, in some embodiments camera 603 can capture other types of data, such as depth data. For example, in some embodiments, camera 603 also captures depth data using speckle, time-of-flight, parallax, or focus-based techniques. The image data that device 600 captures using camera 603 includes data corresponding to a portion of the light spectrum for the scene within the camera's field of view. Additionally, in some embodiments, the captured image data also includes depth data for the light data. In some other embodiments, the captured image data includes sufficient data to determine or generate depth data for the data of the portion of the light spectrum.

In some examples, electronic device 600 includes a depth camera (eg, as part of camera 603), such as an infrared camera, a thermographic camera, or a combination thereof. In some examples, the device further includes a light emitting device (eg, light projector) such as an infrared floodlight, structured light projector, or a combination thereof. The light-emitting device is optionally used to illuminate a subject during image capture by a visible light camera and a depth camera (e.g., an IR camera), and information from the depth camera and visible light camera is transmitted to the visible light camera. is used to determine depth maps for different parts of the object captured by . In some embodiments, a depth map (eg, depth map image) contains information (eg, values) about distances from a viewpoint (eg, camera) to objects in a scene. In one embodiment of the depth map, each depth pixel defines where the corresponding two-dimensional pixel is located in the Z-axis of the viewpoint. In some examples, the depth map is made up of pixels, each pixel defined by a value (eg, 0-255). For example, a '0' value represents the pixel located farthest in a '3D' scene, and a '255' value is located closest to the viewpoint (e.g. camera) in a '3D' scene. Represents a pixel that is In another example, the depth map represents the distance between objects in the scene and the plane of the viewpoint. ) In some embodiments, the depth map provides information about the relative depths of various features of the object of interest in the view from the depth camera (e.g., the relative depths of the eyes, nose, mouth, and ears on the user's face). include. In some embodiments, the depth map includes information that allows the device to determine the z-direction contour of the object of interest. In some embodiments, the lighting effects described herein combine parallax information from two cameras (e.g., two visible light cameras) for the rear image and and depth information from a depth camera combined with image data from a visible light camera. In some embodiments, the same user interface is used to generate lighting effects when determining depth information using two visible light cameras and when determining depth information using a depth camera. Users get a consistent experience even when using disparate techniques to determine what information to use when doing so. In some embodiments, while applying one of the lighting effects to display the camera user interface, the device detects the selection of the camera switching affordance and detects the selection of the front camera (e.g. depth camera and visible light camera). ) to the rear camera (e.g., two visible light cameras spaced apart from each other) (or vice versa), while maintaining the display of the user interface controller for applying lighting effects and switching the display of the front camera's field of view to the rear camera. Replace with the field of view of the camera (or vice versa).

As shown in FIG. 6A, device 600 displays messaging user interface 604 of a messaging application. Messaging user interface 604 includes message area 606 containing messages 608A-608D in a message conversation between a group of participants ("The Dream Team") that includes three or more participants.

While displaying messaging user interface 604, device 600 displays data (e.g., live communication receive an invitation to join a session). In some embodiments, the live communication session is an audio communication session or an audio/video communication session.

In response to receiving data indicating that the live communication session is active, device 600 displays a notification indicating that the live communication session is active. As shown in FIG. 6B, the notification includes notification 614 in message area 606 as a message from participant Pablo, indicating that Pablo has started a live communication session. Notification 614 indicates the type of live communication session by the text "video" and a video camera icon. Notification 614 also indicates the number of currently active participants (“1 active”) in the live communication session. In some embodiments, notification 614 indicates the names of one or more of the participants currently active in the live communication session. Notification 614 also includes live communication participation affordance 614 A, which, when selected, causes device 600 to initiate the process of participating in the live communication session represented by notification 614 . Techniques for participating in a live communication session are described in greater detail below.

Optionally, in response to receiving data indicating that the live communication session is active, device 600 generates audio output indicating that the live communication session is active. In some embodiments, the audio output is different from audio output associated with messaging applications (eg, incoming message notification) and audio output associated with telephony applications (eg, ring tone). In some embodiments, the audio output is a hybrid of audio associated with message notifications and audio associated with incoming call notifications. In some embodiments, the audio output shares audio characteristics (eg, common tone, timbre, rhythm) with both call and message notification audio outputs.

In response to receiving data indicating that a live communication session is active, device 600 also displays live communication affordance 610 . In some embodiments, the live communication affordance 610 is displayed (eg, grayed out) before the live communication session begins and is visually highlighted (eg, highlighted) once the live communication session begins. light) or modified (e.g., non-grayed out). In some embodiments, selection of live communication affordance 610 or option affordance 612 initiates the process of joining a live communication session. In some embodiments, notifications 614 and/or live communication affordances 610 are audio and/or haptic outputs (eg, as they appear in an active message conversation that the user is likely already focused on) and/or tactile outputs (eg, phone calls). displayed without incoming call output).

As shown in FIG. 6C, notification 614 moves up message area 606 to different locations as additional messages are added to the message conversation in message area 606 . In some embodiments, device 600 moves notification 614 to a location outside the message area (eg, notification 614 transitions from message to a banner at the top of display 602 outside message area 606).

In some embodiments, device 600 moves notification 614 in response to notification relocation criteria being met. In some embodiments, the notification relocation criteria are elapse of a predetermined amount of time (e.g., elapses after notification 614 is displayed), ceasing to display messaging user interface 604, or closing of the messaging application. and/or that the entire notification 614 or a portion of the notification 614 is hidden within the message area 606 (e.g., by having new incoming messages occupy the message area 606) Including the above criteria. In some embodiments, notification 614 is moved to the bottom, right, or left of display 602 . In some embodiments, notification 614 is a banner notification displayed near the top of display 612 in direct response to receiving data indicating that a live communication session is available.

In some embodiments, device 600 receives reminder data indicating a reminder to join a live communication session (eg, reminders sent by participants during a conversation or after a predetermined time). As shown in FIG. 6D, device 600 displays full screen notification 616 and outputs audio output 618 in response to receiving data indicating a reminder to join a live communication session. In some embodiments, audio output 618 is different from audio output associated with messaging applications (eg, received message notification) and audio output associated with telephony applications (eg, ring tone). In some embodiments, audio output 618 is a hybrid of audio associated with message notifications and audio associated with incoming call notifications. In some embodiments, audio output 618 is shorter than audio output associated with telephony applications (eg, ringtones) and longer than audio output associated with messaging applications (eg, received message notification). In some embodiments, the ringtone includes multiple reproductions of the corresponding audio output, and the audio output 618 is one reproduction of the corresponding audio output. In some embodiments, the audio output 618 shares audio characteristics (eg, common tone, tone, rhythm) with audio outputs for both phone calls and message notifications. In some embodiments, device 600 outputs haptic output. As shown in FIG. 6D, full screen notification 616 displays the name of the group (“THE DREAM TEAM”), the names of the currently active participants in the live communication session (“PABLO, STEPHEN”), and the live communication session at the beginning. and the name of the participant (“PABLO”) who initiated or issued the reminder. In some embodiments, full screen notification 616 includes the names of participants invited to join the live communication session (eg, according to the order in which they joined the group). The full screen notification 616 also displays an answer affordance 620 for joining a live communication session (e.g., joining a live communication session immediately or displaying a menu containing options to join a live communication session) and a full screen Also includes a "later" affordance 622 to decline the notification 616 and a message affordance 624 to return to the messaging user interface 604 or send a message to the Dream Team group. The full-screen notification also displays an image (e.g., the user's video of device 600 from camera 603) (e.g., as a preview of the user's image that other participants in the live communication session will see once the user joins). Also includes

As shown in FIG. 6D, device 600 receives (eg, detects) user input 650A (eg, tap) corresponding to selection of “later” affordance 622 . In response to user input 650A, device 600 ceases displaying full screen notification 616 and returns to messaging user interface 604, as shown in FIG. 6E.

As shown in FIG. 6E, device 600 receives (eg, detects) user input 650 B (eg, tap) corresponding to selection of option affordance 612 . In response to receiving user input 650B, device 600 expands header of messaging user interface 604 to include audio affordance 626A, video join affordance 626B, and group details affordance 626C.

As shown in FIG. 6F, device 600 receives (eg, detects) user input 650C (eg, tap) corresponding to selection of video participation affordance 626B. In some embodiments, device 600 participates in the live communication session in direct response to receiving user input 650B (eg, without requiring additional input). In some embodiments, device 600 participates in a live communication session and transmits both video and audio. In some embodiments, device 600 participates in a live communication session and only transmits audio (without transmitting video). In some embodiments, depending on the selection of audio affordance 626A, device 600 either joins the live communication session with audio only or initiates a separate conference call with group participants.

As shown in FIG. 6G, in response to receiving user input 650C, device 600 displays live communication interface 628 of the live communication application. Live communication interface 628 provides a view of the group (“THE DREAM TEAM”), images (eg, video of the user of device 600 from the front-facing camera of device 600 (eg, camera 603)), live communication sessions currently active. Includes an indication of the participant (“PABLO, STEPHEN”), an indication of the participant who started the live communication session (“Video Call from PABLO”), and a call control menu 630 . Call control menu 630 includes handle affordance 631, effect affordance 632, call affordance 634, and menu affordance 636, and so on.

As shown in FIG. 6G, device 600 receives (eg, detects) user input 650D on menu affordance 636 . As shown in FIG. 6H, in response to user input 650D, device 600 expands call control menu 630 to display additional information and controls regarding the live communication session. In some embodiments, device 600 expands call control menu 630 in response to an upward swipe starting near handle affordance 631 . Call control menu 630 is expanded to include audio on/off affordance 638, audio source menu affordance 640, video on/off affordance 642, group name 644 (with indication of number of participants in group), group message Further includes affordance 646 , list 648 containing affordances 648 A- 648 E corresponding to group participants, and add participant affordance 652 . In some embodiments, in response to receiving selection of group message affordance 646, device 600 launches and/or displays messaging user interface 604 (eg, FIG. 6F) for sending messages to group participants. do.

As shown in FIG. 6H, the participant affordances of list 648 include an indication of each participant's communication status for the live communication session. Representation 648A indicates that participant Pablo has connected to a live communication session and is providing video data (eg, a live media stream including video and audio data). Representation 648B indicates that participant Stephen has connected and is providing audio data (eg, no video). Representations 648C and 648D indicate that participants Marcel and Allison, respectively, are not connected to the live communication session (eg, not providing video and audio data). A participant who is not connected either has never participated in a live communication session or has previously participated in a live communication session but left the session. Exemplary types of participant communication states are audio-only (e.g., the participant is communicating using audio only) and video (e.g., the participant is talking using video and audio). and processing the participant's video stream for paused video (e.g. participant's video is paused) and undecoded video (e.g. delay issues, video stream format issues, etc.) exited (e.g., the participant has left the live communication session), and waiting to join (e.g., the participant has been invited to the live communication session but has not yet joined the live communication session). and are mentioned.

In some embodiments, live video communication participants include group participants currently connected to a communication session and group participants not currently connected to a live video communication session. That is, all group participants are considered participants in the live communication session, regardless of connection status. When a live communication session is active, participants can join (connect) or leave (disconnect) the live communication session.

As shown in FIG. 6I, device 600 receives (eg, detects) user input 650E (eg, tap) corresponding to selection of video on/off affordance 642. As shown in FIG. As shown in FIG. 6J, in response to receiving user input 650E, device 600 modifies video on/off affordance 642 to indicate a video off state and modifies call affordance 634 to show the video of FIG. 6I. The camera representation is modified to the phone representation of FIG. 6J to show the audio-only state. In the audio-only state, device 600 participates in a live communication session with audio only (eg, device 600 does not provide video data).

As shown in FIG. 6J, device 600 receives (eg, detects) user input 650 F (eg, tap) corresponding to selection of call affordance 634 . As shown in FIG. 6K, in response to receiving user input 650F, device 600 initiates a connection to participate in a live communication session in audio-only mode. Device 600 ceases to display expanded call control menu 630 (eg, device 600 collapses call control menu 630) and displays connection screen 654, as shown in FIG. 6K. In some embodiments, device 600 initiates a connection to join the live communication session and displays connection screen 654 in response to user input 650C on video join affordance 626B. In some embodiments, notification 614 includes affordances (eg, a “Join” icon on notification 614 or notification 614 as a whole is selectable). In some embodiments, device 600 initiates a connection to participate in a live communication session and displays connection screen 654 in response to selection of notification 614 .

As shown in FIG. 6L, after device 600 participates in a live communication session, device 600 displays representation 658 of user of device 600, representation 670A of participant Pablo (eg, the participant who initiated the live communication session), and Display live communication user interface 656, including representation 670B of participant Stephen. According to the communication states shown in list 648, participant Pablo's representation 670A includes a live video stream. Participant Stephen has provided only audio data, so the representation 670B of participant Pablo includes an avatar. In some embodiments, the layout of representations on live communication user interface 656 is based on the number of participants (eg, two participants and the user of device 600) currently connected to the live communication session. The layout shown in FIG. 6L is referred to as a two-to-one layout, with a first position corresponding to the position of representation 670A and a second position corresponding to the position of representation 670B.

In some embodiments, when the device 600 participates in a live communication session, the device 600 detects that the number of participants (e.g., either active or full (active and invited) exceeds a predetermined threshold number). determine whether there is Upon determining that the number of participants exceeds the threshold, device 600 displays an overflow region with overflow representations (eg, thumbnails), as described in more detail below. Alternatively, upon determining that the number of participants does not exceed the threshold, device 600 does not display an overflow region with an overflow representation.

When device 600 participates in a live communication session, device 600 transmits live media streams to other participants (eg, participants participating in the live communication session). When a participant associated with another device participates in the live communication session and device 600 also participates, device 600 receives the media stream corresponding to that participant.

As shown in FIG. 6L , while displaying live communication user interface 656 , device 600 receives (eg, detects) user input 650 G (eg, tap) on display 602 . In response to receiving user input 650G, device 600 displays call control menu 630, as shown in FIG. 6M. Representations 670A and 670B are reduced in size and representations 670B and 658 are moved up on display 602 so that call control menu 630 does not overlap the representation. When the call control menu 630 is displayed in FIG. 6M, the representations 670A and 670B overlap and are offset from each other, in contrast to the configuration shown in FIG. 6L, in which the representations 670A and 670B are placed one above the other and do not overlap.

As shown in FIG. 6N, device 600 receives (eg, detects) user input 650 H (eg, tap) on menu affordance 636 within call control affordance 630 . As shown in FIG. 6O, in response to receiving user input 650H, device 600 expands call control menu 630 (eg, as described with respect to FIG. 6H). In some embodiments, the device 600 displays reminder affordances in the call control menu 630 to representations of participants who are not connected to the live communication session. As shown in FIG. 6O, list 648 indicates that participants Marcel and Allison are not connected to the live communication session. Because participants Marcel and Allison (eg, devices associated with participants Marcel and Allison) are not connected to the live communication session, expanded call control menu 630 displays representations of participants Marcel and Allison in list 648. Includes outgoing affordances 662, 664 at 648C and 648D, respectively. Upon selection of outgoing affordances 662 or 664, device 600 causes a notification (eg, audio output such as a call or outgoing call) to be provided at a device associated with the participant corresponding to the selected outgoing affordance. In some embodiments, the notification is a new notification different from previous notifications sent to participants. In some embodiments, the new notification is more intrusive than the previous notification (e.g., with an audio ringtone, haptic output, or full screen notification compared to a banner notification without audio or haptic output). be). In this way, the user can be notified live that a live communication session is active and that participants can join (similar to how the user of device 600 is reminded by the full screen notification shown in FIG. 6D, for example). Participants who are not in the communication session can be reminded.

As shown in FIG. 6O, device 600 receives (eg, detects) user input 650I (eg, tap) corresponding to selection of video on/off affordance 642 . In response to receiving user input 650I, device 600 provides a live video stream (e.g., from a camera on device 600) to the live communication session and confirms that the video is on, as shown in FIG. 6P. Modify video on/off affordance 642 to indicate . 6Q, device 600 receives (eg, detects) user input 650J (eg, swipe down) on display 602 to collapse call control menu 630. In FIG. As shown in FIG. 6R, when call control menu 630 is collapsed, representation 658 corresponding to the user of device 600 includes video from camera 603 of device 600 .

As shown in FIG. 6S, device 600 receives (eg, detects) user input 650K (eg, tap) corresponding to selection of call affordance 634 . In response to user input 650K, device 600 disconnects from the live communication session. FIG. 6T shows an embodiment of messaging user interface 604 of a messaging application displayed after device 600 disconnects from a live communication session. Device 600 has disconnected from the live communication session, but the live communication session remains active. In some embodiments, even if a participant who initiated a live video communication session leaves the session, the live communication session remains active as long as at least one of the participants is connected. As shown in FIG. 6T, device 600 maintains display of notification 614 and live communication affordance 610 to indicate that a live communication session is active. Notification 614 is updated to indicate the duration of the live communication session (eg, 3 minutes and 42 seconds, which corresponds to the time since the live communication session started).

While displaying notification 614 indicating that a live communication session is active, device 600 receives data indicating that the live communication session is no longer active (eg, terminated). In some embodiments, if no participant has connected to the live communication session (e.g., none of the participants that started or joined the live communication session have connected), the live communication session is finish. In response to receiving data indicating that the live communication session is no longer active, device 600 may stop displaying notification 614 and/or display notification 614 to indicate that the live communication session is no longer active. Update. FIG. 6U illustrates an embodiment of a messaging user interface 604 of a messaging application that is displayed after a live communication session has ended (eg, in response to receiving data indicating that the live communication session is no longer active). As shown in FIG. 6U, the notification 614 is updated to include an indication that the live communication session is no longer active (eg, the text "call ended") and the live communication affordance 610 is removed. In some embodiments, live communication affordance 610 and/or notification 614 are grayed out to indicate that the live communication session is no longer active. In some embodiments, notification 614 is disabled after the live communication session ends. In some embodiments, after the live communication session ends, notification 614 remains selectable and can be selected to initiate a new live communication session with the group. In some embodiments, if device 600 did not participate in the live communication session, device 600 updates notification 614 to indicate that the user missed the live communication session (eg, "missed call").

As shown in FIG. 6U, device 600 receives (eg, detects) user input 650L (eg, a tap) corresponding to selection of return affordance 611 . As shown in FIG. 6V, in response to receiving user input 650L, device 600 displays messaging application user interface 671, which includes affordances 673A-673G for accessing respective message conversations. Contains list 673 . Affordance 673D corresponds to a message conversation between a group of three participants (Matthew, Danielle, and the user of device 600). Affordance 673D includes live communication indicator 675, which indicates that there is an active live communication session for this group. In contrast, affordance 673A corresponding to The Dream Team does not include live communication indicator 675 because the live communication session for The Dream Team is not active.

Turning to FIG. 6W, device 600 is shown displaying an interface other than messaging user interface 604 . In FIG. 6W, device 600 displays lock screen 668 indicating that device 600 is in a locked state. Lock screen 668 includes notifications 677A and 677B. Notification 677A indicates that a message from user John Appleseed was received five minutes ago. Notification 677B represents a live communication session that is no longer available (eg, a missed video call). A missed video call notification includes the invited participants (John, Amy, Rodrigo), the status of the live communication session (“Away”), and either the time the live communication session started or ended (“ 1 hour ago”). In the locked state, device 600 does not display active applications.

While in the locked state, device 600 receives data indicating that there is a live communication session to join. In some embodiments, the response of device 600 receiving data indicating that there is a live communication session to join depends on the context of device 600, the live communication session available, or other factor(s). do. In some embodiments, device 600 displays a first type of notification in response to a live communication session being available between two participants (e.g., a one-to-one communication session). . As shown in FIG. 6X, the first type of notification includes full screen notification 672 and audio output 674. In FIG. In some embodiments, audio output 674 is different than audio output associated with call notifications and audio output associated with message notifications. In some embodiments, audio output 674 shares audio characteristics (eg, common tone, tone, rhythm) with audio output for both phone calls and message notifications. Alternatively, device 600 displays a second type of notification in response to a live communication session being available between three or more participants (eg, group live communication session). As shown in FIG. 6Y, the second type of notifications includes banner notifications 676 (eg, not full screen) and haptic outputs 678 (eg, no audio output). In some embodiments, in response to a live communication session being available between three or more participants (e.g., a group live communication session), device 600 displays (e.g., the full screen notification shown in FIG. 6E After first displaying a full screen notification (similar to 616), a banner notification 676 is displayed (eg, transition to banner notification). In some embodiments, notification 676 includes an affordance that, when selected, initiates the process of joining the corresponding live communication session (eg, as described above with reference to FIGS. 6F-6L).

In contrast to notification 677B, which corresponds to a missed video call, notification 676 corresponds to an active live communication session. Notification 676 includes an indication of the group (The Dream Team), the participant (Pablo) who took the action that triggered the presentation of the notification, and the time (“Now”) when the live communication session started. In some embodiments, notification 676 is updated to accurately reflect when the live communication session started and the state of the live communication session. In some embodiments, when the live communication session represented by notification 676 ends, the text of notification 676 is updated from "Join video call from PABLO" to "Missed video call from PABLO."

In some embodiments, upon determining that no active applications are being displayed (as in FIG. 6W), device 600 responds to receiving data indicating that there is a live communication session to join. to view full-screen notifications. In some embodiments, device 600 displays a non-full screen notification (eg, banner notification) upon determining that an active application (eg, messaging application) is not displayed.

In some embodiments, upon determining to alert a user of device 600, device 600 outputs an alert (e.g., notification or audio or haptic output) and does not alert user's device 600. According to the determination, the device 600 refrains from outputting the alert. In some embodiments, the determination to alert the user is based on disturbance criteria (eg, whether the device is locked or unlocked). In some embodiments, in response to the device 600 being locked (eg, as shown in FIG. 6Y), the device outputs an alert (eg, haptic output 678) and outputs an alert (eg, haptic output 678) (eg, as shown in FIG. 6Z). Device 600 being unlocked (as shown) causes device 600 to display a notification (eg, banner notification 676) and refrain from providing alerts (eg, no haptic output).

In some embodiments, the disturbance criteria indicate whether the user is looking at device 600 (eg, based on data from sensors on the front of device 600). FIG. 6AA shows a user 684 holding device 600 . The user's line of sight 680 is not directed toward the device 600 . Device 600 determines if user 684 is not looking at device 600, and upon determining that user 684 is not looking at device 600, device 600 provides non-visual notification 682 (e.g., haptic or audio output). )I will provide a. As shown in FIG. 6AB, the user's line of sight 680 is directed toward the device 600 . Upon determining that user 684 is looking at device 600, device 600 refrains from providing non-visual notifications (eg, device 600 displays banner notifications without haptic or audio output). In some embodiments, device 600 uses an image sensor configured to identify the user's face to determine if the user is looking at device 600 .

Turning to FIGS. 6AC-6AF, device 600A includes display 602A (eg, a touch-sensitive display), rotatable input mechanism 601A, and mechanical buttons 601B. In some embodiments, device 600A includes features of devices 100, 300, 500, and 600. FIG. In some embodiments, device 600A communicates with device 600 (eg, via two-way wireless communication). In some embodiments, device 600A is in a pairing relationship with device 600 .

In some embodiments, device 600A receives data (eg, from device 600) indicating that a live communication session available to the three or more participants is active. As shown in FIG. 6AC, in response to receiving data indicating that a live communication session is active, device 600A displays user interface 604A, which includes notification 605, live communication participation affordance 607, and rejection affordance. 609 included. In some embodiments, notification 605 includes the characteristics of notification 614 described above (eg, name of participant, number of participants, name of participant that triggered the notification). In some embodiments, device 600A generates audio and/or haptic output in response to receiving data indicating that a live communication session is active. In some embodiments, device 600A produces audio and/or haptic outputs that are different than those produced by device 600 in response to receiving data indicating that a live communication session is active. Generate. In some embodiments, in response to receiving initial data indicating that a live communication session is active, a notification (e.g., display, audio and/or haptic output) indicating that a live communication session is active is ( For example, notification may be provided at both device 600 and device 600A in response to receipt of reminder data provided at device 600 (and not at device 600A) indicating a reminder that a live communication session is active. In some embodiments, a notification (e.g., visual, audio and/or haptic output) indicating that a live communication session is active is provided to the device in response to receiving initial data indicating that a live communication session is active. Notification is provided at device 600 (eg, but not at device 600A) in response to receipt of reminder data provided at both 600 and device 600A and indicating a reminder that a live communication session is active.

In some embodiments, device 600A receives user input corresponding to selection of live communication participation affordance 607 (eg, a tap on live communication affordance 617). In response to receiving the selection of live communication participation affordance 607, device 600A causes device 600 and/or device 600A to participate in a live communication session or initiate the process of participating in a live communication session. In some embodiments, in response to receiving a selection of live communication affordance 607, device 600A directs live communication user interface 628 (FIG. 6G), 654 (FIG. 6K), or 656 (FIG. 6L) to device 600. to display.

FIG. 6AD shows user interface 604A displayed on device 600A after participating in a live communication session. In some embodiments, device 600A includes a microphone for users to provide audio input to live communication sessions via device 600A. 6AD, user interface 604A includes volume controller 611, live communication session participant indication 613 (“PABLO and 2 others”) and duration (“9 minutes 45 seconds”), call end affordance 615, audio on/ Includes Off Affordance 617 and Menu Affordance 619 .

In FIG. 6AE, device 600A receives (eg, detects) user input 650M (eg, an upward swipe gesture starting at or near menu affordance 619). As shown in FIG. 6AF, in response to receiving user input 650M, device 600A modifies display 613 (“Party of 4”) and keypad affordance 621 (eg, using device 600). Display an audio source affordance 623 for participating in a live communication session, and a list 625 of participants (eg, participants who have joined and/or invited participants) in the live communication session. In some embodiments, device 604A displays list 625 in response to receiving user input 650M on user interface 604 of FIG. 6AC (eg, prior to participating in a live communication session). In some embodiments, list 625 includes an indication of the participant's communication status. In some embodiments, list 625 includes reminder affordances that can be selected to send reminder notifications to participants not participating in the live communication session.

7A-7F are flow diagrams illustrating methods according to some embodiments. Method 700 is performed on a device with a display (eg, 100, 300, 500 or 600). Some acts of method 700 are optionally combined, some acts are optionally reordered, and some acts are optionally omitted.

As discussed below, method 700 provides, among other things, an intuitive way of providing interactive notifications (eg, 614, 672, 676) for live communication sessions between multiple participants. This method creates a more efficient human-machine interface by reducing the cognitive burden on users using innovative techniques that provide notification of live communication sessions between multiple participants. For battery-operated computing devices, it is faster and more efficient for users to be able to quickly interact with notifications of live communication sessions between multiple participants, thus saving power and extending battery charge intervals. do.

At block 702, the device displays a messaging user interface (eg, 604) of the messaging application. A messaging user interface (eg, 604) includes a message area (eg, 606). The message area contains multiple messages (eg, 608A-608D) in a message conversation between three or more participants.

At block 704, the device receives first data indicating that a live communication session (eg, an audio communication session or an audio/video communication session) available to the three or more participants is active. In some embodiments, participants in a live communication session are "currently participating" (e.g., providing live media streams to other participants, also referred to as "connected" or "active"), or "Currently Not Joined" (e.g., a participant who has never joined a session, or a participant who was previously in a session but left the session, also known as "not connected" or "invited") has a state of In some embodiments, participants in a live communication session include participants currently participating in a live communication session and participants not currently participating in a live communication session (eg, 648A-648D). When a live communication session is active, participants join (connect to) or leave (disconnect from) the live communication session. When a participant associated with another device joins the live communication session, the device receives media streams corresponding to that participant. When a device user participates in a live communication session, the device transmits live media streams to other participants (eg, participants participating in the live communication session). In some embodiments, a live communication session remains active as long as at least one of the participants is connected (e.g., even if the participant who initiated the live communication session has left the session). .

At block 706, in response to receiving first data indicating that a live communication session between three or more participants is active, the device sends a notification (e.g., 614 ). In some embodiments, the notification that a live communication session is active includes a live communication participation affordance (eg, 614A). In some embodiments, the notification includes an indication of the number of participants in the live communication session (eg, number of active participants in the session, number of participants invited to the session). In some embodiments, the notification is sent to a participant who has taken an action on the display that triggers the presentation of the notification (e.g., a participant who initiated a live communication session, or a user of the device who has (e.g., name, initials, photo, or avatar) of the participant who selected the "recall" button to be reminded. In some embodiments, in response to receiving first data indicating that a live communication session between three or more participants is active, the device outputs a live Generating an audio output (e.g., 618) indicating that the communication session is active, wherein the audio output indicating that the live communication session is active includes audio output associated with incoming notification and audio output associated with message notification. is different. In some embodiments, the audio output indicating that a live communication session is active shares audio characteristics (eg, common quality, tone, rhythm) with both incoming call and message notification audio outputs. Feedback is provided to the user regarding the current state of the live communication session by displaying a notification (eg, 614) and/or generating an audio output (eg, 618) indicating that the live communication session is active. , visual feedback is provided to the user indicating that certain actions related to the notification and/or the live communication session will occur when the user activates the notification (e.g., when the notification includes a join affordance). be. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at block 704, the device displays a notification in a first position of the message area (eg, 606) indicating that the live communication session is active. In some embodiments, the notification that a live communication session is active (eg, 614) includes a live communication participation affordance (eg, 614A).

At block 710, while displaying a notification (e.g., 614) indicating that a live communication session is active, the device indicates that the live communication session between the three or more participants is no longer active. Receive second data.

Optionally, at block 712, the device moves a notification indicating that a live communication session is active from a first location (eg, bottom of message area 606) to a second location (eg, bottom of message area 606). center or top, or top, bottom, right, and/or left of the display (602)). In some embodiments, movement of notifications (eg, 614) occurs in response to notification relocation criteria being met. In some embodiments, the notification relocation criteria are elapse of a predetermined amount of time (e.g., elapses after the notification is displayed), ceasing to display the messaging user interface (e.g., 604), is closed, or a new incoming message occupies the message area (eg, 606) so that the entire notification (eg, 614) or a portion of the notification (eg, 614) is within the message area (eg, 606). contains one or more criteria, such as being hidden in Feedback is provided to the user on the current state of the live communication session by moving the notification indicating that the live communication session is active and, if selected, the notification and/or the specific status associated with the live communication session. Visual feedback is provided to the user to indicate that an action is taking place, reducing the number of inputs required to perform the action. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be. In addition, by reducing the number of inputs required to perform an operation, the operability of the device is further enhanced, making the user-device interface (e.g., providing more accessible inputs to the user and reducing the number of inputs In addition, by allowing the user to use the device more quickly and efficiently, the device uses less power and improves battery life.

Optionally, at blocks 714 and 716, the device receives a first user input (eg, 650C) corresponding to selection of a live communication participation affordance (eg, 614A or 626B). In response to receiving a first user input corresponding to selecting a live communication participation affordance, the device participates in a live communication session (e.g., participates in a live communication session and broadcasts both video and audio from the electronic device). and/or participate in a live communication session and transmit audio only (without transmitting video) from the electronic device). Participating in a live communication in response to receiving a selection of a live communication participation affordance included in the notification reduces the number of inputs required to operate to participate in a live communication session. Device usability is further enhanced by reducing the number of inputs required to perform an operation, making the user-device interface (e.g., providing more accessible input to users and participating in live communication sessions). (by reducing the number of inputs required), and by allowing the user to use the device more quickly and efficiently, the device uses less power and improves battery life. .

Optionally, at blocks 718 and 720, the device sends reminder data (eg, FIG. 6D) indicating a reminder to join the live communication session (eg, a reminder sent by the participant during the conversation or after a predetermined time). receive. In response to receiving reminder data indicating a reminder to participate in the live communication (e.g., the reminder is sent by a participant in the conversation), the device indicates that the live communication session is active by one or more audio output devices of the device. Generate an audio output (eg, 618) (eg, a ringtone output) indicating that the In some embodiments, audio output (eg, 618) is generated in conjunction with redisplaying the notification (eg, 616). In some embodiments, display of the notification is in progress when the reminder data is received. In some embodiments, the audio output indicating that a live communication session is active is different than the audio output associated with incoming call notification and the audio output associated with message notification. In some embodiments, the audio output indicating that a live communication session is active shares audio characteristics (e.g., common tone quality, tone, rhythm) with both incoming call and message notification audio outputs. In some embodiments, the initial audio output indicating that the live communication session is active is replaced by a reminder audio output indicating that the live communication session is active (e.g., the participant has selected the "call" affordance). different (eg, shorter, quieter, or weaker) than the audio output generated based on the In some embodiments, the initial audio output indicating that the live communication session is active is a single repetition of the repeated audio output used for incoming phone calls indicating that the live communication session is active. The reminder audio output is multiple repetitions of the repeated audio output used for incoming phone calls, or the same audio output used for incoming phone calls. In some embodiments, in response to receiving reminder data indicating a reminder to participate in a live communication, the device displays a representation (e.g., name, initials, photo, or avatar) of the participant in the live communication session that initiated the reminder. ). By generating a reminder that includes an audio output when predetermined conditions are met, users can quickly recognize that a live communication session is available if they miss the visual notification (e.g., 614). become. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at blocks 722 and 724, in embodiments where the notification includes a menu access affordance (eg, 626B), the device receives a second user input (eg, 650C) corresponding to selection of the menu access affordance. . In response to receiving a second user input corresponding to the selection of a menu access affordance, the device displays a video join option (eg, 634) (eg, affordance, drop-down menu, checkbox) and an audio-only option (eg, 634). display a menu (eg, 630) containing (eg, affordances, dropdowns, checkboxes) (eg, 642). By displaying a menu containing different options for participating in a live communication session in response to receiving a menu access affordance selection, additional control options are provided without cluttering the UI with additional controllers being displayed. provided. By providing additional control options without cluttering the UI with additional controllers being displayed, the usability of the device is enhanced and the user-device interface (e.g., when operating/interacting with the device (by assisting the user in providing the correct input and reducing user error), as well as reducing the power usage of the device by allowing the user to use the device more quickly and efficiently. is reduced and battery life is improved.

Optionally, at blocks 726 and 728, the device receives a third user input while displaying the menu (eg, 630). Upon receiving a third user input (eg, 650F) corresponding to selecting a join video option (eg, 634), the device joins the live communication session using audio and video transmitted from the electronic device. In some embodiments, the device prompts the user to confirm if they would like to join audio-only and/or video-only (eg, "Would you like to join with video?"). Participating in a live communication session using audio and video transmitted from an electronic device when predetermined conditions are met (e.g., in response to a video participation option being selected) allows the user to Live communication sessions with audio and video can be participated in without having to select or modify additional inputs and outputs (eg, turn on camera and/or turn on microphone). Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at block 730, in response to the third user input corresponding to selection of an audio-only join option (eg, 642), the device causes the electronic device to participate without transmitting video data from the electronic device. Participate in a live communication session with audio transmitted from (eg, audio recorded by one or more of the device's microphones). In some embodiments, the device displays an indicator (eg, 658 in FIG. 6L) that the user has joined the live communication using audio only. This gives the user peace of mind that the video is not being shared). Participating in a live communication session with audio transmitted from the electronic device when predetermined conditions are met allows the user to select or modify additional inputs and outputs of the device (e.g. turn off the camera). , and/or turn on the microphone), and can participate in a live communication session using only audio. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

At block 732, in response to receiving second data indicating that the live communication session between the three or more participants is no longer active, the device ceases displaying the notification (eg, 614); Update notifications to indicate that the live communication session is no longer active. In some embodiments, while the live communication session is active, the notification is updated to indicate the current duration of the live communication session. In some embodiments, when the live communication session is no longer active, the notification ceases displaying the duration of the session and displays a notification that the session has ended (eg, "call ended"). visual indication that certain actions associated with the notification and/or the live communication session are no longer available by ceasing display of the notification upon receipt of data indicating that the live communication session is no longer active; Feedback is provided to the user. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be. In addition, by ceasing display of the notification upon receipt of data indicating that the live communication session is no longer active, the action is automatic without further user input when a set of conditions are met. is performed on Device operability is enhanced by performing optimized behavior without requiring further user input when a set of conditions are met, and the user-device interface (e.g., make it more efficient (by reducing unnecessary elements, assisting the user in providing appropriate input when operating/interacting with the device, and reducing user error); By enabling quick and efficient use, the device uses less power and improves battery life.

Optionally, at blocks 736, 738, and 740, the device receives third data indicating availability of a second live communication session between the two or more participants. In response to the availability of a second live communication session between the two participants, the device sends a first type notification (e.g., 672) indicating that the second live communication is active ( (e.g. full screen notifications). In some embodiments, the first type of notification (eg, 672) is a menu (eg, , 630). In response to a second live communication session being available between the three or more participants, the device sends a second notification, different from the first type, indicating that the second live communication is active. 2 types of notifications (eg, 676) are displayed. In some embodiments, the second type of notification is a banner notification (eg, 676) that is not full screen or persistent full screen. Optionally, at block 742 the device initially displays the second type of notification in a first state (eg, full screen state 672). Optionally, in step 744, the device converts the second type of notification (eg, 672) to a second state (eg, 676) (eg, a persistent non-full-screen state (eg, on top of the display). Persistence banners placed on the edges), prompting the user based on predetermined conditions by displaying certain types of notifications that a live communication session is active based on when predetermined conditions are met. Allows users to quickly become aware that a live communication session is available (e.g., available between two participants) while providing more and/or less distraction reduction. (display a full-screen notification when there is a live communication session available, display a banner when there is a live communication session available between two participants), and when a set of conditions are met, a further Optimized behavior without the need for user input enhances device usability and improves user-device interfaces (e.g., providing appropriate input when operating/interacting with the device). In addition, by enabling users to use the device more quickly and efficiently, the power usage of the device is reduced and battery life is reduced. Life expectancy is improved.

Optionally, at blocks 746, 748, 750 and 752, the device receives fourth data indicating that a third live communication session is available. In response to receiving fourth data indicating that a third live communication session is available, an active application (e.g., a messaging application, or a third party provided by an application developer other than the developer of the operating system). (e.g., an application in place of the device's lock screen (e.g., 668) or home screen (e.g., 1400)), such as a user application, is displayed, the device performs a third Display a notification (eg, 614 or 676) of a third type (eg, a non-full screen state (eg, a banner placed at the top edge of the display)) indicating that a live communication session is available. In response to receiving fourth data indicating that a third live communication session is available, and determining that the active application is not being displayed (eg, FIG. 6W), the device performs the third provides a fourth type of notification (eg, 678 or full screen notification (eg, 616)) indicating that a live communication session is available. Displaying certain types of notifications of live communication sessions based on when predetermined conditions are met may provide more and/or less reduced scrutiny to the user based on the predetermined conditions (e.g., if the device has an application , and a banner when the device is not displaying the application), while making the user quickly aware that a live communication session is available. enable. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at blocks 754, 756, 758, and 760, the device receives fifth data indicating that a fourth live communication session is available. In response to receiving fifth data indicating that a fourth live communication session is available, the current state of the electronic device (e.g., the current operating mode of the device (e.g., the device is do-not- disturb, DND) mode, a specific application or hardware is operationally active, the device is locked or unlocked, or the user is looking at the display. sensors detect) meets the alert notification criteria, the device provides a non-visual notification (eg, 678) (eg, audio or haptic output) while displaying a notification (e.g., 676) that a fourth live communication session is available.In some embodiments, the output or alert provides a notification (e.g., , 614), in response to receiving fifth data indicating that a fourth live communication session is available, and receiving a determination that the current status of the electronic device does not meet the alert notification criteria. , the device provides a notification that a fourth live communication session is available (e.g., 614).In some embodiments, the first type of alert is not announced, while the second type of alert (e.g., a visual alert such as a banner notification (e.g., 676)) is provided. In some embodiments, the relevant status of the device is whether the device is in a locked state or an unlocked state, hi some embodiments, when the device is locked, the first type is announced (e.g. audio or haptic output), and if the device is not locked (e.g. the user is actively manipulating/interacting with the device), no alert is announced ( In some embodiments, the relevant context of the device indicates that the user is looking at the display (e.g., via one or more sensors (e.g., camera, infrared sensor)). In some embodiments, when the device is locked, a first type of alert (e.g., audio or haptic output) is announced.In some embodiments, the user A first type of alert (eg, 682) is generated when the device detects that the is not looking at the display (eg, FIG. 6AA). In some embodiments, the first type of alert is not generated (eg, suppressed) when the user is actively manipulating/interacting with the device (eg, FIG. 6AB).

In some embodiments, the alert notification criteria include a requirement that the device be locked for the alert notification criteria to be met. In some embodiments, the alert notification criteria include a requirement that the device not detect user attention to the device for the alert notification criteria to be met. (eg, based on user input detected by an input device of the electronic device, based on the user's gaze (eg, 680) detected by eye tracking, face tracking, etc.). Determining whether to display a notification (e.g., 614) and provide a non-visual notification based on when predetermined conditions are met while providing more and/or less interruptions based on predetermined , allowing the user to quickly become aware that live communication is available. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

It should be noted that the processing details described above with respect to method 700 (eg, FIGS. 7A-7F) are also applicable in an analogous manner to the methods described below. For example, method 700 optionally includes one or more of the various method features described below with respect to methods 900 , 1100 , 1300 , and 1500 . For example, method 700 uses methods 1300 and 1500 to initiate a live communication session, and initiation and execution of a live communication session can include methods 900 and 1100 . For the sake of brevity, these details are not repeated below.

8A-8BK illustrate exemplary user interfaces for a live communication session, according to some embodiments. The user interfaces of these figures are used to illustrate the processes described below, including the processes of Figures 9A-9K.

FIG. 8A shows a device 600 displaying a messaging user interface 604 for a message conversation between groups The Dream Team. Live communication sessions between group participants are not active (eg, as indicated by the absence of live communication affordances 610). As shown in FIG. 8A, the header of messaging user interface 604 is expanded to show video join affordance 626B (as described above). As shown in FIG. 8A, device 600 responds to user input 850A (e.g., on video join affordance 626B) to display a user interface for a live communication session between participants of participant group The Dream Team. tap). As shown in Figure 8A, the request includes a request to initiate a new live video communication session. In some embodiments, the request includes a request to join an existing live communication session (eg, as described above with respect to Figure 6F).

In response to user input 650A, device 600 displays message 800 in message area 606 and display 802 indicating that a live communication session is being initiated. In some embodiments, message 800 includes some of the aforementioned characteristics of notification 614 . When a live communication session is initiated, device 600 displays live communication user interface 804, which includes representations of multiple participants in the live communication session simultaneously.

Upon determining that the number of participants in the live communication session is greater than a threshold number of participants (eg, more than 5 participants including the user of device 600), device 600 Simultaneously display representations of the first plurality of participants other than at one or more sizes (e.g., three different sizes) larger than their respective threshold sizes, while not included in the first plurality of participants Displaying representations of a second plurality of participants including one or more participants in the overflow region, wherein representations of the one or more other participants displayed in the overflow region are of sizes smaller than their respective threshold sizes. is displayed.

In the embodiment shown in FIGS. 8A-8BK, the threshold number of participants is five participants (including the user of device 600). Since the group The Dream Team includes 15 participants, device 600 determines that the number of participants in the live communication session is greater than the threshold number of participants. As shown in FIG. 8C, in response to this determination, live communication user interface 804 displays four primary representations 810A-810D of each of the four participants other than the user of device 600 ( collectively 810), a user representation 806 (corresponding to the user of device 600) of the view of the camera of device 600, and the user of device 600 and the participants other than those represented in canvas area 811. , and an overflow area 820 with overflow representations 822A-822J (collectively 822) corresponding to participants in the live communication session, a subset of which (eg, 822A-822D) is always displayed. In some embodiments, the layout is one of a plurality of predetermined layouts in which representations of participants are fixed at different positions of user interface 804 . FIG. 8AW shows another embodiment of user interface 804 . In the embodiment shown in FIG. 8AW, overflow area 820 includes overflow representations of all participants other than the user of device 600 in the live communication session, and the participants represented in canvas area 811 are also represented in overflow area 820.

If the number of group participants is 5 or less, upon determining that the number of participants in the live communication session is 5 or less, device 600 participates without displaying an overflow representation in the overflow area. The primary representation of the person and the representation of the user of the device 600 are displayed.

Each primary representation 810 is displayed in one of three predetermined representation sizes (small, medium, and large). Primary representation 810A is displayed in small size, primary representation 810B is displayed in large size, and primary representation 810C and primary representation 810D are displayed in medium size. In FIG. 8C, each primary representation 810 overlaps at least one other primary representation. Overflow representations 822 are all displayed at the same size, which is smaller than the small representation size associated with main representation 810 . In the illustrated embodiment, each threshold size described above is larger than the size of the overflow representation 822 and smaller than the small representation size of the primary representation 810 . User representation 806 is the same size as overflow representation 822 and may be aligned with or displayed within overflow region 820 .

As shown in Figure 8C, the live communication session user interface 804 displays three full overflow representations and one partial overflow representation at a time. The rest of the overflow representation is hidden and scrolls through the overflow area 820 (eg, in response to either user input or detected events such as participants actively participating by speaking or moving). can be displayed by In FIG. 8C, overflow representation 820D is partially shaded to indicate that there are more overflow representations and/or that overflow region 820 can be scrolled. In some embodiments, the order of overflow representations 822 within overflow region 820 depends on the order in which participants join the live communication session. In some embodiments, the order of overflow representations 822 within overflow region 820 changes over time (e.g., as participants join and leave a live communication session, or as described in more detail below). (eg, when participants enter or leave canvas area 811, either by user selection of an overflow representation or by active participation by participants represented in the overflow area). In some embodiments, the order of overflow expressions within overflow region 820 is fixed (eg, the order in which the overflow expressions are displayed does not change over time). In some embodiments, the order of the overflow representation is based on the order of the list associated with the group (eg, the order in which participants were added to the group when the group was created). In some embodiments, the display of primary representation 810 (eg, a live video feed) is updated at a higher rate than the display of overflow representation 822 .

FIG. 8C shows the live communication session interface 804 immediately after starting the live communication session and before any participant (other than the participant associated with device 600) joins the live communication session. As shown in FIG. 8C, main representation 810 and overflow representation 822 contain placeholders for participants who are not connected to the live communication session. As shown in FIG. 8C, the placeholders include a representation of the participant's name (eg, first name or initials of first and last name) and representation with colored objects. In some embodiments, the color and/or pattern of each colored object is different or selected from a predetermined set of colors and/or patterns. Optionally, the placeholder has a status indicator indicating that the participant is waiting to be connected to a live communication session and a reminder to send a reminder to the corresponding participant that the live communication session is available. Affordances (“outgoing”). In some embodiments, the reminder affordance is not initially included in the placeholder and is displayed after a predetermined amount of time upon determination that the corresponding participant has not participated in the live communication session.

As shown in FIG. 8C, device 600 receives (eg, detects) user input 850A (eg, tap) corresponding to selection of key representation 810C. As shown in FIG. 8D, in response to receiving user input 850A, device 600 responds with additional identifying information (eg, participant's name, Marcel) and a reminder that a live communication session is available. Modify the key representation 810C to include a reminder affordance 808A to send to the participant who wants to participate.

FIG. 8D also shows the participant corresponding to key representation 810B participating in a live communication session. In some embodiments, device 600 receives data indicating that the participant corresponding to key representation 822B has joined the live communication session. As shown in FIG. 8D, in response to the participant corresponding to key representation 822B joining the live communication session, device 600 replaces the placeholders of key representation 822B with live video data from the corresponding participant. . In some embodiments (eg, FIG. 8AX), device 600 replaces overflow representation 822B corresponding to the participant with live video data from the participant. As used herein, the phrase "live video data from a participant" (or its equivalent) refers to a device associated with the participant (e.g., smart phone, laptop computer, desktop computer or tablet computer) or Includes live video data transmitted by devices associated with accounts associated with participants (eg, email accounts or cloud storage accounts).

As shown in FIG. 8D, device 600 receives (eg, detects) user input 850B (eg, tap) corresponding to selection of reminder affordance 808 . In response to input 850B, device 600 sends an instruction to cause a new notification (e.g., call, ringtone, etc.) to occur in a device associated with the corresponding participant indicating that a live communication session is available; Here, the new notification is different from previous notifications sent to participants. In some embodiments, the new notification is more intrusive than the previous notification (e.g., audio transmission, haptic output, or full-screen notification compared to banner notification without audio or haptic output). ). In FIG. 8D, device 600 also displays (eg, reminders sent by another participant currently connected to a live communication session, such as the participant represented by main representation 810B, or previously sent by the user of device 600). A reminder affordance 808B is also displayed on the primary representation 810D to indicate that the participant corresponding to the primary representation 810D is currently being reminded that a live communication session is available (such as by a reminder sent). . Device 600 also modifies reminder affordance 808A to indicate that the corresponding participant has been notified (eg, by replacing "calling" with "calling" as shown in FIG. 8E). .

As shown in FIG. 8E, upon determining that a predetermined amount of time has passed since the live communication session began, device 600 ceases displaying overflow representation 822 and increases the size of user representation 806 . In some embodiments, main representation 810 is resized and/or moved when overflow representation 822 is removed. In the illustrated embodiment, the primary representation 810D is moved down. In some embodiments, the device 600 has no more than a threshold number of participants (eg, 5 participants including the user of the device 600) currently participating in the live communication session. In response to the determination, the display of the overflow expression 822 is stopped.

At the time corresponding to FIG. 8E, two participants are participating in the live communication session. In some embodiments, after an additional amount of time, device 600 ceases displaying one of primary representations 810 and changes the layout of live communication user interface 804 to a three-to-one layout. In some embodiments, a 4:1 layout and a 3:1 layout are included in a set of predetermined layouts that have similar relative placements of primary representations 810 . In some embodiments, having relatively similar placement of primary representations includes placement where primary representations remain in similar positions on user display 602 when there is a transition between layouts within a set. (eg, the positions of representations in one layout overlap with the positions of representations in another layout of the same set).

In some embodiments, the device 600 determines whether the corresponding participant is participating in a live communication session (eg, if the corresponding participant is participating, the representation is not deleted) and whether the participant is reminded. (e.g., the representation of the participant being reminded is not removed for a predetermined amount of time after selecting the corresponding reminder affordance), or the placement of the participant in the participant list (e.g., the previous representations of participants later in the list are removed before representations of later participants in the list), and placement of representations on live communication user interface 804 (e.g., representations closer to the top of display 602 are displayed 602 above lower expressions are removed before) to determine which primary expressions to remove. As shown in FIG. 8F, device 600 stops displaying primary representation 810A and moves primary representation 810B up (eg, so that representations 810B and 810C overlap less), while remaining representations are displayed. (eg representation 810B is larger than representation 810C and above representation 810C, representation 810C is above representation 810D and is the same size as representation 810D). FIG. 8F also shows that reminder affordance 808A is modified to "repeat," indicating that the corresponding participant was reminded at least once and is not currently being reminded. Reminder affordance 808B is also displayed (e.g., from "calling" to "calling" and in a different color) to indicate that the corresponding participant was previously reminded, but not by the user of device 600. Fixed.

At the time corresponding to Figure 8F, the live communication session still has only two participants. In some embodiments, after a further period of time, the device 600 ceases displaying another primary representation 810 and changes the layout of the live communication user interface 804 (eg, 4:1 and 2:1 layout (in the same set of predefined layouts as the 3:1 layout). As shown in FIG. 8G, device 600 stops displaying primary representation 810D and expands primary representations 810B and 810C. FIG. 8G shows a three-participant (eg, two-to-one) layout in which the key representations of the two participants other than the user of device 600 are equally sized and placed one above the other. The top-down order of primary representations 810B and 810C is maintained, with primary representation 810B above primary representation 810C. In the layout of FIG. 8F, there is no overlap between primary representations, and user representation 806 overlaps primary representation 810C.

At the time corresponding to FIG. 8G, the live communication session still has only two participants. In some embodiments, after a further period of time, device 600 ceases displaying another primary representation and changes the layout of live communication user interface 804 to a one-to-one layout. As shown in FIG. 8H, device 600 stops displaying primary representation 810C and expands primary representation 810B to full screen size. In some embodiments, device 600 changes from the layout of FIG. 8G to the layout of FIG. 8H at a predetermined time after selection of reminder affordance 808A. FIG. 8H is a two-participant (eg, one-to-one) layout in which the remaining primary representations 810B are displayed in full screen and the user representations 806 are overlaid. As shown in Figure 8H, the remaining primary representation 810B and user representation 806 are square shaped primary representation 810 and user representation 806 in the 4:1, 3:1, and 2:1 layouts shown in Figures 8E-8G, respectively. is a tall rectangle (eg, in profile orientation) compared to .

Turning to FIG. 8I, a third participant Stephen is participating in a live communication session. As shown in FIG. 8I, in response to participant Stephen joining the live communication session, device 600 changes from the display of the one-to-one layout of FIG. Transition to a two-to-one layout above the existing participant representation 810B (other than the user of device 600). In some embodiments, device 600 displays notification 812 and/or generates audio or haptic output indicating that the participant has joined the live communication session, as shown in FIG. 8I. In some embodiments, a new featured representation is displayed when participant Stephen joins the live communication session, so device 600 assumes the role of the display indicating that the participant has joined the live communication session. Refrain from displaying notification 812 or generating audio or haptic output for representation 810E to serve.

Participant Stephen joins the live communication session in audio-only mode, providing live audio data without video. Participant Stephen's audio-only communication status is indicated by main representation 810E and notification 812. FIG. A primary representation 810E indicates the audio-only communication status by displaying the participant's name and the avatar associated with the participant. Notification 812 indicates the communication status in text (“You have joined this call with audio”). As shown in FIG. 8J, device 600 ceases displaying notification 812 (eg, after a predetermined amount of time after participant Stephen joins the live communication session).

In some embodiments, in response to receiving data indicating that an audio-only participant is actively participating (e.g., speaking), device 600 responds to display a visual indication in the representation of In FIG. 8J, a representative image (eg, an avatar) within key expression 810E is animated (eg, in a “voice circle”) to show participant Stephen speaking.

Turning to Figure 8K, a fourth participant, Marcel, is participating in a live communication session. As shown in FIG. 8K, in response to a fourth participant joining the live communication session, the device 600 changes from the display of the two-to-one layout to the primary representation 810F of the most recently joined participant (device Transition to a 3:1 layout (eg, as shown in FIG. 8F) above existing participant primary representations 810B and 810E (other than 600 users).

Participant Marcel joins the live communication session in video mode and provides live video and audio data. Participant Marcel's video communication status is indicated by key representation 810F. The main representation 810F shows the video communication status by displaying the live video stream. As shown in FIG. 8K, device 600 does not immediately display a notification that participant Marcel has joined the live communication session.

Turning to FIG. 8L, a fifth participant Hoan is participating in a live communication session. As shown in FIG. 8L, in response to participant Hoan joining the live communication session, device 600 selects from the 3-to-1 layout that the most recently joined participant's primary representation 810G is the smallest primary representation. There is a transition to a 4:1 layout, located above and to the right of the main representation 810F. In some embodiments, upon determining that both participants Marcel and Hoan have joined the live communication session within a predetermined grace period, the device 600 may direct participants Marcel and Hoan to participate in the communication session as shown in FIG. displays a combined notification 816 indicating that has joined the live communication session. In some embodiments, combined notification 816 is displayed following a determination that the participants joined in the same communication state (eg, video).

As shown by the live video stream displayed in the main representation 810G and the video camera representation 816A in the notification 816, the participant Hoan may send the video to the device 600 so that the device 600 receives the live video and audio data associated with the participant Hoan. Join a live communication session in mode.

As shown in FIG. 8L, device 600 receives (eg, detects) user input 850C (eg, double-tap) at location of primary representation 810F on display 602. As shown in FIG. In some embodiments, in response to receiving user input 850C, device 600 scales primary representation 810F to a maximum representation size (while maintaining a 4:1 layout), as shown in FIG. 8M. Zoom in and reduce the primary representation 810E to a medium size. As shown in FIG. 8M, device 600 ceases displaying notification 816 (eg, after a predetermined amount of time after participant Hoan joins the live communication session).

In some embodiments, the composition of the layout is dynamically changed in response to receiving data indicating that participants are actively participating in a live communication session. In FIG. 8M, device 600 receives data indicating that participant Hoan corresponding to main expression 810G is actively participating. As shown in FIG. 8M, in response to receiving data indicating that participant Hoan corresponding to primary representation 810G is actively participating, device 600 converts primary representation 810G to a small representation size to medium representation. size, and reduce the primary representation 810B from the medium representation size to the small representation size. In this way, the 4:1 layout maintains one small representation, two medium representations, and one large representation.

As shown in FIG. 8M, device 600 receives (eg, detects) user input 850D (eg, double-tap) at location of primary representation 810G on display 602. As shown in FIG. In some embodiments, in response to receiving user input 850D, device 600 expands primary representation 810G and brings it to the forefront in front of other displayed primary representations, as shown in FIG. 8N. Extrude representation 810G. The background behind the enlarged primary representation 810G is grayed out except for the user representation 806. FIG.

As shown in FIG. 8O, device 600 receives (eg, detects) user input 850E (eg, tap) on cancellation affordance 818 . As shown in FIG. 8P, in response to receiving user input 850E, device 600 reduces the size of primary representation 810G, causing primary representation 810G to be enlarged instead of primary representation 801F reduced to the medium representation size. The display reverts to the 4:1 layout displayed prior to user input 850D, except that it now occupies the primary representation of size.

Turning to Figure 8Q, a sixth participant joins the live communication session. As shown in FIG. 8Q, in response to a sixth participant joining the live communication session and in response to the number of connected participants exceeding a predetermined threshold of five participants. Accordingly, device 600 displays overflow representation 822 A of the sixth participant in overflow region 820 and moves user representation 806 to overflow region 820 such that overflow representation 822 A is to the left of user representation 806 . Overflow representation 822A and user representation 806 are aligned in the center of overflow region 820, as shown in FIG. 8Q. In some embodiments, the device 600 replaces the minimal primary representation (eg, 810B) with a representation of the newly joined participant (eg, the sixth participant) and replaces the overflow representation of the participant. is displayed in overflow region 820 (eg, representation 810B is reduced in size and moved to the position of overflow representation 822A within overflow region 820). In some embodiments, in response to a sixth participant joining the live communication session and in response to the number of connected participants exceeding a predetermined threshold of five participants. 8AY, the device 600 displays an overflow representation of the currently participating participant or all participants (currently participating participant and invited participants) in the overflow area, for example, as shown in FIG. .

As shown in FIG. 8Q, overflow region 820 and primary representation 810 do not overlap. In some embodiments, device 600 reduces the size of one or more of the primary representations and/or translates them up or down to make room for the overflow region. In some embodiments, the device 600 reduces the size of the medium size primary representation and maintains the size of the small and large size primary representations. In some embodiments, the device 600 reduces the vertical separation and/or increases the vertical overlap between key representations.

Turning to Figure 8R, a seventh participant is participating in a live communication session. In response to the seventh participant joining the live communication session, device 600 adds overflow representation 822B of the seventh participant to the left of overflow representation 822A in overflow area 820 . As shown in FIG. 8R, overflow representation 822A and user representation 806 are shifted to the right (compared to FIG. 8Q) so that overflow representations 822A and 822B and user representation 806 are centered in overflow region 820. FIG. In some embodiments, a seventh participant's overflow representation 822B is added to the right of overflow representation 822A, between overflow representation 822A and user representation 806 . In the embodiment shown in FIG. 8AZ, in response to a seventh participant joining the live communication session, device 600 replaces the minimal key representation (eg, 810H in FIG. 8AY) with key representation 810I, and participant Change the overflow representation 822F corresponding to Pablo from a placeholder to a live video stream.

As shown in FIG. 8S, an eighth participant has joined the live communication session. In response to the eighth participant joining the live communication session, device 600 adds eighth participant's overflow representation 822C to the left of overflow representation 822B in overflow region 820 . As shown in FIG. 8S, overflow representations 822A and 822B and user representation 806 are aligned to the right (compared to FIG. 8Q) such that overflow representations 822A, 822B and 822C and user representation 806 are centered in overflow region 820. be moved.

Turning to FIG. 8T, a ninth participant is participating in a live communication session. In response to the ninth participant joining the live communication session, device 600 adds overflow representation 822D of the ninth participant to the left of overflow representation 822C in overflow area 820 . As shown in FIG. 8T, overflow representations 822D, 822C, and 822B and user representation 806 are fully displayed. Overflow representation 822A is partially displayed, overlaid by user representation 806 on the right side. Overflow representation 822A is shaded (eg, by a shading effect) where overflow representation 822A meets user representation 806 . In the illustrated embodiment, the sizes of overflow representation 822 and user representation 806 remain the same as when the seventh, eighth, and ninth participants joined the live communication session. FIG. 8T illustrates device 600 displaying combined notification 824 indicating that participants Mary, Mick, John, and Tom (represented by overflow representations 822A, 822B, 822C, and 822D, respectively) have joined the live communication. show. In some embodiments, as additional participants join the live communication session (eg, beyond the ninth participant), existing overflow representations shift right within overflow region 820 to create new overflows. Making room for representations, the user representation 806 remains fully displayed on the right side of the overflow area 820, and the newly joined participant's overflow representation is added to the left edge of the existing overflow representation (e.g., the existing The overflow representation is scrolled to the right, "below" the user representation 806). In some embodiments, the newly joined participant's overflow representation is added to the right edge of the existing overflow representation while the user representation 806 remains fully displayed to the right of the overflow area 820 . In some embodiments, representations of newly joined participants that are added to the right edge of existing overflow representations are not immediately displayed when the overflow region is fully occupied (e.g., new overflow representations are displayed as you scroll the representation to the left).

Turning to FIG. 8U, device 600 receives data indicating that the participant corresponding to overflow representation 822B is actively participating in the live communication session. In some embodiments, the device 600 responds to determining that the actively participating participant's activity level has risen above the participant's activity level represented by the primary representation. is replaced by the representation of the actively participating participant. In the illustrated embodiment, in response to receiving data indicating that the participant corresponding to overflow representation 822B is actively participating in the live communication session, device 600 converts the minimal primary representation into active participation. and display the participant's overflow expression corresponding to the replaced main expression. As shown in FIG. 8U, the primary representation 810B (the smallest primary representation) begins to shade or fade, and the overflow representations 822B of the actively participating participants begin to shade or fade out and increase in size. Overflow representations 822C and 822D enter overflow region 820 from the left edge of display 602 to the left of overflow representation 822D, as new overflow representation 822E, which is scaled down and corresponds to the participant of primary representation 810B (the replaced primary representation), enters overflow region 820. is shifted right. In some embodiments, the main representation 810B is slightly enlarged (eg, to emphasize that it is being replaced). As shown in FIGS. 8V-8W, the primary representation 810B is replaced by the active participant's primary representation 810H, the overflow representation 822B continues to shrink until it is removed, and the overflow representation 822E completely collapses. Overflow expressions 822C and 822D continue to shift right until overflow expression 822C occupies the position of overflow region 820 previously occupied by displayed, deleted overflow expression 822B. 8U-8W, overflow representation 822A and user representation 806 remain in the same position.

In some embodiments, device 600 visually distinguishes user representation 806 in response to the active participation of the user of device 600 . In the embodiment shown in FIG. AZ, device 600 displays a thick border around user representation 806 and highlights initials bars below user representation 806 .

Turning to FIG. 8X, device 600 is rotated from a portrait (vertical) orientation to a landscape (horizontal) orientation. As shown in FIG. 8X, as device 600 is rotated, device 600 maintains representations within respective regions of display 602 that were occupied prior to rotation of device 600 (e.g., user representations). and each primary and overflow representation occupy the same area on display 602 before and after rotation of device 600). The images in the representation are rotated so that the participant is upright with respect to the landscape orientation. In some embodiments, the device 600 displays an animated rotation of the primary representation or an image within the primary representation (eg, 90 degrees opposite the direction of rotation of the device 600). In some embodiments where the overflow representations are not square, when the device 600 is rotated, the participant's video data is cropped and/or cropped to include (eg, center) the user's face in each overflow representation. are moved in parallel. In some embodiments where the primary representations are not square, as the device 600 is rotated, the participant's video data is cropped and/or translated to center the user's face on the respective primary representation. In some embodiments, the participant's video data may be captured whether the device 600 is rotated (e.g., when the received video data has a different aspect ratio or format than the displayed representation). cropped to include the user's face.

As shown in FIG. 8X, the participant corresponding to key expression 810H begins actively participating. In response, device 600 enlarges primary representation 810H and reduces the size of primary representation 810E, as shown in FIG. 8Y. In the embodiment shown in FIGS. 8BA-8BB, device 600 visually distinguishes overflow representations 822F corresponding to active participants (eg, highlighted initials bars).

In some embodiments, in response to a participant joining a live communication session, device 600 replaces the existing primary representation with the primary representation of the newly joined participant and places the replaced participant in the overflow region. Go to 820. Turning to Figure 8Z, participant Pablo is participating in a live communication session. In response, device 600 replaces minimal primary representation 810E (Stephen) with primary representation 810I for newly joined participant Pablo, and displays overflow representation 822F (Stephen) in overflow area 820 . Overflow representations 822A, 822C, 822D, and 822E are scrolled (eg, upwards depending on the orientation of device 600 shown in FIG. 8Z) to make room for overflow representation 822F. Overflow representation 822A is no longer visible and overflow representation 822C is partially obscured by user representation 806, as shown in FIG. 8Z.

Turning to FIG. 8AA, device 600 is rotated from a landscape (horizontal) orientation to a portrait (vertical) orientation. In response, the device 600 rotates the images of the participants in the representations while maintaining the area on the display occupied by each representation (e.g., the device 600 rotates the device from portrait orientation to landscape orientation). flips the rotation done in response to being rotated in the direction of ).

In FIG. 8AB, the participant corresponding to key expression 810G has left (disconnected) the live communication session. Optionally, in response, device 600 displays notification 828 indicating that the participant has left the live communication session, as shown in the embodiment shown in FIG. 8BC. As the number of participants remains greater than five, device 600 maintains a four-to-one layout with display of overflow area 820 . 8AC, primary representation 810G is replaced by primary representation 810J of participant Stephen with the next highest activity level, and overflow representation 822F corresponding to participant Stephen is removed from overflow region 820. FIG. When the overflow representation 822F is deleted, the remaining overflow representations 822A, 822C, 822D, 822E are scrolled to the left. In the embodiment shown in FIG. 8BD, participant Hoan's overflow representation 822D displays a placeholder indicating that participant Hoan is not connected and a reminder affordance to request participant Hoan to rejoin. is updated to

In some embodiments, when the number of participants in the live communication decreases from 6 to 5 when participants leave the live communication session, device 600 stops displaying overflow area 820 and optionally Typically, increase the size and/or vertical spacing of one or more of the primary representations.

In FIG. 8AC, device 600 receives (eg, detects) user input 850F (eg, double-tap) corresponding to participant Allison's selection of overflow representation 822E. As shown in FIG. 8AD, in response to receiving user input 850F, device 600 displays enlarged representation 830 of participant Allison in front of the currently displayed primary representation 810 in a 4:1 layout. The display of expanded representation 830 is similar to the display of expanded representation 810G of FIG. 8N, except that expanded representation 830 is not pushed forward from the main representation of the 4:1 layout.

As shown in FIG. 8AD, device 600 receives (eg, detects) user input 850G (eg, a tap) corresponding to selection of closing affordance 818 . As shown in FIGS. 8AE-8AF, in response to receiving user input 850F, device 600 prompts participant Stephen (eg, the participant with the lowest activity state among the participants with the current primary representation). 810J for participant Allison is displayed. The main representation 810K (at least initially) includes a ribbon (or band) 842A along the lower portion of the rim, which includes the participant's name and a label for expanding the representation (eg, as shown in FIG. 8AD). Including affordances. Figures 8AE-8AF show an exemplary animation of replacing primary representation 810J with primary representation 810K. The primary representation 810K is first displayed at a small representation size (similar to the primary representation 810J in FIG. 8AC) and then expanded to a medium representation size, while the primary representation 810H is reduced from medium to small. . Overflow representation 822E corresponding to participant Allison decreases in size and/or fades out, while overflow representation 822F of the replaced participant translates into overflow region 820. FIG. In some embodiments, ceasing to display primary representation 810J and displaying primary representation 810K includes tinting that creates a cross-fade and/or "flash" effect from primary representation 810J to primary representation 810K. Figures 8BE-8BH show an embodiment in which the main representation 810J is deleted with a flash effect.

In some embodiments, overflow area 820 is scrollable. Referring to the embodiment shown in FIGS. 8BH-BI, the device 600 may display an overflow area 820 in overflow region 820 in response to active participation by a participant whose corresponding overflow representation is not currently being displayed or is not fully displayed. to scroll the overflow representation 822 . Participant overflow representation 822F (corresponding to primary representation 810I) is partially hidden within overflow region 820, as shown in FIG. 8BH. In FIG. 8BH, the participant corresponding to key representation 810I actively participates in the live communication session. As shown in FIG. 8BI, in response to receiving data indicating that the participant corresponding to primary representation 810I is actively participating, device 600 expands primary representation 810I (e.g., The overflow representation 822F is visually indicated by scrolling the overflow representation 822F so that it is fully displayed (near the middle of the displayed portion) and highlighting the initial bar at the bottom portion of the overflow representation 822F.

Turning to FIG. 8AG, device 600 detects contact 850H moved horizontally from right to left over overflow area 820 while still in contact with display 602 . As shown in FIG. 8AH, in response to receiving contact 850H, device 600 is configured such that overflow representation 822F is partially off the left edge of display 602 and user representation 806 no longer overlaps overflow representation 822A. Scroll the overflow representation 822 to the left on the display 602 . ISE, contact 850H and overflow representation 822 move horizontally by the same amount.

While scrolling the overflow area 820, the main representation 810 remains unchanged. Also, the user representation 806 remains within the overflow region 820 .

In the embodiment shown in FIGS. 8BJ-8BK, contact 850H begins on the left side of overflow region 820 and moves horizontally to the right. In response, device 600 scrolls overflow representation 822 to the right on display 602 . 8BJ-8BK, the device 600 removes (eg, stops displaying) the representation from the display 602 when the representation reaches the user representation 806. In FIGS.

Turning to FIG. 8AH, device 600 receives (eg, detects) user input 850I (eg, a tap) at a location on display 602 where primary representation 810 is not displayed. As shown in FIG. 8AI, in response to receiving user input 850I, device 600 displays call affordance 834, effect affordance 836, menu affordance 838, and handle affordance 840 (eg, to exit a live communication session). Display the call control menu 832 including

As shown in FIG. 8AI, the call control menu 832 does not overlap the overflow area 820 and the main representation 810. FIG. To make room for the call control menu 832, the device 600 reduces the size of one or more of the primary representations 810 and/or translates them up or down (eg, reduces the size of the canvas area 811). , moves the overflow region 820 upward on the display 602 . In some embodiments, the device 600 reduces the size of the medium size primary representation and maintains the size of the small and large size primary representations. In some embodiments, the device 600 reduces the vertical separation and/or increases the vertical overlap between key representations.

In response to receiving user input 850I, device 600 also displays ribbons 842A-842D in each of the key representations. Ribbons 842A-842D include corresponding participant names and affordances for expanding the presentation (eg, as shown in FIG. 8O). In some embodiments, the vertical height of ribbons 842A-842D is the same as the amount of vertical overlap between primary representations.

As shown in FIG. 8AJ, device 600 receives (eg, detects) user input 850J (eg, tap) corresponding to selection of menu affordance 838 . In some embodiments, user input 850J includes an upward swipe starting from call control menu 832 (eg, on handle affordance 840). As shown in FIG. 8AK, in response to user input 850J, device 600 displays additional information and controls regarding the live communication session, similar to the additional information and controls described with respect to call control menu 630 of FIG. 6H. Expand the call control menu 832.

As shown in FIG. 8AL, device 600 receives (eg, detects) user input 850K (eg, tap) corresponding to selection of effect affordance 836 . As shown in FIG. 8AM, in response to receiving user input 850K, device 600 displays enlarged image 843 from camera 603 of the user of device 600 and modifies call control menu 832 . In modified call control menu 832, effect affordance 836 is highlighted and effect option affordances 844A-844E are displayed. 8AN, device 600 receives (eg, detects) user input 850L (eg, tap) corresponding to selection of effect option affordance 844A. As shown in FIG. 8AO, in response to receiving user input 850L, device 600 modifies call control menu 832 to include avatar option affordances 846A-846C. In FIG. 8AO, device 600 receives (eg, detects) user input 850M (eg, tap) corresponding to selection of avatar affordance 846C. As shown in FIG. 8AP, in response to receiving user input 850M, device 600 displays effect 843A over the user's face in image 843 and scrolls call control menu 832 to the currently selected Place the affordance 846C of the avatar present in the center of the call control menu 832. 8AP, device 600 receives (eg, detects) user input 850N (eg, tap) on cancellation affordance 848. In FIG. As shown in FIG. 8AQ, in response to receiving user input 850N, device 600 returns call control menu 832 to the configuration of FIG. 8AQ, device 600 receives (eg, detects) user input 850O (eg, a tap) on display 602 outside enlarged image 843 and call control menu 832. In FIG. In response to receiving user input 850O, device 600 ceases displaying enlarged image 843 and displays effect 843A over the user's image in primary representation 806, as shown in FIG. 8AR. When effect 843A is activated, the user's live video stream of device 600 includes effect 843A, as in FIG. 8AR.

As shown in FIG. 8AR, call control menu 832 includes media affordances 844E. Device 600 receives (eg, detects) user input 850P (eg, a tap) corresponding to selection of media affordance 844E. As shown in FIG. 8AS, in response to receiving user input 850P, device 600 modifies call control menu 832 to include media option affordances 852A-852D. 8AS, device 600 receives (eg, detects) user input 850Q (eg, tap) corresponding to selection of media option affordance 852A (sketch). In response to receiving user input 850Q, device 600 displays in canvas area 811 a representation of the media content associated with media option affordance 852A.

As shown in FIG. 8AT, in response to receiving user input 850Q, device 600 replaces primary representation 810K (eg, minimal primary representation) with representation 810L of the selected media item. 8AT, device 600 receives (eg, detects) user input 850R (eg, a downward swipe on call control menu 832) corresponding to a request to dismiss call control menu 832. In FIG. In response to receiving user input 850R, device 600 ceases to display call control menu 832, as shown in FIG. 8AU.

In some embodiments, selection of media option affordance 852A causes related media content to be shared with other participants in the live communication session (e.g., displayed in a canvas area on the display of another participant's device). ). In some embodiments, participants in a live communication session can interact with shared media content (eg, through a series of one or more inputs in the user interface of the live communication application). FIG. 8AV shows an embodiment in which an additional feature 810L-1 has been added to the media content of representation 810L. In some embodiments, in response to a participant's interaction with the media content associated with representation 801L, device 600 may (e.g., select after the primary representation has been expanded or have the participant corresponding to the representation active) expand representation 801L). As shown in FIG. 8AV, the representation 810L is expanded (compared to FIG. 8AT) in response to the participant's interaction with the relevant media content. FIG. 8AV also shows a representation 810M of other content shared in the live communication session (eg, a game between two participants).

9A-9K are flow diagrams illustrating methods according to some embodiments. Method 900 is performed on a device with a display (eg, 100, 300, 500 or 600). Some acts of method 900 are optionally combined, some acts are optionally reordered, and some acts are optionally omitted.

As discussed below, method 900 provides, among other things, an intuitive way of providing an adaptive and intelligent live communication user interface. Additionally, among other things, method 900 provides an intuitive way for a user to interact with a live communication user interface. This method creates a more efficient human-machine interface by reducing the user's cognitive burden by providing an adaptive and intelligent live communication interface based on the number of participants in the live communication. For battery-operated computing devices, enabling a user to view a live communication user interface based on a predetermined number of participants in the live communication is faster and more efficient in power saving. , the battery charging interval becomes longer.

At block 901, a device receives a request (eg, 850A) to display a user interface for a live communication session between two or more participants.

At block 902, in response to receiving a request to display a user interface for a live communication session between two or more participants, the device displays representations of the multiple participants (e.g., 810A) in the live communication session. 810D), displaying a live communication user interface (eg, 804) and displaying representations of multiple participants in a live communication session includes blocks 902 and 903. In some embodiments, the live communication session user interface may be a first participant (e.g., a participant other than the user of the electronic device) who has been invited to the live communication session but has not yet joined the live communication session. , participants who have not yet joined the communication session and have been invited to join the communication session)) (e.g., 810A in FIG. 8C) (e.g., a graphical placeholder (e.g., images or virtual avatars), placeholders for text (eg names or initials)). In some embodiments, if participants are sharing a live media stream that includes a live video stream, the representation includes an image of the live video stream (eg, 810B in FIG. 8D). In some embodiments, when participants are sharing an audio-only live video stream, the representation includes each participant's avatar (eg, 810B in FIG. 8I). In some embodiments, the representation of multiple participants in a live communication session can be an audio-only state (e.g., participants are communicating using audio only), a video state (e.g., participants are communicating with video and audio). ), video paused state (e.g. participant's video is paused), video undecoded state (e.g. due to delay issues, video stream format issues, etc.). Participant's video stream cannot be processed), Leaved state (e.g., participant has left a live communication session), and Waiting to Join state (e.g., participant was invited to a live communication session but not yet joined).

At block 903, upon determining that the number of participants in the live communication session is greater than the threshold number of participants, the device generates a representation (e.g., 810A) of a first plurality of participants other than the user of the electronic device. 810D) are simultaneously displayed at one or more sizes greater than their respective threshold sizes, while a representation of a second plurality of participants (eg, 822) is displayed in an overflow region (eg, 820) and a second The two plurality of participants includes one or more participants not included in the first plurality of participants, and representations of the one or more other participants displayed in the overflow region are each threshold size displayed in a smaller size. In some embodiments, the currently active participant (e.g., 810E in FIG. 8J) (e.g., most active participant, participant speaking or moving to meet a criteria set of active participants) person) is displayed as part of the first plurality of participants. In some embodiments, the currently active participants are displayed in the main area (eg, 811) of the user interface (eg, the area where the first plurality of participants are displayed). In some embodiments, if a participant not currently displayed in the main area (e.g., the participant corresponding to 822B in FIG. 8U) becomes the currently active participant, that participant's representation (e.g., 8V 810H) is displayed in the main area. In some embodiments, representations of the first plurality of participants (eg, 810) are displayed in two or more different sizes (eg, more active participants or more recently active participants Inactive participants or more recently inactive participants are displayed at a larger size), and a representation of a second plurality of participants (eg, 822) within the overflow region is displayed at the same size.

In some embodiments, the overflow region (eg, 820) further includes representations (eg, 810A-810D) of the first plurality of participants (eg, the participants displayed in the main region of the user interface). The representation is also displayed in the overflow area) (eg FIG. 8AX). In some embodiments, the overflow area (e.g., 820) is a placeholder representation (e.g., 822A in FIG. 8AW) for live communication session invitees who are not currently connected to the live communication session (e.g., avatars, name, initials). Representation of participants in a live communication session by displaying them differently based on whether predetermined conditions are met (e.g., based on whether the live communication session is greater than a certain number of participants) more easily or more efficiently. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at blocks 904 and 905, the device displays (eg, arranges, multiple positions, sizes, locations, visual representations of, representations) of the first plurality of participants (eg, 810A-810D). , visual content, and/or video feed) are updated at a first rate (e.g., representations of the first plurality of participants are updated at a typical video refresh rate such as 24, 30, or 60 frames per second or higher). is a live video feed updated at a higher frame rate than ), display of representations of a second plurality of participants (e.g. 822) in the overflow region (e.g. placement of representations, multiple positions, sizes, positions , visual display of the presentation, visual content of the presentation, and/or video feed) at a second rate, the second rate being lower than the first rate. ) (e.g., thumbnails in the overflow region update over time, but at a rate lower than the video refresh rate of representations of multiple participants in the canvas region 811, such as 1, 5, or 10 frames per second ). By updating the display of the set of representations at different rates, the device is informed about the current state of the representations in live communication while conserving processing power to provide the set of representations at a faster rate than the other sets of representations. Allows feedback to be provided to the user, thereby proving to the user the ability to view a higher priority set of representations at a higher rate. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at block 906, the device displays a representation of a second plurality of participants (eg, 822) in an overflow area (eg, 820) while displaying the electronic device's camera view (eg, self view). A representation (eg, 806) of the camera's view) is displayed in the overflow region. Displaying a representation of the electronic device's camera view provides the user with visual feedback regarding the current state of the user's camera view (e.g., what the user is showing to other participants in the live communication session). be. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at blocks 907, 908, 909, and 910, the device detects input (eg, 850H) corresponding to the overflow region (eg, horizontal finger drag). In response to detecting input corresponding to the overflow region, the device stops displaying (eg, hides) a subset of representations of the second plurality of participants (eg, 822G and 822H in FIGS. 8BJ-8BK). ), scrolling the overflow region (eg, receiving input to scroll representations of the second plurality of participants in the first direction). While scrolling the overflow area, the device maintains a representation of the view of the electronic device's camera (eg, 806) within the overflow area (eg, FIGS. 8BJ-BK). In some embodiments, scrolling the overflow region displays an animation of one or more representations of the second plurality of participants that appear to scroll below representations of the device's camera view. including. Maintaining the representation of the electronic device's camera view while scrolling the overflow area allows the user the ability to scroll representations of the participants within the overflow area while allowing the user to view the view of the electronic device's camera. Continuous feedback is provided to the user regarding the current state of the (eg, what the user is showing to other participants in the live communication session). Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at blocks 911, 912, and 913, an input (e.g., 850F ) (eg, a tap gesture corresponding to an expression within the overflow region 820 or a gesture having a characteristic intensity exceeding an intensity threshold corresponding to an expression within the overflow region 820). In response to detecting input corresponding to selection of a representation of the second plurality of participant representations within the overflow region, the device selects a representation of the first plurality of participant representations (e.g., 810J in FIG. 8AC) with the representation of the participant corresponding to the selected representation in the overflow region (e.g., 810K in FIG. 8AE) (e.g., replace the least recently active participant in the main display). . In some embodiments, replacing the display of one of the representations of the first plurality of participants with the representation of the participant corresponding to the selected representation within the overflow region comprises the representation of the representation of the participant within the overflow region. discontinuing display of selected representations of the representations of the two plurality of participants. In some embodiments, ceasing to display the selected representation within the overflow region includes an animation of the selected representation shrinking. In some embodiments, other representations within the overflow region are translated to fill the gap in which the selected representation was displayed. In some embodiments, replacing the display of one representation of the first plurality of participant representations with the representation of the participant corresponding to the selected representation in the overflow region includes: After displaying the corresponding representation of the participant at the first size, enlarging the representation of the participant corresponding to the selected representation to a second size larger than the first size. In some embodiments, the representation of the selected participant is sized (e.g., medium representation size), while other representations (corresponding to participants with the next lowest activity level in the canvas area) are reduced in size (eg, to the small representation size).

Providing additional control options without cluttering the UI by replacing the display of representations corresponding to the selected representation in the overflow region and/or discontinuing display of the selected representation in the overflow region. , by helping the user quickly replace the representation with the representation of the user's choice, the user has more control over the display of the device and the representation. By allowing additional controls without cluttering the UI with additional displayed controllers, the usability of the device is enhanced and the user-device interface (e.g., when operating/interacting with the device (by assisting the user in providing the correct input and reducing user error), as well as reducing the power usage of the device by allowing the user to use the device more quickly and efficiently. is reduced and battery life is improved.

Optionally, at block 914, the device displays a first animation effect affecting certain ones of the representations of the first plurality of participants (eg, FIGS. 8AA-8AC) (eg, The representation becomes larger then smaller, optionally with cross-fades and tints that produce a "flash" effect). Optionally, at block 915, the device applies a second animation effect affecting selected ones of representations of a second plurality of participants that share animation properties with the first animation effect. display (eg, FIGS. 8BF-8BH) (eg, the rate of fading/contraction from the main display can correspond to the flashing rate of the overflow display). Displaying a second animation that corresponds to the first animation provides visual feedback to quickly identify which representations are to be switched and/or replaced. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, further at block 916, upon detection of an input corresponding to an overflow region (eg, 850H), the device displays a new (eg, previously hidden) representation of the second plurality of participants. (eg, 822A in FIGS. 8AG-8AH) are displayed in the overflow area. Displaying the participant's novel representations in the overflow region in response to input makes it easier and more efficient for the user to view the representations by increasing the accessibility of the displayed representations. . Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at block 917, upon receiving data indicating that a new participant has joined or been added to the live communication session, the device selects one of the second plurality of participants within the overflow region. At the end of the continuation presentation, the new participant's presentation (eg, 822B in FIG. 8R) is displayed in the overflow area.

Optionally, at blocks 918, 919, 920, and 921, the device receives second data regarding activity (e.g., speech, locomotion) level of the first participant in the second plurality of participants. . In response to receiving the second data, the device outputs a second number in the representation of the second plurality of participants within the overflow region by increasing the activity level of the first participant in the second plurality of participants. Visual properties (e.g., size, highlight representation (e.g., display border), animation (e.g., pulsating avatar), image push to the front of representation (e.g., 822B in FIG. 8U) corresponding to one participant's representation ) is emphasized (eg increased or started). In response to receiving the second data, the device reduces the activity level of the first participant in the second plurality of participants such that the second plurality of participants in the representation of the second plurality of participants within the overflow region. Deemphasize (e.g., increase or Start). By enhancing and/or de-emphasizing visual representations when predetermined conditions are met, users quickly perceive activity levels of participants, including the most and/or least active participants. be able to. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. Additionally, by enhancing and/or de-emphasizing when predetermined conditions are met, visual feedback is provided to the user indicating the activity level of each participant. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at blocks 922 and 923, while displaying the live communication user interface, the device selects a second participant (e.g., , most active participants, participants speaking or moving to meet a set of criteria for active participants). in response to detecting a second participant of the second plurality of participants to become the currently active participant, and the representation of the second participant of the second plurality of participants overflows. Upon determining that the region is not currently displayed, the device displays a representation of a second participant of the second plurality of participants (eg, 822F in FIGS. 8BH-8BI) in the overflow region. to scroll the overflow area. By dynamically scrolling the overflow area to display the active participants in the overflow area, it is possible to see representations of the active participants in the overflow area without selecting any additional inputs. Device operability is enhanced by reducing the number of inputs required to perform an operation, and the user-device interface (e.g., user interaction) to provide appropriate input when operating/interacting with the device. and reduce user error), as well as enabling users to use devices more quickly and efficiently, thereby reducing device power usage and improving battery life. be done.

At block 924, upon determining that the live communication session includes a number of participants less than or equal to the threshold number of participants, the device renders representations of participants other than the user of the electronic device less than or equal to their respective threshold sizes. Instead of displaying in one or more sizes, display in one or more sizes larger than the respective threshold size (eg, FIG. 8P). In some embodiments, the electronic device displays (eg, prior to displaying a user interface for the live communication session) a plurality of live media streams associated with respective participants of the plurality of participants in the live video communication session. to receive In some embodiments, the number of live media streams received corresponds to the number of participants currently participating in the live video communication session and does not include participants using devices. Display representations of participants differently based on whether certain conditions are met (e.g. display representations without overflow regions based on whether a live communication session is equal to or less than the number of participants) ) allows users to more easily or more efficiently view representations of participants in a live communication session. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

In some embodiments, multiple representations of remote participants (e.g., participants other than the user) following a determination that the number of participants in the live communication session is less than the second threshold number of participants do not overlap each other (eg, FIG. 8J). In some embodiments, multiple representations of remote participants (e.g., participants other than the user) upon determining that the number of participants in the live communication session is greater than or equal to the second threshold number of participants. overlap each other (eg, FIG. 8K).

Optionally, at blocks 925 and 926, the number of participants in the live communication session is less than a second threshold number of participants (eg, two participants including the user of the electronic device (eg, other 810B in FIG. 8H) (e.g., rectangular aspect ratio). ratio). Upon determining that the number of participants in the live communication session is greater than a second threshold number of participants (e.g., three or more participants), the device reduces representations of participants other than the user of the device. At least one is displayed with a second aspect ratio (eg, square aspect ratio) different from the first aspect ratio (eg, 810B in FIG. 8I). By displaying representations of participants in different aspect ratios based on the number of participants in the live communication session, based on meeting predetermined conditions, the device maximizes the size of the representation and/or organization of representations. can provide a better user experience based on the number of participants, and can provide an intelligent live communication user interface, which allows users to more easily and easily express themselves on their devices. can be seen more effectively. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at blocks 927 and 928, upon determining that the number of participants in the live communication session is less than a second threshold number of participants (eg, two participants), the device: A representation of the view of the device's camera (eg, the self-view from the device's camera) is displayed in a third aspect ratio (eg, a rectangular aspect ratio) (eg, 806 in FIG. 8H). Upon determining that the number of participants in the live communication session is greater than or equal to a second threshold number of participants (e.g., three or more participants), the device displays the view of the device's camera (e.g., the device's self-view from the camera) is displayed in a fourth aspect ratio (eg, square aspect ratio) that is different from the third aspect ratio (eg, 806 in FIG. 8J). By displaying a representation of the device's camera view in different aspect ratios based on the number of participants in the live communication session, based on meeting predetermined conditions, the device can determine the size of the representation of the camera's view to the user. can be provided to provide an intelligent live communication user interface by maximizing , and allowing users to more easily and efficiently view representations on their devices. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at blocks 929 and 930, in response to the number of participants in the live communication session being the first number of participants, the device arranges representations of the participants in a first layout (e.g., arrangement or pattern) (eg, FIG. 8J). In some embodiments, the first layout is one of a plurality of predetermined layouts (eg, a randomly selected layout) in which the user's representation is fixed at various positions of the user interface. In some embodiments, the layout (eg, placement) of representations of the first plurality of participants (eg, other than the user of the device) is determined based on the number of participants in the live communication session. In response to the second number of participants in the live communication session, the device displays representations of the participants in a second layout that differs from the second layout (e.g., FIG. 8K) (eg, the layout changes as participants join/leave the live communication session). In some embodiments, the first layout and the second layout are included in a family of predetermined layouts having similar relative placement of the user's representation. (For example, a given family of layouts includes layouts that work well together). In some embodiments, a family of predetermined layouts includes layout elements for multiple possible numbers of participants (e.g., a first element for one participant other than the user of the device (e.g., 1-up layout) , a second element for two participants other than the user of the device (e.g., a 2-up layout), etc. In some embodiments, the electronic device includes multiple families of predetermined layouts, each family elements share a common characteristic: By displaying representations in different layouts with similar relative placement of user representations based on the number of users, users can navigate easier and more efficiently with high visibility. The operability of the device is enhanced by performing optimized actions without requiring further user input when a set of conditions are met, and the user-device make the interface more efficient (e.g., by helping the user to provide appropriate input and reducing user error when operating/interacting with the device), as well as allowing the user to use the device more quickly; and efficient use of the device reduces power usage and improves battery life.

Optionally, at blocks 931, 932, 923 and 934, while displaying representations of participants other than the user of the device, the device displays data indicating that the number of participants in the live communication session has increased. receive. In response to receiving data indicating that the number of participants in the live communication session has increased, the device reduces the size of one or more representations of participants other than the user of the device (e.g., FIGS. 8J-8K ). In response to receiving data indicating that the number of participants in the live communication session has increased, the device adds new participants in the live communication session (e.g., those who have just recently joined the group prior to the original display). participants) are displayed (eg, FIGS. 8J-8K).

Reducing the size of one or more representations and displaying a new representation provides the user with feedback on the current state of the live communication session and provides a visual indication that a new participant has joined the live communication. Feedback is provided to the user. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at blocks 935 and 936, the device receives input (eg, 850B) corresponding to selection of the first reminder affordance (eg, 808). In response to receiving input corresponding to the selection of the first reminder affordance, the device sends a new notification (e.g., call, ring tone, etc.) that is different from previous notifications sent to the first participant. Send to the first participant (eg, the second notification is more coherent than the first notification). In response to receiving input corresponding to the selection of the reminder affordance, by sending a notice to the first participant that is different from the previous notice sent to that participant, the input required to perform the operation. decrease in number. By reducing the number of inputs required to perform an operation (e.g., the number of options a user needs to remind a participant), device usability is enhanced and user-device interfaces (e.g., device (by assisting the user in providing appropriate input when operating/interacting with the device, reducing user error), as well as enabling the user to use the device more quickly and efficiently Doing so reduces device power usage and improves battery life. Additionally, by automatically performing an action, such as sending a notification that differs from the first notification, without further user input, the operability of the device is enhanced and the user-device interface (e.g., device (by assisting the user in providing appropriate input when operating/interacting with the device, reducing user error), as well as enabling the user to use the device more quickly and efficiently Doing so reduces device power usage and improves battery life.

Optionally, in blocks 937, 938 and 939, in embodiments in which the placeholders and representations of the first plurality of participants are arranged in a first layout, the device determines whether the first participant is live communicating. Upon determining that the session has not been attended for a predetermined amount of time (eg, has not participated in the live communication after a predetermined amount of time), display of the placeholder is discontinued (FIGS. 8E and 8F). Further, upon determining that the first participant has not participated in the live communication session for a predetermined amount of time, the device arranges the representations of the first plurality of participants differently than the first layout. change to a second layout (e.g., including enlarging and/or moving one or more of the representations of the participants (e.g., displaying a layout in which the first participant is not included in the layout); In some embodiments, the device receives data from a server or other device to determine that the first participant has not participated in the live communication session for a predetermined amount of time. , the device may determine that the first participant has not participated in the live communication session for a predetermined amount of time by not displaying any representation of the user. By stopping displaying the placeholders and changing the placement of the representation when satisfied, the user maximizes the visibility of the displayed participants and/or placeholders while allowing other users to join the call. Ability to quickly recognize when to participate or not to participate Operability of the device by taking optimized actions when a set of conditions are met, without requiring further user input is enhanced, making the user-device interface more efficient (e.g., by assisting the user in providing appropriate input and reducing user error when operating/interacting with the device); , which reduces device power usage and improves battery life by allowing users to use the device more quickly and efficiently, as well as discontinuing display of placeholders and rearranging displays By providing feedback to the user regarding the current state of the live communication session, the user is provided with notifications and/or visual feedback indicating that certain actions related to the live communication session will occur. Providing user-friendly visual feedback enhances device usability and improves the user-device interface (e.g., assists the user in providing appropriate input when operating/interacting with the device). , by reducing user error), and in addition by allowing the user to use the device more quickly and efficiently, the power usage of the device is reduced and battery life is improved.

Optionally, at block 940, upon determining that the first participant has joined the live communication session, the device modifies display of the first placeholder with a representation of the first participant. (eg, 810B in FIGS. 8C-8D) (eg, the placeholder can accompany the participant's name). In some embodiments, the device may receive data from a server or other device to make this determination. In some embodiments, the device may make this determination because it is not displaying any representation of the user. In some embodiments, animation occurs. By modifying the display of the first placeholder with representations of the participants, the user can quickly recognize which users have joined the call and feedback to the user regarding the current state of the live communication session. visual feedback is provided to the user indicating that certain actions are being taken in connection with the notification and/or live communication session. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at blocks 941 and 942, while displaying representations of multiple participants in the live communication session, the device detects changes in the number of participants in the live communication session. In response to detecting a change in the number of participants in the live communication session, a notification (eg, 812) is displayed indicating that the number of participants in the live communication session has changed. In some embodiments, the notification when a participant leaves a live communication session replaces a representation of the leaving participant with a placeholder graphic (e.g., a graphic containing the participant's name or initials), and then An animation that disappears (eg, after a certain amount of time). In some embodiments, a notification when a new participant joins a live communication session is sent at a later time (eg, after a predetermined time) by a representation of the new participant (eg, from the new participant's device). display of placeholders for the new participant (eg, a graphic containing the participant's name or initials) to be replaced with a representation based on video data. In some embodiments, the notification indicating that the number of participants in the live communication session has changed causes a determination that the number of participants participating in the live communication session exceeds a threshold number of participants. (e.g., when the overflow region contains more than a predetermined maximum number of representations (e.g., three or four representations other than the user of the electronic device). In some embodiments, the device , avoid displaying notifications that the number of participants in a live communication session has changed etc., when there is another visible indication that the participant has joined or left the live communication session.) In some embodiments, the device displays no notification (e.g., the participant provide an audio output indicating that the number of participants in the live communication session has changed, when there is no visible indication that the participants have joined or left the live communication session. includes an indication (e.g., names) of participants who have joined or left the live communication session, hi some embodiments, multiple participants have joined the live communication session within a threshold time period and/or When leaving a session, the device generates a single notification indicating the number of participants who have joined and/or left the live communication session within a threshold time. Participants joining and leaving a live communication session are grouped into separate notifications.In some embodiments, when a participant leaves or joins a live communication session, the device delays the display of the notification until a threshold time has passed to determine if the notification should include information about additional participants joining and/or leaving the live communication session.Participants in a live communication session By displaying a notification indicating that the number of Visual feedback is provided to the user to indicate that providing the improved visual feedback to the user enhances the device's usability and improves the user-device interface (e.g., operating/interacting with the device). making the device more efficient (by assisting the user in providing appropriate input at times and reducing user error), as well as enabling the user to use the device more quickly and efficiently. It uses less power and improves battery life.

Optionally, at blocks 943, 944, 945 and 946, the device renders a representation of the device's camera view (eg, self-view) as part of representations of multiple participants in the live communication session to a first Indicate by size (eg, 806 in FIG. 8AH). While displaying representations of multiple participants in a live communication session, the device receives a first input (eg, 850I) corresponding to a selection to display a first menu (eg, 832) (eg, , triggered based on a single tap or upward swipe on the background). In response to receiving input corresponding to the selection to display the first menu, the device displays the first menu (eg, call control menu 832) on the live communications user interface (eg, 804). , the first menu includes options (eg, 836) for displaying visual effects (eg, creative cameras) on the representation of the participants in the live communication session. In some embodiments, the menu includes other options, such as to exit the call switching camera (eg, from the front camera to the rear camera). In response to receiving a second input (e.g., 850K) corresponding to selecting an option for displaying a visual effect (e.g., 842), the device displays a representation of the view of the device's camera (e.g., 842 ) in a second size that is larger than the first size (eg, overlaid on the participant's representation).

Optionally, at blocks 947, 948 and 949, the device responds to selection to expand a first representation (eg, 810F of FIG. 8L) of representations of multiple participants in the live communication session. A third input (eg, 850C) is received (eg, an expressive double tap in the main region, or an input with a characteristic intensity above the intensity threshold). In response to receiving a third input corresponding to selecting to magnify a first representation of a plurality of participants in the live communication session, the device magnifies the first representation (e.g. , 810F in FIG. 8M). The device reduces the size of at least one representation (eg, 810E in FIGS. 8L-8M) that is different than a first representation of the plurality of participants in the live communication user interface. Dynamically reducing the size of one or more representations in the set of displayed representations and expanding selected representations allows the user to focus on the enlarged representations while minimizing the reduced representations. and provides improved feedback where the same representation can be easily viewed. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at blocks 950, 951, 952 and 953, the device receives data indicating a change in the orientation of the electronic device (e.g., in a first direction (e.g., clockwise 90 degree change)). . In response to receiving data indicating a change in orientation of the electronic device, the device moves live (e.g., in a second orientation (e.g., in a direction opposite the first orientation, such as to maintain the image upright)). Rotate the display of representations of multiple participants in the communication session (eg, FIGS. 8W-8X). While the device rotates the display, the device may determine the relative characteristics of previous displays of representations of multiple participants in the live communication session (e.g., size, position, placement of user interface objects (e.g., representations) of other displays). etc.). By automatically rotating the device when certain conditions (e.g., data indicating that the user is rotating the device) are met, preserving the relative characteristics of previously displayed representations; The user can view the rotated representation in the same display position as before, with minimal visual clutter, without changes in placement, size, or position of the representation. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at block 954, in response to receiving first data indicating a change in orientation of the electronic device (eg, from portrait orientation to landscape orientation), the device performs multiple Modifying the display of a first one of the participant's representations based on the position of the face within the first representation (eg, 822C in FIGS. 8AZ-8BA) (eg, if the representation is not square, Adjust the display so that the participant's face is in the center of the presentation when moving from a vertical presentation (the participant's face may be biased towards the top of the presentation) to a horizontal presentation).

Optionally, in blocks 955 and 956, while displaying a second representation of the plurality of participants, the device displays at least one representation of the plurality of participants in the live communication session. Receive a request to change the aspect ratio of one representation. In response to receiving a request to change the aspect ratio of at least one of the representations of the plurality of participants in the live communication session, the device changes the aspect ratio of the second representation to a sixth aspect ratio. to a seventh aspect ratio (e.g., a square or circular aspect ratio), while the second representation is the second aspect of the live media stream received by the electronic device while at the seventh aspect ratio. portion (eg, less than the first portion, the cropped portion), and the second portion is selected to include faces detected in the live media stream (eg, FIG. 8AZ-FIG. 8A-Z). 822C of 8BA). In some embodiments, the electronic device changes the aspect ratio of the representation based on the live video feed such that the faces to be detected are not lost cropped from the representation. Crop a portion of the live video feed by selecting the portion that contains the face. By maintaining the view of the face within the participant's representation, feedback is provided to the user that allows the user to see the face of the other participant when changing the aspect ratio. By providing improved visual feedback to the user, the usability of the device is enhanced and the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at blocks 957, 958, 959, 960, and 961, while displaying representations of multiple participants in the live communication session, the device is configured to display a menu (e.g., 832). A fourth input (eg, 850I) corresponding to the request (eg, triggered based on a single tap or upward swipe on the background) is received. In response to receiving the fourth input, the device displays a menu (eg, 832) on the user interface for the live communication session. The device renders at least a third representation (e.g., 810H in FIG. 8AI) of representations of multiple participants in the live communication session (e.g., to accommodate the menu, to avoid overlapping with the menu). move. The device (e.g., to accommodate the menu, to avoid overlapping with the menu) represents at least a fourth of the representations of the multiple participants in the live communication session (e.g., 810F in FIG. 8AI) ( For example, a representation that is the same as the third representation, a representation that is different from the third representation) is resized. In some embodiments, the representation is moved up and resized when the menu is displayed at the bottom of the screen. In some embodiments, at least one representation of the plurality of representations is not resized when displaying the menu. In some embodiments, the representation is a representation with three predetermined image sizes (e.g., one "small", two "medium", and one "large"), and one size (eg, the “medium” image(s)) are resized, while the “small” and “large” representations are kept the same size. Add while maintaining visibility of representations in the live communication user interface by moving one or more representations when viewing the menu and resizing one or more representations in the live communication session Control options are provided. Device operability is enhanced by providing additional control options without cluttering the live communication user interface with additional displayed controllers, making the user-device interface (e.g., manipulating/interacting with the device make the device more efficient (by helping the user to provide the appropriate input when use less power and improve battery life.

In some embodiments, the device determines that the activity level of the first participant in the second plurality of participants represented in the overflow region is the first representation of the representations of the first plurality of participants. (e.g., the participant represented in the overflow area rises in activity level above the activity level of one of the participants represented in the canvas area) receive data indicating that a call is to be initiated in a second plurality of participants, and, in response, display a first representation of the representations of the first plurality of participants to the first participation in the second plurality of participants; (eg, replace the participant with the lowest activity level in the canvas area with the representation of the speaking participant) and stop displaying the representation of the first participant in the overflow area. In some embodiments, ceasing to display the representation of the first participant within the overflow region includes an animation of the representation contracting. In some embodiments, other representations within the overflow region are translated to fill the gaps in which the representations were displayed. In some embodiments, replacing the display of the first representation of the first plurality of participant representations displays the representation of the participant corresponding to the replaced first representation in the overflow region. Including.

In some embodiments, while displaying representations of multiple participants in a live communication session, the device receives input corresponding to a request to display a menu. In response to receiving input corresponding to the request to display the menu, the device displays the menu on the live communication user interface. The menu includes options for selecting content items (eg, games, videos, sketches). In some embodiments, the device receives input corresponding to selection of the content item, and in response to receiving input corresponding to selection of the content item, the device displays at a size greater than the respective threshold size. Replace representations of participants to be processed (e.g., representations of participants in the canvas area) with representations of content items that are larger than their respective threshold sizes (e.g., content items are displayed in one).

In some embodiments, displaying representations of the second plurality of participants in the overflow region indicates that the first number of participants (eg, a total of 6 participants) participated in the live communication session. display a first set of representations (e.g., two representations including a representation of the least active participant and a representation of the view of the device's camera) in the overflow region; A first set of representations upon determining that a second number of participants different than one number of participants (e.g., a total of seven participants) are participating in the live communication session. A second set of different representations (e.g., three representations, including representations of the two least active participants and a representation of the view of the device's camera) are displayed in the overflow region, and the second set of representations The set contains a different number of representations than the first set of representations.

It should be noted that the details of the processing described above with respect to method 900 (eg, FIGS. 9A-9K) are also applicable to the methods described/discussed below, in an analogous manner. For example, method 900 optionally includes one or more of the various method features described above with respect to methods 700 , 1100 , 1300 , and 1500 .

10A-10S illustrate exemplary techniques for dynamically adjusting user interfaces, particularly for live communication sessions, according to some embodiments. The user interfaces of these figures are used to illustrate the processes described below, including the processes of Figures 11A-11F.

FIG. 10A shows device 600 displaying user interface 1000 for an active live communication session. User interface 1000 includes representations of five participants in a live communication session displayed in canvas area 1001 . Representation 1010A includes placeholders for participants who are not connected to the live communication session. Representation 1010B includes representative images of participants providing audio data only. Representations 1010C and 1010D each include live video feeds from two participants. Representation 1002 includes a view from camera 603 of device 600 .

Each participant has an activity level that is used to determine the layout of user interface 1000 . In some embodiments, the participant's activity level is based on audio and/or video feeds received from the participant's device (eg, movement or sound in the video feed). In some embodiments, the participant's activity level is filtered audio from an audio feed received from the participant's device (e.g., background noise filtered audio or speech filtered to isolate/amplify speech audio). audio). In some embodiments, a participant's activity level is determined by movement (e.g., movement of a particular type (hand movement, head / based on face movement, background movement (eg non-participant movement).

In some embodiments, the participant's activity level is data indicative of activity detected at the first participant's device (e.g., if the first participant performs a user interface action ( For example, based on data indicating that a selection of attention affordances or visual effect affordances has been made). In some embodiments, the activity level is sent to device 600 after being determined by an external electronic device (eg, server).

As shown in FIG. 10A, the activity levels of the participants in the live communication session are in order from highest to lowest in representation 1010C, representation 1010D, representation 1010B, and representation 1010A. The table shown on the right side of FIGS. 10A to 10O lists the participants in descending order of activity level from highest to lowest. The person with the highest activity level and the person at the end of the list (4) has the lowest activity level. 10A-10O, the representation corresponding to the participant with the highest activity level is the largest representation in canvas area 1001 and the participant represented in canvas area 1001 with the lowest activity level. is the smallest. 10A-10O, the representations corresponding to the participant with the second highest activity level and the participant with the third highest activity level are the representations of the participant with the highest activity level among the participants represented in canvas area 1001. and the size of the representation of the participant with the lowest activity level.

In some embodiments, a participant's activity level varies with respect to other participants. While displaying user interface 1000, device 600 receives data indicating that the activity level of participant Stephen corresponding to representation 1010B has increased above the activity level of participant Nicholas corresponding to representation 1010D. do. As shown in FIG. 10B, the activity level of representation 1010B rises above that of representation 1010D but does not change in size since it has the second highest activity level. In FIG. 10B, participant Stephen continues speaking and becomes the participant with the highest activity level. Correspondingly, as shown in FIG. 10C, the size of representation 1010B is increased, the size of representation 1010C is decreased, and representation 1010A (e.g., avoids excessive overlap with representation 1010B by increasing the size of representation 1002B). ) shift to the right. In some embodiments, device 600 resizes and/or positions more than two representations on user interface 1000 .

As shown in FIG. 10C, representations 1010A-1010D overlap each other. In some embodiments, one or more of the representations 1010A-1010D avoid obstructing the participant's face and/or overlap beyond a predetermined maximum overlap as the size of the representation increases. are reduced in size and/or moved to avoid In some embodiments, the front and back (aka “z-order”) of representations 1010A-1010D do not change when the size of the representation increases (eg, when participants are actively participating). In some embodiments, representation 1010 is (eg, in response to active participation by a corresponding participant, or user input at device 600 selecting the representation (eg, in response to user input 850C of FIG. 8M described above)). ) change the order of one or more representations when resizing (eg, move back to front or front to back). For example, in some embodiments, representation 1010B moves before representation 1010C and/or representation 1010A as it increases in size in FIGS. 10B-10C.

In some embodiments, representations of participants remain in approximately the same area on display 602 when resized/or repositioned. As shown in FIGS. 10A-10C, after representations 1010A-1010D are resized and/or relocated, each rendition occupies at least a portion of the area it occupied prior to being resized and/or relocated. .

Turning to FIG. 10D, participant Stephen of representation 1010B begins to drop the call, and participants in the live video feed of representation 1010A begin to physically move, as shown in FIG. 10E. Participant movement is detected in the live video feed. In some embodiments, the participant is actively participating in the live communication session based on detected movement within the participant's live video feed (e.g., movement indicative of the participant's physical movement). is determined. In some embodiments, the participant's movement results in an increase in the participant's activity level. In some embodiments, a participant is determined to be actively participating by a movement meeting one or more criteria (eg, duration, magnitude, direction, or predetermined gesture).

As shown in FIG. 10F, in response to receiving data indicating that a participant in representation 1010A is actively participating, device 600 increases the size of representation 1010A and decreases the size of representation 1010D. Representation 1010B is moved downward while remaining the same size. Thus, representation 1010A transitions from the smallest representation to the medium size representation, and representation 1010D transitions to the smallest representation.

In some embodiments, in response to receiving data indicating that a participant of representation 1010A (eg, a participant with the lowest activity level) is actively participating, device 600 reduces the size of representation 1010A to From the smallest representation to the largest representation, the size of representation 1010B is reduced. In some embodiments, representation 1010A grows from a minimum representation to a maximum representation by the participant corresponding to representation 1010A becoming the most recent participant actively participating. For example, in some embodiments, the participant corresponding to representation 1010A becomes the most recent participant to actively participate, causing device 600 to convert from the layout and relative representation sizes shown in FIG. 10G to the layout and relative presentation sizes shown (described below).

In some embodiments, participants actively participate in a live communication session by applying visual effects to live video feeds provided to the live communication session. In some embodiments, the participant applies visual effects through the user interface of the participant's device. In some embodiments, a participant is determined to be actively participating in a live communication session based on visual effects included in the participant's live media feed. A visual effect (a robot face) is applied to the live video feed of representation 1010A, as shown in FIG. 10F. As a result, the participant in representation 1010A is determined to be actively participating, and the participant's activity level increases. As shown in FIGS. 10F-10H, upon determining that the participants in representation 1010A are actively participating, device 600 increases the size of representation 1010A to the maximum representation size and the size of representation 1010B. Reduce size to medium representation size. As the participant's activity level associated with representation 1010A increases, the relative z-order of representations 1010A and 101B also changes, moving representation 1010A in front of representation 1010B.

In FIG. 10H, device 600 receives (eg, detects) user input 1050A (eg, tap) on representation 1010C. As shown in FIGS. 10I-10J, in response to user input 1050A, device 600 increases the size of representation 1010C to the maximum representation size. In addition, the participant corresponding to representation 1010D begins to actively participate (eg, speak) causing representation 1010D to grow to a medium size and representation 1010B to shrink to a small size.

In FIG. 10J, device 600 receives (eg, detects) user input 1050B (eg, a tap) at a location on display 602 where a representation of the participant is not displayed. As shown in FIG. 10K, in response to user input 1050B, device 600 displays call control menu 1004 similar to call control menu 630 described above.

As shown in FIG. 10K, the size of canvas area 1001 is reduced so that call control menu 1004 does not overlap representation 1010 when call control menu 1004 is displayed. To make room for call control menu 1004, device 600 modifies the layout of representation 1010 by reducing and/or translating one or more of the representations up. In some embodiments, device 600 reduces the size of medium-sized representations 1010A and 1010D and maintains the size of small and large primary representations, 1010B and 1010C, respectively. In some embodiments, device 600 reduces the vertical gap between representations (eg, representation 1010A and representation 1010C).

In response to receiving user input 1050B, device 600 also displays ribbons 1012A-1012D on each representation except representation 1002 of the user of device 600. FIG. Each ribbon contains the name of the corresponding participant and affordances for expanding the representation.

Turning to FIG. 10L, device 600 is shown in a landscape (horizontal) orientation. The relative activity levels of the participants are the same as in Figure 10J. Compared to the portrait (vertical) orientation of FIG. 10J, the representations occupy the same respective areas of the display 602 as in the portrait (vertical) orientation, and the images in the representations show the participants standing upright. rotated so that it looks like In some embodiments, in response to rotation of device 600 from the portrait orientation of FIG. 10J to the landscape orientation of FIG. , 90 degrees opposite to the direction of rotation of device 600).

Turning to Figure 10M, beginning with the layout and activity level shown in Figure 10J, the sixth participant Pablo joins the live communication session. As shown in FIG. 10M, in response to a sixth participant joining the live communication session and the number of connection participants in the live communication session being a predetermined threshold of connection participants, five connection participants device 600 replaces the image of representation 1010B corresponding to the participant with the lowest activity level with the image of the new connected participant. Device 600 optionally provides notification that a new connection participant has joined the live communication session and displays an overflow representation 1022A of the new connection participant in overflow region 1020. As shown in FIGS. 10M-10O, the size of canvas area 1001 is reduced when overflow area 1020 is displayed. At least some representations 1010A-1010D are reduced in size and/or moved on display 602 .

As shown in FIG. 10N, overflow representation 1010A corresponds to participants not represented by the main representation (eg, the representation above overflow area 1020 in FIG. 10N). 10N, device 600 receives (eg, detects) user input 1050C (eg, a tap) on overflow representation 1022A. As shown in FIG. 10O, in response to user input 1050C, device 600 replaces the image in representation 1010A with the image of the participant corresponding to the selected overflow representation 1022A, causing the replaced participant (Hoan) to Display the corresponding overflow representation 1022B.

In some embodiments, device 600 overflow-represents images in representation 1010A of FIG. Replace with the participant's image corresponding to 1022A. In some embodiments, in response to determining that the participant's activity level corresponding to overflow representation 1022A has increased above the participant's activity level corresponding to representation 1010A of FIG. 10N representation 1010A with the participant's image corresponding to overflow representation 1022A.

Turning to FIG. 10P, device 1008 with display 1014 is shown. In some embodiments, display 1014 has different characteristics (eg, size and/or aspect ratio) than display 602 of device 600 . Device 1008 displays user interface 1016 divided into two portions, portion 1016A (eg, canvas area) and portion 1016B. Portion 1016 A has approximately the same aspect ratio as display 602 of device 600 . Due to the aspect ratio of portion 1016A, portion 1016A includes representations 1030A-1030D corresponding to the participants in representations 1010A-1010D of FIG. 10O. As shown in FIG. 10P, the layout (eg, relative size, position, spacing, and overlap) of representations 1030A-1030D is similar to the layout of representations 1010A-1010D of FIG. 10O.

As shown in FIG. 10P, device 1008 receives (eg, detects) user input 1050D (eg, right swipe) to move divider 1018 to the right. In response, device 1008 decreases the size of portion 1016B and increases the size of portion 1016A. As shown in FIGS. 10P-10S, representations 1030 are magnified and the horizontal spacing between representations 1030 is increased. As shown in FIG. 10S, device 1008 displays menu 1024 with effect affordance 1026, call affordance 1028, and menu affordance 1034 (similar to effect affordance 632, call affordance 634, and menu affordance 636 described above).

11A-11F are flow diagrams illustrating methods according to some embodiments. Method 1100 is performed on a device with a display (eg, 100, 300, 500 or 600). Some acts of method 1100 are optionally combined, the order of some acts are optionally changed, and some acts are optionally omitted.

As described below, method 1100 provides an intuitive way of providing an adaptive and intelligent live communication user interface that highlights one or more active participants in a live communication session. This method creates a more efficient human-machine interface by reducing the user's cognitive burden of viewing active participants in a live communication session. For battery-operated computing devices, allowing users to view active participants faster and more efficiently saves power and extends battery charge intervals.

At block 1102, the device sends a representation of a first participant in the live video communication session (eg, 1010C), a representation of a second participant in the live video communication session (eg, 1010B), and Display a live video communication user interface (eg, 1000) simultaneously displaying a third participant's representation (eg, 1010D). In some embodiments, if the participant is sharing a live media stream that includes a live video stream, the representation includes images of the live video stream (eg, 1010C). In some embodiments, if the participants are sharing an audio-only live video stream, the representation includes each participant's avatar (eg, 1010B). In some embodiments, if the participant is not currently participating in a live video communication session, the representation includes a placeholder image (eg, avatar) (eg, 1010A). In some embodiments, the live communication interface includes a canvas area (eg, 1001) (eg, the canvas area (eg, main area) is different from the overflow area). In some embodiments, the canvas area is adjacent to the overflow area. In some embodiments, a representation of a first participant in the live communication session, a representation of a second participant in the live communication session, and a representation of a third participant in the live communication session are displayed on the canvas area. displayed (eg, FIG. 10A).

At block 1104, while displaying the live video communication user interface, the device determines that the activity level of the second participant (eg, 1010B) in the live video communication session is equal to that of the first participant in the live video communication session. Receive data indicating an increase above an activity level of (eg, 1010C). In some embodiments, data is received from an external source (eg, a server) indicating that the first participant is actively participating. In some embodiments, the device determines whether the first participant is actively participating based on data received (e.g., live video or live audio from a live media feed associated with the first participant). determined to be In some embodiments, prior to receiving data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; One or more of the representations of the first participant, the second participant, and the third participant overlap with at least one other representation of the participant (e.g., 1010B, 1010C, and 1010D).

In some embodiments, the activity level of at least the first participant is (a function of) an audio and/or video feed received from the first participant's device (e.g., movement or audio in the video feed) and/or data indicative of activity detected at the first participant's device (e.g., the first participant performs a user interface action (e.g., selecting a featured affordance) at the first participant's device). based on data showing that In some embodiments, the activity level is determined by an external electronic device (eg, server) and then transmitted to the electronic device. In some embodiments, the activity level of at least the first participant is filtered audio (e.g., background noise filtered audio or speech audio) from an audio feed received from the first participant's device. audio filtered to isolate/amplify). In some embodiments, at least the first participant's activity level is identified in a video feed received from the first participant's device (e.g., identified at a server, identified at an electronic device). based on movements that satisfy a set of movement criteria (movements that are not specific types of movements (e.g. hand movements, head/face movements, background movements (e.g. non-participant movements)) (e.g. 1010A in FIG. 10E).

At block 1106, the activity level of the second participant (eg, 1010B in FIG. 10A) in the live video communication session exceeds the activity level of the first participant (eg, 1010C in FIG. 10A) in the live video communication session. In response to receiving data indicating that the Changing the size and/or position of the representation of the second participant in the video communication user interface and (at block 1112) the size and/or the representation of the third participant (eg, 1010D) in the live video communication user interface Change position. By changing the size and/or position of the representation when predetermined conditions are met, the user can quickly perceive the activity level of the participants, including the most and/or least active participants. . Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. Additionally, the number of inputs required for the user to determine or focus on active participants by changing the size and/or position of the representation only when predetermined conditions are met. decreases. A further reduction in the number of inputs enhances the usability of the device and improves the user-device interface (e.g., assisting the user in providing appropriate input when operating/interacting with the device, By making it more efficient (by reducing errors), in addition to allowing the user to use the device more quickly and efficiently, the device uses less power and improves battery life.

In some embodiments, in response to receiving data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; A representation of a first participant in the live video communication user interface (eg, 1010C in FIG. 10A) is resized and a representation of a second participant in the live video communication user interface (eg, 1010B in FIG. 10A) is resized. is changed to resize the representation of the third participant (eg, 1010D in FIG. 10A) in the live video communication user interface. In some embodiments, if the least active participant becomes the most active participant (e.g., based on activity score), the device increases the size of the representation of the most active participant and reduce the size of all representations of (eg, FIGS. 10E-10F). In some embodiments, if the most active participant becomes the least active participant (e.g., based on activity score), the device reduces the size of the representation of the least active participant, Increase the size of all other representations. In some embodiments, in response to receiving data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; The position of the representation of the first participant in the live communication user interface is changed, the position of the representation of the second participant in the live communication user interface is changed, and the position of the representation of the third participant in the live communication user interface is changed. is changed. In some embodiments, if the least active participant (e.g., based on activity score) becomes the most active participant, then all positions are swapped (e.g., the representation of the most active participant is on top). move under or over the other). In some embodiments, when the most active participant becomes the least active participant (e.g., based on activity score), the representation of the least active participant is under and/or overlaid. Move so that other representations move on or over it.

In some embodiments, a portion of the representation (e.g., 1010C in FIG. 10A) of the participant with the (currently) highest activity level among the first, second, and third participants includes A portion of the representation of another of the first, second and third participants overlaps (eg, a portion of the most active representation is below the other representation). In some embodiments, the representation of the participant with the lowest activity level is displayed without overlapping representations of any other participants (this is, for example, because the representation of the participant with the lowest activity level is (Because it is the smallest of the participants' representations, it is kept on top of other participants' representations to avoid overly obscuring the representation of the participant with the lowest activity level.) In some embodiments, the representation of the participant with the lowest activity level (eg, 1010B in FIG. 10I) overlaps at least a portion of the representation of the participant with the highest activity level (eg, 1010A in FIG. 10I). Is displayed. In some embodiments, a portion of the representation (eg, 1010D in FIG. 10H) of the participant with the lowest (currently) activity level among the first, second, and third participants is the live communication user interface. overlaps a portion of the representation of another of the first, second, and third participants (eg, 1010C in FIG. 10H) (eg, a portion of the least active representation underlies the other representation). be).

Optionally, at block 1114, the device performs facial recognition based on facial recognition data obtained (eg, obtained by analysis) from the second participant's video feed (and/or the third participant's video feed). , changing the size and/or position of the representation of the first participant, and changing the size and/or position of the representation of the first participant obscures the face of the second participant ( and/or obscuring the face of the third participant). changing the size and/or position of a participant's representation in a live communication session without obscuring the face of another participant in the live communication session without obscuring the face of another participant; Feedback about the current activity level is provided to the user so that the user can see the faces of other participants as they change the size or position of the participant whose activity level has changed. By providing improved visual feedback to the user, the usability of the device is enhanced and the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at block 1116, the device changing the size and/or position of the representation of the first participant (eg, 1010B in FIGS. 10A-10C) reduces the size of the representation from the first size. Gradually increasing to the second size (eg, gradually over time). By changing the size and/or position of the representation, feedback is provided to the user regarding the participant's current activity level while minimizing user interface changes. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be. Gradually increasing the size of the representation provides a visual indication that the activity level is changing while minimizing the confusion that abrupt changes in size may cause to the user as the participant's activity level changes. Feedback is provided to the user. By providing improved visual feedback to the user, the usability of the device is enhanced and the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at block 1118, the device triggers a canvas resize event (eg, 1050B, FIGS. 10J-10K) (eg, rotation of the device, introduction of additional elements to the live video communication user interface (eg, participant Detecting one or more call affordances for contacting , adding a representation of a participant displayed outside the canvas area), user input corresponding to a request to resize the canvas area). Optionally, at block 1120, the device resizes the canvas area (eg, 1001) (optionally at block 1124) in response to detecting a canvas resize event. In response to detecting a canvas resize event, the device modifies the layout of representations of the first, second and third participants (eg, 1010B, 1010C and 1010D in FIG. 10K). In some embodiments, the layout of the representations shifts the position of the representations and/or changes the size of the representations (e.g., so that the spacing between representations becomes smaller when the size of the canvas area is reduced). corrected by To enable a user to see the user's representation with minimal changes in the displayed representation when changing the canvas size by intelligently modifying the layout based on the resizing of the canvas area. to reduce the confusion caused to the user while showing the displayed presentation to the user. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at block 1126, the device further resets one of the representations of the first, second and third participants (eg, 1010B, 1010C, 1010D of FIG. 10K) in response to detecting a canvas resize event. Resize one or more. In some embodiments, prior to receiving data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; Representations of the first, second, and third participants in the live communication user interface are displayed (eg, 1010B, 1010C, and 1010D in FIG. 10K) by z-order (eg, axis lines out of the screen) (eg, , "z-order" refers to the order of representations of a participant with respect to an imaginary axis perpendicular to the plane of the display) (e.g., when the positions of two representations overlap, the z-order of representations determines which objects are determines whether it appears before). At block 1128, the device maintains the z-order of the representations of the first, second and third participants (eg, 1010B, 1010C and 1010D in FIG. 10K).

In some embodiments, after changing the size and/or position of the first participant's representation (e.g., 1010B in FIG. 10B) in the live communication user interface, the modified representation of the first participant is: the area occupied by the representation of the first participant in the live communication user interface prior to changing the size and/or position of the representation of the first participant (e.g., the total area, position, pixels, etc. on the display); occupy at least a portion of In some embodiments, after resizing, the participant is displayed in the same area on the display that the participant was previously displayed (e.g., at least a portion of the representation (e.g., a portion, pixels, locations, etc.) continue to touch the previous locations on the display that they touched before the change). In some embodiments, the representation emphasizes the active participant while maintaining the placement/general position of the representation (e.g., the representation shifts as it grows/decreases in size, whereas other images position relative to). In some embodiments, after changing the size and/or position of the second participant's representation (eg, 1010C in FIG. 10B) in the live communication user interface, the modified representation of the second participant is: The area occupied by the representation of the second participant in the live communication user interface prior to changing at least one of the size and position of the representation of the second participant (e.g., total area, position, pixels, etc. on the display) occupy at least a portion of In some embodiments, after resizing, the participant is displayed in the same area on the display that the participant was previously displayed (e.g., at least a portion of the representation (e.g., a portion, pixels, locations, etc.) continue to touch locations on the display that were previously touching before changing). In some embodiments, the representation emphasizes the active participant while maintaining the placement/general position of the representation (e.g., the representation shifts as it grows/decreases in size, whereas other images position relative to). In some embodiments, after changing the size and/or position of the third participant's representation (eg, 1010D in FIG. 10B) in the live communication user interface, the modified representation of the third participant is: The area (e.g., on the display) occupied by the third participant's representation (e.g., prior to the change) in the live communication user interface prior to changing at least one of the size or position of the third participant's representation occupies at least a portion of the total area, location, pixel, etc.). In some embodiments, after resizing, the participant is displayed in the same area on the display that the participant was previously displayed (e.g., at least a portion of the representation (e.g., a portion, pixels, locations, etc.) continue to touch locations on the display that were previously touching before changing). In some embodiments, the representation emphasizes active participants while maintaining placement and/or overall position (e.g., the representation shifts as the size increases/decreases relative to other images). position).

By displaying a representation that occupies at least a portion of the area previously occupied by the representation after changing the size and/or position of the representation, the user can adjust the participant's activity level with minimal changes to the user interface. can be recognized, which enhances the user's ability to focus on the expressions of the participants in a live communication session. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at block 1130 the device displays a representation of the fourth participant (eg, 1010D in FIG. 10E) in the live communication user interface. Optionally, in response to receiving data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session, the device , (at block 1132) change the size and/or position of the representation of the fourth participant in the live communication user interface. In some embodiments, the fourth representation changes size and/or position when all others change size or position. In some embodiments, for example, the first data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session , the fourth representation changes. In some embodiments, some representations, such as the fourth representation, can maintain their size and position even when other representations are changing.

Optionally, at blocks 1134, 1136, 1138, 1140, and 1142, after changing the size and/or position of the representation of the first participant in the live video communication user interface, the device performs Second data is received indicating that the activity level of the first participant has increased above the activity level of the second participant in the live video communication session. In response to receiving second data indicating that the activity level of the first participant in the live video communication session has increased above the activity level of the second participant in the live video communication session, the device Changing the size and/or position of the representation of the first participant in the video communication user interface. The device resizes and/or positions the representation of the second participant in the live video communication user interface. The device maintains the size and position of the representation of the third participant in the live video communication session. In some embodiments, some of the representations displayed on the canvas (e.g., the main area) are maintained while other representations are scaled to size in response to changes in the participant's activity level. or the position changes (eg, two representations may exchange positions based on the determination, and other representations may exchange size or position). In some embodiments, the representation that is maintained corresponds to a participant whose activity level is lower than the activity level of the participant corresponding to the representation that is changed. By maintaining the size and/or position of some representations while changing the size and/or representation of other representations when predetermined conditions are met, the user can minimize user interface changes. can recognize the activity level of the participants, which enhances the user's ability to focus on the expressions of the participants in the live communication session. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at blocks 1144 and 1146 the device outputs the first participant's activity level indicating that the activity level of the second participant in the live communication session has increased above the activity level of the first participant in the live communication session. a visual display (e.g., a highlighted representation (e.g., , display border), animation (eg, avatar pulsation), push image to front, increase size), enhance (eg, increase or start). In response to receiving first data indicating that the activity level of the second participant in the live communication session has increased above the activity level of the first participant in the live communication session, the device (e.g., Visual display (e.g., highlight representation (e.g., display border), animation (e.g., avatar pulsation) of the first participant's representation in the live communication user interface (when the first participant is currently less active) ), pull the image back, reduce the size), reduce (eg, reduce or cancel) the emphasis. By enhancing and/or de-emphasizing visual representations when predetermined conditions are met, users can quickly perceive activity levels of participants, including the most and/or least active participants. can be done. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. Additionally, by enhancing and/or de-emphasizing when predetermined conditions are met, visual feedback is provided to the user indicating the activity level of each participant. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

In some embodiments, prior to receiving data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session (receiving ), the representation of the first participant is displayed in a first size (e.g., the medium size in a set of three or more predetermined sizes, the largest size of the set) and the representation of the second participant is displayed in a second size (eg, the smallest size in a set of three or more predetermined sizes, medium size in the set) (eg, a small representation) that is smaller than the first size. Optionally, at block 1148 the device increases the size of the representation of the second participant from the second size to the first size. By increasing the size and/or position of the representation from a smaller size to a larger size, when the participant's activity level changes, visual feedback is provided to the user indicating that the activity level is changing. . By providing improved visual feedback to the user, the usability of the device is enhanced and the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at blocks 1150 and 1152, receiving data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session. , the device detects an input (eg, 850C) corresponding to the first participant's expression (eg, a tap on the expression). In response to detecting input corresponding to the representation of the first participant (e.g., the user resizes any representation displayed on the live communication user interface (e.g., overflow area, canvas area, etc.) can), the device increases the activity level (eg, increases the activity level value) of the first participant (eg, 810F in FIGS. 8L-8M).

At blocks 1154 and 1156, the device sends data indicating that the fourth participant in the live communication session is more active than the participant with the lowest activity level among the first, second and third participants. to receive In response to receiving data indicating that a fourth participant in the live communication session is more active than the lowest activity level participant of the first, second and third participants, a first, The representation of the participant with the lowest activity level among the second and third participants (eg, 1010A in FIG. 10N) is replaced with the representation of the fourth participant (eg, 1010A in FIG. 10O). replace. In some embodiments, at block 1158, the device scales the representation of the fourth participant to a size smaller than the representation of the participant with the lowest activity level among the first, second, and third participants. Display in large size. By replacing the presentation of the representation of the participant with the lowest activity level with the participant not represented in the first set of representations when a predetermined condition is met, the non-displayed user is replaced with the first set of representations. Representation of the most active participants in a live communication session while causing the user to reduce the user's attention to the least active participants in the live communication session when they are more active than the users shown in the set of can draw attention to. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. of other participants (e.g., the first subset of representations) to the new participant while minimizing changes to the user interface by increasing the size of the newly displayed participant. Feedback is provided to the user regarding their current activity level. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at block 1160, the device displays a representation (eg, 1002) of the electronic device's camera view (eg, within the live communication session user interface (eg, self-view)). At block 1162, the device is further responsive to receiving first data indicating that the activity level of the second participant in the live communication session has increased above the activity level of the first participant in the live communication session. and optionally maintains the size of the representation (eg, 1002) of the view of the electronic device's camera in the live communication session user interface at block 1164. In some embodiments, the self-view is does not change based on changes in a person's activity level). By maintaining the size of the representation of the device's camera's view, while still minimizing layout changes, the user can: expression can be seen. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

It should be noted that details of the processes described above with respect to method 1100 (eg, FIGS. 11A-11F) are also applicable to the methods described/discussed below, in an analogous manner. For example, method 1100 optionally includes one or more of the various method features described above with respect to methods 700 , 900 , 1300 , or 1500 .

12A-12N illustrate exemplary user interfaces for, among other things, initiating a live communication session from a messaging application, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes of Figures 13A-13D.

FIG. 12A shows device 600 (as described above) displaying messaging user interface 1200 of a messaging application. Messaging user interface 1200 contains in message area 1202 messages between participants in a group of 15 participants (“The Dream Team”) involved in a message conversation. The messaging user interface 1200 accesses all call affordances 1206 for initiating live communication sessions with all participants in a message conversation (e.g., all participants in The Dream Team group) and additional information about The Dream Team group. group contact information affordance 1208 for

As shown in FIG. 12A, device 600 receives (eg, detects) user input 1250A (eg, tap) corresponding to selection of overall call affordance 1206 . In response to receiving user input 1250A, device 600 initiates a live communication session with all participants in the message conversation that are included as participants in the live communication session. In some embodiments, in response to receiving user input 1250A, device 600 sends instructions to cause devices associated with members of the group to display notifications that a live communication session is active. In this way, all call affordance 1206 immediately initiates a live communication session with all participants of the message conversation group.

FIG. 12B shows messaging user interface 1200 of a messaging application while a live communication session is being initiated. Messaging user interface 1200 includes message 1212 in message area 1202 and visual indicator 1214 at the top of display 602 to indicate that a live communication session is active.

In some embodiments, in response to receiving user input 1250A, device 600 launches a live communication application. As shown in FIG. 12C, after the live communication session begins, device 600 displays live communication user interface 1216 of the live communication application. Live communication user interface 1216 corresponds to a live communication session initiated by the group The Dream Team.

As shown in FIG. 12C, the live communication user interface 1216 displays primary representations 1210A-1210D (set 1210 ) and a user representation 1218 of the user of device 600 . The live communication user interface 1216 also includes an overflow area 1220 with overflow representations 1222 corresponding to participants in the live communication session (eg, each member of The Dream Team group). Key representation 1210B includes live video from participant Allison, indicating Allison's participation in a live communication session with video data.

Primary representations 1201A, 1210C, and 1210D include placeholders that indicate that the corresponding participant is not participating in (eg, not connected to) the live communication session. The placeholder contains a textual representation of the participant's name (eg, first name or first and last initial). Optionally, the placeholder has a status indicator indicating that the participant is waiting to be connected to a live communication session and a reminder to send a reminder to the corresponding participant that the live communication session is available. affordances (eg, "outgoing" 808A). In some embodiments, the reminder affordance is not initially included in the placeholder and is displayed after a predetermined amount of time upon determination that the corresponding participant has not participated in the live communication session.

In some embodiments, device 600 detects input corresponding to selection of a reminder affordance (eg, a tap on a reminder affordance) and, in response, creates a new An instruction is sent to cause a notification (eg, call, ring tone, etc.) to occur on the device associated with the corresponding participant, the new notification being different from previous notifications sent to the participant. In some embodiments, the new notification is more intrusive than the previous notification (e.g., audio transmission, haptic output, or full-screen notification compared to banner notification without audio or haptic output). ). In some embodiments, reminder affordances are displayed in overflow representation 1222 (eg, according to the same criteria used to display reminder affordances in the main representation). In some embodiments, selection of a Reminder Affordance on an overflow representation results in the same response as described above for Reminder Affordances displayed on the main representation (e.g., a more intrusive new notification to the corresponding participant). ) is provided.

In some embodiments, in response to determining that one of the participants represented by the placeholder has joined the live communication session, device 600 modifies the representation associated with the user (e.g., replaces the placeholder with Replace with live video, eg, see FIGS. 8C-8D). In some embodiments, modifying the representation includes animation. In some embodiments, device 600 receives data from a server or other device for determining whether participants have joined the session. In some embodiments, device 600 determines that a participant has joined a session based on receiving live media data associated with the participant.

In some embodiments, upon determining that one of the participants represented by the placeholder has not participated in the live communication session for a predetermined amount of time, the device 600 prompts the live communication session user interface 1216 to Change the layout of the representations displayed (e.g., the device 600 stops displaying representations corresponding to participants who have not participated in the live communication session for a predetermined amount of time and displays a layout with fewer key representations). or replace one participant's representation with another participant's representation (see, eg, FIGS. 8E-8F). In some embodiments, device 600 receives data from a server or other device to determine if a participant has not participated in a live communication session for a predetermined amount of time. In some embodiments, device 600 determines if a participant has not participated in a live communication session for a predetermined amount of time based on not receiving live media data associated with the participant.

FIG. 12D shows the live communication user interface 1216 at a later time during a live communication session with the group The Dream Team. As shown in Figure 12D, the live communication user interface 1216 includes a call control menu 1224 similar to the call control menu 630 described above. As previously described, in some embodiments, call control menu 1224 is displayed in response to user input (eg, tapping or swiping up on display 602). As shown in FIG. 12D, device 600 receives (eg, detects) user input 1250 B (eg, swipe up) corresponding to a request to expand call control menu 1224 . In response to user input 1250B, device 600 expands call control menu 1224, as shown in FIG. 12E. Expanded call control menu 1224 is similar to expanded call control menu 630 described with respect to FIGS. 6I-6K. As shown in FIG. 12E, call control menu 1224 includes reminder affordances 1226 associated with participant Hoan. In some embodiments, device 600 detects input corresponding to selection of reminder affordance 1226 (eg, a tap on reminder affordance 1226) and, in response, indicates that a live communication session is available. An instruction is sent to cause a new notification (eg, call, ring tone, etc.) to occur on the device associated with participant Hoan, the new notification being different from previous notifications sent to participant Hoan. In some embodiments, the new notification is more intrusive than the previous notification (e.g., audio transmission, haptic output, or full-screen notification compared to banner notification without audio or haptic output). ).

FIG. 12F again shows the live communication user interface 1216 prior to initiation of a live communication session with the messaging group The Dream Team. As shown in FIG. 12F , device 600 receives (eg, detects) user input 1250C (eg, tap) corresponding to selection of group contact affordance 1208 . As shown in FIG. 12G, in response to receiving user input 1250C, device 600 displays group contact user interface 1228, which displays live communication session affordance 1230 and group participant list 1232. Shows additional information about The Dream Team group, including: Group contact user interface 1228 also includes add contact affordance 1234 . In some embodiments, in response to detecting input (eg, a tap) corresponding to selecting add contact affordance 1234, device 600 outputs (eg, a text Live communication by displaying a user interface with input fields and/or add participant affordances for selecting participants from a menu such as a contact list (e.g., as described below with reference to FIG. 14F) Begin the process of adding a new contact to the session.

As shown in FIG. 12H, device 600 receives (eg, detects) user input 1250D (eg, tap) corresponding to selection of live communication session affordance 1230 . In response to receiving user input 1250D, device 600 launches a live communication application and initiates a live communication session with participants of group The Dream Team (eg, as shown in FIG. 12C).

A user may also select a representation of a group participant on group contact user interface 1228 to access additional information and options regarding the corresponding participant. As shown in FIG. 12I, device 600 receives (eg, detects) user input 1250E (eg, a tap) corresponding to selection of participant Allison's list item 1232B. In response to user input 1250E, device 600 displays a personal contact user with additional information related to the selected participant and options 1238A-1238E for personally communicating with the participant via various modes of communication. Display interface 1236 . As shown in FIG. 12J, personal contact user interface 1236 initiates, among other things, a personal message conversation, a personal phone call, and a personal live communication session (eg, video call) with participant Allison, respectively. includes affordances 1238A, 1238B and 1238C for

In some embodiments, device 600 determines if the characteristic strength of user input 1250E exceeds a threshold strength. As shown in FIG. 12K, upon determining that the characteristic strength of user input 1250E exceeds the threshold strength, device 600 selects options 1242A-1242E corresponding to various modes of communication for personal communication with participant Allison. Concomitantly, menu 1240 is displayed over group contact user interface 1228 .

Figures 12L-12N show exemplary 4:1, 3:1, and 2:1 layouts, respectively. In some embodiments, given that the live communication session includes five participants (including the user of device 600), device 600 displays one of the initial layouts shown in FIG. 12L. In some embodiments, given that the live communication session includes four participants (including the user of device 600), device 600 displays one of the initial layouts shown in FIG. 12M. In some embodiments, given that the live communication session includes three participants (including the user of device 600), device 600 displays one of the initial layouts shown in FIG. 12N. In some embodiments, device 600, based on the number of invited and/or connected participants, as an initial layout when starting a live communication session (e.g., instead of the layout shown in FIG. 12C): Display representations of the participants in one of the layouts shown in Figures 12L-12N. For example, in some embodiments device 600 displays representations of the participants according to one of the layouts shown in FIG. 12M given that the live communication session includes a total of four participants.

13A-13D are flow diagrams illustrating methods according to some embodiments. Method 1300 is performed on a device with a display (eg, 100, 300, 500 or 600). Some acts of method 1300 are optionally combined, the order of some acts are optionally changed, and some acts are optionally omitted.

As described below, method 1300 provides an intuitive way to initiate and interact with a live communication session between multiple participants. This method creates a more efficient human-machine interface by reducing the user's cognitive burden by providing intelligent initiation and interaction with a live communication session between multiple participants. For battery-operated electronic devices, the ability for users to interact and initiate live communication sessions faster and more efficiently saves power and extends battery charging intervals.

At block 1302, the device displays a messaging user interface (eg, 1200) of the messaging application. The messaging user interface includes a message area (eg, 1202) that includes multiple messages between participants in a message conversation that includes three or more participants. The messaging user interface provides an affordance (e.g., 1206) for starting a live video communication session in the live video communication application (e.g., message detail affordance or live video communication application graphics) separate from the message area (e.g., 1202). Affordances with indication). In some embodiments, the affordance is an all-call affordance (eg, 1206) that immediately initiates a live communication session with all participants in the messaging application. In some embodiments, the affordance is a group contact affordance (eg, 1208) that displays a group contact user interface for inviting participants to a live communication session.

At block 1340, the device detects an input (eg, 1250A) corresponding to selection of an affordance (eg, 1206). At block 1306, the device initiates a process of starting a live video communication session in the live video communication application in response to detecting input corresponding to the affordance selection, the live video communication session including participants in the message conversation. . In some embodiments, a live video communication application is launched (eg, 1216) and a new live video communication session is started directly in response to the affordance selection. In some embodiments, a messaging application menu (e.g., 1228) is displayed in response to selecting an affordance, and a menu option (e.g., , 1230) is selected. A user may initiate a live communication session by automatically initiating a process to initiate a live video communication session in a live video communication application when predetermined conditions are met (eg, by detecting input). Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. In addition, by providing additional controllers (e.g., affordances) to initiate the process of initiating live communication without cluttering the UI with displayed additional controllers (e.g., group chat UI), the device's operability is enhanced, making the user-device interface more efficient (e.g., by assisting the user in providing appropriate input and reducing user error when operating/interacting with the device); In addition, by allowing the user to use the device more quickly and efficiently, the device uses less power and improves battery life.

Optionally, at block 1308, the device transitions from the messaging user interface (eg, 1200) to the live communication application user interface (eg, 1216) (eg, transitions immediately upon detection of user input). Optionally, at block 1310 the device initiates a live communication session. Optionally, at block 1312, the device displays the group contacts user interface (eg, 1228) and displays the group contacts user interface (eg, messaging user interface) while continuing to display at least a portion of the group contacts user interface. (or a separate interface that replaces the messaging user interface) includes a group contact affordance (eg, 1230) for initiating a live communication session. A user immediately initiates a live communication session by automatically transitioning from the messaging user interface and starting a live video communication session in the live video communication application when predetermined conditions are met (e.g., by detecting input) be able to. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. In addition, transitioning from the messaging user interface to starting a live video communication session in the live video communication application when predetermined conditions are met reduces the number of inputs required to perform an operation. Device operability is enhanced by reducing the number of inputs required to perform an operation, and the user-device interface (e.g., user interaction) to provide appropriate input when operating/interacting with the device. and reduce user error), as well as enabling users to use devices more quickly and efficiently, thereby reducing device power usage and improving battery life. be done.

In some embodiments, the group contact user interface displays a first participant affordance (e.g., 1232B) associated with a first one of the participants of the message conversation (e.g., a personal participant list). It further includes a plurality of participant affordances (eg, 1232) including: In some embodiments, the group contact user interface further includes a new contact affordance (eg, 1234). Optionally, at block 1314 the device initiates a live communication session in the live communication application. Device usability by displaying a group contact user interface that provides additional control options (e.g., affordances) when additional controllers are needed without cluttering the UI with additional controllers being displayed. is enhanced, making the user-device interface more efficient (e.g., by assisting the user in providing appropriate input and reducing user error when operating/interacting with the device); , which reduces device power usage and improves battery life by allowing users to use devices more quickly and efficiently.

Optionally, at block 1316, the device selects a first one of the live communication session participants (e.g., a participant other than the user of the electronic device, not participating in the communication session and not participating in the communication session). placeholders (e.g., 1210A in FIG. 12C) (e.g., graphic placeholders (e.g., images or virtual avatars), text placeholders (e.g., names or initials) associated with participants sent invitations to participate in Display a live communication session user interface (e.g., 1216) (e.g., an interface for a live communication placeholder) including (e.g., first including) In some embodiments, the live communication session user interface includes a second Further includes a second placeholder associated with the participants, hi some embodiments, the first placeholder is a first reminder affordance for a first one of the live communication session participants (e.g. , call button) (eg, FIG. 12C) In some embodiments, the live communication session user interface further includes a second placeholder associated with the second participant (eg, 1210C). In some embodiments, the first placeholder and the second placeholder are arranged according to a first layout, and by displaying the live communication session user interface when the live communication is initiated. The user is provided with feedback regarding the current state of the live communication interface, and visual feedback is provided to the user indicating that an operation related to the live communication interface may be performed.By providing the improved visual feedback to the user, Device usability is enhanced, making the user-device interface more efficient (e.g., by assisting the user in providing appropriate input and reducing user error when operating/interacting with the device) In addition, by allowing the user to use the device more quickly and efficiently, the device uses less power and improves battery life.

Optionally, at block 1318, upon determining that the first participant has joined the live communication session, the device displays the first placeholder (eg, 1210A in FIG. 12C) to the first Replace with a representation of the participant (eg, 1210A in FIG. 12D) (eg, the placeholder can accompany the participant's name). In some embodiments, an animation occurs (eg, Figures 8T-8W). In some embodiments, the device receives data from a server or other device to make this determination. In some embodiments, the device makes this determination because it is not displaying any representation of the user. By modifying the display of the first placeholder with representations of the participants, the user can quickly recognize which users have joined the call, and feedback to the user on the current state of the live communication session. Visual feedback is provided to the user indicating that certain actions are being taken in connection with the notification and/or live communication session. By providing improved visual feedback to the user, the usability of the device is enhanced and the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at block 1320, upon determining that the first participant has not participated in the live communication session for a predetermined amount of time (eg, has not participated in the live communication after a predetermined amount of time), the device stops displaying the first placeholder (optionally at block 1322), and optionally at block 1324, the device arranges the placement of the second placeholder in a different layout than the first layout. 2 layout (eg, display a layout where the first participant is not included in the layout). In some embodiments, the device may receive data from a server or other device to make this determination. In some embodiments, the device may make this determination because it is not displaying any representation of the user. By stopping the display of placeholders and rearranging the displays when predetermined conditions are met, users can maximize the visibility of displayed participants and/or placeholders while allowing other users to You can quickly recognize when you are participating in a call or when you are not. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. Additionally, by discontinuing the display of placeholders and changing the placement of the display, feedback is provided to the user regarding the current state of the live communication session, and notification and/or specific actions related to the live communication session are performed. Visual feedback is provided to the user to indicate that the Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

Optionally, at blocks 1326 and 1328, the device receives input (eg, 850B) corresponding to selection of the first reminder affordance (eg, 808). In response to receiving input corresponding to the selection of the first reminder affordance, the device sends a new notification (e.g., call, ring tone, etc.) different from previous notifications sent to the first participant. Send to the first participant (eg, the second notification is more coherent than the first notification). By sending a notification to the first participant that is different from the previous notification sent to that participant in response to receiving the input corresponding to the selection of the reminder affordance, the input required to perform the operation. decrease in number. By reducing the number of inputs required to perform an operation (e.g., the number of options required for a user to invoke a participant), the usability of the device is enhanced and the user-device interface (e.g., the device more efficient (by assisting the user in providing appropriate input and reducing user error when operating/interacting with the device), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. In addition, by automatically performing actions such as sending notifications that differ from the first notification, without further user input, the operability of the device is enhanced and the user-device interface (e.g., device (by assisting the user in providing appropriate input when operating/interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently Doing so reduces device power usage and improves battery life.

Optionally, at blocks 1330 and 1332, the device receives input (eg, 1250C) corresponding to selection of a group contact affordance (eg, 1208) to initiate a live communication session. In response to detecting input corresponding to selecting a group contact affordance to initiate a live communication session, the device initiates a live communication session in the live communication application. (eg, transition to the user interface of a live communication application). In some embodiments, the group contact user interface includes a participant list (eg, 1232) for live communication sessions. In some embodiments, the participant list is provided without providing options and/or affordances to personally join or initiate a video or audio conference with a given participant. be. In some embodiments, the messaging user interface provides two affordances for starting a live video communication session in a live video communication application: a first affordance that, when selected, results in display of a group contacts user interface; and a second affordance that, when selected, causes transition to and initiation of a live communication application user interface (eg, immediately starting a live communication session). Live communication in a live communication application upon selection of group contact affordances that provide additional control options (e.g., affordances) when additional controllers are needed without cluttering the UI with additional controllers displayed Starting a session enhances the usability of the device, improves the user-device interface (e.g., assists the user in providing appropriate input when operating/interacting with the device, and corrects user error). In addition, by allowing the user to use the device more quickly and efficiently, the device uses less power and improves battery life. Additionally, initiating a live communication session in a live communication application in response to selection of a group contact affordance can reduce the number of inputs required to perform an operation. Device operability is enhanced by reducing the number of inputs required to perform an operation, and the user-device interface (e.g., user interaction) to provide appropriate input when operating/interacting with the device. and reduce user error), as well as enabling users to use devices more quickly and efficiently, thereby reducing device power usage and improving battery life. be done.

Optionally, at blocks 1334, 1336, 1338, and 1340, after initiating communication, the device responds to determining that the first participant has not participated in the live communication session for a predetermined amount of time: Display reminder options associated with the first participant. In some embodiments, the reminder option is displayed above or near the first participant affordance of the group contact user interface. By displaying reminder options associated with participants when predetermined conditions are met, users can quickly recognize users who are not participating in the call and call those particular users. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. While displaying reminder options associated with the first participant, the device receives input corresponding to selection of the reminder option. In response to receiving input corresponding to selecting the reminder option, the device then sends a notification (e.g., to the first participant's device) indicating that a live communication session is available to the first participant. For example, send an audible, visible, tactile) notification that causes the notification to be output. Sending a notification to the first participant in response to receiving input corresponding to selection of a reminder option associated with that participant reduces the number of inputs required to perform an operation. By reducing the number of inputs required to perform an operation (e.g., the number of options required for a user to invoke a participant), the usability of the device is enhanced and the user-device interface (e.g., the device more efficient (by assisting the user in providing appropriate input and reducing user error when operating/interacting with the device), as well as enabling the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at blocks 1342 and 1344, the device detects input (eg, 1250E) corresponding to selection of a first participant affordance (eg, 1232B) associated with the first participant in the message conversation. . In some embodiments, these participants are participants in a live communication session (e.g., participants in a live communication session can include one or more users that are different from the users from the message conversation). , may not include participants from the message conversation). In response to detecting input corresponding to selection of a first participant affordance associated with a first participant in the message conversation, the device displays a personal participant user interface (e.g., 1236 ) (eg, the individual participant user interface has details about the individual user (eg, user identifier such as name, initials, user contact information, communication mode for contacting the user)). On the individual participant user interface (e.g., group contact UI) in response to input detection without cluttering the UI (e.g., group contact UI) with additional controllers displayed when additional controllers are needed. (Participant Affordances for each individual participant) enhances device usability and aligns the user-device interface (e.g., to provide appropriate input when operating/interacting with the device). (by assisting the user in the process and reducing user error), and by allowing the user to use the device more quickly and efficiently, reducing device power usage and improving battery life. is improved.

Optionally, at blocks 1346 and 1348, the device detects an input (eg, an input having a characteristic strength exceeding a strength threshold) corresponding to selecting an option for selecting a communication mode (eg, 1238A-1238C ). Upon detecting input corresponding to selection of a communication mode option, the device initiates the process of selecting a commute mode for the first participant. In some embodiments, initiating the process includes displaying multiple communication option affordances (eg, mail affordances, call affordances, messaging affordances).

Optionally, at blocks 1350 and 1352 the device detects input corresponding to selection of a new contact affordance (eg, 1234). In response to detecting input corresponding to selection of a new contact affordance, the device initiates the process of adding the new contact to the live communication session (e.g., user interface). By initiating the process of adding new contacts to a live communication session, users can quickly recognize which users are being added to the call, and feedback is provided to the user on the current status of the live communication session. Visual feedback is provided to the user indicating that certain actions are being taken in connection with the notification and/or live communication session. By providing improved visual feedback to the user, the usability of the device is enhanced and the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

It should be noted that the details of the processes described above with respect to method 1300 (eg, FIGS. 13A-13D) are also applicable to the methods described below/above in an analogous manner. For example, method 1300 optionally includes one or more of the various method features described with respect to methods 700 , 900 , 1100 , or 1500 .

14A-14I illustrate exemplary user interfaces for, among other things, initiating a live communication session from a live communication application, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIGS. 15A-15C.

FIG. 14A shows a device 600 (as described above) displaying a user interface 1400 (eg, home screen or springboard) with affordances for launching applications. As shown in FIG. 14A, device 600 receives (eg, detects) user input 1450A (eg, tap) corresponding to selection of live communication application affordance 1402 .

As shown in FIG. 14B, in response to receiving user input 1450A, device 600 launches the live video communications application and displays live video communications user interface 1404 of the live video communications application. The live video communication user interface includes a list 1408 of participant affordances 1408A-1408K corresponding to each participant or group of participants. In some embodiments, based on previous (e.g., recent) live communication sessions with each participant or group of participants (e.g., two or more participants other than the user of device 600) in a live communication application , participant affordances are included in list 1408 . In some embodiments, a previous live communication session includes communications that were successfully connected and attempted by either the user of device 600 or another participant. Live video communication user interface 1404 includes new session affordance 1406 for selecting participants for a new live communication session.

In some embodiments, another application (e.g., a messaging application (e.g., affordance 1408A, e.g., group chat participant list), a phone application (e.g., affordance 1408C, e.g., participants in a conference call), a calendar application (e.g., affordance 1408B, e.g. Participant affordances are included in list 1408 based on participant groups associated with an email application (affordance 1408D, e.g., email recipient list)). For example, participant affordance 1408A corresponds to a participant group of a previous or existing group messaging conversation (e.g., text or instant messaging) and is displayed as an offer in list 1408 (e.g., a user may select a participant group does not need to be created manually). In some embodiments, a participant affordance corresponds to a participant group of a conference call call associated with the telephony application. Participant affordances 1408A-1408K include a representation of a participant or group of participants, status indicators, detailed affordances, and optionally associated dates. As shown in FIG. 14B, the status indicator may be the source of the participant or group (e.g., "From Messages", "From Phone", "From Calendar", "From Mail") or the communication status of the associated communication (e.g. , video, audio, absence, etc.). In some embodiments, the source of participants or groups is indicated by a graphical or textual display, a color associated with a particular application, or a logo for the application (eg, phone icon, video camera icon, or speech bubble icon).

In some embodiments, device 600 visually distinguishes participant affordances associated with groups in which active live communication sessions exist. In FIG. 14B, participant affordance 1408E is associated with the group whose live communication session is currently active. An active live communication session is shown by the name of the participant in bold text in participant affordance 1408E compared to the non-bold text and contact detail icon (e.g., 1408A-1) displayed in other participant affordances; Indicated by a video camera icon 1408E-1 and a "tap to join" message.

In some embodiments, the appearance of participant affordances associated with previous live communication sessions is based on whether the user of the device participated in the corresponding live communication session. In FIG. 14B, participant affordance 1408H relates to a previous live communication session in which the user of device 600 did not participate. As a result, participant affordance 1408H separates the italicized text from the non-italicized text of other participant affordances (e.g., 1408F) associated with previous live communication sessions in which the user of device 600 participated. include.

As shown in FIG. 14C, device 600 receives user input 1450B (eg, a tap) corresponding to selection of participant affordance 1408A for the aforementioned participant group, The Dream Team, which includes fifteen participants (eg, a tap). To detect).

In response to receiving user input 1450B, device 600 initiates the process of starting a new live video communication session with the participant corresponding to selected participant affordance 1408A (The Dream Team). In some embodiments, a new live video communication session is started immediately upon selection of participant affordance 1408A. For example, as shown in FIG. 14D, in response to receiving user input 1450B, device 600 displays live communication user interface 1410 and initiates a live communication session with group The Dream Team without further user input. to start. The live communication user interface 1410 includes a representation of the user of the device 600 (eg, video from the front-facing camera of the device 600) and a call control menu 1412 similar to the call control menu 630 discussed.

In some embodiments, in response to selection of participant affordance 1408A, a menu of live video communication applications is displayed and a menu option is selected to initiate a new live video communication session. For example, in some embodiments, device 600 displays live communication user interface 1410 (as shown in FIG. 14D) without initiating a live communication session. Instead, while displaying the live communication user interface 1410, the device 600 receives (eg, detects) user input (eg, a tap) corresponding to selection of the call affordance 1414 on the call control menu 1412, In response to selection of live communication initiation affordance 1414, a live communication session is initiated with the group The Dream Team.

FIG. 14E also shows live communication user interface 1404 . Instead of receiving a selection of a participant affordance in list 1408 (as shown in FIG. 14C), device 600 receives (eg, detects) user input 1450C (eg, tap) corresponding to selection of new session affordance 1406. ). In response to user input 1450C, and in response to user input 1450C corresponding to selection of new session affordance 1406, device 600 selects a participant and creates a new session between the selected participant and the user of device 600. Start the process of starting a live video communication session. As shown in FIG. 14F, the live communication user interface 1404 has been replaced by a user interface 1416, which includes a text entry field 1418 for entering the participant's name using a keyboard 1420, and a menu (e.g., an add-participant affordance 1422, an audio-only affordance 1424, and a video affordance 1426 for selecting participants from a contact list).

14F, in response to entering the text "JOE" into text entry field 1418, device 600 displays two communication options 1417A and 1417B for contact "JOE SMITH." In some embodiments, device 600 visually distinguishes communication options associated with devices that can participate in a live communication session. In FIG. 14F, the communication mode (eg, email) represented by communication option 1417B relates to devices that can participate in a live communication session, as indicated by the bolded text. In contrast, the communication mode represented by communication option 1417A (e.g., home phone) is shown in normal (non-bold) text to indicate that the communication mode is not related to devices that can participate in a live communication session. Is displayed.

As shown in Figure 14G, the participants named Joe, Ashley, and Sam have been selected. Device 600 receives (eg, detects) user input 1450D (eg, tap) corresponding to selection of video affordance 1426 .

In response to receiving user input 1450D, device 600 initiates a new live communication session with the selected participant (as shown in FIG. 14H) and provides live media streams including video and audio data. do. In some embodiments, in response to receiving user input corresponding to selection of audio-only affordance 1424, device 600 initiates a new live communication session with the selected participant, accompanied by video data. provide a live media stream that includes audio data.

In some embodiments, after a new live communication session has started, device 600 displays a representation of the new live communication session in an application other than the live communication application. After initiating a live communication session with participants Joe, Ashley, and Sam, the user of device 600 navigates to the messaging application. As shown in FIG. 14I, after initiating a live communication session, device 600 displays messaging interface 1428 of the messaging application. Messaging user interface 1428 displays a group message conversation including participants in a live communication session, and message 1430 is displayed in message area 1432 indicating that the live communication session has started. In some embodiments, after initiating a live communication session, device 600 displays a group including participants in the live communication session in a list of message conversations in the messaging application. In some embodiments, device 600 displays messaging user interface 1428 (described above) in response to selection of a group in a list of message conversations.

15A-15C are flow diagrams illustrating methods according to some embodiments. Method 1500 is performed on a device with a display (eg, 100, 300, 500 or 600). Some acts of method 1500 are optionally combined, the order of some acts are optionally changed, and some acts are optionally omitted.

As described below, method 1500 provides an intuitive way to initiate and interact with a live communication session between multiple participants. This method creates a more efficient human-machine interface by reducing the user's cognitive burden for providing intelligent initiation and interaction with a live communication session between multiple participants. For battery-operated electronic devices, the ability for users to interact and initiate live communication sessions faster and more efficiently saves power and extends battery charging intervals.

At block 1502, the device displays a live video communications user interface (eg, 1404) of the live video communications application. At block 1504, the live video communication user interface includes a plurality of participant affordances (eg, 1408A-1408K) each corresponding to one or more participants, the plurality of participant affordances representing groups of three or more participants. includes a first participant affordance (eg, 1408A) corresponding to . At block 1506, the live video communication user interface includes a new session affordance (eg, 1406) for starting a new live video communication session. In some embodiments, a participant affordance is a previous (e.g., recent) video communication (e.g., successfully connected (by a participant using the device or associated with an external device) or attempted) and includes an indication (eg, 1408F) of the participant(s) of the video communication. In some embodiments, a participant affordance represents a group associated with another communication application (eg, a messaging application or a telephony application (eg, recent conference calls)). In some embodiments, the live communication user interface includes a list of recent live communication sessions (eg, a list of recent video chats can be individual and/or group chats). By displaying a live communication user interface that includes a list of recent live communication sessions, the user can quickly view recent live communication sessions in which the user or device has been involved (e.g., invited, attended, communicated, etc.). The access and session can be implemented, feedback is provided to the user regarding the current state of the live communication session, and visual feedback is provided to the user indicating that certain actions related to the live communication session will be performed. provided. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

In some embodiments, multiple participant affordances are determined based on data (e.g., usage data) from a second application (e.g., telephony application, messaging application) different from the live video communication application. Include a second participant affordance (eg, 1408B) corresponding to a suggested participant group (eg, a suggested participant group that was not manually selected/created by the user).

In some embodiments, the suggested group of participants are participants (e.g., 1408A) of a group conversation (e.g., group messaging session, group conference call) conducted in the second application or Corresponds to a participant (eg, 1408C) in the conference call that was made. By displaying a live communication user interface that includes affordances corresponding to proposed groups of participants, users can quickly access and initiate live communication with proposed groups and select groups. Reduces the number of inputs required to perform operations related to setting up a live communication session involving each member of the Device operability is enhanced by reducing the number of inputs required to perform an operation, and the user-device interface (e.g., user interaction) to provide appropriate input when operating/interacting with the device. and reduce user error), as well as enabling users to use devices more quickly and efficiently, thereby reducing device power usage and improving battery life. be done. In addition, displaying affordances corresponding to suggested groups provides additional control operations without cluttering the UI with additional controllers being displayed. By providing additional control operations without cluttering the UI with additional displayed controllers, the usability of the device is enhanced and the user-device interface (e.g., when operating/interacting with the device is appropriately more efficient (by assisting the user in providing correct input and reducing user error), as well as reducing the power usage of the device by allowing the user to use the device more quickly and efficiently. reduced and battery life improved.

In some embodiments, the second participant affordance is a display (e.g., a graphical or textual display (e.g., "suggested from message") of the second application (e.g., the source application for the suggestion); application-specific colors, or application-related logos). A user can live communicate with a proposed group by displaying a live communication user interface further comprising affordances corresponding to the proposed group of participants, corresponding to participants in the group conversation in an application different from the live communication application. to quickly access and initiate communications and to perform operations related to setting up a live communication session involving each member of the selected group (e.g. navigating different applications) decrease in the number of Device operability is enhanced by reducing the number of inputs required to perform an operation, and the user-device interface (e.g., user interaction) to provide appropriate input when operating/interacting with the device. and reduce user error), as well as enabling users to use devices more quickly and efficiently, thereby reducing device power usage and improving battery life. be done. In some embodiments, the multiple participant affordances are from a third application (e.g., telephony application, messaging application, email application, calendar application) that is different from the live video communication application and different from the second application. data (e.g. usage data) (e.g., suggested participant groups not manually selected/created by the user) (e.g. group or groups from recent mail events). By displaying a second participant affordance that includes an indication of the application from which the proposed group originated, feedback is provided to the user regarding the current state of the live communication session, and specific actions associated with the second application are provided. Visual feedback is provided to the user to indicate. By providing improved visual feedback to the user, the usability of the device is enhanced and the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be. In some embodiments, the live communication user interface further includes a participant selection affordance (eg, "+").

At block 1508, the device detects input (eg, 1450B) corresponding to a request to initiate processing to initiate a new live video communication session. At blocks 1510, 1512, and 1514, in response to detecting the input, the device selects a group of three or more participants in response to the input corresponding to selection of a first participant affordance (eg, 1408A). Initiate the process of starting a new live video communication session, including: In some embodiments, a new live video communication session is started immediately upon selection of the first participant affordance (eg, FIG. 14D). In some embodiments, in response to selecting the first participant affordance, a menu of live video communication applications (eg, 1416) is displayed and a menu option is selected to initiate a new live video communication session. be. In response to detecting input, the device initiates the process of selecting two or more participants in response to the input corresponding to the selection of the new session affordance, and the participant associated with the device and the two or more participants. selected participants (eg, FIG. 14F). In some embodiments, the user interface provides a first affordance (e.g., 1422) for selecting participants and initiates a live video communication session (e.g., a live a second affordance (eg, 1424 and 1426) for immediately starting a video communication session). In some embodiments, participants are selected by entering contact information into a text field (eg, 1418) or selecting a contact from a list of contacts. In some embodiments, a new live video communication session is started immediately upon selection of an affordance (eg, 1408A). By initiating the process of initiating a new live communication when predetermined conditions are met, the device can provide different live video communication sessions to the user based on the number of participants selected. Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. Additionally, by initiating a different process to initiate a new live communication when predetermined conditions are met, the number of inputs required to perform an operation is reduced. Device operability is enhanced by reducing the number of inputs required to perform an operation, and the user-device interface (e.g., user interaction) to provide appropriate input when operating/interacting with the device. and reduce user error), as well as enabling users to use devices more quickly and efficiently, thereby reducing device power usage and improving battery life. be done.

Optionally, at blocks 1516, 1518, 1520, 1522, 1524, and 1526, the device receives a second input corresponding to selection of participant selection affordances. In response to receiving a second user input corresponding to selection of a participant selection affordance, the device displays a text entry field (eg, 1418) (eg, To: field) for identifying one or more participants. ), keyboard (eg, 1420), audio-only affordances (eg, 1424), and video affordances (eg, 1426). While displaying the group selection user interface, the device receives a third user input selection. In response to receiving the third user input, the device transmits video data from the electronic device without transmitting video data from the electronic device in response to the third user input corresponding to the selection of the audio-only affordance. start a new live communication session with audio received (eg, audio recorded by one or more microphones of the device). By initiating a new live communication session with audio transmitted from the electronic device when predetermined conditions are met (e.g., in response to the selection of audio-only affordances), the user is enabled to add additional inputs to the device. You can participate in a live communication session with audio without having to select or modify the output (eg, turn off the camera and/or turn on the microphone). Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life. In some embodiments, the system displays an indicator that the user has participated in a live communication using audio only. In some embodiments, the device displays a phrase such as "Do you want to connect using video?" on the user display to determine if the user wants to use audio only or video and audio. This can give the user peace of mind that the video is not being shared). In response to receiving the third user input, and in response that the third user input corresponds to the selection of the video affordance, the device performs a new video using the audio and video transmitted from the electronic device. start a live communication session with By initiating a live communication session using audio and video transmitted from the electronic device when predetermined conditions are met (e.g., in response to the video affordance option being selected), the user can Live communication sessions with audio and video can be participated in without having to select or modify additional inputs/outputs (eg, turning on the camera and/or turning on the microphone). Device operability is enhanced by taking optimized actions when a set of conditions are met, without the need for further user input, to improve the user-device interface (e.g., / make it more efficient (by assisting the user in providing appropriate input when interacting with the device, reducing user error), as well as allowing the user to use the device more quickly and efficiently This reduces device power usage and improves battery life.

Optionally, at blocks 1528 and 1530 the device initiates a new live communication session. After initiating the new live communication session, the device displays a representation of the new live communication session in a fourth application (eg, messaging or phone application) different from the live communication session. In some embodiments, live communication participants are displayed as group participants of group messages in a list of message conversations in the messaging application. In some embodiments, group messages in the list of message conversations include an indication of whether there are active live communication sessions involving group participants (eg, 1408E). In some embodiments, the live communication participants are displayed as group participants of the group call in the list of recent calls in the phone application or call application. In some embodiments, live communication participants are displayed as calendar event invitees in calendar applications. Displaying a representation of the live communication session in a different application than the live communication session provides feedback to the user regarding the current state of the live communication session in multiple applications, allowing the user to simply, in some embodiments, view the representation. (e.g., later starting another new live communication session in a live communication session from a different application with the same participants who were in the previous live communication session), notification and/or Visual feedback is provided to the user indicating that certain actions related to the live communication session are taking place. Providing improved visual feedback to the user enhances the usability of the device and improves the user-device interface (e.g., the user to provide appropriate input when operating/interacting with the device). It reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently), as well as by helping the user reduce user error. be.

In some embodiments, the plurality of participant affordances (eg, 1408) includes fourth participant affordances (eg, 1408A, 1408E) corresponding to groups of three or more participants. In some embodiments, following a determination that the fourth participant affordance corresponds to an active live communication session (eg, 1408E), the fourth participant affordance has a first visual appearance (eg, The first visual appearance is displayed with the font color or font style of the participant affordance, or an icon or glyph displayed at or near the participant affordance), and the fourth participant affordance is active. (e.g., 1408A), the fourth participant affordance has a second visual appearance that differs from the first visual appearance (e.g., a second visual appearance is a different font color or style than the first visual appearance, or the second visual appearance does not include icons or glyphs displayed near the representation of the potential participant, or (including icons or glyphs that differ from the visual appearance of 1). In some embodiments, the appearance of participant affordances associated with a previous live communication session is based on whether the user of the device participated in the corresponding live communication session (e.g., missed live communication session participants The affordance (e.g., 1408H) is different from or included in the participant affordance (e.g., 1408F) of the live communication session in which the user of device 600 participated. etc., where the participant affordance contains text of one color if the user of the device joined the session, and text of a different color if the user of the device did not join the session before the session ended. including). In some embodiments, the appearance of participant affordances associated with an active live communication session changes when the session is no longer active (e.g., from bold text and a video camera icon to non-bold text without a video camera icon). to the text of the notation).

In some embodiments, initiating the process of selecting two or more participants displays representations (e.g., 1417A, 1417B) of potential participants (e.g., contacts in the contact list). including doing In some embodiments, the representation of the potential participant is rendered in a first visual appearance (e.g., a first The visual appearance is displayed with the font color or font style of the potential participant's representation, or an icon or glyph displayed near the potential Following the determination that the potential participant's representation is not relevant to the devices that can participate in the session, the representation of the potential participant is given a second visual appearance that differs from the first visual appearance (e.g., the second visual appearance is Is a different font color or font style than the font color or font style of the first visual appearance, or the second visual appearance includes icons or glyphs displayed near the representation of the potential participant. (including icons or glyphs that are absent or different from the first visual appearance).

It should be noted that details of the processes described above with respect to method 1500 (eg, FIGS. 15A-15C) are also applicable to the methods described above in an analogous manner. For example, method 1500 optionally includes one or more of the various method features described above with respect to methods 700 , 900 , 1100 , or 1300 .

The foregoing description, for purposes of explanation, has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. The embodiments have been chosen and described in order to best explain the principles of the technology and their practical application. Others skilled in the art are thereby enabled to best utilize the techniques and various embodiments with various modifications appropriate to the particular uses envisioned.

Although the disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are intended to be included within the scope of the disclosure and examples as defined by the claims.

As noted above, one aspect of the present technology is the collection and use of data available from various sources to improve multimedia communications. This disclosure indicates that, in some cases, this collected data is personal information that can be used to uniquely identify a particular person or to contact that person or locate that person. It is contemplated that it may contain data. Such Personal Data may include demographic data, location-based data, phone number, email address, Twitter ID, home address, data regarding the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information). or may include records, dates of birth, or any other identifying or personal information.

The present disclosure understands that use of such personal information data of this technology can be used to the benefit of the user. For example, personal information data can be used to facilitate multimedia communication between users. The use of such personal information data thereby enables users to improve such communications. Additionally, other uses of personal information data that benefit the user are also contemplated by this disclosure. For example, health and fitness data may be used to provide insight into a user's overall wellness, or as useful feedback to individuals who use technology to pursue wellness goals. may

This disclosure contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other use of such personal information data adhere to robust privacy policies and/or practices. Specifically, such entities will, at all times, implement privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for the confidentiality and confidentiality of personal information data. should be used. Such policies should be easily accessible to users and updated as data collection and/or use changes. Personal information from users should be collected for the entity's lawful and legitimate use and should not be shared or sold except for those lawful uses. Further, such collection/sharing should only occur after informing and obtaining consent from the user. In addition, such entities protect and secure access to such Personal Data and ensure that others who have access to that Personal Data comply with their privacy policies and procedures. all necessary measures should be taken to Further, such entities may themselves be evaluated by third parties to certify their adherence to widely accepted privacy policies and practices. In addition, policies and practices should be tailored to the specific types of personal information data collected and/or accessed and to be consistent with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, collection or access to certain health data may be governed by federal and/or state laws, such as, for example, the Health Insurance Portability and Accountability Act (HIPAA); may be subject to regulatory and policy ramifications and should be dealt with accordingly. Different privacy practices should therefore be maintained for different types of personal data in each country.

Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively prevents use of or access to personal information data. That is, the present disclosure contemplates that hardware and/or software elements may be provided to prevent or inhibit access to such personal information data. For example, in the case of multimedia communications, the technology is configured to allow users to choose to "opt-in" or "opt-out" of participation in the collection of personal information data during registration for the service or at any time thereafter. can do. In another example, a user may choose not to provide location data and/or availability. In yet another example, the user may choose to limit the length of time recordings of communication sessions are maintained, or to prohibit retention of such data altogether. In addition to providing "opt-in" and "opt-out" options, this disclosure contemplates providing notice regarding access or use of personal information. For example, upon downloading an app, the user may be notified that their personal data will be accessed, and then reminded again just before the personal data is accessed by the app.

Further, it is the intent of this disclosure that personal information data should be managed and handled in a manner that minimizes the risk of unintended or unauthorized access or use. Risk can be minimized by limiting data collection and deleting data when it is no longer needed. Additionally, and where applicable, also in certain health-related applications, data de-identification can be used to protect user privacy. De-identification includes, where appropriate, removing certain identifiers (e.g. date of birth, etc.), controlling the amount or specificity of data stored (e.g. location data rather than address level). city level), controlling how data is stored (e.g., aggregating data across users), and/or in other ways.

Thus, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, this disclosure does not disclose that those various embodiments also It is also contemplated that it can be performed without the need to access such personal information data. That is, various embodiments of the present technology are not rendered inoperable by the lack of all or part of such personal information data. A minimum amount of non-personal data or personal information, such as, for example, content requested by a device associated with a user, other non-personal information available with a communications service provider, or publicly available information; A communication session can be established based solely on

Claims (132)

In an electronic device with a display,
displaying a messaging user interface of a messaging application, the messaging user interface including a message area containing a plurality of messages in a message conversation between three or more participants;
receiving first data indicating that a live communication session available to the three or more participants is active;
displaying a notification indicating that the live communication session is active in response to receiving the first data indicating that the live communication session between the three or more participants is active;
Receiving second data indicating that the live communication session between the three or more participants is no longer active while displaying the notification indicating that the live communication session is active. and,
stop displaying the notification or the live communication session is no longer active in response to receiving the second data indicating that the live communication session between the three or more participants is no longer active; updating the notification to indicate that
A method, including displaying the notification that the live communication session is active includes displaying the notification that the live communication session is active in a first position of the message area; teeth,
2. The method of claim 1, further comprising moving the notification indicating that the live communication session is active from the first location to a second location in the message area. wherein the indication that the live communication session is active includes a live communication participation affordance, the method comprising:
receiving a first user input corresponding to selection of the live communication participation affordance;
Participating in the live communication session in response to receiving the first user input corresponding to selecting the live communication participation affordance;
3. The method of claim 1 or 2, further comprising: the notification indicating that the live communication session is active is displayed without generating a corresponding audio output, the method comprising:
receiving reminder data indicating a reminder to participate in the live communication;
Generating, by one or more audio output units of the device, an audio output indicating that the live communication session is active in response to receiving the reminder data indicating the reminder to join the live communication. ,
4. The method of any one of claims 1-3, further comprising: wherein the notification indicating that the live communication session is active includes a menu access affordance, the method comprising:
receiving a second user input corresponding to selection of the menu access affordance;
displaying a menu including a video join option and an audio only option in response to receiving the second user input corresponding to the menu access affordance selection;
receiving a third user input while displaying the menu;
Participating in the live communication session using audio and video transmitted from the electronic device in response to the third user input corresponding to selection of the video participation option;
joining the live communication session using audio transmitted from the electronic device without transmitting video data from the electronic device in response to the third user input corresponding to selecting the audio-only join option; to participate and
5. The method of any one of claims 1-4, further comprising: 6. A method according to any preceding claim, wherein said notification indicating that said live communication session is active is displayed in said message area. receiving third data indicating that a second live communication session is available between the two or more participants;
displaying a first type notification indicating that the second live communication is active in response to the second live communication session being available between two participants;
A second notification, different from the first type, indicating that the second live communication is active in response to the second live communication session being available between three or more participants. displaying two types of notifications;
7. The method of any one of claims 1-6, further comprising: Displaying the notification of the second type includes:
initially displaying the notification of the second type in a first state;
transitioning the notification of the second type to a second state;
8. The method of claim 7, comprising: receiving fourth data indicating that a third live communication session is available;
in response to receiving the fourth data indicating that the third live communication session is available;
Displaying a third type of notification indicating that the third live communication session is available upon determining that an active application is being displayed;
Displaying a fourth type notification indicating that the third live communication session is available upon determining that no active applications are being displayed;
9. The method of any one of claims 1-8, further comprising: receiving fifth data indicating that a fourth live communication session is available;
in response to receiving fifth data indicating that the fourth live communication session is available;
the fourth live communication session while providing a non-visual notification that the fourth live communication session is available upon determination that the current status of the electronic device meets alert notification criteria; displaying a notification that is available; and
without providing the non-visual notification that the fourth live communication session is available upon determination that the current status of the electronic device does not meet the alert notification criteria; displaying the notification that a live communication session of 4 is available;
10. The method of any one of claims 1-9, further comprising: 11. The method of claim 10, wherein the alert notification criteria include a requirement that the device be locked for the alert notification criteria to be met. 11. The method of claim 10, wherein the alert notification criteria include a requirement that the device not detect user attention to the device for the alert notification criteria to be met. 5. The method of claim 4, further comprising, in response to receiving the reminder data indicating the reminder to participate in the live communication, displaying an indication of the participant in the live communication session that initiated the reminder. . 5. The method of claim 4, wherein the audio output indicating that the live communication session is active is different than audio output associated with telephone call notifications and audio output associated with message notifications. 15. A method according to any preceding claim, wherein said notification includes an indication of the number of participants in said live communication session. 16. The method of any one of claims 1-15, wherein the notification includes an indication of the participant who has performed an action on the display that triggers presentation of the notification. one or more audio output devices of the device in response to receiving the first data indicating that the live communication session between the three or more participants is active; generating a second audio output indicating that the live communication session is active, the second audio output indicating that the live communication session is active, the audio output associated with a telephone call notification and a message notification; 17. A method according to any one of claims 1 to 16, different from audio output. 2. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs comprising: 18. A non-transitory computer-readable storage medium comprising instructions for performing the method of any one of Claims 1-17. a display;
one or more processors;
and a memory storing one or more programs configured to be executed by said one or more processors, said one or more programs being any of claims 1 to 17. An electronic device comprising instructions for performing the method of claim 1. a display;
means for performing the method of any one of claims 1 to 17;
An electronic device comprising: A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs comprising:
displaying a messaging user interface of a messaging application including a message area containing multiple messages in a message conversation between three or more participants;
receiving first data indicating that a live communication session available to the three or more participants is active;
displaying a notification indicating that the live communication session is active in response to receiving the first data indicating that the live communication session between the three or more participants is active;
receiving second data indicating that the live communication session between the three or more participants is no longer active while displaying the notification indicating that the live communication session is active;
stop displaying the notification or the live communication session is no longer active in response to receiving the second data indicating that the live communication session between the three or more participants is no longer active; A non-transitory computer-readable storage medium comprising instructions for updating the notification to indicate that the notification has occurred. a display;
one or more processors;
and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising:
displaying a messaging user interface of a messaging application including a message area containing multiple messages in a message conversation between three or more participants;
receiving first data indicating that a live communication session available to the three or more participants is active;
displaying a notification indicating that the live communication session is active in response to receiving the first data indicating that the live communication session between the three or more participants is active;
receiving second data indicating that the live communication session between the three or more participants is no longer active while displaying the notification indicating that the live communication session is active;
stop displaying the notification or the live communication session is no longer active in response to receiving the second data indicating that the live communication session between the three or more participants is no longer active; an electronic device comprising instructions for updating the notification to indicate that the notification has been received. a display;
means for displaying a messaging user interface of a messaging application including a message area containing a plurality of messages in a message conversation between three or more participants;
means for receiving first data indicating that the live communication session available to the three or more participants is active;
means for displaying a notification indicating that the live communication session is active in response to receiving the first data indicating that the live communication session between the three or more participants is active;
means for receiving, in response to displaying the notification indicating that the live communication session is active, second data indicating that the live communication session between the three or more participants is no longer active;
stop displaying the notification or the live communication session is no longer active in response to receiving the second data indicating that the live communication session between the three or more participants is no longer active; updating the notification to indicate that
An electronic device comprising: In an electronic device with a display,
receiving a request to display a user interface for a live communication session between two or more participants;
simultaneously displaying representations of multiple participants in the live communication session in response to receiving the request to display the user interface for the live communication session between the two or more participants. displaying a live communication user interface;
and displaying representations of the plurality of participants in the live communication session, comprising:
Upon determining that the number of participants in the live communication session is greater than a threshold number of participants, reducing representations of a first of the plurality of participants other than a user of the electronic device below a respective threshold size. simultaneously displaying at one or more sizes larger than the first plurality of participants, while displaying in an overflow region a second plurality of representations of the participants, including one or more participants not included in the first plurality of participants; representations of the one or more other participants displayed in the overflow region are displayed at a size smaller than the respective threshold size;
One or more representations of the participants other than the user of the electronic device that are equal to or less than the respective threshold size upon determining that the number of participants in the live communication session is equal to or less than the threshold number of participants. displaying representations of the participants other than the user of the electronic device at one or more sizes greater than the respective threshold size, without displaying at a size of
A method, including updating the display of the representations of the first plurality of participants at a first rate;
updating the display of the representations of the second plurality of participants within the overflow region at a second rate lower than the first rate;
25. The method of claim 24, further comprising: the representations of the first plurality of participants are displayed in two or more different sizes;
26. A method according to claim 24 or 25, wherein said representations of said second plurality of participants within said overflow area are displayed with the same size. 27. The method of claims 24-26, further comprising displaying a representation of a camera view of the electronic device in the overflow area while displaying the representation of the second plurality of participants in the overflow area. A method according to any one of paragraphs. detecting an input corresponding to the overflow region;
In response to detecting the input corresponding to the overflow region,
scrolling the overflow region to cease displaying a subset of the representations of the second plurality of participants;
maintaining the display of the representation of the view of the camera of the electronic device within the overflow area while scrolling the overflow area;
28. The method of claim 27, further comprising: Displaying representations of the plurality of participants in the live communication session includes:
at least one of the representations of the participants other than the user of the device upon first determining that the number of participants in the live communication session is less than a second threshold number of participants; display with an aspect ratio of
at least one of the representations of the participants other than the user of the device upon determining that the number of participants in the live communication session is greater than the second threshold number of participants; displaying at a second aspect ratio different from the first aspect ratio;
29. The method of any one of claims 24-28, further comprising: Displaying representations of the plurality of participants in the live communication session includes:
displaying a representation of a camera view of the device in a third aspect ratio in response to determining that the number of participants in the live communication session is less than a second threshold number of participants;
the representation of the view of the camera of the device to the third aspect ratio upon determining that the number of participants in the live communication session is greater than or equal to the second threshold number of participants; displaying at a different fourth aspect ratio;
30. The method of any one of claims 24-29, further comprising: receiving data indicating that the number of participants in the live communication session has increased while displaying representations of the participants other than the user of the device;
in response to receiving said data indicating that the number of participants in said live communication session has increased;
reducing the size of one or more of the representations of participants other than the user of the device;
displaying one or more representations of new participants in the live communication session;
31. The method of any one of claims 24-30, further comprising: Displaying representations of the plurality of participants in the live communication session includes:
displaying the representation of the participants in a first layout in response to the number of participants in the live communication session being a first number of participants;
displaying the representation of the participants in a second layout different from the first layout in response to the number of participants in the live communication session being a second number of participants;
32. The method of any one of claims 24-31, further comprising: 33. The method of claim 32, wherein the first layout is one of a plurality of predetermined layouts in which user representations are fixed at different positions of the user interface. 34. A method according to claim 32 or 33, wherein said first layout and said second layout are included in a family of predetermined layouts having similar relative placement of user representations. 35. Any of claims 24-34, wherein the live communication session user interface further comprises a placeholder associated with a first participant who has been invited to the live communication session but has not yet joined the live communication session. 1. The method according to item 1. The placeholder includes a first reminder affordance for the first participant, the method comprising:
receiving input corresponding to selection of the first reminder affordance;
Sending a new notification to the first participant that is different from previous notifications sent to the first participant in response to receiving the input corresponding to selection of the first reminder affordance. and,
36. The method of claim 35, further comprising: The placeholders and the representations of the first plurality of participants are arranged in a first layout, the method comprising:
Upon determining that the first participant has not participated in the live communication session for a predetermined amount of time,
ceasing to display the placeholder;
changing the arrangement of the representations of the first plurality of participants to a second layout different from the first layout, including enlarging and/or moving one or more of the representations of the participants. and
37. The method of claim 35 or 36, further comprising: 38. The method of claims 35-37, further comprising modifying said display of said placeholder to represent said first participant upon determining that said first participant has participated in said live communication session. A method according to any one of paragraphs. Displaying representations of the plurality of participants in the live communication session includes:
none of the plurality of representations of remote participants overlap each other following a determination that the number of participants in the live communication session is less than a second threshold number of participants;
at least a portion of the plurality of representations of the remote participants overlapping each other following a determination that the number of participants in the live communication session is greater than or equal to the second threshold number of participants;
39. The method of any one of claims 24-38, comprising detecting a change in the number of participants in the live communication session while displaying representations of the plurality of participants in the live communication session;
responsive to detecting the change in the number of participants in the live communication session, displaying a notification indicating that the number of participants in the live communication session has changed;
40. The method of any one of claims 24-39, further comprising: detecting an input corresponding to a selection of one of the representations of the second plurality of participants within the overflow region;
in response to detecting the input corresponding to selection of one of the representations of the second plurality of participants within the overflow region;
replacing the representation of one representation of the representations of the first plurality of the participants with representations of the participant corresponding to the selected representation within the overflow region;
41. The method of any one of claims 24-40, further comprising: replacing the representation of one of the representations of the first plurality of the participants with the representation of the participant corresponding to the selected representation in the overflow area in the first 42. The method of claim 41, comprising displaying a first animation effect affecting said one of representations of a plurality of participants. Replacing the representation of the representation of the representations of the first plurality of participants with the representation of the participant corresponding to the selected representation in the overflow region is performed in the second plurality of representations. 43. The method of claim 42, further comprising displaying a second animation effect that shares animation properties with the first animation effect that affects the selected ones of the representations of the participants of the Method. 44. Any one of claims 24 to 43, further comprising displaying a novel representation of the second plurality of participants in the overflow area further in response to detecting the input corresponding to the overflow area. The method described in section. at the end of the representation of the second plurality of participants in the sequence within the overflow region in response to receiving data indicating that a new participant has joined or been added to the live communication session; 45. The method of any one of claims 24-44, further comprising displaying a representation of the new participant in the overflow area. receiving second data regarding an activity level of a first participant of the second plurality of participants;
In response to receiving the second data,
the second one of the representations of the second plurality of participants within the overflow region in response to the activity level of the first one of the second plurality of participants increasing; emphasizing visual characteristics of representations corresponding to said representations of one participant;
the second one of the representations of the second plurality of participants within the overflow region in response to the activity level of the first one of the second plurality of participants decreasing. de-emphasizing the visual properties of the representation corresponding to one participant;
46. The method of any one of claims 24-45, further comprising: 47. The method of any one of claims 24-46, wherein currently active is displayed as part of the first plurality of participants. detecting, while displaying the live communication user interface, that a second one of the second plurality of participants becomes the currently active participant;
responsive to detecting that the second participant of the second plurality of participants becomes the currently active participant, and the joining of the second of the second plurality of participants; displaying the representation of the second one of the second plurality of participants in the overflow area upon determining that the representation of the second participant is not currently displayed in the overflow area. scrolling the overflow region;
48. The method of any one of claims 24-47, further comprising: 49. The method of any one of claims 24-48, wherein the overflow region further comprises representations of the first plurality of participants. 50. The method of any one of claims 24-49, wherein the overflow area further comprises a placeholder representation for invitees of the live communication session who are not currently connected to the live communication session. displaying representations of camera views of the device at a first size as part of representations of the plurality of participants in the live communication session;
receiving a first input corresponding to a selection to display a first menu while displaying representations of the plurality of participants in the live communication session;
said first menu including options for displaying visual effects on representations of participants in said live communication session in response to receiving said input corresponding to selection to display said first menu; displaying a menu on the live communication user interface;
scaling the representation of the view of the camera of the device to a second size larger than the first size in response to receiving a second input corresponding to selecting the option for displaying a visual effect; be displayed in a size of
51. The method of any one of claims 24-50, further comprising: receiving a third input corresponding to a selection to expand a first one of the representations of the plurality of participants in the live communication session;
expanding the first representation in response to receiving the third input corresponding to a selection for expanding the first representation of the representations of the plurality of participants in the live communication session; and
reducing the size of at least one of the representations of the plurality of participants in the live communication user interface that is different from the first representation;
52. The method of any one of claims 24-51, further comprising: A representation of the plurality of participants in the live communication session includes a user communication state selected from the group consisting of an audio only state, a video state, a video paused state, a video undecoded state, an exited state, and a waiting to join state. 53. The method of any one of claims 24-52. receiving data indicative of a change in orientation of the electronic device;
In response to receiving the data indicative of a change in orientation of the electronic device,
rotating the display of the representations of the plurality of participants in the live communication session;
rotating the display while maintaining relative characteristics of previous displays of the representations of the multiple participants in the live communication session;
54. The method of any one of claims 24-53, further comprising: responsive to receiving the first data indicative of a change in orientation of the electronic device, one of the representations of the plurality of the participants in the live communication session based on the position of the face within the first representation; 55. The method of any one of claims 24-54, further comprising modifying the representation of the first representation. The representation of the plurality of participants in the live communication includes a second representation comprising a first portion of a live media stream received by the electronic device and displayed in a sixth aspect ratio, the method comprising:
for changing the aspect ratio of at least one representation of the plurality of representations of the participants in the live communication session while displaying the second one of the representations of the plurality of participants; receiving a request;
changing the aspect ratio of the second representation to the second representation in response to receiving the request to change the aspect ratio of the representation of at least one of the representations of the plurality of participants in the live communication session; changing from an aspect ratio of 6 to a seventh aspect ratio, wherein the second representation is a second representation of the live media stream received by the electronic device while at the seventh aspect ratio; wherein the second portion is selected to include faces detected in the live media stream;
56. The method of any one of claims 24-55, further comprising: receiving a fourth input corresponding to a request to display a menu while displaying representations of the plurality of participants in the live communication session;
In response to receiving the fourth input,
displaying a menu on the user interface for the live communication session;
moving at least a third representation of the plurality of participants in the live communication;
resizing at least a fourth one of representations of the plurality of participants in the live communication session;
57. The method of any one of claims 24-56, further comprising: replacing the representation of the representation of the representations of the first plurality of participants with the representation of the participant corresponding to the selected representation within the overflow area comprises: 42. The method of claim 41, further comprising ceasing display of said selected ones of said representations of a second plurality of participants. Replacing the representation of the representation of the representations of the first plurality of participants with the representation of the participant corresponding to the selected representation in the overflow region comprises the selected representation. after displaying the representation of the participant corresponding to the representation in a first size, expanding the representation of the participant corresponding to the selected representation to a second size larger than the first size. 42. The method of claim 41, further comprising: a participation in which an activity level of a first participant of the second plurality of participants represented in the overflow region corresponds to a first one of the representations of the first plurality of participants; receiving data indicating an increase above the activity level of the individual;
wherein the activity level of the first participant of the second plurality of participants represented in the overflow area corresponds to the first representation of the representations of the first plurality of participants; in response to receiving said data indicating an increase above said activity level of the corresponding said participant;
replacing the display of the first representation of the representations of the first plurality of participants with a representation of the first participant of the second plurality of participants;
Ceasing display of the representation of the first participant in the overflow region;
60. The method of any one of claims 24-59, further comprising: Replacing the display of the first one of the representations of the first plurality of participants displays a representation of the participant corresponding to the replaced first representation in the overflow region. 61. The method of claim 60, comprising: receiving input corresponding to a request to display a first menu while displaying representations of the plurality of participants in the live communication session;
Displaying the first menu including options for selecting content items on the live communication user interface in response to receiving the input corresponding to the request to display the first menu. and,
receiving input corresponding to selection of a content item;
responsive to receiving the input corresponding to the selection of the content item, displaying a representation of the participant displayed at a size greater than the respective threshold size; by replacing the expression with
62. The method of any one of claims 24-61, further comprising: displaying representations of the second plurality of participants in the overflow region;
Displaying a first set of representations in the overflow region upon determining that a first number of participants are participating in the live communication session;
including a different number of representations than the first set of representations upon determining that a second number of participants, different from the first number of participants, are participating in the live communication session. , displaying in the overflow region a second set of representations different from the first set of representations;
63. The method of any one of claims 24-62, comprising: 25. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs comprising: 64. A non-transitory computer-readable storage medium comprising instructions for performing the method of any one of Claims 1 to 63. a display;
one or more processors;
and a memory storing one or more programs configured to be executed by said one or more processors, said one or more programs being any of claims 24 to 63. An electronic device comprising instructions for performing the method of claim 1. a display;
means for performing the method of any one of claims 24-63;
An electronic device comprising: A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs comprising:
receiving a request to display a user interface for a live communication session between two or more participants;
Simultaneously displaying representations of multiple participants in the live communication session in response to receiving the request to display the user interface for the live communication session between the two or more participants. including instructions for displaying a live communication user interface,
Displaying representations of the plurality of participants in the live communication session includes:
Upon determining that the number of participants in the live communication session is greater than a threshold number of participants, reducing representations of a first plurality of the participants other than the user of the electronic device to a size greater than a respective threshold size. displaying representations of a second plurality of said participants in said overflow region while simultaneously displaying at one or more larger sizes, said second plurality of participants being included in said first plurality of participants; said representation of said one or more other participants displayed in said overflow region is displayed at a size smaller than said respective threshold size;
One or more representations of the participants other than the user of the electronic device that are equal to or less than the respective threshold size upon determining that the number of participants in the live communication session is equal to or less than the threshold number of participants. displaying representations of the participants other than the user of the electronic device at one or more sizes greater than the respective threshold size, without displaying at a size of
A method, including a display;
one or more processors;
and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising:
receiving a request to display a user interface for a live communication session between two or more participants;
simultaneously displaying representations of multiple participants in the live communication session in response to receiving the request to display the user interface for the live communication session between the two or more participants. displaying a live communication user interface, wherein displaying representations of the plurality of participants in the live communication session comprises:
Upon determining that the number of participants in the live communication session is greater than a threshold number of participants, rendering representations of a first plurality of participants other than the user of the electronic device larger than their respective threshold sizes. displaying representations of a second plurality of participants in an overflow region while simultaneously displaying in one or more sizes, wherein the second plurality of participants is not included in the first plurality of participants; comprising more than one participant, wherein the representation of the one or more other participants displayed in the overflow area is displayed at a size smaller than the respective threshold size;
One or more representations of the participants other than the user of the electronic device that are equal to or less than the respective threshold size upon determining that the number of participants in the live communication session is equal to or less than the threshold number of participants. displaying representations of the participants other than the user of the electronic device at one or more sizes greater than the respective threshold size, without displaying at a size of
A method, including a display;
means for requesting to display a user interface for a live communication session between two or more participants;
simultaneously displaying representations of multiple participants in the live communication session in response to receiving the request to display the user interface for the live communication session between the two or more participants. and means for displaying a live communication user interface, wherein displaying representations of the plurality of participants in the live communication session comprises:
Upon determining that the number of participants in the live communication session is greater than a threshold number of participants, reducing representations of a first plurality of the participants other than the user of the electronic device to a size greater than a respective threshold size. displaying representations of a second plurality of said participants in said overflow region while simultaneously displaying at one or more larger sizes, said second plurality of participants being included in said first plurality of participants; said representation of one or more other participants displayed in said overflow region is displayed at a size smaller than said respective threshold size;
One or more representations of the participants other than the user of the electronic device that are equal to or less than the respective threshold size upon determining that the number of participants in the live communication session is equal to or less than the threshold number of participants. means for displaying representations of participants other than the user of the electronic device at one or more sizes greater than the respective threshold size, without displaying at a size of
An electronic device comprising: In an electronic device with a display,
A live video simultaneously displaying a representation of a first participant in a live video communication session, a representation of a second participant in the live video communication session, and a representation of a third participant in the live video communication session. displaying a communications user interface;
The activity level of the second participant in the live video communication session rises above the activity level of the first participant in the live video communication session while displaying the live video communication user interface. receiving data indicating that the
in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;
changing the size and/or position of the representation of the first participant in the live video communication user interface;
changing the size and/or position of the representation of the second participant in the live video communication user interface;
changing the size and/or position of the representation of the third participant in the live video communication user interface;
A method, including After changing the size and/or position of the representation of the first participant in the live communication user interface, the modified representation of the first participant is the representation of the first participant. occupying at least a portion of the area occupied by said representation of said first participant in said live communication user interface prior to changing at least one of said size or position of
After changing the size and/or position of the representation of the second participant in the live communication user interface, the modified representation of the second participant is the representation of the second participant. occupying at least a portion of the area occupied by said representation of said second participant in said live communication user interface prior to changing at least one of said size and position;
After changing the size and/or position of the representation of the third participant in the live communication user interface, the modified representation of the third participant is: 71. The method of claim 70, occupying at least a portion of the area occupied by said representation of said third participant in said live communication user interface prior to changing at least one of said size or position. in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;
changing the size of the representation of the first participant in the live communication user interface;
changing the size of the representation of the second participant in the live communication user interface;
the size of the representation of the third participant in the live communication user interface is changed;
72. The method of claim 70 or 71, comprising: in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;
changing the position of the representation of the first participant in the live communication user interface;
changing the position of the representation of the second participant in the live communication user interface;
the position of the representation of the third participant in the live communication user interface is changed;
73. The method of any one of claims 70-72, comprising: displaying a representation of a fourth participant in the live communication user interface;
in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; changing the size and/or position of the representation of the fourth participant in the live communication user interface;
74. The method of any one of claims 70-73, further comprising: after changing the size and/or position of the representation of the first participant in the live video communication user interface, the activity level of the first participant in the live video communication session is receiving second data indicating an increase above the activity level of the second participant in
in response to receiving the second data indicating that the activity level of the first participant in the live video communication session has increased above the activity level of the second participant in the live video communication session. hand,
changing the size and/or position of the representation of the first participant in the live video communication user interface;
changing the size and/or position of the representation of the second participant in the live video communication user interface;
maintaining the size and position of the representation of the third participant in the live communication session;
75. The method of any one of claims 70-74, further comprising: 76. The method of any one of claims 70-75, wherein the activity level of at least the first participant is based on audio and/or video feeds received from the first participant's device. 77. The method of claim 76, wherein the activity level of at least the first participant is based on filtered audio from an audio feed received from the device of the first participant. 78. The claim 76 or 77, wherein the activity level of at least the first participant is based on movement meeting a set of movement criteria identified in a video feed received from the device of the first participant. the method of. further in response to receiving said first data indicating that said activity level of said second participant in said live communication session has increased above said activity level of said first participant in said live communication session; hand,
enhancing a visual representation of the representation of the second participant in the live communication user interface;
de-emphasizing a visual representation of the representation of the first participant in the live communication user interface;
79. The method of any one of claims 70-78, further comprising: prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;
the representation of the first participant is displayed in a first size;
the representation of the second participant is displayed in a second size that is smaller than the first size;
changing the size and/or position of the representation of the second participant increases the size of the representation of the second participant from the second size to the first size; 80. The method of any one of claims 70-79, comprising changing the size and/or position of the representation of the first participant;
modifying the size and/or position of the representation of the first participant based on facial recognition data obtained from the second participant's video feed; changing the size and/or position of the representation does not result in obscuring the face of the second participant;
81. The method of any one of claims 70-80. prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; 82. Any of claims 70-81, wherein one or more of representations of the first participant, the second participant and the third participant overlap with at least one other representation of the participant. or the method according to item 1. prior to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; representations of the first, second, and third participants in the live communication user interface are displayed according to z-order;
Changing the size and/or position of the representations of the first, second, and third participants in the live communication user interface may include: 83. The method of claim 82, comprising preserving the z-order of representations. a portion of the representation of the participant with the highest activity level among the first, second, and third participants of the first, second, and third participants in the live communication user interface; 84. A method according to any one of claims 70 to 83, wherein a portion of said representation of another of the participants overlaps. a portion of the representation of a participant having the lowest activity level among the first, second, and third participants of the first, second, and third participants in the live communication user interface; 85. A method according to any one of claims 70 to 84, overlapping a portion of said representation of another of the participants. after receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session; detecting an input corresponding to said expression of one participant;
86. A method according to any one of claims 70 to 85, in response to detecting said input corresponding to said representation of said first participant of said first participant. receiving data indicating that a fourth participant in the live communication session is more active than the lowest activity level participant of the first, second and third participants;
in response to receiving said data indicating that said fourth participant in said live communication session is more active than the least active one of said first, second and third participants. replacing the display of the representation of the participant with the lowest activity level among the first, second, and third participants with the display of the representation of the fourth participant;
87. The method of any one of claims 70-86, further comprising: replacing the representation of the representation of the participant with the lowest activity level among the first, second, and third participants with the representation of the fourth participant,
displaying the representation of the fourth participant at a size greater than the size of the representation of the participant with the lowest activity level among the first, second, and third participants; 88. The method of claim 87. from claim 70, wherein changing the size and/or position of the representation of the first participant comprises gradually increasing the size of the representation from a first size to a second size; 88. The method of any one of clauses 88. The live communication interface includes a canvas area, the representation of the first participant in the live communication session, the representation of the second participant in the live communication session, and the third participant in the live communication session. 90. The method of any one of claims 70-89, wherein said representation of a participant is displayed in said canvas area. detecting a canvas resize event;
In response to detecting said canvas resize event,
resizing the canvas area;
modifying the layout of the representations of the first, second and third participants;
91. The method of claim 90, further comprising: 92. The method of claim 91, further comprising resizing the one or more of the representations of the first, second, and third participants further in response to detecting the canvas resize event. the method of. displaying a representation of a camera view of the electronic device in the live communication session user interface;
further in response to receiving said first data indicating that said activity level of said second participant in said live communication session has increased above said activity level of said first participant in said live communication session; hand,
maintaining the size of the representation of the camera's view of the electronic device in the live communication session user interface;
93. The method of any one of claims 70-92, further comprising: 70. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs comprising: claim 70 94. A non-transitory computer-readable storage medium comprising instructions for performing the method of any one of Claims 1-93. a display;
one or more processors;
and a memory storing one or more programs configured to be executed by said one or more processors, said one or more programs being any of claims 70 to 93. An electronic device comprising instructions for performing the method of claim 1. a display;
means for performing the method of any one of claims 70-93;
An electronic device comprising: A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs comprising:
A live video communication simultaneously displaying a representation of a first participant in the live video communication session, a representation of a second participant in the live video communication session, and a representation of a third participant in the live video communication session. display the user interface,
the activity level of the second participant in the live video communication session exceeds the activity level of the first participant in the live video communication session while displaying the live video communication user interface; receive data indicating that it has risen;
in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;
changing the size and/or position of the representation of the first participant in the live video communication user interface;
changing the size and/or position of the representation of the second participant in the live video communication user interface;
A non-transitory computer-readable storage medium comprising instructions for changing the size and/or position of the representation of the third participant in the live video communication user interface. a display;
one or more processors;
and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising:
A live video communication simultaneously displaying a representation of a first participant in the live video communication session, a representation of a second participant in the live video communication session, and a representation of a third participant in the live video communication session. display the user interface,
the activity level of the second participant in the live video communication session exceeds the activity level of the first participant in the live video communication session while displaying the live video communication user interface; receive data indicating that it has risen;
in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;
changing the size and/or position of the representation of the first participant in the live video communication user interface;
changing the size and/or position of the representation of the second participant in the live video communication user interface;
An electronic device comprising instructions for changing the size and/or position of the representation of the third participant in the live video communication user interface. a display;
A live video communication simultaneously displaying a representation of a first participant in the live video communication session, a representation of a second participant in the live video communication session, and a representation of a third participant in the live video communication session. means for displaying a user interface;
the activity level of the second participant in the live video communication session exceeds the activity level of the first participant in the live video communication session while displaying the live video communication user interface; means for receiving data indicative of an increase;
in response to receiving the data indicating that the activity level of the second participant in the live video communication session has increased above the activity level of the first participant in the live video communication session;
changing the size and/or position of the representation of the first participant in the live video communication user interface;
changing the size and/or position of the representation of the second participant in the live video communication user interface;
means for changing the size and/or position of said representation of said third participant in said live video communication user interface;
An electronic device comprising: In an electronic device with a display,
Displaying a messaging user interface of a messaging application, the messaging user interface comprising:
a message area containing a plurality of messages between participants in a message conversation containing three or more participants;
an affordance separate from the message area for initiating a live video communication session in a live video communication application;
detecting an input corresponding to the affordance selection;
responsive to detecting the input corresponding to the selection of the affordance, initiating a process for initiating the live video communication session in the live video communication application, wherein the live video communication session comprises the including the participants; and
A method, including Initiating the process of initiating the live communication session in the live communication application includes:
transitioning from the messaging user interface to a messaging user interface of the live communication application;
initiating the live communication session;
101. The method of claim 100, comprising: Initiating the process of initiating the live communication session in the live communication application includes displaying a group contact user interface including group contact affordances for initiating the live communication session, the method comprising: ,
detecting input corresponding to selection of the group contact affordance to initiate the live communication session;
initiating the live communication session in the live communication application in response to detecting the input corresponding to selecting the group contact affordance for initiating the live communication session;
101. The method of claim 100, further comprising: After initiating said live communication session,
Displaying a reminder option associated with the first participant on the group contact user interface in response to determining that the first participant has not participated in the live communication session for a predetermined amount of time. ,
receiving input corresponding to selection of the reminder option while displaying the reminder option associated with the first participant;
Sending a notification indicating that the live communication session is available to the first participant in response to receiving the input corresponding to selecting the reminder option;
103. The method of claim 102, further comprising: The group contact user interface further includes a plurality of participant affordances including a first participant affordance associated with a first one of the participants of the message conversation, the method comprising:
detecting input corresponding to selection of the first participant affordance associated with the first participant of the message conversation;
In response to detecting the input corresponding to selecting the first participant affordance associated with the first participant of the message conversation, displaying a personal participant user interface including characteristics of the first participant. and
103. The method of claim 102, further comprising: The first participant affordance includes an option for selecting a communication mode for contacting the first participant of the message conversation, the method comprising:
detecting input corresponding to selection of the option for selecting the communication mode;
responsive to detecting the input corresponding to selection of the communication mode option, initiating a process of selecting the commute mode for the first participant;
105. The method of claim 104, further comprising: The group contact user interface further includes a new contact affordance, the method comprising:
detecting input corresponding to selection of the new contact affordance;
initiating a process of adding a new contact to the live communication session in response to detecting the input corresponding to selection of the new contact affordance;
106. The method of any one of claims 103-105, further comprising: Initiating the process of initiating the live video communication session includes:
initiating the live communication session in the live communication application;
displaying the live communication session user interface, the live communication session user interface comprising:
including a placeholder associated with a first one of the participants of the live communication session;
107. The method of any one of claims 100-106, comprising The first placeholder includes a first reminder affordance for the first one of the participants of the live communication session, the method comprising:
receiving input corresponding to selection of the first reminder affordance;
Sending a new notification to the first participant that is different from previous notifications sent to the first participant in response to receiving the input corresponding to selection of the first reminder affordance. and,
108. The method of claim 107, further comprising: The live communication session user interface further includes a second placeholder associated with a second participant, the first placeholder and the second placeholder arranged according to a first layout, the method comprising:
Upon determining that the first participant has not participated in the live communication session for a predetermined amount of time,
ceasing to display the first placeholder;
changing the arrangement of the second placeholders to a second layout different from the first layout;
109. The method of claim 107 or 108, further comprising Claim 107, further comprising modifying said display of said first placeholder to represent said first participant upon determining that said first participant has participated in said live communication session. 109. The method of any one of 100. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device comprising a display, the one or more programs comprising: claim 100 110. A non-transitory computer-readable storage medium comprising instructions for performing the method of any one of 110. a display;
one or more processors;
and a memory storing one or more programs configured to be executed by said one or more processors, said one or more programs being any of claims 100 to 110. An electronic device comprising instructions for performing the method of claim 1. a display;
means for performing the method of any one of claims 100-110;
An electronic device comprising: A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device comprising a display and a touch-sensitive surface, the one or more programs ,
A messaging user interface for a messaging application, comprising:
a message area containing a plurality of messages between participants in a message conversation containing three or more participants;
an affordance separate from the message area for initiating a live video communication session in a live video communication application;
detecting an input corresponding to a selection of said affordance;
instructions for, in response to detecting the input corresponding to the affordance selection, initiating, in the live video communication application, a process to initiate the live video communication session including the participants in the message conversation; A non-transitory computer-readable storage medium. a display;
one or more processors;
and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising:
A messaging user interface for a messaging application, comprising:
a message area containing a plurality of messages between participants in a message conversation containing three or more participants;
an affordance separate from the message area for initiating a live video communication session in a live video communication application;
detecting an input corresponding to a selection of said affordance;
instructions for, in response to detecting the input corresponding to the selection of the affordance, initiating, in the live video communication application, a process to initiate the live video communication session including the participant in the message conversation; electronic device. an electronic device,
a display;
means for displaying a messaging user interface of a messaging application, said messaging user interface comprising:
a message area containing a plurality of messages between participants in a message conversation containing three or more participants;
an affordance separate from the message area for initiating a live video communication session in a live video communication application;
means for detecting an input corresponding to selection of said affordance;
means for, in response to detecting the input corresponding to the selection of the affordance, initiating a process for initiating the live video communication session in the live video communication application, wherein the live video communication session comprises the a means comprising a participant;
An electronic device comprising: In an electronic device with a display,
A live video communication user interface for a live video communication application, comprising:
a plurality of participant affordances each corresponding to one or more participants, including a first participant affordance corresponding to a group of three or more participants;
displaying a live video communication user interface including a new session affordance for starting a new live video communication session;
detecting input corresponding to a request to initiate a process to initiate a new live video communication session;
In response to detecting said input,
responsive to the input corresponding to the selection of the first participant affordance, initiate a process to initiate a new live video communication session including the group of three or more participants;
Initiating a process of selecting two or more participants in response to the input corresponding to the selection of the new session affordance, the participant associated with the device and the two or more selected participants. initiating a new live video communication session comprising
A method, including 118. The method of claim 117, wherein the live communication user interface further includes a list of recent live communication sessions. 4. The plurality of participant affordances includes a second participant affordance corresponding to a proposed group of participants determined based on data from a second application different from the live video communication application. The method according to 117 or 118. 120. The method of claim 119, wherein the suggested group of participants corresponds to participants in a group conversation conducted in the second application or a conference call conducted in the second application. 121. A method according to claim 119 or 120, wherein said second participant affordance comprises displaying said second application. The plurality of participant affordances correspond to a proposed second group of participants determined based on data from a third application that is different from the live video communication application and also different from the second application. 122. The method of any one of claims 119-121, comprising a third participant affordance. The live communication user interface further includes a participant selection affordance, the method comprising:
receiving a second input corresponding to selection of the participant selection affordance;
in response to receiving the second user input corresponding to selection of the participant selection affordance;
a text entry field for identifying one or more participants;
a keyboard;
Audio-only affordances and
displaying a group selection user interface including a video affordance;
receiving a third user input selection while displaying the group selection user interface;
In response to receiving the third user input,
In response to the third user input corresponding to the selection of the audio-only affordance, the new live communication session is initiated using audio transmitted from the electronic device without transmitting video data from the electronic device. and
initiating the new live communication session with audio and video transmitted from the electronic device in response to the third user input corresponding to the video affordance selection;
123. The method of any one of claims 117-122, further comprising: initiating the new live communication session;
displaying a representation of the new live communication session in a fourth application that is different from the live communication session after initiating the new live communication session;
124. The method of any one of claims 117-123, further comprising: the plurality of participant affordances includes a fourth participant affordance corresponding to a group of three or more participants;
upon determining that the fourth participant affordance corresponds to an active live communication session, the fourth participant affordance is displayed having a first visual appearance;
Upon determining that the fourth participant affordance corresponds to an inactive live communication session, the fourth participant affordance has a second visual appearance different from the first visual appearance. 125. The method of any one of claims 117 to 124, displayed as . initiating the process of selecting two or more participants includes displaying representations of potential participants;
displaying said representation of said potential participant having a first visual appearance following a determination that said potential participant is associated with a device capable of participating in a live communication session;
The representation of the potential participant has a second visual appearance different from the first visual appearance following a determination that the participant is not associated with a device that can participate in a live communication session. 126. A method according to any one of claims 117 to 125, displayed as . 117. A non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device comprising a display, the one or more programs comprising: claim 117; 127. A non-transitory computer-readable storage medium comprising instructions for performing the method of any one of 126. a display;
one or more processors;
and a memory storing one or more programs configured to be executed by said one or more processors, said one or more programs being any of claims 117 to 126. An electronic device comprising instructions for performing the method of claim 1. a display;
means for performing the method of any one of claims 117-126;
An electronic device comprising: A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs comprising:
A live video communication user interface for a live video communication application, comprising:
a plurality of participant affordances each corresponding to one or more participants, including a first participant affordance corresponding to a group of three or more participants;
displaying a live video communication user interface including a new session affordance for starting a new live video communication session;
detecting input corresponding to a request to initiate a process to initiate a new live video communication session;
In response to detecting said input,
responsive to the input corresponding to the selection of the first participant affordance, initiate a process to initiate a new live video communication session including the group of three or more participants;
Initiating a process of selecting two or more participants in response to the input corresponding to the selection of the new session affordance, the participant associated with the device and the two or more selected participants. A non-transitory computer-readable storage medium comprising instructions for initiating a new live video communication session comprising: a display;
one or more processors;
and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising:
In an electronic device with a display,
A live video communication user interface for a live video communication application, comprising:
a plurality of participant affordances each corresponding to one or more participants, including a first participant affordance corresponding to a group of three or more participants;
displaying a live video communication user interface including a new session affordance for starting a new live video communication session;
detecting input corresponding to a request to initiate a process to initiate a new live video communication session;
In response to detecting said input,
responsive to the input corresponding to the selection of the first participant affordance, initiate a process to initiate a new live video communication session including the group of three or more participants;
Initiating a process of selecting two or more participants in response to the input corresponding to the selection of the new session affordance, the participant associated with the device and the two or more selected participants. An electronic device comprising instructions for initiating a new live video communication session comprising: a display;
A live video communication user interface for a live video communication application, comprising:
a plurality of participant affordances each corresponding to one or more participants, including a first participant affordance corresponding to a group of three or more participants;
means for displaying a live video communication user interface comprising: a new session affordance for starting a new live video communication session;
means for detecting input corresponding to a request to initiate a process to initiate a new live video communication session;
In response to detecting said input,
responsive to the input corresponding to the selection of the first participant affordance, initiate a process to initiate a new live video communication session including the group of three or more participants;
Initiating a process of selecting two or more participants in response to the input corresponding to the selection of the new session affordance, the participant associated with the device and the two or more selected participants. means for initiating a new live video communication session comprising
An electronic device comprising:

Images