HK1237090B - Method and system for executing multi threads on computer processor core - Google Patents

Description

Method and system for executing multiple threads on a computer processor core

Background Art

The present invention generally relates to coordinated start of interpreted execution exit for multithreaded processors, and more particularly to providing wait state and warning tracking in conjunction with coordinated start of interpreted execution exit in a multithreaded environment to reduce resource costs in the multithreaded environment.

Generally speaking, multithreading increases the number of processor threads that can operate in parallel within a single processor core. Multithreading provides this increased capacity by enabling one or more processor threads to use portions of the hardware of a single processor core that are not currently being used by one or more other processor threads running on the single processor core. For example, during a delay caused by a cache miss or other delay in a first processor thread, one or more other processor threads can utilize core resources that were assigned to the first processor thread during the cache miss, thereby improving utilization of those core resources.

While multithreading provides hardware savings, adding another thread incurs more coordination costs at the software level than using an additional individual processor core to provide increased capacity. In many cases, after achieving a certain scaling ratio, the overhead of coordinating core resources between threads (whether running on a single or shared processor core) is significant and can reduce or even outweigh the benefits of independent processor threads.

Summary of the Invention

According to one embodiment of the present invention, a method for executing multiple threads on a computer processor core is provided, the multiple threads including a first thread and a group of remaining threads, the method comprising: determining that a start interpret execute exit condition exists; determining that the computer processor core is within a grace period; the first thread entering a start interpret execute exit synchronization loop without signaling any of the group of remaining threads; and the first thread remaining in the start interpret execute exit synchronization loop until the grace period expires or each of the remaining threads enters a corresponding start interpret execute exit synchronization loop.

Additional features and advantages are achieved through the techniques of the present invention. Other embodiments and aspects of the present invention are described in detail herein and are considered a part of the claimed invention. For a better understanding of the present invention with these advantages and features, reference is made to the description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of this specification. The foregoing and other features and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

FIG1 is a schematic diagram showing a computing device of a multi-threaded system;

FIG2 shows a schematic diagram of a processor of a multi-threaded system;

3A-3B illustrate a processing flow regarding a wait state in a multi-threaded system;

4A-4B illustrate another processing flow of a multi-threaded system regarding a wait state in a core dispatch environment;

FIG5 shows a process flow of warning tracking in a core dispatching environment of a multi-threaded system; and

6A-6C illustrate another process flow for warning tracking in a core dispatch environment of a multi-threaded system.

DETAILED DESCRIPTION

As noted above, the coordination cost in multithreading between processor threads is substantial and can reduce or even outweigh the benefits of independent processor threads. Therefore, a need exists for a multithreaded environment that provides wait states and warning tracking in conjunction with coordinated start-interpret-execute-exit to reduce resource costs in a multithreaded environment.

Generally speaking, embodiments of the invention disclosed herein may include a multithreaded system, method, and/or computer program product that utilizes software to efficiently manage infrastructure on a core at a granular thread basis to reduce core resource costs. This is achieved by allowing a hypervisor running on a single thread to dispatch multiple guest threads on a single core using core dispatching, and utilizing wait states and warning tracking in conjunction with coordinated start of interpret execution exit.

The multi-threaded system, method, and/or computer program ("multi-threaded system") will now be described as utilizing core dispatching via coordinated start of interpreted execution ("SIE") exit. That is, via core dispatching, the multi-threaded system allows a single-threaded hypervisor to dispatch multi-threaded guests onto its cores using a single instruction (note that each multi-threaded guest represents a guest logical processor or guest thread). The operands of the single instruction may specify a single state description encompassing the states of all guest threads, or a set of state descriptions, e.g., each state description representing the state of a single guest thread. Furthermore, to support the use of core dispatching and to accommodate single-threaded hypervisors, the multi-threaded system provides coordinated SIE exits to enable all guest threads to exit simultaneously.

For example, when each thread of a guest core determines that it must exit interpreted execution mode, it enters the SIE Exit state and waits in an initial SIE Exit synchronization loop until all other active threads of the same core are also ready to exit. In some cases, each thread signals the other threads to exit before entering this synchronization loop.

Wait states in a non-multithreaded and/or multithreaded environment of a multithreaded system will now be described. With respect to a non-multithreaded environment, when a guest thread has completed a task from a queue and there are no additional tasks on the queue, the multithreaded system loads a wait state code or bit into the program status word ("PSW"). The wait state bit in the PSW causes the guest thread to pause instruction execution until an interrupt is provided. When a guest thread is running in a dedicated non-multithreaded environment (e.g., a physical processor is used exclusively by a single guest thread), and the single guest thread enters an enabled (i.e., guest-enabled for asynchronous interrupts) wait state, the single guest thread will remain dispatched on the physical processor until an interrupt is recognized. If the guest thread is running in a shared environment (i.e., a physical processor is shared between different guest logical processors), when the guest thread enters a wait state, the shared environment will exit interpreted execution via a wait state interception, allowing the hypervisor to, if applicable, dispatch a different guest thread with work to perform.

In a multi-threaded environment, if a guest thread on a core is still executing guest instructions, it is more efficient for the core to continue running until all active threads on the core have entered a wait state or a coordinated SIE exit is required for another reason. In addition, a thread in an enabled wait state enters a firmware wait state loop; and if an interrupt is presented to the thread before all other threads have entered a wait state, the thread can process the interrupt and exit the firmware wait state loop (e.g., the interrupt includes any request for a coordinated SIE exit by another thread).

Referring now to FIG. 1 , an example of a multi-threaded system 100 including a computing device 112 is shown. The multi-threaded system 100 is merely an example of a suitable computing node and is not intended to represent any limitation on the scope of use or operability of the embodiments of the present invention described herein (indeed, additional or alternative components and/or implementations may be used). That is, the multi-threaded system 100 and the components therein may take a variety of different forms and include multiple and/or alternative components and facilities. In addition, as described herein, the multi-threaded system 100 may include and/or employ any number and combination of computing devices and networks utilizing various communication technologies. Regardless, the multi-threaded system 100 is capable of being implemented and/or performing any operability described herein.

In the multi-threaded system 100, there is a computing device 112 that operates with a variety of other general-purpose or special-purpose computing system environments or configurations. Systems and/or computing devices, such as the multi-threaded system 100 and/or computing device 112, can employ any of a number of computer operating systems, including, but not limited to, versions and/or flavors of the AIX UNIX and z/OS operating systems distributed by International Business Machines Corporation of Armonk, New York, the Microsoft Windows operating system, Unix operating systems (e.g., the Solaris operating system distributed by Oracle Corporation of Redwood Shores, California), the Linux operating system, Mac OS X and iOS operating systems distributed by Apple Inc. of Cupertino, California, the BlackBerry OS distributed by Dynamic Research Corporation of Waterloo, Canada, and the Android operating system developed by the Open Handset Alliance. Examples of computing systems, environments, and/or configurations suitable for use with computing device 112 include, but are not limited to, personal computer systems, server computer systems, thin clients, complex clients, handheld or laptop computer devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputer systems, computer workstations, servers, desktop computers, laptop computers, network appliances, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

The computing device 112 can be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computing device 112 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in both local and remote computer system storage media, including storage devices.

As shown in FIG1 , the computing device 112 in the multi-threaded system 100 is shown in the form of a general-purpose computing device that is improved based on the operation and functionality of the multi-threaded system 100, its methods, and/or its components. Components of the computing device 112 may include, but are not limited to, one or more processors or processing units (e.g., a processor 114 including at least one core 114a that supports multiple threads 115; for example, the multi-threaded system 100 includes a core 114a of the processor 114 that includes two or more threads 115), memory 116, and a bus 118 that couples various system components including the processor 114 and the memory 116. The computing device 112 also typically includes a variety of computer system-readable media. Such media can be any available media that can be accessed by the computing device 112, and includes both volatile and non-volatile media, and both removable and non-removable media.

The processor 114 may receive computer-readable program instructions from the memory 116 and execute these instructions, thereby performing one or more processes defined by the multi-threaded system 100. The processor 114 may include any processing hardware, software, or a combination of hardware and software utilized by the computing device 114 to execute computer-readable program instructions by performing arithmetic, logical, and/or input/output operations. Examples of the processor 114 and core 114a include, but are not limited to: an arithmetic logic unit that performs arithmetic and logical operations; a control unit that fetches, decodes, and executes instructions from memory; and an array unit that utilizes multiple parallel computing elements.

FIG2 illustrates one embodiment of a computing environment including a processor 114 coupled to a controller 215. In one example, the computing environment based on the z/Architecture includes a System z server provided by International Business Machines Corporation of Armonk, New York. For example, the processor 114 may include one or more partitions (e.g., logical partitions LP1 through LPn), one or more physical cores (e.g., core 1 through core m), and a level-0 hypervisor 214 (e.g., a logical partition manager). The controller 215 may include centralized logic responsible for arbitrating between different processors issuing requests. For example, when the controller 215 receives a memory access request, it determines whether access to the memory location is permitted and, if so, provides the contents of the memory location to the processor 114 while maintaining memory coherency between processors within the complex. Another controller 215 may manage requests to and from the I/O interface 130 and/or network adapter 132 shown in FIG1 .

A physical core comprises physical processor resources allocated to a logical partition. A logical partition may include one or more logical processors, each of which represents all or a portion of the physical processor resources allocated to that partition. A physical core can be dedicated to a logical core of a particular partition, so that the physical processor resources of the underlying core are reserved for that partition; or it can be shared with a logical core of another partition, so that the physical processor resources of the underlying core are potentially available to the other partition. Each logical partition is capable of functioning as a separate system. That is, each logical partition can be independently reset, initially loaded with an operating system (e.g., operating system OS1 through operating system OSn) (if necessary), and operated using different programs. An operating system or application running in a logical partition appears to have access to the entire complete system, but in reality, only a portion of the entire system is available. A combination of hardware and authorized internal code (also known as firmware, microcode, or millicode) prevents programs in one logical partition from observing, accessing, or interfering with programs in a different logical partition. This allows several different logical partitions to operate on a single or multiple physical cores in a time-sliced manner. In one embodiment, each physical core includes one or more central processing units (also referred to herein as "physical threads"). 2, each logical partition has a resident operating system, which may be different for one or more logical partitions. Each logical partition is an instance of a virtual machine or guest configuration in which an operating system can run.

In the embodiment shown in FIG2 , logical partitions LP1 through LPn are managed by a level-0 hypervisor 214 implemented by firmware running on physical cores 1 through m. Logical partitions LP1 through LPn and hypervisor 214 each comprise one or more programs residing in respective portions of central storage (memory) associated with physical cores 1 through m. One example of hypervisor 214 is Processor Resource/Systems Manager (PR/SM ^™ ), available from International Business Machines Corporation in Armonk, New York.

Returning to FIG1 , memory 116 may comprise a tangible device that retains and stores computer-readable program instructions (as provided by multi-threaded system 100) for use by processor 114 of computing device 112. Memory 116 may comprise computer system-readable media in the form of volatile memory, such as random access memory (RAM) 120, cache 122, and/or storage system 124. Bus 118 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

By way of example only, a storage system 124 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown and typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media). In such cases, each drive may be connected to the bus 118 by one or more data media interfaces. As will be further depicted and described below, the memory 116 may include at least one program product having a set (e.g., at least one set) of program modules configured to perform operations of embodiments of the present invention. The storage system 124 (and/or the memory 116) may include a database, a data repository, or other data storage and may include various types of mechanisms for storing, accessing, and retrieving various types of data, including a hierarchical database, a collection of files in a file system, an application database in a proprietary format, a relational database management system (RDBMS), and the like. The storage system 124 may generally be included within the computing device 112 (which, as shown, employs a computer operating system such as one of the operating systems described above) and accessed via a network in any one or more of a variety of ways.

By way of example and not limitation, program/utility 126 having a set (at least one set) of program modules 128, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 116. Each or some combination of the operating system, one or more application programs, other program modules, and program data may include an implementation of a network connection environment.

Computing device 112 may also communicate via input/output (I/O) interface 130 and/or via network adapter 132. I/O interface 130 and/or network adapter 132 may comprise physical and/or virtual mechanisms used by computing device 112 to communicate between components within and/or external to computing device 112. For example, I/O interface 130 may communicate with one or more external devices 140, such as a keyboard, pointing device, display 142, etc.; one or more devices that enable a user to interact with computing device 112; and/or any device that enables computing device 112 to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Furthermore, computing device 112 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet), via network adapter 132. Thus, I/O interface 130 and/or network adapter 132 may be configured to receive or send signals or data within or for computing device 112. As depicted, I/O interface 130 and network adapter 132 communicate with the other components of computing device 112 via bus 118. It should be understood that, although not shown, other hardware and/or software components may be used in conjunction with computing device 112. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

1 illustrates a single item for multi-threaded system 100 (and other items), these representations are not intended to be limiting and, therefore, any item may represent multiple items. For example, processor 114 may include multiple processing cores, each of which executes multiple threads and is capable of handling the coordinated SIE exits described herein.

The multi-threaded system will be described with reference to Figures 3A-3B, which illustrate one example of a process flow 300 regarding wait states and coordinated SIE exits.

As shown, process flow 300 is executed by multiple threads (e.g., thread X, thread Y, and thread Z). At blocks 310, 312, and 314, thread X (e.g., the main thread) executes a guest instruction, loads an enabled wait state PSW, and enters a firmware wait state loop. Thread X does not exit the SIE at this point because both thread Y and thread Z (e.g., helper threads) are still executing guest instruction streams (e.g., blocks 330 and 340). Thread X remains in the firmware wait state loop until a guest asynchronous interrupt becomes pending at block 316. At block 318, the guest asynchronous interrupt causes thread X to exit the wait state loop and enter the firmware asynchronous interrupt handler. The firmware asynchronous interrupt handler saves the old interrupt PSW and interrupt information to a fixed storage location in the first block of guest memory and loads the new guest interrupt PSW into hardware. Next, at block 320, thread X begins executing the guest software interrupt handler. When thread X completes executing the software interrupt handler, the guest operating system reloads the enabled wait PSW at block 322, which again calls/enters the firmware wait state loop.

Meanwhile, thread Y independently executes a guest instruction at block 330, loads an enabled wait state PSW at block 332, and enters a firmware wait state loop at block 334. Similarly, thread Z independently executes a guest instruction at block 340, loads an enabled wait state PSW at block 342, and enters a firmware wait state loop at block 344. That is, each thread Y, Z loads a corresponding enabled wait state PSW (blocks 332, 342) after a respective time Y1, Z1, allowing these events to occur independently. Additionally, each thread Y, Z enters a corresponding firmware wait state PSW (blocks 334, 344) after a respective time Y2, Z2, allowing these events to occur independently. At block 350, thread X determines that all active threads of the core (e.g., threads X, Y, Z) are now in wait states and calls an SIE exit for wait state interception. Next, at line 352, thread X reaches the initial SIE exit synchronization point, which causes thread X to signal threads Y and Z to exit the SIE using an internal interrupt mechanism, as shown in block 354. The process of detecting that all threads are in a wait state and signaling the other threads to exit the SIE can be performed by any thread and is typically completed by the last thread to enter a wait state.

After receiving the internal interrupt signal from block 354, threads Y and Z independently exit. That is, threads Y and Z independently exit their firmware wait state loops in response to the SIE exit request, handle the internal interrupt, and call SIE exit (as indicated in blocks 356 and 358). Finally, threads Y and Z reach the initial SIE exit synchronization loop at lines 362 and 364. Once all active threads (e.g., threads X, Y, and Z) have reached the initial synchronization point (line 366), each thread independently completes updating its state description and indicates wait state interception (e.g., blocks 370, 372, and 374). Once each thread (e.g., thread Y and Z) has completed updating its state description, the thread sets a hardware control bit. Specifically, at block 382, thread Y sets a hardware control bit to indicate that the final SIE exit synchronization point has been reached and instruction execution ceases. Similarly, at block 384, thread Z sets a hardware control bit to indicate that the final SIE exit synchronization point has been reached and instruction execution ceases. Next, at block 390, thread X (e.g., the main thread) waits for all threads Y and Z to reach the final synchronization point, which means that all threads have completed all state description updates and completed the coordinated SIE exit. Finally, at block 392, thread X then presents the multi-threaded SIE intercept to the host, which processes the multi-threaded SIE intercept.

As another example of wait states and coordinated SIE exits in a multi-threaded system, process flow 400 will be described with reference to Figures 4A-B. Similar to process flow 300, process flow 400 is performed by multiple threads (e.g., thread X, thread Y, and thread Z). To begin process flow 400, at blocks 410, 412, and 414, thread X executes a guest instruction, loads an enabled wait state PSW, and enters a firmware wait state loop. Similarly, at blocks 420 and 430, thread Y and thread Z also execute guest instructions, which causes thread X to remain in a wait state loop rather than invoking an SIE exit.

Additionally, at block 432, while thread X is still in a firmware wait state loop, thread Z performs an operation that results in a call to SIE Exit for type N interception. At line 434, thread Z reaches an initial SIE exit synchronization point, which in turn signals other threads to exit the SIE using the corresponding internal interrupt mechanism (e.g., block 436). In response, at block 440, thread X exits the firmware wait state loop for the SIE exit request, enters the firmware interrupt handler for the SIE exit, and calls SIE Exit. Thus, thread X reaches an initial SIE exit synchronization point at line 442. Next, at block 444, a guest asynchronous interrupt becomes pending on thread X, but it is not presented because the thread is already in an SIE exit. When thread Y reaches an interruptible point, it takes an internal interrupt for the SIE exit request at block 450 and calls SIE Exit. Thread Y then reaches an initial SIE exit synchronization loop at line 452.

Once all active threads (e.g., threads X, Y, and Z) have reached the initial synchronization point (e.g., line 460), each thread independently completes its state description update and indicates that an intercept is applicable (e.g., blocks 470, 472, and 474). For example, thread X takes a wait state intercept at block 470 because PSW bit 14 is equal to 1; thread Y takes a no-op intercept at block 472; and thread Z takes a primitive type N intercept at block 474. As described above with respect to process flow 300, in this example, once each thread Y and Z has completed all of its state description updates, it has reached the final SIE exit synchronization point and ceases its own instruction execution (472, 474). For example, at blocks 482 and 484, threads Y and Z set hardware control bits to indicate that the final SIE exit synchronization point has been reached and cease instruction execution. Next, at block 490, thread X waits for all threads to reach the final synchronization point, which means that all threads have completed all state description updates and completed the coordinated SIE exit. Finally, at block 492, Thread X then presents the multi-threaded SIE intercept to the host, which processes the multi-threaded SIE intercept.

For example, a non-multithreaded guest operating system can be signaled via a warning trace interrupt that its time slice is about to expire. The hypervisor uses the host CPU timer to determine when a guest processor core's time slice has expired, allowing another guest core to begin its time slice. Additionally, the hypervisor can remotely indicate from another processor that a warning trace interrupt should be presented to a guest logical core by setting a bit in the target state description. A machine enters a grace period when it detects a host timer interrupt as pending during a grace period of inactivity, or when it detects a warning trace interrupt as pending in the state description. The grace period is implemented by adding a minimum extra time to the time slice and, when enabled, signals the guest operating system via a warning trace interrupt that its time slice is about to expire. This extra time allows guests that are aware of the warning trace interrupt to take appropriate action (often referred to as "cleanup") before leaving the processor; such action may include, for example, releasing a system-level lock. Once a guest has completed its cleanup, it is expected to signal the completion of the cleanup by issuing a diagnostic acknowledgement (e.g., the guest executes the DIAGNOSE instruction with the code "049C" indicating a warning trace acknowledgement). If a DIAG "049C" warning trace interrupt confirmation is issued before the grace period expires, the system intercepts the interrupt back to the hypervisor through instruction interception, and the hypervisor handles this as the end of the time slice. Conversely, if the grace period expires before the guest operating system issues a DIAG "049C" warning trace interrupt confirmation, a host timer interrupt is generated, and the hypervisor handles the end of the time slice.

The warning trace interrupt ("WTI") processing performed in the multi-threaded system will now be described. When multi-threading is active in a guest, the hypervisor manages guest time slices on a core-by-core basis. A single host timer exists, and its associated interrupt indicates when a guest time slice has ended for all threads in the core. Additionally, when the hypervisor sets a WTI pending in the primary state description, it is the multi-threaded system's responsibility to set a WTI interrupt pending in the secondary state description. When a host timer interrupt occurs, or when a pending WTI is detected in the primary state description, the core enters a grace period. The resulting WTI interrupt remains pending and, when enabled, is presented to each active guest thread. As described above, a guest loaded with enabled waits enters a firmware wait state loop when appropriate. However, if the grace period is active or a WTI is pending when an enabled wait state is loaded, the guest will instead enter an SIE exit synchronization loop. The difference between the two loops is that the wait state loop will process pending interrupts, which may extend the time required to complete an SIE exit and directly contradict the desired result of responding to a WTI.

The multi-threaded system will be described with reference to FIG5 , which illustrates an example of a process flow 500 for WTI and core dispatching. As shown, process flow 500 describes the handling of a host processor timer ("CPT") interrupt from the perspective of alert tracking. If the CPT becomes pending and the host is enabled to take the interrupt, firmware is called to handle the interrupt. That is, process flow 500 checks at decision block 502 whether the processor core is running in host mode. If the core is running in host mode, process flow 500 proceeds to block 504 where the host CPT interrupt is presented (e.g., as indicated by a "yes" arrow). If the core is not running in host mode (e.g., as indicated by a "no" arrow), it is running in guest mode and process flow 500 proceeds to decision block 506.

At decision block 506, process flow 500 checks whether a grace period is active. If a grace period is not in progress, process flow 500 proceeds (e.g., as indicated by a "no" arrow) to determine at decision block 508 whether warning tracing is supported for the guest. If warning tracing is supported, process flow 500 proceeds to block 510 (e.g., as indicated by a "yes" arrow) to set the T bit in the core's main state description, which resides in the state description and indicates that a warning tracing request is pending. Otherwise, if a grace period is not in progress and/or warning tracing is not supported, process flow 500 proceeds to block 512. At block 512, an SIE exit is called, thereby enabling a host CPT interrupt to be presented to the host, and in guest multithreaded mode, other threads are signaled to exit the SIE. This behavior is identical to WTI behavior in a non-multithreaded environment, except that in a multithreaded environment, other threads are signaled to exit the SIE when a host CPT interrupt is to be presented.

The WTI in conjunction with the coordinated SIE exit of the multi-threaded system will now be described. To implement the signaling of other threads to exit the SIE when a guest interrupt is taken, such signaling should not be performed while a WTI is pending or the core is within a grace period. Instead, before an SIE exit occurs, the multi-threaded system will provide other threads with an opportunity to complete their cleanup and issue a diagnostic acknowledgement (e.g., the guest executes a DIAGNOSE instruction with a code "049C" indicating a warning trace acknowledgement). If the grace period expires before all active threads have synchronized, the remaining threads will be signaled to exit.

In a multithreaded environment, any active processor will remain in the SIE exit synchronization loop until all other active processors 1) enter a wait state within the grace period, 2) are in a wait state when the core enters the grace period, 3) exit the SIE for independent reasons, including due to an instruction intercept after issuing DIAG 049C to indicate cleanup completion, or 4) exit the SIE in response to signaling from another thread. The typical and expected situation is that all threads will independently complete cleanup and issue DIAG 049C WTI confirmation before the grace period expires, or will be in a wait state when the grace period is entered on that core. A multithreaded specific behavior of the warning trace is the delay of SIE exit signaling within the grace period to attempt to allow the operating system on each thread to clean up before being removed from the processor.

The multi-threaded system will be described with reference to Figures 6A-6C, which illustrate an example process flow 600 for warning tracking in a core dispatch environment. Generally speaking, process flow 600 illustrates the behavior of any given thread once a warning tracking request is pending. Thus, process flow 600 begins at block 602 when the T bit is enabled for any thread (e.g., a main thread or an auxiliary thread). The T bit in the main state description can be set by the multi-threaded system (as shown in Figure 5) or by the hypervisor when processing a host timer interrupt. Once the multi-threaded system detects the T bit, it determines at decision block 604 whether the core is within a grace period. If the core is not within a grace period (e.g., as indicated by a "no" arrow), the multi-threaded system determines at decision block 606 whether the core is running in a guest multi-threaded environment. If the core is running in a guest multi-threaded environment, process flow 600 proceeds to block 608, where the multi-threaded system sets the T bit in the auxiliary state description for all active auxiliary threads in the state description group. Note that the T bit is propagated to the state description of the auxiliary thread only when this process flow is being executed on the main thread. Furthermore, regardless of whether the core is running in a guest multithreaded environment, process 600 proceeds to block 610, where the core enters a grace period (e.g., assuming the T bit is on, as seen in block 602, and not entering a grace period, as seen in block 604). Entering a grace period involves extending the guest time slice by a small amount and setting an indication that the grace period is active. The grace period begins even if a WTI cannot be presented to any or all threads. Next, at decision block 612, the multithreaded system determines whether the guest thread is enabled for WTI. If the multithreaded system determines that the guest thread is enabled for WTI, an interrupt is presented at block 614, and the guest operating system can take appropriate action at 616, such as initiating cleanup (process flow 600 then proceeds to decision block 618 of FIG. 6B via process connector "a"). When the multithreaded system determines that the guest is not enabled for WTI, then WTI cannot be presented and guest execution continues until it becomes enabled or until another SIE exit condition occurs (process flow 600 then proceeds to decision block 642 of FIG. 6C via flow connector “b”).

As shown in FIG6B , once the WTI interrupt has been presented to the guest, the multi-threaded system monitors for certain conditions while completing the cleanup. Process flow 600 proceeds to decision block 618, where the multi-threaded system determines whether a grace period has expired before the cleanup is complete. If the grace period has expired (e.g., as indicated by a "yes" arrow), process flow 600 proceeds to block 620, where an SIE exit is called and the other threads are signaled to exit. Note that if an SIE exit is called within the grace period for any other reason, the other threads are not signaled to allow them time to complete their cleanup. If the grace period has not expired (e.g., as indicated by a "no" arrow), process flow 600 proceeds to decision block 622, where the multi-threaded system determines whether a wait-state PSW has been loaded before the cleanup is complete. If a wait-state PSW has been loaded (e.g., as indicated by a "yes" arrow), process flow 600 proceeds to block 624, where a wait-state intercept is presented on the thread. If the wait state PSW is not loaded (e.g., as indicated by a "no" arrow), process flow 600 proceeds to decision block 626, where the multi-threaded system determines whether a guest intercept occurs before the cleanup is complete. If a guest intercept occurs (e.g., as indicated by a "yes" arrow), process flow 600 proceeds to block 628, where the intercept is taken on the thread. If a guest intercept does not occur (e.g., as indicated by a "no" arrow), process flow 600 proceeds to decision block 630, where the multi-threaded system determines whether a host interrupt occurs before the cleanup is complete. If a host interrupt occurs (e.g., as indicated by a "yes" arrow), process flow 600 proceeds to block 632, where an SIE exit is called so that the interrupt can be presented. If no host interrupt has occurred (e.g., as indicated by a "no" arrow), process flow 600 proceeds to decision block 634, where the multi-threaded system determines whether a SIE exit request has been received from another thread before cleanup is complete. If a SIE exit request has been received from another thread (e.g., as indicated by a "yes" arrow), process flow 600 proceeds to block 636, where a no-op intercept occurs on that thread. If no SIE exit request has been received from another thread (e.g., as indicated by a "no" arrow), process flow 600 proceeds to block 638, where a DIAG "049C" WTI acknowledgment is issued if the guest operating system is able to complete cleanup and before any other conditions occur. Next, at block 640, a DIAG instruction intercept occurs. Once all threads have reached the SIE exit final synchronization cycle, the guest intercept(s) and/or host interrupt are presented to the host on the main thread.

As shown in FIG6C , if the WTI interrupt is not enabled, the guest thread continues to monitor for the WTI interrupt to enable so that cleanup will occur, and monitors for other conditions, including the expiration of the grace period. Process flow 600 proceeds to decision block 642, where the multi-threaded system determines whether the grace period has expired before the WTI becomes enabled and the interrupt is presented to the software. If the grace period has expired (e.g., as indicated by the “yes” arrow), process flow 600 proceeds to block 620, where the SIE exit is called and the other threads are signaled to exit. Note again that if the SIE exit is called within the grace period for any other reason, the other threads are not signaled to provide them with time to complete their cleanup. If the grace period has not expired (e.g., as indicated by the “no” arrow), process flow 600 proceeds to decision block 644, where the multi-threaded system determines whether the wait-state PSW was loaded before presenting the WTI. If a wait-state PSW is loaded (e.g., as indicated by a "yes" arrow), process flow 600 proceeds to block 624, where a wait-state intercept is presented on the thread. If a wait-state PSW is not loaded (e.g., as indicated by a "no" arrow), process flow 600 proceeds to decision block 646, where the multi-threaded system determines whether a guest intercept is required for the thread. If a guest intercept occurs (e.g., as indicated by a "yes" arrow), process flow 600 proceeds to block 628, where an intercept is taken on the thread. If a guest intercept does not occur (e.g., as indicated by a "no" arrow), process flow 600 proceeds to decision block 648, where the multi-threaded system determines whether a host interrupt occurs before presenting the WTI. If a host interrupt occurs (e.g., as indicated by a "yes" arrow), process flow 600 proceeds to block 632, where an SIE exit is called so that the interrupt can be presented. If a host interrupt has not occurred (e.g., as indicated by a "no" arrow), process flow 600 proceeds to decision block 650, where the multi-threaded system determines whether an SIE exit request has been received from another thread. If an SIE exit request has been received from another thread (e.g., as indicated by a "yes" arrow), process flow 600 proceeds to block 636, where a no-op intercept occurs. If an SIE exit request has not been received from another thread (e.g., as indicated by a "no" arrow), process flow 600 proceeds to decision block 612 of FIG. 6A (via flow connector "c"), where the multi-threaded system determines whether the guest is enabled for WTI.

WTI in a multithreaded environment can also be extended to include additional signaling to the operating system before performing the SIE exit synchronization signaling. In cases where the expected hypervisor response to an interrupt or intercept that causes a SIE exit is to dispatch a different guest core, it is often beneficial to use WTI to warn still executing threads that they are about to be removed from the processor.

In general, a computing device may include a processor (e.g., processor 114 of FIG. 1 ) and a computer-readable storage medium (e.g., memory 116 of FIG. 1 ), wherein the processor receives computer-readable program instructions (e.g., from the computer-readable storage medium) and executes these instructions, thereby performing one or more processes, including one or more of the processes described herein.

Computer-readable program instructions may be compiled or interpreted from a computer program generated using compiler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object program code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, or the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on a computing device, partially on a computing device, as a stand-alone software package, partially on a local computing device and partially on a remote computing device, or entirely on a remote computing device. In the latter case, the remote computer may be connected to the local computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits (including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs)) can execute computer-readable program instructions by personalizing the electronic circuits using state information of the computer-readable program instructions so as to perform aspects of the present invention. The computer-readable program instructions described herein can also be downloaded from a computer-readable storage medium to a corresponding computing/processing device or to an external computer or external storage device (e.g., any combination of computing devices and connections that support communication) via a network. For example, the network can be the Internet, a local area network, a wide area network, and/or a wireless network, including copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers, and utilizing a variety of communication technologies, such as radio technology, cellular technology, etc.

A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device (e.g., a computing device as described above). A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device (such as a punch card or a raised structure in a groove on which instructions are recorded), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium itself is not understood to be a temporary signal, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated via a waveguide or other transmission medium (e.g., a light pulse transmitted via an optical cable), or an electrical signal transmitted via a wire.

Thus, the multi-threaded system and method and/or its components may be implemented as computer-readable program instructions on one or more computing devices, these instructions being stored on a computer-readable storage medium associated therewith. A computer program product may include such computer-readable program instructions stored on a computer-readable storage medium for executing and/or causing a processor to execute the operations of the multi-threaded system and method. The multi-threaded system and method and/or its components (as implemented and/or claimed) improve the functionality of the computer and/or processor itself because the wait state and warning tracking utilized in conjunction with the coordinated start of interpreted execution exit reduces resource costs. In addition, the multi-threaded system, method and/or computer program product provides a more efficient means of providing wait state and warning tracking support in a multi-threaded environment. For example, a modified wait state of a first thread running in a shared environment delays the start of interpreted execution exit until all threads of the shared environment are in the modified wait state. During a grace period, warning tracking delays signaling between threads to initiate a coordinated start of interpreted execution exit so as to provide each thread with "cleanup" time before they exit.

In other words, the improvements include providing mechanisms to: keep a core dispatched until all threads on that core have entered wait states in a shared physical processor environment; recognize that threads in a core are in the "cleanup" process and delay signaling other threads for "immediate" SIE exit to provide them with an opportunity to complete cleanup before being forced to exit; and leverage the use of warning trace interrupt facilities to more efficiently exit the SIE in a multi-threaded environment.

Various aspects of the present invention are described herein with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a component for implementing the operations/actions specified in one or more flowcharts and/or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions can direct the computer, programmable data processing device, and/or other equipment to operate in a specific manner, so that the computer-readable storage medium storing the instructions includes an article of manufacture, which includes instructions for implementing multiple aspects of the operations/actions specified in one or more flowcharts and/or block diagrams.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the operations/actions specified in one or more flowcharts and/or block diagram blocks.

The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, operability and operation of possible implementations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment or portion of an instruction, which includes one or more executable instructions for implementing a specified logical operation. In some alternative embodiments, the operations mentioned in the blocks may not occur in the order mentioned in the accompanying drawings. For example, depending on the operability involved, two blocks shown in a continuous manner may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the opposite order. It should also be noted that each block of the block diagram and/or flowchart, and the combination of blocks in the block diagram and/or flowchart, can be implemented by a hardware-based dedicated system that performs the specified operation or action or executes a combination of dedicated hardware and computer instructions.

The descriptions of the various embodiments of the present invention have been presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the marketplace, or to enable those skilled in the art to understand the embodiments disclosed herein.

The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention. As used herein, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural forms as well. It will be further understood that when used in this specification, the term "comprising" specifies the presence of stated features, integers, steps, operations, elements, and/or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and/or combinations thereof.

The flowcharts depicted herein are merely examples. Multiple variations of the figures or steps (or operations) described herein may exist without departing from the scope of the present invention. For example, the steps may be performed in a different order or steps may be added, deleted, or modified. All of these variations are considered part of the claimed invention.

While the preferred embodiment to the invention has been described, it will be understood that those skilled in the art, both now and in the future, may make various improvements and enhancements which fall within the scope of the claims which follow. These claims should be construed to maintain the proper protection for the invention first described.

Claims (11)

1. A method for executing multiple threads on a computer processor core, said multiple threads including a first thread and a group of remaining threads, the method comprising: It has been determined that an exit condition for initiating interpretation and execution exists; It was determined that the computer processor core was within the grace period; The first thread enters the process of interpreting and executing before exiting the synchronization loop without notifying any of the remaining threads in the group. The first thread remains in the start-interpretation-execution-exit synchronization loop until the grace period expires or each of the remaining threads enters the corresponding start-interpretation-execution-exit synchronization loop. It has been determined that the grace period for the computer processor core has expired; and Upon the expiration of the grace period, the first thread signals each of the remaining threads to proceed to the corresponding start of interpretation execution and exit the synchronization loop. 2. The method of claim 1, further comprising: It was determined that the first thread was pending in response to the warning tracing interruption; The first thread propagates the warning trace interruption pending condition to each of the remaining threads; and The first thread causes the computer processor core to enter the grace period. 3. The method of claim 1, further comprising: It is determined that for the first thread, the waiting state is pending; It was determined that for the first thread, a warning tracing interruption pending condition existed; and Based on the determination that the warning tracking interruption pending condition exists, the process enters the "start interpretation execution exit synchronization loop" state. 4. The method of claim 3, wherein the start of interpretation execution exits the synchronization loop based on determining the existence of the warning trace interruption pending condition, without presenting the warning trace interruption and without signaling each of the remaining threads. 5. The method of claim 1, wherein the first thread is the main thread among the plurality of threads. 6. A system for executing multiple threads on a computer processor core, the multiple threads including a first thread and a set of remaining threads, the system including the processor and memory: The processor is configured to: It has been determined that an exit condition for initiating interpretation and execution exists; It was determined that the computer processor core was within the grace period; The first thread enters the process of interpreting and executing before exiting the synchronization loop without notifying any of the remaining threads in the group. The first thread remains in the start-interpretation-execution-exit synchronization loop until the grace period expires or each of the remaining threads enters the corresponding start-interpretation-execution-exit synchronization loop. It has been determined that the grace period for the computer processor core has expired; and Upon the expiration of the grace period, the first thread signals each of the remaining threads to proceed to the corresponding start of interpretation execution and exit the synchronization loop. 7. The system of claim 6, wherein the processor is further configured to: It was determined that the first thread was pending in response to the warning tracing interruption; The first thread propagates the warning trace interruption pending condition to each of the remaining threads; and The first thread causes the computer processor core to enter the grace period. 8. The system of claim 6, wherein the processor is further configured to: It is determined that for the first thread, the waiting state is pending; It was determined that for the first thread, a warning tracing interruption pending condition existed; and Based on the determination that the warning tracking interruption pending condition exists, the process enters the "start interpretation execution exit synchronization loop" state. 9. The system of claim 8, wherein the start of interpretation execution exits the synchronization loop based on determining the existence of the warning trace interruption pending condition, without presenting the warning trace interruption and without signaling each of the remaining threads. 10. The system of claim 6, wherein the first thread is the main thread among the plurality of threads. 11. A computer-readable storage medium having program instructions contained thereon for executing a plurality of threads on a computer processor core of a processor, the plurality of threads including a first thread and a set of remaining threads, the program instructions being executable by the processor to cause the processor to: It has been determined that an exit condition for initiating interpretation and execution exists; It was determined that the computer processor core was within the grace period; The first thread enters the process of interpreting and executing before exiting the synchronization loop without notifying any of the remaining threads in the group. The first thread remains in the start-interpretation-execution-exit synchronization loop until the grace period expires or each of the remaining threads enters the corresponding start-interpretation-execution-exit synchronization loop. It has been determined that the grace period for the computer processor core has expired; and Upon the expiration of the grace period, the first thread signals each of the remaining threads to proceed to the corresponding start of interpretation execution and exit the synchronization loop.