GB2353119A - Testing a distributed software system by manipulating the arrival times of incoming network communication events - Google Patents

Abstract

A method of testing a distributed software system which includes a first network communicating process and a second network communicating process, includes steps of: intercepting network communication level events being sent from the first network communicating process 10a-10c to the second network communicating process; accumulating the intercepted network communication level events in a central location 13, and forwarding the network communication level events on to the second network communicating process 12 in a time controlled manner, so as to directly control the arrival times of the network communication level events at the second network communicating process for the purpose of testing the distributed software system.

Description

2353119 TESTING A DISTRIBUTED SOFTWARE SYSTEM BY MANIPULATING THE ARRIVAL

TIMES OF INCOMING NETWORK COMMUNICATION EVENTS

Field of the Invention

The present invention relates to the field of computation in a distributed environment where a plurality of computational processes are communicating with each other (e.g., via a network) in order to carry out an overall data processing task. More particularly, the invention relates to the testing of such a distributed data processing environment in order to discover, diagnose, recreate and eliminate software defects.

Background of the Invention

In a distributed computing environment, client application processes communicate with server application processes, typically over a network, in order to communicate requests to the server application processes which then process the requests and, if appropriate, return replies back to the client application processes. In addition to this simple client-server interaction, similar behaviour is present in any distributed architecture that involves software processes communicating over a network (for example, multi-tier client-server architecture or peer-to-peer architecture).

Software defects can emerge during operation if one or more of the communicating processes is unusually stressed by incoming network communications. For example, when a server application process experiences a very high rate of incoming client application process requests. Even at moderate rates, defects may emerge due to the timing pattern of incoming communication arrivals. More complicated distributed architectures are coming into favour as the popularity of distributed systems increases. Toolkits (for example Enterprise Java Beans) for building integrated systems of network communicating components and legacy systems are gaining broad acceptance. This trend increases the number and types of processes that are vulnerable to stress caused by incoming network communication. For example, as electronic commerce over the Internet increases in popularity, existing client/server transactional software products will undoubtedly experience problems due to the increased network traffic load on server application processes managing the on-line transactions.

2 Software system testing products have been developed in order to perform testing on traditional client-server systems to determine what kinds of load levels the application server is able to handle. These products are used by software testers to find defects and raise confidence in the system by creating anticipated loads prior to deployment of the system. They are also used to recreate defects found in the deployed system. For example, Mercury Interactive's LoadRunner (trademark), Sequels SilkPerformer (trademark) and IBM's TPNS (trademark) products have all been developed for this purpose. See, also, US Patent No. 5,812,780 to Microsoft Corp. However, Applicant has found that the type of testing that can be done with the present state of the art is very limiting.

For example, the LoadRunner product uses client emulators to simulate a plurality of expected usage scenarios. To create a new simulation the tester creates a script or changes settings via a graphical user interface.

The tester input defines the parameters of the simulation (e.g., the number of concurrent users, synchronisation points between users, business logic of the user's actions, rates of user actions and delays between actions) The simulation creates a server load that is similar to that which would result in a real usage scenario.

Thus, the tester has control of only the usage scenario parameters, with the resulting server loads being dependent on such settings. This results in an indirect and imprecise control over the timing patterns of the server load. Further, this prior art product deliberately focuses the testing on a narrow range of server load patterns, i.e., those that follow from anticipated usage scenarios, in order to find defects that arise in the anticipated usage scenarios. However, the present Applicant has found that there are many defects that are sensitive to small variations in the timing properties of the server load ' and the inability of the prior art to directly create a wide range of such variations makes such defects more difficult to detect.

Further, the prior art is limited to performing testing of a server process by creating a simulation of a particular client environment.

Applicant has found that it would be highly desirable to extend a test to cover a plurality of processes in more complicated distributed architectures. For example, in a three tier architecture, it would be useful to test a backend database (bottom tier) by controlling the timing pattern of incoming communication from a mid-level application server. For testing efficiency, it would be useful to control the timing properties of 3 incoming communication to both the middle tier application server as well as the bottom tier database server so as to simultaneously test two tiers.

in the emerging location independent component architectures (for example systems that use Enterprise Java Beans) there are potentially many more processes receiving network communication, thus the prospect of simultaneously testing multiple processes becomes even more advantageous.

Having noticed these limitations in the present state of the art, Applicant has invented the present invention to overcome these and other limitations in the prior art which will be discussed below.

Summary of the Invention

According to a first aspect, the present invention provides a method of testing a distributed software system which includes a first network communicating process and a second network communicating process, the method comprising steps of:

intercepting network communication level events being sent from the first network communicating process to the second network communicating process; accumulating the intercepted network communication level events in a central location; and forwarding the network communication level events on to the second network communicating process in a time controlled manner, so as to directly control the arrival times of the network communication level events at the second network communicating process for the purpose of testing the distributed software system.

According to second and third aspects, the invention provides an apparatus for carrying out the method of the first aspect, and a computer program product stored on a computer- readable storage medium for, when run on a computer, carrying out the method steps of the first aspect.

Thus, the present invention provides direct and precise control over the arrival times of a sequence of network level events (e.g., packets) arriving at a particular network communicating process. This direct and precise control allows a broad range of timing patterns to be used in a highly selective fashion in order to create a very versatile testing environment for a distributed software system.

4 Brief Description of the Drawings

The invention will be better understood by the below described description of the preferred embodiments thereof which will be provided with reference to the following drawing figures:

Figure 1 is a block diagram of a client/server network which will be used to describe an example of a preferred embodiment of the present invention; Figure 2 is a flowchart showing the steps involved in scheduling the forwarding of packets from the intermediate computer to the server computer; and Figure 3 is a flowchart showing the method steps involved in the testing technique of the preferred embodiment of the present invention.

Detailed Description of the Preferred Embodiments

In the preferred embodiment of the present invention network communication level events (e.g., packets) being sent between one or more pairs of processes which are communicating via a network are intercepted after leaving the first process and accumulated in a central location, where the network communication level events are later sent to the second process in a time controlled fashion. An example will be given below by reference to Fig. 1 where the two communicating processes are a client process (10a, 10b, 10c) and a server process 12. However, this is only an example to illustrate the usefulness of the invention and one of the important features of the invention is that more than one pair of processes may be selected for the interception and forwarding of network communication events. While the example uses the packetized TCP/IP protocol, the invention could be used with networks that use other sorts of protocols.

In Fig. 1 a plurality of client computers 10a, 10b and 10c are shown in communication with a server computer 12 and an intermediate computer 13 via a network 11 (which can be, for example, a Local Area Network (LAN), or a Wide Area Network such as the Internet). According to a preferred embodiment of the present invention, the clients communicate with the server 12 by sending requests in the form of packets (an example of a network communication level event). For example, the packet format could be that which is commonly used in the TCP/IP protocol of the Internet.

Intermediate computer 13 is provided for the purpose of conducting testing on the server software running on the server computer 12, and the operational steps involved in such testing will be described with reference to the flowchart of Fig. 3.

That is, if the client computers 10a, 10b and 10c are all involved in a test of the server computer 12's software then each packet being sent by clients 10a, 10b, 10c is intercepted (step 31 of Fig. 3) and redirected to the intermediate computer 13. the clients 10a, 10b, 10c had originally intended that the packets be directed to the server 12, however, according to the preferred embodiment of the present invention, the packets are redirected, upon leaving the clients, to the intermediate computer 13.

The specifics of this interception/ redirection of packets will now be discussed.

There are two ways to redirect packets to the intermediate buffering computer. while both approaches will be described in terms of a TCP/IP ethernet network, similar approaches could be taken with other sorts of networks and protocols. one method is to construct an atypical physical network. The intermediate computer is equipped with multiple network interface cards (NICs). One dedicated NIC is required for each computer that sends or receives the packets that are buffered by the intermediate computer. We will refer to the computers that send and receive test related packets as "test computers". The NIC of each test computer is directly cabled to it's dedicated NIC on the intermediate computer. There must be no other network routes between the test computers. The routing table of each test computer is modified to designate the intermediate computer to be the "next hop" for the only route to each of the other test computers. It is desirable that the intermediate computer have an uplink to the ordinary test lab LAN, and that it be configured to forward packets to all non-test computers that may be reached by way of the uplink.

Likewise the ordinary LAN should be configured such that the intermediate computer is the designated gateway to all of the test computers. The 3S intermediate computer should be configured to forward packets sent from all non-test computers to all test computers. Thus, test packets are redirected to the intermediate computer by modifying the physical network.

Modifying the physical network is cumbersome, expensive, and the maximum number of test computers is limited to the maximum number of NICs that the intermediate computer can accommodate. IBM's copending US patent 6 application serial number 163,211 filed September 29, 1998 (IBM docket no.

CR9-98-037) describes a simple, inexpensive and scalable approach that does not require modification of the physical network. In the most simple case, all test computers are on the same subnet. The ARP mechanism is disabled on all test computers. On each test computer, for each computer that the test computer needs to communicate with (via the network), a static entry is made in the table that maps MAC addresses to IP addresses (sometimes called the ARP table). The static entries for the other test computers are modified such that the IP address of each of the other test computers is mapped to the MAC address of the intermediate computer. The normal packet addressing mechanisms use these modified entries, and thus direct all packets destined for other test computers to the intermediate computer.

When test computers are located on more than one subnet, the intermediate computer would preferably have a NIC connected to each of the subnets is containing test computers. The ARP table of each test computer is modified as described above for each of the other test computers residing on the same subnet. Further, the intermediate computer is designated in the routing table of each test computer to be the "next hop" of the primary route to each of the other test computers that reside on a different subnet. If the intermediate computer is unable to have a NIC on each of the subnets containing test computers, then'modi f i cations of ARP and routing tables of the various devices on each of the paths, between each test computer and all the other test computers, must be made such that the intermediate computer is a device (or more loosely speaking "a hop") on all of the paths connecting any pair of test computers. It is easier if all test computers are on one or two subnets but this is not possible in all testing situations. while this approach is preferable when using TCP/IP ethernet, it may not be compatible with all network types and protocols, and in that case modification of the physical network (described above) maf be used.

Once the redirected packets arrive at the intermediate computer 13 the packets are accumulated (step 32) in a FIFO (first in first out) buffer (not shown). Once accumulated in the buffer the packets are acted upon by well known real time programming techniques at the computer 13 in order to control the forwarding (step 33) of the buffered packets on to the server 12 via the network 11 in a precise and direct time controlled manner.

Specifically, a request arrival rate (i.e. the rate of arrival of packets at the destination) can be directly and precisely controlled in a repeatable manner. For example, the arrival rate can be made higher than 7 normal by forwarding the accumulated packets in a shorter timespan than it took to accumulate the packets. The arrival rate can also be made volatile, i.e., changing in value for successive forwarding operations, in a precise, time controlled, and repeatable manner. The accumulating of packets in the buffer provides a holding of the packets so that they may be acted upon under software control, resulting in the forwarding of each packet in the queue according to a schedule. Any distribution of interarrival times of the packets at the server could be used for the schedule- For example, the intervals between the arrival of successive packets could follow a random exponential distribution that is commonly used for system engineering simulations. Two kinds of events can trigger the forwarding of the packets to the destination. The forwarding of packets can simply be triggered by the number of accumulated packets in the queue. Second, a trigger that depends on the semantics of the system being tested can be detected by examining the data content of the buffered packets. For example, forwarding of packets could be triggered by accumulating a specific number of packets containing a specific kind of service request from the clients. Multiple trigger events may be defined and they may be overlapping in their triggered effect. For example, all buffered packets are forwarded (the triggered effect) to computer a, when (trigger event 1) 100 packets arrive from computer b, or (trigger event 2) packets arrive from computer c.

The buffering and forwarding technique could follow various algorithms. The buffering and forwarding technique used in the preferred embodiment of the present invention is shown in the flowchart of Fig. 2.

This diagram is simplified to show the embodiment for only one pair of communicating computers. Another simplification is that the diagram suggests that only one trigger event may be in effect at any one time, whereas multiple trigger events may be defined and "watched for", and they may be overlapping in their triggered effect. In the preferred embodiment, only one memory buffer contains all relevant packets. The preferred embodiment avoids copying packet data so as to maximise performance. various sequences and other relationships between the packets are captured by multiple linked lists superimposed on the buffered packets.

For example, the sequence of packets sent from one source to one destination is captured in one linked list, while the sequence of packets sent from multiple sources to a single destination is kept in a second linked list. Additional linked lists may be used to capture relations that depend on the semantics of the system being tested as well as the underlying communication protocols. Thus, the packets may be eventually 8 forwarded to the destination in a variety of orders depending on the testing effect desired. Most commonly, all buffered packets destined for a single computer would be forwarded, at one time, in the order in which they would have otherwise arrived at the destination computer.

Well known "packet filtering" techniques may be used such that selected packets are immediately forwarded to their destination and thus bypass the delaying buffer. For example, it is desirable to immediately forward packets that control the test harness or those packets that provide basic communication services between the computers involved in the test.

For example, the packets used by a telnet session between two computers that are involved in a test should flow unimpeded, even though other packets between the same two computers are buffered, delayed, and later forwarded.

At step 21, the tester defines the statistical distribution to use for choosing packet interarrival intervals. At step 22, the tester chooses to use a semantic or length trigger. At step 23, if length trigger is chosen, the tester chooses the statistical distribution for the lengths of successively buffered packet sequences. At step 24, the program chooses the length of the next packet sequence. At step 25 a sequence of the chosen length is accumulated or a timeout occurs that moves processing to step 26 before the desired sequence length has been accumulated. Setting this timeout value is omitted from this diagram because it is usually set only once to a value compatible with the communication protocol being used.

The handling of protocol specific "keep alive" messages is also omitted from this diagram. At step 26, the delivery of packets is scheduled according to the previously defined statistical distribution of intervals.

At step 27, the sequence of packets is forwarded to the server according to the schedule. At step 28, the user chooses whether to change the interval distribution or forwarding trigger. If a change is desired, processing loops back to step 21. Processing branches at step 29 depending on the type of trigger being used. If only sequence length is being used, then processing loops back to step 24, otherwise processing loops back to step 31. If, at step 22, the semantic trigger is chosen, the semantic trigger event is defined at step 30 and then control flows to step 31 (described above).

During the accumulation phase, any "keep alive" messages must be handled by the preferred embodiment in a protocol specific manner. A "keep alive" message is sent by the implementation of the communication protocol 9 on the computer where the buffered packets originate. A "keep alive" (sometimes called a "server ping") message is sent to determine whether a duplicate of the original communication needs to be sent for some reason.

For example, a "keep alive', message may be sent to determine whether the original packet has been lost due to network malfunction. Normally the receiver of the original packet is expected to respond to the "keep aliveu message but this cannot happen if the "inquired about" packet is still buffered on the intermediate computer. In some protocols, the "keep alive" messages do not need to be responded to immediately, and there is no need to respond after the normal response is made to the original message (for example after an RPC response is made to the original RPC request). In all cases, the "keep alive" message should not be buffered and eventually forwarded to the server because it may arrive at the server after the normal response has already been made to the client and thus violate the is protocol. In some protocols, it is acceptable for the intermediate buffering computer to simply discard the "keep alive" messages. Should a protocol require a response to a "keep alive" message, then the intermediate computer would forge what looks like a protocol -compliant response from the receiving computer. A typical response to a "keep alive" message indicates that the server is still busy working on the original request and that the client computer should wait. In some protocols, there may be additional situations that require a similar remedy.

while the technique could be extended to other communication protocols, the preferred embodiment uses a structured publicly documented protocol such as DCE RPC, CORBA IIOP, JRMP, or DCOM. This sort of protocol is preferable because it simplifies the use of semantic triggers based on the data content of the packets. For the purposes of this invention, using these protocols with UDP rather than TCP is easier. Also, the preferred embodiment uses a general purpose network emulator (for example, WANspoof) (such as the network emulator described in IBM's copending US patent application serial number 163,211 filed September 29, 1998, the contents of which are herein incorporated by reference (IBM docket no.

CR9-98-037)) running on the intermediate computer 13 to intercept and buffer the packets. The software of the general purpose network emulator is extended according to well known real time programming techniques to provide the precise and direct control over the forwarding of the buffered packets on to the destination when triggered (e.g., server 12 in the example of Fig. 1).

The source and destination of each packet determine the link that a packet traverses over the network 11 (there is a link between each pair of possible sources and destinations). For each link that is subject to buffering and forwarding, the preferred embodiment of the present invention provides a FIFO queue containing all of the traffic on that link in the intermediate computer 13 (except the "keep alives" and similar superfluous packets). The traffic in each queue accumulates and is later forwarded to the destination according to a schedule and ordering determined by various parameters. For example, forwarding of traffic is controlled by various triggers, principally counts of RPCs on a link. Also, timeouts can be used (e.g., in DCE RPC, requests that do not receive a response in 3 0 seconds are cancelled by the originator of the RPC).

In a particular server process under test, Applicant has found that concurrent threads overlap in time such that the sequences of events taking place in different threads interleave. Deadlocks or race condition defects usually result from the sequence of events in only two or three threads interleaving in a detrimental pattern. Applicant has further found that the highly volatile request arrival rates that are inherent at large customer sites create a broad variety of thread overlap patterns. Thus, a testing environment should be able to recreate this volatility and consequent variety of thread overlap patterns in order to maximise test coverage. The preferred embodiment of the present invention allows the forwarding of buffered packets in a controlled and repeatable manner to thus provide a broad range of volatile request arrival rates and consequently a broad variety of thread overlap patterns can be set up in the test lab.

The preferred embodiment allows the buffering of requests generated over a relatively long period of time and the subsequent forwarding of the requests over a relatively short period of time, thus high request arrival rates and consequently high peak server loads can be attained with a small number of client machines (even a single client machine 10a could be used. ) Further, the great variety of high and volatile server loads are attained without custom programming of test client emulations (as was required in the prior art) and thus the preferred embodiment of the present invention can be used with existing test programs to extend their coverage.

Preserving the investment in and extending the value of existing client test programs is very desirable.

11 Because a live client can be used (rather than a client emulator), multistep dialogues can be conducted between the client and the server during the test. Such dialogues may dynamically branch in subsequent steps depending on server responses in earlier steps of the dialogue. This is especially important in simulation- scenario based testing where the course of the test depends on the intermediate results that follow from the randomly chosen actions of other unsynchronized test clients. one such test, in the area of online transaction processing, might include a simulated stock market with test clients placing buy and sell orders with stops and limits, the results of which and the course of the test would depend on the actions of other clients and the state of the simulated market.

in the above discussion the main focus has been on buffering and is bursting client requests in a two-tiered client-server architecture.

However, the technique can be used between any concurrent set of pairs of network communicating processes in most architectures. Prior art techniques that require custom programming of test client emulations have much less generality and much less widespread applicability. This increases efficiency because multiple components can be tested in the same timespan. For example, the technique of the preferred embodiment of the present invention can be used to buffer and burst:

a) requests from a middle-tier application server to a bottom-tier database server; b) responses from a bottom-tier database server to a middle-tier application server; c) peer-to-peer requests and/or responses between replicated servers; or d) all or some combination of the first three possibilities simultaneously.

A typical large distributed software system contains computers of several types with different relative computation power. The number of computers of a particular type in a system tends to get smaller as the computation power increases. For example, a three tiered architecture may contain several mainframes, many mid-level application servers, and a vast multitude of user level desktop PCs. This presents a problem for testing 12 because it is expensive to obtain enough computers to create sufficiently high loads on the more powerful machines. Using this technique, the load generated by a small number of computers can be forwarded in a shorter time than it took to generate the load. Thus, with a relatively small number of each type of computer that is present in a system, it is possible to test the software running on each type at high loads simultaneously, albeit the more powerful computers are tested intermittently for shorter periods of time. This provides an economical means to test the behaviour of the system under realistic high and volatile loads repeatedly sustained for short periods of time. This testing is valuable because in real software systems such peak loads also tend to be sustained for short periods of time.

Further, since all of the relevant network traffic associated with a testis funnelled through a single program (at the intermediate computer), this gives a global view of the behaviour of a system which is distributed across many pieces of hardware, such a global view can then be used to advantage. For example, packets originating from all three tiers (client tier, application server tier and database server tier) of a three tier distributed architecture can be intercepted and buffered. Then, the rate of events in one part of the distributed system can be used to set the rate for forwarding packets to another part of the distributed system and consequently set the rate of events in yet another part of the system (creating a "cruise control" function).

An extension to the preferred embodiment, that may be preferential under some circumstances, would be to buffer packets in the communication stack of each destination computer below the level of the software that is being tested. When, and in what timing pattern the packets are forwarded upward in the stack could be either locally or centrally (based on a global view) managed. This approach provides more precision in the arrival times of the communication events at the software being tested because there is less indeterminate delay in the lower levels of the communication stack.

However, it is more difficult to implement and it intrusively changes part of the software system that is being tested. A similar technique could be used for interprocess communication on a single computer system or between processes that communicate over a bus rather than a network. It would also be possible to buffer the packets at their respective sources, but this would add complexity without advantage.

13 While preferred embodiments of the present invention have been described in the above description, the present invention is not to be limited by the same but only by the properly interpreted scope of the appended claims.

Claims (1)

1. CLAIM

   1 A method of testing a distributed software system which includes a f irst network communicating process and a second network communicating process, the method comprising steps of:

   intercepting network communication level events being sent from the first network communicating process to the second network communicating process; accumulating the intercepted network communication level events in a central location; and forwarding the network communication level events on to the second network communicating process in a time controlled manner, so as to directly control the arrival times of the network communication level events at the second network communicating process for the purpose of testing the distributed software system.

   2. The method of claim 1 wherein the network communication level events are packets.

   3. The method of claim 1 wherein the intercepting step is carried out via a network emulator.

   4. The method of claim 1 wherein the central location associated with the accumulating step is a first-in-first-out (FIFO) buffer.

   S. The method of claim 4 wherein the FIFO buf f er is included in a network emulator which is used to carry out the intercepting step.

   6. The method of claim 1 wherein the first network communicating process is a client and the second network communicating process is a server.

   7. The method of claim 1 wherein the forwarding step forwards the network communication level events to the second network communicating process using a plurality of interarrival intervals of time between arrival of successive packets, each interval being directly and precisely specified by the tester.

   B. The method of claim 1 wherein the data content of the network communication level events is examined to determine the timing of the forwarding of the events on to the second network communicating process.

   9. An apparatus for testing a distributed software system which includes a first network communicating process and a second network communicating process, the apparatus comprising:

   means for intercepting network communication level events being sent from the first network communicating process to the second network communicating process; means for accumulating the intercepted network communication level events in a central location; and is means for forwarding the network communication level events on to the second network communicating process in a time controlled manner, so as to directly control the arrival times of the network communication level events at the second network communicating process for the purpose of testing the distributed software system.

   10. The apparatus of claim 9 wherein the network communication level events are packets.

   11. The apparatus of claim 9 wherein the means for intercepting is carried out via a network emulator. 12. The apparatus of claim 9 wherein the central location associated with the means for accumulating is a first-in-first-out (FIFO) buffer. 30 13. The apparatus of claim 12 wherein the FIFO buffer is included in a network emulator which carries out the function of the means for intercepting. 35 14. The apparatus of claim 9 wherein the first network communicating process is a client and the second network communicating process is a server. is. The apparatus of claim 9 wherein the means for forwarding forwards 40 the network communication level events to the second network communicating process using a plurality of interarrival intervals of time between arrival 16 of successive packets, each interval being directly and precisely specified by the tester.

   16. The apparatus of claim 9 wherein the data content of the network communication level events is examined to determine the timing of the forwarding of the events on the second network communicating process.

   17. A computer program product stored on a computer readable storage medium for, when run on a computer, carrying out the method of claim 1.