US20180337817A1

US20180337817A1 - Performance evaluation of applications that access external resources

Info

Publication number: US20180337817A1
Application number: US15/898,053
Authority: US
Inventors: Lewis Karl Cirne; James Robert Gochee; Saxon Michael D'Aubin
Original assignee: New Relic Inc
Current assignee: New Relic Inc
Priority date: 2009-09-10
Filing date: 2018-02-15
Publication date: 2018-11-22
Also published as: US9935812B1

Abstract

Evaluating application performance includes: obtaining a request, processing the request, including invoking a function that accesses an external resource, determining whether a response time associated with invoking the function exceeds a threshold, and in the event that the response time exceeds the threshold, capturing a runtime attribute associated with the function.

Description

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of co-pending U.S. patent application Ser. No. 12/584,833, entitled PERFORMANCE EVALUATION OF APPLICATIONS THAT ACCESS EXTERNAL RESOURCES filed Sep. 10, 2009 which is incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

As computer software becomes more sophisticated, performance evaluation has also become a more complex task. In some client-server applications, for example, many different calls are made to the server application and it can be difficult and time-consuming to pinpoint performance bottlenecks. One technique often employed by software developers is to manually insert logging code to facilitate performance evaluation. The addition of logging code, however, can negatively impact the overall performance of the system and may only provide limited information on how the bottlenecks occurred.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

FIG. 1 is a block diagram illustrating an embodiment of a performance evaluation system.

FIG. 2 is a flowchart illustrating an embodiment of an application monitoring process.

FIG. 3 is a flowchart illustrating another embodiment of another application monitoring process.

FIG. 4 is a flowchart illustrating an embodiment of a process configured to perform post processing.

FIG. 5 is a diagram illustrating an example user interface displaying performance monitoring results of an application.

FIG. 6 is a diagram illustrating an example user interface displaying the diagnostic information pertaining to a selected call.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
FIG. 1 is a block diagram illustrating an embodiment of a performance evaluation system. System 100 includes an application server 102, on which one or more applications 104 are installed and executed. The application server receives requests such as 120. The requests may be sent by client devices, other applications (either on the same server or on a different server), function calls within application 104, or other appropriate request generating source. For example, in some embodiments, the application server is a web server, the application is a web application, and the request may be a Hypertext Transfer Protocol (HTTP) request.
To process a request, the application will access an external resource (i.e., a resource that is not an integral part of the application), such as database 106, web service 108, network service 110, or other appropriate resource. An external resource is typically installed and executed on a device other than the application server, but may sometimes be installed and executed on the same application server in a different process. The application is programmed to accesses the external resource using a suitable protocol. For example, to access a database such as Oracle™ or Microsoft SQL Server™, Structured Query Language (SQL) calls are made; to access web services such as credit card processing service or message queuing service, HTTP calls are made; and to access network services such as socket services, networking calls such as open_socket are made. These external resource access calls sometimes cause performance bottlenecks, and can be specifically identified and diagnosed using techniques described below.
A performance monitor 110 is configured to monitor the performance of application 104. In some embodiments, the performance monitor is implemented as special instrumentation code (also referred to as probes) that cooperates with the application to detect any slowdown in the application's access to external resources. For example, in some embodiments where the application is implemented using Ruby On Rails™ (Rails), the source code of certain external resource access classes is replaced with performance monitoring code that carries out the same function and additionally performs monitoring functions. In a more specific example, SQL database access calls all flow through a class named ActiveRecord::Base, which is replaced with instrumented code that invokes the original code, and additionally collects performance information such as the amount of time spent on accessing the external resource (referred to as the response time) and other runtime attributes. In some embodiments where the application is implemented using JAVA™, the byte code of certain external resource access functions and libraries is replaced with byte code with additional monitoring functions. Other implementations such as binary code replacement may be used as appropriate.
FIG. 2 is a flowchart illustrating an embodiment of an application monitoring process. Process 200 maybe performed by application servers such as 102. At 202, a request is received. The request may be received from a client, another server, a different application running on the same server, or a function call in the same application. At 204, the request is processed. A response, if required, is generated and provided to the requester based on results of the invocation. In some embodiments, the processing includes invoking a function that accesses an external resource such as a database, a web service, a network connection, etc. The function has been previously instrumented to include monitoring code to determine the function's response time. In some embodiments, the total response time, i.e., the total amount of time spent processing the request (e.g., a Process_HTTP_Request function) is determined. In some embodiments, the remote function call response time, i.e., the amount of time required to execute a remote function call to the external resource (e.g., a SQL_Select call that is invoked by the Process_HTTP_Request function) is determined. In some embodiments, both types of response time are recorded.
At 206, it is determined whether the response time exceeds a predefined threshold. Depending on whether the response time is a total response time or a remote function call response time, different thresholds may be set. In some embodiments, both types of response time are compared with their respective thresholds. If the response time does not exceed the threshold, the performance of the function's invocation is considered normal and no further process is required. If, however, the response time exceeds the threshold, at 208, further investigation is needed and one or more runtime attributes associated with the invocation of the function are captured. By examining these runtime attributes, the software developer can identify certain causes for the slowdown. For example, in some embodiments, the runtime attributes include a stack trace of the function's stack frame(s), which can be used to pinpoint certain code that invokes the external resource access and causes the slowdown. In some embodiments, the runtime attributes include parameter(s) associated with the invocation of the function. For example, an incorrect parameter can slow down the function call. In some embodiments, the runtime attributes include the return value/payload information of the function, which may indicate that too much data is being passed and therefore slowing down the access. In some embodiments, the runtime attributes include performance statistics, which may indicate that resource constraints or a heavy load on the external resource is causing the slowdown.
FIG. 3 is a flowchart illustrating another embodiment of another application monitoring process. In this example, at 302, a request is received at the application server. At 304, the request is processed, and a database access function is invoked in response to the request. For example, the request may ask for account information of a user. To generate the response, the application performs a database access function to get the account information from a user database. At 306, the start and finish times of the database access function are measured based on previously inserted instrumentation code in the database access function. Thus, the response time of the database access function call is determined based on the measured start time and finish time. At 308, it is determined whether the response time that is measured exceeds a predefined threshold. No further action is required if the response time does not exceed the predefined threshold. If, however, the threshold is exceeded, at 310, one or more runtime attributes are captured. In addition to the runtime attributes described above in connection with process 200, in some embodiments additional runtime attributes are saved for post processing. For example, database connection information of the database access call, which includes information pertaining to user credential, database host, port, and/or any other appropriate information required for making the database connection, is saved in data structures or other memory that is accessible by the post processing code. The function is marked for post processing.
At 312, post processing is performed to get more in-depth diagnostic information. Since post processing operations can be expensive to perform, they are sometimes performed asynchronously, in a thread or a process that is separate from the function invocation to more efficiently batch process multiple calls and reduce performance impact on the main application.
The following is a pseudocode example of an instrumented database access function:


	Function Instrumented_SQLCall

	record start time
	execute Original_SQLCall
	record finish time
	if (finish time − start time) > threshold

	save configuration
	get runtime attributes such as stack trace, etc.

FIG. 4 is a flowchart illustrating an embodiment of a process configured to perform post processing. In this example, process 400 starts by locating database access calls that are deemed to be too slow (i.e., execution duration exceeding threshold), at 402. In some embodiments, such function calls are marked and/or stored in memory so that they can be located by the post processing steps. In this example, for each slow function call, at 404, database connection information is obtained from the server application. In some embodiments, the connection information is included in a configuration file of the application or statically encoded in the application. In some embodiments, the connection information is stored in memory location(s) and/or data structure(s) accessible by the post processing steps. The database connection information includes the user credential, database host, port, etc. associated with the original database access call.
At 406, a new database connection is opened using the same database connection information as the original call. At 408, an explain plan is generated for the original database call, over the new database connection. The explain plan is a database feature that describes how the database would execute the database access function. For example, whether indexes are used, which indexes are used, what type of database operation is performed, if temporary tables are used, and how many rows would be accessed. Analysis of the explain plan can be helpful for determining the cause for the slowdown. For example, queries on tables that don't use an index run quickly when the number of rows is small but progressively get slower as the number of rows grows. Knowing that a query is slow, looking at an explain plan and seeing that a large number of rows is accessed and that an index is not being used is enough information for a software developer to quickly see that adding an index will speed up queries.
In some embodiments, post processing further includes invoking a debug version of the function. The debug function is invoked under substantially the same condition, using the same parameters as the invocation of the original function. The debug version of the function will provide extra diagnostic information such as log information, break points, or any other appropriate debugging facilities.
The following is a pseudo code example for a post processing function that operates in a separate thread and diagnoses slow SQL calls:


Function Post_Process_SQL

for each slow SQLCall

	get database connection information
	open new database connection
	generate explain plan (SQLCall(new database connection))

FIG. 5 is a diagram illustrating an example user interface displaying performance monitoring results for a service call to an application. The access duration of calls to external resources, including various SQL calls and memory cache (MemCache) access calls, are measured. In this example, call times in the “Exclusive” column correspond to call durations minus any nested calls to child functions, which more accurately represent how long the calls take than the times recorded in the “Duration” column, which includes time spent on nested calls. A threshold of 100 ms is used in this example. Calls that took longer than 100 ms to complete, such as the RealAgent#find_by_sql call that occurred at time 0.036, find_and_aggregate call that occurred at 1.520, and layouts/application.rhtml Template call at 276.643 are highlighted, indicating that these calls warrant special attention since their execution time exceeded the threshold.
The software developer can select a call from the table to view further performance monitoring details. FIG. 6 is a diagram illustrating an example user interface displaying the diagnostic information pertaining to a selected call. In this example, performance information of a SQL call that is made at time 0.036 and lasted for 640 ms is displayed (note that entries in the SQL select statement are largely omitted for purposes of simplicity). The explain plan for the SQL call indicates that a large number of rows (172,753) are returned, and a filesort operation (invoked by the “Order By” operation of the SQL call) is performed. These factors contribute to the slow execution of the call. Additionally, the stack trace shows the functions and line numbers of the chain of calls that lead to this SQL call, starting at “load_time_zone” of the “memory_watcher” file and ending at the “most_recent_run” of the “virtual_agent” file. Knowing the causes and locations of the slowdown can help the software developer make decisions about modifying the database access call to reduce the amount of access time.
Performance evaluation of applications that invoke function calls that access external resource has been described. Although database access and using explain plan to identify bottlenecks in the access calls have been discussed extensively for purposes of example, the technique can also be used to diagnose slowdowns in other types of external resource access.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims

What is claimed is:

1. (canceled)

2. A system for evaluating application performance, comprising:

a processor configured to:

process a request, including invoking a function that accesses an external resource;

determine whether a response time associated with the invocation of the function exceeds a threshold;

in response to a determination that the response time exceeds the threshold:

mark the invocation of the function for post processing to obtain a marked invocation of the function; and

capture a runtime attribute associated with the marked invocation of the function; and

post process the marked invocation of the function to obtain additional performance information; and

a memory coupled to the processor, configured to provide the processor with instructions.

3. The system of claim 2, wherein:

the runtime attribute includes a stack trace associated with the marked invocation of the function; and

the processor is further configured to post process the marked invocation of the function comprises to:

analyze, based on the stack trace, the marked invocation of the function to identify code invoking the accessing of the external resource.

4. The system of claim 3, wherein the post processing is performed asynchronously.

5. The system of claim 2, wherein the function is a database access function.

6. The system of claim 5, and the processor is further configured to generate an explain plan of the database access function.

7. The system of claim 6, wherein the runtime attribute includes database connection information associated with the invocation of the database access function, and generating the explain plan includes opening another database connection using the database connection information.

8. The system of claim 7, wherein the database connection information includes user credential information, database host information and port information.

9. The system of claim 2, wherein the runtime attribute includes a parameter that is used to invoke the function.

10. The system of claim 2, wherein the runtime attribute includes a return value of the function.

11. The system of claim 2, wherein the runtime attribute includes performance statistics.

The system of claim 2, wherein the function was previously modified to include instrumentation code configured to measure response time.

12. The system of claim 2, wherein the external resource includes a web service.

13. The system of claim 2, wherein the external resource includes a networking resource.

14. A method for evaluating application performance, comprising:

processing a request, including invoking a function that accesses an external resource;

determining whether a response time associated with invoking the function exceeds a threshold;

in response to a determination that the response time exceeds the threshold:

marking the function for post processing to obtain a marked invocation of the function; and

capturing a runtime attribute associated with the marked invocation of the function; and

post processing the marked invocation of the function to obtain additional performance information.

15. The method of claim 14, wherein the function is a database access function.

16. A computer program product for evaluating application performance, the computer program product being embodied in a computer readable storage medium and comprising computer instructions for:

in response to a determination that the response time exceeds the threshold:

17. The computer program product of claim 16, wherein the function is a database access function.