DE202016008054U1 - Performance monitoring in a distributed storage system - Google Patents

Abstract

A non-transitory computer-readable medium having stored thereon usable instructions that, when executed, cause the at least one processor to perform operations for measuring performance metrics in a distributed storage system comprising operations: identifying requests sent by clients to the distributed storage system where each request includes request parameters; Generating probe requests based on the identified requests, wherein the probe requests include probe request parameter values representing a statistical sample of the request parameters included in the identified requests; Sending the generated probe requests to the distributed storage system over a network, wherein the distributed storage system is configured to prepare for satisfaction of the respective probe request in response to receipt of the probe request; Receiving replies to the probing requests from the distributed storage system; and outputting at least one performance metric for the distributed storage system based on the received responses.

Description

BACKGROUND

This specification generally relates to monitoring performance in a distributed storage system.

Under protection and subject to the utility model are, according to the provisions of the utility model law, only devices as defined in the appended claims, but no method. Wherever in the description, if appropriate, reference is made to methods, these references are merely illustrative of the device or devices set forth in the appended claims.

In distributed systems, different performance metrics can be tracked to determine the overall health of the system. For example, it can be monitored how long the system takes to respond to a client request (i.e., latency) to ensure that the system responds in a timely manner.

SUMMARY

In general, one aspect of the subject matter described in this specification may be embodied in systems and methods performed by data processing devices that include the actions of identifying requests sent by clients to the distributed storage system, each request containing query parameter values for request parameters ; Generating the probe requests based on the identified requests, wherein the probe requests include probe request parameter values for probe request parameter values representing a statistical sample of the request parameters included in the identified requests; Transmitting the generated probe requests to the distributed storage system over a network, wherein the distributed storage system is configured to prepare for servicing each probe request to receive the probe request; Receiving replies to the probe requests from the distributed memory system and outputting the at least one power metric value that measures a current performance state of the distributed memory system based on the received responses.

Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects and potential advantages of the subject matter will become apparent from the description, drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 Figure 10 is a diagram of an example environment for monitoring the performance of a distributed storage system.

2 Figure 11 is a swim-line diagram of an example process for processing a client request in a distributed storage system.

3 Figure 11 is a swim-line diagram of an example process for monitoring performance in a distributed storage system.

4 Figure 4 is a flow chart of an example process for monitoring performance in a distributed storage system.

5 Figure 13 is a diagram of computing devices that may be used to implement the systems and methods described in this document.

Corresponding reference numbers and markings in the various drawings indicate corresponding elements.

DETAILED DESCRIPTION

In a distributed storage system, there are many factors that can affect system performance. For example, clients can access the system over a public network, such as the Internet. In such a case, performance issues anywhere in the public network may affect the performance of the distributed storage system from the perspective of the particular client. Problems with the distributed storage system itself, e.g. Hardware failures, internal network failures, software bugs or other issues can also impact the performance of the system as perceived by the client.

In some cases, the relationship between the distributed storage system and its clients may be subject to a Service Level Agreement (SLA). An SLA generally includes performance goals that the distributed storage system provider has agreed upon when meeting client requests. For example, an SLA may indicate that the distributed storage system provider guarantees a latency of no more than 10 ms when processing a request. In some cases, the SLA may include actions that are taken if the performance goal is not met, e.g. B: that the provider Refunds to customers. Such arrangements may also include provisions that the distributed storage system provider is not responsible for performance issues beyond its control (eg, public network failures, customer network failures, customer equipment issues).

Even in cases where there is no SLA, a distributed storage system vendor may want to monitor the state of the system in a non-intrusive manner that does not affect the performance of the system in meeting requests for clients ,

Accordingly, the present disclosure describes techniques for monitoring performance in a distributed database system by profiling client requests. An example method involves identifying requests sent by clients to the distributed storage system. These requests may include request parameters, e.g. A request type, simultaneity parameter, and a request destination indicating the data concerning the request. In some cases, requests may be identified based on information sent by clients of the distributed storage system. Based on these identified queries, probe requests are generated containing probe request parameters representing a statistical sample of the request parameters contained in the identified request. The generated probe requests are sent to the distributed memory system which responds by making preparations for the fulfillment of the respective probe request, but does not actually access the data indicated by the request destination. For example, in response to an inquiry request request type "read", the distributed storage system may prepare to read the data indicated by the request destination (e.g., search the data, place the request in the queue, etc.), but can read a field associated with the data that the client can not access. This allows a careful profiling of the system without disrupting the processing of normal client requests. When replies to the probe requests are received from the distributed storage system, performance metrics can be calculated.

In some cases, the process may be performed by the computing device that is in a shared location with the storage system (eg, within the same internal network) to measure the performance of the distributed storage system alone, such that problems beyond the vendor's control (eg failures of public networks) are not indicated by the process.

The techniques described herein can provide the following advantages. By profiling actual client requests to generate probe requests, the present technique enables the monitoring of a distributed storage system by approximating client requests currently being met by the distributed storage system. In addition, by monitoring the state of the distributed storage system without accessing data accessible to clients, the performance impact of such monitoring on actual client requests can be minimized. Further, by monitoring the distributed storage system isolated from other factors beyond the control of the system provider, a more accurate view of the system's true performance can be obtained, wherein the true performance includes the consistent performance a client perceives when requesting service from the distributed storage system, e.g. , In terms of latency. Such information may be useful in determining compliance with an SLA. The techniques may also provide statistical information about the distribution of the customer requests themselves (eg, whether a particular data item is more popular than others), which may inform the provider and the customer as to whether the workload should be adjusted (eg, at Avoid hotspots). In addition, the technique enables the profiling of various aspects of the distributed storage system, e.g. Queue time, actual processing time, server site time, along with the server's current CPU and memory usage, queue length, and other metrics. Furthermore, the techniques may allow easier management of SLA / SLO compliance because the profiling is completely controlled by the distributed storage system vendor.

1 Figure 10 is a diagram of an example environment for monitoring the performance of a distributed database system. As shown, the environment includes 100 a distributed storage system 110 with a variety of servers 112 , each containing a variety of data groups 114 manage. In operation, clients send 120 a request 122 to the distributed storage system 110 , The distributed storage system 110 processes the request 122 and send answers 124 to the clients 120 , The clients 120 send inquiry information 126 to a sounder 130 , This request information 126 Include information about the requests made by a particular client 120 to the distributed storage system 110 were sent, z. For example, the request type, the request parameters, and the request destination of each request. The sounder 130 receives the request information 126 and sends based on the request information 126 exploratory inquiries 132 to the distributed storage system 110 , The distributed storage system 110 processes the sounding requests and gives answers 134 to the program 130 back. The sounder 130 analyzes the answers and gives performance metrics 140 off, which is the current performance of the distributed storage system 110 , certain server 112 in the distributed storage system 110 , certain data groups 114 in the distributed storage system 110 or the current performance of other components in the distributed storage system 110 specify.

The distributed storage system 110 can be a distributed system with a large number of servers 112 which are connected by a local or private network (not shown). In some cases, the local or private network may be located entirely in a single permanent establishment, while in other cases the local or private network may cover a larger area and connect multiple sites. The servers 112 can communicate with each other to the client request 122 to be fulfilled by the clients 120 store, retrieve and update requested data. In some cases, the distributed storage system 110 a distributed database, a distributed file system, or some other type of distributed storage. The distributed storage system 110 can also components to manage and organize the operations of the server 112 in the system.

In the distributed storage system 110 can any server 112 a computer with a processor and a storage device, e.g. A hard drive for storing data from the distributed storage system 110 be managed. In some cases, data can be sent to the different servers 112 distributed according to the distribution policy. For example, a distribution policy may indicate that a particular table or file is in the distributed storage system 110 on a certain number of servers 112 must be stored in order to maintain the redundancy. The distribution policy can also specify that data must be stored in several different locations to maintain geographic redundancy. In some cases, the server can 112 an external storage device or system, e.g. B. use a distributed file system, instead of a directly connected persistent storage.

Each of the servers 112 manages one or more data groups 114 , The data groups 114 can include the entire record from the distributed storage system 110 is managed. Each data group 114 may be data from a portion of a table in a distributed database, one or more files from a distributed file system, or other portions of data in the distributed storage system 110 represent / included. In operation, any request 122 and sounding request 132 through the distributed storage system 110 be analyzed to determine the particular data set based on the request destination 114 to which the request or probe request belongs. The distributed storage system may then request or probe a particular server 112 direct the specific data group 114 managed.

In some cases, the clients can 120 Distributed storage system user 110 be. The clients 120 can also be entities (such as Web sites or applications) that use the distributed storage system 110 to store and retrieve data. Each of the clients 120 can provide information about each request 122 record it to the distributed storage system 110 sends. In some cases, every client can 120 Records the entire request, such as the distributed storage system 110 sent, save. Each client may also provide a summary of the request 122 that has been sent to the distributed storage system, it can store, for. For example, store a number of requests sent with the same set of request parameters. The client 120 For example, it may record the fact that five requests with the request type "read", a simultaneity parameter "weak" and a request destination of the table named "customers" have been sent. This request information 126 For example, at regular intervals by each client 120 to the sounder 130 be sent. The request information 126 can from the clients 120 via a public network, e.g. The Internet, to the sounder 130 be sent. In some cases, the request information 126 be collected through a software process or library that runs on the client and collects information about requests sent by the client. In some cases, the software library may be provided by the distributed storage system vendor.

The sounder 130 Can by the clients 120 analyze received request information and generate a probe profile representing a statistical approximation of the requests made by the request information 126 to be discribed. The sounder 130 For example, request information 126 which will generate 10,000 read-type queries and 5,000 write-type queries, and generate an exploratory profile indicating that 1,000 "read" exploratory requests and 500 "write" exploratory requests should be sent to the Performance of the distributed system processing the original request 122 to emulate and identify. In some cases, the sounder 130 a number of probing requests to be generated 132 so select that the number is big enough for the request 122 to be representative of the clients 120 is sent, but small enough to minimal impact on the performance of the distributed storage system 110 to have.

Based on the sounding profile, the sounder sends 130 the exploratory inquiry 132 to the distributed storage system 110 , In some cases, the sounding request 132 in the format with the requests 122 be identical to that of clients 120 can be sent but may contain a notice that they are probing requests and not requests from clients 120 are. The distributed storage system 110 can sound inquiries 132 receive and process them in the same way as the client requests 122 except that the shared storage system can not access the actual data specified by the request destination of each probe request. In some cases, the distributed storage system 110 instead, access data specifically for the probe 130 are assigned, for. For example, a specific field, column, or metadata value or other value associated with the data specified by the request destination. Thus, the performance of several aspects of the distributed storage system 110 be profiled without processing client requests 122 disturb. For example, probe requests that include simultaneity parameters that would cause the placement of a lock on the requested data may instead cause the distributed memory system 110 places a lock on the probe-specific data so as not to disturb the processing of the client request. Such functionality enables the profiling of the concurrency features of the distributed storage system 110 without affecting the processing of client requests.

As shown, the probe produces 130 one or more performance metrics 140 based on the sent inquiry request 132 and the answers received 134 , In some cases, the performance metrics may include the total system latency as measured by the average time that the distributed storage system has 110 for answering an exploratory inquiry. The performance metrics may also include availability as measured by the ratio of failed probe requests to successful probe requests. The performance metrics may also include local network latency, server queue latency (e.g., the average time that each probe request 132 wait for processing on the server), volume or memory latency, or other performance metrics.

2 Figure 11 is a swim-line diagram of an example process for processing a client request in the distributed storage system. In this 205 the client sends 120 a request to the distributed storage system 110 , at 210 hits the distributed storage system 110 Preparing for the fulfillment of the request. The distributed storage system 110 Preparations made may include, for example, analyzing the request, identifying a particular server storing the requested data, sending the request to the particular server, generating an execution plan for the particular request (eg, determining which tables are being accessed and how the data is manipulated to fulfill the request) or other operations. at 215 accesses the distributed storage system 110 to the data specified by the particular request. This is in contrast to the handling of probe requests by the distributed storage system 110 because the data specified by the requests is not accessed (as described below). at 220 sends the distributed storage system 110 an answer to the request to the client. For example, if the client is reading certain data from the distributed storage system 110 requested, the answer may include the requested data.

3 Figure 12 is a swim-line diagram of an example process for monitoring performance in a distributed database system. at 305 the client sends 120 Requests to the distributed storage system 110 that comes with answers to the requests 310 responding. at 315 asked the client 120 Information about the sent request for the sounder 130 ready. at 320 the sounder generates 130 an exploratory profile based on information about the requests sent by the client. In some cases, the probe may receive information about sent requests from multiple clients and generate the probe profile based on that information. at 325 sends the sounder 130 Exploratory inquiries based on the probing profile for the distributed storage system 110 , at 330 hits the distributed storage system 110 Preparations for the fulfillment of the respective exploratory inquiry. at 335 accesses the distributed storage system 110 to probe-specific metadata associated with the data indicated by the request destination of the respective probe request. at 340 sends the distributed storage system 110 Answers to the sounding requests to the sounder 130 , at 345 the sounder calculates performance metrics based on the answers to the sounding requests.

4 Figure 3 is a flow chart of an example process for monitoring performance in a distributed database system. at 405 Incoming requests sent by clients to the distributed storage system are identified, each request including request parameters. In some cases, the request parameters include a request type, concurrency parameter, and a request destination indicating data in the distributed storage system to which the request belongs.

at 410 Probe requests are generated based on the identified requests. The probe requests include probe query parameters that represent a statistical sample of the query parameters contained in the identified request. In some cases, the generation of the probe requests involves generating a number of probe requests that is less than the number of identified requests. The generation of the probe requests may involve generating a number of probe requests having a particular request type, request parameters, and a particular request destination that is proportional to the number of identified requests with the particular request type, the specific concurrency parameters, and the particular request destination.

at 415 The generated probe requests are sent over a network to the distributed storage system. The distributed storage system is configured to prepare for satisfaction of the respective probe requests in response to receiving the probe request. In some cases, the distributed storage system is configured to read or write data accessible to clients as it prepares to complete the respective probe request in response to receiving the probe request. In some implementations, the data in the distributed storage system includes a probe field that can not be accessed by clients, wherein the distributed memory system is configured to respond to the probe field associated with a request destination in a probe request in response to the reception the probe request accesses.

at 420 Responses to the probe requests are received by the distributed storage system. at 425 At least one performance metric for the distributed storage system is issued based on the received responses. In some cases, outputting at least one performance metric includes outputting a weighted average of the at least one performance metric for a particular data group of the distributed storage system based on the replies to probe requests with request destinations identifying data in the particular data group. The at least one performance metric may include at least one of the following: availability, media latency, queue latency, request preemption latency, or internal network latency. In some cases, the performance metrics may include weighted averages, where the weights used in the calculation are from the query information 126 be derived. For example, if there are 10 times as many requests for weak simultaneity as for high concurrency, the performance data in the subsequent performance metrics for soft simultaneity probes may be 10 times heavier than for probes for high simultaneity.

In some cases, the process involves 400 comparing the at least one performance metric with a service level objective (SLO) having a target value for the at least one performance metric for the distributed storage system. The SLO may be included in a Distributed Storage System Service Level Agreement (SLA). The process 400 may further include determining that the at least one performance metric does not meet the target value, and outputting an indication that the at least one performance metric is not meeting the target value.

5 is a block diagram of the computing devices 500 . 550 which may be used to implement the systems and methods described herein, either as a client or as a server or as a plurality of servers. The computer device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes and other suitable computers. The computer device 550 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smart phones and other similar computing devices. In addition, the computers can 500 or 550 also include USB (Universal Serial Bus) drives. The USB storage media can store operating systems and other applications. The USB flash drives can handle input / output components such as B. a radio transmitter or USB port included, which can be plugged into a USB port of another computer. The components, their connections and relationships and their functions shown herein are intended to be exemplary only and not intended to limit implementations of the inventions described and / or claimed in this document.

computing device 500 includes a processor 502 , a store 504 , a storage device 506 , a high-speed interface 508 that deals with memory 504 and high-speed expansion ports 510 connects, and a low-speed interface 512 that deals with the 514 and storage device 506 combines. Each of the components 502 . 504 . 506 . 508 . 510 and 512 are interconnected using different buses and may be mounted on a common motherboard or otherwise. The processor 502 may instructions for execution within the computing device 500 process, which contain instructions in the memory 504 or on the storage device 506 are saved to graphic Information for a GUI on an external input / output device, such as display 516 that interface with high speed 508 connected to display. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and storage types. There can also be several computer devices 500 Each device provides portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multiprocessor system).

The memory 504 stores information within the computing device 500 , In one implementation, the memory is 504 one or more volatile storage units. In other implementations, the memory is 504 one or more non-volatile storage devices. The memory 504 may also be another form of computer-readable medium, for example a magnetic or optical medium.

The storage device 506 is capable of mass storage for the computing device 500 provide. In one implementation, the storage device may 506 a computer readable medium such as a floppy disk drive, hard disk drive, optical disk device, magnetic tape drive, flash memory or other similar solid state storage device or a variety of other devices, including devices in a storage network or other configurations. A computer program product may be embodied concretely in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer- or machine-readable medium such as the memory 504 , the storage device 506 or memory on a processor 502 ,

The high-speed controller 508 Manages bandwidth-intensive operations for the computing device 500 while the low-speed controller 512 managed less bandwidth-intensive operations. Such an assignment of functions is only an example. In one embodiment, the high-speed controller 508 with memory 504 , Display 516 (eg via a graphics processor or accelerator) and with high-speed expansion ports 510 coupled, which can accommodate various expansion cards (not shown). In the implementation is low-speed controller 512 with storage device 506 and low-speed expansion port 514 coupled. The slow expansion port, which may include various communication ports (eg, USB, B, Ethernet, wireless Ethernet), may be connected to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or Router, z. B. be coupled by a network adapter.

The computer device 500 can be implemented in a number of different forms, as shown in the figure. For example, it can act as a standard server 520 or implemented multiple times in a group of such servers. It can also be considered part of a rack server system 524 be implemented. Additionally it can work in a personal computer like a laptop computer 522 be implemented. Alternatively, components of computer device 500 combined with other components in a mobile device (not shown), e.g. B. device 550 , Each of these devices can be one or more of computing devices 500 . 550 included and an entire system can be made up of multiple computing devices 500 . 550 Being composed communicate with each other.

computing device 550 includes among other components a processor 552 , a store 564 , an input / output device such as a display 554 , a communication interface 566 and a transceiver 568 , The device 550 can also be used with a storage device z. As a microdrive or other device to provide additional memory. Each of the components 550 . 552 . 564 . 554 . 566 and 568 are interconnected using different buses, and several of the components may be mounted on a common motherboard or otherwise.

The processor 552 can be instructions in the computer device 550 execute, including in memory 564 stored instructions. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor can be implemented with any number of architectures. So can the processor 510 for example, a Complex Instruction Set Computer (CISC) processor, a Reduced Instruction Set Computer (RISC) processor, or a Minimal Instruction Set Computer (MISC) processor. For example, the processor may coordinate the other components of the device 550 such as control of user interfaces, applications, by the device 550 running, and wireless communication through device 550 provide.

processor 552 can with a user via control interface 558 and display interface 556 that with an ad 554 connected, communicate. the display 554 For example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED display (Organic Light Emitting Diode) or other suitable display technology. The display interface 556 can be an appropriate circuit as a driver of the ad 554 to present graphical and other information to a user. The control interface 558 can receive commands from a user and send them to the processor 552 convert. In addition, an external interface 562 in conjunction with processor 552 be provided to close range communication of device 550 to allow with other devices. The external interface 562 For example, in some implementations, it may be a wired connection or a wireless connection in other implementations, and multiple interfaces may be used.

The memory 564 stores information within the computing device 550 , The memory 564 can be implemented as a computer-readable medium or as one of several computer-readable media, as a volatile memory or as a non-volatile memory or non-volatile memory. extended memory 574 can also be provided and with the device 550 via extension interface 572 For example, it may include a single in-line memory module (SIMM) card interface. Such an expansion memory 574 can add extra storage space for device 550 can also provide applications or other information for device 550 to save. In particular, expansion memory can 574 It also contains instructions for completing or completing the processes described above, and it may also contain secure information. Accordingly, expansion memory 574 for example, as a security module for device 550 be provided and may be programmed with instructions for safe device use 550 enable. In addition, secure applications can be provided via the SIMM cards, along with additional information such as storing identification information on the SIMM card in a manner that can not be hacked.

The memory may include, for example, flash memory and / or NVRAM memory as discussed below. In one implementation, a computer program product is tangibly implemented in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 564 , the memory extension 574 or the processor memory 552 , for example via the transceiver 568 or the external interface 562 can be received.

device 550 can via communication interface 566 which may include, if necessary, a digital signal processing circuit, communicate wirelessly. The connection interface 566 can establish connections to various types of communication or protocols, including GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS among others. Such communication may be, for example, by radio frequency transceivers 568 occur. Additionally, short distance communication may take place, such as using a Bluetooth, WLAN or other such transceivers (not shown). In addition, GPS (Global Positioning System) receiver module 570 additional wireless device data associated with navigation and location 550 may be used by applications that may be on device 550 be executed.

device 550 can also be done using the audio codec 560 who audibly communicate spoken information from a user and convert it into usable digital information. Audio codec 560 may also generate audible tones to a user, for example through a speaker, for example, in a handset of the device 550 , These tones may include tones of voice telephone calls, may include recorded sounds (eg, voice messages, music files, etc.), and may also include sounds from on-device applications 550 were created.

The computer device 550 can be implemented in a number of different forms, as shown in the figure. For example, it can be used as a mobile phone 580 be implemented. He can also be part of a smartphone 582 , a Personal Digital Assistant or similar mobile device.

Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuit, in specially designed ASICs (application specific integrated circuits), in computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be a special purpose processor or processor, and for receiving Data and instructions from and for transmitting data and instructions to a storage system, at least one Input device and at least one output device is coupled.

These computer programs (also known as programs, software, applications, or code) contain machine instructions for a programmable processor and may be translated into a high-level procedural and / or object-oriented programming language and / or assembly / machine language. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, device, and / or device (eg, magnetic disks, optical disks, memory, programmable logic devices (PLDs)) that uses to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

For interaction with a user, the techniques and systems described herein may be used on a computer with a display (eg, a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user and with a keyboard and a pointing device (eg, a mouse or a trackball) through which the user can forward input to the computer. Other types of devices may also be used to provide interaction with a user; for example, a feedback provided to the user may be some form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and an input from the user may be received in any form, including acoustic, voice or touch inputs.

The systems and techniques described herein may be implemented in a computer system that includes a back-end component (eg, as a data server) or that includes a middleware component (eg, an application server) or a front-end Component (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the systems and techniques described herein) or any combination of such back end, middleware, or front End components includes. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), peer-to-peer networks (with ad hoc members and permanent members), network computer infrastructures and the Internet.

The computer system may include client and server. A client and server are generally remote from each other and typically interact over a communications network. The relationship between client and server arises because of computer programs that run on the respective computers and that have a client-server relationship with each other.

Other implementations are summarized in the following examples:

Example 1: A computer-implemented method executed by one or more processors for measuring performance metrics in a distributed storage system, the method comprising: identifying requests sent by clients to the distributed storage system, each request including request parameter values for request parameters; Generating the probe requests based on the identified requests, wherein the probe requests include probe request parameter values for probe request parameter values representing a statistical sample of the request parameters included in the identified requests; Transmitting the generated probe requests to the distributed storage system over a network, wherein the distributed storage system is configured to prepare for servicing each probe request to receive the probe request; Receiving replies to the probe requests from the distributed memory system and outputting the at least one power metric value that measures a current performance state of the distributed memory system based on the received responses.

Example 2: The method of Example 1, wherein generating the probe requests includes generating a number of probe requests that is less than the number of identified requests.

Example 3: The method of Example 1 or 2, wherein the query parameters include a request type, simultaneity parameter, and a request destination indicating data in the distributed storage system to which the request belongs.

Example 4: The method of Example 3, wherein generating the probe requests includes generating a number of probe requests a certain request type, specific request parameters, and a particular request destination that is proportional to the number of identified requests with the particular request type, the specific concurrency parameters, and the particular request destination.

 Example 5: The method of any one of Examples 1 to 4, wherein outputting at least one performance metric comprises outputting a weighted average of the at least one performance metric for a particular data group of the distributed storage system based on the replies to probe requests with request destinations that identify data in the particular data group , includes.

Example 6: The method of any one of Examples 1 to 5, wherein the at least one performance metric includes at least one of the following: availability, media latency, queue latency, request preemption latency, or internal network latency.

Example 7: The method of any one of Examples 1 to 6, wherein the distributed storage system is configured to not read or write data accessible to clients when preparing to fulfill the respective probe request in response to the receipt the exploratory request.

Example 8: The method of any one of Examples 1 to 7, wherein the data in the distributed storage system includes a probe field that is inaccessible to clients, and wherein the distributed memory system is configured to point to the probe field associated with a request destination in a probe request, accesses in response to receipt of the probe request.

Example 9: The method of any one of Examples 1-8, further comprising comparing the at least one performance metric with a service level objective (SLO) having a target value for the at least one performance metric for the distributed storage system.

Example 10: The method of Example 9, further comprising determining that the at least one performance metric does not meet the target value, and outputting an indication that the at least one performance metric does not meet the target value.

Example 11: The method of Example 9 or 10, wherein the service level target is included in a distributed storage system service level agreement (SLA).

Example 12: The method of any one of Examples 1 to 11, wherein the probe requests are statistical representations of the identified requests.

Example 13: A non-transitory, computer-readable storage medium having stored thereon usable instructions that, when executed, cause the at least one processor to perform operations for measuring performance metrics in a distributed storage system comprising operations: identifying requests made by clients to the computer distributed storage systems are sent, each request containing request parameter values; Generating the probe requests based on the identified requests, the probe requests including probe request parameters representing a statistical sample of the request parameters included in the identified requests; Transmitting the generated probe requests to the distributed storage system over a network, wherein the distributed storage system is configured to prepare for servicing each probe request to receive the probe request; Receiving replies to the probe requests from the distributed storage system and outputting the at least one distributed memory system performance metric value based on the received responses.

Example 14: The computer readable medium of Example 13, wherein generating the probe requests includes generating a number of probe requests that is less than the number of identified requests.

Example 15: The computer readable medium of Example 13 or 14, wherein the query parameters include a request type, simultaneity parameter, and a request destination indicating data in the distributed storage system to which the request belongs.

Example 16: The computer readable medium of Example 15, wherein generating the probe requests includes generating a number of probe requests having a particular request type, request parameters, and a particular request destination that are proportional to the number of identified requests with the particular request type, the determined concurrency parameters, and the particular request Request is.

Example 17: The computer readable medium of any one of Examples 13 to 16, wherein outputting at least one performance metric outputting a weighted average of the at least one performance metric for a particular data group of the distributed memory system based on the replies to probe requests with the destination data in the particular data group identify, includes.

Example 18: The computer-readable medium of any one of Examples 13 to 17, wherein the at least one performance metric includes at least one of the following: availability, media latency, queue latency, request-ahead latency, or internal network latency.

Example 19: The computer readable medium of any one of Examples 13 to 18, wherein the distributed storage system is configured to not read or write data accessible to clients when preparing to fulfill the respective probe request in response to the data request Receiving the probe request performs.

Example 20: A system for measuring performance metrics in a distributed storage system, comprising: identifying requests sent by clients to the distributed storage system, each request including request parameter values; Generating the probe requests based on the identified requests, the probe requests including probe request parameters representing a statistical sample of the request parameters included in the identified requests; Transmitting the generated probe requests to the distributed storage system over a network, wherein the distributed storage system is configured to prepare for servicing each probe request to receive the probe request; Receiving replies to the probe requests from the distributed storage system and outputting the at least one distributed memory system performance metric value based on the received responses.

Although several implementations have been described in detail above, other modifications are possible. In addition, the logical operations in the illustrations do not necessarily require the depicted order or sequential order to achieve the desired results. Additional steps may be added to or omitted from the described procedures, and other components added to or omitted from the described systems. Accordingly, other implementations are within the scope of the following claims.

Claims (8)

A non-transitory computer-readable medium having stored thereon usable instructions that, when executed, cause the at least one processor to perform operations for measuring performance metrics in a distributed memory system comprising the operations of: Identifying requests sent by clients to the distributed storage system, each request including request parameters; Generating probe requests based on the identified requests, wherein the probe requests include probe request parameter values representing a statistical sample of the request parameters included in the identified requests; Sending the generated probe requests to the distributed storage system over a network, wherein the distributed storage system is configured to prepare for satisfaction of the respective probe request in response to receipt of the probe request; Receiving replies to the probing requests from the distributed storage system; and Output at least one performance metric for the distributed storage system based on the received responses. The computer-readable medium of claim 1, wherein generating the probe requests includes generating a number of probe requests that is less than the number of identified requests. The computer readable medium of claim 1, wherein the query parameters include a request type, simultaneity parameters, and a request destination indicating data in the distributed storage system to which the request belongs. The computer-readable medium of claim 3, wherein generating the probe requests includes generating a number of probe requests having a particular request type, request parameters, and a particular request destination that is proportional to the number of identified requests having the particular request type, the determined concurrency parameters, and the particular request destination. The computer-readable medium of claim 3, wherein outputting at least one performance metric includes outputting a weighted average of the at least one performance metric for a particular data group of the distributed memory system based on the responses to probe requests with request destinations identifying data in the particular data group. The computer readable medium of claim 1, wherein the at least one performance metric includes at least one of the following: availability, media latency, queue latency, request preemption latency, or internal network latency. The computer-readable medium of claim 1, wherein the distributed storage system is configured to not read or write data accessible to clients as it prepares to complete the respective probe request in response to receiving the probe request. A system for measuring performance metrics in a distributed storage system, comprising: Memory for storing data; and One or more processors that can be used to access memory and perform operations that include: Identifying requests sent by clients to the distributed storage system, each request including request parameters; Generating probe requests based on the identified requests, wherein the probe requests include probe request parameter values representing a statistical sample of the request parameters included in the identified requests; Sending the generated probe requests to the distributed storage system over a network, wherein the distributed storage system is configured to prepare for satisfaction of the respective probe request in response to receipt of the probe request; Receiving replies to the probing requests from the distributed storage system; and Output at least one performance metric for the distributed storage system based on the received responses.

Images