WO2016039767A1

WO2016039767A1 - Assign error rate to memory

Info

Publication number: WO2016039767A1
Application number: PCT/US2014/055360
Authority: WO
Inventors: Anys Bacha
Original assignee: Hewlett Packard Enterprise Development LP
Current assignee: Hewlett Packard Enterprise Development LP
Priority date: 2014-09-12
Filing date: 2014-09-12
Publication date: 2016-03-17
Anticipated expiration: 2017-03-12

Abstract

Techniques for assigning error rates to memory are described. In one aspect, an indication of physically addressable memory in a system is received. The indication may include available error rates and a range granularity for assigning error rates to ranges of the physically addressable memory. Error rates may be assigned to each range of the physically addressable memory.

Description

ASSIGN ERROR RATE TO MEMORY

BACKGROUND

[00013 Computers include a wide variety of components. For exampie, most computing devices include a processor, volatile memory, such as dynamic random access memory (DRAM), disk drives, input / output controllers, host bus adaptors, network interface controllers, and any number of other components. One thing a large number of these components have in common is the need to be supplied with power to operate. Almost ail components may specify a minimum operating voltage and/or current in order to ensure error free operation of the component.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] FIG. 1 depicts an example system that may utilize the assigning error rate to memory techniques described herein.

[0003] FIG. 2 depicts an example computing device that may utilize the memory error rate assignment techniques described herein.

[0004] FIG. 3 depicts another exampie computing device that may utilize the memory error rate assignment techniques described herein.

[00053 ^IG. 4 depicts an exampie of a flow diagram for assigning error rates to memory according to techniques described herein.

[00063 P'G- ⁵ depicts another exampie of a flow diagram for assigning error rates to memory according to techniques described herein. DETAILED DESCRIPTION

[0007] Power usage by computing devices is one of the larger expenses incurred by operators of computer data centers. Not only do the computers use power to operate, the operation of the computers produces heat. Because the heat may damage the computing systems, additional power may be used to operate equipment, such as liquid and air cooling systems, to dissipate the heat generated by the computers. As such, reducing the power used by a computing system may help to reduce the overall expense of running a data center;

[0008] As mentioned above, many components in a computer may require power to operate. One such component is the memory system. In many implementations, the memory system of a computer is provided by dynamic random access memory {DRAM} or static random access memory (SRAM). Although these two types of memory have different physical and operating characteristics, they both share the common feature that in order to ensure accurate storage of data, a certain voltage level should be supplied to the memory. For purposes of this description, this voltage level is referred to as the nominal voltage. In other words, the memory device provider may specify that so long as at least the specified nominal voltage is supplied to the memory device, the memory device will not introduce any errors based on insufficient voltage.

[0009] Although providing the nominal voltage to the memory device may ensure that no errors are generated by the device, it is not necessarily true that providing less than the nominal voltage to the memory device will result in the memory device introducing errors. The semiconductor manufacturing process used in creating memory devices may be a variable process meaning that every memory device may have slightly different operating characteristics. For example, the operating characteristics of devices produced from different dies on the same semiconductor wafer may have slightly different nominal voltages due to natural variations in the manufacturing process. Manufacturers may specify a nominal voltage that ensures proper operation, but it is quite possible that the device would still not introduce errors at a lower voltage (although such operation would not be ensured by the device manufacturer). [00103 Although not explicitly mentioned above, an assumption is often made in the design of computing systems, and their memory systems in particular, that al! data requires accurate storage. Thus, system designers may design the memory system such that the specified nominal voltage is supplied to the memory system at ai! times, thus ensuring that no errors are introduced due to insufficient voltage. However, this assumption may not always be true. For example, large amounts of data that is stored and processed by computing systems may be referred to as "trivial" data, whose absoiute accuracy may not be necessary. For example, the data may be of such limited value that loss of the data has no meaningful impact. As another example, errors included in the data may not render the data unusable.

[0011] As a more concrete example of cases wherein errors in data may be acceptable, consider one large use of large computing data centers: social media. At the time of submission of the instant disclosure, Facebook™ is the dominant social media platform. Although there may be some important information conveyed via Facebook™, large portions of the data include trivia! items. One use of social media is posting photographs on a social media site and receiving comments on those photographs. An example of an especially trivial use of sociai media is a current trend wherein users order a meai at a restaurant, photograph the meal, and then post the photograph to social media sites, where other users may then comment on the photograph.

[0012] in such cases, absoiute data integrity may not be necessary. For example, on sociai media sites, people may comment, or provide other text. If the memory storing the text data were to introduce errors, such as errors introduced due to voltage less than the nominal voltage being supplied, those errors may manifest themselves as alterations to the text. For example, the errors may appear as typographical errors. The human mind is adept at understanding text, even when typographical errors are included, thus the errors may not significant!y alter the conveyance of information, in the case of a photograph, errors in the memory storing the photograph may manifest as color changes, or portions of the photograph being distorted. Again, in many cases, the image depicted by the photograph may still be ascertainable by a viewer. As such, in many cases, ©yen when there are errors introduced by the memory storing text or a photograph, the information conveyed may still be understood. Errors introduced by memory may not be fatal, as the data can still be understood.

[0013] Even in a case where the errors introduced by file memory were of such a nature that the underlying data becomes unusable, the proper response may be that such a toss of data is acceptable. Given the trivial nature of some data conveyed over social networking, the loss of the data may have such a limited impact that it is inconsequential.

[0014] The techniques described herein provide for the ability of a computer system to reduce overall power usage by providing a mechanism to selectively provide less than the nominal voltage to portions of the memory subsystem. The memory may be periodically characterized to determine the error rate introduced by the memory at voltage levels less than the nominal voltage. The operating system of the computer may configure different portions of memory to operate at different voltage levels, thus effectively assigning different error rates to the different portions of memory. The operating system may then store data according to the level of tolerance of errors. Data that cannot tolerate errors may be stored by memory receiving nominal voltage, while data that is more tolerant of errors may be stored in memory receiving less than nominal voltage.

[00153 Although the previous, description has been in terms of social media, it should be understood that the techniques described herein are not so limited. The techniques deschbed herein are applicable in any application in which the stored data can tolerate a given rate of errors introduced by the memory system. Furthermore, although DRAM and SRAM are mentioned above, it should be understood that the techniques described herein are not dependent on the particular form of memory. Any memory that specifies a nominal voltage to ensure error free operation, but is still able to operate (perhaps with some level of introduced errors) at less than the nominal voltage, is suitable for use with the techniques described herein. [0016] FIG. 1 depicts an example system that may utilize the assigning error rate to memory techniques described herein. System 100 may include a processor 110, a memory 120, a voltage source 130, and a non-volatiie memory 150. The processor may be of any type suitable for use in a computer. The processor may be a single or mu!ticore processor. The techniques described herein are not dependent on any particular type of processor.

[0017] The system may also include memory 120 coupled to the processor. As mentioned above, some exampies of memory may include DRAM and SRAM. However, the techniques described herein are not limited to those types of memory. The memory may have a nominal voltage which may be specified by the memory manufacturer, if the memory is provided with the nominal voltage, the memory manufacturer may provide guarantees that no errors wiii be introduced due to insufficient voitage. As mentioned above, providing less than the nominal voltage may allow the memory to function, but the manufacturer may no longer guarantee that errors due to insufficient voitage wit! not be introduced.

[0O1SJ The system may also include a voltage source 130. The voitage source may be coupled to the memory 120. The voltage source may be a variable voltage source that is able to supply the nominal voltage to the memory as well as a reduced voltage. The reduced voitage may be referred to as a margined voltage. In other words, the margined voitage may be a reduced percentage of the nominal voltage. The processor 110 may be coupled to the voitage source 130 and may determine what voltage level wiil be supplied to the memory 120.

£001SJ The system may aiso include a non-volatile memory 150. The non-volatiie memory may be used to store operating characteristics of the memory 120 when the memory is operating at different voltage ievels. For example, the non-volatile memory may store an expected error rate of the memory for a plurality of possible voitage levels, including both the nominal as well as a plurality of margined voltage ievels. The process of obtaining the operating characteristics of the memory is described in further detail below. [0020] fn operation ^ the system 100 may be initialized The processor 110 may determine that memory 120 does not need to be supplied the nominal voitage. For example, the processor may determine that system 100 can toierate a certain error rate introduced by the memory. The processor may access non-volatile memory 150 to retrieve the voitage level that results in a memory introduced error rate that is acceptable. The processor may configure the voltage source 130 to supply the memory with the voltage levei retrieved form the non-volatile memory.

[0021] FIG. 2 depicts an example computing device that may utilize the memory error rate assignment techniques described herein. Device 200 may include a processor 210, memory 220-{a-c), a voitage source 230, non- transitory processor readable medium 240, and non-vo!atiie memory 250. The processor 210, and non-volatile memory 250 are substantially the same as processor 110 and non-volatile memory 150 described above in FIG. 1. For clarity, the description of the processor and the non-volatile memory are not repeated here.

[0022] The memory 220~{a~c) may be provided in different forms. For example, the memory may be in the form of dual inline memory modules (DIMM)s. Each DIMM may contain one or more memory elements, such as memory chips. The memory as a whole may be thought of as a range of physical addresses. For example, expressed in hexadecimal notation, the memory range of the device shown In FiG. 2 may be from 0x0000 to OxFFFF. Each address may represent a storage location of an individuai byte of data. This range of addresses may be referred to as the physically addressable memory range. Although a specific address range and number of memory blocks are shown, it shouid be understood that this is for purposes of description. The techniques described herein are not dependent on any particular form or layout of memory elements.

[0023] In many devices, memory operations are performed on multiple bytes of data at a time. These multiple bytes of data may be referred to as a cache line. For example, in a system operating with S byte cache lines, instead of manipulating an individual byte of data referenced by an address, an 8 byte cache fine containing the address of interest may be manipulated. Devices exist with different size cache tines, and in some cases the cache line size may be 1 , which is essentially the same as manipulating individual byte sized data. The techniques described herein are not dependent on any particular size of cache line.

[0024] The memory 220-(a-c) may provide memory elements that make up the physicai!y addressable memory range. For example* memory 22G~a may provide the memory elements that store physically addressable memory range 0x0000 - OxOFFF, memory 220-b may provide the memory elements that store physically addressable memory range 0x000™ 0x3FFF, and memory 220-c may provide the memory elements that store physically addressable memory range 0x4000 - OxFFFF. For purposes of this description, a physically addressable memory range refers to a range of memory for which the error rate may be assigned by adjusting the voltage supplied to the memory elements. For example, memory 220-a provides memory elements to store physical addresses OxOOOO-OxOFFF. The actual components, such as individual chips that make up the memory are unimportant What should be understood is that a range of physically addressable memory addresses is a range of addresses for which the supply voltage can be changed In order to produce the desired error rate.

[0025] The granularity of the memories 220~(a-c) may be different As shown in FIG. 2, memory 220-a is 0x1000 bytes long, memory 220-b is 0x3000 byte long, while memory 220-c is OxCOQG bytes long. The granularity of memory determines the ability to assign different error rates. For example, because memory 220-a encompasses the physically addressable memory range from 0x0000 to OxOFFF, ail of those addresses will have the same assigned error rate.

[0026] Device 200 may also include a voltage source 230. Voltage source 230 may provide voltage needed for the operation of memory 220-{a-c). The voltage source may provide both a nominal voltage 225 as well as a margined voltage 226. As described above, when supplied with the nominal voltage, the memory will not introduce any supply voltage induced errors. The margined voltage may be a voltage that is less than the nominal voltage. When supplied with the margined voltage* the memory can no longer be guaranteed to not introduce errors related to supply voltage. The rate of errors introduced may be characterized and stored, as will be described below.

[0027] For ease of description, the device 200 is shown with power supply 230 supplying the nominal voitage and 1 margined voltage 226.

However, this is done for simplicity of explanation and not by way of limitation. The techniques described herein are equally applicable if there are multiple different margined voltages supplied simultaneously.

[0028] The device 200 may also include a non-transitory processor readable medium 240 containing a set of instructions thereon. These instructions, which when executed by the processor, may cause the processor to implement the techniques described herein. For example, the instructions may include operating system instructions 241 to implement an operating system on device 200. The instructions may also include available memory and capabilities instructions 242 to provide an interface for the device to iearn of the memory available within the device as we!i as the error rates associated with that memory. The instructions may also include error rate assignment instructions 243, through which the device may assign error rates to the available memory. The instructions may also include voitage setting

instructions 244, through which the device may control the voltage supplied by the voltage source to each memory.

[0029] in operation, the device may begin a startup procedure. As part of the startup procedure the device may provide the operating system, provided by the operating system instructions 241, with an interface to determine the physically addressable memory ranges within the device, as well as the granularity of those ranges. For example, the interface may be based on the Advanced Control and Power Interface (ACPI)„DSM definition. Using the available memory and capabilities instructions 242 provided by the ACPI„DSM definition, the operating system may receive an indication of the physically addressable memory ranges in the system, the granularity of those ranges, and the error rates that can be assigned to those systems. The data used to provide this information may have been previously stored in the non-volatile memory. Storing the information in the non-voiatHe memory will be described below, but for now it may be assumed that the available memory and capacities have aiready been stored in the non-volatile memory.

[0030] The operating system may then determine which error rate is to be assigned to each physically addressable memory range. For example, using the error rate assignment instructions 243, the operating system may determine that 0x3000 bytes of data may need to be stored in memory that is ensured to not introduce errors, while it would be acceptable for the remainder of the memory to introduce errors. As such, the operating system may determine that memory range 0x10G0~0x3FFF (which is 0x3000 bytes iong), provided by memory 220-b, should be connected to the nominal voltage 225, The remainder of the memory 220~a,c which can tolerate errors should be connected to the margined voltage 226.

[0031] Using the voltage setting instructions 244 the device may configure the voltage source 230 to set the margined voltage level and to provide either the margined or nominal voltage level to each memory. In the present example, the device may connect memory 220-b to the nominal voltage 225 to ensure no memory introduced errors. The device may also determine an acceptable memory introduced error rate and the margined voltage that is associated with that rate. The device may then set the voltage source to produce the determined margined voltage 226 and connect that margined voltage to memory 220-a,c.

[0032] FIG. 3 depicts another example computing device that may utilize the memory error rate assignment techniques described herein. Many of the elements described with respect to FIG. 2 are also included in FIG. 3. For example, the processor 210, memory 220-ia-c), margined and nominal voltage 225, 226, voltage source 230, non-transitory medium 240 (including instructions 241-244), and non-voiatiie memory 250 are substantially the same as processor 310, memory 320-(a~c), margined and nominal voltage 325, 326, voltage source 330, non-transitory medium 340 (including instructions 341-344), and non- voiatiie memory 350. For the sake of clarity, the descriptions of those elements are not repeated here. [0033] The non-transitory medium 340 may also include error rate characterization instructions 345. The error rate characterization instructions may be used by the device to determine the error rates for the memory, given various operating voltages, in one exampie implementation, the error rates of memory may be determined by setting the voltage supplied to the memory, and then repeatedly running a diagnostic test on that memory. The diagnostic test may exercise the memory by repeatedly reading to and writing from the memory. The testing may determine the rate of errors introduced by the memory for each given level of supplied voltage.

[00343 The error rate summary / store instructions 346 may be used to summarize and store the error rate characterization information. For exampie, the characterization information may be stored in the non-volatile memory 350 as an error rate table 350-a. The table may indicate for each possible memory range, and for each possible voltage level, an expected error rate. The expected error rate may have been determined from the error rate

characterizations instructions described above.

[00353 in some implementations, the error rate characterization instructions may only run periodically. In the short term, it is not expected that given a specific margined voltage, that the error rate would vary widely. Thus tt may not be necessary to characterize the memory upon every system restart. For example, the characterization may be done upon every 10* 100^,h, or 1000^th restart of the system. In other implementations, the characterization may be done oh a system restart if the time since the last characterization exceeds a threshold (e.g. last characterization was greater than ? days ago). By periodically performing the characterization, the average delay in system startup may be reduced, as the characterization is not performed on every startup.

[0036] in operation, the system 300 may start and it may be determined that the memory should be characterized. The system may then set the voltage level for each memory device to the nominal voltage and run through a series of diagnostic tests to ensure that no errors are introduced, in some

implementations, this step may be omitted, as it may be assumed that memory operating at the nominal voltage will not introduce errors. In any case, as shown in table 350-a, for the nominal voltage, all address ranges indicate 0 bits / line as the expected error rate.

[0037] The process of characterization may continue by setting each range of physically addressable memory to a margined voitage level (depicted here as V1 , V2, and V3) and running the diagnostic tests. For each range of memory addresses, for each tested margined voltage, an error rate may be determined. The error rate may be stored in table 35G-a_r It should be noted that table 350-a depicts the error rate as bits / line (cache line). However, the techniques described herein are not dependent on any specific unit of measurement. The error rate couid be specified as a probability of error, an expected error rate per 1000 bytes, or any other suitable form. What should be understood is that the error rate may be used by the operating system to when determining which error rates to assign to which ranges of memory, and where to store those ranges of memory.

[0038] FiG. 4 depicts an example of a flow diagram for assigning error rates to memory according to techniques described herein. The biocks described in FIG. 4 may be executed by a system such as the system depicted in FIG.. 3. In other words, the medium 240 may contain instructions that when executed by the processor 210 cause the processor to implement the blocks described below.

[0039] In block 410, an indication of physicaliy addressable memory in a system may be receivsid. The indication may include avaiiable error rates and a range of granularity for assigning error rates to the ranges of physicaliy addressable memory. In other words, an indication of the available memory in a system may be received by the operating system of a computer. The indication may include information to notify the operating system about various ranges of memory that may be configured with different error rates. As explained above, this may be referred to as the granularity of assignment of error rates.

[0040] In block 420, error rates may be assigned to each range of the physicaliy addressable memory. As explained above, the granularity of the addressable memory ranges may determine the address ranges that may be assigned different error rates. As explained above, error rates may be determined based on the voltage supplied to the memory elements comprising the address range. In some implementations, each memory range may be supplied with either the nominal voltage or a margined voltage. Memory supplied with the margined voltage may introduce errors, as was described above.

[0041] FIG. 5 depicts another example of a flow diagram for assigning error rates to memory according to techniques described herein. The blocks described in FIG. 5 may be executed by a system such as the system depicted in FIG. 3. In other words, the medium 240 may contain instructions that when executed by the processor 210 cause the processor to implement the blocks described below.

[0042] in block 502, the voltage supplied to memory elements making up the physically addressable memory may be set. In block 504, diagnostic tests may be run on the memory to determine the error rate of the memory elements, in block 506, the error rate for the memory elements may be stored. The dotted outline depicted as reference numeral 508 indicates that this process may be an iterative process. For example, the voltage may be set to the nominal voltage and blocks 502, 504, and 506 executed to determine the error rate of the memory. The voltage may then be reduced, and the blocks re-executed to determine the error rate at the reduced voltage. This process may continue for a plurality of different voltage levels.

[0043] The end result may be a table, such as table 350-a which indicates the expected error rate for each range of memory addresses, given a specified voltage. As mentioned above, the table may be referred to as th£ memory characterization table. Furthermore, in some implementations, it may not be necessary to characterize the memory upon each restart of the system, as the expected error rates for the memory elements are not expected to vary in the short term. As such, the process of characterizing the memory may occur periodically. For example, the characterization process may occur on every 10^¾ system restart. As another example, the characterization process may occur one the first system restart of any given month. The particular period for characterizing the memory Is unimportant. What should be understood is that the memory need not be characterized on every system restart.

[0044] in block 510 just as above in block 410, an indication of physically addressable memory in a system may be received. The indication may include available error rates and a range of granularity for assigning error rates to the ranges of physically addressable memory.

[0045] in block 515, error rates may be assigned to each range of the physically addressable memory, wherein assigning error rates to each range of the physically addressable memory further comprises setting a voltage level supplied to memory elements providing storage for the range of physically addressable memory, wherein the voitage level determines the error rate for the range of physically addressable memory. As explained above, ranges of memory may be provided by one or more memory elements (e g. individual memory chips}. The particular arrangement of the chips may determine the memory address ranges that can be assigned different error rates, in block 515, each range of physically addressable memory may be assigned an error rate by setting the voltage supplied to the memory elements that are used to supply storage for that range of physically addressable memory.

[0046] In block 520, a request for the error rate of a particular range of physically addressable memory may be received. As mentioned above, the system may provide an interface that allows the system to determine the current operating characteristics of a particular range of memory, in block 525, the error rate for the particular range of physically addressable memory may be provided. In other words, block 525 may be the response to the request from block 520.

[0047] in block 530, it may be determined that the error rate of a candidate range of memory is inappropriate for data to be stored in the candidate range of memory. For example, a range of memory may be assigned to an error rate specified by the nominal voltage (e.g. no errors introduced by insufficient voltage). This error rate may be inappropriate for trivial data, as there is no need to waste the power used by providing nominal voltage to the memory range. As another example, the error rate may be greater than that provided by the nominal voitage, and as such the error rate may be

inappropriate for data that cannot toierate errors,

[0048] fn block 535, an acceptable error rate for a data element may be determined, in other words, it may be determined if a data element can or cannot toierate errors. In block 540, an appropriate error rate may be assigned to the candidate range of memory prior to storing the data element In other words, if it is determined that the error rate for a candidate address range that is to be used to store a data element is in appropriate, the system may be able to reassign the error rate for the range of memory. This reassignment may occur by altering the voitage supplied to the memory. Furthermore, the reassignment may occur dynamically while the system is stiii running. In block 545, the data eierriertt may be stored in the range of physically addressable memory having an appropriately assigned error rate.

Claims

I Claim:

1. A method comprising:

receiving an indication of physically addressable memory in a system, the indication including available error rates and a range granularity for assigning error rates to ranges of the physically addressable memory; and

assigning error rates to each range of the physically addressable memory.

2. The method of claim 1 further comprising;

determining the error rate of a candidate range of memory is

inappropriate for data to be stored in the candidate range of memory; and

assigning an appropriate error rate to the candidate range of memory prior to storing the data.

3. The method of claim 1 wherein assigning error rates to each range of the physically addressable memory further comprises:

setting a voltage level supplied to memory elements providing storage for the range of physically addressable memory, wherein the voitage level determines the error rate for the range of physically addressable memory.

4. The method of claim 1 further comprising:

for each of a plurality of voltages:

setting the voltage supplied to memory elements making up the physically addressable memory;

determining the error rate of the memory elements; and storing the error rate for the memory elements.

5. The method of claim 4 wherein the setting, determining, and storing is done periodically.

6. The method of claim 1 further comprising: determining an acceptable error rate for a data element; and

storing the data element in the range of physically addressable memory having an appropriately assigned error rate.

7. The method of claim 1 further comprising:

receiving a request for the error rate of a particular range of physically addressable memory; and

providing the error rate for the particular range of physically addressable memory.

8. The method of claim 1 wherein error rates are specified as bits per cache line.

9. The method of claim 1 wherein assigning error rates is done through an advanced control and power interface (ACPI).

10. A non-transitory processor readable medium containing a set of instructions thereon which when executed by the processor cause the processor to:

provide an operating system with an indication of available memory and capabilities of the available memory, the capabilities including error rates and granularity for assigning error rates;

receive an assignment of error rates for the available memory; and set a voltage supplied to the available memory based oh the assigned error rate, wherein the voltage supplied to the available memory determines the error rate.

11. The medium of claim 10 farther comprising instructions to:

characterize the error rates of the available memory; and

provide a summary of the error rates to the operating system.

12. The medium of claim 11 further comprising: provide an advanced contra! and power interf ace (ACPI) methods and data tabies to receive the capabilities of the available memory and set the error rates of the available memory.

13. The medium of claim 11 further comprising instructions to:

store the characterization of the available memory across system restarts; and

periodically re-characterize the available memory, wherein the recharacterization does not occur on every system restart.

14. A system comprising:

a processor;

a memory coupled to a voltage source and the processor; and the voltage source to selectively provide a nominal voltage or a margined voitage to the memory, wherein an error rate of the memory is based on the supplied voltage, wherein the selection of supplied voltage is based on the desired error rate to assign to the memory.

15. The system of ciaim 14 further comprising:

a nonvolatile memory to store a table of voltage levels and error rates for the memory for a plurality of margined voltages.