CN111367926A - Data processing method and device for distributed system - Google Patents

Abstract

The invention discloses a data processing method and device of a distributed system. Wherein, the method includes: receiving a data storage instruction; acquiring data to be stored, and segmenting the data to be stored according to a specified size; separately calculating the verification data of the data to be stored in each segment, and The verification data of the data to be stored in each segment is stored as a verification record. The invention solves the technical problem that the distributed storage system occupies too much system performance when verifying data in the prior art.

Description

Data processing method and device for distributed system

technical field

The present invention relates to the field of data storage, in particular, to a data processing method and device of a distributed system.

Background technique

In a distributed storage system, data consistency is very important. The silent error of the disk itself, hardware anomaly or software vulnerability will lead to user data loss or inconsistency, which brings huge losses to users. Therefore, in the distributed storage system, data verification is performed through MD5 or CRC, and the size of each data block is recorded in the storage system. When reading data, the copy is determined by calculating the checksum of the data and the stored checksum. data consistency

The data verification device in the prior art needs to perform data verification when reading data from a distributed storage system, and this process has a great impact on system performance, which is a defect of the current distributed storage system.

Aiming at the problem that the distributed storage system occupies too much system performance when verifying data in the prior art, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a data processing method and device for a distributed system, so as to at least solve the technical problem in the prior art that the distributed storage system occupies too much system performance when verifying data.

According to an aspect of the embodiments of the present invention, a data processing method for a distributed system is provided, including: receiving a data storage instruction; acquiring data to be stored, and segmenting the data to be stored according to a specified size; The verification data of the data to be stored in each segment is stored, and the verification data of the data to be stored in each segment is stored as a verification record.

Further, the above method also includes: receiving a data reading instruction; reading data according to the data reading instruction, and calculating the current verification data of the read data; The fetched data is compared with the original verification data of the read data; if the comparison is successful, it is determined that the read data is the data to be read indicated by the data read instruction.

Further, the above method also includes: receiving a data zero write instruction or a data clearing instruction; clearing the zero data to be written or the verification data corresponding to the data to be cleared, and updating the zero data to be written or the segment where the data to be cleared is located verification data.

Further, the preset verification data cache space stores verification data, before the current verification data is compared with the original verification data of the read data when the read data is written. , the method further includes: extracting the original verification data from the verification data cache space.

Further, extracting the original verification data from the verification data cache space includes: if the original verification data exists in the verification data cache space, extracting the original verification data from the verification data cache space. verification data; if the verification data cache space does not store the original verification data, extract the original verification data from the disk, and write the original verification data extracted from the disk into the verification data Check the data cache space.

Further, comparing the current verification data with the original verification data of the read data when the read data was written, comprising: calling an asynchronous readback module, wherein the asynchronous readback After the module submits the read request, the current thread is completed to support asynchronous reading; the current verification data is compared with the original verification data of the read data when the read data is written by the asynchronous read back module. Compare.

According to an aspect of the embodiments of the present invention, there is provided a data processing apparatus of a distributed system, including: a receiving module for receiving a data storage instruction; an obtaining module for obtaining data to be stored, and storing the data to be stored Segmentation is performed according to the specified size; the first calculation module is used to calculate the verification data of the data to be stored in each segment respectively, and use the verification data of the data to be stored in each segment as a verification data The test records are stored.

Further, the above-mentioned device further includes: a receiving module for receiving a data reading instruction; a second calculating module for reading data according to the data reading instruction, and calculating the current verification data of the read data; A pairing module is used to compare the current verification data with the original verification data of the read data when writing the read data; a determination module is used to determine when the comparison is successful The read data is the data to be read indicated by the data read instruction.

According to an aspect of the embodiments of the present invention, a storage medium is provided, the storage medium includes a stored program, wherein when the program runs, a device where the storage medium is located is controlled to execute the above-mentioned data processing method of a distributed system.

According to an aspect of the embodiments of the present invention, a processor is provided, and the processor is configured to run a program, wherein the program executes the above-mentioned data processing method of a distributed system when the program is run.

In the embodiment of the present invention, a data storage instruction is received, data to be stored is acquired, and the data to be stored is segmented according to a specified size; the verification data of the data to be stored in each segment is calculated separately, and the The verification data of the data to be stored in each segment is stored as a verification record. For the distributed storage system based on ceph, the minimum IO size of each operation is 4KiB, and each 4KiB of data is encoded by the crc32 algorithm to obtain a 4-byte data check value. If the data check value of an object is used as a check data record , then every 4K IO update requires the operation of the entire object, but in the above-mentioned segmentation method, only the verification data records of the segment where the operation object data is located are loaded, updated or cleared each time, and then the corresponding data records are persisted or deleted. rocksDB records, thereby reducing the impact of enabling data verification on performance, and solving the technical problem that the distributed storage system takes up too much system performance when verifying data in the prior art.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached image:

1 is a flowchart of a data processing method for a distributed system according to an embodiment of the present invention;

2 is a schematic diagram of segmented storage of verification data according to an embodiment of the present invention;

3 is a schematic diagram of storing verification data based on a crc cache according to an embodiment of the present invention;

FIG. 4 is a flow chart of reading data according to an embodiment of the present invention; and

FIG. 5 is a schematic diagram of a data processing apparatus of a distributed system according to an embodiment of the present invention.

Below, in order to facilitate the understanding of this embodiment, the technical terms that appear in this embodiment are explained:

Distributed storage system: In simple terms, it is to store data in multiple storage servers, and form a virtual storage device with these scattered storage resources to provide data storage services.

PG: Placement Group, a placement group (PG) gathers a series of objects into a group and maps this group to a series of OSDs.

OSD: OSD (Object Storage Device) refers to a process responsible for data placement.

Silent error: Silent Data Corruption, the core mission of the hard disk is to store data correctly, read data correctly, and throw an exception alarm in a timely manner when an error occurs. Disk abnormalities may include hardware errors, firmware bugs or software bugs, power supply problems, media damage, etc. These conventional problems can be captured normally and exceptions are thrown, and the most terrifying thing is that data processing is normal. The data is not found to be erroneous or corrupt until it is used. This is the silent error.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only Embodiments are part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

Example 1

According to an embodiment of the present invention, an embodiment of a data processing method for a distributed system is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions and, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.

FIG. 1 is a flowchart of a data processing method for a distributed system according to an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:

Step S102, receiving a data storage instruction.

Specifically, the above data storage instruction is used to store data in the distributed storage system.

Step S104: Acquire data to be stored, and segment the data to be stored according to a specified size.

In Ceph's distributed storage system, the size of each object is 4MiB. In the above solution, each object is segmented to obtain multiple segments corresponding to one object. In an optional embodiment, for a 4G object, it can be divided into multiple segments, for example, if the segment is performed according to 256KiB, an object has 16 segments.

Step S106: Calculate the verification data of the data to be stored in each segment respectively, and store the verification data of the data to be stored in each segment as a verification record.

Specifically, the above-mentioned check data is the check value of the data, and the MD5 (Message Digest Algorithm 5) or CRC (Cyclic Redundancy Check, Cyclic Redundancy Check) can be used to determine the value of the data to be stored in each segment. Check data. In the above-described embodiment, if the segmentation is performed according to 256KiB, one object has 16 inspection data records.

FIG. 2 is a schematic diagram of segmented storage of check data according to an embodiment of the present invention. With reference to FIG. 2, for a 4MiB object, it is divided into 16 segments, each segment data_chunk_size=256KiB . Each 4KiBd data is encoded by the crc32 algorithm to obtain a 4byte data check value, where each 256KiB data check value is used as a <key, value> cre_chunk record (4KiB IO has a 4-byte check value, 256KiB data checksum The value is 64 4 bytes, that is, 256 bytes. The check value of the 256 bytes is stored as a piece in the database), where the key is used to indicate the identifier of the segment, and the velue is used to indicate the data check value. The records are stored in the backend RocksDB to obtain 16 verification data records, each verification data record crc_chunk_size=256B, and the sum of the data verification records is 4KiB crcvalue.

It can be seen from the above that the above-mentioned embodiment of the present application receives a data storage instruction, obtains the data to be stored, and segments the data to be stored according to the specified size; respectively calculates the verification data of the data to be stored in each segment, and The verification data of the data to be stored in each segment is stored as a verification record. For the distributed storage system based on ceph, the minimum IO size of each operation is 4KiB, and each 4KiB of data is encoded by the crc32 algorithm to obtain a 4-byte data check value. If the data check value of an object is used as a check data record , then every 4K IO update requires the operation of the entire object, but in the above-mentioned segmentation method, only the verification data records of the segment where the operation object data is located are loaded, updated or cleared each time, and then the corresponding data records are persisted or deleted. rocksDB records, thereby reducing the impact of enabling data verification on performance, and solving the technical problem that the distributed storage system takes up too much system performance when verifying data in the prior art.

After data verification is enabled, it is necessary to ensure that the data is consistent with the existing data verification value after the four operations of write, read, zero, and truncate are executed. For the operation of writing data (write), the written data must calculate its verification data, update the verification data record of the segment to which the data belongs, and persist (store it in RocksDB) the verification data, so as to keep the data. Consistency with check value. The other three operations are described separately.

As an optional embodiment, the above method further includes: receiving a data reading instruction; reading data according to the data reading instruction, and calculating current verification data of the read data; Compare with the original verification data of the read data when writing the read data;

If the comparison is successful, it is determined that the read data is the data to be read indicated by the data read instruction.

In the above solution, data verification needs to be performed for a read operation (read). The method of performing data verification is to calculate the current verification data of the data read according to the data reading instruction, extract the original verification data when the data is stored, and compare the current verification data with the original verification data, If they are the same, it means that the read data has not been tampered with; if they are different, it means that the read data may have been tampered with, thus ensuring the consistency of data writing and reading.

As an optional embodiment, the above method further includes: receiving a data zero writing instruction or a data clearing instruction; clearing the zero data to be written or the check data corresponding to the data to be cleared, and updating the zero data to be written or the Verification data of the segment where the data to be cleared is located.

In the above solution, when performing a data writing zero operation (zero) or a data clearing operation (truncate), the corresponding data check value needs to be cleared. To clear the data check value of the data in the zero write operation or the clear operation, the check record of the segment where it is located also needs to be updated.

As an optional embodiment, the preset verification data cache space stores verification data, and when the current verification data is compared with the original verification data of the read data when the read data is written Before comparing the verification data, the above method further includes: extracting the original verification data from the verification data cache space.

The above verification data cache space (crc cache) adopts a caching mechanism to store the verification data in RocksDB in the verification data cache space, so that the original verification data can be directly read from the verification data cache space without having to Backend RocksDB ingest.

In an optional embodiment, the verification data cache space includes multiple levels, and the search method is to first search at the highest level, and if it does not exist, search at the next level until the verification data cache is found. The last floor of the space. FIG. 3 is a schematic diagram of storing verification data based on crc cache according to an embodiment of the present invention. With reference to FIG. 3 , the designated storage space XStore includes multiple cache segment cache shards, and each cache shard has 8 layers of crc caches (level#1of crc cache, level#1of crc cache...level#8of crc cache), which covers all data in pg.a to pg.z. When searching, firstly search from level#1of crc cache. If the searched verification data does not exist in level#1of crc cache, search from level#2of crc cache until the searched verification data is found. Or find level#8of crc cache, where, in this example, a level is 128M memory, and 128M memory can cache the data check value of 128G data.

As an optional embodiment, extracting the original verification data from the verification data cache space includes: if the original verification data exists in the verification data buffer space, extracting the original verification data from the verification data cache space Extract the original verification data from the cache space; if the verification data cache space does not store the original verification data, extract the original verification data from the disk, and use the original verification data extracted from the disk Data is written into the check data cache space.

In the above scheme, after the crc cache is enabled, the crc cache is used to cache the check value. When reading data, the checksum of the data is calculated, and then compared with the check value of the data in the cache, the data check value Compared in cache, the response speed is faster than reading from RocksDB. If there is no hit in the crc cache, it will be read from RocksDB and then cached in the crc cache.

Fig. 4 is a flow chart of reading data according to an embodiment of the present invention. First, a condition check is performed: judging whether l2_cache is enabled, where l2_cache refers to the local storage engine of xstore (ceph, similar to a file system, and is directly in the same block) The second layer of cache in the components of device interaction), this layer of cache includes the cache of two important data structures, namely onode and crc. The onode cache records which objects have the onode cache, and the onode contains all the metadata information of the object, such as the physical offset of the data block used by the object, extended attributes, object size and other information, similar to the inode of the file system; which objects are recorded in the crc cache Has the crc cache, and which pieces of the object's data contain the data check value and the data check value itself. Turning on l2_cache means enabling onode and crc caches. After enabling, when object read and write operations are involved, onode and crc caches need to be updated. If the judgment result is ON, go to the next step, and if the judgment result is not ON, exit directly.

In the case that the judgment result is that l2_cache is enabled, the validity is checked. If it is meta_pg, temporary object, non-block object, etc., it does not need to be cached, just exit, otherwise go to the next step.

Determine whether only_cache_crc=true is satisfied (only_cache_crc=true means that only the crc cache is updated), if it is satisfied, the onode is not cached (the onode cache update will be performed in other parts of the process, here only the crc cache is updated to avoid repeated updates and improve operations efficiency), otherwise cache the onode and go to the next step.

Condition check again. If there is no record in the onode to enable data verification, exit, if the switch of crc cache is not enabled, exit, otherwise go to the next step. Specifically, l2_cache contains the crc cache, but the opening of l2_cache does not mean that the crc cache is enabled, but the opening of the crc cache means that the l2_cache must be enabled. Therefore, it is necessary to judge whether the crc cache is enabled. At the same time, if the object itself does not have the data verification function enabled (this property is recorded in the onode), there is no need to update the crc cache.

Check whether the object is cached in the crc cache. If there is no cache, write all the record objects in the crc_chunk_map (used to record all the data check values saved when the object is currently running) into the crc cache, otherwise only its internal records will be updated. , The check value of segmented data that is marked for deletion or newly read from RocksDB is written to cec cache. Specifically, the object is a basic operation unit of ceph, similar to a file in the file system, and the object has metadata information (onode) and data information (actually record the content written by the user). The crc cache capacity has an upper limit, and it is impossible to include the crc cache of all objects in the ceph cluster. Therefore, before updating the crc cache, it is necessary to first determine whether the object is in the crc cache, and if so, write the part changed in this operation into the cache ; If not, write all data check values (recorded in crc_chunk_map) saved at the current runtime of the object to the crc cache. Segmented data that is updated, marked for deletion, or read from rocksDB is part of the above changes. Being updated means that the old value is stored in the crc cache, and the old value needs to be overwritten with the new value of this operation. The segment has been deleted, so the data check value in the crc cache needs to be cleared; the segment data read from rocksDB means that the crc cache does not have this data check value, and this part of the data check value needs to be added to the crc cache.

As an optional embodiment, comparing the current verification data with the original verification data of the read data when the read data was written includes: calling an asynchronous readback module, wherein , the asynchronous readback module completes the current thread after submitting the read request to support asynchronous reading; through the asynchronous readback module, the current verification data is compared with the read data when the read data is written. The original verification data of the data is compared.

The read of the ceph-based distributed storage system is divided into synchronous read and asynchronous read, which is the semantics of the read request issued by the PG. If it is a synchronous read, the worker thread of the osd will be blocked until the data is returned from this disk or cache. This is equivalent to when the disk does IO, the thread cannot schedule other IO, and the performance is poor. Therefore, more asynchronous reads are used, and only a small number of services are synchronous reads, such as scrub data read requests.

However, if data verification is enabled, the data crc needs to be verified when reading. Due to the internal force waiting for the read to return, this thread has been holding the pg lock at this time, so other IOs inside the pg cannot be processed, and asynchronous reading is done internally. Degenerates to synchronous read.

In order to solve the above problems, data verification is added to the asynchronous read callback module. This module is used to perform data verification after the asynchronous read operation returns from the bottom layer. If there is data inconsistency, the object is recorded. The data verification is put into the asynchronous read callback module and executed by the asynchronous read callback module. The semantics of asynchronous read is that after submitting to the cache, the execution of the thread is completed, and the pg lock is released at the same time to support asynchronous reading, which greatly reduces performance. Impact.

Example 2

According to an embodiment of the present invention, an embodiment of a data processing apparatus of a distributed system is provided. FIG. 5 is a schematic diagram of a data processing apparatus of a distributed system according to an embodiment of the present invention. As shown in FIG. 5 , the apparatus It includes the following steps:

The receiving module 50 is used for receiving a data storage instruction.

The obtaining module 52 is configured to obtain the data to be stored, and segment the data to be stored according to a specified size.

The first calculation module 54 is configured to calculate the verification data of the data to be stored in each segment respectively, and store the verification data of the data to be stored in each segment as a verification record.

As an optional embodiment, the above-mentioned device further includes: a receiving module, configured to receive a data reading instruction; a second computing module, configured to read data according to the data reading instruction, and calculate the value of the read data. current verification data; a comparison module for comparing the current verification data with the original verification data of the read data when the read data is written; a determination module for comparing the read data In the case of success, it is determined that the read data is the data to be read indicated by the data read command.

As an optional embodiment, the above-mentioned apparatus further includes: a second receiving module, configured to receive a data zero writing instruction or a data clearing instruction; a clearing module, configured to clear the zero data to be written or the check data corresponding to the data to be cleared , and update the zero data to be written or the check data of the segment where the data to be cleared is located.

As an optional embodiment, the above-mentioned apparatus further includes: an extraction module, which is used to store the verification data in the preset verification data cache space, and compares the current verification data with the data written into the read data. Before the data is compared with the original verification data of the read data, the original verification data is extracted from the verification data cache space.

As an optional embodiment, the extraction module includes: a first extraction sub-module, configured to extract the original verification data from the verification data cache space if the original verification data exists in the verification data cache space verification data; a second extraction sub-module for extracting the original verification data from the disk if the verification data cache space does not store the original verification data, and extracting the original verification data from the disk The verification data is written into the verification data cache space.

As an optional embodiment, the comparison module includes: a calling submodule for calling an asynchronous readback module, wherein the asynchronous readback module completes the current thread after submitting a read request to support asynchronous reading; the comparison A sub-module configured to compare the current verification data with the original verification data of the read data when the read data was written by the asynchronous read-back module.

Example 3

According to an embodiment of the present invention, a storage medium is provided, and the storage medium includes a stored program, wherein when the program runs, a device where the storage medium is located is controlled to execute the data processing method of the distributed system according to Embodiment 1.

Example 4

According to an embodiment of the present invention, a processor is provided, and the processor is used to run a program, wherein the data processing method of the distributed system described in Embodiment 1 is executed when the program is run.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are only illustrative, for example, the division of the units may be a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims (10)

1. a data processing method of a distributed system, is characterized in that, comprises: receive data storage instructions; Acquiring data to be stored, and segmenting the data to be stored according to a specified size; The verification data of the data to be stored in each segment is calculated respectively, and the verification data of the data to be stored in each segment is stored as a verification record. 2. The method according to claim 1, wherein the method further comprises: Receive data read instructions; Read data according to the data read instruction, and calculate the current check data of the read data; comparing the current verification data with the original verification data of the read data when writing the read data; If the comparison is successful, it is determined that the read data is the data to be read indicated by the data read instruction. 3. The method according to claim 1, wherein the method further comprises: Receive data write zero command or data clear command; Clear the zero data to be written or the verification data corresponding to the data to be cleared, and update the zero data to be written or the verification data of the segment where the data to be cleared is located. 4. The method according to claim 2, wherein the preset check data cache space stores check data, and when the current check data and the read data are written, the read Before comparing the original verification data of the acquired data, the method further includes: extracting the original verification data from the verification data cache space. 5. The method according to claim 4, wherein extracting the original verification data from the verification data cache space comprises: If the original verification data exists in the verification data cache space, extract the original verification data from the verification data cache space; If the verification data cache space does not store the original verification data, extract the original verification data from the disk, and write the original verification data extracted from the disk into the verification data cache space . 6. The method according to claim 2, wherein comparing the current verification data with the original verification data of the read data when writing the read data, comprising: Calling the asynchronous readback module, wherein the asynchronous readback module completes the current thread after submitting the read request to support asynchronous reading; The current verification data is compared with the original verification data of the read data when the read data is written by the asynchronous readback module. 7. A data processing device of a distributed system, characterized in that, comprising: a receiving module for receiving data storage instructions; an acquisition module, used for acquiring data to be stored, and segmenting the data to be stored according to a specified size; The first calculation module is used for separately calculating the verification data of the data to be stored in each segment, and storing the verification data of the data to be stored in each segment as a verification record. 8. The apparatus according to claim 7, wherein the apparatus further comprises: The receiving module is used to receive data reading instructions; A second calculation module, configured to read data according to the data read instruction, and calculate the current verification data of the read data; A comparison module for comparing the current verification data with the original verification data of the read data when the read data is written; A determination module, configured to determine that the read data is the data to be read indicated by the data read instruction when the comparison is successful. 9 . A storage medium, characterized in that the storage medium comprises a stored program, wherein when the program is run, a device where the storage medium is located is controlled to execute the distributed program according to any one of claims 1 to 6 The data processing method of the system. 10 . A processor, wherein the processor is used to run a program, wherein when the program is run, the data processing method for a distributed system according to any one of claims 1 to 6 is executed.

Images