What methods does QES use to perform data deduplication?

Deduplication is a data reduction technique that works by eliminating duplicate copies of repeated data. There are different methods of performing deduplication, which can be split into 3 categories: point of application, granularity, and time of application.

Point of Application

• Source-based deduplication: The client device (workstation, mobile device) deduplicates the data. The client device then transfers the deduplicated data to the target storage server.
• Target-based deduplication: The client device transfers the raw source data to the target storage server. The target storage server then deduplicates the data.

QES uses target-based deduplication, which means that QES performs deduplication on data after it has been transferred from a client to the NAS. This method requires less processor and memory resources on the client.

Granularity

• File-level deduplication: The deduplication process looks for files with identical content. This is a simple approach which is effective in scenarios where many files are shared between clients, for example in email and messaging systems.
• Block-level deduplication: The deduplication process looks for data blocks with identical content. This is a more thorough approach, as it can detect and eliminate smaller repeated chunks of data within one file or in many different files.

QES uses block-level deduplication, as in most cases it is more effective at reducing data size.

Time of Application

• Inline deduplication: Deduplication is performed on data after it has been transferred to the storage server but before it has been written to disk.
• Offline deduplication: Deduplication is performed on data after it has been written to disk, according to a schedule or other trigger.

QES uses inline deduplication, which means that it deduplicates data after receiving it from a client but before writing it to storage. Inline deduplication reduces the amount of data written to drives, which reduces drive wear. This is especially important for NAS devices containing SSDs, as SSD lifespan is highly dependent on the total number of write operations. Inline deduplication also requires less storage space, as the storage server does not need to store the larger raw source data before deduplicating it.

The disadvantages of inline deduplication are that it requires more processor and memory resources while data is being written to the storage server, and it is usually slightly less effective than offline deduplication at reducing data size.