FileSharer Development Notes 5 — Designing a Cache to Avoid Rebuilding Folder ZIPs
FileSharer Development Notes · 05 / 10
When sharing a folder, compression may be the reason the first response is slow, rather than the network. If the entire folder is turned into a ZIP after the recipient clicks Download, no file bytes reach them while that work is underway. Repeating the compression after a dropped connection makes the wait even longer.
FileSharer's ZipCache separates this preparation cost from the transfer. When a folder is shared, the application builds its ZIP in the background. At download time, it transfers the prepared archive like an ordinary file. This also allows the same Range handling to be applied to ZIP downloads.
Separate locks for the dictionary and compression
The cache has an Entry for each share ID, and each entry has a BuildLock. Access to the dictionary when finding or creating an entry uses mapLock; the actual ZIP creation holds that entry's BuildLock.
This separation can be read as a way to keep compression of one large folder from holding up dictionary lookups for other shares. Threads requesting the same share concurrently wait for the same result. This does not mean that a work queue limiting CPU or disk usage has been implemented. Compression for different shares can run concurrently.
Keep unfinished ZIPs under a different name
GetOrBuild writes the ZIP to a .part file, then moves it to the final filename. The compression step is as follows:
ZipFile.CreateFromDirectory(
item.FilePath, partPath, CompressionLevel.Fastest, false);
File.Move(partPath, finalPath);
The design aims to prevent reuse of an output that is still being written simply because its final filename exists. The compression level is set to Fastest, but this investigation did not compare its execution time or archive size with other levels. No claim about a speedup factor is therefore made.
Deciding when to invalidate the cache is harder
A folder fingerprint determines whether a cached archive can be reused. The original TryStampFolder combines three values:
File count : Total bytes : Most recent file modification time
This is simple to calculate and compare without hashing all the folder contents again. It is not, however, a complete identifier for those contents. It does not directly include the list of filenames or the contents of each file.
I checked this distinction using the original function. I changed a sample file's contents from AAAA to BBBB and set its modification time back to the same value. The file count and length stayed the same.
| Change | Fingerprint comparison |
|---|---|
| Contents changed; size and modification time preserved | Same |
| Length changed from 4 bytes to 5 bytes | Different |
The first result means that the cache check cannot distinguish this kind of change. The original GetOrBuild reuses an archive when the fingerprint matches and the cached file exists, so the source-code flow indicates that it could return a stale ZIP. This test directly checked the fingerprint function's results; it was not an integration test that downloaded and compared complete ZIP archives.
Match the refresh policy to the material being shared
If users bundle material once and leave it unchanged, creating a snapshot at the time of sharing provides a clear model. For a working folder that is continually edited, stronger change criteria could be considered: a manifest containing paths, file sizes, and individual modification times; content hashes; or explicit version increments.
None of these approaches is free. Content hashing requires reading the files, and files can change even while a manifest is being assembled. The product first needs to define whether a share means “the bundle as it was when shared” or “the latest folder contents.” That decision determines what the cache must guarantee.
Cache cleanup is part of lifecycle management
The original code includes functions to delete the cache for an individual share and to clean up all cached archives on exit. Before compression, it attempts to check for free disk space equal to the source size plus a 256 MiB margin. There is also a path that allows the operation to proceed if the check itself fails. This threshold alone cannot rule out running out of disk space during compression.
The benefit of a ZIP cache comes from reusing a prepared result. Defining how long that result still represents the same material makes it possible to explain both the performance improvement and the correctness of its contents.
FileSharer.cs:1239–1329 ZipCache.GetOrBuild · FileSharer.cs:1336–1349 PrebuildAsync · FileSharer.cs:1394–1427 TryStampFolder / HasFreeSpace · FileSharer.cs:4951–4970 ZIP transfer
Proposed improvements are distinguished from implemented behavior.
댓글
댓글 쓰기