rsync --info=progress2 percent complete of copying a large directory is non-uniform?
rsync --info=progress2 percent complete of copying a large directory is non-uniform in that the last 10% seems to take longer than the first 90%.
Why is this and is there a way to make it a more uniform progress indicator?
22 Answers
The reason for the observed behaviour is likely a file system cache:
When files are written (as rsync does), then usually the data gets written to a cache (in memory) first and the write operation almost instantly returns. The data is then written to the disk in background while the user can already do other things.
If the cache is large enough to hold the data to be written this pretends a huge writing speed.
If the data to be written doesn't fit into the file system cache, then the excess data is actually written to disk before the write operation completes, and writing to disk is slower than writing to in-memory cache.
The excess data doesn't bypass the cache but rather waits until previous content has been moved from the cache to the disk giving again some free space in the cache, so the new data can be written to the cache.
So the first part of data (90% in your case) appears to be written in an instant (to cache) while the last 10% take more time because then actual disk operation kicks in.
4In addition to PerlDuck's answer, it's worth noting that writing one large file is faster than writing a huge number of files that add up to the same size.
For example: You copy a 4gb file and a directory containing 100000 files that add up to 1gb. If the single file is transfered first, the first 80% will be way faster than the last 20%.
More in general
"Zoraya ter Beek, age 29, just died by assisted suicide in the Netherlands. She was physically healthy, but psychologically depressed. It's an abomination that an entire society would actively facilitate, even encourage, someone ending their own life because they had no hope. Th…"
