Microbenchmarking Flash Drives

1. Page sizes observed in testing bunch of random devices: 64 KiB, 100 KiB, 128 KiB, 256 KiB, 400 KiB, 1 MiB. Interesting. I would have assumed the pages to be powers of two, but probably some of the pages contain error correction data, so the true flash capacity isn't visible to the file system.

2. Also the page processing latency was very different on different drives. Usually when the pages get larger, the latency also gets larger, unless the drive is faster. And this is exactly what I've been observing without this test.

3. On some "fast drives" are horrible when using NTFS, because the blocks are so large that the metadata management overhead makes the drive ultra slow, even if it's "fast". Every random 4k update ends up updating 1 MiB block.

4. Some drives were very consistent with the latency and others showed different latency levels. Like small latency increase over every 50 KiB and large latency increase every MiB.

5. And with many drives the number of 512 sectors per page waried, some showed interesting values like 251 sectors per page. But I guess it's to the manufacturer to decide how large part of that page is

6. Reserved for error correction data. Some drives were extremely consistent, others had more jitter. Micro benchmarking seems to be working well revealing the page sizes and alignment. All the tests were repeated 20 times, and for each offset median timing was selected to exclude "random variation" due to other things potentially messing up the timing / write / io process.

7. Difference for writes crossing the page boundary were huge. As example many of the older Flash USB drives write 1024 bytes in around 4-2 milliseconds. But when page boundary is crossed, the write might take 1.15 seconds. The slowest of the sticks with large 400KiB pages took whopping 2.5 seconds to write 512 bytes which crossed the page boundary.

8. Testing method? Writing 512 bytes aligned with 512 bytes and then writing 512 bytes misaligned by 256 bytes. That formed one measurement block, and this is block which was latency monitored for differences. So no single measurement was perfectly aligned by 512 bytes sector. And then syncing the data to disk, raw block device access was being used. For statistical reasons I also stored information about writes aligned with 512 bytes versus misaligned writes. It was interesting to observe that it was mostly meaningless difference for most drives. With one drive the measurement was for aligned writes 49.9 seconds (total test time) versus misaligned writes 57.6 seconds. Sure, it shows that the writes are misaligned, but the difference isn't nearly as huge, compared to the situation where the misaligned write did hit the page boundary and increased by several orders of magnitude.

9. Why? Well, I just wanted to see, if the page sizes and boundaries can be easily found by microbenchmark timings. When some people claim that those things aren't visible to the user space.

10. Kibibyte (KiB), Mebibyte (MiB) - Not to be confused with kilobyte kB and megabyte MB.

2021-01-17