Blog‎ > ‎

Small file handing & file queues (Linux vs Windows)

posted Feb 6, 2015, 11:24 PM by Sami Lehtinen   [ updated Feb 11, 2015, 8:05 AM ]
Got bit annoyed due to poor disk I/O performance with one of my projects when using Windows. I just synced one of my git repository with about 2000 files and it seemed to take along time.
 
I found out that it's really true. Even server with SSD gives 3x worse disk I/O performance that el cheapo VPS. Even bigger the difference is when we compare results to Linux server with SSD. Test generates files, gets list of those files and then delete files by name, just to test out file queue (in directory) processing speeds, using Python 3.4 64bit and standard library.
 
10k files 4k / file chart, lower is better.


I often develop stuff on Linux systems, but production runs Windows. Production systems run on Windows servers, but when I'm bored I often develop stuff at home and I'm not going to use Windows at home. Funny thing is that really slow el cheapo USB flash stick gives you better actually performance on Linux than server SSD gives when using Windows. Yeah, that's life.You might be wondering the results, but the point is that the test app does NOT force fsync with disk as most of programs won't do.

This leads to situation where same program is working lot faster on Linux platform than on Windows. Of course this might be a problem if you assume that all data would be persisted on disk without using fsync. I've written about this topic earlier.
 
Raw data and extended tests with larger files, times are in seconds required to run the loop with code.
 
Windows Server (SSD) NTFS:
Create 3.185
Delete 0.955
 
Linux Server (SSD) ext4:
Create 0.276
Delete 0.051
 
Linux Server (HDD using Writeback) ext4:
Create 0.270
Delete 0.054
 
Linux Server (Slow (4MB/s write) USB Flash) NTFS:
Create 1.735
Delete 0.393
 
Linux VPS 1 ext4, el cheapo, minimal resources:
Create 1.061
Delete 0.324
 
Linux VPS 2 ext4, el cheapo, minimal resources:
Create 0.916
Delete 0.279
 
Same tests with 100x 16M / file:
 
Windows server (SSD) NTFS:
Create 10.359
Delete 0.124
 
Linux Server (SSD) ext4:
Create 2.416
Delete 0.379

Additional edit 2015-02-11:

Same test with NTFS using same slow USB2 stick with Windows 8.1. This is a good apples to apples comparison, because same program was run using Windows and Linux using exactly the same file system and storage medium:
Create 133.55
Delete 51.88
This really should blow your socks off. No, it's not 70% slower, it's over 70 times slower. then the Linux run. Deleting files is over 130x slower.