RFID, BtrFS, DHCP, obfsproxy, GAE/python27, sql pivot, ext4, extents, SSD, NCQ, NTFS

Post date: Feb 12, 2012 4:22:25 PM

• Tested and installed RFID end client identification system.
• Tested and compared BtrFS none / lzo / zlib compression algorithms and disk performance on virtual servers.
• e4defrag is finally becoming ready, just when BtrFS is becoming slowly ready for production use? BtrFS provides autodefrag mount option.
• Closed case with Clonzeilla and TFTPDS32 DHCP protocol issue. It's now clear that Clonezilla sends invalid request packets and are being ignored by DHCP server application. To find out all the details, tried both Wireshark and (MS) Network Monitor.
• There is also strange issue with Clonezilla, Linux, pxelinux network driver. When using one Realtek Gigabit Ethernet adapter with pxelinux, it seems that something goes so badly wrong that network card is "permanently disabled". Even booting computer to Windows, where it did work before this boot attempt doesn't fix the situation. Some engineers thought that it's better to send those computers for warranty repair, but before they were able to send those. I told them that have they really rebooted those computers? They said yes. Then I unplugged the power cord, pressed power button a few times, and plugged cable back in. - Yup, now you got working network adapter again. No, it's not real cold reboot, if you keep motherboard powered all the time. Unfortunately even latest pxelinux.0 file won't fix this issue. Yes, I have reported it.
• Compiled and installed obfsproxy to virtual server. I think information flow should be free and available to everyone. There are now over 20 users using it all the time.
• Created new (git) branch for 9oxnet project. Plan is to make required changes, so I can use python27 runtime with Google App Engine. After very quick testing I found out that upgrading from 2.5 runtime to 2.7 requires more changes than I initially thought. I also need to confirm that application is truly threadsafe, otherwise interesting issues could arise.
• I had to produce one report, there were multiple options how to collect data for it. But I decided that it's not bad idea to be able to fully master pivot queries, so I used a few hours to learn it throught and managed to pull all required information with one query including all required multitable joins etc.
• Listened just a few too many Social Engineering podcasts, those weren't that good. Won't listen more.
• Checked what are the actual fragmenting effects of sparsely allocated files on ext4 filesystem. It seems that even if large file is created as sparse file, and filled up in fully random order, ext4 manages to create in average 2 megabyte extents. Files are of course fragment up between extents, but that's not excessive. Without extent feature, results could be very much worse. Like they are when doing similar test is executed using NTFS file system. It leads to incredibly many fragments, just a few contiguous blocks here and there. If you want even worse results, fill up compressed NTFS file in random order. It seems that 7-zip doesn't pre-allocate files even when extracting data. It would be smart to pre-allocate space for file because final file size is of naturally known.
• During doing that I found out that there is currently failure in Ubuntu power management. It sets ext4 commit time always to 0 if AC power is being used. I had to modify /usr/lib/pm-utils/power.d/journal-commit file to get 60 second commit time. That's what I thought would have been using all the time. When I checked mounts I found out that there was commit=60,commit=0 entries on ext4 mount rows. Unfortunately I haven't been able to enable NCQ (and AHCI) with my Intel P35 Express motherboard and Linux kernel. I have tried all basic tricks.
• Checked out (again) latest info about SSD drives and issues realated to those: file systems designed for SSD drives (log filesystem), flash file system, wear leveling, write amplification, data remanence, trim. Low level wear leveling can caused serious drive fragmentation which isn't visible at all to operating system. Performance of some Intel drives suffer very badly. Running OS file system level defrag only makes this situation often worse. (ubifs, logfs, jffs2, ahci, ncq, trim, lba