Clonezilla (Partclone, SMB), Peewee (ORM)

• Even more testing and analyzing. I've been given disinformation by colleagues. There's nothing wrong at all with Clonezilla or Partclone. I continued straddling and made several independent tests of different components, and now it seems that the issue is probably linked to SMB 3.0 protocol usage with CIFS or the network. But the time when the issue occurs is such that it's more likely to be protocol error than network error. Issue always comes up, when complete file is being finally closed. I'll try to repeat those tests using dd / rsync alone, just to see what happens, when writing and closing huge image files from Ubuntu. Yet, same problem does not occur if same files and tests are done using Windows 10 computer to same share. Yet it also seems that latest SMB is so smart, that if you just copy files, it does the actual copy remotely, without looping the data via client, so even if that test passes, it's testing protocol failure. Btw. I didn't know that, but it's great. I always loathe which trivial and obvious optimizations like that are omitted. Let's loop the images via local disk, which also separates download / upload bandwidth usage. Network performance is solid, on downloads and uploads, so it's unlikely that the network is the actual problem. Protocol issue seems even more likely. More testing is required. After some testing, it seems that the SMB 3.0 is broken and that's the problem. File I/O repeatedly totally hangs and dies. Let's see how long it takes before some kind of timeout occurs. Of course this might be caused yet again by several different factors. After quick testing, it's obvious that the SMB 3.0 totally hangs on several occasions causing everything to fail. SMB 3.0 seems to be causing all kind of extremely strange errors. As example python gave OSError Errno 22 Invalid Argument when writing a test file. It's clear that there's some kind of IO issue for some unknown reason. Also writing to SMB3 share seems to consume a ton of CPU time compared to writing to local system with the exactly same program. Next question is if this is caused by protocol, network or the destination server itself for some strange reason. Btw. Now it also fails with Windows 10 client, way interesting and annoying. Made a small program for testing which writes 10 x 10 GB files, and repeats this until fail. And now it seems that running multiple parallel instances won't fail. But running single instance will. I'm also testing different storage systems on the image server. As quick work-a-round images could be restored over HTTP / FTP transports, it's trivial to setup. Because the image server already runs servers for both protocols. If the problem persists, workaround is easiest way forward. Interestingly it seems that higher network load makes things less likely to fail. Two clients, with two parallel write operations, seem to work without any problems.
• Peewee got so many new interesting features, like this, it didn't exist long ago: BitBitField - Now you can do bit field operations directly and there's bitmap data type. Support for FTS5 - So many goodies, like JSON1 - Also most of the features I used to use with execute_sql like lot of PRAGMA stuff is now directly supported using Peewee API. ZeroBlob and BlobFileAccess awesomeness. Not forgetting CompressedField type. This is kind of interesting, because I've actually implemented these features my self on top of old peewee. Just like the bit fields and compressed fields in many of my projects, using blob... And my own class... But now these are standard features with Peewee. Except, I've got also lzo and lzma compression, where Peewee offers only gzip / bz2. Oh well, I could once again refactor so much stuff, based on these awesome changes and improvements. ManyToManyField, omg, so much joy. Yet another great improvement. Which makes code a lot clearer and simpler. JSONField, even moar perfection. At this point it starts to feel that some projects should be 100% refactored, duh. So many things to code, learn and improve. Learning is never ending task.
• Something different? Clock of the Long Now aka 10000 year clock, single-camera setup vs multiple-camera setup and sRGB color space.

2019-07-02