Duplicati / Restic - Backup Program Comparison

Initially I planned to post this stuff quite a long ago. But I've held back this post for a while, so I do have the experience and results to deliver immediately.

Duplicati (@ duplicati.com) vs Restic (@ restic.net):

For reference if anyone cares at all: Restic version 0.16.2 and Duplicati version

Two open wishes for Duplicati team:

Two open wishes to Restic team:

Duplicati issues, in past

Dupliati had serious data corruption issues: Duplicati Forum (@ forum.duplicati.com).

Duplicati - Just one thing is interesting, how it's possible that the test and restore functions are so different. Afaik, it tells that the application is pretty much guaranteed to be flawed. If it would have something to do with actually data corruption in the remote store, test should catch it. Or if it doesn't, then the test is even worse broken than I thought. Yet again, my primary suspect is the same that we've covered earlier. Repair / Database Rebuild <- That combination is broken, with some other things being broken, leading to total mess. - That other thing is backup data transactions. When program is shutdown it might leave a mess behind, and when it runs again, it doesn't know anymore, which data is current and which is stale so the correct way from resuming that situation just fails. Repair does something, but the result could be good or bad, quite randomly depending on the aborted operation stage.

But sure I could try the task with more durable storage, yet I'm doubtful it would fail in any reasonable time, nor that the root problem actually is the storage durability in this case. The storage system is used for many other things, which are encrypted and hashed, and none of those got any integrity issues. It's always and always, Duplicati RESTORE only issue. Verify nor Test never fails. Which points to software logic issues, afaik. I've seen lots of bad software, I've also written lots of bad software and learned in the process of failure.

Also at one point they had broken lz77 (xz) compression library, which corrupted data. I didn't ever see any kind of confirmation that the problem would have been actually fixed or the broken library disabled. But hopefully that's done. For me, it just did mean that I didn't use lz77 compression anymore.

Btw. Just now I'm running a full test to one of the backups sets, just to figure out what's the problem. I did start too tasks, restore with database and restore without the database. And if that makes any difference, it just pretty much is direct proof about all the things I've repeatedly said in the Duplicati forums. Yet the problem doesn't appear immediately, it's some unsafe data operations, which break the data set. I've even provided detailed logs earlier to the dev team.

Just to be sure, I'll run the tests on the corrupted data sets, to see, if any of the data is actually corrupted or if the app either incorrectly tests or just badly restores it. On some situations there's separate recovery tool, which allowed to restore the data. But why did the process get so messed up? Why normal recovery (based on journal) and or recovery using repair, didn't detect or remedy these issues?

Note. I haven't been following early Restic development, I'm sure they have had their issues as well.

Finally there's a Duplicati version which should fix these issues:
They have also added an option --repair-force-block-use, which I assume would help recovering the unrestorable backups, which might have just issues with the index files.

Phew. I really hope, I'm done with this subject now.