Chats, Services and Open Source

1. Matrix data retention issue (@ GitHub) - Nice. So even rooms, which were complete abandoned were lingering around with all past messages. Because the delete unlinked stale stuff ie. garbage collect process was broken.

2. Updated SimpleX servers with latest version of SMP & XFTP server software, allowing setting extended data retention. Now servers keep undelivered data for 30 days. Allowing clients which aren't running 24/7 to collect and sync with the data later. Default values seem to be 21 days for messages and only 2 days for media attachments. I personally think it's not enough. Servers got anyway absolute excess of resources (RAM, Storage, Bandwidth, etc.), so there's no need to limit the retention due to server resources.

3. Duplicati is finally trying to fix the problems I've been reporting for years. Ref: - Atomic Transactions - Yet afaik, this still doesn't fix the problem with database rebuilding. If there are some old junk files in the path, it shouldn't completely prevent rebuild & restore. As far as I remember it's also two tiered problem. A) First leave junk B) Get confused by the junk. When those are fixed, then I would say that Duplicati is "ready", because well known corruption flaws are fixed as far as I know. - Just a minor wish. If a single file is added to backup set, instead of directory. The restore process gives Code: 2 always, because it created "a missing directory". Well, I don't see that as being a reason to change exit code or report it as error at all.

4. SimpleX.chat - About the server load management, it could be for the server when creating new queues to give a hint, that this server is under high load. Which would make the client to use another server (if possible). Flow, create a new queue on server. Server replies that it's under high load, please user another server if possible. -> Then the client decides if it really wants to create queue on that server or use another server with lower load. - I often see that many protocols lack this kind of server feedback channel and the communication is too much dictated by the clients.

5. Thank you Oracle, all of my VPS instances lost IPv6 connectivity. I sent two hours for trouble shooting and truth is that, everyone else I've asked from is having the same problem. Well, I hope it'll get restored at some point. - Can't really complain, when I'm using their awesome Free Tier. But still bit annoyed. - Based on some discussions days later, it's likely that they've just broken their RA / DHCPv6 setup and that's why the connectivity is lost. I can of course add manual network configuration. - Actually to be honest, I've seen exactly similar problems with at least two providers. In both cases, their response to "fixing the problem" was manual setup. Ok, why they then offer DHCP if manual setup is the right way? - Everyone can draw their own conclusions about that attitude. It's broken, but you can use workaround. - Thank you for that. - I also deeply hate the Oracle Cloud management system, but the Cloud Console seems to be okish. I've disabled IPv4 management access to the servers completely, so I can't login without using alternate console access.

6. Data hygiene, upgraded some NVMe wiping scripts, now the scripts use several overlapping methods. Full trim, secure trim, full over write and secure erase. Depending on disks some or all work, hopefully producing as good wiping results as we can realistically expect, without physically destroying the drives.

7. Microservices - Worked hard creating a few business critical microservices, aim was 100% reliability and recovery. Of course utilizing full transactionality, whatever happens, whenever the process dies for whatever reason, it should reliably resume working from that point on. As well as clean up any garbage potentially left around. Actually, this is how I always plan my code. Initially it felt hard, but when you just internalize the steps necessary for this, it's actually pretty trivial. Also some repeated tasks are optimized for speed, locked records are skipped and if updates fail due to locks, that's also ok, because the next update run would overwrite the data anyway. Etc. All timers got jitter implemented, quadratic back-off and other basic stuff.

2024-04-28