Blog

Google+
My personal blog is about stuff I do, like and dislike. If you have any questions, feel free to contact. My views and opinions are naturally my own personal thoughts and do not represent my employer or any other organizations.

[ Full list of blog posts ]

AMP, asyncpg, Python Secrets, WA 2FA, LAN protocols, LinkedIn, Crypto, PostgreSQL, Ubuntu 17.10

posted Oct 20, 2017, 8:09 AM by Sami Lehtinen   [ updated Oct 20, 2017, 8:11 AM ]

  • Several articles about AMP. Well, AMP is slower than fast website with CDN. So why would anyone use AMP? I personally don't like it too much. I believe in lightweight bleep free websites.
  • Played a little with asyncpg. That's fast. I'll use it if and when required with suitable setup / configuration / use case.
  • Tested and played with Python 3.6.0 secrets module.
  • Am I the only one who finds the WhatsApp constant two-factor authentication (2FA) reminders extremely annoying? Of course I go the 2FA key stored safely. It doesn't mean that I would need to remember it.
  • Reminded my self about a few things before local area network (LAN) management meeting:
    MSRPC, mDNS, WMI, IPC, SMB, and neat tutorial about calling RPC functions over SMB.
  • Whoa, LinkedIn SMS 2FA is working again. It was broken, and they claimed it was my fault. It took them a week to admit and fix the issue, but now it's working again. I'm not happy about the initial response. But in average, the end result is better than usual. Most of helpdesks just feed you FAQ lies and don't even bother to look at the problem. Getting the problem actually fixed is quite rare. I'm still asking LinkedIn to add backup codes to 2FA, and allowing TOTP as alternative 2FA method.
  • Person being held prison as not divulging the password? You'll just need to XOR these together: random: ea74f9e9db9c514f data? : 8c019a82fbe53e3a Got my point? Random can be anything. As well as data can be anything. So it's possible to produce whatever 'evidence' is required. Often when talking about encryption, the random is pseudo random stream derived using some algorithm and then used with cipher. But it might or might not be that way. Everything is possible. Mixing bits around is trivial, making it in secure way is harder.  In some cases there's no password. The data can be on one device and the 'random' can be on other device. Also the random or the data part can be encrypted. - This is actually where 'standard encryption' is bad. Because it makes it pretty easy to know if it's cracked or not. Everyone also says that using standard crypto is a good idea, and doing something non-standard is a bad idea. But it's not always that simple. When using non-standard crypto, it makes many things very much harder. And at least the attacker needs to use valuable resources like cyrpto experts to try to decrypt to data. That's why some times bad crypto might be actually better than high end standard crypto.
  • Is PostgreSQL good enough? - I've believe, that the answer is usually yes. I'm also often thinking that SQLite3 is also good enough for most of cases. Of course it's possible to mix dozens of technologies and then spend months or years having issues with those, as well as complicating setup, configuration, installation, version management, etc all with excess complexity. When you could just use a simpler approach. Some projects are large, require different technologies. But for many projects, using mess of technologies is guaranteed way to hinder development. You'll end up tinkering with different cool tech toys, instead of getting the job done which pays the bills. We're making products / services to solve customer problems. Not to do academic research on different ways how to to accomplish same thing, using other neat technologies. - It's easy to forget how much research and study is required to make any new tool actually usable in production and so that everyone understands how it works. - Because it's often a real challenge to get that done even with one technology. - Afaik, it's newbie mistake trying to mix every possible design, paradigm, library, framework, something cool, and new, in a single project. - That's why you should have separate study projects and learning time, when you can play with those. And not trying to push all that stuff into production projects.
  • Ubuntu 17.10 release notes - GNOME, Kubernetes, Linux, Visual Studio, Snapcraft.io - Snaps with delta updates, Robot Operating System (ROS), OPAL storage - Quite a nice list.

Future Retail, CF, IP Spoofing, Pigz / GZip, Browsers / Apps, Cubes-OS, Projects, XLSX / CSV

posted Oct 15, 2017, 12:06 AM by Sami Lehtinen   [ updated Oct 15, 2017, 12:08 AM ]

  • Watched a set of future retail store videos from Youtube. Some of those videos were kind of funny. I've got a very clear vision of fully automated retail store. But in some of the videos there were extra manual steps which were mostly requiring additional labor and making the process awkward.
  • What's my vision for future retail? Afaik, perfect concept would be 100% automation. So basically it's like shopping at any web shop. Some Finnish chains are already offering 'pick-up' service. But it still requires someone to pick-up the goods. In my concept, store is fully automated, and there's no precollection of goods based on orders. Order is only automatically collected when you arrive or are arriving. So if you can't come for the pickup, good won't get collected. If collection system is fast enough, and not the bottleneck of the system, precollection isn't required at all. Adding precollected storage / delivery unit could be used if the collection process is the bottleneck. Let's say that the peak load of the system is 17:00 after people leave from work. In that the collection and temporary storage process could get started a few hours earlier. Minimizing precolection allows keeping the products and goods in right temperature as long as possible. Of course the option currently provided in Finland is great precursor for this. Because the basic concept to end customer remains the same. Web / mobile order & pickup / delivery. If people love and use that. Then it's easy to calculate where the break even for full automation goes. Because it's going to be expensive, really expensive. That delivery option also is perfect fit for future automatic cards / delivery vehicles. These things can be easily integrated to any existing web-shop.
  • CloudFlare servers getting MITMed? Yep, so it seems by Airtel. This is just another reason why backlinks from CDN edge nodes should be also encrypted. It's also good to read implications section of this post.
  • Strange Loop - IP Spoofing - Awesome talk about DDoS IP Spoofing and attacks. It would make so much sense to filter spoofed traffic, but there are still lot of ISPs not doing it. Nothing new in this post, but it's great overall summary.
  • Pigz, which stands for Parallel Implementation of GZip. That's nice. I usually often opt for 7-zip LZMA2 compression when compressing files. Totally unrelated but the multi-threaded parallel PAR2 I'm using seems to segfault often on smaller files.
  • Browsers, not apps - I love web browser vs apps. Especially for cases where I'm not using those too often. I don't want too many BS apps. No I won't install your app, go away. Unfortunately many websites try to force users to use their BS app. For no reason whatsoever. I'm sorry that you've got so incompetent web developers and no, I still won't install your crap app.
  • Played with Cubes-OS. I really like the application isolation. I would love to get that tab isolation on browsers too. But currently I don't have use cases for that technology.
  • Watched a documentary about Google Moonshots aka Google X projects. I liked their approach, it's very similar to mine. Let's take the hardest problem first, if that's not resolved then we know that the project is done, aka canceled. I think that's the best way of taking on projects.
  • Had long discussions with one team about XLSX and CSV. They wanted to use XLS / XLSX. I said, XLS sucks, let's use CSV. It took quite a while to make them realize that MS Excel files are horrible mess, when UTF-8 CSV is something beautiful which just works and is compact. It's possible to turn CSV data into XML bloat, but it doesn't actually bring any extra value. It just makes parsing lot slower and might expand the amount of bytes (not data) by order or magnitude or even more. Why bytes not data? Well, that's just because the data is actually the exactly same what would have been provided in CSV format. So, don't ever ask me for excel import or export, I'm gonna hate it. Yes, I can do it,I've got naturally preselected and tested libraries for that, but it's not a really good thing to begin with. As far as I remember there has been only one project so far which has selected XLSX for the official data transport format for integration.

LinkedIn 2FA, MVP, Server Problems, NTFS file system

posted Oct 7, 2017, 8:59 PM by Sami Lehtinen   [ updated Oct 7, 2017, 9:00 PM ]

  • I'm wondering if anyone else is 'locked out' from LinkedIn because their 2FA is broken. To begin with, LinkedIn 2FA sucks. They do not provide backup codes, for cases where 2FA fails. Nor they provide TOTP.  So only way to receive the authentication codes is SMS. Then they did break their 2FA authentication by them selves. First it was unreliable, and now it isn't working at all. When I complained about that, they're asking me to provide government id copy, like a passport, so they can disable 2FA for my account. First of all, if you go and bleep the things. Is it my fault? Why should I need to do something, I don't really mind really doing, to regain access? When it's completely your fault. Because the login process is totally crappy and badly designed and then after all you go and bleep it up. - I just wondering. You made me go Linus. I don't even know if I want to use such anymore. Just so much fail. I appreciate some organizations being so. That's not even a strong way to authenticate user. Let's say hacker would have gained access to my email. I travel a lot, it's highly probable that they'll find a copy of my passport from my email box anyway. Therefore, it's not a sane way to strongly authenticate user for disabling 2FA. - Note this is from my really long backlog(!)
  • Wondered the normal business practices in a meeting room. 1) Ethernet sockets locking broken on switch. 2) Ethernet cables twisted and broken. 3) Ethernet connector locking clips broken at the end of switch. Wrong color of Ethernet cables being connected to wrong sockets. And so on. Can't stop loving totally hopeless people. The usual story, very unfortunately.
  • A friend gave immediate feedback about EasyCrypt that's it's dubious project, because they don't provide proper legal business information at their site.
  • Had long discussion with one client about RFID, ERP and traceability of goods. How the processes should work, and what costs are involved and which benefits the new tracking solution would provide to the organization.
  • Long talks about MVP and minimal approach. There are some service providers providing e-salary, which delivers electronic salary calculation reports. I laughed at the site, it's as MVP as it can be. Login, password, list of PDF files with link. Neat. Who said launching new services needs to be complex? Just stick to the basic and essential. That service does everything it's supposed to, yet it's extremely simple.
  • When discussing 'server problems'. I encountered again that old concept, correlation doesn't prove causation. In one discussion it was mentioned that all the problems lately have been related to vendor X. Yeah, sure. That's because we've been only deploying systems lately with vendor X. There are also many other sourced of problems than the server platform itself. Often people also misjudge the problems and draw all kind of conclusions without any facts. It doesn't work, it's caused by X? Is there any proof that it's caused by X? No, but we just assumed so. That's of course totally normal, and happens easily, unless you're paying attention and requiring factual evidence. Sometimes obtaining the factual hard data based evidence can be extremely hard and time consuming. It's totally normal that people claim that there's something wrong with the server if something doesn't work. No, it doesn't mean that at all.
  • Again random numbers and whitening. Related to 33c3 wheel of fortune talk and bad random numbers. Von Neumann extractor & Bernoulli sequence.
  • I'm so happy I'm using NTFS for my USB sticks. Once again something is corrupted, let's see if it fully recovers after proper chkdsk on Windows. On Linux some directories are just inaccessible due to corruption. Lack of proper chkdsk / fsck for NTFS stops me from fixing the errors right now. But NTFS is so robust, it shouldn't matter if I don't fix the errors right now. - As expected chkdsk on Windows fixed the issues, which fsck on Linux didn't. As a pro-tip, don't try to fix NTFS volumes using Linux. This is something fsck can't fix properly. Yet the NTFS is so robust, that even if I knew the volume was corrupted, I weren't afraid about writing to it. exFAT would be radically faster, especially with small files. But when storing valuable information, source codes etc. It's just not an viable and reliable option.

    Chkdsk log:
    Stage 1: Examining basic file system structure ...
    Deleting corrupt file record segment 301.
    Truncating corrupt attribute list for file 74921.
    Deleting corrupt attribute record (0xA0, $I30)
    from file record segment 0x124A9.
    Deleting corrupt attribute record (0xB0, $I30)
    from file record segment 0x124A9.
    Stage 2: Examining file name linkage ...
    Removing corrupt index $I30 in file 74921.
    Recreating deleted index $I30 in file 74921.
    CHKDSK is scanning unindexed files for reconnect to their original directory.
    Recovering orphaned file ### (140) into directory file 74921.
    ...
      3 unindexed files scanned.
      3 unindexed files recovered to original directory.
    CHKDSK discovered free space marked as allocated in the
    master file table (MFT) bitmap.
    CHKDSK discovered free space marked as allocated in the volume bitmap.
    Windows has made corrections to the file system.

Learning vs Performance, OB, VBox, MFA, Batteries, Project Management, SPF, Testing

posted Oct 1, 2017, 12:31 AM by Sami Lehtinen   [ updated Oct 1, 2017, 12:31 AM ]

  • Just listened a lesson about professional life and studying. I agree with it. When working on project, it's important to get it done quickly and efficiently. That's when you should use all the boring tech you've got. It's done and works reliably. Then there's free time, when you can study and experiment. In that case, your primary goal is to test things out and see what happens. It doesn't need to be productive nor reliable. It's just quick ad-hoc bleeding edge and maximizing the learning experience, even if it would be total disaster. That's just why I've got my hobby projects. Decades ago I sometimes tried to use something new cool for production, and ended up shooting my own leg badly. Lessons learned. kw: learning zone, performance zone
  • Installed OpenBazaar 2.0 developer version and checked it out. Looks good so far! Also played with it's REST API a bit. This also means that I had to install go lang and compile my very first ever go project. kw: OB2, OB1, Peer 2 Peer (P2P), Free, Crypto, Market, Bitcoin
  • Read a few start-up guides, just for fun. At least this was nice and short. - How to start a startup without ruining your life
  • Played with VBoxManage and tested everything out. Sometimes it was annoying when referring to .vdi files that referring to files failed, because the .vdi UUID is already active. It doesn't matter, then you can use the UUID to refer to the file, but ... It would be nice if that wouldn't require manual extra step.
  • Once again had several hours long workshop about system backup & restore. How to make sure that data integrity is preserved, how to backup different databases without bringing down the database service, still maintaining data integrity, etc. How to make sure that even in case where hacker could access some of the key systems, they still can't access / delete the data.
  • What's the point of using MFA and when to use "application passwords" without 2FA. When the static application passwords are much weaker than the primary password I'm using even if I've got the 2FA enabled. Hmm. Strange approach. Many services seem to provide somewhat insecure static passwords.
  • Evening reading: Lithium-air battery , Lithium-sulfur battery. We all know how horrible Lithium battery news have been lately. But it doesn't need to be that way. Lithium batteries can be totally safe and double the capacity, and be even cheaper than today.
  • Wondered one project manager. He was astonished about the fact, that there were new requirements popping up during the software development project. Scope creep is hardly anything new. I think it's usually more norm than exception. That's why it's so important to make agreements right and keep project in-line and to manage customer expectations.
  • Reminded my self about basic stuff like: Jevons paradox and Griffen good.
  • Nice article about web bloat. Shouldn't be a surprise to anyone. Most of websites are absolutely full of s...t. Like a news site, with a short 1 KB article, might require you to load 10 Megabytes of junk before you can see that 1 KB article. That's well, bit inefficient, slightly said.
  • Helped a friend to beta test their silent launch with anonymous email forwarding using custom domains. It's awesome and working. I'm now using it for a few of my domains too. It makes email configuration super easy and simple as well as address management. Also featuring email reply mapping / masking. Which allows you to use your custom domain, with any existing email service, which doesn't support custom domains. That's very neat extra feature.
  • Using simple SPF include rule is very simple and easy configuration to understand. There's also benefit that, if you change the SPF info. It'll get auto-updated for all users. Otherwise you might end up having lot of angry people. This is also why outlook is doing exactly same. Yet, I think it's stupid that they're chaining the rules, and not forking directly. Aka, it would be better to have one spf rule which refers to others. Than having SPF rule which refers to SPF rule which refers to SPF rule, as outlook is doing right now. Some verification sites claim that the depth of the lookups is too great, and not all email services are going to check all the SPF rules due to excess nesting.

Learning, OB2, Startup, VBox, DR, 2FA, Batteries, Project Scope, Web Bloat, Silent Launch, SPF

posted Sep 23, 2017, 11:34 PM by Sami Lehtinen   [ updated Sep 23, 2017, 11:35 PM ]

  • Just listened a lesson about professional life and studying. I agree with it. When working on project, it's important to get it done quickly and efficiently. That's when you should use all the boring tech you've got. It's done and works reliably. Then there's free time, when you can study and experiment. In that case, your primary goal is to test things out and see what happens. It doesn't need to be productive nor reliable. It's just quick ad hoc bleeding edge and maximizing the learning experience, even if it would be total disaster. That's just why I've got my hobby projects. Decades ago I sometimes tried to use something new cool for production, and ended up shooting my own leg badly. Lessons learned. kw: [learning zone vs performance zone].
  • Installed OpenBazaar 2.0 developer version and checked it out. Looks good so far! Also played with it's REST API a bit. This also means that I had to install golang and run the go compiler for first time.
  • Read a few start-up guides, just for fun. At least this was nice and short. - How to start a startup without ruining your life.
  • Played with VBoxManage and tested everything out. Sometimes it was annoying when referring to .vdi files that referring to files failed, because the .vdi UUID is already active. It doesn't matter, then you can use the UUID to refer to the file, but ... It would be nice if that wouldn't require these annoying manual extra steps.
  • Once again had several hours long workshop about system backup & restore. How to make sure that data integrity is preserved, how to backup different databases without bringing down the database service, still maintaining data integrity, etc.
  • What's the point of using 2FA and then "application passwords" without 2FA. When the application passwords are usually much weaker than the primary password I'm using with the 2FA. Hmm. Strange approach. Afaik, the static application password should be very strong passwords.
  • Evening reading: Lithium-air battery , Lithium-sulfur battery. We all know how horrible Lithium battery news have been lately. But it doesn't need to be that way. Lithium batteries can be totally safe and double the capacity, and be even cheaper than today.
  • Wondered one project manager. He was astonished about the fact, that there were new requirements popping up during the software development project. Scope creep is hardly anything new. I think it's usually more norm than an exception. That's why it's so important to make agreements right and keep project in-line.
  • Reminded my self about basic stuff like: Jevons paradox and Griffen good.
  • Nice article about web bloat. Shouldn't be a surprise to anyone. Most of websites are absolutely full of s...t. Like a news site, with a short 1 KB article, might require you to load 10 Megabytes of junk before you can see that 1 KB article. It's well, bit kind of inefficient.
  • Helped a friend to beta test their silent launch with anonymous email forwarding using custom domains. It's awesome and working. I'm now using it for a few of my domains too. It makes email configuration super easy and simple as well as address management. Also featuring email reply mapping / masking. Which allows you to use your custom domain, with any existing email service, which doesn't support custom domains. That's very neat extra feature.
  • Using simple SPF include rule is very simple and easy configuration to understand. There's also benefit that, if you change the SPF info. It'll get auto-updated for all users. Otherwise you might end up having lot of angry people. This is also why outlook is doing exactly same. Yet, I think it's stupid that they're chaining the rules, and not forking directly. Aka, it would be better to have one spf rule which refers to others. Than having SPF rule which refers to SPF rule which refers to SPF rule, as outlook is doing right now. Some verification sites claim that the depth of the lookups is too great, and not all email services are going to check all the SPF rules due to excess nesting / recursivity.

Cyberwar, OVH SMB, User Identification, GAE, Integrations, Telegram

posted Sep 16, 2017, 10:05 PM by Sami Lehtinen   [ updated Sep 16, 2017, 10:06 PM ]

  • Friend watched Zero Days Cyberwar Stuxnet documentary and got worried. Don't worry. There's practically nothing you can do about it. If you're getting targeted, they'll get it. So it's pointless to worry about it. You can harden systems, but it won't stop certain actors. Worrying about it, is like worrying how you can protect your summer cottage from intercontinental nuclear weapons. Well, you can't.
  • About OVH and TCP/445 blocking / filtration. They claim it's mandatory to filter TCP/445 RPC/SMB traffic on network edge. Because without filtering hackers would get it. I guess it's better I don't say my honest opinion about that rationalization. But oh well. Just like with many rules, if someone is stupid, it doesn't mean everyone would be stupid. But that's often a concept which is way too hard for many to understand. Yet on the other hand they claim that all ports are open, which is obviously bs, when some ports are being filtered. Traditional we're lying to customers making things seem better.
  • Web and cloud services are being used as a proxy in monitoring and identity tracking? Some services require sending them information about official identification documentation. This is just a way to gather information about the users. As example account is suspended until official identification information is provided, even if any rules wouldn't have been violated. Usually if basic rules are violated account is immediately suspended without option to reactivate it. Of course this isn't anything new and it's just one way to abuse and harness power. Let's say you open account for some confidential use, then some parties get interested about it. Then they ask the service provider to close down the account and ask for official identification like a copy of passport. Then it's up to the user to decide if they want to provide that information, if they will provide invalid information or if they simply accept the suspension and continue without the service. This is huge problem with cloud services, you can basically get locked out of the service at anytime. And it's totally arbitrary when this will happen. On the other hand, as possible  service provider, I also understand this. If I'm running service X, I should be able to select the users totally freely. I can delete whatever I want, whenever I want as well as block users from accessing the service or add additional requirements for users to get continued access to the service. No warranties whatsoever. Yet, this is kind of risk to the users, which naturally should accept it the possibility for this. Yet asking such information might undermine security and privacy, in some specific cases, and denying pseudonymous operations, even if any law isn't being violated.
  • Previous statement is also one of the reasons why I'm not running anything on Google Cloud Platform / Google App Engine anymore. They could at anytime without any reason close the service(s) down. Of course I've got multiple off-site & cloud backups. But in general that's a huge risk, because App Engine apps aren't easily transportable to "any" provider around the world. Same applies for the people going for AWS or Azure. It's huge risk. Your business might be gone tomorrow and fixing it to run on other platforms, depending on how badly it's designed, might take months.
  • About integrations, I'm just  mentioning that also shared memory, database, message queues, message passing, FTP, HTTP, REST, XML, JSON, SOAP, CSV, PUB/SUB, pure TCP, UDP, etc, whatever can be naturally used. - It's just the same stuff, in different packet. Some people seem to think it's big deal, but it really isn't. It's just getting "message through" using whatever means available. I've seen so many guys seriously stuck with minor details without seeing the large abstract picture. When building house you could spend two years selecting suitable nails, and discussing if screws or nails should be used and what kind of metal alloy those should be. But it'll be probably highly unproductive. It's of course different story if you're doing your PhD about different nail alloys and plating techniques and how those last decades of different environmental factors.
  • Telegram's HTML5 web implementation is awesome. Even if there would be a marginal platform which wouldn't have native Telegram client, you could always fall back to their great HTML5 implementation, with push notifications etc.
  • Something different: Reminded my self about XB-1 from Boom Supersonic and Kh-55.

SQLite3, Unicode, httpbin, SLO/SLA, HTTP Pipelining, Android slow(er) WiFi

posted Sep 12, 2017, 9:52 AM by Sami Lehtinen   [ updated Sep 12, 2017, 9:53 AM ]

  • Multi-Threaded SQLite3 without the OperationalErrors - Yet another totally awesome post by Charles Leifer. - Thanks! - Thoughts: I've had pretty similar thoughts. Yet as said, I often use bit different approach to keep the writes as short as possible. I read data, process it and then check if the record getting updated is still the same. So processing data might take 5 seconds, but at the end, updates happen in very short block. Of course this might lead to situation where value has changed and there's need for retry. Aka opportunistic locking. So this isn't perfect solution for very volatile data. With some of the most volatile data, which doesn't require absolute consistency I've got very different approach. I'll update in RAM counters, and then just flush updated values to disk, ever 15 seconds, or 15 minutes, whatever seems to be the reasonable interval. On graceful shutdown data is flushed to disk. This allows 'very high transaction rates'. Also individual processes maintain own counters, so there's no memory lock contention either. About Charles approach, it's very nicely wrapped and pretty obvious. Just use single thread for writes -> removes all locking issues. It can also cause bottle neck, but actually in this case the bottle neck already exists and it's the SQLite's single active writer limitation. About the gotcha, that's obvious. As many other performance improving techniques, it requires some planning and writing program logic in a way that it works efficiently with the underlying system. I'm waiting for the SqliteQueueDatabase. I've got a few projects which really could use it. - fopen, ha. Yeah. Sure. - I'm using several different databases for different projects. PostgreSQL, Raima SQL, Microsoft SQL Server. Yet for most of projects with small requirements I do choose SQLite3. It's good enough, and doesn't require additional setup, planning, discussion possibly expensive licenses etc.
  • Had some Unicode play. Usually Unicode is so transparent you don't even need to think about it. But now I had an issue. I had to work with Unicode Mahjong tiles. Now I know what's the difference between \u (16 bit) and \U (32 bit) and that chr() integer -> code point can be also used with Unicode nicely. It's all so simple and easy and too just a couple of minutes to figure out. But I haven't had any reason to do it earlier, because most of programs just handle Unicode without any issues. Also with Python 3 the ord() -> code point to integer works with Unicode seamlessly too. Awesome. Some people complain about Python 3 Unicode. But I think it's actually just the way it should be and that's awesome. 
  • Here's a nice Unicode password generator. I thought I could make one, but now I don't need to.
  • Another interesting observation is that when OVH delivers Windows Servers, the Windows Firewall is disabled by default. Meaning that all 'basic windows network services' are accessible from whole Internet. Maybe this is the reason why they're blocking SMB/TCP445 on their network edge? Maybe, maybe not. I don't know. But that's one thing which has been also baffling me for quite a while. (Btw. They have fixed this while this post was in backlog). But they're still blocking SMB/TCP445.
  • About great tools, httpbin - HTTP Request & Response Service is totally essential service. Especially if doing something quickly and ad hoc. Otherwise you could setup your own test systems, but for quick fixes testing against something online is just way to get it done now.
  • Service Level Objective (SLO) key component of Service Level Agreement (SLA).
  • A very nice post about HTTP/1.1 and Pipelining requests. Using Japronto Python HTTP server and micro framework. Lovely optimizations, can't say anything else.
  • Just suspecting that Android 6.0 update made 5 GHz WiFi slower on one old device. Earlier WiFi Speeds were over 90 Mbps, but after upgrade those have dropped to around 30 in the very same environment. Got any scientific proof. Nope, got adequate sample size, nope. But this just happened at the same time when Android was upgraded. Why I'm posting this? To see if anyone else has had similar experiences.

Linkedin 2FA, Caching Performance, OVH TCP 445 SMB, IP address, Network Reachability

posted Sep 2, 2017, 9:50 PM by Sami Lehtinen   [ updated Sep 2, 2017, 9:51 PM ]

  • LinkedIn 2FA repeatedly failing. Enhancing security is easy, if you just disable login option completely and prevent everyone from logging in. That also efficiently prevents build up of data which might need protection. That's awesome. I can honestly claim, I'm running the worlds most secure website. It's in my head, and I haven't even told everyone about it, and there's no hardware or software involved. So there aren't any bugs ore exploits. Nor there's any user data. It's perfection! (FYI, this entry is from a long backlog)
  • About read caching and performance. Once back in DOS and CD-ROM days. We bought pretty awesome drive. It was 7 CD changer with 4x CD-ROM drive. Due to minimal buffering and multi user access to the BBS system, the most awesome trolling method was accessing several disks from the CD-ROM drive simultaneously. Changing disk took like 10 - 15 seconds. And then user would get a few kilobytes of data, and then it would trigger change to access the next disk. Only a few releases later the software contained option to cache the downloads to HDD, so that users downloading stuff from multiple disks wouldn't basically bring the whole system down due to extreme I/O lag. Btw. Because the system didn't contain parallel I/O queues. The CD-ROM trick also blocked any HDD I/O and CPU usage, due to I/O wait and blocking I/O operations. Making it totally impossible to use the system for the users. Like trying just to read private messages. Of course because the operations wasn't being spooled, this situation usually lead to case, where users would get extremely annoyed and probably disconnect, until there were only one user downloading from CD-ROM changer disks again. The system was usually on high demand, so this caused funny spike in login attempts. Due to lag, and users disconnected and the system became free for more users. Which find out that it's so slow it can't be used and disconnected. It's just like the lagged websites, those might get highest user visitor count ever, due to users trying to reload the site, until giving up. Therefore getting high visitor count can be gotten by providing extremely bad service on purpose. On normal day the system could handle something like 20 - 50 users with dual access lines. But on this kind of day, the login count could be easily 10x more. The same users came back a few hours later trying again to read the messages. If it still failed, they would try again later.
  • OVH and TCP port 445 which is used for SMB protocol is being blocked. I also used portquiz.net to trouble shoot some networking issues, and guess what. The service is also hosted at OVH, so that just confused me even more, before I realized what was happening. I also used one Linux server for trouble shooting, and everything worked fine. But it was hosted on OVH and that's why it worked, because it was OVH intranet traffic. I got totally confused before I figured out the facts. After rechecking everything and starting to sum up the data I've collected. I thought that I'll try from one Windows server hosted on OVH and it worked. So it seems that OVH is blocking TCP PORT 445 SMB even if they claim they're not. All ports open, except 445. World wide total blocking of port 445 doesn't make any sense after all. This is one of the reasons why you should really well know your tools and configurations. Because otherwise those might just add extra confusion, instead of actually providing solutions / answers. But of course this is nothing new. Keep in mind that correlation does not imply causation. Bit extra testing from home where I know my network configuration 100% and tracing using TCP SYN packets confirmed it. SMB / TCP 445 port traffic is dropped at OVH network edge. Sigh. Testing with several random ports, and using several different ISPs result is always the same. Yet everything works if the same tests are run from another server inside OVH network. Ok, that's it. One thing which also mislead me is that if the ports 135-139 are accessible, the error message while connecting to server changes. Giving impression that those traditional NB ports could be used for communication. But after all everything fails, because 445 is blocked. For inbound connections GRC ShieldsUp scan is a true classic. I even had long discussion with OVH helpdesk, and they said that blocking TCP port 445 is mandatory. I guess we should thank Microsoft providing so bad code that it leads to global TCP port blocking by port number? Eh.
  • In P2P network discussions there was often question how to define if node got "full Internet access or not". My answer was that it can't be determined. And that's the only truth. Having access to A, B, C doesn't mean full access. Nor not having access, means that there wouldn't be access to any other parts. Working connection now, doesn't mean that the same connectivity would be there 15 minutes later, and so on. Also the IP address for outbound traffic can be different than for inbound traffic. Or the IP can be dynamically assigned on connection creation from a pool. So address changes per TCP connection / UDP outbound 'connection'. As well as the IP / port for any inbound / outbound address can be defined separately. So if you connect from port 500 to port 1000 to IP a. It doesn't mean that port 600 to port 500 to ip B would work. Nor that the IP address would be same for that connection than it was for the previous connection. My firewall allows me to define route and IP based on any basic criterion. protocol, source, destination, port, ip, etc. Yes, it means that SMTP out can even use different AS and service provider than connection to port 80... Even if the destination IP is the same. 

Safety, Pipenv, SSH ChaCha20, User Accounts, SPF, Sysfs, Avaiability, TCP stack

posted Aug 27, 2017, 5:47 AM by Sami Lehtinen   [ updated Aug 27, 2017, 5:48 AM ]

  • Trident Safety - Well, as said. It doesn't really matter if IT companies got really ridiculously bad security. But when bit more 'serious businesses' got bad security, it's bit more serious. Yet nobody seem to care about that either. Tail gating, my classic favorite, made me smile. Not checking what you're bringing with you. Just the standard procedure. Well, I've been pretty happy with airport security lately. Yet here are still frequent fails. At least on basic level, they even try to follow security procedures and those aren't being completely ignored. At least on international airports. Well, every business got it's up'n'downs. Nothing too special.
  • Checked out pipenv - Awesome tool. I've been using virtualenv and pip naturally with my projects. Yet usually I hope I don't need it. Why? I prefer to use limited number of libraries and usually the most recent 3.X python version. Which means that the configuration is so standard, I don't need different setup & configuration for each project. Of course this is due to my very limited scope of projects. There are some other projects which I'm using, which do heavily rely on pip and virtual environment for dependency & version management. But even these projects provide ready packaging, unless you're building everything your self and using developer mode. 
  • OpenSSH added new cipher: chacha20-poly1305@openssh.com, yet GCM modes have been supported for quite a long time. Some older SSH clients dno't yet support other than AES-CTR modes. Newish ciphers: aes128-gcm@openssh.com, aes256-gcm@openssh.com
  • More "standard personal user account management processes". Lulz. How about sending about two hundred user names in one mass listing to everyone listed. Just pick your own credentials from the list. This is always as funny as it happens, and seems to be the normal way of doing things. We anyway trust everyone, so there's no need to keep private and personal credentials personal. It's also very handy, if I don't have access to project X, I can just pick someone else credentials who got the access and it's not going to be show stopper or slow down the process. - Awesome. - We get the things done, and do not focus or waste time and effort on non productive security bs discussions. - Only good thing this time was the possibility that I could login with the 'everyone knows' password using my accuont and change it to use actually personal password only known to me. But as I've reported earlier, often that option has been disabled / prevented. Because it wouldn't be handy at all, if someone wouldn't know my password and couldn't do management tasks which are only assigned to small group of people. - Why this reminds me about that Trident Safety post strangely.
  • Once again never ending discussion about SPF and spam. Duh. It works, if you configure it correctly. If you mis-configure stuff and don't do it right. Well, it works just and exactly as it's supposed to work, aka not work. What is the problem here?
  • LinkedIn 2FA again broken. It's nice to have safer system. But if it makes in unusable or inaccessible, is it actually better at all? (This is from backlog too)
  • Reminded my self about Deadline scheduler's sysfs tunables.
  • Nice post in Google Cloud Platform's blog. Available or not - Things they mentioned: Availability, Mean Time Between Failures (MTBF), Mean Time to Repair (MTTR), Error Budget, measurement, user-expectations, business objectives, Site Reliability Engineering (SRE), cost/benefit trade-offs, opportunity costs, developer operations.
  • I see all the time messages how complex TCP stack is. Well, I've seen pretty compact implementations. telnet.com during DOS times got one. It maintained only minimal information for a single TCP connection and worked pretty well, as well as used fixed RWIN. I've also seen pretty interesting implementations on RS-232C - Ethernet / TCP adapters. If you have to implement cheapest ever way of maintaining TCP connection with minimal code & RAM & CPU power. It can be done in pretty compact space. Oh why telnet.com? Well, whole point of .com files is that maximum size was 64 kilobytes. With today's bloat crap ware, it seems miraculous that anyone could have ever done anything with less than 64KB apps.

MSSQL, Data Polling, RDP, Mobile Auth, Security, Credentials, Mental Models

posted Aug 20, 2017, 2:32 AM by Sami Lehtinen   [ updated Aug 20, 2017, 2:33 AM ]

  • Read a few more articles about MS SQL (Transact SQL, T-SQL) performance fixes and issues. Most of the tips were of course extremely obvious. Like fixing I/O, limiting I/O, making sane queries, reuse queries (for query plan reuse), use indexing (don't over use), separate data & log, don't use production database / server for scratch / temp data, too small VLFs, do not over allocate memory for MS SQL (causing memory deprivation of other processes and operating system), semi-slow queries which are run repeatedly like polling. Nothing but obvious stuff. But it's surprising how often these facts are forgotten.
  • One polling query is a good example. It's not actually slow, but it's run 50 milllion times per day. Even if it wouldn't return anything, it still requires lot of resources from server side. Especially if there's data which needs to be filtered and or sorted on server side.
  • More interesting observations about Remote Desktop Protocol / Remote Desktop Connection design fails. It seems that there isn't any kind of activity / networking timeout. Addresses getting banned on firewall level can linger as established TCP connections indefinitely. I guess this is also one of the reasons the RDP is so crappy and extremely easy to DoS. Negotiating connections with server to certain state and then just disappearing leaves the server with tied resources lingering forever. - Great, just great. Some protocols are just (a lot) better than others. I would understand this kind of 'quality' if it would be my code for a customer which wanted 'cheapest possible crappy ad-hoc' implementation. Build something which mostly works in a few hours. Copy paste sample code from net and make extreme naive shoddy experimental piece of code which just works when everything is OK. But when production code from major corporation is just as bad. Well, it is. Nothing more to say. Restarting remote desktop service, throws out all active users. As well as terminates these lingering connections.
  • So guys, next time you're writing production server code, just copy paste something like this. Python http.server - Who needs nginx or anything else, when we've got full featured robust and attack resistant web server which we can simply use. Actually I've been planning to do exactly that. But only for a project which handles a small quantity of request from a trusted sources and IP addresses.
  • One important mobile user identification application by DNA, doesn't allow user changing personal PIN code at all. That's just absolutely wonderful. There's no way to change PIN. Except than to terminate the contract agreement with customer service and then re-enabling it with new PIN. I'm not talking about lost PIN code recovery. I'm talking about changing known PIN code.
  • Even after double and triple checking, situation remains the same: For some reason discard doesn't seem to be working for my SSD with ext4. Funnily it works great with vfat on same drive. Should I see discard on #mount option row when checking what mount says? I would assume it should read there. What's the best way to verify that discard is actually active? I did see tons of guides with mostly only bad hints and incorrect ways of checking it.
  • Python 3.6.0 standard library hashlib also includes scrypt, blake2, shake and SHA-3 aka Keccak - Awesome - It's very important to have modern and compatible tools for key derivation, password protection & data hashing. Dupe from previous post, but doesn't matter. I studied and played quite a lot with that stuff.
  • Some security / design flaws are just so devastatingly horrible that those can't be even mentioned. - So I'll shut up. - But these are really serious. - Let's hope those get fixed, but I'm highly skeptical.
  • Also found out tons of basic stuff, like using default credentials. Which basically means that key business data is not protected at all. But actually, nobody cares, or gives a bleep. And this is the norm, most of companies actually got. So no news here. At least there is authentication, even if it requires attacker to guess the default credentials.
  • Excellent article: Mental Models I Find repeatedly Useful - This article covers many many models, which I've been talking about as well as plenty which I haven't. Especially liked the Deciding section: business case, opportunity cost, intuition, local vs global optimum, decision trees, sunk cost, availability bias, confirmation bias, loss aversion. Yet naturally all of the listed items were familiar. Virtual team is something, I've been talking for decades, and often been a part of from early 90s. I like high-context documents. There are many things, which are 'obvious' and therefore doesn't need to be mentioned. Technical Debt, such a classic. Unfortunately it's often hard or nearly impossible not to end up collecting (lots) of technical debt. It's a constant struggle. Zawinski's Law, hmm. Uncomftable laugh. Metcalfe’s Law aka Network Effect. Classics, MVP, Product/Market Fit. "First-mover advantage vs First-mover disadvantage", that's a very good question.

1-10 of 508