My personal blog is about stuff I do, like and dislike. If you have any questions, feel free to contact. My views and opinions are naturally my own personal thoughts and do not represent my employer or any other organizations.

[ Full list of blog posts ]

Endianness, Credentials, PSD2, Hosting, Boring Stack = Happiness, Version Control

posted by Sami Lehtinen   [ updated ]

  • Had to read one annoying old binary format. Enjoyed tuning with endianness. World is so full of all kind of strange joys.
  • Never ending discussions is using globally available services with default credentials is a good idea. Yes, you would think this is some kind of joke, but it isn't. It's business as usual.
  • It seems that more and more Finnish sites and services are hosted on UpCloud. I don't wonder that at all. They provide great bang for buck. If you just look at basic stats and monthly fees, those might seem bit expensive. But when you compare practical performance, the cost ratio is also great. As we all know, VPS performance can be really flaky on many service providers. With UpCloud you don't get into that trouble at all. It's just rock solid performance.
  • Studied EU's PSD2 FAQ document - It's awesome that banks are forced to provide common APIs. It has been traditionally painful to deal with banks, because each bank had it's own standards. At least in this sense EU has been getting lot of good things done. For POS vendors this is highly beneficial thing. We used to have direct access to banks, but since PCI-DSS we've lost it. New direct payments and third party payment service providers (TPPs, PISP, AISP, XS2A) will allow access again. Payment Initiation Services have been very popular in Finland for a long time. kw: SEPA, fintech, digital banking, open bank, open data, my data, direct payments, Payments Services Directive 2, AIS, PIS, Strong Customer Authentication (SCA), European Banking AUthority (EBA). - All this will also bring the basic question up again which I've been always wondering. Many organizations alway seem to think that payment = credit card payment. I've been like, "Why? what's the point of always dealing with credit card which adds extra costs." Many mobile payments platforms are just digital extensions of existing credit cards which is kind of funny.
  • Scaleway - is adding new data center(s). That's nice. So you can have multiple availability zones with one provider. Yet I usually prefer using independent providers. Trusting everything on single provider doesn't mitigate all risks. Yet funnily their routes from Helsinki to Amsterdam loop via Paris. That doesn't make sense at all. Bad bad routing.
  • Ceph performance issues finally resolved. A few wrong values in configuration caused all the pain. But it took quite a while to realize what was behind that latency. That's all it takes to make system 200x slower than what it's supposed and expected to be. Average read speed for cold data remains still at around 15 MB/s as well as average I/O latency is around 40 ms. Which is quite much. Reason for the problems was configuration which caused hot spots and lead to extreme latencies on some I/O requests.
  • Nice post IPv6 Performance Revisited - Nothing to add. I'm sure some of the problems will be solved when IPv6 will be mainstream.
  • Firefox Quantum engine - Nice. Finally. I've been annoyed by slow sluggish and hanging Firefox for a long time.  Wide use of Rust, nice. See Quantum Wiki.
  • Happiness is a boring stack - That's funny, but something I completely agree about. Yes, I haven't included all the latest junk in my project. As well as I don't use 10 different shared, high, performance, cluster, queue, storage, distributed projects with it. I just make really boring and simple stack and only add something new if I really have to. Each new added technology will just make everything more complex, harder to maintain, and there's considerable learning curve where you just end up causing more trouble than solving it. That being said. business critical applications are best being kept really boring. And that's awesome! If you love having 'interesting problems' which will bring the production system down every now and then, or cause extreme performance degradation. Go for it. But I just really happen to like boring, aka working systems.
  • It's very important to have version control for all kind of stuff. But even more important it's for important stuff, like credentials repository. I had incident where credentials were lost from latest version for unknown reason. Had to restore old version and fetch part of the credentials back. Phew, that could have been a very bad day. But proper version control managed to save me.

Secure Environments, WinSxS / Dism Cleanup, VPS, Project Management, Toshiba HDD

posted Jan 15, 2017, 2:19 AM by Sami Lehtinen   [ updated Jan 15, 2017, 2:20 AM ]

  • Sometimes I just wonder what engineers are thinking. As example, why they install in highly secure production environment stuff like AskToolBar aka Teoma Media Search App and then open up firewall for it to be globally accessible. I've got just a few options: A) Yes, average operations engineer is just really that brain dead. They're so stupid they can't be even held accountable. Yet, question remains, why would anyone ever hire such people? B) They're on purpose sabotaging production systems and endangering security, trolling everyone and laughing. C) Something else? It's bit hard to come to any conclusion other than A or B. - This is yet another example of standard system security. Someone could imagine this is a bad joke. But it isn'. It's just how things often are.
  • Here's the essential way to reduce Windows 2012 R2 disk space consumption by repacking WinSxS folder content and practically removing previous versions of files. Warning, it takes quite a while to run and the next boot can take a long time. So make sure there's suitable window for next reboot to take up to a few hours.
    dism /Online /Cleanup-Image /StartComponentCleanup /ResetBase
  • Tested some VPS platforms. Virtual Machine Manager by Microsoft (Hyper-V) and compared results with other platforms. Making comparisons is quite hard, unless you've got the "real word" load on servers. All kind of testing is 100% debatable if it's faster or not. As example on this tiered storage system the Hyper-V allows writing to storage up to 1 GB/s speed. For a while, after that it grinds to complete halt and after quite long pause resumes at 1 MB/s and finally to under 100 KB/s. Is that fast or not? It depends, if you won't ever run out of that "burst zone" then the actual application speed can be high. On the other platform the write speed was pretty consistent 155 MB/s and it seemed 'really slow' to begin with. But in reality, it turned out to be faster than the 1 GB/s when writing data for extended periods. One platform got faster disk & network, but much slower CPU and some got really fast CPU but slow storage. Some and so on. Totally normal. Worst part of public cloud is that these values can be instance and time specific. If I now test something and conclude it's good or bad, it doesn't mean that the results would be same after one week or month or year. Sigh.
  • We all know how software projects are managed. Someone puts black hood on head, and does something. When the "ready date comes" the result is announced to be ready. That's also the first time the customer sees the result. I've seen projects like office renovation being done in that way, and usually the results aren't great, not even good. But that's the way. Initial design and execution are both extremely poor. Methods where feedback is given during design and negotiated during practical execution, as well the project schedule being monitored with the progress... That's something never heard. Everyone's just blindly expecting spectacular results without any feedback. The perfection just miraculously pops up from somewhere. - Yet that rarely happens, but at least often people are hoping for such miracle. It would be nice if it would be like that, I personally would wish for it too.
  • Friend suffered from similar Toshiba HDD failure than I did earlier. Tests run on the drive gave also similar results. So smart data is just one big lie. My tests - I didn't ask full report from my friend, but smart data was all good, but Windows was crashing all the time and badblocks -nsfv showed tons of bad sectors. So typical. Not telling that the drive is broken just makes consumers suffer more because they might do something silly like get a new laptop or waste a lot of time re-installing windows, because they just don't get that the darn HD is broken. Similarly they might lose data, because drive degenerates and corrupts data, but still doesn't clearly say. Drive is going to die soon, copy everything you still can and run to get a new drive. - Thanks

Double bind, I/O latency, /tmp tmpfs, Social loafing, WhatsApp, Social Credit

posted Jan 15, 2017, 2:16 AM by Sami Lehtinen   [ updated Jan 15, 2017, 2:27 AM ]

  • Double bind - Aah, excellent topic. World is just so full of double binds. Every project and almost every task contains major contradictions how things should be done. This topic is so highly depressing, I'm not even starting about it. This is a subject for several books. - Also the new Finnish Spy law will be naturally full of this stuff. You must do it, but you must not do it, etc. Because utilizing this right would violate rights of others etc. - Actually this is quite closely related to the 'everything is a trade-off' thinking. There's no absolute right / correct solution to many problems.
  • Wrote a small test application which measures time of 4k file creation and stores it. This is perfect method for monitoring underlying storage systems performance. As well as doing random 4k reads (slowly) in the space of whole storage system. This has been very beneficial. It's easy to see that small number of storage writes seem to be extremely low, compared to average. But as said, this shouldn't be anything new to storage / performance / cloud people. Generally it works, but at times, it's just extremely slow. Like 1000000 times slower than it's supposed to be. This can be caused of course by multiple reasons. I personally think that the primary source has some kind of problem, and it causes timeout and then operation is tried on some other source, which might as well timeout. Leading to cascading of timeouts, which leads to extremely low compared to normal operation.
  • Run base line tests on HDD and SSD storages as well as on multiple cloud storages. It seems that with Cloud Storage it's highly likely you'll be getting consumer spinning rust kind of performance. Instead of screaming fast SSD / Enterprise server performance. - Of course everyone's expecting that cheap systems would be extremely fast enterprise systems, right? - At least I'm not expecting that. But still, latencies need to be reasonable. As well as single thread blocking programs severely suffer if I/O latencies grow.
  • Fsyncing on /tmp still writes to underlying storage. I actually didn't know that. I thought it would be RAM stored, and only if there's memory pressure it would get flushed to underlying storage. Yet, after I examined this stuff bit more on the test computer. I did find the reason. I assumed /tmp/ would use tmpfs, but it doesn't on Debian & Ubuntu. So that's the reason why it still wrote to disk. Changed fstab so that /tmp/ is now stored on tmpfs. /etc/fstab "tmpfs /tmp tmpfs nodev,nosuid,noexec,nodiratime,size=1G,mode=1700   0 0" Unsurprisingly the tmpfs based /tmp is also a bit faster.
  • Endless discussions about storage latency. Now they claim that the test software is reason for the latency. That's BS. I always like to check affecting factors and I can tell that on ramdisk the 4k file creation latency is around 0.6 ms on my desktop. So anything above 1 ms is guaranteed to be caused by the storage subsystem.
  • Wanted to watch the ExoMars and TGO & Schiaparelli landing via live stream. But the stream kept hanging all the time. Sigh. So typical, we live stream this and then the live stream won't work.
  • Social loafing - Just so normal. Everybody thinks that somebody will take care of it. Haha, so classic and traditional.
  • EFF's article about WhatsApp security. Interestingly they had exactly the same points I've mentioned earlier. Having top notch all the time changed encryption keys is totally meaningless, if there are multiple circumvention methods left in place.
  • In some earlier posts I've said that trust and reputation is always context related. But in China they seem to think that the trust can be widened. Generally if you're 'being trusted and keep your word' it usually means that you could be trusted on other aspects too? Or does it? It depends. Arguments for both directions can be made and of course examples found. But this is what China is planning a National 'trust / social credit' score. In a way that's good, because now breaching trust on some level, will also cause issues on others. Which will make you less likely to want breach trust in cases where you think it just won't matter even if I don't keep my word or do the right thing. In many important cases the generic integrity of your word is counted more important than what you actually did. If they ask if you have done something you promised to do. Well, of course it's bad if you haven't. But if you then lie about it, it's even worse. Then it's immediately clear that you can't be trusted on even really small things. So how you could be trusted on more important things? In Finland trust and reputation is highly valued. In other countries on cultures it seems that it doesnt't matter, if you got really bad reputation. Yet continuous lack of trust makes many things really hard to organize. Isn't it great if you can be sure that the counter party got probity a complete and confirmed integrity. It's much better than just a plain credit score. In Finland people don't have credit score. Everyone's "trusted by default", you'll only lose your credit score if you screw up. It's like having or not having a criminal record. Even if we know that practically everyone has broken the law. Of course if that kind of system is being abused, it would be pretty horrible. It's like the double construction cases. If the 'military intelligence' spots something which isn't right, but isn't about military, should they rat on it? Isn't that still abuse of power, or is it just for generic greater good? Don't know, there are hard things to make right conclusions about. Finland it's right now crafting laws about these issues. Do you need privacy, if you don't do anything wrong? If the lack of privacy is handled correctly and not abused, does it matter and so on. If we accept that 'military intelligence' or 'state security agency' or whatever, got basically full access to everything. Is that a problem? It shouldn't be or it could be? Who knows.

Security Approach, Legal Snooping, Privacy, Finland, Slow Storage, Tiered Hybrid Storage

posted Jan 8, 2017, 7:35 PM by Sami Lehtinen   [ updated Jan 8, 2017, 7:36 PM ]

  • It's interesting to notice how different security approaches different hotels got. I were in Spain, where there was a guy in reception complaining that everything from his room has been stolen, including passport etc, and he needed to call embassy and police. Only thing the reception was interested about, was if the guy can pay for the calls. - Points for that, very nice approach! - In Miami situation was quite similar, also the hotel room was broken. The guy in the reception was mostly interested who's going to pay for the broken door. - Then bit different approach in Hong Kong. We were really jet lagged so we wake up pretty late and just got up and went for breakfast. I guess the room was quite a mess when we just left shortly. When we were having breakfast, the hotel security called us and asked that we should return to the room because they suspect it might have been searched and stuff stolen. Yes, that's the way to do it. Yet I felt pretty ashamed when I told that nope, it's totally normal mess and you don't need to worry about it. I guess the cleaning stuff noticed that the room wasn't like it's usually supposed to be and contacted security instead of cleaning the room. - That's awesome, even if personally I wasn't very proud about the state we left the room in. We were just going to get back after the breakfast. As afterthought I should have left a very nice tip for that action. - I guess there are many more interesting travel stories out there to be told by people whom travel more than I do.
  • Read lot of discussion about Finnish new snooping law, which would allow intercepting Internet traffic and hacking & spying in Finland, as well as outside Finland when it seems necessary. Officially this is of course called reconnaissance / intelligence law. Of course Finland wasnt's NSA like spying capabilities because other states have shown what kind of benefits those can provide. There's nothing surprising about that. Maybe it has been done before that, but it has been against the constitution law. Of course law can be circumvented when situation demands it by entities "above the law". But in general, it's illegal. One reason for this legalization is protection of intelligence employees. Currently they 'might' be doing illegal stuff, and that isn't great if they happen to get prosecuted for that. Of course the norm is that it's being watched through the fingers, but you'll never know what can happen. In that case, it's highly likely that they claim that "this individual person" broke the law by doing illegal operations on his own, and we don't know anything at all about it. - Not great or cool at all, if it was organized by the chain of command. - But that might happen. - It would be less bad, than they just admitting that the whole organization has been breaking the constitution all the time. - Related news link
  • I think I've mentioned this earlier. But Finland is crafting legalization about Internet Snooping and Spying right now. It's interesting to see if we're going to see cases like this in future. UK spy agencies broke privacy rules. There are some references in past, which I'm not going to repeat, but shade strange light about Finnish legalization and how these things are handled in court.
  • Wondered how slow storage can be. Hitting just a very few slow I/O transactions will completely ruin system performance. As people should know, average performance is very bad measure. 99th percentile is much better. In this case 99th percentile is 1000 ms, but worst times are over 30000 milliseconds and median is 47 milliseconds and average is 10 milliseconds. - Using values like average it's very easy to forget that there are some extremely slow ops there. But this shouldn't be any kind of surprise with normal distribution to anyone.
  • It seems that almost all service providers are now offering tiered hybrid storage systems, which combine SSDs and HDDs to provide performance and capacity with reasonable price.

Cloudflare CDN + API, Self-hosting, Technology, Thunderbird, NTFS Extents Write-Amplification

posted Jan 8, 2017, 7:31 PM by Sami Lehtinen   [ updated Jan 8, 2017, 7:32 PM ]

Had long reasoning with a friend who's running a tech startup, about if it's better to connect from client directly to server's API or if API requests should pass via CDN. - That's a very good question. I'm afraid that there's no simple or right answer to that question. There are so many factors affecting the question. But here's my reasoning:
  • One project I've been developing provides JSON RESTful API. After long testing and experimenting I decided to pass the API traffic through Cloudflare CDN.
  • Main reason for that is DDoS protection and the fact that the API latency doesn't really matter. Because that API is low traffic and it's bot which call the API so there isn't even user waiting for instant interactive response.
Other factors which also affect this discussion are:
  • HTTP/2 (H2) - It depends from API usage, if it's utilizing H2 and efficiently using existing connections. If the API users are mostly bots and other backend systems, those can keep a few H2 connections open all the time and pass asynchronous traffic. In this case it's probable that CDN will make the performance worse.
  • Mobile Devices & Packet Loss - On mobile devices high latency and possible packet loss turns the table around. Now it's important that potential packet loss is detected quickly, and using CDN probably will make the end user experience better. It's also a great question how smart the CDN is. If the CDN does then maintain a few H2 connections to the source server and aggregates traffic from clients using this path, then the CDN can provide great benefit in over all latency and experience. But this is a extremely complex question and requires experimenting and fine tuning, a lot probably.
  • API response cacheability - Are the API calls / responses such that those can be cached? If so, CDN should provide great benefits. Let's say we have a service which gives you bitcoin exchange rate which is updated every minute or so. All the calls from clients to the API during that minute can get the cached response, that's great.
  • TLS / SSL - It's nice if the CDN can handle the TLS/SSL connection, which requires a few round trips at least with older implementations. I think the zero round trip implementation is coming, but it's only for repeated connections as well as clients which are smart enough to utilize it. Problem with these optimizations are that many of these are very complex. Which means that many won't be using those. It's like any other complex optimization, it's done, when it's absolutely required. Doing it from the very beginning can divert resources from more important things and is just a premature optimization. Of course using a suitable library can help, which handles all the complexity seamlessly.
  • DDoS protection - If you're trusting CDN for DDoS protection and don't want to reveal key subsystem IP addresses, then passing through CDN is great idea. Without CDN even if you would use load balancer outside the backend's address space, DDoS at least will take down the API even if rest of site would remain running.
Afaik, these are the most important key points we discussed, I'm sure there's plenty of other aspects. This is a deep topic.

Other stuff just gathered with this post:
  • Friends self-hosted email server had extensive down time. Yeah, that happens. That's one of the reasons why I actually stopped running my own mail server, even if I naturally could. But as we all know, it does break down just when you are unavailable to do anything about it, even if you normally could react swifly and get it fixed in no time, or even completely reinstalled in a few hours.
  • Some technology seems to go forward. But some won't. It seems that new washing machines got generally much louder electric engines than the old ones. It's kind of annoying.
  • Thunderbird, bad code, bad code. It seems that the IMAP message downloading code is broken. It hangs forever while fetching message from mail folder. I've been wondering why some messages are downloaded in real time, when I've got download option enabled. But the reason seems to be the fact that the downloader isn't working. I tried to look in configuration for downloader timeout, but there isn't such option. Great, just great. So message downloader will remain stuck, until Thunderbird is restarted. - Seems to confirm the software industry norm, broken software is the norm. If something actually works, it's rare exception. And you've probably been just very lucky.
  • Also noticed that this blog post entry is now over 4k bytes long. And being stored in two separate NTFS extents. Now when I save this file, with write amplification. It'll mean that at least 16 megabytes is written to the flash. Neat. No wonder it takes a while. No, size alone doesn't make it to be stored like that, but just on this file system and with this file and current allocation situation it seems to be split.

2016 topic dump continued (3/3)

posted Dec 30, 2016, 6:48 AM by Sami Lehtinen   [ updated Dec 30, 2016, 6:48 AM ]

  • Cloudflare HTTP/2 demo - and under the hood technology description.
  • Using Google's Physical Web URL Eddystone beacons (Bluetooth Low Energy, BLE) to spread geographically targeted malware?
  • Had some issues with Peewee ORM doing data aggregation, but unfortunately I don't remember anymore the exact issues.
  • Login forms over HTTPS - That's obvious. Nothing new there.
  • Important URLs for all Google users - Quick links to manage your privacy, security and history settings.
  • Europol Public Awareness and Crime Prevention Guides
  • Watched a great documentary series about cyber fraud and crime.
  • Let's Encrypt leaked email addresses. - It's funny how services which are security oriented can leak user data. So what would you expect from 'normal' non-security oriented sites?
  • Watched a long documentary / lesson series about deep learning. Very easy or hard, depends about the level you're diving into it.
  • NIST Special Publication 800-63B Digital Authentication Guideline - Well, as said. SMS is insecure way of authenticating people.
  • Studied some Skein and Threefish related source code and Tweakable Authenticated Encryption (TAE) mode.
  • It's so easy to be blind to own mistakes, especially if quickly iterating between different versions. Some of the documentation had painful inconsistencies, which most likely are caused by the developer changing his/her mind on something, and only partially updating documentation / source comments. Ugh! Yet, I'm personally very guilty to the same fails quite often.
  • Another thing is hastily written text where there are blatant typos / spelling errors. Ouch, that hurts. It's just like by blog. I'm just dumping stuff, not writing a book.
  • HTTP/2 might be vulnerable to Slow Read, HPACK Bomb, Dependency Cycle Attack and Stream Multiplexing Abuse. Unless the server application is hardened against these attacks.
  • Quickly played with Redmine. But I don't think I've got enough time to try all the project management tools out there properly.
  • Mobile Privacy & Security: It's only safe to assume that nothing you do on Android phone would be actually private.
  • Schneier IoT Security - Nothing but the truth. Security will suck, and it will suck even more!
  • IoT privacy. It will be interesting to find out how much data all the IoT devices will be storing and who's going to get access to that data and based on what, etc. But this is so hot topic, that it's certainly guaranteed that we will find it out sooner or later.
  • DBaaS is coming more and more popular. Yet it's not nearly suitable for all use cases. One of the major issues is latency. This means that there are many applications where DBaaS can't be used, because the Application Server and the database needs to run multiple queries and make processing. And then deliver final data to the client. I've seen "chatty apps", which start to get slow or extremely slow as soon as the database latency gets above 1 ms. Yet of course this is nothing new. If it's possible to get something simple, like whole document based on key, then it's great. Yet any REST API would do it, instead of using DB protocols.
  • Absolutely awesome article about Data Projects failing. Been there, done that.
  • Quicky checked SAS language - Yet I don't see any personal use for it. Skipping.
  • I don't understand Python AsyncIO. Neat. Could we make this bit more complex plz? One of the things I love about Python. Is being quite simple and straight foward. This doesn't look anything like it.
  • Found out a few interesting ways to generate indirect spam for marketing purposes. Hah, it can be trivially and fully automated. But because it works well, can't tell more about it here.

2016 topic dump continued (2/3)

posted Dec 30, 2016, 6:47 AM by Sami Lehtinen   [ updated Dec 30, 2016, 6:48 AM ]

  • A good rule of thumb for a CONSUMER is three copies of your data: 1) primary, 2) on-site backup, and 3) off-site backup. If you are a business that will lose millions of dollars if a programmer makes a mistake or an IT guy is disgruntled, add 4) another off-site backup with a totally different vendor that doesn't share a single line of code with 1-3 and has separate passwords. As well as maintain proper versioning and history for long enough time
  • As confirmed in my blog earlier, C-Lion cable saves around 7 milliseconds to Central Europe from Finland. Now latency from data center to data center is around 20 ms. (C-Lion1, Sea Lion, Cinia)
  • NxtVn announced building huge data center park in Finland. But no news since this initial announcement? Yet NxtVn web page gives impression that the data center park would be up'n'running. Interesting. It's bit funny that if they operate 'world class data center', their web site is still server by GoDaddy.
  • Read: Learning from the enemy: The GUNMAN Project. (NSA publication - Center for Cryptologic History National Security Agency)
  • Bing being replaced with other search engine alternatives in Windows 10? This is interesting development. I'm curious what consideration and decisions have lead to that? Maybe other search engines are paying so well, it doesn't just make sense to push bing? Or maybe Bing is just so bad it doesn't make sense to offer customers sub par experience? Or maybe the other search engines receive much more ad money and therefore it's just more profitable to bring money in via that than using own solution?
  • Read: More Concurrency: Improved Locking In PostgreSQL.
  • Read & Thought: Disinformation & Counter information & Propaganda, using false information as weapon. Black Propaganda.
  • XM25 Counter Defilade Target Engagement System, Thermobaric Weapon - FN P90 - TERCOM DSMAC - Barracuda Submarine - Shaped Charge - Boeing_P-8_Poseidon - Advanced Airbone Sensor
  • I'm really sick'n'tired of MS software quality. It seems that their firewall isn't reliable, technically it's working, yet it can open key ports without any warning. So you really can't trust it as well as it requires constant monitoring. Well, monitoring is always good anyway. - Another thing which really annoys me is the RDP remote desktop protocol reliability issues. It just is utter crap and causes constant burden, suffering and pain. Can I sue Microsoft for causing extended discomfort and mental strain?
  • Python vs R: head to head data analysis - Neat, confirmed that I've got no need for R either.
  • Efficient Use of Asynchronous Operations in Google App Engine - More async & concurrency. 
  • Checked out Onename, which is yet another naming scheme using bitcoin blockchain to store claim data. Similar to Namecoin, but uses Bitcoin chain instead.
  • lost some data. Again. This seems to happen to them, over and over again. All confidence on it and their cloud services should be totally lost. Anyone doing business with them in future should feel foolish. Yet that's very unfortunate for businesses running on-line services. If your service goes down, you could lose all of your user base really easily to competitors. Yet in this particular case, it's the only sane way to do.
  • I'm just wondering if Facebook, Google or some other major site is able to **** up things so badly that they end up in the same situation. Use our cloud service, your data is safe forever. Oops, we just lost it and you can't access it anymore. I believe most of people don't keep adequate local / alternate service / backups.
  • Hands up - How many of you got full backup of your Google / Gmail / Facebook data? Do you update it frequently enough? - I actually DO. But I don't know many whom got it all saved up.
  • Digitalization would allow storing everything in Phone. Keys, Wallet, Travel documents, Identification, etc.
  • Finland wants to be in lead of drone development and utilization in commercial applications.
  • Read: An Empirical Analysis of Email Delivery Security and PostgreSQL Parallel Sequential Scan is Committed!

2016 topic dump continued (1/3)

posted Dec 30, 2016, 6:33 AM by Sami Lehtinen   [ updated Dec 30, 2016, 6:34 AM ]

This is end of 2016 dump post. Some of the links and stuff is quite old. It seems that I had backlog going back for 18+ months. Now I'm dumping almost everything from the backlog, out of order with optional short comments. 
  • Discussions about CDN optimizations, server locations. With one global provider. Why Finland would be a good location for POP. How's Finnish ICT skilled staff would help company. How great Finnish Startup scene is, etc
  • Cloudflare Keep-Alive - Nice post about HTTP keep-alive benefits.
  • OneWeb satellite constellation - Let's see if this project ever realise.
  • SourceForge & Malware, classic theme. Anyway, users should know what they're downloading. Any unexpected payload is very bad. Some sources say NotePad++ download would be hosted by GitHub, but now it seems to be hosted by OVH.
  • When Solid State Drives are not that solid - No news. Everything fails.
  • One colleague had IPv6 problems. It was the classic IPv6 ICMPv6 Type 130 issue. I'm sure many users are going to encounter it when configuring firewalls. See MLD. Also related Neighbor Discovery (ND), Neighbour Announcement (NA), Neighbor Discovery Protocol (NDP)
  • Lot's of Lets Encrypt stuff.
  • Some privacy issues with Google Chrome Audio Search. Privacy Defense in Depth.
  • Vega Rocket
  • IPv6 Explicit Cognestion Notification (ECN).
  • Cloudflare how to achieve low latency - A very nice post. Yet requires more tuning than what most people are willing to do. Great for nerds though.
  • Computers leak encryption keys on RF spectrum (RadioExp). Well well, TEMPEST is old concept.
  • Azure Chaos Engineering - Yep. Everything should work, even if random things happen. Then you can call it actually robust.
  • Channel Tunnel - Wasn't great investment. Who would make tunnel between Helsinki and Tallinn would make any sense?
  • WiFi DSSS - FHSS - OFDM - Chirp Modulation
  • Blog posts can be always signed with anonymous public key. In that case it's trivial to prove that you're the author if that's required. You can also have strong pseudonymous identity.
  • Akamai HTTP/2 demo
  • HTTPS, SPDY and HTTP/2 performance comparison
  • Once again advanced remote video surveillance turned out to be beneficial. It's always good to be trained and prepared on both physical and on cyber security. These solutions can help you to maintain privacy and security of premises and also to give an advance warning if something is happening / going to happen.
  • Discussions about web shop and integrated multichannel customer experience. As well as how to automate customer product recommendations based on purchase and browsing history. Size and favorite brand and style information. As well as possibility to track customers inside store and automatically identifying them using several different technologies.
  • Finvoice, ARTS, XBRL, e-Receipts.
  • Telegram end-to-end (e2e) encryption explained. - Yet, Infinite garble extension (IGE) is still something which of many really do not like.
  • A lot more discussions about e-Estonia and keeping UK businesses in EU market area. As well as running light virtual company in Estonia cheaply, etc.
  • Cloud, containers, big data, data analytics & insights, machine learning, IoT, beacons, mobile payments, everything is a API, going green.

Access Control, It Works, Random or Trend, Business Analytics (BI), Consistenty vs Availability

posted Dec 25, 2016, 1:50 AM by Sami Lehtinen   [ updated Dec 25, 2016, 1:51 AM ]

  • Access Tokens and reality. It's so funny when people claim that electronic access tokens or phone authentication makes things more secure. Truth is that people often think that they'll find it later when they lose it. Based on this fact, it's common that lost access token or even phone might take several days to be reported. If access to key systems is access controlled by those measures it leaves just plenty of time to abuse those systems. So yet another massive security fail. It's also usual that in these kind of situations they might gladly borrow co-workers access token or credentials to access systems, because their own credentials / token / phone is just 'temporarily lost'. - Yet another reason why so many security system claims are totally false. As widely reported, it might well be that lost tokens remain active for several years or new ones will be issued without invalidating the old ones at some point much later. - Yet in case of physical keys are being used, it's even more likely they won't report it. Because it would know much more trouble than just losing electronic tag which is "seemingly easy" to disable. Even if it might take long time still, as reported earlier. - Just interesting security culture observations. I think it's more like norm instead of exception. I think it's rare exception if these things are handled properly and promptly. Usually people avoid telling such news. The story goes on. It's rumored that the PIN code required with that access token was written on the access token itself. Ah god, I love humans. They're smart, I mean ruining all of the security that the system has been developed and planned with.
  • I've mentioned sometimes et Jugaad is one fine invention. Chinese seem to use Chabuduo and I'm not exactly sure if there's a single word in Finnish which would mean the same thing. Good suggestions anyone? I'm not meaning something would be done very badly. It's done in a way, that's hackings and will work well enough to do what's required, but is far from pretty or great solution. On the same time you can ask, what's the true value of doing something better than required. Isn't it waste of resources? See: Chà bu duō  (差不多), System D, Jugaad
  • Google starts to provide VPS services in Finland, that's only a small part of the Google Cloud Platform (GCP).
  • Read some stuff about enterprise and data center SSD usage and endurance estimation. Related keywords: SSD wear out, Random Fail, Write Operations per Second (WOPS), Drive Fill Ratio, Drive Writes per Day (DWPD).
  • Read: Differentiable neural computers and Neural Turing Machine
  • Had once again a long discussion about alerts and warning to users. Alerts and warnings are very important, but those must not cause alarm fatigue. It's important that there's basic sanity checks and only if those fail the warning is being shown.
  • Another question was that, if critical near miss of huge accident will lead to learning and avoidance of the problem in future. Or if it's actually just indication of a trend. It's very hard to say. I'm inclining it's indication of trend. Having really bad mishap can lead to short term reversal, but based on practical experience I believe in trends. User who often does something catastrophically wrong, disregards security aspects, forgets password or anything similar, will do it again in future. Even if you would scold them very hard about what happened. They're not doing it on purpose or intentionally, it's just what they do and they'll do it again.
  • Even more discussion about IoT security disasters and the only thing guaranteed, it's going to get much worse before, it might get better in future.
  • Read too many articles about BI and Business Analytics Systems and also refreshed my memory about Data Loss, Disaster Recovery And System Crashing matters, by reading articles for several hours. Yet as said, nothing new in either sectors.
  • A great post by Julia Evans - Consistency vs Availability - Basic stuff, but yes, that's how it is. Even if in reality things are really shades of gray. CAP theorem is related.

Cloud Service Level, Traceability, TLS Nonce, OVH @ US, Windows Server 2016, PyPy3, Bitbucket

posted Dec 25, 2016, 1:42 AM by Sami Lehtinen   [ updated Dec 25, 2016, 1:43 AM ]

  • Typical cloud service mode is that you'll start with good service and performance. Then gradually worsen the service after trapping customers. That's why vendor lock-in is so important. They can provide absolutely crappy service and still keep charging, because cost of changing vendors / platform is annoyingly high. - This also applies to many other businesses other than cloud services. But in this case it's blatantly true. - Even hardware vendors to this. They firt release model SupaSpeed600. When it receives rave reviews, they'll replace it with model SupaSpeed600(b) which is very slightly cheaper, but actually performs at least 50% worse. This happens all the time. Some times they try to hide the plan B really hard. It might be just some extremely small change in revision or something. Yet customers thinking they'll get the fast model get just screwed. - This is also one of the reasons why vendor lock-in is extremely bad thing. They can just keep cashing and almost blackmailing you, if you don't have any viable options.
  • More fun with 'traceability'. Now I'm being asked to remove 'duplicates' from database. That's freaking awesome. But how do we know if something is a duplicate or not if it doesn't have unique consistent tracking id? - It doesnt. -> We can't know for sure if something is a duplicate or not. - That's freaking awesome. - If duplicates aren't removed, there's too many records, but if "duplicates" are removed, it's very highly likely that some data is being detected as duplicate even if it isn't. - How to fix this mess? Well, it's very hard to fix this mess afterwards. Only way to get this kind of stuff would not to mess up the data in the very first place. Best way would be to delete all data and reload it from the trusted source(s) where there are no dupes.
  • Studied more Ceph documentation to understand better how it works. Including: RADOS Block Storage / RADOS Block Device (RBD) related configuration and NVMe caching.
  • Cloudflare TLS nonce-nse - This article also shows how easy it's fail crypto and how many ways there are implement seemingly similar solution. KW: Initalization Vector, Encryption, Message, Key, Cipher, Nonce, TLS, RC4, CBC, GCM, ChaCha20-Poly1305, BEAST.
  • How OVH expand in the USA while preserving its European identity - A very nice article by OVH.
  • Checked out Windows Server 2016 feature list. All the Hyper-V, Containers, Virtual TPM, JEA, Shielded VMs, Identity Management stuff isn't really viable. It's great. But personally I don't see use for it right now. Storage Spaces Direct seems to be something which is actually nice for small scale operations. Server containers sound something which could be highly interesting. I've been complaining about Windows Licensing costs with Virtualization, which is usually more expensive than the hardware per month. As well as the huge over head caused by running bloat ware called Windows. Licensing + Overhead make Windows cloud operations highly inefficient. That's why Nano Server image is very interesting. Docker support is also very neat.
  • PyPy3 5.5.0 released, but it's not yet plug-in replacement for Python 3. I wish it would be. I don't have real performance issue with Python, so I won't be using PyPy3 as long as it requires 'extra work' to get it working.
  • Studied Bitbucket's new features - Pipelines, Large File Storage (LFS), Smart Mirroring and Merge checks. I've been using Bitbucket for my personal projects with great success. Continuous Delivery using Bitbucket Pipelines. The LFS implementation is pretty obvious and very basic, but it works. I've done basically the very same without using LFS. By keeping file hash references in flat files (hex) but not storing the file data itself inside the git repository. For replication purposes and automated replication those could be torrent magnet links, or any content based addressing systems keys. Like IPFS, Freenet, etc, list goes on.

1-10 of 450