My personal blog about stuff I do, like and I'am interested in. If you have any questions, feel free to mail me! My views and opinions are naturally my own and do not represent anyone else or other organizations.

[ Full list of blog posts ]

Aurora, Ethereum, Tor, 3DXM, Security, TKIP/RC4, Hosting, OpSec, QUIC, FireChat

posted Aug 1, 2015, 9:45 PM by Sami Lehtinen   [ updated Aug 1, 2015, 9:45 PM ]

  • TeliaSonera (Sonera, Telia) confirmed that they'll be building Finland's largest data center at Pitäjänmäki. Which is open to customers for co-location, etc. It'll be located about 200 meters from where I'm currently working.
  • Checked out Amazon Aurora - Amazon's blog post about Amazon Aurora - Yet I don't have currently any use for it.
  • Checked out Ethereum and read Ethereum White Paper and Ethereum Developer Tutorial - I've been having bit similar thoughts about OpenBazaar and Smart Contracts.
  • Checked out Augur - A decentralized future of prediction markets? At one point I was very interested about prediction markets.
  • Checked out EtherX - A fully decentralized cryptocurrency exchange, of course based on Ethereum.
  • Checked out OnionCat - It's an IPv6 VPN over Tor or I2P network. Allowing location privacy and strong security and defeating IP spoofing. It's excellent too for maintaining anonymous servers on the net. Which are hard to track back to the real administrators.
  • Reminded my self about some Tor  stuff: node types, directory authority, guard, middle, exit, relays, fast, stable, hsdir, v2dir, valid, flags, consensus algorithm, authority operators, padding, 3des, cipher suites, AES, padding, cells, certs, authenticate, authorize, TAP, ntor, curve25519, ECDH, pluggable transports, signatures, usage statistics, qunite items, GeoIP digest, bandwidth and stream counts, keepalives, path selection, rendezvous point relay, attacker, probability, random, math, network and user traffic profiling, fingerprint attacks, traffic correlation and confirmation attacks, countermeasures, bandwidth scanner, load balancing, proportional-integral-derivative controller, bridges, censorship, recurring, obfs3, obfs4, scramblesuit, fte, meek, bananaphone, stegotorus, skypemorph, dust, dust2, dlopd, sshproxy, git, generates random bytes and traffic patterns, randomizes packet sizes & timings, Format Transforming Encryption (FTE), Deep Packet Inspection (DPI) evasion, Markov Chains, maps data to text, Stegotorous, splits data over multiple paths and makes those look like HTML/JS/PDF etc, collateral freedom (meek), Flash Proxies, faciliator, intermediator, middle man, hidden services, introduction points, random value, nonce, cookie, CTR, Public Key, Encrypted, ephemeral single-use public key, traffic correlation, recognize traffic signature, HTTPS Everywhere, NoScript, Reproducible Builds, protocol improvements, directory mirrors, Post-Quantum Key Exchange, revocation keys. Hidden Services 2.0 will implement new much longer .onion addresses, that's wonderful, ring location randomization, directory authority voting, correlation attacks, entrance traffic and exit traffic, dragnet data collection. It was a good read.
  • What was new to me in that latest Tor spec? A new longer addresses, Post-Quantum Key Exchange and BananaPhone were new stuff to me. Otherwise everything was pretty much old stuff and known or 'obvious' development, like kicking DHE and replacing it with ECC. It seems that I'll have to read this separately. Actually the BananaPhone was something I've been thinking about too. Hiding encrypted data into English text, so it's text Steganography.
  • I used hidden service to access some servers (administrative) at one point. But after all I felt it's not a good idea and dropped that project.
  • Just as general comment. Many of this tech stuff is getting really deep. Unless you'll study it continuously and update your information monthly, it can take months of years to catch up!
  • 3D Xpoint memory - Yet another storage layer to be added to multi-tiered storage system. So cpu registers, cpu cache (multiple layers, at least 2x), ram, ramdisk, xpoint memory, ssd, hdd. That's quite a chain of different technology layers for data to flow through.
  • Feeling so tired about how bad Microsoft Server operating systems are. Those got constant DoS (Denial of Service) issues with Remote Desktop Service / Protocol (RDS/RDP) and they're doing nothing to fix it. Issue has persisted for several years with multiple Windows Server versions. Extremely annoying, causing unplanned random system boots because Windows is just so much fail. - This is my personal honest opinion.
  • "No one can hack my mind" Comparing expert and non-expert security practices [PDF] - This is just so awesome. It clearly how differently experts vs normal users think about security.
  • "All Your Biases Belong To US" Breaking RC4 in WPA-TIKIP and TLS [PDF] - Excellet paper about Wifi (RC4/TKIP) hacking and fails of RC4 + IVs.
  • A few sites I want to share with you: and those are excellent information and news sources.
  • I did read more stuff about Docker and played a bit with it. Yet as said, I don't (yet) see any use for it. But it's just like ready virtualbox images, there might be use it for it. But it's probably not needed in daily user. As well as discussed issue with development, staging and production environment differences and how docker could help in that field.
  • Actually some of the interesting projects overlap nicely. Outernet (Satellites) will be complement and over lapped by Project Loon (Balloons) which will be overlapped by Titan platform (Plans) and when there's network coverage then by Facebook can be delivered to users for free. Awesome and nice. I completely agree with this stuff, if it's free, then you can't whine about net neutrality. If you want free access to all content, feel free to pay for it.
  • Studied Google QUIC Experiments [PDF] document - Providing 0 RTT and 1 RTT at times (~25%) connectivity. Also reminded my self about RENO and CUBIC differences. TCP congestion-avoidance algorithm - - Also reminded my self about TCP timestamps and PAWS (TCP Sequence number wrapping).
  • Played a little with crunch, airmon-g, airodump-ng, aircrack-ng, reaver and other standard WIFI / WLAN hacking & cracking stuff.
  • Studied High-speed Onion Routing at the Network Layer (HORNET) [PDF] - After quick reading, some of the claims sound bit far fetched without technical proof. Also it doesn't protect against the confirmation and correllation attacks, duh. As well as 'high speed' is more linked to node speed than the actual platform. Coded in Python, hmm. Isn't that 'computationally' expensive? It depends, there's so many things you could speculate about based on that paper alone.
  • Have been doing some comparisons between OVH, Hetzner, UpCloud, Sigmatic and Capnova about hosting solutions. I'll write more about this bit later. Google Compute Engine (GCE) also offers three zones in St. Ghislain, Belgium.
  • Had long discussions with friends how beneficial IPv6 is compared to IPv4. Without NAT there's no more need for  constant keep-alive traffic things work as they were supposed to work, before Internet got broken. True stateless connectivity available and so on. That's wonderful!
  • OpSec is really hard for most of people. It's practically impossible to get them to follow any reasonable OpSec procedures. As example: What kind of moron first creates a message draft on Gmail, writes it there. Then encrypts it using PGP and send it? Aww double Aww... Didn't he/she realize that the Gmail is going to store the unencrypted draft version too?
  • Checked out Helion Energy - Hmm, lots of promises, light on details, but where's the deliverable? Yep, it would be nice to have fusion reactor in mobile phone so it wouldn't run out of power in next 10 years. Some how reminds me from SCRAM jet engines . What could be simpler than SCRAM jet? Yet it seems to be pretty hard.
  • Wondered new version of FireChat. Yet my thoughts are: I would prefer combining different networking technologies, because mesh and flood casts got serious inherent problems. I would only relay messages on mesh network to reach "a better connected node" and try to optimize routing. So use Internet if avail, if not, then try to find path to the recipient or Internet. Both ends could of course have a 'mesh' relay network but the primary path between relay networks could be Internet. This would help in many cases, if one operator is out or so on. There's still somebody with connectivity which you can use to piggyback. Yet keeping system efficient without using too much store and messaging for updating forward & routing tables can be a quite interesting challenge. -> Leads to lot of 'administrative / management / control traffic' -> consumes resources -> Not something you want to run on mobile. - Yet we often do something similar when traveling in group. We get one local prepaid with data plan, and then just tether rest of users to it. Computing OSPF tree isn't a light task for a mobile device in a large network. Using Internet gateways would also limit the size of mesh network that needs to be kept known and routable.
  • SMTP was great when it was open, nowadays it seems that email deliverability is really sucking. There are so many systems which refuse to handle email based on multiple reasons. Basically email isn't a generally working solution any more.
  • Windows 10 is taking the snooping of users to new levels or should we just say to the norm of today. All your data are belong to us.
  • This Akamai GNET CDN interactive map is just beautiful.
  • Studied DO-178B standard. - Gives great example how software can be more reliable. But usually customers don't want good software, they want cheap software and fast.
  • Read A look inside Google's Data Center Networks - They're using Jupiter Network with Jupiter Fabrics. Software Defined Networking (SDN), Andromeda.
  • Checked out meta coin and colored coins. These can open so many interesting possibilities in future. Yet I don't like some examples. Like in case of Namecoin, they give example that people could get names like 'George' based on first to come policy. Lol. Everyone knows where that leads to. Immediately when names are globally shared and unique all the good ones are taken. So instead of 'George' you'll end up with 'georgeb-882' or something similar, which isn't so fun anymore. There has been long discussions why people utilize so lame and limited name spaces.
  • Studied VVER reactor design and benefits of heavy water reactors.
  • Studied Bitcoin Thin Client Security and Simple Payment Verification (SPV) protocol.

DNS-SD, Learn, SSBJ, Security, RA/DHCPv6, 5GHz, IPAM, gRPC, IV, RC4NOMORE, Data Retention

posted Jul 28, 2015, 7:20 AM by Sami Lehtinen   [ updated Jul 28, 2015, 7:21 AM ]

  • DNS based Service Discovery - DNS-SD, RFC 6763
  • Finnish recommendation for Internet Service Providers ISP to deliver IPv6 connectivity to end users [PDF, Finnish].
  • How to actually learn data science. That's what I do. I usually like to setup a project which requires a certain skill set. The skills I don't have, I have to study and learn and then execute the stuff. Creating a actually working implementation will give you a much better insight into problems than just reading about those.
  • Hunted for hot spots in VMware ESXi environment. Some tasks which should take only seconds, can now seemingly take 10 minutes. Ehh, that's not really an optimal situation. Need to investigate more. After some hunting found a memory leak in one application, which reserved huge number of small memory segments and then caused those to be swapped out. Actually that creation didn't cause the problem. Problem was only caused when that huge swap was getting released suddenly on several parallel servers causing absolutely unacceptable amount of disk I/O and 'freezing' the host in process.
  • Youtube documentary Zero days - security leaks for sale. Hacker / hacking / Internet / security documentary.
  • Checked out: Textron AirLand Scorpion and Supersonic Business Jets (SSBJ), and many similar concept designs, but it's really hard to know from limited information resources which of those projects are pure fiction and if some are actually making some progress.
  • It seems also to be really complex to tell if system is using DHCPv6 or RA / SLAAC. Microsoft Windows gives very confusing and misleading information about that. Sometimes addresses are labeled as Public or DHCP but who says that you can't get public address via DHCP? As well as DHCP based entries do not show lifetime but SLAAC based entries do and so on. I'm sure if there are problems, it's going to be horrible to provide customer support because everything is so messed up and it's hard to get reliable information. In somecases it seems that only way to get reliable information what's actually happening is to dump the network traffic and analyze it. Tools, logs and user interfaces are so badly designed and confusing that you can't really trust those. Yep, this isn't first time nor the last. IPsec is similar. There's no way to trust the user interfaces or logs, everything can be more or less wrong. Well after playing with this stuff for a long time, you'll find out which are the places you can trust and which provide conflicting or wrong information. But it's always so annoying when things are inconsistent. It's just like bad or misleading documentation, which makes troubleshooting real nightmare, because you can't trust any information. You'll simply have to go through all possibilities and try to find some reliable source of information (like packet dumps) when you can't trust any other information.
  • Also had separate problems at one hosting company. Well you'll get what you pay for. More money = More dedicated resources. Yet it seems that things turned good. After I made the complaint and clearly said, I'll move all my systems out if this happens again. It hasn't been happening again. Luck or did they actually change something? Sounds pretty unlikely that they would really care. Or maybe I'm underestimating their interest to customer satisfaction, which also sounds unlikely.
  • Noticed that some teams aren't using automated monitoring for their production systems. That's really bad. If you don't monitor service quality & availability it's highly likely there will be exteded down time.
  • Once again ended up in a discussion where I had to remind my self about OTP, OFB, LRW, XTS, XEX. For simplicity of implementation team decided to use standard CTR with AES128. I really like asking some things from cryptology professor, helped to deepen my understanding about a few things. Which I already know how those should be done, but I really didn't understand why. Now I know it too.
  • OWASP Cryptographic Storage Cheat Sheet
  • Hackers remote kill a jeep on the highway. This is the future, everything is connected to the Internet, remote controllable and of course hackable. kw: uconnect, CAN bus, remote, exploit
  • Reminded my self about why and when Initialization Vector (IV) is needed.
  • Studied Python types library, "Dynamic type creation and names for built-in types". - It allows you to generate new classes dynamically. Yep, full classes, not only instances as usual.
  • gRPC Google's Remote Procedure Call system utilizing bidirectional HTTP/2 single connection multiplexed RPC. Really a nice way to utilize HTTP/2.
  • Checked Charles Leifer's post about Python UnQLite bindings. Looks really interesting. I have to check out if I could use unqlite instead of SQLite for some of my projects. Answer is most probably yes. Yet I'm familiar with SQLite3 and if there's no reason to switch, there's no reason to switch. Peewee already offers dictionary like interface for SQLite3 which I'm using with some key, value tables.
  • OpenBazaar project started weekly progress updates in their Blog.
  • Checked out a few IPAM products, yet I believe I don't have any need for those in future either. Managing just a few networks, is trivial, and even more trivial when IPv6 comes along, because you can easily allocate own /64 for every subnet required. Currently my ISPs are offering /48 for businesses and /56 for home users. It's interesitng that the Wikipedia article says that IPAM is more in demand for IPv6, I personally think that it's less required. Also firewalling comes much easier when you can refer directly to required subnet level. Or if you want just to croasely restrict traffic you can easily whitelist whole ISP, instead of going to through tens of even hundreds of different IP subnets they're using. This can be naturally combined with DDNS when required.
  • Quickly tried HaCi and Netmagis - Which just confirmed what I thought earlier. Using one smallish spreadsheet for required data is ok way to manage all I need to manage.
  • Checked out WLAN (Wifi) 5 GHz channels in Europe. I'll need to setup one network and wanted to be informed about channel usage. Ok, I wanted to see also the international differences, I'm curious so I did read it too.
  • Had long discussion with a friend about 'academic research' versus 'efficient execution'. How huge difference there is how things can be done.
  • At one salary comparison site I really wondered about about lack of units and definitions. They just got question like, what's your salary with dropbox containing several ranges from 500 to 200k+. But salary, in which currency? For which period? Weekly, daily, hourly, monthly, yearly? I really tried to look for the definition on the site and I couldn't find one. Also in heavily taxed countries there are big differences if you'll get paid vacations or not and if the salary is before or after taxes. Does it include potential bonuses, extras or overwork or not and so on.
  • HTML5 can be used to hide malware. Surprise? No.
  • Still had strange problems with IPv6 and one Linux server. It's probably related to IPv6 and UFW configuration. Yet I'm not exactly aware what's causing the problem. I changed some settings and if the problem reoccurs then I'll have to do larger changes. I just prefer not to change too many things at once, because then there's no way to tell which particular setting fixed the issue.
  • RC4NOMORE - Yep, RC4 shouldn't be used. As they say, attacks only get better. Here's improved and further developed clever attack against RC4.
  • Actually the DHCPv6 vs SLAAC poll is interesting, because even if address is assigned using SLAAC the DNS and other information can be delivered with RA O flag using DHCPv6-Lite protocol, which does not require M flag. So the host IP address is autoconfigured using SLAAC but the DNS information is still fetched over DHCPv6. This makes the question if you're using SLAAC or DHCPv6 quite confusing. There should be three options, A,M,O which flags are being used or if the address is being configured manually.
  • Studied Veeam backup & replication for VMware or Hyper-v, yet I concluded that I don't have use case for it right now.
  • Carefully studied and commented OpenBazaar's upcoming contract schema. I'll be blogging more about my findings. The schema version which I commented is still under construction and so far 'lightly discussed', so there are many things to fix. But I'll be posting about my OpenBazaar related observations later, and it will be a long post.
  • Studied unqlite-python documentation. - - Nice, I like it. It's fits very well with Pythonic design. Iterable, lists and dictionaries.
  • Data retention, privacy, law and leaks / data theft: What's the problem? Everyone is talking about big data and stuff. Isn't one key factor of that, that any data ever obtained whatever means, won't be deleted, ever. You don't ever know when you might need it. Yet if it leaks, too bad. It wasn't 'our' data necessarily in the first place, we just happened to have it.
    This can be a problem, because some corporations have data retention policies which explicitly forbids deletion of any data, even if it would be required by law. Who's going to audit that anyway.
    Just as example:  If Gmail, Facebook or Dropbox leaks all customer data, including your private messages, chats, email attachments, anything you ever synced (photos, excel sheets) to the service in past 10 years. They can just say s*t happens. Not our problem. This came as complete surprise and we'll be making some improvements in future. Sorry.
    If that happens in future. Don't feel bad. You should have been expecting this to happen when you send your stuff to 'cloud'. So there's nothing to whine about.
    Why so? Because data isn't properly classified when it's generated / received, it leads to situation where there's so much 'random' data that nobody wants to go through it and decide what should be removed. Therefore it's just much simpler to keep everything forever. As well as many developers are lazy, inserting data into relation database is really easy, but nobody bothers to build the data structures so that data could be removed from the database in some sane way without breaking relations and this leads to situation where nothing gets ever deleted.
  • Finally something light, it's a cloud story time! What's the silliest thing you've encountered with cloud stuff? Here's my story.
    Once upon a time, at one customer, they had advance awesome private cloud.  It was really top notch. When we needed resources from that cloud it turned out to the project managers that getting resources from private cloud would require so much bureaucracy, paperwork and meetings, that we'll do it otherwise. We just ordered a few physical servers and installed those to the corner of the office. This was cheap, fast and efficient.
    Isn't flexible cloud stuff awesome or what? Nowadays it would be just as simple to get the servers from UpCloud or similar service provider, but the company's own cloud was a joke. Shadow IT working hard!
    Got any juicy stories to tell? I got tons of those! Share on G+ with me.

No Estimates, Eddystone, AltBeacon Schema, Modulation, OpenBazaar, DNS, DR, DISM

posted Jul 20, 2015, 10:17 AM by Sami Lehtinen   [ updated Jul 20, 2015, 10:44 AM ]

  • I just so much agree with this No Estimates concept. Because truth is that estimates are usually horribly wrong and not counting multiple factors. As well as so many of the details are missing that estimate is really a complete guess.
  • UK is again considering banning of encryption. This is strange trend. Don't they realize how much it can harm economy? Yet it won't be a problem for people who are willing to use encryption even if it's illegal. You'll just need to camouflage it so it isn't obvious. Crypto Wars are back - Should all encryption contain backdoor so it can easily be decrypted if required?
  • Had extensive discussions about international trade and business arrangements with a few friends.
  • Telegram was under massive 200Gbit/s DDoS Attack. Attckers were using Tsunami SYN Flood.
  • Checked out new contract schema drafts for OpenBazaar .
  • Also studied pre-existing schemas at - I love standards, but I always want that the standard is extensible. Most of standards really aren't in any easy way. Does unknown field cause an error or is it silently ignored? Well, if it causes process to fail, it's not extensible, because you're creating new standard for adding something simple into old standard.
  • I like standards really, but I also acknowledge the need for extensible standards. Especially in cases where quite simple things are being done using some heavy standard is a good example when I don't like standards too much. In such case studying standard can require a lot of time, there can be several complex traps in the standard as well as the implementation being build probably shortcuts most of the standard. Then you have a 'standard' solution with extremely limited functionality which causes errors when anyone with fuller implementation tries to talk with it.
  • OpenBazaar DHT and long term data storage: All data stored in distributed network / DHT should have TTL as well as most probably re-balancing (republishing) at quite rare intervals. These are the things I've been tuning with GNUnet guys back a long time ago. Originally they didn't have any expiry and it was bit strange, only new nodes stored new data as old nodes were full of old data. Duh! Yet this is the case where potential spam / flooding can get really dangerous and problematic, potentially hindering functionality of whole network for extended period.
  • What's new in uWSGI 2.0.11 - No HTTP/2 - support yet. I guess they haven't figured out what's the best way of doing Server Push.
  • Firefox starts to block Flash as default (Until most serious vulnerabilities are fixed). Yay! It has been causing so much security trouble. Now it's a must to start using HTML5 instead of Flash. Everyone has recommeded this for years, but well, u know, people and organizations are really slow making changes until they have to.
  • Reminded my self about QAM, OFDM and SSB. Interested? See modulation @ Wikipedia that's a good starting point.
  • Checked mobile power consumption 3G vs 4G on in my typical usage environment. Difference is really small, and 4G speed and low latency makes things nicer so it's a win for 4G (no surprise there).
  • Frawned once again about security procedures (total lack of those). Everything is installed and configured randomly and not even fixed in case there are reports of serious misconfiguration.
  • Well how's that different from Adobe Flash issues? Well it isn't. Who cares if there are serious exploits or bugs. If there's no widely used exploit for those, it doesn't matter. It only matters when it's actually happening, before that it's only theoretical threat.
  • Reminded my self about Paravirtualization.
  • Studied Google's Eddystone and their Blog entry about it. It's a flexibe iBeacon replacement. Also see Electric beacon. This is also a concept which could bring new business to small startups dealing with those. The Eddystone's telemetry frame (Eddystone-TLM) is also very interesting from this aspect when combined with Beacons Diagnostics. It's really nice that the Eddystone supports URL beacon instead of UUIDs alone. Problem with UUID is that for most of people it really doesn't mean or represent anything at all. UUID is about as useful as MAC address of WiFi base station. It can be meaningful to you, but in most of cases, it just doesn't mean anything at all. There are also some encrypted frame types like Ephemeral Identifiers (EIDs). It's also good to knowledge related technologies like Weave, Thread and Brillo all of this also realtes to Internet of Things (IoT).
  • Checked out AltBeacon. Read the AltBeacon protocol specification and frame type. Yet AltBeacon is super simple and only sends really short UUID making it also as useless as iBeacon is without external database. Useless? Well, I just now got 6415712610302 in my hand. Of course you should know what it is! 
  • Reminded my self about Bluetooth Low Energy (BLE).
  • Also checked out Google's Physical Web project. Yet it's merging to use Eddystone technology. I also love the concept, because I personally would prefer almost always HTML5 application over native application. I just hate installing tons of junk on my phone, when I really rarely need those. Using a properly designed HTML5 website, when I need one would be a lot better option.
  • Frowned to Microsoft, I guess they're working hard to make things as annoying as possible. Running CleanMgr.exe is really annoying on 2008 R2 or 2012 R2. I think Windows is even harder to use than Linux. There's absolutely no user friendliness what so ever, they've made it about as annoying and complex as it can get. I just posted one solution to the problem here.
  • Read some deliciously enjoyable stuff like: Potato paradox, Ham sandwich theorem, Pizza theorem, Pancake sorting, Fair cake cutting
  • Checked out and PeerJS for efficient P2P direct in browser JSON utilizing WebRTC communication without needing to pass data via server.
  • Checked out OpenBazaar contract types: Physical Goods, Digital Content, Services and process flow charts for Physical goods (flow), Digital content (flow), Services (flow) - Getting a contract expiry is a great thing. There's also a new way to host images ant vendor's node. Which probably means that there will be some kind of new API call to fetch data in case data can't be fetched directly over HTTP. I also want to get the data so that it doesn't need to be refetched when contract is refreshed, so the image data can remain static, even if other parts of the contract get changed. Also the process used to encrypt address using XOR and nonce makes me think, but no conclusion yet. I have to find out why this is being done. I heard that they got cryptography professor, I hope it helps!
  • OpenBazaar is generally very interesting project. Networking, P2P, DHT, Reputation management, Transaction Ratings, Python, OpenPGP (PGP), E-Commerce, Encryption, Digital Standardized Ricardian Contracts using contract type based schema, BitCoin, Multisignature (multisig), Escrow, Moderators, ECC, Cryptography, Semantic data, Digital signatures, Cryptographic hash, JSON, databases and all that stuff, Financial Power combined with global free P2P trade! Connecting vendors and buyers around the world. Minimizes personal identifiable information (PII) leak yet provides strong identity using GUIDs, metadata, network data. This is exactly the kind of project I've been looking for several years and have been wondering why nobody sees the potential for it!
  • Checked out a Passcard - a Bitcoin based identity and authentication solution. Ok, I had to register too. Here's my Onename profile.
  • Reminded my self about DNS Glue Records and circular dependencies.
  • Had not so fun with DISM and Windows Servers. It's huge mess with bad instructions & documentation. I would really like to cleanup winsxs from all uncesessary junk, with Windows 2012 R2 it's reasonable, but with 2008 R2 I can't find similar commands? It seems that the things work differently with every WIndows version, hor annoying is that?
  • Reported a few seriously bad IPv6 routing issues to corresponding NOC's (,,
  • Studied Google's Disaster Recovery (DR) Planning Guide and Cookbook.
  • Now when IPv4 addresses are running out it's interesting to see traffic from IPv4/8 addresses where you never used to see traffic earlier like 1. 2. and 5. I actually got severs my self in 5. which use to be 'used by Hamaci' because nobody uses it. Smile.
  • Launched a poll in IT Professionals group, if you use SLAAC, DHCPv6, Static/Manual or some other method to configure IPv6 addresses.
  • It seems that it's hard to get for some people that when IPv6 starts to be used, and no IPv4 is being used, they have to start using IPv6 too, there's no other way to get things to work. Even if they still got 'enough addresses' behind their NAT. Smile.
  • I don't know if it's really necessary so often, it seeems that my home network triggers ICMP6, neighbor solicitation, ICMP6, neighbor advertisement for all IPv6 addresses every minute.
  • From my G+ post: " Well DHCPv6 doesn't always help with audit, because in some cases it won't help compared to SLAAC at all. Unless there's some additional authentication layer, it's really hard to get any information who's using which address and logs won't provide enough information. Even if logs would contain MAC you can naturally trivially change it.
    This is the area where many things need to be changed before things work out really well out of the box. Well, ok, not all DHCPv4 servers neither log mappings nor traffic, so that's not a new issue either.
    You can also log NDP traffic when using SLAAC and gain basically same information you would get from (working) DHCPv6 logging. "
  • And " Full port security was also first thing come into my mind, but that's pretty expensive solution. Most of networks do not require that kind of security. It's just enough that's some way to detect users. It's also interesting to see what kind of problems arise from network filtering or lack of it. I've already noticed that filtering MLD causes loss of connectivity at some cases and well of course not filtering some messages has similar results if someone just purposefully injects those to network like rogue RAs.
  • I remember good times when you could bring major systems down by hijacking just IP on switched large network or running rogue DHCP server. Smile. "
  • What's the difference between LAN and WAN in future, none? 'LAN, service provider is often responsible for WAN. But because we're talking about the Internet, why you should have lan, you can just bridge WAN to make it a LAN. same stuff, no router needed, just a switch. In many environments I don't have separate 'LAN' at all, it's just switched Internet and depending where packets are going those go to LAN or WAN.'
  • Studied UHV power transmission in China.
  • Debugging one network with tcpdump required me to refresh my memory about RA MO flags.
  • Quite nice and a simple post how backpropagation works on neural networks. A good read if you haven't ever really thought about it.
  • Glanced OpenBazaar Docs Documentation site. - There's a ton of stuff which I have to study later, it's all so good stuff.
  • A nice Infographics by BBC about Artificial Intelligence.
  • Once again thought why we do not yet have universal strong identity for ehh, for lifeforms (I said universal). Ok, let's say for humans. Many people are using IBAN it shouldn't be impossible to provide a global strong identity for people. Issued by governments.
  • Just a post how to learn data science. It's a guide basically, how to get started. I personally couldn't agree more. That's how I often get things done. I pick interesting topic and then I create related project. To get my stuff done, I'll have to learn how to get it done. keyword: learn by doing.

Back log is still building up. I'll really need to create one what I did during the summer dump.

Download CleanMgr.exe for Windows Server 2012 R2 & 2008 R2

posted Jul 17, 2015, 8:32 AM by Sami Lehtinen   [ updated Jul 17, 2015, 8:40 AM ]

Download Disk Cleanup Manager for Windows Server 2012 R2 64 bit.
Download Disk Cleanup Manager for Windows Server 2008 R2 64 bit.
Just drop content of archive C:\ and that's it. Then you can run CleanMgr.exe and get the job done. 
This assumes that you're using c:\windows path as most are. If path is different, just drop cleanmgr.exe in system32 and the cleanmgr.exe.mui in en-US in system32. Because system32 is in path, then you can just run cleanmgr.exe to clean up the system.
Again Microsoft has made everything really unnecessarily hard and complex. There are only bad ways of dealing without proper download link.
1) Follow complex instructions with super annoying long filenames, path and commands, which most likely won't work with Windows Server 2012 R2 64 bit version. 
2) Install Desktop Experience package which contains load of junk. Isn't purpose of disk cleanup to get rid of that junk, not install more? Desktop Experience installs stuff like Ink and Handwriting Services, Media Foundation, which naturally are totally useless on average server.  *
3) My solution? Provide as simple download link for needed software and install it in a seconds. Why isn't this provided by Microsoft?

MS Description for this application:
"You can use Disk Cleanup to reduce the number of unnecessary files on your drives, which can help your PC run faster. It can delete temporary files and system files, empty the Recycle Bin, and remove a variety of other items that you might no longer need. The option to cleanup updates helps reduce the size of the component store."
Wikipedia Description for this application:
"Disk Cleanup (cleanmgr.exe) is a computer maintenance utility included in Microsoft Windows designed to free up disk space on a computer's hard drive. The utility first searches and analyzes the hard drive for files that are no longer of any use, and then removes the unnecessary files. There are a number of different file categories that Disk Cleanup targets when performing the initial disk analysis"

*) Desktop Experience junk load contains a ton of unnecessary junk like:
  • Windows Media Player
  • Desktop themes
  • Video for Windows (AVI support)
  • Windows SideShow
  • Windows Defender
  • Disk Cleanup
  • Sync Center
  • Sound Recorder
  • Character Map
  • Snipping Tool
KW: WinSxS, Disk Cleanup, Remove Junk, Purge, Compact, reduce disk space usage, winsxs consumes a lot of disk space, FAQ, Instructions, step by step, commands, without installing desktop experience, disk cleanup utility, enable, install, run, access, 2008r2, 2012r2.

How to persistently disable IPv6 privacy addressing on Windows 2012 R2 Server

posted Jul 13, 2015, 6:45 AM by Sami Lehtinen   [ updated Jul 18, 2015, 11:11 AM ]

Here's the solution to the question how to persistently disable Windows 2012 R2 server IPv6 privacy addressing.

All you need to do is open Administrative PowerShell and run following commands:

Set-NetIPv6Protocol -RandomizeIdentifiers Disabled
Set-NetIPv6Protocol -UseTemporaryAddresses Disabled

That's it! Now it's done and also remains that way. The old netsh based settings are reset after a system reboot.

So if you're like me, you've probably tried the commands seen below. Just to find out that these settings remain only active until the system is restarted. I don't know why restart causes the configuration settings to get lost even if there's store=persistent option there.

netsh interface ipv6 set global randomizeidentifiers=disabled store=active
netsh interface ipv6 set global randomizeidentifiers=disabled store=persistent
netsh interface ipv6 set privacy state=disabled store=active
netsh interface ipv6 set privacy state=disabled store=persistent

I know many people suggested using Group Policy editor (gpedit.msc) or editing the Windows registry keys (regedit.exe) but those simply won't work.

This allows auto configure a address which remains the same (except the prefix part) even if it's moved to another network, address will be MAC based. More information is available in RFC4941.

keywords: IPv6, privacy addressing, MAC based IPv6 address, Windows, Server, configuration, registry key, registry, settings, networking, Internet, static, statically, loses, lost, losing, randomized identifiers, use temporary addresses, addressing, netipv6protocol, set-netipv6protocol, net, netsh, interface, randomize, randomizes, config, set, disable, disabled, inactivate, network, stack, configure, setup, install, permanent, permanently, persist, store, stored, key, value, reboot, restart, shutdown, boot, start., support, Microsoft, question, questions, answer, answers, operating system, os, IP6, without DHCPv6, DCHP, SLAAC, RFC4941, disabling Privacy Extensions for Stateless Address Autoconfiguration in IPv6, RFC3041, RFC, 3941, 3041, IPv4, stable address, storage, disable randomized interface identifier, disable usage of temporary addresses, temp, tmp, constant, constants, reference, references, automatic, automatically, protocol, process, standard, standards, IEEE, same node, outgoing sessions, global scope, prefix, suffix, subnet, mask, RA, router advisement, preference, preferences, step by step, easy, instructions, guide, tutorial, commands, IP6, stabile, unchanging.

AAAA, DHCPv6, Agile, Scrum, IPv6, !bang, Go @ GAE, Intelligence Algorithm, Happy Eyeballs

posted Jul 11, 2015, 1:58 AM by Sami Lehtinen   [ updated Jul 11, 2015, 2:15 AM ]

  • Added AAAA records for many servers. So far everything has been IPv6 enabled and working, but yet not published in DNS. Only used for pilots and testing. But now, it's fully laid out.
  • DHCPv6 Client Link-Layer Address Option - Checked out RFC6939 which explains why many DHCPv6 servers can't identify the client as DHCPv4 servers do. Also read RFC4361 - Node-specific Client Identifiers for Dynamic Host Configuration Protocol Version Four (DHCPv4). As well as checked out IPv6 ULA Unique Local Address
  • Finished a books about Agile software development and Agile Scrum.
  • Also checked out a few related articles: Agile Is Dead (Long Live Agility), AGILE must be destroyed, once and for all, Top 12 Things Every Software Engineer Should Know, Mob Programming, #NoEstimates, Empirical Project (Process) management & control model,,
  • Basic stuff like Return on Investment (RoI), Time to Market (TTM), Total Cost of Ownership (TCO), deadline, feature set, quality, cost, prioritization, Agile management team, Growing software, organic growth based on real needs and situation instead of fixed waterfall, Story Points, Agile Contracts, Proof of Concept (POC), public sector projects and laws, sub contracting, out sourcing, scrummaster.
  • It's so familiar situation that customers asks what IT costs even if they don't have any clue what IT is. So annoying, yet they're afraid that using agile methods will lead to costs ballooning out of control. Well, is it my fault if their scope balloons out of control? Ha!
  • Now I've been dealing with 8 different service providers when enabling IPv6 for all systems I'm responsible of. It's just wonderful of many different problems you can have. SLAAC, DHCPv6, manual configuration, no documentation, gateway address issues, DNS failing, and so on. But yep, when you just decide that you'll get it done, you'll get it done. I guess i've made something like 40 tickets about different problems and also found out that service providers aren't 100% IPv6 ready, because they have problems on their side and helpdesk might not be aware about all required things. Yet, it's good to have practical experience from other cases. Network card driver issues, Virtualization platform issues and so on. So much fun! Why it is MY JOB to tell the hosting company what drivers they should use with their vitualization platform when they're cluess. Aww. Well, this isn't first or last time when something like this will happen. It's not my task to tell you, but how about just ...
  • Tons of server migrations, from virtualization platform to another, as well as re-arranging servers between service providers and data centers, providing awesome cost benefits.
  • Something different: Travelling wave reactor - Mutualism (economic theory) - Confidence trick and a list of tricks.
  • OpenBazaar Ratings, Reviews and Reputation - explained 106 slides, Ricardian Contracts, Identity, Rating, Privacy, Storage.
  • Fixed !robtex !bang syntax for (DDG) search engine.
  • App Engine for Go is now generally available. Need a fast and easy to scale PaaS platform? This is the one. You'll focus on your code and business processes and Google takes care of the rest, no worries. Unfortunately I haven't had time to play with Go, but I'm sure it works as well as Python with Google App Engine. I would still love to see App Engine with Python 3.x support. Actually that's one of the reasons I'm not currently developing anything for App Engine.
  • Is there a simple algorithm for intelligence? Interesting article, I guess will find out later. Just like todays computers, things are based 'really simple' princilple of gates, but layered to build something which is absolutely hugely complex. If you compare modern laptops versus old four function pocket calculator - - , both use exactly the same primitives. Don't forget things which underlined the principles used with electronic systems, like the Charles Babbage's Difference Engine.
  • Apple is catching up with IPv6 - Dual stack implementation and DNS lookups with Happy Eyeballs aka Quick Fallback.
  • Happy Eyeballs - Yes it's related to IPv6? What? Read the article.
  • Sonera is building a new large multi-tenant data center in Helsinki. Do they offer multiple carriers? Who knows, they didn't mention.
  • Reminded my self about current state of IPv4 address pool exhaustion.

Cloud, Certificates, s2n, TEMPEST, data mining, IPv6, PostgreSQL, Let's Encrypt, Latency, Data Protection

posted Jul 6, 2015, 6:13 AM by Sami Lehtinen   [ updated Jul 6, 2015, 6:13 AM ]

  • Checked out Trusted Cloud Europe - A policy documentation how to create secure and trusted cloud environment and agreements, and how Europe could get such ecosystem built?
  • Something different: Meteor missile
  • RCC - check your system's trusted root certificate store. Do you have some certificates in your system which you really wouldn't want to have? I got this one: Number of 'interesting' items: 1 (Not part of baseline RCC1_STANDARD_MCP): 32F30882622B87CF8856C63DB873DF0853B4DD27: VeriSign Class 3 Public Primary
  • s2n -  a new open source TLS implementation - Is (?) there's a need for OpenSSL alternatives.
  • Stealing Keys from PCs using a Radio: Cheap Electromagnetic Attacks on Windowed Exponentiation - TEMPEST stuff is nothing new.
  • Tested servers in UpCloud's new Frankfurt Zone with MaxIOPS, it all looked good! Frankfurt is slightly bettery location to host services for whole Europe than London and quite much better for Nordics like Denmark, Norway, Sweden and Finland.
  • Excellent article Top 10 data mining algorithms in plain English. - It's really worth of reading it!
  • Ublock is better alternative to AdBlock. - I'm now using it.
  • Had to disable IPv6 privacy extensions on some servers which are causing constant trouble using commands:
    netsh interface ipv6 set global randomizeidentifiers=disabled store=active
    netsh interface ipv6 set global randomizeidentifiers=disabled store=persistent
    netsh interface ipv6 set privacy state=disabled store=active
    netsh interface ipv6 set privacy state=disabled store=persistent
  • Enjoyed some issues with IPv6 privacy addressing (RFC 4941) and SLAAC (RFC 4862). It seems that the privacy addressing might not remain disabled for unknown reasons. So annyoing. Well, I'm pretty sure I'll find a solution for that. When I reboot the system, privacy addressing is again enabled. Aww... It seems that only viable way to work around this is to configure a static address.
  • What's new with PostgreSQL 9.5 - BRIN (Block Range) Indices look really interesting. - A full listing of changes
  • When checking servers logs, there's amazing amount of different bots which crawl the sites, I just saw these two: PaperLiBot/2.1, uMBot-LN/1.0 and probably those aren't going to be the last new bots I see.
  • They're so right, people continuously downplay risks related to ICT systems. They always seem to think that we don't have any valuable data, nor nobody's interested about it. Well, that's exactly why many systems are so easy targets, because nobody thinks those would be targets in the very first place. Also most of project related people always seem to think that security doesn't really matter at all. I've seen this happening over and over again.
  • Let's encrypt is good and bad, it's always just how people consider things. Afaik, it's also kind of bad that after let's encrypt launches all sites serving malware will also be 'secure'. So it's up to the users to understand that using encryption has nothing to do with being secure.
  • It seems that service providers (Yes, at least three individual onces) treat IPv6 as second class citizen clearly. If there's a networking issues IPv4 problems are resolved quickly, but fixing IPv6 issues or even noticing those, can take considerable time. Up to days. Well, I'm pretty sure that situation is going to change in future. One issue is still on going, it's ridiculous. Repeated outages and this time it's NOT about the configuration issue with MLD.
  • Maybe this is going to change that? - North America out of IPv4 space officially.
  • Once again really enjoyed reading The Economist, it's just excellent magazine.
  • CloudFlare guys play with network stack and try to shave off latency. A really nice post.
  • Finished reading the Handbook on European Data Protection Law (EU, European Union). - That was well, a really horrible thing to read, but topic is good. If you need to get some sleep, try reading this.
  • Had once again long discussions with colleagues as well as with friends about different cloud services, pros, cons and so on. It's impossible to say anything without proper case study and testing. Of course you can set some kind of rule of thumb, like where the service is mainly used and do you just use it for backups, do you transfer lot of data, what kind of access speed you need. Do you need a lot of ram, CPU power, both, networking, SSD or traditional disks. Do you run Windows, Linux or do you want some kind of SaaS / PaaS and so on.
  • Google+ seems to think that you're probably liking posts you try to mute, because those activate you... So you'll end up getting just more of the stuff you didn't like! It's a good strategy if your goal is to annoy people. Everything what really upsets them, is the stuff that will make them react. Even more than stuff what they like. ;)

Duplicati 2.0 - Tiered Block Storage & Server Side Scrubber

posted Jul 5, 2015, 7:54 PM by Sami Lehtinen   [ updated Jul 5, 2015, 8:31 PM ]

I just one day thought how Duplicati 2.0 would 'scrub' it's data blocks which contain too much expired / stale data. Currently it just reads the blocks and replaces the stale data with fresh one. But could it be optimized more? I think it could, and that potentially would make big difference. Another immediate idea was using a server side scrubber application, if writing a full client / server solution is too heavy.

Tiered Block Storage

When data blocks are scrubbed, instead of mixing old (yet still valid) and new data, just separate that data into old and new blocks. This would prevent mixing volatile data with data that doesn't change too often, if at all. Whenever scrubbing those old and even older blocks, do the same. Over time this should lead to massive amount of network and disk I/O reduction. Because there's no need to juggle the old data around so much. Amount of network traffic reduction could be even amplified by using the Server Side Scrubber.

Server Side Scrubber

Duplicati doesn't have a full client server model, where scrubbing could be done on the server side alone. Client server solution would let the client to upload only the new data without need to touching the old data. My question is that if it would be possible to run a scrubbing as separate server side scheduled process ? This should reduce amount of network & disk I/O quite a lot. Combine this with Tiered Block Storage and the network and disk I/O should be reduced quite a lot.

Any thoughts? Feel free to comment my G+ post.

SSD, MLD, ICMPv6, OCP, ECN, BitTorrent, LavaBoom, OVH, RamNode, Web, Yeti, DVR, Python, ATHEX

posted Jul 5, 2015, 6:50 PM by Sami Lehtinen   [ updated Jul 5, 2015, 6:50 PM ]

Some light stuff during summer vacation.

  • Many Samsung SSD drives got badly broken firmware leading to data corruption. Ha! No news, isn't it more or less normal that everything is broken when you start looking things in detail. Often it's just better to accept it, because when you start doing it in the detail, results are often as those are in this case. It just seems to work, and we don't even want to know all the dirty secrets
  • Had interesting problem with one VPS server. It has lost it's Internet connectivity. Everything seems to be ok, except IP addresses aren't getting assigned via DHCP. VM ware configuration ok, Windows configuration ok, but yet not working. Strange. Console does work, and I got 'a few' similar servers which all also work correctly. Strange. It's always so fun to troubleshoot more or less random stuff. Here's actually link to the problem with MLD / ICMPv6 / Neighbor Discovery.
  • Reminded my self about Explicity Cognestion Notification (ECN), RFC3540 - Reminds me from Source Quench (SQ) messages. - Enabled it on most of servers switching tcp_ecn parameter from 2 to 1.
  • Some ISPs seem to have funny attitude about IPv6. They will happily give you /22 subnet for IPv4 but for IPv6 they're only allowing you one /128 address.
  • Checked out Open Container Project (OCP) - This is great no it's AWESOME! Now there's really strong coalition (Google, Microsoft, Amazon, Docker) with open governance to create really powerful standard (which is widely supported) containerization technology. Not forgetting small companies like Fujitsu, EMC, HP, Huawei, IBM, Intel, Red Hat, VMware, CoreOS. Container as a Service (CaaS) will be the future of the cloud. Also see runC
  • Someone is flooding fake peer information to BitTorrent network - Lol. New attack? Hardly, such attacks are known for P2P networks for ages. First step is NOT TO delegate unconfirmed peer information to whole network. Actually his is exactly what I did for BitMessage at one time. Everything needed to tackle this is to prefer known good nodes and trusting data got from good nodes via peer exchange on relativity basis. So unknown fake peers are quickly left out from 'web of trust'. To deal with this trivial attack requires just a little bit of code to defend against.
  • Once again wondered 'normal' server administration style: Disable updates -> because updates cause problems, Open firewall -> because firewalls cause problems, Use default credentials -> because passwords cause problems, Do not allow users to change passwords -> because changing passwords causes problems, Passwords shouldn't expire ever -> because changing passwords causes problems. Phew, nuff said.
  • Checked out (yet another) social network. My Minds profile. Yep, site is clearly alpha, but working okish.
  • Elisa is also starting to offer IPv6 connectivity on mobile via tunneling.
  • Something different: Comparison of orbital launch systems, Vega, Falcon Heavy, Long March 5, Ariane 6, OneWeb satellite constellation, Electrically powered spacecraft propulsion, Electrodynamic tether, Magnetic Sail, Photonic laser thruster (Wow, that's complex stuff), Personal reconnaissance satellite - S-400 SAM / S-500 missile
  • Google's global surveillance is progressing. Now their goal is to listen and catch all conversations of the world. Not OK, Google! - Good writing about this important topic. 
  • Eat your own dog food is excellent method. I've found so many bugs during my own usage of LclBd that if someone else would have been reporting those, I would just easily claimed. It's working, you're just doing it wrong. Of course without specifying in detail what's wrong and what's not. Just fixed one Unicode URL spam control issue, where certain Unicode characters caused problems with Google Safe Browsing web API.
  • Had some not so interesting loss of network events with Windows. Not amusing at all. Even more debugging is required to locate the problem. Aww.
  • Tested out secure email service LavaBoom. It's pretty sleek and sweet. I might be using it as temp account service. It got PGP support too. Yet after bit more extensive testing I found it pretty buggy and there are some usability issues. Ok, you can get things to work, but those are pretty confusing, before you know how to exactly get that stuff done. I hope they'll get it fixed. Basically LavaBoom is just like Hushmail, but JavaScript has replaced Hushmail's Java Applet.
  • Had another day long meeting about change management and process development. There are so many things to consider, which none are new. But ok, let's say it's good to talk about these. Even if anyone who's been working for a long in this sector knows all the usual stuff. Then there's the buy the book and optimal way and then there's the reality with different priorities and people.
  • Quickly checked out Google Cloud Repositories
  • Problems with OVH Classic seem to be quite unbearable, I re-checked Kimsufi, Vultr, DigitalOcean and RamNode. I'll think I'll try RamNode next.
  • Web feels more broken every day? - My view - I think you've just forgotten how things used to be. Like if you encounter BSOD or something, you'll get upset. But do you remember how often Windows 3.1 freeware / shareware apps crashed? When I create web sites, I'll always focus on performance. Like most of pages require just single item to be downloaded and no, I'm not in-lining stuff. I'm just reusing already downloaded stuff. And even amount of that stuff is minimized. Benefit? On slow mobile connection my pages load in under a second, versus many sites which might take minutes(!) yes, that's right to load.  I'm sure anyone using NoScript has noticed how full of s*t many popular web sites are today. My sites all load from single domain using HTTPS, no 3rd part junk.
  • Yeti DNS Project launched - It's an IPv6 only DNS project supporting DNSSEC.
  • Digital Video Recorder (DVR) file system ... Big blocks, simple chaining ... I might write another post about this. Well, thoughts for simple file system for managing data in big blocks to avoid need to defragment. This just came into my mind, because my DVR is at times really loud and seems to be doing some kind of background maintenance tasks, when you would expect it to be quiet. Which practically sound just like defragmention. Tons repeated seeks when nothing should be happening.
  • Ubuntu 15.10 Wily Werewolf going to use Python 3.5 default. Finally!
  • Quickly checked out line chat app which is perfect Skype replacement.
  • I'm following ATHEX 20 closely, it could be possible to get really good investments from Greece with reasonable price. Or maybe not, who knows, what will follow. But I just think there could be a good time to buy when the crisis escalates and money tries to escape. Even if there are good longterm investment possibilities? I've been also discussing this scenario with my trusted friends extensively. It's nice to see how many different arguable scenarios out can get out of this.
  • I've been helping my friends to setup their IPv6 connectivity, either using tunnel from, 6rd, 6to4 or native connection from ISP.
  • I just encountered a Finnish company that uses Yubico Yubikey for remote employee authentication. That's nice. Most of companies are still using static passwords.
  • It's funny to see how hyper-convergence is coming so hard again. We can use PC's as PC's, yep, that's what is has been all the time. Yet there are positive and negative sides on that. As well as cases where software is run on generic hardware and then there are situations where ASIC or specialized hardware is being used. This is pretty never ending battle, just the shifts back and forth are amusing at times. I still miss IBM NetStation, smile. On the other hand JavaScript and in-browser clients are pushing traditional web server back to browsers and so on.
  • Web is getting slower? - Nobody uses you to use websites which are full of junk. As example, LclBd isn't.
I'm starting to build a backlog on these posts again. There might be just a 'dump' post soon again.

IPv6 networking issues - ICMPv6 Type 130 Multicast Listener Discovery (MLD)

posted Jul 5, 2015, 6:25 PM by Sami Lehtinen   [ updated Jul 5, 2015, 8:13 PM ]

I wondered why I'm having occasional IPv6 outages, everything seems to be working fine and then it just stops. Something wrong, but what? More interesting fact was that after pinging (ICMPv6 ping) the host, everything started working again, just as usual.

Reason for all this was my fail, that I were blocking the Multicast Listener Dsicovery (MLD) aka ICMPv6 Type 130 datagrams. Those are used by the network to maintain state of the devices attached to the network. Debugging the situation was bit funky. When I pinged the node using ICMPv6 PING, the router initiated Neighbor Discovery (NDP) and everything worked. But in normal operation router uses periodic MLD queries to see if the clients are still there. If not, it stops forwarding / routing traffic to right destination.
I also got confused about that, because so far other networks I've been using so far have been always using NDP all the time, and not using the Multicast Listener Discovery v2 protocol (MLD) for periodic network state refreshes. Also the network started after system reboot due Router Advisement RA traffic and pings and other networking worked for a while due NDP.

Some traffic dump for analysis. As you can see neighbor advertisement and solicitation are clearly being used. Shouldn't that be enough as it is in many networks.

11:25:51.981048 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 64) fe80::1 > ip6-allnodes: [icmp6 sum ok] ICMP6, router advertisement, length 64
        hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
          source link-address option (1), length 8 (1): 00:05:73:a0:05:1e
            0x0000:  0005 73a0 051e
          mtu option (5), length 8 (1):  1500
            0x0000:  0000 0000 05dc
          prefix info option (3), length 32 (4): 2a04:3540:1000:310::/64, Flags [onlink, auto], valid time 2592000s, pref. time 604800s
            0x0000:  40c0 0027 8d00 0009 3a80 0000 0000 2a04
            0x0010:  3540 1000 0310 0000 0000 0000 0000
11:27:03.009866 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 24) > fe80::2: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is, Flags [solicited]
11:27:08.022639 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a8aa:aaff:fe80:470d > fe80::2: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::2
          source link-address option (1), length 8 (1): aa:aa:aa:80:47:0d
            0x0000:  aaaa aa80 470d
11:27:08.025583 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::2 > fe80::a8aa:aaff:fe80:470d: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::2, Flags [router, solicited]
11:27:13.029685 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::2 > fe80::a8aa:aaff:fe80:470d: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a8aa:aaff:fe80:470d
          source link-address option (1), length 8 (1): c4:7d:4f:8c:27:40
            0x0000:  c47d 4f8c 2740
11:27:13.029750 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 24) fe80::a8aa:aaff:fe80:470d > fe80::2: [icmp6 sum ok] ICMP6, neighbor advertisement, length 24, tgt is fe80::a8aa:aaff:fe80:470d, Flags [solicited]

Lack of these messages as responses was the problem causing the outage. So if that is omitted it seems that the IPv6 routing in that particular network just stops working at some point.

11:28:06.117828 IP6 (class 0xe0, hlim 1, next-header Options (0) payload length: 36) fe80::2 > ip6-allnodes: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener query v2 [max resp delay=10000] [gaddr :: robustness=2 qqi=125]
11:28:12.134638 IP6 (hlim 1, next-header Options (0) payload length: 36) fe80::a8aa:aaff:fe80:470d > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff80:470d is_ex { }]
11:29:12.481671 IP6 (class 0xe0, hlim 255, next-header ICMPv6 (58) payload length: 64) fe80::1 > ip6-allnodes: [icmp6 sum ok] ICMP6, router advertisement, length 64
        hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0s, retrans time 0s
          source link-address option (1), length 8 (1): 00:05:73:a0:05:1e
            0x0000:  0005 73a0 051e
          mtu option (5), length 8 (1):  1500
            0x0000:  0000 0000 05dc
          prefix info option (3), length 32 (4): 2a04:3540:1000:310::/64, Flags [onlink, auto], valid time 2592000s, pref. time 604800s
            0x0000:  40c0 0027 8d00 0009 3a80 0000 0000 2a04
            0x0010:  3540 1000 0310 0000 0000 0000 0000

Just to clarify it out. I didn't choose to block MLD. It was just default configuration for UFW on Ubuntu / Linux. It seems that Windows Server is by default allowing such traffic. But UFW isn't. So when ever enabling firewall you'll have to consider a few new things. Actually this isn't too different from IPv4 and when firewalls came to exist. People were blocking packets required for DHCP to operate and wondered why their network connection dies when DHCPv4 address renewal failed.

Other issues

It clearly seems that IPv6 isn't yet important enough. Serious networking issues can take days to get fixed, because it's not that important. I've seen that happening with multiple service providers. Also routing is really broken at times. See: Google+ discussion in IT professionals.

1-10 of 266