My personal blog is about stuff I do, like and dislike. If you have any questions, feel free to contact. My views and opinions are naturally my own personal thoughts and do not represent my employer or any other organizations.

[ Full list of blog posts ]

34C3 - TUWAT - Talks - Part 7

posted Apr 15, 2018, 10:07 AM by Sami Lehtinen   [ updated Apr 15, 2018, 10:07 AM ]

  • BGP and the Rule of Custom - This talk should be interesting. I haven't ever really dug deep with BGP routing, but I've read the basics several times. They started with interesting history talk. Mesh protocol, routing between AS numbers. Interconnections between organizations are public knowledge. Tier 1 providers, Tier 2 operators and customers.
  • Mobile Data Interception from the Interconnection Link -  Lots of interesting attacks against different mobile networks. And why you shouldn't use SMS for OTP authentication. And finally in Q&A, security costs money, yet everybody seems to expect it to be free (or something similar). The usual mantra.
  • MQA - A clever stealth DRM-Trojan - Hmm, curious what have they done. This talk should provide a "A critical look on a new audio Format". Master Quality Authenticated (MQA). Digital Rights Management (DRM). Lots of talk about media formats and hardware being used and so on. Lossy compression. 24 bit quantization using 192 Hz sampling rate for audio. Licensing fees.
  • Low Cost Non-Invasive Biomedical Imaging - An Open Electrical Impedance Tomography (EIT) Project. Preventative scans combined with AI analytics, that would be awesome. How about getting full body scan every week or month? Resolution, cost and time discussed. Now they got to theelectode arrays, that's the first obvious thing which I thought about when thinking how the stuff could work. Interesting and good talk. Nothing new afaik, but let's hope this technology got bright future.
  • Briar - Fresh talk about Briar Resilient P2P Messaging for Everyone. Good talk, encouraged me to test the beta with friends. Encrypted communication over avian carriers, nice, classic. Of course not forgetting secure sneaker net and dead drop communication. Lessons about cryptographic algorithms being used. No protection from global passive network observer. Briar uses mesh networking and net split is totally normal situation. Branched chats. Decentralize all the things. Let's hope Briar adds alternate key exchange, now they only offer QR code and Bluetooth. It's underestimating users that they assume that there can't be other secure ways of exchanging keys. "You must meet up with the person you want to add as a contact. This will prevent anyone from impersonating you or reading your messages in future". That's just making it hard for users.
  • Electromagnetic Threats for Information Security - Ways to Chaos in Digital and Analogue Electronics. Of course I can start by saying, this is nothing new at all. But let's see if they come up with anything really interesting or new. Electromagnetic & RF communications Security. TEMPEST was mentioned. On high level this of course leads to Electromagnetic Warfare (EW). Attack rating: availability / cost, dimensions / mobility, capabilities. Required technical knowledge. Effective range, target knowledge, target specific? Effects detection, effects classification, impact estimation, propagation chain estimation. Radiated / conducted, coupling front-door / back-door and so on. Lots of basics, let's hope they've got a great demonstrations. Well, just some quite boring demonstrations. But this is important aspect when doing Information Security (InfoSec) and  functional safety risks analysis and planning required countermeasures.
  • The Noise Protocol Framework - Secure channel protocols, TLS, IPsec, SSH. Two parties online, atuh + key agreement. Authenticated Key Exchange (AKE). Forward secrecy, mutual or one-way authentication, preknowledge of identities, type of crytpography used (signatures, DH, encryption). Signatures for authentication and Diffie-Hellman for key exchange. Lots and lots of basics. Noise Protocol - the project link. The talk didn't contain anything  new at all, just lost of basics and truly classic stuff. Read this The Noise Protocol Framework instead of watching the video. Yet event that document doesn't contain anything new, because it was on way too high level without enough technical details.
  • Forensic Architecture. Timing and locating things from video / photographs. Time space relations. Correcting invalid meta data. Measuring items in video / photographs. This is also very important to remember, if you think you're "anonymizing" video / photos. Data and information leak is inevitable, unless you're extremely careful with all information being sent out. In reality, you should always assume that information will leak.

CDNs, Postgres, JSON, datetime, PS tail, WebSites, Fish Shell, User Identification, Scrum / Agile / CI

posted Apr 15, 2018, 9:09 AM by Sami Lehtinen   [ updated Apr 15, 2018, 9:13 AM ]

  • Are CDN businesses being consolidated? Highwinds -> StackPath, NetDNA -> MaxCDN, MaxCDN -> StackPath. Sounds like there are too many CDN operators and those are being consolidated as usually happens when business starts to mature. Yet operating smaller CDNs with cost effective hosting can be a market sector which doesn't interest the larger players with higher price point and service level(s).
  • Why Use Postgres - Nothing to add. Postgres is awesome, that's why I love PostgreSQL. JSONB is awesome, because often manual compex JSON -> SQL mapping is just simply painful. GIN and GIST indexes are used for full text and geospatial indexing. Also check out PostGIS and OpenGIS. Upsert is also nice, yet with simple use cases I can't stop loving SQLite3 Replace. Which does update / insert fro whole row based on primary key. It's just incredibly handy. You'll read / create data and then just store it, not caring about if it's already in the database or not. I like that workflow because for update you often need to read the data and verify some things anyway. After that logic storing the updated data back, should be as simple as possible.
  • So much tuning with different kind of JSON APIs and working between Python and Go, well. After all it worked out as expected. Good project.
  • Sigh, datetime with Python is such a mess, with timezones. time.time() is nice, as well as time delta. But datetime timezone is so messed up. But I'll manage. This is one of the reasons why I always prefer using UTC time stamps. The only sane way to go. It seems that there are annoying variations of the ISO time stamp. Some prefer to separate timezone hours and minutes with : and some others don't. Sigh. Also the standard strptime function doesn't detect 'Z' as valid timezone / UTC offset.
  • Tail -f using PowerShell: Get-Content -Wait -tail 50 -Path logfile.etc
  • That will nicely list 50 last lines and any new lines getting written to the log. That's very useful when monitoring logs in near real time.
  • Cyber weapons being leaked. Who's trolling who. Great question and a good blog post by Schneier.
  • I just seriously hate websites which do strange assumptions. Like DB X-Trackers ETF site. They assume that if you're English client, you would like to use Pounds. Or if you want to buy ETF from XETRA you would prefer German language. How annoying is that. What if I want to have my stuff in English and I want to buy stuff from Xetra with EUROs.
  • Switched my default shell from bash to fish shell. In the good old days I used to use zsh, but that's decades ago. Also upgraded default Python version to 3.6, which is nice. It seems that the fish adopts really much stuff from Python. That's no surprise. I've often used python script to generate shell or SQL scripts. It's always good to use the tools you know very well.
  • One of the reasons why many sites aren't using strong official identifying authentication is costs of it. First of all, you have to setup secure infrastructure, make integrations, and even then you'll need to pay running costs of that system. So far in Finland the authentication cost has been 0.50€ / authentication. But now it has been limited to maximum of 0.10€ / authentication. That's still quite high for many use cases. I know that even many major players have provided official authentication just for a while, while acquiring new customers. After that they've reverted back to classic user/pass login, because it's just so much cheaper! I just would love to get 50 cents on every Facebook, Google, Twitter logic. I'm also curious if there will be new service providers entering the nationally rusted official legally binding authentication market in Finland. If you think this is silly, I've heard there are still countries which use methods like utility bills for identification and other utterly ridiculous methods. - It was just lately in news that government is looking to reduce on-line user identification costs, because it's costing in range of several millions per year.
  • There are now news that Scrum is too slow. That's just one of the reasons why I haven't been using it with projects for a long time. That's why continuous integration (CI) is being used. We've received a lot of feedback from happy customers due to CI practices. Why? Well in some companies getting things done takes 6 months, in some companies it takes around 4 - 2 weeks due to Scrum latency. But when things get serious and the speed is the key, CI is there to save you. In last integration project it took usually just a few hours to get stuff pushed into production when customer confirmed clearly what they want. Why software changes should take weeks, or months? Especially in cases where changes aren't that big. As I've written so many times, it's clear that whenever something is created, multiple iterations are required. This project isn't yet complete, but we've already logged 173 versions of the integration software deployed in production. What if the delivery latency would have been longer, due to some archaic release cycles or even two weeks due to Scrum? In some cases when there's a hour long Skype for Business meeting. If something is agreed in the meeting, the requested changes have been already deployed during the meeting, and it's possible to get second and or third iteration too within that hour. No problem! And trust me, the customers do value this.

DNS performance tests - Cloudflare, Google, ISP

posted Apr 8, 2018, 12:26 AM by Sami Lehtinen   [ updated Apr 8, 2018, 12:27 AM ]

Run DNS server benchmark. Well sure, Cloudflare ( is doing great, but ISPs servers aren't that bad either, as usually is being claimed. People often claim that Google DNS is great, but it isn't. Many of the other "better alternatives" are also horrible compared to ISPs or Cloudflare's DNS servers, and also provide consistently bad performance. In this case the Telia DNS servers are the old Telia DNS servers 18.18 and 19.19 which have been used for long time. And it seems that for fiber connections the DHCP delivers alternate DNS servers, which are the two last DHCP servers. As far as I can see, the ISP DNS isn't bad at all, it's actually quite good. For local public DNS tests, I also included DNS which got quite an deviatio, because ICMP Echo time (PING) is just 4 ms.

                  test1   test2   test3   test4   test5   test6   test7   test8   test9   test10  test11  test12  test13  test14  test15  test16  test17  test18  test19  test20  Average
cloudflare        1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms      1.00
level3            34 ms   34 ms   31 ms   33 ms   34 ms   34 ms   31 ms   31 ms   31 ms   32 ms   32 ms   123 ms  84 ms   34 ms   48 ms   34 ms   38 ms   36 ms   33 ms   32 ms     40.95
google            9 ms    8 ms    9 ms    20 ms   29 ms   134 ms  8 ms    20 ms   9 ms    9 ms    117 ms  34 ms   19 ms   50 ms   116 ms  31 ms   54 ms   30 ms   30 ms   109 ms    42.25
quad9             27 ms   30 ms   29 ms   31 ms   30 ms   30 ms   28 ms   27 ms   29 ms   30 ms   30 ms   47 ms   59 ms   83 ms   238 ms  30 ms   57 ms   34 ms   31 ms   194 ms    54.70
freenom           30 ms   30 ms   30 ms   30 ms   30 ms   155 ms  30 ms   30 ms   30 ms   31 ms   152 ms  33 ms   69 ms   67 ms   46 ms   47 ms   82 ms   60 ms   68 ms   242 ms    64.60
opendns           57 ms   16 ms   17 ms   36 ms   33 ms   32 ms   17 ms   43 ms   42 ms   17 ms   75 ms   119 ms  39 ms   17 ms   45 ms   39 ms   36 ms   36 ms   38 ms   143 ms    44.85
norton            32 ms   33 ms   29 ms   28 ms   30 ms   32 ms   29 ms   30 ms   27 ms   32 ms   30 ms   29 ms   30 ms   29 ms   30 ms   29 ms   31 ms   28 ms   28 ms   32 ms     29.90
cleanbrowsing     30 ms   30 ms   30 ms   31 ms   28 ms   30 ms   33 ms   30 ms   32 ms   29 ms   40 ms   35 ms   29 ms   29 ms   28 ms   46 ms   42 ms   41 ms   31 ms   32 ms     32.80
yandex            18 ms   2 ms    19 ms   18 ms   26 ms   107 ms  2 ms    66 ms   18 ms   50 ms   170 ms  20 ms   164 ms  74 ms   90 ms   185 ms  96 ms   84 ms   33 ms   69 ms     65.55
adguard           38 ms   32 ms   30 ms   30 ms   35 ms   30 ms   30 ms   30 ms   30 ms   34 ms   36 ms   39 ms   30 ms   30 ms   30 ms   45 ms   36 ms   185 ms  31 ms   30 ms     40.55
neustar           29 ms   29 ms   32 ms   31 ms   29 ms   29 ms   29 ms   30 ms   30 ms   29 ms   29 ms   30 ms   29 ms   30 ms   30 ms   29 ms   29 ms   29 ms   29 ms   29 ms     29.50
comodo            44 ms   46 ms   44 ms   45 ms   60 ms   45 ms   45 ms   44 ms   42 ms   45 ms   76 ms   60 ms   43 ms   46 ms   44 ms   56 ms   194 ms  67 ms   66 ms   44 ms     57.80
telia dns 1       2 ms    1 ms    1 ms    2 ms    1 ms    1 ms    1 ms    1 ms    2 ms    2 ms    1 ms    4 ms    60 ms   1 ms    1 ms    1 ms    33 ms   1 ms    2 ms    1 ms      5.95
telia dns 2       2 ms    2 ms    2 ms    10 ms   2 ms    2 ms    2 ms    2 ms    2 ms    2 ms    2 ms    6 ms    2 ms    2 ms    43 ms   2 ms    2 ms    2 ms    2 ms    2 ms      4.65           4 ms    43 ms   329 ms  4 ms    200 ms  143 ms  43 ms   23 ms   4 ms    79 ms   55 ms   9 ms    4 ms    4 ms    4 ms    245 ms  292 ms  272 ms  4 ms    4 ms      88.25
telia dhcp 1      1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    2 ms    1 ms    1 ms    2 ms    1 ms    4 ms    1 ms    2 ms    2 ms    2 ms    2 ms    1 ms    1 ms    1 ms      1.45
telia dhcp 2      1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    2 ms    1 ms    2 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms    1 ms      1.10

Yet currently it seems that the Cloudflare DNS does break some DNS based geolocalization features. So even if the DNS lookup is fast, the client could be directed to servers which aren't optimal for rest of the process due to high latency and lower bandwidth.

kw: DNS performance, Finland, results, test, testing, tests, latency, delay, Telia.

3D printing, CORS, CSP, XHR, I Spy, Aurora, Net Neutrality, HSTS, Bits vs Bytes

posted Apr 8, 2018, 12:11 AM by Sami Lehtinen   [ updated Apr 8, 2018, 12:12 AM ]

  • FreeCAD, Repetier, Slicer and Cura. I've been working with those during last few months and phew... So many bugs I can't even believe it. Almost everything can bug extemely annoyingly. When you change something, something disappears. Programs crash. Even basic boolean operations fail. Objects which are ok in editor, aren't ok anymore when exported for 3D printing etc. So it's software as usual, it's not working well. Well working code is actually quite rare occurrence. Bad code and bad software is the norm.
  • Latest version of duplicati isn't called experimental anymore. That's awesome news. It's becoming stable and production ready? That's great. I've got just tons of servers and other systems waiting for it. Maybe I'll setup it next time on my home systems too, when I would need to run full backup next time. So that will happen in a month or so.
  • is losing even more email. Duh, this happens all the time. Funniest thing is that I can see the mails in sent folder but the mails never arrive at the destination. When I got the right mood, I'll migrate to some other service provider. Outlook sucks way too much. Outlook SMTP confirms that email has been received, but it never appears in inbox. They have some kind of hidden censorship engine installed. Who would trust that kind of messaging service?
  • Reminded my self about Cross-origin resource sharing (CORS) and Content_Security_Policy (CSP) for one project where I needed those. CSP is a whitelist which lists allowed data sources for the site to be loaded. This can be very easy or hard, depending what kind of mess you're running. In best case the source is the server itself. And if required CORS can be used to allow sending cookies to other domains to request authenticated resources from cross-domain (XHR) resources using Ajax etc. Most of projects I do, only use the site itself as source, and only potentially load a few libraries from some CDN. Which is easy to specify with CSP and doesn't require CORS. Because I prefer making sites secure and private.
  • I Spy - Funny story about how paranoid security agencies were and probably are. Well, good luck wasting resources.
  • Deleted my experimental Brand Account @ Google+
  • Studied products from Retail Next and their Aurora product. Very interesting stuff. But as we know this of surveillance and tracking technology has been creeping in for a long time. Actually it's quite easy to spot the tech being used in local malls, if you just look carefully.
  • Watched Digits Series Season 1. A very nice set of documentaries about information technology's past and future. Where we are heading? Only the future will tell.
  • News about: Network attacks utilizing subcontractors as ingress route. Yep, we're all very aware about that risk, nothing new.
  • War on Open Internet - Absolutely awesome and thought waking post about Internet business. Yet it's not actually anything new at all. We've been heading to this direction for a long time? How someone can send me a message over the Internet that Internet isn't working. Well, they weren't talking about Internet, they were talking about Facebook or Google, because some people seem to think that those are the Internet. Are we losing Net Neutrality?
  • Watched a documentary of 2016 year in science. As well as nice documentary about all the problems related to Brexit from this point on.
  • Telcos and chronic lying and spreading customer disinformation. One operator now advertises that they provide 1 GB speeds. When they should say they provide 1 Gbit/s speeds. Once again the question is, do the telcos just got totally incompetent staff Do they think their customers are braindead or are they just being whatever and bleeping their customers on purpose. They can choose any of these three options. Right attitude, right? Positive customer service experience. Sometimes trolling is irresistible fun, but trolling paying customers might not be a good idea.
  • Nice HSTS post by Julia Evans - Nothing new on it of course. But if you're not familiar with HSTS it's a good read.

34C3 - TUWAT - Talks - Part 6

posted Apr 8, 2018, 12:02 AM by Sami Lehtinen   [ updated Apr 8, 2018, 12:02 AM ]

  • Tightening the Net in Iran. Interesting topic. Yet - Internet censorship - on different levels is being done in most of countries. They call their Internet as Filternet. That's funny. Cyber Crime Law. Logging all network traffic. Nice samples of different versions of the censorship page being shown. Localized content for absolute access and content control. Incentives for developers to create local / national apps. Removing apps which do not co-operate with censorship. Telegram is very popular app in Iran.
  • Growing Up Software Development - Nice programming lessons,yet nothing new. Pretty joyable stores after all.
  • Humans as software extension - Bit strange talk, let's see if there's anything actually interesting. - Using humans as bots. Quite interesting concepts, but nothing new really.
  • How Alice and Bob meet if they don't like onions - This should be pretty nice Survey of Network Anonymization Techniques. Let's see if it got any new information. Interestingly JonDonym is something I haven't heard or noticed earlier. But it seems to trying to be commercial Tor, so no thank you. Some alternate networks mentioned. They also mentioned older networks like Freenet, GNUnet, I2P, Tor. I haven't heard either, like The Loopix Anonymity SYstem and AN.ON Next Generation. I think I've checked out the Vuvuzela network in past. Side track: USENIX Security 2017 talk about Loopix. Talk also mentioned Freenet caching and distributed storage, which is one of the features, I really liked about it. They also mentioned GNUnet's F2F mode. They also mentioned that none of these services protect against global passive observer. Freenet and GNUnet provide plausible deniability. My brain hurts, amount of hype bs... About Q&A section and especially the Q part, not A part or the talk itself - IPFS uses blockchain, argh! GNUnet and Freenet are quite similar to IPFS. No, this is just what I hate about some things. Please check your timeline. IPFS is quite similar to GNUnet and Freenet, not the other way around. How about just saying content addressable distributed storage, which is way older concept than Freenet / GNUnet and so on. Too hard for you guys, is it, really? Cached with blockchain algorithm, what kind of ... my mind is blown. Now please someone, could you explain Blockchain caching algorithm, please. Because me or anyone else, isn't aware about it. - This is exactly what put me off with IPFS originally, straight out lies and hype. How about sticking to the facts? Nothing wrong with the technology, but I'm extremely allergic to unproven marketing hype and clams which could be defined to pure lies without technical proof and documentation how it's achieved. Back to the talk, in the Q&A section they well explained the Relationship Anonymity which can reveal users identity to global passive observer. Just like the water "bubbles" in water hose in the cartoons. Flows in and flows out. If statistics isn't enough alone. Using active (indirect) measures, you can cause latency / bandwidth changes and so on, making the proof much stronger.
  • Loopix Anonymity System talk at Usenix Security Symposium (2017 - 26th) - Talk - Explained Mixnets Background. Mixnets add latency and with cover traffic limit scalability. Onion-routing isn't resistant against global passive adversaries. Loopix allows tunable trade-off between latency and genuine and cover traffic volume. I think this sounds very much like GNUnet which allows similar kind of option. Off-line message storage, inbox (queue) in situations where messages can't be delivered to the recipient. Drop cover traffic, loop cover traffic. Uses bi-directional cover traffic, in and outbound messages. Loop traffic allows detecting n-1 attack. It seems that Loopix is addressing many of the issues I've blogged about several times. Poisson mixing, exponential delay, no synchronization rounds required. Memoryless property of the mixing pool. Loopix does address global passive adversary, corrupt mixes and corrupt providers. Yet the corrupt provider is most important, because it will break receiver unobservability and receiver anonymity. Nice performance and throughput analysis. Partition attacks and Denial of Service attacks discussed in QA. It was also obvious that it's research network currently. Many of the "practical" aspects, weren't covered.

exFAT (Extended File Allocation Table) vs UDF (Universal Disk Format) - File System for Flash Drives

posted Apr 2, 2018, 11:03 PM by Sami Lehtinen   [ updated Apr 2, 2018, 11:06 PM ]

Simple tests exFAT vs UDF on Flash Drive, performance, compatibility, just for fun.

Test data set & results

Large file: 1 x 1 GB
Small files: 10k x 1 KB = 10 MB
Time Unit: minutes:seconds.second_decimals
Large file: 2:24.90
Small files: 1:07.81

Large file: 0:41.06
Small files: 0:14:25

Large file: 0:00.90
Small files: 0:22.27


Large file: 2:20.77
Small files: 0:18.96

Large file: 2:11.32
Small files: 0:19.10

Large file: 0:00.36
Small files: 0:08.44


After every test all IO buffers and caches were flushed (and it's included in the time) and cleared (no data re use from cache)
All test steps were repeated three times, and results were consistent enough and the values shown are the median values.
Some of the results are so confusing, I've gotta recheck... Let's verify...
Just to be sure, I recreated all the tests using another, but 100% similar storage device.
After all, it's clear that the file systems behave very differently in different situations. Yet, the absolutely most interesting part was the large file read performance. How it can differ so much with same storage media? And repeated tests confirmed that. Amazing and confusing at the same time.
Oh one more remark, the exFAT device was using 4K cluster size, which is disadvantage for exFAT small files handling. UDF uses 512 byte "blocks" on the used device. So this can explain why the small files read is bit slower on exFAT.
The flash drives used for testing are drives which are generally quite slow, but provide good random access performance. Results could be different with some of the Flash Drives, which are fast, but provide devastatingly poor random access performance, especially when writing.


There's no silver bullet or simple answer. Both solutions do work, and provide different performance characteristics. Yet probably which ever you're using, you're not using it inherently for performance reasons, but for compatibility. Anyway, if exFAT works for you, use it. As far as I can see, there's no clear benefits for using UDF.
Reasons why I bothered even to check out UDF, were the fact that exFAT free space bitmap gets often corrupted (lost disk space) and you can't get it fixed on Linux without reformat. Yet I also managed to corrupt UDF on unclean dismout tests quite quickly, ending up with file which couldn't be read or deleted. Just as with exFAT there's no way to fix this, except to format the device again.
Other more or less interesting stuff? I created the UDF systems on Linux using mkudffs and when I run chkdsk on Windows 10, the results were curious:
With partition table, yet Windows removed partition table from USB drive when chkdsk was run ... This is amazing...
Also all files written with Linux were read-only files when being used with Windows, but it was enough to remove the read-only flag.

Tech stuff for nerds

On this device, I did create MBR and partition with UDF. Windows removed the partition table. What if I would have had multiple partitions on the device? I didn't test that, but it could be ahem, quite bad end result. Anyway, in both of cases the list of things getting corrected is quite scary. It's horrible to see, how there can be so different views how UDF should be implemented and operated.

The type of the file system is UDF.
Logical Volume Integrity Descriptor at block 273 is corrupt.
Creating new Logical Volume Integrity Descriptor at block 15654592.
Reserve Volume Descriptor Sequence is corrupt or unreadable.
Volume MBRPRT is UDF version 2.01.
CHKDSK is verifying ICBs ...
ICB verification completed.
CHKDSK is looking for orphan ICBs ...
Search for orphans completed.
CHKDSK is verifying ICB links ...
ICB link verification completed.
CHKDSK is verifying link counts and parent entries ...
Link count and parent entry verification completed.
CHKDSK is checking system files.
CHKDSK is checking the directory tree for cycles.
CHKDSK is identifying lost files.
Lost file identification completed.
CHKDSK is verifying object size for ICBs with alternate data streams ...
ICB object size verification completed.
Correcting errors in Space Bitmap Descriptor at block 0.
Correcting directory count to 1903 in Logical Volume Integrity Descriptor.
Correcting file count to 19789 in Logical Volume Integrity Descriptor.
Correcting partition size for partition 0 to 15654317 in Logical Volume
Integrity Descriptor.
Correcting partition free space for partition 0 to 12144363 in Logical Volume
Integrity Descriptor.
Creating Reserve Volume Descriptor Sequence at sector 15654841.
Correcting errors in Unallocated Space Descriptor in the Volume
Descriptor Sequence.
Correcting errors in the Volume Descriptor Sequence.
Correcting errors in Anchor Volume Descriptor Pointer.
Windows has made corrections to the file system.
No further action is required.
   7827158 KB total disk space.
   1750573 KB in 19789 files.
      2474 KB in 1903 directories.
         0 KB in 1 stream directories.
      1929 KB in use by the system.
   6072181 KB available on disk.
       512 bytes in each allocation unit.
  15654317 total allocation units on disk.
  12144363 allocation units available on disk.
With this device, I created pure UDF file system directly on the block device without partition table as UDF is supposed to be created.

The type of the file system is UDF.
Logical Volume Integrity Descriptor at block 273 is corrupt.
Creating new Logical Volume Integrity Descriptor at block 15646400.
Reserve Volume Descriptor Sequence is corrupt or unreadable.
Volume UDFPURE is UDF version 2.01.
CHKDSK is verifying ICBs ...
ICB verification completed.
CHKDSK is looking for orphan ICBs ...
Search for orphans completed.
CHKDSK is verifying ICB links ...
ICB link verification completed.
CHKDSK is verifying link counts and parent entries ...
Link count and parent entry verification completed.
CHKDSK is checking system files.
CHKDSK is checking the directory tree for cycles.
CHKDSK is identifying lost files.
Lost file identification completed.
CHKDSK is verifying object size for ICBs with alternate data streams ...
ICB object size verification completed.
Correcting errors in Space Bitmap Descriptor at block 0.
Correcting directory count to 1903 in Logical Volume Integrity Descriptor.
Correcting file count to 19789 in Logical Volume Integrity Descriptor.
Correcting partition size for partition 0 to 15646125 in Logical Volume
Integrity Descriptor.
Correcting partition free space for partition 0 to 12136683 in Logical Volume
Integrity Descriptor.
Creating Reserve Volume Descriptor Sequence at sector 15646649.
Correcting errors in Unallocated Space Descriptor in the Volume
Descriptor Sequence.
Correcting errors in the Volume Descriptor Sequence.
Correcting errors in Anchor Volume Descriptor Pointer.
Windows has made corrections to the file system.
No further action is required.
   7823062 KB total disk space.
   1750573 KB in 19789 files.
      2218 KB in 1903 directories.
         0 KB in 1 stream directories.
      1928 KB in use by the system.
   6068341 KB available on disk.
       512 bytes in each allocation unit.
  15646125 total allocation units on disk.
  12136683 allocation units available on disk.

kw: USB, flash, performance, benchmark, test, testing, reading, writing, windows, linux, exFAT, UDF, file system, fs, flash drive, drives, exFAT vs UDF, UDF vs exFAT, comparison.

Fintec, Scaleway, Enum, AsyncIO, Aura, F-35, Concurrency, SQL, Pulseaudio

posted Apr 1, 2018, 12:35 AM by Sami Lehtinen   [ updated Apr 1, 2018, 12:36 AM ]

  • Studied real-time open payment ecosystem and subscribed newsletter (e-commerce, fintec). I don't usually like attending meetings, but sometimes I do. If travel time is very reasonable and the agenda of the meeting is interesting enough. kw: PSD2, P2P, PFM, ARPP, eCommerce, In Store Payments. 
  • Scaleway released ARMv8 based servers. Seems to be powered by Cavium ThunderX SoC ARM64 Processors with 48 cores, those CPUs also support dual socket, which would sound like a good idea for hosting.  All of my Python software naturally runs just as well on ARM as it does on X86, I really don't care. Maybe I'll take a spin.
  • Studied Python 3.6 enum - , yet I'm not 100% sure yet, what the benefits are compared to dictionary. As usual, similar functionality can be achieved by multiple different ways. Which way and why is the best, is awesome question. Yet one feature which is something I've been actually looking for is the IntFlag, which allows efficiently bit mapping information and still making easy comparisons, without calling separate masking / comparison function or method.
  • Write Fast Apps Using Async Python 3.6 and Redis - Nice article. Unsurprisingly the Redis Python is very similar to Peewee ORM which can be used with SQLite PostgreSQL etc. Just like Flask is very similar to Bottle. Still haven't had a change to develop something which would seriously use asyncio. But I'll try to utilize that with some project which would actually benefit from it.
  • Aura Home Security System - That's interesting one. This one is just cheaper version for home users of the matrix / matrix detection devices I've blogged earlier. kw: Tomographic motion detector, Cognitive radio, Radio spectrum.
  • When F-35 (A) stelth fighters flew to Estonia, there were several RC-135 (U & W) planes monitoring the Russian signaling. Afaik, that's obvious action. Many people are sometimes surprised by some operations, but if you would be running those operations many actions would be natural. This is just what we do, and it's pretty much standard procedure. If you assume that the adversary is using their most valuable assets, then you should do the same. In this process when they try to acquire information, you'll be acquiring it too. This is also the reason why using advanced cyber weapons is a huge risk, because if the adversary is prepared, you'll be just giving your weapons to them. As we've seen happening. Just wondering if Finland used that change to operate our CASA C-295 - SIGINT plane with Dragon Shield.
  • Go Concurrency Visualize - I think I've covered this same topic several times over. But there's the model where subworkers are being used. That's the model I'm typically using with Python. Because I run several processes, which each runs it's own threads. This is due to Python GIL mostly. So I have to balance suitable count of processes and threads to achieve maximum performance. Number of threads usually depends from underlying latencies and number of processes is usually 1-2 x the number of cores or threads on the system. Sometimes I do use one dispatcher thread which passes work to all other sub processes and threads. But more often I'm using model where the processes are independent and control threads. Basically this model also allows running those processes on several different computers and or NUMA systems efficiently, because there's no need for SMP.
  • SQL Server 2017 will allow Python scripting as native T-SQL stored procedures. That's awesome. Love it! Gotta test it as soon as it's widely available.
  • Finally found reason for that darn audio distortion in Ubuntu which is so annoying that it seems more like trolling code. Yet no need to reboot whole system anymore, just killing the pulseaudio does the trick. pulseaudio -k and it's fixed (for a while). I just wonder why the root cause doesn't get fixed. Afaik, I think it was working for a while. Because after latest updates I've been experiencing this audio distortion problem much more often than I used to.

34C3 - TUWAT - Talks - Part 5

posted Apr 1, 2018, 12:26 AM by Sami Lehtinen   [ updated Apr 1, 2018, 12:26 AM ]

  • The Snowden Refugees under Surveillance in Hong Kong - Flashbacks, I think these are the talks which are given every year even if there wouldn't be actually anything new to add to that story.
  • Financial Surveillance - This should be quite interesting. Risk Management solutions via Risk Intelligence using profiling of heightened risk individuals and entities globally. Detecting international money laundering and terrorism funding. "World-Check.json", data structure, format description and content description. "World Check - Reducing risk through intelligence". Detecting political corruption and bribes. Data source quality was highly questionable. Proper data filtering and verification completely missing. Stale data in the database. Competing data providers: LexisNexis and Dow Jones. - Good talk, system is clearly flawed.
  • Holography of Wi-Fi radiation -. This is interesting talk, let's see how they've done it. Even if technically it should be "simple". Wi-Fi is like laser, but with longer wave length. Yep, electromagnetic radiation (they said waves, what about duality? They also mentioned Microwave Light). Also good reminded about light / photon holography. Phase-coherent recording. Wi-Fi based holography. Microwave camera, phase array, amplitude array, signal correlation. Phase delay and attenuation recorded using reference signal. Building imaging view using data from signal using numerical reconstruction and virtual focus plane. Using multiple frequencies in parallel together even more information. Spying and imaging using Wi-Fi as security attack talked a lot. Wi-Fi provides poor resolution, using dedicated Ultra-Wideband (UWB) Radar  emitter is much better. Walabot mentioned. Tracking radio emitters in buildings. With higher computing power full tomography is possible. In door localization and tracking is well possible using receiver network.
  • Bulletin Board System (BBS) and early Internet access (1990's) - This is something which I don't need to listen talks about. But I still liked to listen it. Let's see if there's anything worth of commenting. I remember that I personally had problems compiling Telegrad BBS software with Turbo Pascal, because my system was running out of RAM. I guess it was the largest piece of software I ever compiled with Turbo Pascal. There were also strange problems, like "lines out" from local exchange being all full. Does it sound familiar that "uplink" is oversold? Also at one point they added DTMF support, but it was really funny, because it was only support for it. No they didn't process the DTMF signals, they had DTMF converter! So even if you used DTMF to dial the number, it actually was just as slow as using pulse dial, because there was just a module which converted that DTMF back to pulse dial. Yep, even in good old days they had all the tricks to sell you something as being more advanced than it really was. Cool the talk showed Smartmodem, (US Robotics OLD) Courier, there was also the newer Sportster and of course the legendary ZyXEL U-1496E v.32bis with v.42bis support. Nice hardware in the photos. Proprietary protocols like US Robotics HST I think it as 14,400 / 2400 bps. But it could actively renegotiate the faster link to any direction, without dropping the connection using "retrain". Some modems were able to change link speed without retrain, but most of modems couldn't do that. Talker seemed to like Telemate, I used Telemate a lot. They also mentioned TELIX. Also Terminate was popular in later times and before Telemate many used Procomm Plus. Telemate & Terminate allowed using Avatar instead of ANSI which wasted lot of bytes when changing colors, repositioning cursor etc. Telemate was nice because it allowed light co-operative 'multitasking' like reading documents while downloading files, or running shell prompt commands, etc. FIDOnet point, polling, offline messaging formats like Bluewave and QWK mentioned. All so familiar stuff. FIDO (Netmail + Echomail) later also UUCP (Usenet Mail + News). Fidonet hierarchy, Regional routing / addressing introduced in 1986. But the sample address they used didn't contain point. 2:2490/1343 but point could have been added to it with .23 which is leaf of node usually operated by an individual. Many BBSs worked as Internet gateways via shell, providing shell and slip or using slirp from shell. And Trumpet Winsocket and NCSA Mosaic of course! And this is what lead to slow but guaranteed death of BBS systems. UUencoding. No multitasking? Hmm, that's not true, we had - DESQview - - of course. POTS / ISDN / PCM. ISDN Terminal Adapter, 64000 bps or 128000 bps (using dual channel). Lots of talk about Internet connectivity providing regional member associations, etc. They had really nice network diagrams in the talk. DOOM mentioned. - Basically this is just like the networks in Cuba. Owned and operated by individual users whom network together.

Web Security, Learning, Functional Requirements, SQL, APFS, EM-Drive, TSQL / MSSQL CXPACKET, Projects

posted Mar 25, 2018, 3:41 AM by Sami Lehtinen   [ updated Mar 25, 2018, 3:41 AM ]

  • Cross-origin resource sharing (CORS). It's being used in a few projects, but I'll need to fully study and implement it. Security features like SRI & CORS & CSFS protection can't be ignored. Subresource integrity (SRI), Cross Site Request Forgery (CSRF).
  • Thematic anaylsis. Sounded pretty much something which should be completed using machine learning, unless data is pretty static, analyzing system got very limited resources and high amounts of data are being processed.
  • Life long learning, learning by doing, reading from net versus expensive formal education. That was quite an interesting article. "You can become an expert in anything you choose." - Just wondering how unemployed people aren't using their time to study stuff and develop themselves.
  • Microsoft licensing. It's horrible, absolutely horrible. It's also very hard to get information how to get things right. Yet another reason why I haven't been using any Microsoft products for a decade. No wonder you'll need expensive specialist staff or outside experts to take care licensing, bringing extra cost to already expensive licences.
  • Gone through hundreds of pages of different functional requirements for one interesting project. I really enjoy cases, where customer can present clear requirements documentation.
  • Laughed how bad Microsoft product naming is. Today I've seen several emails which talk about Skype. Which is of course total fail, when it should say Skype for Business. - But we all know how badly Microsoft handles matters like this.
  • One document said that messages are store into SQL queue. Wasn't that classic anti-pattern? - I've actually personally argued that it's totally ok to use SQL database as queue. It that sounds too bad, you can also use directory with files, it has been proven to be a working solution for decades. Or just large append files which are rotated when fully processed. This is more efficient than individual files for small messages.
  • A nice post about APFS filesystem format - But there are still plenty of details missing. Anyway, databases, file systems, memory allocation, etc. Are very interesting topics and as usual, when designing those everything is a trade-off. When you make something faster, you might make some other use cases a lot slower. I've been often wondering why file systems won't use small file packing. Yet 'inlining' is one way to get that done. Another way to get kind of same result, but with seek, would be using single cluster with offset to store data. Yet as it seems storage is growing so fast, that there's no need for that mostly. It's more important to manage large files and free space allocation even more efficiently. Can waste a ton of storage space, if small files are being written.
  • EM Drive is working? - Confirmed by NASA peer rewview. Hmm, that's interesting. Does this mean that Newton got the third law wrong?
  • Analyzed TSQL performance issues on MS SQL Server. It seems that the CXPACKET is the worst bottleneck right now. Of course this is very expected situation. There are certain database access related issues, which always arise on every project, unless especially dealt with. Related links: CXPACKET wait statistics, Parallel Execution Plans, Troubleshooting the CXPACKET, CXPACKETs not your performance problem. Also reminded my about clustered indexes.
  • Interesting article about Fusion Reactors. What kind of problems there are to be expected? Some claim this article has been totally debunked, but afaik interesting points after all.
  • Response to how to never complete anything - This is actually very good post, and among the lines I've written about again and again in my blog. Another interesting related response post: How to complete a side project.
  • is now shutdown and domain has expired. When I run last archive run with wget, it seemed that very high speed full mirroring of the site consumed about 10% of CPU time (after disabling rate limits). So it worked very efficiently. I'm quite happy about that. It was a good experimental project and I learned a lot while creating and maintaining the site.

Outlook, Onecoin, Atomicity, OWASP, SoftEther, SSTP, BBS, IPv6

posted Mar 17, 2018, 10:54 PM by Sami Lehtinen   [ updated Mar 17, 2018, 10:54 PM ]

  • Did more tests, it seems that Outlook is actually losing emails. I'll do some final configuration changes, and if that's still true. I have to change service provider. I got one idea why that's happening, but I need to dig a lot of logs through to see if that's really the cause. Two main suspects based on my experience are their spam filtering and SPF rules. This is also one of the reasons I didn't use spam filtering on my own mail server at all. Because there's no way to tell if something is spam or not. - Outlook seems to sometimes filter mails covertly, so that the sender nor the recipient gets any information about the filtration. - Sigh.
  • Outlook 2FA OTP login seems to be again failing exactly once, but second attempt always works. This is ridiculous engineering.
  • Read a few posts about Onecoin cryptocurrency (scam). Well well, future will tell. Luckily I'm not involved in anyway.
  • Had once again, long discussion how to handle atomic transactions (atomicity) correctly or as two phase commit, so there's always clear roll backward, roll forward separation with everything. It's surprisingly common, that programs do not work properly. Correctly written program can be aborted at any time, and restarted later and that shouldn't cause any problems. Same also applies to parallel instances. It shouldn't matter if there's one or billion instances running, the end result should be exactly same.
  • OWASP vuneralibities report 2017 included Unauthenticated API's. - No surprises there. It's quite common that only some parts of API are authenticated, and some other parts might completely lack proper authentication or rights management.
  • Studied SoftEther VPN - It's a nice free Open Source VPN solution, but currently I don't have any use case for it. The most beneficial feature is that it supports multiple protocols: SoftEther VPN Protocol, L2TP/IPsec VPN aka L2TP over IPsec, MS-SSTP VPN Protocol (PPP over HTTPS) and of course the very traditional OpenVPN Protocol over TCP/UDP. As example MS SSTP is really easy to use with Windows 10 workstations without installing additional clients, which makes it practically pretty handy. Also options to use bridging / routing are nice including IPv6.
  • This very nice BBS post - Reminded me from yet another configuration fail with one BBS. They had configured 60 minute user time limit / session. But the program used incorrectly DTE to DCE for time estimates and not DCE to DCE speed. Because I had 14.4k modem and the DTE to DCE speed was 115.2k I could hoard stuff to be downloaded for roughly 8 hours or even much longer, if connection quality was bad. This worked for quite a while, before I got locked out due abuse. Some data transfer applications also got the minutes remaining information from BBS system, so those would disconnect user, if download was taking too long. But most of systems didn't do that. Another way to hog down the system for very long times, was to upload something. Most of systems didn't enforce time limits on uploads. So you could first download and then upload for a long time. Of course this was beneficial to the BBS because that's the way systems got the content they were providing to users.
  • Lack of IPv6 addresses. - Say what - Yes. Some ISPs provide just one /64 per customer. At home, I'm using Telia 6rd and I've got several separated networks here. The key to dealing with that is the router which you're using. There are several ways to make work-a-rounds like you said. Depending what kind of router you're using, you can configure it in many more or less annoying ways. My current configuration is based on packet filtering. Technically I've got just one /64 as you said. But it's split to several independent networks using filtration. All of the ports are on same /64 but DHCPv6 assigns different address ranges to the ports and intra port traffic is limited by firewall rules. So yes, it's horrible kludge, but it works. Other things I've tried, is using smaller subnets with DHCPv6.  As well as manual (static) configuration, which worked well with almost any subnet smaller than /64. Also tried software filtration on router so that each devices gets only /128 and all traffic passes via router. - Once again, not optimal, especially if you've got fast network and bad router. On first time building the setup I literally spent a few weeks swearing and wondering what's broken and how. But at the moment, everything is good. (knock knock) At work I've naturally got /48 and there I've done the configuration just as everyone would expect it to be. Splitting that /48 to /64 subnets.

1-10 of 543