Ricardian Contract, SIGAINT, NoTCP, PEP484, RUDP, PM, Business Culture, Efficiency, DBaaS, Hyper-Convergence

Post date: May 3, 2015 4:46:00 PM

• Studied Bounded Futures Ricardian Contract for OpenBazaar
• Docker without Docker - Docker is just standardized bunch of techniques which already existed
• Studied: CopSSH and played with it to familiarize with the product. Simple and easy to use and understand, which is generally a good thing.
• Once again wondered how efficiently software engineers handle data transfers. Their way of doing it? Opening a remote desktop connections to two servers and then using clipboard to transfer ~200 GB of files and then complaining it's being slow and often unreliable especially in cases where there are individual files which are like 50GB in size. They used two days trying to transfer that data from server to another. I just couldn't stand it. Solution? Compress data using 7-zip and transferring it using SSH. Result? Compression time ~40 minutes transfer time, transfer time ~20 minutes and decompression time ~30 minutes. That's it. I think it makes more like 1,5 hours than several days. Even if you think people can do stuff, it still seems matters who's getting it done. There are huge efficient benefits gained from doing things efficiently. Yes, no surprise there. I've earlier witnessed cases where engineers try to move stuff from computer to another using USB stick. It's also laughable. Usually they transfer lot of stuff that's not needed to be transferred, secondly they transfer tens of hundreds of thousands of small files, which is for sure totally inefficient. They might try to transfer larger files using fat32, which of course fails. In this case efficiency can be hugely improved and when you do it inefficiently enough it's not going to work at all which is a bonus on top of it being just darn slow. So please. Only transfer what's needed, compress it, move efficiently (USB or network) and decompress. It's not that hard, yet it seems to be really hard for many.
• Privacy is really hard. You should train for it. So you have the capability if and when needed. All this surveillance forces journalists to hink and act like spies.
• I were really excited about the future prediction markets at one time. OpenBazaar allows easy P2P contracts directly between individuals and that could bring totally new kind of financial and predictive speculation (aka betting) markets. As well as one interesting field of development would be fully automated notaries, which would read data from reliable sources and automatically quickly and efficiently deal with any payment contracts which are linked to outcome of those events. Like sports, politics, stock prices and so on.
• Studied: General Data Protection Regulation (GDPR). Related: Data Protection Officer (DPO), Data Protection Authorities (DPA), EU, European Union.
• SIGAINT project targetted Tor Exit Nodes. No news there, it's well known point of interception. It would be much smarter to use encrypted communication if you're worried about security. Or just run hidden service as to begin with.
• Type Hints: PEP-484 Adding Gradual Typing to Python 3.5.
• Refreshed memory and current status about Nofollow usage.
• I've greatly improved my CSS skills. I get things done reasonably quickly. I'm not expert, but getting stuff done isn't pure swearing and reading more and more documentation and experimenting for hours why this ... just doesn't work out. I've got clear idea how to do what I want to accomplish.
• One of major challenges I did was large database migration script stuff for one project. Tons of tables getting renamed, fields, added, removed and most importantly remapped to new structure using additional conversion program (Python) which I had to write. I got it done. Of course it was kind of trivial in my mind, but making sure that everything goes absolutely right when script is run during short service break got me bit stressed at first. But after testing, and fixing stuff I got confident about getting it done. I also found a few bugs from the main application which I wrote the script for that got also fixed during same update.
• E-receipt and m-receipt points: Fragmented environment, Investment efficiency (investment / benefit), Comfortably and usability (off-line environment in e-society), Accounting and reporting, Analytics and cost management.
• My first time I used findstr on Windows Server. It's always nice to learn something new.
• NoTCP - Nothing new there. One of the TCP works well enough, is easy to use. Implementing data transfer over UDP in sane way is monumental task. As well as some of the benefits are lost with it. The fact that TCP uses three way handshake prevents spoofing. With UDP you have to do something similar, otherwise your system is just going to be one huge DDoS amplifier which is well, not a good idea. I've seen some projects use stuff like RUDP and I can say the communication over UDP is a lot worse than over TCP. I commented on one discussion:"Tons of discussion and nobody mentioned this: https://en.wikipedia.org/wiki/Reliable_User_Datagram_Protoco... Reason for using TCP is that using UDP requires application specific implementation and making it good and efficient would require more work than making rest of the software. That's why the applications only use UDP where it really does make difference. Others just won't bother because it isn't worth of it.."
• Reminded my self about ARQ and NAK. yet these are familiar from good old phone line modems and of course PAR which is used by TCP. Related: NACK, SACK. Related: RFC 4077
• Microsoft launched .NET for Linux. That's nice. No need to fight with old Mono version and software compatibility problems. (Hopefully)
• Microsoft added PackageManagement for Windows 10. That's absolutely great news. I've been using Chocolatey but hopefully in future it's possible to use native package manager.
• I keep wondering why so many Finnish sites are hosted at Amazon EU West, even if it's clearly slower than Amazon EU Central when being accessed from Finland. If there are options to choose between Frankfurt and Amsterdam, that's though call because it varies from operator to operator. But choosing between Frankfurt and Dublin is not. Latency is always worse from Finland to Dublin. Some routes from Sonera Finland to OVH Roubaix seem to fluctuate between Frankfurt and Amsterdam. When ever running trace, you don't know which path the packets are going to take.
• Some stuff on this post is really surprisingly negative. But actually we should learn from it. It is what it is, but how we can fix it?
• Lol, some say programming is hard. But it's same thing with everything. I was really laughing when one of my friends had huge struggle with GIMP and LibreOffice Writer to get things right. He kept cursing for hours because things simply won't work out. I told her that duh, that's just like programming. But eventually you will learn, just keep banging your head. GIMP layers and adding text as well as thend page numbering with LibreOffice documents can be real pain in the ... if you don't know how to do the thing. How about using paragraphs which reset page number or using title pages and page number offset? But if you don't know that, it's rage and taking a quite good while reading tens of different more and less incorrect instructions how to accomplish that. Yes, it's exactly like tuning Linux, Windows or programming. Lol again. Just lookup how to do X using CSS or how to configure Linux network stack to ... And you're going to have fun penty of fun. Or like in my case, joining stuff using Peewee in very specific way where I want to also get the rows which do not have joinable references.
• World is so full of *t software... I'm still highly amused about PDF files. First of all, people seem to think that you can fill PDF forms, nope you can't. It sucks. Another really amusing misunderstanding is that PDF is somehow signed and encrypted format and you can't modify it. Which is yet another purely delusion belief. World is just so full of crap software. Even basic things like office applications are just so hopelessly badly coded. Due these limits I always have all on my stuff on my own computer, pdf, odf, docx, doc and other formats. As well as I have the working application to show etc. Some formats embed fonts and others won't and some systems render unknown fonts really funnily, like Windows shows Ubuntu font which is missing as Script etc. So if you save document as docx or odf and recipient doesn't have the font installed document or presentation gets totally screwed up and so on. It's just baffling to notice how full of bad and really bad software world is and how much suffering, pain and loss of productivity all that crap causes. Yet another thing is that how overly complicated software is "cool", I think that software which is fast, light, simple and does what's essential is the coolest thing ever. Any complex standards or really super complex standards are horrible, making everyone suffer, users, developers, just everyone. It also takes resources away from the truly profitable stuff. So why is anyone making such a horrible software? Maybe there's just some kind of need for it? Because we're paid to do it? Yuck! Lean, mean and working. That's the way.
• I've seen projects which have used millions of euros to produce, well, absolutely nothing, but confusing and poor very high level documentation without any practical details. Worst part of all this is that the solution is so obvious that just a few nerds meeting for a half day would have solved it. But that would be just so wrong, getting something done, instead of keeping useless meetings, where discussion starts from absolute zero over and over again and nothing is actually getting produced. Good thing is that the catering service was good. I think this is excellent concept for the pointy-haired boss from Dilbert. But I don't personally like it, I would get the thing done and that's just it. Actually I was at one point totally shocked how accurately Dilbert describes software and IT industry. Smile.
• Yet another thing I really hate is let's just discuss this matter. Well, does the discussion change any of the facts? No? Does it help in anyway? No? What about going straight to the facts and looking for some real solution, instead of wasting our time over dinner talking about some high level talks. It doesn't solve the problem and won't be in any other meaning full way helpful. Like show me the API documentation and tell me what it costs. Talking about what kind of organizational history or how cool tech you're using, won't help. It won't make things happen like fixing the problem or creating the integration, or making stuff work like the customer wants it to work. This is not personal, this is business and getting the things done, delivered and solved as fast and efficiently as possible is preferred. Please send clear agenda and documentation before the meeting, describing what is being offered, what's the plan and what it costs and what are the benefits. When you do that, I can decline before the meeting and we both save time. It's just pointless to have pointless meetings.
• It's interesting to notice that in Estonia, they do the things which Finnish people talk about. And in Sweden, aww. They're still planning to have a meeting about preparing the meeting and thinking what we should talk about at the meeting. (When Estonians already got the specifications done and they're now working (actually) hard on the product.) There's huge difference between looking like you're working and busy and actually getting the thing done.
• Just be straight, honest and go straight to topic, if you can't get it done, just say so. If you say you will get it done, try really hard to get it done, if you won't well, that's too bad, you look bad and if you do it a couple of times or when things are important enough, you won't be trusted anymore. I value being straight and honest more than being polite or politically correct. I really don't like hedging topic, just say how things are and that's it. We save time, and can proceed to fixing potential problem. Instead of trying to guess what the problem is.
• I've seen some of the open source projects do absolutely great progress. Team of 5-10 competent and efficient people who want to get the thing done, can achieve miracles. Compared to organizations where all kind of 'overhead' is being done and years pass and stuff which would be already completed is still being considered if we should do it, and can it be done, and how. How about stopping to wonder things and getting it done. It's kind of analysis paralysis. Which I admit I'm at times suffering personally, especially when talking about stock market or investing.
• Some people say it's good to be always happy and always smiling. But it's not, when things won't work out. I'm happy and smiling when results are excellent. From larger organizations I've noticed that people are totally disconnected from reality, what's productive and what's not. It would be interesting to see those guys as independent www.entrepreneur.com working alone. I've heard a few complaining about the salary and how 'easy' it would be millions as alone developer. Smile. Especially if you're focus and productivity is ahem, bit lost. And that's slightly said.
• Strange system freezes at OVH ended when I complained about it. It might be that the HOST was running out of memory and swapping. Yes, that's also reasonable optimization when trying to run as many servers as possible on cheapest possible hardware. Even if the VM isn't out of memory the HOST could still swap memory out and VM might now reval it. Totally reasonable optimization, because systems do reserve and keep a lot of memory which isn't usually being used. And now it's usable, it's just bit slower when needed.
• Studied EU Data Retention Directive (DRD)
• Tried Debian GNU/Hurd in VM. Yes, it works, and does look just like any linux bash level and application level. Of course there are major gaps. I assigned 8 gigs of memory for it and on boot it said limiting memory to ~1.9 gigabytes. Etc. But getting so far is already a really great thing.
• Grooveshark shutdown. That's sad. I really liked Grooveshark, it was the best music site on net afaik. So sad to see it's gone. I also recommended it to all of my friends, instead of Spotify. I also liked Pandora but it doesn't work in my country.
• 6 Tips for Goole App Engine from Streak. No non-obvious surprises there.
• Mozilla wants to deprecate HTTP and make whole web 'secure'. Whole point of certs is to verify the site ownership. If getting certs is too easy, well those are then worthless. As it happens to be already. Email verification of domain ownership isn't good verification at all. These certs even if trusted are no different from self signed certs. IMHO.
• Pentagon announces new strategy for Cyberwarfare. MAD for Internet, cool?
• It seems that my personal email server is handling about 6k mails / month. That's like 190 / day. That's actually quite horrible when you think about it.
• Checked out EmDrive and Cannae Drive. Related: RF resonant cavity thruster, spacecraft propulsion, microwaves, magnetron, reaction massless propulsion.
• Some analysis on WhatsApp security. As usual and expected, it seems to be weak.
• Rehearsed my Morse code and reminded my self about tap code messaging protocols.
• The data center and server arrangements made lower purchase and production costs for many services for over 80%. That's immense! When you think how much it will add to the profit margin if the customer prices aren't cut. Good article how reducing costs on YouTube possibly saved it.
• What we can learn from history, that if there's a way to communicate at all. Then there's also a way to covertly communicate over that channel. It's just encoding matter. It's nearly impossible to prevent that. Camp X, Clandestine Warfare, Covert Sabotage, Guerilla Warfare,
• DataBase-as-a-Service (DBaaS) aka Cloud Database. That surely fits some purposes great, and others very badly. But it's good to have that option available for cases where it's suitable and usable option.
• I still can't stop deeply hating bad code, documentation and error messages. It's so extremely frustrating when things which really trivial and should take less than a minute end up taking days, weeks or won't even ever get solved and require some kind of complex obsecure workaround which poses tons of new potential for risk and catastrophic failure. I guess we all know that. It's just like selling cheap crap devices which are unreliable and you'll end up wasting tons of time to trying to get that stuff to work, and it might even randomly work before once again failing and causing even more loss of resources, time, money and mental energy.
• Peewee add_column works well, drop_columm fails with OperationalError message which doesn't give any hint what's wrong. Compared to the fact how simple the command is, it's quite clear that there's some kind of annoying problem somewhere and it's better to add and drop the columns once again using pure sql that trying to waste your time playing with ORM which obfuscates things and makes things generally extremely painful. - After researching this deeper and trying all kind of combinations. I think the problem was that due to libraries I imported for migration some of those connected database and getting exclusive lock was impossible. yet, it would be really nice to get better than OperationalError exception. Unable to exclusively lock database, would have been nice message. But now everything works again.
• This of course seems to pretty normal in business and managing things in general. Competent developers or managers, won't bother to do something which is clearly required and useful. They push the problem downward to other people. One guy, could have done it quickly and efficiently with good knowledge and shared there results with rest of the organization just won't bother to do it.. What's the way to get it done then? Just pass the problem to be solved by 1000 incompetent guys who don't talk to each other. The Win? Now we as organization managed to waste just about 100000x the resources that would have been required to solve the problem. Compared to situation where it would have been taken care efficiently. As bonus of that during the time the incompetent guys tried to solve the problem they possibly seriously messed up things and caused further customer dissatisfaction, bad PR, arguments, crisis meetings with customers and general huge loss of energy, mood , money, time and resources in all departments. (Hopefully not loss of life) Yes, that's just how things seem to work.
• This applies just so much to stuff like software installation, upgrade scripts and everything at least in manufacturing in general. Yes yes, I know the quality of the cement or steel used to build the bridge was inadequate. But no problem, the bride won't fall a part before we can get rid of this fake company and run with the money. It's then someone else's problem to deal with that emerging catastrophe. I guess this is pretty much the mentality when building buildings around the world.
• PSYOP - They're manipulating you! Don't listen!
• Read about Saab Kockums A26 submarine
• Read a book about Industrial Intenet, Data Analytics, Data Collection, Data Discovery, IoT, Sensor Networks, Remote Monitoring, pre-emptive service, pre-failure alerts, digitalization, business models. Fiware, Data Visualization, fragmented standards, user interfaces, usability, efficiency, cost beneftis, productivity, investments. Message queue telemetry transport (MQTT), AMQP, CAP, XMPP.
• Smile, some say that IoT needs to be on same level as m2m to break through. That's not going to happen ever? Why? Well, it's just like your random desktop application isn't on same level when comparing to flight management system of modern airliner. The resources needed for it, are quite different. For IoT it's mostly enough if it works (if it even does that!) and for m2m, industrial internet software requirements are absolutely different. Yet this somehow reminds me from the Airbus overflow problem which could cut power from whole airliner and prevent flight controls from working. But that's something which happens with consumer products all the time. I assume you got hard drives? When have you updated firmware for those? Do you know what version you got? Do you even know if there are new version available and what the changelog says? Of course you don't. And this is just as things are going to be in future too. Do you know what firmware version your digital camera is using? No, do you know firmware of your toaster, fidge or ... You get the point. Most don't know and even those who know, mostly don't care. Not before things actually stop working. Have you checked if there's a firmware update for your monitor or tv? Yes, botnet of appliances will be here and it seems just inevitable. Lol, one article said that securing networks can be done using 20 usd/eur firewalls. Did you btw know that those got often security issues, device require firmware updates and so on.
• Hyper-Convergence - It's kind of cool to re-invent thigns. Getting rid of SAN. Who said that there should have been SAN in the first place. Just install servers and use software to coordinate everything including replication to different disks. SAN was a bad solution because in most of cases the disk space prices were way too high compared to other kind of solutions. (Software-defined storage, software-defined networking), data center, rack, servers.
• LclBd.com now uses fully threaded discussion model. Making it much more friendly for long and deep conversions. Next step is to optimize per user customized views, because currently compiling those from database can take too long for web app. I guess some kind of message that data is being processed is first step and second is that the intermediate results will be stored as cache to improve performance so the most computationally or I/O expensive (database) steps do not need to be recomputed on every reload.