Data Center Security, BackBlaze B2, Crypto Monoculture, Thread Pools

Post date: May 29, 2016 4:19:56 AM

• Data center security and design best practices by Google - Really nice article. Using machine learning for data center optimization is awesome.
• Lack of proper all around hydraulic or otherwise heavy duty Vehicle Entry Barriers is something I've been observing missing in Finland from many sites (not referring to Google, just the article and as general observation about also other sites than data centers). They say site is secure, but you still can drive a truck with trailer next to it's wall or even in some worst cases under the building. - I'm still laughing at the Donald Duck cartoon where the The Beagle Boys try to break into Scrooge's money bin. They try absolutely everything against the massive impenetrable door. At the end when the 'camera zooms out' the only thing remaining is the massive door. Walls around the door have collapsed a good while ago. This seems to be the norm in many cases. The primary route might be extremely well protected in show off style, but what about the alternate routes? Who would think about those. Guests arrive using the primary route, and then the cleaning, maintenance and security staff uses backdoor for 'service people' which lacks all the security measures massively implemented for the main entrance in a show off way.
• BackBlaze B2 integration requirements. Pretty obvious I would say. But as I've mentioned in my blog several times, obvious things might not be obvious to everyone.
• Crypto Monoculture - A great post at cryptome.
• Now I've done one integration to dedicated Financial Management Software (FMS) too. Yeah, there are just too many of these to count. Budgeting, Billing, Invoicing, etc. Basically and technically all the same stuff, BI, CRM, ERP, ECM, SCM. Just some data and stuff being done with it, from integration point of view, it doesn't really make any difference. Here's the data, you'll just do whatever you want with it, I really don't care. But as required I'll make it happen and work for you. No problem.
• Thread Pools by Julia Evans - Nice timing. I've been just spending easter by implementing Multiprocessing and Threadpools. For one project. She didn't say it, but if you're using Python it makes perfect sense to mix processes & threads, because multiprocessing doesn't suffer from GIL as multithreading does. - https://wiki.python.org/moin/GlobalInterpreterLock -. With python there's funny thing that the queue for apply_async execution is unbound. If you want to fix that you'll need to subclass the Pool, or use additional bound semaphore which is being called back and releasing space for new items. Most annoyingligly the queue in thread / process spool is also processed further, so you can't even check the queue length, before submitting more stuff, because unless you're submitting really fast, it's likely going to be empty. So the fact that the pool queue is empty, doesn't really mean that there wouldn't be any queue. Yep, easily confusing. If you don't do that, you'll end up pushing everything into in memory processing queue and running out of memory just like Julia said. Using multiprocessing.BoundedSemaphore(queue_len) helps you to limit the depth of the queue. Just acquire when submitting and release when getting results back. Annoying, yes, but works very well. I'm not the only and first one cursing this, but that's not a problem when you'll get used to it. Yet sometimes exceptions and stuff like that can break havoc, if the release stuff doesn't get executed. So that's why it would be much better to have internal queue length limit, but there isn't. So this is a trick: pool. _taskqueue.qsize() <- it doesn't reveal the actual internal queue length. It only reveals very shortly kept "input" queue. It's like thinking that at airport you can get straight to the plan from taxi drop off, if you didn't see any queue when jumping out of taxi. But the actual internal queues are still hidden from you. Alternate way is not using pool at all and writing your own implementation with required features. Yet, the callback for apply_async is being called even if there's unhandle exception, so that's pretty safe way to do it.