A file sharing cluster with automated management

Post date: Nov 30, 2013 3:15:44 PM

This is a project I did a very long time ago and just for fun. I guess it was at end of year 2001 or so. Just writing old stuff, but I liked it. I just happened to have excess hardware and bandwidth.

Concept

A cluster which automatically downloads and seeds everything released at Sharerector file sharing indexing site.

Technology

EDonkey2000 command line client

Java management system on one node

Nine Windows servers

10 Megabit symmetric internet connection

CLI over TCP/IP was used from management system to control individual ED2K clients on servers.

High level description

Java management system scraped Sharereactor releases page every 15 minutes for new releases. If new release was found, it was added to seed list. When there was new stuff on seed list, management system selected a node to seed the file on. Node selection was based on server's network load, free disk space, file popularity (sources + downloader count), seed ratio (sources / downloader ratio). Same parameters and file age were also used to evict files from nodes with certain loading factors.

How it worked in real world?

Well it did pretty well, and everything worked really nice. But I found out that unless my main goal is to keep seeding old stuff, only one server was more than enough to saturate bandwidth with high on demand new releases. Therefore additional servers and disk space didn't provide any extra bandwidth, but it allowed me to seed and store files way longer than those were in high demand. After all it was a very nice test / practice project for automatically managed file sharing cluster. After I had tested things and played with it for about two weeks I shut it down.

KW: emule, edonkey 2000, ed2k, file sharing, server, hosting, automated, automatically, automation, cluster, clustering, server, servers, disk space, bandwidth management, Java, Sharereactor, Overnet, ED2K, "Harness the power of 2000 electronic donkeys".