File transfer basics for developers

Post date: Dec 9, 2013 4:28:22 PM

All this should be very clear for everyone, but it unfortunately isn't. I have seen so many broken file transfer implementations. Even if it's not hard at all. I simply hate when things need to be manually fixed, because software written to take care of the task is so bad that it gets stuck with different simple things. I also hate programs which leave junk files everywhere which require manual purging every now and then.

  • If transaction is already running go to connect remote host.
  • Create temp filename
  • Start local transaction containing source, temp and destination filenames and sizes
  • Connect remote host
  • Check if final filename exists locally check size
  • If local file doesn't exist go to finish transaction
  • Check if final filename exists remotely check size
  • If destination file exists is right size go to rename ... step
  • Check if remote temp file exists, check size
  • If file is smaller than local file try to resume transfer if possible
  • If file size matches, good, go to Temp file ...
  • If resume is not possible, delete destination temp file and re-upload whole file
  • Temp file transfer is complete
  • Check temp file size
  • If there's mismatch retry
  • If size matches and you want to be extra sure and protocol allows you to do it, check local and remote file hash at this point
  • Rename temp file to it's final destination filename
  • Rename, delete, move, local file, what ever needs to be done when file had been successfully transferred
  • Finish file transfer transaction
  • Daily or so, purge temp files from destination which are older than N days, hours or so.

Of course if these things are running in parallel, you can use the transaction table to lock files so that multiple threads / processes aren't handling same file.