File transfer basics for developers
Post date: Dec 9, 2013 4:28:22 PM
All this should be very clear for everyone, but it unfortunately isn't. I have seen so many broken file transfer implementations. Even if it's not hard at all. I simply hate when things need to be manually fixed, because software written to take care of the task is so bad that it gets stuck with different simple things. I also hate programs which leave junk files everywhere which require manual purging every now and then.
- If transaction is already running go to connect remote host.
- Create temp filename
- Start local transaction containing source, temp and destination filenames and sizes
- Connect remote host
- Check if final filename exists locally check size
- If local file doesn't exist go to finish transaction
- Check if final filename exists remotely check size
- If destination file exists is right size go to rename ... step
- Check if remote temp file exists, check size
- If file is smaller than local file try to resume transfer if possible
- If file size matches, good, go to Temp file ...
- If resume is not possible, delete destination temp file and re-upload whole file
- Temp file transfer is complete
- Check temp file size
- If there's mismatch retry
- If size matches and you want to be extra sure and protocol allows you to do it, check local and remote file hash at this point
- Rename temp file to it's final destination filename
- Rename, delete, move, local file, what ever needs to be done when file had been successfully transferred
- Finish file transfer transaction
- Daily or so, purge temp files from destination which are older than N days, hours or so.
Of course if these things are running in parallel, you can use the transaction table to lock files so that multiple threads / processes aren't handling same file.