Fast File Downloader

A simple CLI application to download files over HTTP with the goal of maximum throuput by parallelizing downloads. Optimized for downloading large quantities of files from zippyshare as fast as possible.

Features

• Automatically download files from a URL list
• Automatically fetch actual download URLs for zippyshare and sendcm links
• Download multiple files at the same time
• Use multiple connections to download non overlapping chunks of the same file in parallel
  • This can be used to gain more download speed than a server would normally provide to a single connection
  • Works really well with zippyshare
  • This requires high random write performance for the local storage and will bottleneck on HDDs
  • Only works if the host supports the HTTP Range header and doesn't otherwise block it

Performance

On a 1000Mbit connection (actual speed ~800Mbit) I can download with about 90MB/s through a wireguard VPN from zippyshare when using 5 files in parallel with 10 connections each (-n 5 -c 10).

When downloading the same content manually through a browser I get about 5MB/s per file, dropping to 0.5-1MB/s after a few seconds. Also I of course have to start all the downloads by hand which is quite annoying.

Usage

Simply create a textfile and enter the URLs one per line. Comments using # and empty lines are allowed.

$ cat list.txt
# My encrypted documents backup
https://www71.zippyshare.com/v/XXXXXXX/file.html
https://www72.zippyshare.com/v/XXXXYYY/file.html
https://www73.zippyshare.com/v/XXXXZZZ/file.html
https://www74.zippyshare.com/v/Xagdgfh/file.html

# My encrypted video backup
https://www75.zippyshare.com/v/sdsgfds/file.html
https://www76.zippyshare.com/v/dfsdxfd/file.html
https://www75.zippyshare.com/v/dsgsdgf/file.html
https://www76.zippyshare.com/v/drtdrtd/file.html
https://www75.zippyshare.com/v/erdfghd/file.html
https://www76.zippyshare.com/v/87654rd/file.html
https://www75.zippyshare.com/v/dfghdfg/file.html
https://www76.zippyshare.com/v/hkjghjk/file.html

And then download the files using the desired levels of concurrency (e.g. 5 files at once with 10 connections each):

$ ffdl -l list.txt -o ~/Downloads/ -c 10 -n 5

CLI Arguments

$ ffdl --help

FFDL - Fast File Downloader 0.1.2
Download files fast

USAGE:
    ffdl.exe [OPTIONS]

OPTIONS:
    -c, --connections <CONNECTIONS PER FILE>
            The number concurrent connections per file download. Increasing this number will
            increase the download speed of individual files if supported by the server but setting
            this number too high may cause the download to fail.
            NOTE: This mode will write cause random writes and for that reason won't work on HDDs.
            WARNING: Files started with multiple connections currently can't be continued. [default:
            1]

    -d, --download <URL>
            Download only the one file from the specified url

    -h, --help
            Print help information

    -l, --listfile <URL LISTFILE>
            Download all files from the specified url list file

    -n, --num-files <PARALLEL DOWNLOADS>
            Specify the number of files from that should be downloaded in parallel. Increasing this
            number will increase the total download speed but won't improve the download speed for
            individual files [default: 1]

    -o, --outdir <OUTPUT DIR>
            Set the output directory in which the downloads will be stored. The directory will be
            created if it doesn't exit yet [default: ./]

    -V, --version
            Print version information