From 4ea898efb24bc3e546c4c0942c19afc25e29dd7f Mon Sep 17 00:00:00 2001 From: Daniel M Date: Thu, 16 Jun 2022 01:35:53 +0200 Subject: [PATCH] Add README --- README.md | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..bec1361 --- /dev/null +++ b/README.md @@ -0,0 +1,86 @@ +# Fast File Downloader +A simple CLI application to download files over HTTP with the goal of maximum throuput by parallelizing downloads. Optimized for downloading large quantities of files from [zippyshare](https://zippyshare.com/) as fast as possible. + +## Features +* Automatically download files from a URL list +* Automatically fetch actual download URLs for [zippyshare](https://zippyshare.com/) and [sendcm](https://send.cm/) links +* Download multiple files at the same time +* Use multiple connections to download non overlapping chunks of the same file in parallel + * This can be used to gain more download speed than a server would normally provide to a single connection + * Works really well with zippyshare + * This requires high random write performance for the local storage and will bottleneck on HDDs + * Only works if the host supports the HTTP Range header and doesn't otherwise block it + +## Performance +On a 1000Mbit connection (actual speed ~800Mbit) I can download with about 90MB/s through a wireguard VPN from zippyshare when using 5 files in parallel with 10 connections each (`-n 5 -c 10`). + +When downloading the same content (backup files using the free cloud storage ofc) manually through a browser I get about 5MB/s per file, dropping to 0.5-1MB/s after a few seconds. + +## Usage +Simply create a textfile and enter the URLs one per line. Comments using `#` and empty lines are allowed. + +``` +$ cat list.txt +# My encrypted documents backup +https://www71.zippyshare.com/v/XXXXXXX/file.html +https://www72.zippyshare.com/v/XXXXYYY/file.html +https://www73.zippyshare.com/v/XXXXZZZ/file.html +https://www74.zippyshare.com/v/Xagdgfh/file.html + +# My encrypted video backup +https://www75.zippyshare.com/v/sdsgfds/file.html +https://www76.zippyshare.com/v/dfsdxfd/file.html +https://www75.zippyshare.com/v/dsgsdgf/file.html +https://www76.zippyshare.com/v/drtdrtd/file.html +https://www75.zippyshare.com/v/erdfghd/file.html +https://www76.zippyshare.com/v/87654rd/file.html +https://www75.zippyshare.com/v/dfghdfg/file.html +https://www76.zippyshare.com/v/hkjghjk/file.html +``` + +And then download the files using the desired levels of concurrency (e.g. 5 files at once with 10 connections each): +``` +$ ffdl -l list.txt -o ~/Downloads/ -c 10 -n 5 +``` + +## CLI Arguments + +``` +$ ffdl --help + +FFDL - Fast File Downloader 0.1.2 +Download files fast + +USAGE: + ffdl.exe [OPTIONS] + +OPTIONS: + -c, --connections + The number concurrent connections per file download. Increasing this number will + increase the download speed of individual files if supported by the server but setting + this number too high may cause the download to fail. + NOTE: This mode will write cause random writes and for that reason won't work on HDDs. + WARNING: Files started with multiple connections currently can't be continued. [default: + 1] + + -d, --download + Download only the one file from the specified url + + -h, --help + Print help information + + -l, --listfile + Download all files from the specified url list file + + -n, --num-files + Specify the number of files from that should be downloaded in parallel. Increasing this + number will increase the total download speed but won't improve the download speed for + individual files [default: 1] + + -o, --outdir + Set the output directory in which the downloads will be stored. The directory will be + created if it doesn't exit yet [default: ./] + + -V, --version + Print version information +``` \ No newline at end of file