A Comprehensive Guide to the wget Command

Common wget Command Examples

wget is a non-interactive command-line utility for downloading files from the web, supporting HTTP, HTTPS, and FTP protocols. Here are practical examples for various scenarios.

1. Download a Single File

Download a file and save it in the current directory.

wget http://example.com/file.zip

Progress is shown, including percentage, bytes transferred, current speed, and estimated time remaining.

2. Specify Output Filename with -O

By default, wget uses the last part of the URL as the filename. For dynamic URLs (e.g., download.php?id=1), use -O to set a specific name.

wget -O wordpress.zip http://www.example.com/download.php?id=1080

3. Limit Download Speed with --limit-rate

Limit bandwidth usage to avoid interfering with other network tasks.

wget --limit-rate=300k http://example.com/large.iso

4. Resume Interrupted Download with -c

If a download is interrupted, resume it without starting over.

wget -c http://example.com/large.iso

5. Download in Background with -b

For large files, run wget in the background. Output is logged to wget-log.

wget -b http://example.com/large.iso

Monitor progress with:

tail -f wget-log

6. Spoof User-Agent with --user-agent

Some sites block non-browser requests. Simulate a browser.

wget --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' URL

7. Test Link Validity with --spider

Check if a URL is reachable without downloading the file.

wget --spider URL

Returns 200 OK if valid, or 404 Not Found if broken.

8. Increase Retry Attempts with --tries

Default is 20 retries. Increase for unstable connections.

wget --tries=40 URL

9. Batch Download with -i

Save URLs to a file (e.g., urls.txt), one per line, then download all.

wget -i urls.txt

10. Mirror a Website with --mirror

Download an entire website locally.

wget --mirror -p --convert-links -P ./LOCAL URL

• --mirror: Enables mirroring (equivalent to -r -N -l inf --no-remove-listing).
• -p: Downloads all page resources (images, CSS).
• --convert-links: Converts links for local viewing.
• -P ./LOCAL: Saves files to the specified directory.

11. Filter File Types with --reject

Exclude specific file types, e.g., all GIF images.

wget --reject=gif URL

12. Log Output with -o

Save download output to a log file instead of the terminal.

wget -o download.log URL

Note: lowercase -o logs output; uppercase -O sets the output filename.

13. Limit Total Download Size with -Q

Set a quota for recursive downloads (does not apply to single files).

wget -Q500m -i filelist.txt

14. Recursively Download Specific File Formats

Download all files of a certain type, e.g., PDFs.

wget -r -A.pdf URL

Use --accept and --reject for finer control.

15. FTP Downloads

Download from FTP servers.

Anonymous FTP:

wget ftp://example.com/file.zip

Authenticated FTP:

# Method 1: Credentials in URL
wget ftp://user:[email protected]/file.zip
# Method 2: Separate parameters (safer for special characters)
wget --ftp-user=user --ftp-password=pass ftp://example.com/file.zip

16. Recursive Website Download

Download a site recursively. Use with caution.

wget -r http://example.com/

Limit depth with -l:

wget -r -l 2 http://example.com/

Use -m (mirror) for a more complete mirror that respects robots.txt.

17. Connection Control

Control retries and timeouts.

wget -c -t 100 -T 120 http://example.com/large.iso

• -t: Retry count (-t 0 for infinite).
• -T: Timeout in seconds.

18. HTTP Basic Authentication

Access password-protected HTTP sites.

wget --http-user=USER --http-password=PASS URL

Note: wget does not support client certificate authentication; use curl for that.

19. Use a Proxy Server

Configure via ~/.wgetrc or command line.

In ~/.wgetrc:

http_proxy = http://proxy:port/
ftp_proxy = http://proxy:port/

Command line:

wget --proxy=on --proxy-user=USER --proxy-password=PASS URL

20. Handle Non-ASCII Filenames

Prevent encoding of characters like Chinese by restricting filename processing.

wget -r --restrict-file-names=nocontrol ftp://example.com/测试.txt

Quick Reference of Common Options

• -V, --version: Show version.
• -h, --help: Show help.
• -O FILE: Write output to FILE.
• -nc, --no-clobber: Do not overwrite existing files.
• -N, --timestamping: Download only newer files.
• -nd, --no-directories: Do not create directories.
• -x, --force-directories: Force directory structure.
• -A LIST, --accept LIST: Accept file patterns.
• -R LIST, --reject LIST: Reject file patterns.
• -D LIST, --domains LIST: Allowed domains.
• --exclude-domains LIST: Excluded domains.
• -L, --relative: Follow relative links only.
• -H, --span-hosts: Go to external hosts.
• -I LIST, --include-directories LIST: Allowed directories.
• -X LIST, --exclude-directories LIST: Excluded directories.