Wget (Linux File Downloader) Command Examples in Linux.

Stan Lee. July 30, 2016 Comments

In this post we are going to review wget utility which retrieves files from World Wide Web (WWW) using widely used protocols like HTTP, HTTPS and FTP. Wget utility is freely available package and license is under GNU GPL License. This utility can be install any Unix-like Operating system including Windows and MAC OS. It’s a non-interactive command line tool. Main feature of Wget of it’s robustness. It’s designed in such way so that it works in slow or unstable network connections. Wget automatically start download where it was left off in case of network problem. Also downloads file recursively. It’ll keep trying until file has be retrieved completely.

 

First, check whether wget utility is already installed or not in your Linux box, using following command.

$ rpm -qa wget
#wget-1.12-1.4.el6.i686

Please install it using YUM command in case wget is not installed already or you can also download binary package at http://ftp.gnu.org/gnu/wget/.

$ yum -y install wget

The -y option used here, is to prevent confirmation prompt before installing any package.

 

1. Single file download

The command will download single file and stores in a current directory. It also shows download progress,size, date and time while downloading.

$ wget http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz

--2012-10-02 11:28:30--  http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: wget-1.5.3.tar.gz
100%[===================================================================================>] 446,966     60.0K/s   in 7.4s
2012-10-02 11:28:38 (58.9 KB/s) - wget-1.5.3.tar.gz

 

2. Download file with different name

Using -O (uppercase) option, downloads file with different file name. Here we have given wget.zip file name as show below.

$ wget -O wget.zip http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz

--2012-10-02 11:55:54--  http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: wget.zip
100%[===================================================================================>] 446,966     60.0K/s   in 7.5s
2012-10-02 11:56:02 (58.5 KB/s) - wget.zip

 

3. Download multiple file with http and ftp protocol

Here we see how to download multiple files using HTTP and FTP protocol with wget command at ones.

$ wget "http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz" "ftp://ftp.gnu.org/gnu/wget/wget-1.10.1.tar.gz.sig"

--2012-10-02 12:11:16--  http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: wget-1.5.3.tar.gz
100%[===================================================================================>] 446,966     56.7K/s   in 7.6s
2012-10-02 12:11:29 (57.1 KB/s) - wget-1.5.3.tar.gz
--2012-10-02 12:11:29--  ftp://ftp.gnu.org/gnu/wget/wget-1.10.1.tar.gz.sig
=> wget-1.10.1.tar.gz.sig
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /gnu/wget ... done.
==> SIZE wget-1.10.1.tar.gz.sig ... 65
==> PASV ... done.    ==> RETR wget-1.10.1.tar.gz.sig ... done.
Length: 65 (unauthoritative)
100%[===================================================================================>] 65          --.-K/s   in 0s
2012-10-02 12:11:33 (2.66 MB/s) - wget-1.10.1.tar.gz.sig
FINISHED --2012-10-02 12:11:33--
Downloaded: 2 files, 437K in 7.6s (57.1 KB/s)

 

4. Read URL’s from a file

You can store number of URL’s in text file and download them with -i option. Below we have created tmp.txtunder wget directory where we put series of URL’s to download.

$ wget -i /wget/tmp.txt

--2012-10-02 12:34:12--  http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.10.1.tar.gz.sig
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: wget-1.10.1.tar.gz.sig
100%[===================================================================================>] 446,966     35.0K/s   in 10s
2012-10-02 12:34:23 (42.7 KB/s) - wget-1.10.1.tar.gz.sig
--2012-10-02 12:34:23--  http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
Resolving mirrors.hns.net.in... 111.91.91.34, 2401:4800::111:91:91:34
Connecting to mirrors.hns.net.in|111.91.91.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1761607680 (1.6G) [application/octet-stream]
Saving to: CentOS-6.3-x86_64-LiveDVD.iso
45%[==========================================                                          ] 1,262,000   51.6K/s  eta 8h 17m

 

5. Resume uncompleted download

In case of big file download, it may happen sometime to stop download in that case we can resume download the same file where it was left off with -c option. But when you start download file without specifying -c optionwget will add .1 extension at the end of file, considering as a fresh download. So, it’s good practice to add -cswitch when you download big files.

$ wget -c http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso

--2012-10-02 12:46:57--  http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
Resolving mirrors.hns.net.in... 111.91.91.34, 2401:4800::111:91:91:34
Connecting to mirrors.hns.net.in|111.91.91.34|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 1761607680 (1.6G), 1758132697 (1.6G) remaining [application/octet-stream]
Saving to: CentOS-6.3-x86_64-LiveDVD.iso
51% [=================================================                                  ] 3,877,262   47.0K/s  eta 10h 27m ^

 

6. Download file with appended .1 in file name

When you start download without -c option wget add .1 at the end of file and start with fresh download. If .1already exist .2 append at the end of file.

$ wget http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso

--2012-10-02 12:50:49--  http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
Resolving mirrors.hns.net.in... 111.91.91.34, 2401:4800::111:91:91:34
Connecting to mirrors.hns.net.in|111.91.91.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1761607680 (1.6G) [application/octet-stream]
Saving to: CentOS-6.3-x86_64-LiveDVD.iso.1
18% [==================                                                                 ] 172,436     59.2K/s   

See the example files with .1 extension appended at the end of the file.

$ ls -l CentOS*
-rw-r--r--. 1 root root 3877262 Oct  2 12:47 CentOS-6.3-x86_64-LiveDVD.iso
-rw-r--r--. 1 root root  181004 Oct  2 12:50 CentOS-6.3-x86_64-LiveDVD.iso.1

 

7. Download files in background

With -b option you can send download in background immediately after download start and logs are written in/wget/log.txt file.

$ wget -b /wget/log.txt ftp://ftp.iinet.net.au/debian/debian-cd/6.0.5/i386/iso-dvd/debian-6.0.5-i386-DVD-1.iso
Continuing in background, pid 3550.

 

8. Restrict download speed limits

With Option –limit-rate=100k, the download speed limit is restricted to 100k and the logs will be created under /wget/log.txt as shown below.

$ wget -c --limit-rate=100k  /wget/log.txt ftp://ftp.iinet.net.au/debian/debian-cd/6.0.5/i386/iso-dvd/debian-6.0.5-i386-DVD-1.iso
/wget/log.txt: Scheme missing.
--2012-10-02 13:16:21--  ftp://ftp.iinet.net.au/debian/debian-cd/6.0.5/i386/iso-dvd/debian-6.0.5-i386-DVD-1.iso
=> debian-6.0.5-i386-DVD-1.iso
esolving ftp.iinet.net.au... 203.0.178.32
Connecting to ftp.iinet.net.au|203.0.178.32|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /debian/debian-cd/6.0.5/i386/iso-dvd ... done.
==> SIZE debian-6.0.5-i386-DVD-1.iso ... 4691312640
==> PASV ... done.    ==> REST 2825236 ... done.
==> RETR debian-6.0.5-i386-DVD-1.iso ... done.
Length: 4688487404 (4.4G), 4685662168 (4.4G) remaining (unauthoritative)
0% [                                                                                    ] 3,372,160   35.5K/s  eta 28h 39m

 

9. Restricted FTP and HTTP downloads with username and password

With Options –http-user=username, –http-password=password & –ftp-user=username, –ftp-password=password, you can download password restricted HTTP or FTP sites as shown below.

$ wget --http-user=narad --http-password=password http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
$ wget --ftp-user=narad --ftp-password=password ftp://ftp.iinet.net.au/debian/debian-cd/6.0.5/i386/iso-dvd/debian-6.0.5-i386-DVD-1.iso

 

10. Find wget version and help

With Options –version and –help you can view version and help as needed.

$ wget --version
$ wget --help

 

11. Send Request with wget without downloading actual file.

Use q flag for quiet mode, and tell wget to output to stdout with O- (uppercase o) and redirect to /dev/null to discard the output:

wget -qO- $url &> /dev/null

> redirects application output (to a file). if > is preceded by ampersand, shell redirects all outputs (error and normal) to the file right of >. If you don't specify ampersand, then only normal output is redirected.

./app &>  file # redirect error and standard output to file
./app >   file # redirect standard output to file
./app 2>  file # redirect error output to file

if file is /dev/null then all is discarded.

 

In this article we have covered Linux wget command with options for daily administrative task. Do man wget if you want to know more about it. Kindly share through our comment box or if we’ve missed out anything, do let us know.

  • linux
  • Wget