Pages

Friday, January 18, 2013

Some Curl Tips


cURL

A command line tool for getting or sending files using URL syntax.
Since cURL uses libcurl, it supports a range of common Internet protocols, currently including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, LDAP, LDAPS, DICT, TELNET, FILE, IMAP, POP3, SMTP and RTSP (the last four only in versions newer than 7.20.0 or 9 February 2010).

The name of the project is a play on 'Client for URLs', originally with URL spelled in uppercase to make it obvious that it deals with URLs. The fact that it can also be pronounced 'see URL' also helped; it works as an abbreviation for "Client URL Request Library" or the recursive version: "Curl URL Request Library".
Daniel Stenberg started writing cURL in 1997, as a way to transfer files via protocols such as HTTP, FTP, Gopher, and many more, via a command-line interface. Several other people made important and crucial contributions to the project. Distributed under the MIT License, cURL is free software



Below are some Curl examples for testing URL Redirection for any site.
To start with, Below are some basic commands :

Syntax :
$curl [options] [URL...]

1.)   Check redirection :

OPTIONS :
-I, --head
(HTTP/FTP/FILE) Fetch the HTTP-header only! HTTP-servers feature the command HEAD
which this uses to get nothing but the header of a document. When used on an FTP or FILE file,
curl displays the file size and last modification time only.

-L, --location
(HTTP/HTTPS) If the server reports that the requested page has moved to a different location
(indicated with a Location: header and a 3XX response code),



HTTP/1.1 301 Moved Permanently
Content-Length: 177
Content-Type: text/html
Server: Apache
Date: Sun, 23 Dec 2012 11:34:42 GMT
Connection: keep-alive

HTTP/1.1 403 Forbidden
Content-Length: 480
Content-Type: text/html;charset=utf-8
Server: Apache
Date: Sun, 23 Dec 2012 11:34:43 GMT
Connection: keep-alive



HTTP/1.1 301 Moved Permanently
Content-Length: 180
Content-Type: text/html
Server: Apache
Date: Sun, 23 Dec 2012 11:37:00 GMT
Connection: keep-alive

HTTP/1.1 302 Moved Temporarily
Content-Length: 189
Content-Type: text/html
Server: Footprint Distributor V4.8
Date: Sun, 23 Dec 2012 11:37:00 GMT
Connection: keep-alive

HTTP/1.1 200 OK
Content-Length: 18364
Content-Type: text/html; charset=utf-8
ETag: "116c113-47bc-4d1812618f9c0"
Last-Modified: Sun, 23 Dec 2012 08:48:47 GMT
Accept-Ranges: bytes
Server: Apache
Date: Sun, 23 Dec 2012 11:37:00 GMT
Connection: keep-alive


2.)    Checking Redirection on based of cookies value :
Suppose we have asked L3 to redirect on basis of cookies i.e. it will first look for cookies values. If it didn’t found any cookies then it will redirect on HTTP headers. We can verify the same using below examples.

OPTIONS :
-b, --cookie
    If no ’=’ symbol is used in the line, it is treated as a filename to use to read previously stored cookie lines from, which should be used in his session if they match. Using this method also activates the "cookie parser" which will make curl record incoming cookies too, which may be handy if you’re using this in combination with the -L, --location option. The file format of the file to read cookies from should be plain HTTP headers or the Netscape/Mozilla cookie file format.
           
-H, --header
(HTTP) Extra header to use when getting a web page. You may specify any number of extra headers. Note that if you should add a custom header that has the same name as one of the internal ones curl would use, your externally set header will be used instead of the internal one. This allows you to make even trickier stuff than curl would normally do. You should not replace internally set headers without knowing perfectly well what you’re doing. Remove an internal header by giving a replacement without content on the right side of the colon, as in: -H "Host:". If you send the custom header with no-value then its header must be terminated with a semicolon, such as −H "XCustom-Header;" to send "X-Custom-Header:".
           
a.)   Using same value for Header and Cookies :

$ curl -IL http://m.staging.mcdonaldsarabia.com/mobilearabia/kuwait  -H"Accept-Language:ar" -b"cookies:country=kumait; language=ar"

HTTP/1.1 301 Moved Permanently
Content-Length: 180
Content-Type: text/html
Server: Apache
Date: Sun, 23 Dec 2012 12:05:15 GMT
Connection: keep-alive

HTTP/1.1 302 Moved Temporarily
Content-Length: 192
Content-Type: text/html
Server: Footprint Distributor V4.8
Date: Sun, 23 Dec 2012 12:05:15 GMT
Connection: keep-alive

HTTP/1.1 200 OK
Content-Length: 9388
Content-Type: text/html; charset=utf-8
ETag: "1064056-24ac-4d1826f884d80"
Last-Modified: Sun, 23 Dec 2012 10:20:54 GMT
Accept-Ranges: bytes
Server: Apache
Date: Sun, 23 Dec 2012 12:05:15 GMT
Connection: keep-alive

b.)   Now Different Values for Header and Cookies, But we will notice that which one will notice get “preference” as we have asked L3 to priorities “cookies value over HEADER”.

$curl -IL http://m.staging.mcdonaldsarabia.com/mobilearabia/kuwait  -H"Accept-Language:ar" -b"cookies:country=kumait; language=en"

HTTP/1.1 301 Moved Permanently
Content-Length: 180
Content-Type: text/html
Server: Apache
Date: Sun, 23 Dec 2012 12:09:16 GMT
Connection: keep-alive

HTTP/1.1 302 Moved Temporarily
Content-Length: 192
Content-Type: text/html
Server: Footprint Distributor V4.8
Date: Sun, 23 Dec 2012 12:09:16 GMT
Connection: keep-alive

HTTP/1.1 200 OK
Content-Length: 9779
Content-Type: text/html; charset=utf-8
ETag: "cfc025-2633-4d18272c04700"
Last-Modified: Sun, 23 Dec 2012 10:21:48 GMT
Accept-Ranges: bytes
Server: Apache
Date: Sun, 23 Dec 2012 12:09:17 GMT
Connection: keep-alive

3.)  Redirection based of User Agent(Means checking Redirect for different devices e.g. Mobiles/Desktop/ipad etc )

            OPTIONS :
-A, --user-agent
            (HTTP) Specify the User-Agent string to send to the HTTP server. Some badly done CGIs fail if this field isn’t set to "Mozilla/4.0". To encode blanks in the string, surround the string with single quote marks. This can also be set with the -H, --header option of course. If this option is used several times, the last one will be used.

c.)    Check using without any option by default will act as Desktop :


HTTP/1.1 302 Moved Temporarily
Content-Length: 186
Content-Type: text/html
Server: Footprint Distributor V4.8
Date: Sun, 23 Dec 2012 12:37:09 GMT
Connection: keep-alive

HTTP/1.1 200 OK
Content-Length: 75605
Content-Type: text/html; charset=utf-8
ETag: "8b8495-12755-4d1653ad85200"
Last-Modified: Fri, 21 Dec 2012 23:30:16 GMT
Accept-Ranges: bytes
Server: Apache
Date: Sun, 23 Dec 2012 12:37:09 GMT
Connection: keep-alive

d.)   Specifying User Agent (Checking with nokia)

$ curl -IL http://www.mcdonalds.com/mcnuggets -A "nokia"

HTTP/1.1 302 Moved Temporarily
Content-Length: 148
Content-Type: text/html
Server: Footprint Distributor V4.8
Date: Sun, 23 Dec 2012 12:38:10 GMT
Connection: keep-alive

HTTP/1.1 200 OK
Date: Sun, 23 Dec 2012 12:38:11 GMT
Server: Apache/2.2.15 (Red Hat)
X-Powered-By: PHP/5.3.3
Set-Cookie: PHPSESSID=e8dk52fi05h62h8kukt3mq3je5; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Content-Type: text/html; charset=utf-8
Set-Cookie: BIGipServerPOOL_74.205.90.114-MOMS=1929488576.20480.0000; path=/

e.)   Testing with other Agent, we have asked L3 to redirect from some specific user agent to move to different location, Here we goes :

$ curl -IL http://www.mcdonalds.com/mcnuggets -A "iphone*os*5*mobile*"

HTTP/1.1 302 Moved Temporarily
Content-Length: 192
Content-Type: text/html
Server: Footprint Distributor V4.8
Date: Sun, 23 Dec 2012 12:39:58 GMT
Connection: keep-alive

HTTP/1.1 200 OK
Content-Length: 8162
Content-Type: text/html; charset=utf-8
ETag: "8b873c-1fe2-4d1653a8c06c0"
Last-Modified: Fri, 21 Dec 2012 23:30:11 GMT
Accept-Ranges: bytes
Server: Apache
Date: Sun, 23 Dec 2012 12:39:58 GMT
Connection: keep-alive

Hope this will be useful.