Interacting with the Web

In this section, you will learn how to interact with the Web through HTTP requests, both for getting data and posting data to the Web. You will learn about sending and getting requests to and from websites and also analyzing those responses.

Getting ready

Start by downloading and installing the Requests.jl package of Julia, which is available at Pkg.add("Requests").

Make sure that you have an active Internet connection while reading and using the code in the recipe, as it deals with interacting with live websites on the Web. You can experiment with this recipe on the website http://httpbin.org , as it is designed especially for such experiments and tutorials.

This is how you use the Requests.jl package and import the required modules:

1. Start by importing the package:

   Pkg.add("Requests")
   

2. Next, import the necessary modules from the package for quick use. The modules that will be used in this recipe are get, post, put, and delete. So, this is how to import the modules:

   import Requests: get, post
   

How to do it...

Here, you will learn how to interact with the Web through the HTTP protocol and requests. You will also learn how to send and receive data, and autofill forms on the Internet, through HTTP requests.

GET request

1. The GET request is used to request data from a specified web resource. So, this is how we send the GET request to a website:

   get("url of the website")
   

2. To get requests from a specific web page inside the website, the query parameter of the GET command can be used to specify the web page. This is how you do it:

   get("url of the website"; query = Dict("title" => 
           "page number/page name"))
   

3. Timeouts can also be set for the GET requests. This would be useful for identifying unresponsive websites/web pages. The timeout parameter in the GET request takes a particular numeric value to be set as the timeout threshold; above this, if the server does not return any data, a timeout request will be thrown. This is how you set it:

   get("url of the website"; timeout = 0.5)
   

   • Here, 0.5 means 50 ms.
4. Some websites redirect users to different web pages or sometimes to different websites. So, to avoid getting your request repeatedly redirected, you can set the max_redirects and allow_redirects parameters in the GET request. This is how they can be set:

   get("url of the website"; max_redirects = 4)
   

5. Now, to set the allow_redirects parameter preventing the site from redirecting your GET requests:

   get("url of the website"; allow_redirects = false)
   

   • This would not allow the website to redirect your GET request. If a redirect is triggered, it throws an error.
6. The POST request submits data to a specific web resource. So, this is how to send a post request to a website:

   post("url of the website")
   

7. Data can be sent to a web resource through the POST request by adding it into the data parameter in the POST request statement:

   post("url of the website"; data = "Data to be sent")
   

8. Data for filling forms on the Web also can be sent through the POST request through the same data parameter, but the data should now be sent in the form of a Julia dictionary data structure:

   post("url of the website"; data = Dict(First_Name => "abc",
           Last_Name => "xyz" ))
   

9. Data such as session cookies can also be sent through the POST request by including the session details inside a Julia Dictionary and including it in the POST request as the cookies parameter:

   post("url of the website"; cookies = Dict("sessionkey" => "key"))
   

10. Files can also be sent to web resources through the POST requests. This can be done by including the files in the files parameter of the POST request:

    file = "xyz.jl"
    post("url of the website"; files = [FileParam(file), "text/julia", 
            "file_name", "file_name.jl"])
    

There's more...

There are more HTTP requests with which you can interact with web resources such as the PUT and DELETE requests. All of them can be studied in detail from the documentation for the Requests.jl package, which is available at  https://github.com/JuliaWeb/Requests.jl .