Python’s Requests Library Tutorial

There are a lot of libraries in the Python that can take care of HTTP for us. To name a few: requests, urllib, httplib2, urllib3, grequests, aiohttp.. Most popular is requests and the reason why is simple: this library builds on top of “urllib3,” but it allows you to tackle the majority of HTTP use cases in code that is short, pretty, and easy to use.

Installing requests can be done easily through pip. Execute the following in a command-line window (the “-U” argument will make sure to update an existing version of requests should there already be one):

pip install -U requests

Next, create a Python file (“tutorial.py” is a good name), and enter the following:

import requests
url = 'https://en.wikipedia.org/wiki/Python_(programming_language)'
r = requests.get(url)
print(r.text)

Let’s take a look at what’s happening in this short example:

• First, we import the requests module. If you’ve installed requests correctly on your system, the import line should simply work without any errors or warnings.

• We create a variable url with the content of https://en.wikipedia.org/wiki/Python_(programming_language). Try opening this web page in your browser.

• We use the requests.get method to perform an “HTTP GET” request to the provided URL. In the simplest case, we only need to provide the URL of the page we want to retrieve.

• The requests.get method returns a requests.Response Python object containing lots of information regarding the HTTP reply that was retrieved. Requests takes care of parsing the HTTP reply so that you can immediately start working with it.

• r.text contains the HTTP response content body in a textual form.

Status Codes

>>> r.status_code
200

By accessing .status_code, you can see the status code that the server returned: 200
For example, a 200 OK status means that your request was successful, whereas a 404 NOT FOUND status means that the resource you were looking for was not found.

Headers

The headers can give you useful information, such as the content type of the response payload and a time limit on how long to cache the response. To view these headers, access .headers:

>>> r.headers
{'date': 'Mon, 07 Nov 2022 20:36:26 GMT', 'server': 'mw1322.eqiad.wmnet',... }

.headers returns a dictionary-like object, allowing you to access header values by key. For example, to see the content type of the response payload, you can access Content-Type:

>>> r.headers['content-type']
text/html; charset=UTF-8

Whether you use the key 'content-type' or 'Content-Type', you’ll get the same value.

Using GET Parameters

We use the GET parameters to pass information in a key-value pair format to a web server through an URL. The get method allows us to pass a dictionary of key-value pairs using the params argument.

We use the [httpbin.org] (http://httpbin.org/) website in this tutorial. The httpbin tool is a free and simple HTTP request-and-response service that provides a set of URL endpoints. We use these endpoints to test various ways of working with HTTP operations. We will use the httpbin tools because it helps us focus on learning the Python Requests library without setting up a real web server or using an online web service.

url = 'http://httpbin.org/get'
payload = {
    'website':'mmkernel.com',
    'courses':['Python','SQL']
    }
r = requests.get(url, params=payload)
print('Response Content:\n',r.text)

Run the code above. You’ll see the following output:

Response Content:
 {
  "args": {
    "courses": [
      "Python", 
      "SQL"
    ], 
    "website": "mmkernel.com"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.28.1", 
    "X-Amzn-Trace-Id": "Root=1-636a452a-01df123f7aae4bd80565eebc"
  }, 
  "origin": "2.211.174.26", 
  "url": "http://httpbin.org/get?website=mmkernel.com&courses=Python&courses=SQL"
}

The response content is in JSON format, and the key-value pairs that we passed through the params argument appear in the args section of the response. Also, the url section contains the encoded URL along with the parameters passed to the server.

Using POST Request

We use the POST request to submit data collected from a web form to a web server. To do this in the Requests library, you need to first create a data dictionary and assign it to the data argument of the post method. Let’s look at an example using the post method:

url = 'http://httpbin.org/post'
payload = {
    'website':'mmkernel.io',
    'courses':['Python','SQL']
    }
r = requests.post(url, data=payload)
print('Response Content:\n',r.text)

Run the code above. You’ll see the following output:

Response Content:
 {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "courses": [
      "Python", 
      "SQL"
    ], 
    "website": "mmkernel.io"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "46", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.28.1", 
    "X-Amzn-Trace-Id": "Root=1-636a4700-7dba801755a7a16a12e9bdb2"
  }, 
  "json": null, 
  "origin": "2.211.174.26", 
  "url": "http://httpbin.org/post"
}


Process finished with exit code 0

This time, the submitted data through the post method appears in the form section of the response rather than in the args because instead of sending the data as part of the URL parameters, we sent them as form-encoded data.

Handling Exceptions

Some exceptions may occur while communicating with a remote server. For instance, the server can be unreachable, the requested URL doesn’t exist on the server, or the server doesn’t respond in a proper timeframe. In this section, we’ll learn how to detect and resolve HTTP errors using the HTTPError class of the Requests library.

import requests
from requests import HTTPError

url = "http://httpbin.org/status/404"
try:
    r = requests.get(url)
    r.raise_for_status()
    print('Response Code:', r.status_code)
except HTTPError as ex:
    print(ex)

We learned about one of the most powerful and downloaded Python libraries. We discussed the basics of HTTP and different aspects of the Python Requests library. 

To be continued…