Introduction

One of the most popular libraries in the Python ecosystem is the infamous requests library. Requests is used primarily for creating, sending, and parsing HTTP requests and responses. Requests tag line is "HTTP for Humans" which is very appropriate for this easy to use and wholesome project.

You will find many clients written in Python utilizing this library for communication with HTTP RESTful API services. In this post, we will look at the library and talk about some common scenarios when dealing with web APIs and how to solve them cleanly with the library. We will look at basic and more advanced usages of the library which will help you write concise pythonic code.

The Basics

The top level module exposes several functions which correspond to the standard set of HTTP verbs.

import requests
result = requests.get('https://httpbin.org/get', params={'search':'this is a search term'})
assert 'search' in result.json()['args']

This is the most basic functionality of the library in which you can send HTTP requests to a given URL and a HTTP response object is returned with attributes such as the status code, headers, cookies, and response data easily accessible.

If your code only utilizes these functions, you will want to stay tuned for the more advanced usages as they can really step up your HTTP game.

The Session

When accessing a standardized web API, it is generally advised to use a Session context object. With a session created, all subsequent requests called from within the session benefit from HTTP persistence as well as shared headers and cookies among other functionality.

For example, if an API requires all requests to have a certain HTTP header set, a session will allow you to define this default behaviour.

from requests import Session

session = Session()
session.headers.update({'X-Custom-Header': 'any value'})

result = session.get('https://httpbin.org/headers')
data = result.json()

assert 'X-Custom-Header' in data['headers']
assert data['headers']['X-Custom-Header'] == 'any value'

Event Hooks

Another important, but often overlooked functionality within requests, is the ability to register event hooks. With event hooks, we can transform the HTTP response and its data before it reaches our client code.

Also, event hooks allows us to write common error handling routines in a much simpler fashion. For example, we can catch certain status codes in an event hook and handle the error within. This is a really powerful concept since the validation will automatically be applicable to all requests sent from the session.

import requests

def forbidden_error_handler(response, *args, **kwargs):
    if response.status_code == requests.codes.forbidden:
        raise Exception('Raise some custom exception')

session = requests.Session()
session.hooks['response'].append(forbidden_error_handler)

session.get('https://httpbin.org/status/{}'.format(requests.codes.forbidden))

Authentication

There are several protocols for authenticating with HTTP services using requests.

Basic auth is a standardized authentication method supported natively in which an encoded value, given a username and password, is set in the headers of the requests.

from requests import Session
from requests.auth import HTTPBasicAuth

session = Session()
session.auth = HTTPBasicAuth('username', 'password')

result = session.get('https://httpbin.org/headers')
result_data = result.json()
assert 'Authorization' in result_data['headers']
assert result_data['headers']['Authorization'].startswith('Basic')

After registering the authentication method, every request will initiate the callable with the request. The request is then modified with the necessary authentication before transport.

Token Based Authentication

Some APIs require registering with a specific endpoint to obtain an authentication token. This token is used for subsequent calls to the API. An example of this can be found in the Django REST framework in TokenAuthentication.

If this token has a time to live, you may need to write your own authentication. For custom authentication, the AuthBase can be subclassed directly to define your own authentication.

A good example to consider is the HTTPDigestAuth class. This authentication class registers authentication specific event hooks prior to sending the request. The hooks determine if the status code of the response is that of one missing authentication. If so, an auth token is generated and the request is re-sent.

Unfortunately, this particular use case requires a good bit of logic due to the complexity of request data that can be sent with the library. For example, it must roll back iterables and file descriptors before re-sending the request. Based on a particular use case, you may be able to trim down your custom authentication class considerably.

Transport Adapters

Not for the faint of heart, Transport Adapters are used for modifying the underlying connection engine within the requests library. The adapters can be mounted to a session in order to supply specific functionality to a particular set of protocols, domains, or routes.

One common use case for supplying an adaptor is for automatic retry logic. The Retry class from urllib3 can be used to specify this information.

import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

retry_policy = Retry(3, status_forcelist=[requests.codes.server_error])
adapter = HTTPAdapter(max_retries=retry_policy)

session = requests.Session()
session.mount('https://httpbin.org/status/', adapter)

url = 'https://httpbin.org/status/{}'.format(requests.codes.server_error)

try:
    result = session.get(url)
except requests.exceptions.RetryError as e:
    print('Retries exceeded. Success!')

Conclusion

As can be seen, Requests has much more functionality than meets the eye. I hope this deep dive has given you an idea of how you can improve your uses of the library to write more clean and modular code.