Of course! Let's dive deep into HTTP headers in Python. They are a fundamental part of web communication, and Python provides excellent tools to work with them.

We'll cover:

1. What are HTTP Headers? (A quick refresher)
2. Working with Headers in the requests Library (The most common and recommended way)
3. Working with Headers in http.client (Python's built-in, lower-level library)
4. Common Headers and Their Uses

What are HTTP Headers?

HTTP headers are key-value pairs that are sent between a client (like your Python script) and a server (like a website) as part of an HTTP request or response. They provide metadata about the request or response, such as:

• Content-Type: The format of the data being sent (e.g., application/json, text/html).
• Authorization: Credentials to authenticate a user (e.g., for an API key).
• User-Agent: Information about the client making the request (e.g., MyCoolApp/1.0).
• Content-Length: The size of the request/response body in bytes.
• Cookies: Data stored on the client to remember state.

Working with Headers in the requests Library

The requests library is the de facto standard for making HTTP requests in Python. It's built on top of urllib3 and provides a much simpler, more human-friendly API.

Installation

If you don't have it installed:

pip install requests

A. Sending Custom Headers in a Request

You pass headers as a dictionary to the headers parameter of any request function (get, post, put, etc.).

Key Point: The dictionary keys should be strings, and the values should also be strings. The requests library will automatically handle the formatting for the HTTP request (e.g., capitalization).

Example: Sending a User-Agent and an Accept header

import requests
# Define the headers as a Python dictionary
headers = {
    'User-Agent': 'MyPythonScraper/1.0',
    'Accept': 'application/json' # We want JSON data back
}
# URL of the API we want to call
url = 'https://httpbin.org/get'
# Make the GET request with our custom headers
response = requests.get(url, headers=headers)
# Check if the request was successful
response.raise_for_status() # Raises an exception for bad status codes (4xx or 5xx)
# Print the server's response to see our headers were received
print("Status Code:", response.status_code)
print("Response JSON (showing received headers):")
# The 'headers' key in the response JSON shows what the server received
print(response.json()['headers'])

Output:

Status Code: 200
Response JSON (showing received headers):
{
  'Accept': 'application/json',
  'Accept-Encoding': 'gzip, deflate',
  'Host': 'httpbin.org',
  'User-Agent': 'MyPythonScraper/1.0',
  'X-Amzn-Trace-Id': 'Root=...'}

Notice how requests correctly sent our User-Agent and Accept headers.

B. Accessing Headers from a Response

When you receive a response, you can access its headers using the headers attribute. This attribute is a case-insensitive dictionary-like object.

Example: Accessing Content-Type and Content-Length

import requests
url = 'https://api.github.com'
response = requests.get(url)
# Access headers using the .headers attribute
# It's case-insensitive, so 'content-type' and 'Content-Type' both work
content_type = response.headers['content-type']
content_length = response.headers['content-length']
print(f"Content-Type: {content_type}")
print(f"Content-Length: {content_length}")
# You can also use .get() for safe access (returns None if key doesn't exist)
server_header = response.headers.get('Server')
print(f"Server: {server_header}")

Output:

Content-Type: application/json; charset=utf-8
Content-Length: 6623
Server: GitHub.com

C. Working with Cookies

The requests library has a dedicated Session object for handling cookies and persistent connections, which is more efficient than making individual requests.

import requests
# Create a session object
session = requests.Session()
# 1. First request to log in (set a cookie)
# This is a hypothetical login endpoint
login_url = 'https://httpbin.org/cookies/set'
# Set some cookies
session.get(login_url, cookies={'session_id': '12345', 'user_token': 'abcdef'})
# 2. Second request to a protected page
# The session automatically includes the cookies from the previous request
protected_url = 'https://httpbin.org/cookies'
response = session.get(protected_url)
print("Cookies sent by the session:")
print(response.json())

Output:

Cookies sent by the session:
{
  'cookies': {
    'session_id': '12345',
    'user_token': 'abcdef'
  }
}

Working with Headers in http.client (Built-in)

For simpler scripts or environments where you can't install external libraries, Python's built-in http.client module can be used. It's more verbose and lower-level.

This example demonstrates how to add headers to a request and read them from a response.

import http.client
import json
# For httpbin.org, we use the 'host' and 'port'
host = 'httpbin.org'
port = 443 # Use 443 for HTTPS
# --- 1. Sending a Request with Headers ---
print("--- Sending Request ---")
# Create an HTTPS connection object
conn = http.client.HTTPSConnection(host)
# Define headers as a dictionary
headers_to_send = {
    'User-Agent': 'MyBuiltInApp/1.0',
    'Accept': 'application/json',
    'X-Custom-Header': 'Pythonista'
}
# The path and the headers dictionary are passed to the request method
conn.request("GET", "/get", headers=headers_to_send)
# Get the response from the server
response = conn.getresponse()
print(f"Status: {response.status} {response.reason}")
print("Request Sent Headers:")
# To see what the server received, we can inspect the response body
# httpbin.org echoes back the request in its JSON response
data = response.read().decode('utf-8')
response_data = json.loads(data)
print(json.dumps(response_data['headers'], indent=2))
# --- 2. Accessing Headers from a Response ---
print("\n--- Reading Response Headers ---")
# The response object has a .getheader() method for case-insensitive access
content_type_header = response.getheader('Content-Type')
server_header = response.getheader('Server')
print(f"Content-Type from response: {content_type_header}")
print(f"Server from response: {server_header}")
# Always close the connection
conn.close()

Output:

--- Sending Request ---
Status: 200 OK
Request Sent Headers:
{
  "Accept": "application/json",
  "Host": "httpbin.org",
  "User-Agent": "MyBuiltInApp/1.0",
  "X-Amzn-Trace-Id": "Root=...",
  "X-Custom-Header": "Pythonista"
}
--- Reading Response Headers ---
Content-Type from response: application/json
Server from response: nginx

Common Headers and Their Uses

Summary: Which one should I use?