57.8 Mocking HTTP in Tests with responses and httpretty
Testing HTTP interactions presents a unique challenge. Unlike many parts of an application, you cannot control the external server’s behavior, its response time, or its availability. Relying on live APIs for tests leads to a fragile, slow, and non-deterministic test suite. The solution is to intercept HTTP requests at the library level and return predefined responses, a practice known as mocking. For the modern httpx library, two of the most robust tools for this task are the responses library and httpretty.
The Philosophy of HTTP Mocking
The core principle behind libraries like responses and httpretty is to patch the low-level socket communication or the HTTP client’s internal transport mechanism. When you register a mock response, these libraries instruct the underlying networking code to bypass the actual network call. Instead, they immediately return a response object constructed from your provided data (status code, headers, body). This approach is superior to mocking the httpx client object itself because it tests the entire stack—your code, the client, and the request-building logic—ensuring your application builds the correct request URL, headers, and body. It avoids the false confidence that can come from mocking at too high a level.
Using the responses Library with httpx
The responses library is a popular, powerful choice specifically designed for mocking the requests library, but it also provides excellent support for httpx. It uses a decorator-based API that is both intuitive and flexible.
import httpx
import responses
import pytest
@responses.activate # Activates the mocking for this function
def test_get_user_success():
# Register a mock response for a specific URL
mock_json = {"id": 123, "name": "Alice"}
responses.get(
"https://api.example.com/v1/users/123",
json=mock_json,
status_code=200,
)
# Your code that uses httpx runs as normal, but is intercepted
client = httpx.Client()
response = client.get("https://api.example.com/v1/users/123")
data = response.json()
# Assert on the response your code received
assert response.status_code == 200
assert data["name"] == "Alice"
# Verify the request was made exactly once (optional but recommended)
assert len(responses.calls) == 1
assert responses.calls[0].request.url == "https://api.example.com/v1/users/123"
def test_post_user_created():
responses.post(
"https://api.example.com/v1/users",
json={"id": 999, "name": "Bob"},
status_code=201,
match=[responses.matchers.json_params_matcher({"name": "Bob"})] # Match request body
)
client = httpx.Client()
response = client.post(
"https://api.example.com/v1/users",
json={"name": "Bob"} # This request body will be matched
)
assert response.status_code == 201
Using httpretty for Low-Level Control
httpretty takes a slightly different approach by mocking the socket layer itself. This makes it protocol-agnostic and able to work with virtually any HTTP client library, including httpx, without requiring special adapters.
import httpx
import httpretty
def test_with_httpretty():
# Register the mock before activating
httpretty.register_uri(
httpretty.GET,
"https://api.example.com/status",
body='{"status": "ok"}',
content_type="application/json",
status=200
)
# Activate the mocking
httpretty.enable()
try:
# This request will be intercepted by httpretty
response = httpx.get("https://api.example.com/status")
assert response.status_code == 200
assert response.json()["status"] == "ok"
# Check what request was made
last_request = httpretty.last_request()
assert last_request.method == "GET"
finally:
# Critical: Always disable afterwards to avoid leaking mocks
httpretty.disable()
httpretty.reset()
Common Pitfalls and Best Practices
A frequent pitfall is incomplete mocking. If your test makes a request to a URL you haven’t explicitly mocked, responses will raise a ConnectionError, and httpretty will typically allow the real request to pass through, potentially causing a slow or unpredictable test. Always ensure your mocks cover all expected requests.
Best practices include:
- Use
responses.activateas a decorator or context manager to automatically handle the mocking scope, preventing mocks from leaking into other tests. - Always disable and reset
httprettyin afinallyblock or using a test framework fixture (e.g.,pytest’sautouse) to ensure one test’s mocks don’t interfere with another. - Be specific with your URL matching. Using regex patterns or the
matchparameter inresponsesensures your mock only responds to the exact request you expect, catching bugs where the wrong URL is constructed.responses.get( responses.GET, url='https://api.example.com/items/.+', # Regex pattern json={"item": "data"} ) - Test failure scenarios. Don’t just test 200 OK responses. Mock 404s, 500s, and network timeouts to ensure your application handles errors gracefully.
responses.get( "https://api.example.com/error", body=ConnectionError("Network problem"), status=500 ) - Avoid over-mocking. While these tools are powerful, they can also lead to tests that merely confirm your mock was set up correctly. For integration tests, consider using a fake server or the
pytest-httpxpackage, which provides a morehttpx-native mocking interface. The goal is to mock the external dependency (the API), not the internal implementation (the HTTP call).