Python requests Error Handling: Timeouts, Retries, and 429s
Calling an API from Python starts as one line.
data = requests.get(url).json()It works fine locally. Put it on a server for a few hours and it will break — hanging forever waiting for a response that never comes, or throwing from .json() because a 429 came back, or returning 401 because the token expired.
This article is about turning that one line into code that doesn't wake you at 3 a.m.
1. Timeouts — the Most Common and Most Damaging Omission
requests has no default timeout. If the server never responds, your program waits indefinitely. Check back hours later and it's still sitting there. Nothing in the logs. It just quietly stops.
# Bad — waits forever if the server neither dies nor answers
res = requests.get(url)
# Good — (connect timeout, read timeout)
res = requests.get(url, timeout=(3, 10))The tuple means 3 seconds to connect, 10 seconds to receive the body. Giving up quickly on the connection while allowing more room for the data transfer is usually the right shape. If you can't decide on values, even a bare timeout=10 beats having none.
2. Distinguish Between the Failures
Wrap every failure in except Exception and you lose the ability to tell what should be retried from what shouldn't. The requests exceptions break down roughly like this:
Timeout— timed out. Worth retryingConnectionError— connection failure, DNS error, network drop. Worth retryingHTTPError— a 4xx/5xx response. Depends on the status code
By status code:
- 5xx: server-side problem. Retry.
- 429: you called too often. Wait, then retry.
- 401 / 403: authentication problem. A plain retry fails identically. You need to refresh the token and try again.
- 400 / 422: your request is malformed. Retrying is pointless. Send it 100 times and it's wrong 100 times. Log it and give up immediately.
3. Retrying with Exponential Backoff
If every retry waits a flat 0.5 seconds, you're piling requests onto a server that's already struggling. The standard approach is waiting longer with each failure — exponential backoff.
import time, random, requests
RETRIABLE_STATUS = {429, 500, 502, 503, 504}
def request_with_retry(method, url, max_retries=3, **kwargs):
kwargs.setdefault("timeout", (3, 10))
last_err = None
for attempt in range(max_retries):
try:
res = requests.request(method, url, **kwargs)
# Return immediately on failures not worth retrying
if res.status_code < 400:
return res
if res.status_code not in RETRIABLE_STATUS:
res.raise_for_status()
last_err = f"HTTP {res.status_code}"
wait = backoff_seconds(attempt, res)
except (requests.Timeout, requests.ConnectionError) as e:
last_err = str(e)
wait = backoff_seconds(attempt, None)
if attempt < max_retries - 1:
time.sleep(wait)
raise RuntimeError(f"failed after {max_retries} retries: {last_err}")
def backoff_seconds(attempt, res):
# 1s -> 2s -> 4s, plus a random jitter
base = 2 ** attempt
return base + random.uniform(0, 0.5)There's a reason for the jitter — the random value added on top. When several bots fail at the same moment, they all retry at exactly the same instant, and the server gets hit by the same thundering herd again. Mixing in a random 0–0.5 seconds is enough to break that synchronization.
4. On a 429, Read Retry-After
When you get a 429 (Too Many Requests), many servers tell you when to come back. It's in the Retry-After response header. Ignore it and retry on your own schedule and you may extend the block.
def backoff_seconds(attempt, res):
if res is not None and res.status_code == 429:
retry_after = res.headers.get("Retry-After")
if retry_after:
try:
return float(retry_after) + 0.1 # wait as long as the server asked
except ValueError:
pass # date format -> fall through to default
return max(5.0, 2 ** attempt) # no header -> be generous
return 2 ** attempt + random.uniform(0, 0.5)The better answer is not getting 429s at all. Enforce a minimum interval between calls on the client side.
class RateLimiter:
"""Guarantees a minimum interval between calls"""
def __init__(self, min_interval=0.1):
self.min_interval = min_interval
self.last_call = 0.0
def wait(self):
elapsed = time.time() - self.last_call
if elapsed < self.min_interval:
time.sleep(self.min_interval - elapsed)
self.last_call = time.time()5. A 401 Calls for a Token Refresh, Not a Retry
This is the failure you'll meet most often with token-based APIs. Tokens generally have an expiry, and past it you get a 401. What's needed then isn't waiting, it's reissuing.
def call_api(path, **kwargs):
headers = kwargs.pop("headers", {})
headers["Authorization"] = get_token()
res = request_with_retry("GET", BASE + path, headers=headers, **kwargs)
if res.status_code == 401:
refresh_token() # get a fresh token
headers["Authorization"] = get_token()
res = request_with_retry("GET", BASE + path, headers=headers, **kwargs)
res.raise_for_status()
return res.json()Attempt the reissue exactly once. If 401s keep coming and you refresh indefinitely, a genuinely bad key puts you in an infinite loop.
6. .json() Fails Too
The last trap. Even with a 200 status code, the body may not be JSON. The classic case is a proxy in front of the server returning an HTML error page.
try:
data = res.json()
except ValueError:
logging.error("JSON parse failed: %s", res.text[:200]) # keep only the head
raiseLogging the whole of res.text will balloon your log files the moment a response is large. The first 200 characters are plenty to identify the cause.
Checklist
Confirm these before committing API-calling code and you'll prevent most incidents.
- Does every request have a
timeout? - Are you retrying 4xx errors that retrying can't fix?
- Does the wait grow with each failure, and is there jitter?
- Do you read
Retry-Afteron a 429? - Do you refresh the token on a 401 — and only once?
- When everything fails, does the log say what failed and why?
That last item matters more than it looks. Build beautiful retry logic and then swallow it with except: pass, and your bot doesn't die — it just stops doing anything. Failures have to make noise.
This article was written for educational and informational purposes. The code is illustrative; when applying it to a real service, check the target API's documentation and usage policies.
댓글
댓글 쓰기