Handling Rate Limits, Retries, and Failed Imports When Loading Economic Data

Introduction

A data import works perfectly the day you build it. The interesting question is what it does on the day the source is slow, returns half a response, or quietly rate-limits you. Most abandoned trackers did not die from a dramatic crash — they died because an import started failing and nobody found out for weeks. This guide covers the reliability layer that turns a fragile fetch into a load you can leave alone: respecting rate limits, retrying the right failures, and making the wrong ones impossible to ignore.

None of this is specific to one tool. Whether you import with Power Automate, a GitHub Action, or a Cloudflare Worker, the same three habits apply.

Respect the rate limit before you hit it

Public economic APIs publish limits, and they vary widely. FRED allows a generous number of requests but will throttle a tight loop. The World Bank and Eurostat are lenient for occasional pulls but unhappy with bursts. The BLS API distinguishes between registered and unregistered use, with much lower limits for the latter. Before you build a loop, read the source’s documented limit and design under it.

Two practices keep you safely inside any limit. First, space out requests — if you are pulling fifty series, do not fire fifty simultaneous calls; add a small delay between them so you make a steady trickle rather than a spike. Second, pull only what changed — most economic series update on a known cadence, so requesting just the latest observation (or only since your last successful run) is both faster and far gentler on the source than re-downloading history every time.

💡 Tip: A surprising amount of rate-limit trouble comes from re-fetching full history on every run. Store the date of your last successful observation and request only newer data. Your import gets faster, cheaper, and politer all at once.

Retry transient failures — but only transient ones

Failures come in two kinds, and they need opposite responses.

Transient failures are temporary: a timeout, a 429 Too Many Requests, a 503 Service Unavailable, a brief network blip. These usually succeed if you simply try again a moment later. The correct response is a retry with exponential backoff — wait a second, then two, then four — so you give the source room to recover instead of hammering it. Cap the retries (three or four attempts is plenty) so a truly down source does not trap your job forever.

Permanent failures are structural: a 401 because your key is wrong, a 404 because the series ID changed, a 400 because the request is malformed. Retrying these accomplishes nothing except wasting calls and hiding the real problem. The correct response is to stop and report.

The single most common reliability bug we see is retrying everything, including the permanent failures. That converts a clear “your API key expired” into a vague “the import is slow,” and buries the one error you needed to see.

📌 Note: A 429 response often includes a Retry-After header telling you exactly how long to wait. When it is present, honor it instead of guessing. Respecting Retry-After is the difference between a well-behaved client and one that gets blocked.

Make failure loud

Retries handle the failures that fix themselves. The failures that do not fix themselves are the ones that kill trackers, so the non-negotiable habit is this: a load that fails, or that succeeds but brings back no new data, must produce a signal a human will actually see.

The cheapest effective version is a freshness check. After each run, record the timestamp of the newest data you successfully stored. Separately — ideally in a different place from the import itself, so a total outage cannot silence the alarm — check whether that timestamp is older than it should be for each series’ update cadence. If unemployment data has not advanced in ten days when it updates monthly, that is fine; if a daily series has not advanced in three days, that is an alert.

Where the alert goes matters less than that it exists: a message to a Teams or Slack channel, an email, or a status badge on your own site all work. What does not work is a log file nobody opens. The test is simple: if your import broke today, how many days until you would know? If the honest answer is “weeks,” you have monitoring to add.

Make reruns safe

The last reliability habit is making your import idempotent — safe to run twice. When a load fails partway and you rerun it, or a retry double-writes, you do not want duplicate rows or a corrupted series. Design the write so that loading the same observation twice is harmless: upsert on a natural key like (series_id, observation_date) rather than blindly appending, or de-duplicate on that key after loading. Idempotent writes mean recovery is just “run it again,” with no cleanup, which is exactly what you want when you are recovering something you half-forgot.

Conclusion

Reliability for a data import is three habits, not a framework. Stay under the source’s rate limit by spacing requests and fetching only what changed. Retry transient failures with backoff and a cap, but let permanent failures stop and shout. And above all, make a stalled or empty load produce a signal you will see, because silence is the failure mode that actually ends trackers. Add idempotent writes so recovery is a single rerun. Build these in once and your import earns the right to be ignored — which, for anything meant to run for years, is the whole point.