World Bank Data

What You're Getting

World Bank Open Data covers 1,600+ development indicators across 260+ economies — GDP per capita, poverty rates, health metrics, education enrollment, infrastructure access, and more. Data is organized by indicator code (e.g. NY.GDP.MKTP.CD), country code (ISO 3166 alpha-3), and year. The Indicators API returns JSON with nested value, country, indicator, and date fields.

Ingestion Strategy

Use the REST Indicators API — no authentication needed. Preferred pattern: pull by indicator, requesting all countries at once with ?country=all&per_page=10000. Page through results using ?page=N if needed. For a full load, iterate over your target indicator list. Store country ISO3 code, indicator code, year, and value — plus decimal and unit metadata from the indicator definition endpoint.

import requests

def fetch_indicator(code):
    url = f"https://api.worldbank.org/v2/country/all/indicator/{code}"
    params = {"format": "json", "per_page": 5000, "mrv": 50}
    r = requests.get(url, params=params)
    meta, data = r.json()
    return data  # list of {country, indicator, date, value, ...}

Rate Limits & API Details

• No authentication required; no published rate limit but be polite (1 req/sec)
• Bulk download at databank.worldbank.org for CSV/Excel dumps of entire databases
• WDI (World Development Indicators) bulk CSV: ~200MB zipped, updated annually
• API base: api.worldbank.org/v2

Schema Stability

Indicator codes are stable long-term but occasionally retired. Country codes follow ISO 3166 with World Bank additions (e.g. XKX for Kosovo). Methodology changes happen — track lastupdated field in indicator metadata. Regional aggregates (e.g. WLD, ECS) are included alongside country rows — filter by region.id != "NA" to exclude aggregates.

Data Quality Gotchas

• Sparse coverage: many indicators have large gaps for low-income countries — model as nullable
• Aggregates mixed with countries: rows include World Bank regional groups; separate these in your staging layer
• Annual cadence: most indicators update once a year (April/May WDI release); don't poll more frequently
• Unit inconsistency: some indicators switch from current to constant USD over time — check sourceNote field

Tool Compatibility

• Python: wbdata or world_bank_data PyPI packages
• R: WDI package (CRAN)
• Airbyte: no official connector; community scripts available
• Bulk CSV download works well for warehouse loads via dbt seeds or Snowflake COPY

Benchmarking Approach

For NY.GDP.MKTP.CD (GDP current USD), expect values for ~215 countries × ~60 years ≈ 12,000+ rows with many nulls for early years. Spot-check US GDP for 2022 against IMF WEO figure — should be within 1% (different methodology). Alert if row count drops more than 5% between annual releases.