Google Keyword Intelligence

Architecture

Google Keyword Intelligence has three layers around one set of domain models.

Layers

  1. Core: settings, market validation, envelopes, providers, cache, rate limits, storage, research scenarios, saved-run execution, scoring, clustering and reports under src/google_keyword_ai/.
  2. Thin wrappers: Typer exposes the core as gkai; MCP exposes the same use cases and envelopes over stdio. Neither wrapper owns research logic.
  3. Agent skill: .claude/skills/researching-google-keywords/ selects workflows, orders cheap and expensive work, and preserves interpretation caveats.

Provider I/O is asynchronous so HTTP pacing, concurrent expansion and blocking vendor clients can yield without freezing orchestration. Synchronous Google client calls are moved to worker threads. Public use cases give both wrappers a simple synchronous boundary; MCP tool functions are deliberately synchronous because the SDK offloads them from its stdio event loop.

Response contract and MCP

Every use case returns Envelope[T]: schema_version, typed data, warnings, errors, completeness, optional reason and optional run ID. CLI JSON and MCP structured output serialize this same envelope.

MCP tools must have one concrete result shape. Planning and execution are separate plan_research and research_keywords tools because a union return type makes the SDK wrap data under an extra result key and breaks CLI/MCP parity.

A refused request travels in that same envelope: completeness: empty with the reason, printed by the CLI on exit 1 and returned by MCP as ordinary structured output. Refusals are therefore raised as project errors before any provider work — a non-positive limit selects nothing and so is impossible rather than small, and an unknown expansion strategy is named against the list of known ones, because MCP publishes that argument as free strings while the CLI parses an enum.

An empty answer always names what produced it. A budget that stopped a crawl before it collected anything, or a batch of skipped requests, is reported in place of the flat "no keywords" — the two read very differently to a caller deciding whether the niche is worth another look.

Providers

All providers expose Provider.info (name, official status, stability) and is_available(). Typed capabilities add provider-specific operations:

  • Autocomplete: suggestions; unofficial and undocumented.
  • Expander: bounded fan-out over Autocomplete strategies.
  • Trends: normalized timelines, regions and related searches; unofficial fallback.
  • Google Ads: official keyword ideas and historical planning metrics.
  • Search Console: official properties and Search Analytics rows.

Failures cross the boundary as project errors and become honest partial or empty envelopes. Details: Autocomplete, expansion, Trends, Google Ads and Search Console.

Cache, throttling and storage

SQLite cache keys include provider, endpoint, canonical parameters, account scope and parser version. A setting counts as a parameter when it shapes the answer rather than the fetch: the Search Console row cap bounds the request itself, so it belongs in the key, or raising it would keep serving the short answer it was raised to replace. The Trends timezone offset and the Google Ads API version are in for the same reason -- the first decides where each bucket of the timeline begins, the second decides which API answered, and parser_version covers only our side of the parse. Per-provider TTLs limit staleness, and they measure how fast an answer goes stale rather than how long an outage lasts: a Trends fetch whose every widget was refused is not written to the cache, because storing it would make one rate-limited moment the answer for the rest of the TTL. What decides that is which widgets answered, not whether the answer had rows in it -- a keyword Google has no interest data for comes back just as empty and is cached like any other. Async rate limiters pace single-process calls; Google Ads also uses a file lock and timestamp for a shared interprocess limit.

Retries honour a Retry-After header in either form the standard allows -- a delay in seconds or an HTTP-date -- but only up to http_max_retry_after_seconds (60 by default). A longer wait is a quota to come back for rather than a retry: sleeping it out would run straight through the budget's runtime ceiling with nothing on screen, so the request fails instead and the advertised delay is reported in the error details.

SQLite uses WAL, a busy timeout, foreign keys and numbered forward migrations. Schema v1 added cached payloads; v2 added runs, stages, fingerprints and checkpoints; v3 added the rest of the original request to a run — its seed keyword and result limit — so resume and rerun repeat the question that was actually asked. Saved configuration snapshots use masked values. See pipeline and runs.

Milestones

  • M1: package, configuration, errors, logging, market, envelope and SQLite.
  • M2: HTTP, retry, cache, rate limiting, normalization and Autocomplete.
  • M3: bounded keyword expansion and language data.
  • M4: Google Trends provider and parsing.
  • M5: Google Ads ideas, metrics and interprocess throttling.
  • M6: Search Console queries and opportunities.
  • M7: niche, competitor and existing-site pipelines with budgets and dry-run.
  • M8: durable runs, checkpoints, resume, rerun and schema v2 (v3 added the seed keyword and result limit, so a repeat asks the question that was asked).
  • M9: scoring, lexical clustering, reports and analysis commands.
  • M10: CLI/MCP documentation, Claude skill and documentation consistency tests.

Scoring formulas and confidence are documented in scoring.md.

Edit on GitHub · 2026-09-04