Usage-API Sync — System Overview

The Problem

Cloudflare has two billing systems that must agree: PAYGO (self-serve) and Enterprise (contract). When the PAYGO team changes how a product is billed, the Enterprise side has no way to know. Changes slip through, causing billing discrepancies that can go unnoticed for months.

The Solution

A system that automatically watches the PAYGO billing repo for changes, figures out which Enterprise products are affected, and alerts the team within 24 hours — with a JIRA ticket and a proposed code fix ready to go.

How it works

🔎

1. Watch for Changes

Every day, the system checks the PAYGO billing repo for newly merged code changes. It looks specifically at the files that define billing routes — the rules for how each product's usage is calculated.

Runs daily

🔬

2. Parse What Changed

For each code change, the system reads the diff and identifies: was a billing route added, modified, or removed? Did the query logic change? Did the data source change?

Automated

🗺

3. Map to Enterprise Products

A hand-built mapping file connects all 129 PAYGO billing routes to their Enterprise counterparts. The system looks up each changed route to find which Enterprise product is affected — accounting for the fact that names often differ between the two systems.

Context layer

💬

4. Alert the Team

A formatted notification is sent to the team's Google Chat space with the affected products, a summary of what changed, links to the code changes, and who made them.

Automated

📋

5. Create a Tracking Ticket

A JIRA ticket is generated with all the details: which routes changed, which Enterprise files need updating, and a proposed action plan.

Engineer-triggered

🚀

6. Propose the Fix

The system reads the PAYGO code change and the Enterprise SQL side-by-side, proposes the equivalent edit, and opens a Draft merge request for an engineer to review. For brand-new products, it hands off to the pipeline scaffolding tool.

AI-assisted

What it produces

💬

Chat Alert

Daily notification to the team with affected products, change summary, and action items

📋

JIRA Ticket

Pre-filled ticket with change details, affected files, and proposed action plan

🚀

Draft Code Fix

A merge request with proposed SQL changes, ready for engineer review

129

PAYGO billing routes mapped

Enterprise products covered

<24h

Detection time (was months)

Phases: detect, notify, ticket, fix

Before

Manual checks (or no checks at all)
Billing drift discovered months later
No mapping between PAYGO and Enterprise product names
Engineers write JIRA tickets from scratch
No proposed fix — start from zero

→

After

Automated daily detection
Team alerted within 24 hours
129 routes mapped to Enterprise products
JIRA tickets auto-generated
Draft code fix proposed by AI

usage-api → variable-usage sync

Detect PAYGO billing changes · Notify VB team · Create tickets · Draft MRs

nihit/DIA-41528

⚙

cloudflare/data/usage-api

PAYGO / self-serve billing · GitLab project 4410 · 129 billing routes

SOURCE OF TRUTH

▼engineer merges MR touching internal/sources/

⏱ Automated Path

Python CLI runs on schedule (cron / Airflow)
uv run main.py check --notify
Polls GitLab REST API every day

✍ Interactive Path

Engineer loads the skill in OpenCode
Load usage-api-sync skill
Uses GitLab MCP (already authed)

▼

Automated Script

scripts/python/usage_api_monitor/

Fetch Merged MRs

gitlab_client.py calls GitLab REST API
GET /projects/4410/merge_requests?state=merged
Lookback window: configurable (default 7 days)

httpxREST API

Filter Billing Diffs

For each MR, fetch /changes endpoint.
Keep only files under internal/sources/

sources.go · *_reader.go

Parse Diffs

diff_parser.py extracts route-level changes:
• New Route{} structs → added
• Removed Route{} structs → removed
• Changed Source/Reader → modified
• *_reader.go changes → query logic

regex

Resolve to VB Products

route_mapper.py loads the YAML mapping.
Each route → vb_product, vb_table, vb_config_group
Flags unmapped routes as NEW

YAML

Dedup & Notify

state.py checks .monitor_state.json — skips already-seen MRs.
notifier.py builds Card V2 JSON and POSTs to GChat webhook.

httpx POST

CONTEXT LAYER

Shared mapping file read
by both paths

129 routes

101 mapped

28 PAYGO-only

← reads →

route_mapping.yaml

route_name → vb_product
route_name → vb_table
route_name → config_group
+ source, retention, notes

▼

resolution

DECISION ENGINE

Mapped? → update VB SQL
Unmapped? → new pipeline
PAYGO-only? → verify w/ PM
Removed? → flag for review

OpenCode Skill

.opencode/skills/usage-api-sync/

Fetch Merged MRs

AI agent uses gitlab-mcp-server tool
gitlab.request({ path: '/projects/4410/merge_requests' })
Auth handled by MCP — no tokens needed

GitLab MCP

Filter & Parse Diffs

Agent fetches /changes for each MR,
filters for internal/sources/,
uses LLM reasoning to classify changes

MCP

Resolve & Analyze

Reads route_mapping.yaml from disk.
Applies decision rules from SKILL.md.
Produces English-language summary of impact.

YAMLLLM

Generate JIRA Content

Pre-fills ticket: summary, description table,
affected VB files, proposed action.
Uses jira CLI or Jira MCP to create.

Jira CLI

Draft MR via GitLab MCP

Proposes SQL edits to VB query files.
Opens a Draft MR via POST /merge_requests.
For new products: invokes
setup-variable-usage-pipeline skill.

GitLab MCP

▼

💬

GChat Notification

Card V2 message to
"DIA <> Variable Billings" space

Products affected, change summary,
MR links, authors, action required

📋

DIA JIRA Ticket

Project: DIA · Type: Task
Labels: variable-billing, usage-api-sync

Change details table, affected VB files,
proposed action (update SQL / new pipeline)

🚀

Draft MR

Branch with proposed changes to
VB SQL / config / mappings

Auto-generated from usage-api diff
via GitLab MCP POST /merge_requests

// Decision Rules

IFRoute changed AND vb_product is not nullUPDATE VB SQL

IFNew route AND not in mapping file at allNEW PIPELINE + TICKET

IFNew route AND vb_product: null (marked PAYGO-only)VERIFY WITH PM

IFRoute removed (possible product sunset)FLAG FOR REVIEW

IFReader function changed (query logic) but route definition unchangedHIGH IMPACT — TICKET

IFSource field changed on a route (upstream table swap)TICKET + VERIFY SOURCE

IFOnly DaysRetention changedNO VB ACTION

IFOnly Description changedUPDATE MAPPING NOTES

IFMR title contains "fix" or "bug"APPLY SAME FIX TO VB

// File Map

context/usage_api_route_mapping.yaml

129 routes → VB products (shared context)

.opencode/skills/usage-api-sync/SKILL.md

4-phase skill (detect, notify, JIRA, MR)

scripts/python/usage_api_monitor/main.py

CLI entrypoint (check, jira-content, reset-state)

scripts/python/usage_api_monitor/gitlab_client.py

GitLab API client (fetch MRs, diffs)

scripts/python/usage_api_monitor/diff_parser.py

Unified diff parser (routes, readers)

scripts/python/usage_api_monitor/route_mapper.py

YAML loader, route → VB resolution

scripts/python/usage_api_monitor/notifier.py

GChat Card V2 webhook sender

scripts/python/usage_api_monitor/state.py

Dedup state (JSON, tracks seen MR iids)

usage-api:internal/sources/sources.go

Upstream: all 129 PAYGO route definitions

Keeping Enterprise Billing in Sync with PAYGO

The Problem

The Solution

1. Watch for Changes

2. Parse What Changed

3. Map to Enterprise Products

4. Alert the Team

5. Create a Tracking Ticket

6. Propose the Fix

Chat Alert

JIRA Ticket

Draft Code Fix

Before

After

usage-api → variable-usage sync

cloudflare/data/usage-api

⏱ Automated Path

✍ Interactive Path

Automated Script

Fetch Merged MRs

Filter Billing Diffs

Parse Diffs

Resolve to VB Products

Dedup & Notify

CONTEXT LAYER

route_mapping.yaml

DECISION ENGINE

OpenCode Skill

Fetch Merged MRs

Filter & Parse Diffs

Resolve & Analyze

Generate JIRA Content

Draft MR via GitLab MCP

GChat Notification

DIA JIRA Ticket

Draft MR

// Decision Rules

// File Map