swarm-gcp: The same pipeline, on managed pieces
The lead swarm ported to Cloud Run jobs, BigQuery and Pub/Sub in Terraform, with one identity per component and a walkthrough that teaches the stack. Built, not yet deployed.
Solo — worker, Terraform, and the walkthrough · 2026
Built and tested, not yet deployed. The numbers below are from the code and the pricing pages, not from a running system, and this page will say so until that changes.
The problem
The lead swarm runs on GitHub Actions, which is a good place for a cron and a bad place for everything after it. State lives in a spreadsheet shaped database, there is no query surface, and nothing downstream can subscribe to a result.
The port was an excuse to answer a real question: what does this pipeline look like when each piece is a managed service with its own identity, and does that cost anything at this volume.
The answer is one job, one table, one topic, and a bill that stays inside the free tier at twelve runs a day.
Architecture
Two identities, one job, and a pipeline that costs nothing when it is not running.
- Cloud Scheduler triggers every two hours as its own service account
- A Cloud Run job starts, does one pass, and exits
- Collect from public sources, dedupe by external id, score in code
- Every row lands in BigQuery, partitioned by day and clustered by source
- Only what clears the threshold is published to Pub/Sub
- Undeliverable messages dead letter after five attempts
- The job service account can write one table and publish to one topic, nothing else
Key decisions
A job, not a service
The work is periodic. A Cloud Run service would sit there between runs being billed for readiness it does not need. A job starts, runs one pass, exits with a status, and the scheduler owns the question of when. That single choice is the difference between a few cents and a few dollars a month at this volume.
One identity per component, sized to the job
The worker can write one BigQuery table and publish to one topic. The scheduler can start one job. Neither can do the other’s work. If either credential leaked, that is the whole blast radius, and it is short enough to say out loud.
Retries are safe because writes are idempotent
Cloud Run retries a failed job, and a retry is a rerun. Rows carry an insert id derived from the source and the external id, so BigQuery drops the duplicate instead of adding a second copy of a lead. The alternative is a pipeline that punishes you for having failure handling.
Partitioning and clustering are the cost control
BigQuery bills for bytes scanned, so the table is partitioned by day and clustered by source and qualified. A query over yesterday reads one partition. The walkthrough has you delete the where clause and watch the bytes jump, because that is the moment the pricing model stops being abstract.
A dead source degrades the run, it does not fail it
One source failing is recorded in the structured log and the run continues. Every source failing exits non zero. A pipeline that fails completely when one API has a bad minute teaches you to ignore its alerts.
Numbers
- 2 — service accounts (one to start the job, one to do the work)
- ~7,200 — vCPU seconds a month (against a free tier of 180,000)
- 12 — runs a day (every two hours, by Cloud Scheduler)
- 8 — steps in the walkthrough (each with a command and a question to answer)
- 1 — command to remove it all (terraform destroy, then delete the project)
This has not been deployed yet, so the cost figures come from the pricing pages and the usage the code implies, not from a bill. Nothing here has run on a schedule, met a quota, or hit a real API outage in production. The Terraform validates against the provider and the worker’s tests pass, which proves the shape is right and proves nothing about the operational reality. When it runs, this page gets the real numbers and this paragraph gets shorter.
github.com/brandononchain/swarm-gcp