Pipeline examples¶
Choose a task, then use its canonical recipe to set up, validate, deploy, run and inspect a pipeline with synthetic data.
| Task | Flow | Recipe |
|---|---|---|
| Enrich API orders while keeping unknown customers | HTTP + PostgreSQL → S3 JSON | Customer enrichment |
| Reconcile payments, refunds and unmatched ledger entries | S3 CSV + PostgreSQL → BigQuery | Settlement reconciliation |
| Apply spreadsheet overrides before CRM upsert | S3 XLSX + PostgreSQL → HubSpot | Contact overrides |
| Project document inventory into relational reporting | MongoDB → MySQL | Inventory projection |
| Archive a file without changing its bytes | HTTP → S3 | File archive |
The public examples repository contains the pipeline definitions, operations, binding templates, sample data and expected results. Use its manifest to find inputs programmatically.
Before you run¶
Start with the repository's shared setup and execution guide. It covers authentication, Git authoring, connections, saved operations, bindings, deployment, run inspection and cleanup. Use dedicated test resources and grant the provider permissions listed in each recipe.
Build the CLI from the same Core revision as your application. See CLI installation and pipeline specifications for setup.
The original runs used the source revision recorded with each recipe. Separate offline checks used public CLI v0.1.0-alpha.6; alpha.5 lacked the required left-join support. Build a matching CLI for your application before running them.
Test fixtures and results¶
The five recipes above were run locally and their destination output was compared with the expected results. PostgreSQL, MongoDB, MySQL and S3-compatible storage used local containers. HTTP used a synthetic fixture, while BigQuery and HubSpot used local protocol mocks. To run against Google Cloud or HubSpot, configure your own account and permissions.
The examples repository also includes an advanced settlement review demo with local execution and reporting-table checks. Its specification passed offline validation with public CLI alpha.6.
Each recipe records its executed version, run and step results. The coverage and friction ledger lists test scope and known failures, including a transform-contract mismatch and a missing binding.
For a guided introduction with downloadable inputs, follow the PostgreSQL-to-S3 walkthrough.