dbt and Airflow implementation: tested transformation layers, DAG design, CI/CD for data, and migration off fragile cron jobs and stored procedures.
Adopted the tools, kept the problems
Installing dbt does not make a project maintainable. We regularly review dbt repos with one flat directory of 200 models, no tests, and business logic copy-pasted across a dozen files. Airflow tends to arrive with the same pathology: DAGs with hard-coded dates, tasks that are not idempotent so a retry corrupts data, and no alerting, so failures are noticed when someone opens the UI. The tools are correct, the practices around them are what break.
What we implement
- dbt project structure with staging, intermediate, and mart layers plus clear naming conventions
- Test coverage on the assumptions models depend on: uniqueness, referential integrity, accepted values, freshness
- Incremental model strategies so large tables do not fully rebuild every run
- Airflow DAG design with idempotent tasks, correct dependencies, sensible retries, and real alerting
- CI that runs dbt build on every pull request, so a broken model never reaches production
- Generated documentation and lineage, plus enablement so your analysts own the project
How we work
Review the current project or scripts and identify what is fragile versus merely untidy
Restructure the transformation layer with tests, refactoring duplicated logic into shared models
Rebuild orchestration with idempotent tasks and alerting that reaches a person
Add CI, then train the team on the workflow so it holds after we leave
Typical stack
Frequently asked questions
Airflow has the larger ecosystem, more managed options, and a much bigger hiring pool, which is why we default to it. Dagster has a genuinely better developer experience and treats data assets as first-class, which suits teams with strong engineering practices starting fresh. Either is defensible; the deciding factor is usually who maintains it.
Almost always, and it is usually worth doing. dbt gives you version control, testing, documentation, and lineage, none of which stored procedures offer. The migration is mostly mechanical, and the real gain is that transformation logic becomes reviewable in pull requests instead of living undocumented in the database.
Every model should assert its primary key is unique and not null, and every relationship it depends on should be tested. Beyond that, test the assumptions that would cause a wrong number rather than a failed run, since those are the ones that reach clients. Perfect coverage is not the goal; catching silent wrongness is.
Both, and the training matters more. A dbt project nobody on your team can extend has just moved the bottleneck to us, which is not a good outcome for you. Handover includes the repo, the CI, the docs, and working sessions with your analysts.
Go deeper
Review your dbt and Airflow setup
Point us at the repo or describe how transformations run today, and we will report what is fragile and what to fix first.
Start a project