Mats Olsen is the co-founder and CTO of Dune, the world’s leading crypto data and analytics platform. Dune’s infrastructure is designed to serve one of the largest open-source data modelling communities in existence, enabling users to build and share crypto-related queries, dashboards, and models. Dune’s team of engineers powers this open analytics layer with a robust transformation stack that supports thousands of models.
But with scale came problems, especially with Dune’s legacy tooling. The company originally adopted dbt™ as its transformation layer in 2020, and for a time, it worked well. But over the years, the cracks began to show.
“We had a blast initially, but later on, we suffered from dbt’s programming model, because everything is solved with a macro. It's impossible to reuse the code. At a philosophical level, it felt naive in terms of how you have to refresh, and to only think about SQL as text, when it really is code. Plus, dbt’s updated pricing model really kicked us in the butt.”
Challenges Dune Faced With dbt
As Dune’s data needs grew, their dbt setup became difficult to manage. The team has one of the largest public dbt projects in the world, known as Spellbook. But this scale came with a number of friction points:
- Rigid programming models: dbt’s approach to SQL-as-text limited Dune’s ability to reason about logic, test effectively, or modularize code. Its reliance on macros made the codebase complex and difficult to manage, which only caused frustration.
- Backfill and refresh pain-points: Updating schemas or handling large datasets required manually writing individual models for each monthly slice, adding complexity and eating up valuable engineering time.
- Dramatic cost increase: dbt Cloud’s shift from seat-based to run-based pricing led to a dramatic cost increase. Given Dune’s community-driven platform, model runs could be triggered without permission, further compounding the problem.
Mats and the Dune team knew they needed a better path forward. This wasn’t just about cost. It was about building something that would last.
Discovering Tobiko: From SQLGlot to SQLMesh
It was time for a change. Dune discovered Tobiko almost by accident. While looking for a way to transpile queries during their migration from Spark SQL to Trino, Mats stumbled upon SQLGlot - Tobiko’s SQL transpiler that powers SQLMesh.
That initial collaboration left a lasting impression, and after reaching out to Toby, Tobiko’s CTO, for help with adapting SQLGlot to Dune’s custom dialect, Mats was introduced to SQLMesh, Tobiko’s next-gen alternative to dbt Core. It immediately sparked his interest.
“When Toby started talking to me about SQLMesh - from the first time I looked at it, it was obviously a better paradigm.”
Mats established a personal relationship with Toby, but when it came down to the adoption decision, he handed off the evaluation process to his data engineering team, who adopted SQLMesh because it “ended up solving problems for them.”
Still, migration wasn’t instant. With thousands of dbt models in production, moving everything over was going to be difficult. Instead, Dune carved out their hardest-to-manage workloads, starting with a token price pipeline as an initial testbed for SQLMesh.
How SQLMesh Transformed Dune’s Workflow
From there, SQLMesh has become Dune’s go-to tool for new development and complex workloads. The team quickly realized that the “grass is greener over on the SQLMesh side”.
“Whenever we have issues with something in dbt, we now say, ‘we should just move this to SQLMesh now’”
The Dune team saw a major step up over dbt as a result of SQLMesh’s hosted orchestration and virtual data environments. These features remove the burden of managing pipelines and saves engineering time by automatically breaking huge backfills into bite-sized and easily managed chunks. Tobiko Cloud helped to eliminate manual scheduling and rerunning failed jobs, which are also common frictions with dbt.
Another win was the ability to run Python models. Dune’s internal token metadata APIs are now pulled directly into their data lake using Python-based SQLMesh models, something that wasn’t possible with their previous dbt setup. This shift allowed the Dune team to focus on what mattered most. As Mats put it:
“For us, the question is: What do I want my engineers to be doing? Do I want my engineers to work on the business logic or hosting and orchestrating pipelines? There's a cutoff where it's worth doing it yourself, but certainly in this case, it was worth working with Tobiko Cloud to just not have to worry about it at all.”
What’s next for Dune and SQLMesh
Dune is just getting started with SQLMesh and Tobiko Cloud, with a broader adoption of SQLMesh across its ecosystem on the horizon. Some of their largest models, like Solana blockchain analytics, are being considered for full migration. Dune hopes to simplify these pipelines using SQLMesh’s smarter model execution.
Conclusion
For Dune, switching from dbt to Tobiko’s SQLMesh and Tobiko Cloud wasn’t just about cost savings or shiny new tech. It was about dissolving the bottlenecks that come with a legacy approach to data modeling. The implementation of SQLMesh has unlocked faster iteration, simpler backfills, reduced costs, and cleaner infrastructure for Dune. And as their data needs grow, Tobiko Cloud and SQLMesh continue to help the Dune team focus on building value, rather than babysitting pipelines.
