Our team at Tobiko puts a lot of thought into reducing the mental load of building and maintaining large data pipelines as part of your daily workflows.

That's why we've made a set of transformative enhancements to the SQLMesh CLI output. Whether you’re a power user or just getting started, these changes will make you more confident that SQLMesh is working exactly as you intended.

Every element you see in your terminal going forward is surgically placed to:

  • Help you understand SQLMesh’s operations at a glance
  • Better pattern match as you get used to the output
  • Minimize scrolling and left-to-right eye strain
  • Reach for verbosity when needed
  • Ensure you'll never need to second-guess whether SQLMesh is doing exactly what you want

Why work on this now?

Developer experience is something near and dear to our hearts.

When you choose to use SQLMesh in production as your daily driver to build data pipelines, it becomes something you touch dozens of times a day (at minimum!). In other words, it plays a meaningful part in shaping how you work.

We’ve had the honor of seeing exponential adoption of SQLMesh (think: we already have multi-repo power users), and we’ve seen what works for the CLI and its outputs in theory vs. practice. This update is the result of face-to-face conversations, posts we've seen from you in our community Slack, thousands of error logs, and watching people learn in real time during screen sharing sessions-all of which led us to overhaul our CLI output.

We want it to feel useful, usable, and beautiful each time you type, press enter, scroll, scan, press y to apply a plan, pattern match, and debug. It should let you instantly know what happened and why, and be confident in your next step.

‍

These questions served as our design guide:

  • Can I tell whether SQLMesh ran my data and unit tests based on code changes?
  • Can I understand all the operations at a glance with what succeeded/failed and at what stage?
  • Can the vocabulary (familiar/new) be reasoned through?
  • Are there any notable changes in the output from the last three times I ran it?
  • Is the CLI output giving me too much or too little information? And how can I adjust verbosity?
  • How does the CLI output push me to read the docs or ask in community Slack to understand what’s going on (or not!)?

Expectations vs. Reality

Great CLI experiences give you just enough information when things are working, without feeling like homework, but all the context and detail you need when things break or “look off”.

You should get enough clarity upfront that you should not need to turn to Slack, docs, or ChatGPT to solve problems - at least, not on the first or second try.

Let’s take a simple example. Assume I’m new to SQLMesh, and want to try out incremental models mixed in with familiar model kinds, such as full tables/views.

When I apply the change with sqlmesh plan dev , here’s what it looks like in our previous output:

It picked up my code changes as I’m adding net new models. It tells me to backfill these same models and identify that two are full refresh and a couple have specific date ranges. I can reason that I’ll get 2 fully refreshed tables and my incremental models will have data ranges to distinguish how they process specific intervals.

Then I notice it’s building physical tables, but one of them is a view, which makes little sense.  Views don’t persist data, so a physical table should not be created I spot as well that there’s evaluation language everywhere. I presume this means that SQLMesh processed data and ran audits (data tests), but the logic is fuzzy. Lastly, I see a virtual update progress bar, but unlike the earlier step, there is no count -so I'm left uncertain about so I’m what it does at that final stage. There are three stages in total, but they do not provide me solid footing to determine what I should do next.

At this point, I might think to myself, maybe I didn’t read the docs well enough. But if it would be my first experience with SQLMesh, it could already feel like homework simply to grasp its basic operations! That’s not ideal, and we don’t want that impression to be the default for newcomers - or any other users. Our design itself should serve as a great teacher before anyone even reaches for the docs.

Here’s what our CLI looks after our revamp:

This picked up my code changes. The plan preview looks similar to the original. However, when I apply the plan, I get an entirely different experience.

The first thing I notice - it’s updating a physical layer, not creating physical tables. Now, I’m no longer second guessing if a table is being created for my view model. Next, I see that the term evaluation has changed to execution, which better reflects what's happening. This new language is also coupled with how each model was executed based on its specific kind, including models with audits!

The CLI shows that 6/6 models successfully moved through each stage: physical layer → execution → virtual layer. And if I wanted extra peace of mind, I can make the CLI output more verbose by passing the -vv flag with my command sqlmesh plan dev -vv.

This tells me SQLMesh is behaving as expected. It is also helpful if I want a way to easily copy and paste the fully qualified model name to copy/paste into my query console for ad hoc exploration.

As for why we did not make this level of verbosity the default - we've observed that most users are primarily interested in execution details. Updates to physical and virtual layers are important operational plumbing, but they introduce noise and don’t convey as much value. This is particularly when scrolling through dozens of model operations. We want our design to surface the most relevant information, while still offering the option to go deeper when needed.

Debugging as First Class

Verbosity defaults are a delicate art. When everything is working as expected, what’s just enough to make you nod your head in solidarity and move onto the next change? And when something fails, how do we deliver you an experience that feels just as intuitive as building your first pipeline? I won’t belabor you with the old vs. new output - let’s just get straight into the latter.

Here’s what happens when audits warn and fail:

I notice that audit failures now appear inline, marked clearly with a ❌ symbol to signal a non-blocking issue without stopping execution.

In contrast, blocking failures are more verbose. They include a clear error message and point to the specific SQL that caused the issue. I can then infer that I mistakenly audited a column that doesn’t exist, something that can easily happen when I’m copying over audits from other models as a starting point.

This new CLI does not simply indicate when something is wrong, it also illuminates what actions I should take. I'm obtaining as much information as I need to effectively move forward.

What’s Next?

Now we watch how you use this in the wild. We want you to have your 34th invocation and think, “Wow, it feels really good each time I run this.” We’re hoping to see questions you ask in our Slack community evolve from, “What’s going on here?” to “How do I solve this next meaningful problem?”.

From there, based on your feedback, we’ll continue to iterate. What matters is that you believe in our actions, not just our words, to build you a better DevEx. It matters a lot, and we don’t take you for granted.

Other modifications that are up for consideration:

  • Show specific row count metadata where relevant: [insert/update 387 rows]
    • Helpful to know why it’s running long or backfilled more/less than expected
  • Show specific fingerprint ids in the physical layer per model sqlmesh-example.demo.full_model__91158582
    • Inspect the underlying data directly in a swift copy-and-paste motion into your query console/logs
  • Enhance the --forward-only CLI output to better illustrate complex operations

If you want to learn or talk more, or bring us suggestions, join our community Slack in the #sqlmesh channel: https://tobikodata.com/slack.