[RFC] 🚀 Try Out Our New Selection Syntax! We Need Your Feedback 📝

[salazarm]

Hey everyone!

We’re super excited to introduce a brand-new Selection Syntax to help you visualize and manage your asset lineage/job op/run gantt graphs more easily. 🎉 But before we roll it out to everyone, we’d love for some brave beta testers to give it a spin and share your thoughts!

✨ What’s New?

• New traversal syntax:
  • Old syntax: +++"my_asset"+++ New syntax: 3+key:"my_asset"+3
  • Old syntax: *"my_asset"* New syntax: +key:"my_asset"+
• Boolean logic: Use boolean logic to create complex query expressions: tag:billing or tag:sales
• Combine lineage operations with filters: +tag:billing (everything upstream of assets tagged with billing) .
• Sinks/Roots functions: Find the sinks or roots of a set of assets: sinks(owner:"billing")

🔧 How to Enable It:

1. Go to your User settings in the UI.
2. Find the Experimental Features section.
3. Toggle on Enable new selection syntax for Asset/Op/Run graphs.

💬 We’d Love to Hear From You!

• What you like: What makes this new syntax awesome for you?
• What’s tricky: Any parts that are confusing or hard to use?
• Ideas: Features or improvements you’d like to see.

Happy querying! 🛠️✨

(This flag will be available in release 1.9.7)

Example finding the path between nodes:

Example finding the path between two nodes that go through another node:

Selection in the Run Gantt chart:

Selection in a Job Op graph:

[malow106]

Will try it for sure !

[Terroface]

These look like great additions. One thing I have found myself needing is a way to exclude assets. Is it possible to take all this great selection syntax and use it to filter out assets instead?

[salazarm]

We do plan on adding UI controls to make this easier in the future but first we need the syntax to express it!

[Terroface]

@salazarm that sounds great. I did have some trouble getting it working in some situations. I have an example.

I have several asset groups:

• groupa
• groupb
• groupc
• ...

I want to filter to all groups's assets except those of groupc

I tried the following selection criteria:

• not: groupc
• not groupc

Both, unfortunately filtered to groupc's assets instead of filtering them out.

Any idea what I'm doing wrong here?

[salazarm]

Hi Terro, can you share a screenshot of what you typed in? In general the syntax is not group:"group_name"

[salazarm]

@Terroface Also, this is in 1.9.7 which is coming out today in OSS and came out yesterday in Dagster+. Where are you trying this and on which version?

[Terroface]

@salazarm oh of course! We haven't yet upgraded to 1.9.7 so I'll have to wait before testing this. Thanks for your answers.

[stretcharm]

It looks good. I do find asset selection tricky, not that I have played with it much. In the dbt docs I like the fact that the lineage diagram filters gives me the same code needed in the dbt command line.

Is it also possible to select one or more partitions, if applicable? That would be useful if I want to check the lineage of a particular partition for failures or missing materialisations.

A nice feature would be a way to copy a code snippet to help create jobs based from the UI selection. Or even create a job directly from the UI for our less techy operational colleagues. Code first is a great principle but if an operational team might need to create a job or schedule and don't have the skills or dev support to update the code base they are stuck. Many ops teams come from older tools that have a UI for creating schedules.

[salazarm]

It looks good. I do find asset selection tricky, not that I have played with it much. In the dbt docs I like the fact that the lineage diagram filters gives me the same code needed in the dbt command line.

The CLI and AssetSelection.from_string api's will accept this syntax as well! So you would be able to copy paste it in.

Is it also possible to select one or more partitions, if applicable? That would be useful if I want to check the lineage of a particular partition for failures or missing materialisations.

Not yet, we do plan in the future to expand the syntax to cover more object types such asset checks and partitions though!

A nice feature would be a way to copy a code snippet to help create jobs based from the UI selection. Or even create a job directly from the UI for our less techy operational colleagues. Code first is a great principle but if an operational team might need to create a job or schedule and don't have the skills or dev support to update the code base they are stuck. Many ops teams come from older tools that have a UI for creating schedules.

Thats a good idea, while you can copy paste the selection syntax into the proper APIs if you know them, I think it would also make sense to take it a step further and provide full code samples for common workflows.