Consider never deleting the runs

[Ark-kun]

Some features (e.g. the ability to see where a piece of data came from) rely on the retention of runs. It's also extremely useful to be able to see the logs of those runs.

I propose using some mechanism to hide the runs (e.g. setting the hidden: true label) instead of deleting the runs altogether.

We should carefully gather the feedback about the users' need for run deletion.

[yebrahim]

The current plan is not to completely delete runs, but to archive them.

[Ark-kun]

I'm not sure archiving is good enough. For data provenance and caching it does not matter whether some run failed or not. Each data piece should have a source execution. It's convenient when all executions are in the same table.

[yebrahim]

Not sure I understand.
Archiving will keep runs around, it's just a convenience feature so that when you're listing resources, the API doesn't return everything, and you can keep the interface clean from old, ignored, or failed runs for example. You can still list archived resources and reference them.

[Ark-kun]

when you're listing resources, the API doesn't return everything, and you can keep the interface clean from old, ignored, or failed runs for example.

This looks similar to what I meant by hiding. What do you think about implementing the feature using labels? We can have a default filter like hidden=false owner=<user>

[yebrahim]

I'm not opposed to using labels for this in principle, but I wouldn't like blocking on its implementation. We can represent this in the UI however we like however.

[Ark-kun]

I forgot we do not have labels yet. I wonder whether adding labels to the DB schema would be as easy as adding the archived/hidden flag...

[hongye-sun]

I think we should let user to decide whether to keep the data or not. We might keep the data with a retention period, but I don't think we should keep the data beyond it. Even in managed service, we have to clean up user data if they decide to delete them for privacy reason.

[vicaire]

Resolving. As @hongye-sun pointed out, users may need to delete runs permanently for one reason or another.