[qob][batch] do not list all jobs on failure (plus: types!)

[danking]

CHANGELOG: In Query-on-Batch, the client-side Python code will not try to list every job when a QoB batch fails. This could take hours for long-running pipelines or pipelines with many partitions.

I also added API types to the list jobs end point because I have to go hunting for this every time anyway. Seems better to have this information at our digital fingertips.

[daniel-goldstein]

Just a drive by comment: You can use the query string q to search for failed jobs instead of filtering client-side

[danking]

done

[jigold]

These both need to be versioned. Otherwise, we're going to run into issues with backwards compatibility.

they have been so versioned.

[jigold]

I don't understand how the backwards compatibility is going to work in this case:

1. I have a V1 client, the server is sending me a V2 response with extra fields (or removed fields). What happens here?

[danking]

A V1 client cannot hit the V2 endpoint because it doesn't know it exists.

[danking]

To be clear, all this does is encode in the types what we already guarantee: if version X returns {'foo': int}, then all future versions of the server must at least return a dict containing the 'foo' key with an integer. If those servers start returning {'bar': int}, that would break all the old clients (and violate the types that we wrote in those clients).

[jigold]

Just so I understand correctly, we can add fields to the V1Alpha type later on, but not remove them or change the type?

[jigold]

You're missing the V1Alpha here.

[danking]

I merged main and was forced to fix the pyright errors, which was great! I found two bugs with these new types:

1. It's possible for a spec file to be missing which means the spec is None which would fail our get job front end code when it tries to fix up the resources.
2. In the aioclient, we assumed the attributes was present and a dict but that key can be completely missing from the dict.

I also noticed cost_str has had the wrong type since always.

[danking]

we can add fields to the V1Alpha type later on, but not remove them or change the type?

We should not remove or change fields, the types of fields, or the meaning of fields of JSON dicts returned by our API endpoints because that would break backwards compatibility.

The types on the other hand can change willy nilly, mypy and pyright have no way of knowing that we changed the definition of, for example, GetJobResponseV1Alpha. They will however, go report all the places in our codebase where we expected certain fields to exist if we remove those fields from GetJobResponseV1Alpha.

x

[danking]

@jigold bump