[batch] Add environment variable for batch id in worker

[jigold]

This is needed for Siwei's recursive pipeline to work.

[daniel-goldstein]

I thought I'd proposed this env variable but then we ended up going with this?

[daniel-goldstein]

See comment above

[jigold]

I thought the config file was better for QoB where versioning and backwards compatibility is important and there could be substantially more configuration in the future. I don't think we mount that config file in the worker regardless for DockerJobs, but I could be wrong.

see comment

[daniel-goldstein]

was better for QoB where versioning and backwards compatibility is important

I think we have to fulfill the same backwards compatibility with env variables, though I agree that they are more ergonomic and favorable in this case to a config file. The config file was necessary for JVMJobs because the JVM containers are long-lived, so we cannot update their environment when a new job comes in. Looks like we do write the config in DockerJobs but don't mount it, so let's add this env variable but do the following:

• Remove the use of write_batch_config in DockerJob and move write_batch_config from Job into JVMJob
• Document where write_batch_config is used in JVMJob why it's used instead of an env variable. I think I still don't understand the point of further configuration, since that's still possible with environment variables, we just can't change environment variables in JVMContainers like we can with the filesystem

done

[daniel-goldstein]

awesome, thanks