FList can't be pickled

[ismael-mendoza]

As I'm trying to replace the survey functionality of BTK with galcheat, I realized that I can't pickle the galcheat.survey.Survey class due to it containing a dynamically generated dataclass FList. See here and here.

Would it be possible to replace it with just a dictionary? Or why is the FList necessary? Thanks you! The reason I need to pickle it is to run multiprocessing which internally pickles objects.

One workaround that was suggested was to do FList.__module__ = __name__ (where it is created). But it seems a bit hacky and perhaps easier to replace with a dictionary if possible

@aboucaud

_pickle.PicklingError: Can't pickle <class 'types.LSSTFilterList'>: attribute lookup LSSTFilterList on types failed

[aboucaud]

The reason I created these dynamical lists was to enable the possibility of writing

LSST.filters.u.effective_wavelength

and having autocompletion in ipython or any IDE, instead of writing

LSST.get_filter('u').effective_wavelength

where you actually need to check first which filters are available.

[aboucaud]

It cannot be solved through the trick of writing FList.__module__ = __name__ because FList is just a placeholder for the dataclass created specifically for each survey given its filters : LSSTFilterList for LSST, DESFilterList for DES, etc.

The only option seems to abandon that functionality for a dictionary, but sadly that was one of the coolest features I had created for galcheat 😢

[thuiop]

I spoke about a way to potentially solve this at the meeting today (by replacing multiprocessing by pathos.multiprocessing) ; however I tried it and it does not seem to work :(

[ismael-mendoza]

Thanks @aboucaud and @thuiop for looking into this. And thanks @aboucaud for explaining your reasoning on doing this. It is indeed unfortunate that we need to abandon the FList due to multiprocessing solely and that other workarounds don't seem to work.

I'll think more about this today and let you know if I can come up with a workaround in BTK.

[aboucaud]

Closed by #92