Error when loading checkpoint #3709

greschd · 2020-01-07T16:28:29Z

After moving from 1.0.0 to the latest develop, I'm getting errors where workchains except because their checkpoint could not be loaded (see full traceback below).

These are new processes, i.e. they did not exist before the code update.

At least once, the issue occurred right after the daemon was restarted. This might just be the trigger for actually having to dump / load the checkpoint, though.

I have tried isolating a minimal failing example, but without luck so far. Will update once I've got an example.

@sphuber, could this be related to the switch to safe YAML loading?

2020-01-07 15:35:38 [44621 |  ERROR]: Traceback (most recent call last):
  File "/data/aiida/source/aiida-core/aiida/engine/persistence.py", line 124, in load_checkpoint
    bundle = serialize.deserialize(checkpoint)
  File "/data/aiida/source/aiida-core/aiida/orm/utils/serialize.py", line 230, in deserialize
    return yaml.load(serialized, Loader=AiiDALoader)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 43, in get_single_data
    return self.construct_document(node)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 52, in construct_document
    for dummy in generator:
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 596, in construct_python_object
    state = self.construct_mapping(node, deep=deep)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 210, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 135, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 92, in construct_object 
    data = constructor(self, node)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 420, in construct_undefined
    node.start_mark)
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply: aiida.engine.processes.workchains.awaitable.AwaitableAction'
  in "<unicode string>", line 87, column 11:
      action: !!python/object/apply:aiida.engi ...
              ^

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/aiida/source/aiida-core/aiida/manage/external/rmq.py", line 187, in _continue
    result = yield super()._continue(communicator, pid, nowait, tag)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
     raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/tornado/gen.py", line 307, in wrapper
    yielded = next(result)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/plumpy/process_comms.py", line 546, in _continue 
    saved_state = self._persister.load_checkpoint(pid, tag)
  File "/data/aiida/source/aiida-core/aiida/engine/persistence.py", line 127, in load_checkpoint
    'Failed to load the checkpoint for process<{}>: {}'.format(pid, traceback.format_exc())
plumpy.exceptions.PersistenceError: Failed to load the checkpoint for process<55902>: Traceback (most recent call last):
  File "/data/aiida/source/aiida-core/aiida/engine/persistence.py", line 124, in load_checkpoint
    bundle = serialize.deserialize(checkpoint)
  File "/data/aiida/source/aiida-core/aiida/orm/utils/serialize.py", line 230, in deserialize
    return yaml.load(serialized, Loader=AiiDALoader)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 43, in get_single_data
    return self.construct_document(node)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 52, in construct_document
    for dummy in generator:
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 596, in construct_python_object
    state = self.construct_mapping(node, deep=deep)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 210, in construct_mapping
    return super().construct_mapping(node, deep=deep)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 135, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 92, in construct_object    data = constructor(self, node)
  File "/home/a-dogres/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py", line 420, in construct_undefined
    node.start_mark)
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:aiida.engine.processes.workchains.awaitable.AwaitableAction'
  in "<unicode string>", line 87, column 11:
      action: !!python/object/apply:aiida.engi ...

The text was updated successfully, but these errors were encountered:

sphuber · 2020-01-08T09:06:46Z

This might have to do with the recent update to another version of pyyaml. Can you do a pip freeze? It might have been that you ran this process with older version, then updated aiida-core and hence pyyaml and then restarted the process? In that case the old checkpoint dump is no longer supported by the new version of pyyaml.

greschd · 2020-01-08T09:35:33Z

Running pip freeze | grep -i yaml gives me

aspy.yaml==1.3.0
PyYAML==5.2
ruamel.yaml==0.16.5
ruamel.yaml.clib==0.1.2

which I think is correct for the latest develop, right?

The processes were created after the update.. and I think I restarted everything that holds python state (daemon, jupyter kernels), but can't be 100% sure now.

I think I'll just leave this issue open for a while in case it happens again or someone else has the same issue.

greschd · 2020-01-13T09:02:06Z

Happened again, now I'm fairly certain there weren't any more kernels / daemons alive with the old code version.

sphuber · 2020-01-13T09:08:24Z

can you dump the response of node.checkpoint here

greschd · 2020-01-13T09:14:56Z

Not sure if I should post the whole thing, but this is the offending part:

_awaitables:
- !!python/object:aiida.engine.processes.workchains.awaitable.Awaitable
  action: !!python/object/apply:aiida.engine.processes.workchains.awaitable.AwaitableAction
  - assign
  key: fp_run
  outputs: false
  pk: 58729
  target: !!python/object/apply:aiida.engine.processes.workchains.awaitable.AwaitableTarget
  - process

greschd · 2020-01-13T09:15:27Z

This is one of my workchains, but it had also happened on a pw relax workchain before.

sphuber · 2020-01-13T09:21:16Z

Ok, so it seems that is simply has trouble deserializing the Awaitable instance, which is a sub class of plumpy.AttributesDict. If we follow the mro, we see that it is based completely on custom classes, designed to recreate a simple namespace that was not available in python 2. I am just surprised that this has never cropped up before though. Although of course this is only triggered if people installed develop and the new version of pyyaml. I think we should be able to fix this by adding a constructor to the yaml serializer for the Awaitable class. A potentially better solution is to just use a standard python data structure since really all it is, is just an attribute dictionary.

greschd · 2020-01-13T09:34:16Z

As far as standard python data structures go: I just tried dumping / loading a types.SimpleNamespace instance, and get exactly the same error:

In [12]: class Test(SimpleNamespace):
    ...:     pass
    ...:

In [13]: yaml.dump(Test(x=1))
Out[13]: '!!python/object/apply:__main__.Test\nstate:\n  x: 1\n'

In [14]: yaml.load(_)
/home/a-dogres/.virtualenvs/aiida/bin/verdi:1: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  #!/home/a-dogres/.virtualenvs/aiida/bin/python3
---------------------------------------------------------------------------
ConstructorError                          Traceback (most recent call last)
<ipython-input-14-cef43f020149> in <module>
----> 1 yaml.load(_)

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/__init__.py in load(stream, Loader)
    112     loader = Loader(stream)
    113     try:
--> 114         return loader.get_single_data()
    115     finally:
    116         loader.dispose()

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py in get_single_data(self)
     41         node = self.get_single_node()
     42         if node is not None:
---> 43             return self.construct_document(node)
     44         return None
     45

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py in construct_document(self, node)
     45
     46     def construct_document(self, node):
---> 47         data = self.construct_object(node)
     48         while self.state_generators:
     49             state_generators = self.state_generators

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py in construct_object(self, node, deep)
     90                     constructor = self.__class__.construct_mapping
     91         if tag_suffix is None:
---> 92             data = constructor(self, node)
     93         else:
     94             data = constructor(self, tag_suffix, node)

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py in construct_undefined(self, node)
    418         raise ConstructorError(None, None,
    419                 "could not determine a constructor for the tag %r" % node.tag,
--> 420                 node.start_mark)
    421
    422 SafeConstructor.add_constructor(

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:__main__.Test'
  in "<unicode string>", line 1, column 1:
    !!python/object/apply:__main__.Test
    ^

sphuber · 2020-01-13T09:34:36Z

Hmmm, I created a unit test to try and hit this problem but for me it is working. Can you check out this branch and try to run pytest aiida/backends/tests/engine/test_persistence.py

sphuber · 2020-01-13T09:38:02Z

As far as standard python data structures go: I just tried dumping / loading a types.SimpleNamespace instance, and get exactly the same error:

Might that not just be because you are doing it in a shell or top-level script, causing the module path to be __main__?

greschd · 2020-01-13T09:38:44Z

Might that not just be because you are doing it in a shell or top-level script, causing the module path to be __main__?

Ah, true.

greschd · 2020-01-13T09:39:32Z

Yup, test fails for me..

greschd · 2020-01-13T09:42:01Z

Might that not just be because you are doing it in a shell or top-level script, causing the module path to be __main__?

Ah, true.

Wait, no... I get the same behavior when using a bare SimpleNamespace:

In [1]: from types import SimpleNamespace

In [2]: s = SimpleNamespace(x=1)

In [3]: import yaml

In [4]: yaml.dump(s)
Out[4]: '!!python/object/apply:types.SimpleNamespace\nstate:\n  x: 1\n'

In [5]: yaml.load(_)
/home/a-dogres/.virtualenvs/aiida/bin/verdi:1: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  #!/home/a-dogres/.virtualenvs/aiida/bin/python3
---------------------------------------------------------------------------
ConstructorError                          Traceback (most recent call last)
<ipython-input-5-cef43f020149> in <module>
----> 1 yaml.load(_)

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/__init__.py in load(stream, Loader)
    112     loader = Loader(stream)
    113     try:
--> 114         return loader.get_single_data()
    115     finally:
    116         loader.dispose()

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py in get_single_data(self)
     41         node = self.get_single_node()
     42         if node is not None:
---> 43             return self.construct_document(node)
     44         return None
     45

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py in construct_document(self, node)
     45
     46     def construct_document(self, node):
---> 47         data = self.construct_object(node)
     48         while self.state_generators:
     49             state_generators = self.state_generators

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py in construct_object(self, node, deep)
     90                     constructor = self.__class__.construct_mapping
     91         if tag_suffix is None:
---> 92             data = constructor(self, node)
     93         else:
     94             data = constructor(self, tag_suffix, node)

~/.virtualenvs/aiida/lib/python3.6/site-packages/yaml/constructor.py in construct_undefined(self, node)
    418         raise ConstructorError(None, None,
    419                 "could not determine a constructor for the tag %r" % node.tag,
--> 420                 node.start_mark)
    421
    422 SafeConstructor.add_constructor(

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:types.SimpleNamespace'
  in "<unicode string>", line 1, column 1:
    !!python/object/apply:types.Simp ...

greschd · 2020-01-13T09:42:41Z

Python version is 3.6.8, in case that might matter.

sphuber · 2020-01-13T09:43:05Z

Can you try to do pip install pyyaml==5.1.2 and rerun the test? Because I noticed you have 5.2 and I added pyyaml~=5.1.2 explicitly for plumpy because 5.2 breaks. I think that version became even stricter. I don't remember exactly what broke, but I can test it again.

greschd · 2020-01-13T09:44:48Z

I guess it's this? yaml/pyyaml#364

greschd · 2020-01-13T09:46:59Z

Yeah, that fixes it. Not sure how I ended up with 5.2 - I just installed with pip install -e . after the develop update.

sphuber · 2020-01-13T09:53:44Z

That is probably because currently in aiida-core the spec is pyyaml~=5.1 and the more strict requirement on plumpy is maybe overruled/ignored? Not sure what to do here, really pinning version of pyyaml to ~=5.1.2 or to upgrade and use the UnsafeLoader. Neither seem desirable, but maybe the UnsafeLoader is exactly what they intend to be used in our use case. I don't think it is feasible to write yaml loaders for every possible class we can stick in the process checkpoint.

greschd · 2020-01-13T09:58:10Z

Hmm, when looking at the files on PyPI, plumpy==0.14.4 just has 'PyYAML>=3.13' as dependency.

greschd · 2020-01-13T09:59:47Z

But yeah, I don't think for loading checkpoints it's really needed to do safe loading. Maybe we can strip checkpoints when importing (if we don't already do that) to completely avoid the "sharing malicious code" vector.

sphuber · 2020-01-13T10:11:42Z

Maybe we can strip checkpoints when importing (if we don't already do that) to completely avoid the "sharing malicious code" vector.

We don't yet, but I think this would be indeed an option. In the export file it is just shared as a key in the JSON serialized object. The yaml serialized checkpoint is just a string at that point. When importing the nodes, we can just simply omit all checkpoint attributes. So I think the best approach is to update the requirement to pyyaml~=5.2 for plumpy, use the UnsafeLoader for checkpoints and then update the requirements in aiida-core as well. I can keep the test I have now, to check that things are working.

greschd · 2020-03-30T12:46:06Z

I've just come across this issue again - @sphuber I have a few questions on how to fix this:

Is the AiiDALoader used only for checkpoints, or also elsewhere? If it's only for checkpoints, I think switching the base to UnsafeLoader would be fine, otherwise we might need a second (AiiDAUnsafeLoader) loader class.
I'm not exactly sure where the checkpoint importing happens: Is https://github.com/aiidateam/aiida-core/blob/develop/aiida/tools/importexport/dbimport/backends/sqla/__init__.py#L416 and https://github.com/aiidateam/aiida-core/blob/develop/aiida/tools/importexport/dbimport/backends/django/__init__.py#L295 roughly the right place?

sphuber · 2020-03-30T13:05:41Z

I think it is only used in the engine: specifically it is used by the Communicator to pass inputs and outputs over RabbitMQ and by the persister to dump the state of a Process instance to be stored as an attribute on the node. This is used by the daemon workers to reconstruct the process instance after having received the task to continue it. I think thus that it will be fine if we were to use the unsafe loader in aiida.orm.utils.serialize and mention this clearly in the docstring. The only thing that then remains to be done is to delete the attributes.checkpoint on export and add an additional check on import just to be safe. In any case, normally the attributes.checkpoint is deleted automatically when the process terminates nominally.

greschd · 2020-03-30T13:16:26Z

Ok, thanks!

an additional check on import just to be safe.

If we're concerned about malicious export files, the check on import is definitely necessary (and more important than the export change), because the exporter could always change the code on his side.

sphuber · 2020-03-30T13:18:57Z

If we're concerned about malicious export files, the check on import is definitely necessary (and more important than the export change), because the exporter could always change the code on his side.

very good point 👍

greschd · 2020-09-04T17:04:46Z

Yeah, just realized I still have that as a hotfix in my installation (because I don't need to worry about the malicious checkpoints issue).

@ltalirz would be great if you could make a PR, I think the two things that need doing are:

change AiidaLoader base class to UnsafeLoader
strip checkpoints on import

ltalirz · 2020-09-07T12:57:01Z

Hi @greschd , I haven't followed the thread here and I haven't touched that part of the codebase before - my suggestion was just to upgrade the pyyaml version to 5.2.

I'm happy to make the changes but would need some help - so, should I

update pyyaml minimal version to 5.2 both in plumpy and aiida-core (or only in one place)

replace yaml.Fullloader by yaml.UnsafeLoader . The following comment will need to be updated as well

aiida-core/aiida/orm/utils/serialize.py

Lines 182 to 185 in 999ae3a

    
               .. note:: we subclass the `FullLoader` which is the one that since `pyyaml>=5.1` is the loader that prevents 
        
                   arbitrary code execution. Even though this is in principle only used internally, one could imagine someone 
        
                   sharing a database with a maliciously crafted process instance dump, which when reloaded could execute arbitrary 
        
                   code. This load prevents this: https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation

what do I do to strip checkpoints on import?

greschd · 2020-09-07T13:58:52Z

Simply updating pyyaml won't work - the thread here is about issues I encountered when accidentally installing that version 😄 Since you're reporting issues with 5.1.2, I guess we should now fix the problems we had with 5.2.

* replace yaml.Fullloader by `yaml.UnsafeLoader`  . The following comment will need to be updated as well

Yes. If we remove checkpoints on import the explanation about malicious process instances isn't needed. I guess it'd need double-checking that this is the only place where AiidaLoader is used, and we should put a warning into its docstring.

what do I do to strip checkpoints on import?

Haven't touched this either.. maybe @sphuber give some pointers? EDIT: Well, these were my guesses as to where import stripping should happen - no guarantees that it actually is correct 😄

I'm not exactly sure where the checkpoint importing happens: Is https://github.com/aiidateam/aiida-core/blob/develop/aiida/tools/importexport/dbimport/backends/sqla/__init__.py#L416 and https://github.com/aiidateam/aiida-core/blob/develop/aiida/tools/importexport/dbimport/backends/django/__init__.py#L295 roughly the right place?

update pyyaml minimal version to 5.2 both in plumpy and aiida-core (or only in one place)

The above should fix the 5.2 incompatibility in aiida-core (at least, the one encountered here). But @sphuber also commented:

I added pyyaml~=5.1.2 explicitly for plumpy because 5.2 breaks. I think that version became even stricter. I don't remember exactly what broke, but I can test it again.

So I guess we should also go chasing after that.

sphuber · 2020-09-07T14:31:20Z

A checkpoint is just an attribute stored on the node, with the key checkpoint. In the export we should simply remove this attribute from all process nodes (note not all nodes in general, because dat nodes may happen to have an attribute with the same name but that is fine) and in the import we have to check that we are not importing any process nodes with them. That's all

mikibonacci · 2020-10-14T15:23:22Z

Hi there,
I found the error too. The fact was that I was trying to load stuff like numpy arrays, not allowed: the solution was to convert all the ctx.data into dictionaries/lists. Anyway, using the "unsafe load" seems to be the best solution.
In the meantime, I think that a pytest(for plugins) should check if the stuff that a workchain stores in the context are allowed from pyyaml or not.

greschd · 2020-11-16T13:13:58Z

@mikibonacci I've used the patch below as a workaround.

⚠WARNING:⚠ The code is unsafe in the sense that when importing an archive from an untrusted source, it can lead to arbitrary code execution. The reason is that the archive can contain "checkpoints" crafted to execute any code.
If you do not import archives from untrusted sources however, the patch is fine to apply:

diff --git a/aiida/orm/utils/serialize.py b/aiida/orm/utils/serialize.py
index 23aa68348..70fc09070 100644
--- a/aiida/orm/utils/serialize.py
+++ b/aiida/orm/utils/serialize.py
@@ -176,7 +176,7 @@ class AiiDADumper(yaml.Dumper):
         return super().represent_data(data)


-class AiiDALoader(yaml.FullLoader):
+class AiiDALoader(yaml.UnsafeLoader):
     """AiiDA specific yaml loader

     .. note:: we subclass the `FullLoader` which is the one that since `pyyaml>=5.1` is the loader that prevents

@ltalirz is there any progress on this?

ltalirz · 2020-11-17T16:09:42Z

Sorry @greschd I didn't have time to look into this - please feel free to if you manage.
I just checked plumpy and the dependency there is now pyyaml~=5.1, i.e. upgrading the version in aiida-core should work.

ltalirz · 2020-11-17T16:11:34Z

I just checked plumpy and the dependency there is now pyyaml~=5.1, i.e. upgrading the version in aiida-core should work.

Hm... that is incorrect, sorry https://github.com/aiidateam/plumpy/blob/develop/setup.py#L33

I think I saw it somewhere, but obviously not in the setup.py (also conda-forge feedstock is (correctly) locked to ~=5.1.2.
I probably confused something.

Edit: Ok, I saw it in kiwipy

greschd · 2020-11-17T18:43:57Z

I don't think the resolution of this is blocked by being able to use the new pyyaml - and the problem definitely occurs also (maybe less often, but still) on the old one.

greschd · 2020-11-18T08:31:18Z

please feel free to if you manage.

Yeah, unfortunately this isn't super high on my to-do list; but I think it's actually a fairly important issue because end users sometimes bump up against that.

csadorf · 2021-02-09T10:39:40Z

@greschd @ltalirz What is the status of this? The pyyaml version in plumpy is still constrained to ~5.1.2, but could we maybe loosen the constraint for aiida-core?

greschd · 2021-02-09T11:31:29Z

Not sure about the version compatibilities, but note that this issue is not mainly about the version (i.e., loosening the constraint wouldn't resolve this issue).

The main tasks are:

Strip checkpoints when importing from an archive
Replace yaml.FullLoader with yaml.UnsafeLoader as a base class to AiiDALoader

csadorf · 2021-05-05T08:33:38Z

Just for the record, this issue has increased in priority, because versions prior to 5.3.1 are currently vulnerable to arbitrary code execution when processing untrusted input values, see: GHSA-6757-jp84-gxfx

greschd · 2021-05-05T13:18:44Z

#4730 can in principle be merged.

To allow e.g. numpy arrays to be serialized to a process checkpoint, the `AiiDALoader` is based on `yaml.UnsafeLoader` instead of `yaml.FullLoader`. Since this could pose a security risk when sharing databases with maliciously crafted checkpoints, the checkpoints are removed upon importing an archive. Fixes aiidateam#3709.

greschd added type/bug topic/processes labels Jan 7, 2020

sphuber self-assigned this Jan 13, 2020

sphuber added the topic/persistence label Jan 13, 2020

greschd self-assigned this Mar 30, 2020

greschd mentioned this issue Nov 16, 2020

'aiida_optimize.engines._result_mapping' is not imported exception greschd/aiida-optimize#29

Closed

greschd assigned ltalirz Nov 16, 2020

greschd added the priority/important label Nov 18, 2020

unkcpz mentioned this issue Dec 17, 2020

yaml failed in constructing numpy object from checkpoint #4639

Closed

greschd mentioned this issue Feb 9, 2021

👌 IMPROVE: Allow numpy arrays to be serialized on process checkpoints #4730

Merged

chrisjsewell closed this as completed in #4730 Jun 16, 2021

Error when loading checkpoint #3709

Error when loading checkpoint #3709

Comments

greschd commented Jan 7, 2020 • edited Loading

sphuber commented Jan 8, 2020

greschd commented Jan 8, 2020

greschd commented Jan 13, 2020

sphuber commented Jan 13, 2020

greschd commented Jan 13, 2020

greschd commented Jan 13, 2020

sphuber commented Jan 13, 2020

greschd commented Jan 13, 2020

sphuber commented Jan 13, 2020

sphuber commented Jan 13, 2020

greschd commented Jan 13, 2020

greschd commented Jan 13, 2020

greschd commented Jan 13, 2020

greschd commented Jan 13, 2020

sphuber commented Jan 13, 2020 • edited Loading

greschd commented Jan 13, 2020 • edited Loading

greschd commented Jan 13, 2020

sphuber commented Jan 13, 2020

greschd commented Jan 13, 2020

greschd commented Jan 13, 2020

sphuber commented Jan 13, 2020

greschd commented Mar 30, 2020

sphuber commented Mar 30, 2020

greschd commented Mar 30, 2020

sphuber commented Mar 30, 2020

greschd commented Sep 4, 2020

ltalirz commented Sep 7, 2020

greschd commented Sep 7, 2020 • edited Loading

sphuber commented Sep 7, 2020

mikibonacci commented Oct 14, 2020

greschd commented Nov 16, 2020

ltalirz commented Nov 17, 2020 • edited Loading

ltalirz commented Nov 17, 2020 • edited Loading

greschd commented Nov 17, 2020

greschd commented Nov 18, 2020

csadorf commented Feb 9, 2021

greschd commented Feb 9, 2021

csadorf commented May 5, 2021

greschd commented May 5, 2021

greschd commented Jan 7, 2020 •

edited

Loading

sphuber commented Jan 13, 2020 •

edited

Loading

greschd commented Jan 13, 2020 •

edited

Loading

greschd commented Sep 7, 2020 •

edited

Loading

ltalirz commented Nov 17, 2020 •

edited

Loading

ltalirz commented Nov 17, 2020 •

edited

Loading