Code

Hashing, caching and fast-forwarding#652

Code

Merged

giovannipizzi merged 122 commits intoaiidateam:developaiidateam/aiida-core:developfrom greschd:node-hashinggreschd/aiida_core:node-hashingCopy head branch name to clipboardFeb 9, 2018

+1,153-200

Commits

Commits on Jun 1, 2017

• aiidateam#119 added first small test to check for node hashing

  lekah

  committed
  ce539d8

  Copy full SHA for ce539d8

• aiidateam#119 Added small functions get_hash and get_same_node to Node

  lekah

  committed
  7589afd

  Copy full SHA for 7589afd

• aiidateam#119 Added functionality to store method (only for Django backend) that replaces the _dbnode member if a similar node already exists

  lekah

  committed
  13caff2

  Copy full SHA for 13caff2

Commits on Jun 2, 2017

• aiidateam#119 Try-except clause if hashing fails, and also check for None for hash, which obviously should not be checked in the DB

  lekah

  committed
  ca5f047

  Copy full SHA for ca5f047

• aiidateam#119 Fix in test that was failing because there is now an additional extra (hash)

  lekah

  committed
  87e546f

  Copy full SHA for 87e546f

Commits on Jun 15, 2017

• * Failing test for two small but unequal floats

  Show description for 39d378e
  greschd

  committed
  39d378e

  Copy full SHA for 39d378e

Commits on Jun 16, 2017

• Merge pull request aiidateam#591 from greschd/node-hashing

  Show description for f7e6fe0
  lekah

  authored
  f7e6fe0

  Copy full SHA for f7e6fe0

Commits on Jul 20, 2017

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  28af8ed

  Copy full SHA for 28af8ed

• Merge branch 'node-hashing' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  f12c3cb

  Copy full SHA for f12c3cb

• Give name to '_' in loop

  greschd

  committed
  219d53b

  Copy full SHA for 219d53b

• Update hashing algorithm

  greschd

  committed
  76acbb6

  Copy full SHA for 76acbb6

• Fix get_hash for special case of ArrayData

  greschd

  committed
  2de9bc8

  Copy full SHA for 2de9bc8

• Fix ArrayData issue by hashing folder content for 'pathlib.Path'

  greschd

  committed
  a4ae949

  Copy full SHA for a4ae949

• Explicitly raise error when non-directory path is hashed

  greschd

  committed
  516c44e

  Copy full SHA for 516c44e

• Add pathlib2 requirement

  greschd

  committed
  dcb4a38

  Copy full SHA for dcb4a38

• Print modulename to check failing Travis test

  greschd

  committed
  bbe4c33

  Copy full SHA for bbe4c33

• Try moving ArrayData import inside the test

  greschd

  committed
  fb182cc

  Copy full SHA for fb182cc

• Move numpy import to test

  greschd

  committed
  dc93962

  Copy full SHA for dc93962

• Add comma after checksumdir requirement

  greschd

  committed
  5fef7f2

  Copy full SHA for 5fef7f2

• Add find_same to store_all, add to SQLAlchemy

  greschd

  committed
  f04d3bb

  Copy full SHA for f04d3bb

• Simplify get_same_node

  greschd

  committed
  1bc7c90

  Copy full SHA for 1bc7c90

• Set hash extra on SQLAlchemy nodes

  greschd

  committed
  c5df6bf

  Copy full SHA for c5df6bf

• Change how extra is stored on SQLAlchemy node

  greschd

  committed
  e5c72a1

  Copy full SHA for e5c72a1

• Use Folder interface for hashing the repository folder

  greschd

  committed
  a896eb1

  Copy full SHA for a896eb1

• Add logic to ignore attributes

  greschd

  committed
  d13f8dd

  Copy full SHA for d13f8dd

• Add tests for FolderData with empty files and folders.

  Show description for 533e968
  greschd

  committed
  533e968

  Copy full SHA for 533e968

Commits on Jul 21, 2017

• Add functionality to set the default of caching True / False

  greschd

  committed
  55d1ad0

  Copy full SHA for 55d1ad0

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  29f14ba

  Copy full SHA for 29f14ba

• Add contextmanager to enable / disable caching

  greschd

  committed
  7a0d4ef

  Copy full SHA for 7a0d4ef

• Change caching default in store methods to False.

  Show description for 148770d
  greschd

  committed
  148770d

  Copy full SHA for 148770d

• Add module.__version__ to hash of a Node

  greschd

  committed
  a3bc4b5

  Copy full SHA for a3bc4b5

Commits on Aug 2, 2017

• Add _is_valid_cache, to exclude failed calculations

  greschd

  committed
  bfad737

  Copy full SHA for bfad737

Commits on Aug 7, 2017

• Add debug statements in subcalss run_until_complete

  greschd

  committed
  6f37ddd

  Copy full SHA for 6f37ddd

Commits on Aug 11, 2017

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  ffbdd73

  Copy full SHA for ffbdd73

Commits on Aug 14, 2017

• Implement fast-forwarding in _create_and_setup_db_record

  greschd

  committed
  06acb5c

  Copy full SHA for 06acb5c

• Add inputs hash for WorkCalculation Nodes

  greschd

  committed
  c4027ff

  Copy full SHA for c4027ff

• Add tests for WorkChain fast-forwarding

  greschd

  committed
  6693152

  Copy full SHA for 6693152

Commits on Aug 15, 2017

• Add more tests to WorkChain forwarding

  greschd

  committed
  8e56a54

  Copy full SHA for 8e56a54

• Add Travis hack to install plum from my fork

  greschd

  committed
  e7493c7

  Copy full SHA for e7493c7

• Add insert_data to fast-forwarding workchain test setUp method

  greschd

  committed
  ae485ac

  Copy full SHA for ae485ac

• Call tearDownClass and setUpClass in fast-forwarding tearDown

  greschd

  committed
  fb8367e

  Copy full SHA for fb8367e

• Add _is_valid_cache method to exclude invalid Nodes (e.g. failed Calculations)

  greschd

  committed
  33e6d0d

  Copy full SHA for 33e6d0d

• Remove unused _use_cache attribute

  greschd

  committed
  404a6f6

  Copy full SHA for 404a6f6

Commits on Aug 16, 2017

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  62b8880

  Copy full SHA for 62b8880

• Add 'ignore_errors' option to get_hash, raise errors instead of returning None if 'ignore_errors=False'

  greschd

  committed
  471dcfa

  Copy full SHA for 471dcfa

• Add fast-forwarding for JobCalculations

  greschd

  committed
  f076bfe

  Copy full SHA for f076bfe

• Implement _is_valid_cache at AbstractCalculation level, add abstractmethod for has_finished_ok

  greschd

  committed
  ba052a5

  Copy full SHA for ba052a5

Commits on Aug 17, 2017

• Add option to configure caching / fast-forwarding via config file

  greschd

  committed
  3b02796

  Copy full SHA for 3b02796

• Set Py2 requirements via environment marker

  greschd

  committed
  509b7e8

  Copy full SHA for 509b7e8

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  eb0289f

  Copy full SHA for eb0289f

• Add _get_objects_to_hash method, to de-duplicate get_hash code

  greschd

  committed
  b79fe10

  Copy full SHA for b79fe10

• Add 'expanduser' to getting the cache config

  greschd

  committed
  f989e92

  Copy full SHA for f989e92

Commits on Aug 20, 2017

• - Add **kwargs to make_hash

  Show description for d64add2
  greschd

  committed
  d64add2

  Copy full SHA for d64add2

• Add re-hash command

  greschd

  committed
  f1d7171

  Copy full SHA for f1d7171

• Add '.' for every 100 re-hashed nodes

  greschd

  committed
  7b460ab

  Copy full SHA for 7b460ab

Commits on Aug 21, 2017

• Get only INPUT link in get_inputs_dict for objects to hash

  greschd

  committed
  1df9128

  Copy full SHA for 1df9128

• Add 'CALL' to ignored inputs

  greschd

  committed
  58b2195

  Copy full SHA for 58b2195

Commits on Oct 19, 2017

• Add option to specify PKs (or --all) in verdi rehash.

  greschd

  committed
  7df259f

  Copy full SHA for 7df259f

Commits on Oct 20, 2017

• Simplify verdi rehash code

  greschd

  committed
  4168fa0

  Copy full SHA for 4168fa0

Commits on Oct 27, 2017

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  98e7aaf

  Copy full SHA for 98e7aaf

• Run pre-commit

  greschd

  committed
  537cd89

  Copy full SHA for 537cd89

Commits on Oct 30, 2017

• Update plumpy requirement to 0.7.11

  greschd

  committed
  d809be6

  Copy full SHA for d809be6

• Install plumpy before AiiDA on travis build

  greschd

  committed
  5a63309

  Copy full SHA for 5a63309

• Update RTD requirements

  greschd

  committed
  42cde39

  Copy full SHA for 42cde39

• Ignore '_updatable_attributes' in addition to '_hash_ignored_attributes'.

  greschd

  committed
  e96cd6b

  Copy full SHA for e96cd6b

Commits on Oct 31, 2017

• Enable use of the cache_config for use_cache

  greschd

  committed
  afbb3c5

  Copy full SHA for afbb3c5

• Add tests for caching configuration.

  greschd

  committed
  28e4d60

  Copy full SHA for 28e4d60

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  e8a7d2b

  Copy full SHA for e8a7d2b

• Merge branch 'develop' into node-hashing

  greschd

  authored
  e04b9de

  Copy full SHA for e04b9de

• Merge branch 'node-hashing' of github.com:greschd/aiida_core into node-hashing

  greschd

  committed
  39ac450

  Copy full SHA for 39ac450

• Move caching config tests to db tests

  greschd

  committed
  18cf330

  Copy full SHA for 18cf330

• Implement cache lookup in general/node.py instead of db implementations

  greschd

  committed
  050eb38

  Copy full SHA for 050eb38

Commits on Nov 1, 2017

• Use 'copy' when retrieving cached Node

  Show description for 74a358a
  greschd

  committed
  74a358a

  Copy full SHA for 74a358a

• Keep inputlink_cache when creating cached Node.

  greschd

  committed
  fd09763

  Copy full SHA for fd09763

Commits on Nov 6, 2017

• Add class-level '_cacheable' attribute

  greschd

  committed
  5646de9

  Copy full SHA for 5646de9

• Seal inline calculations

  greschd

  committed
  2b72bff

  Copy full SHA for 2b72bff

• Fix _store_from_cache

  greschd

  committed
  9c6f90a

  Copy full SHA for 9c6f90a

• Simplify _store_from_cache code, add tests for InlineCalculation caching

  greschd

  committed
  e4d101d

  Copy full SHA for e4d101d

• Merge configuration for 'use_cache' and 'fast_forward'.

  greschd

  committed
  d902d69

  Copy full SHA for d902d69

• Add enable_caching and disable_caching contextmanagers.

  greschd

  committed
  644d349

  Copy full SHA for 644d349

• Merge duplicate 'general' implementations of InlineCalculation.

  Show description for 389aa8d
  greschd

  committed
  389aa8d

  Copy full SHA for 389aa8d

• Remove fast-forwarding tests for WorkChain and workfunction

  greschd

  committed
  e4af12d

  Copy full SHA for e4af12d

• Set calculation state to PARSING before adding cached outputs, and match cached state after

  greschd

  committed
  01a9143

  Copy full SHA for 01a9143

• Add caching documentation

  greschd

  committed
  338f952

  Copy full SHA for 338f952

• Update singledispatch requirement to 3.4.0.3

  greschd

  committed
  62ba9d9

  Copy full SHA for 62ba9d9

Commits on Nov 12, 2017

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  7559721

  Copy full SHA for 7559721

Commits on Nov 13, 2017

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  baea092

  Copy full SHA for baea092

• Add cache tests to test_daemon.py

  greschd

  committed
  20691dd

  Copy full SHA for 20691dd

• Add assert for 'cached_from' key in daemon test

  greschd

  committed
  326b258

  Copy full SHA for 326b258

Commits on Dec 10, 2017

• Test that the hash stays the same for completed calculations.

  greschd

  committed
  2a6f475

  Copy full SHA for 2a6f475

• Add 'retrieve_temporary_list' to ignored attributes for calculations.

  greschd

  committed
  8d76026

  Copy full SHA for 8d76026

• Check for hash consistency only for cached calculations.

  greschd

  committed
  fb5f5dd

  Copy full SHA for fb5f5dd

Commits on Dec 12, 2017

• Add has_failed and has_finished methods to AbstractCalculation.

  greschd

  committed
  fe4a5c2

  Copy full SHA for fe4a5c2

• Remove install for git plumpy version in Travis.

  greschd

  committed
  b54d840

  Copy full SHA for b54d840

• Add computer uuid to _objects_to_be_hashed

  greschd

  committed
  dd3ca84

  Copy full SHA for dd3ca84

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  da52523

  Copy full SHA for da52523

• Remove 'self.is_sealed' in _is_valid_cache test -- is not fulfilled for all calculations.

  greschd

  committed
  7e3a771

  Copy full SHA for 7e3a771

• Add a more thorough discussion of links and caching

  greschd

  committed
  2fb2081

  Copy full SHA for 2fb2081

Commits on Dec 13, 2017

• Validate caching of 'cached' calculations when also checking for expected results

  greschd

  committed
  87994f2

  Copy full SHA for 87994f2

Commits on Dec 18, 2017

• Merge branch 'develop' into node-hashing

  giovannipizzi

  authored
  265d058

  Copy full SHA for 265d058

Commits on Jan 29, 2018

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  85c812f

  Copy full SHA for 85c812f

• Merge branch 'node-hashing' of github.com:greschd/aiida_core into node-hashing

  greschd

  committed
  3aa6aa0

  Copy full SHA for 3aa6aa0

• * add 'get_all_same_nodes' function, returns a list of all matching nodes

  Show description for 9717235
  greschd

  committed
  9717235

  Copy full SHA for 9717235

• Add '_iter_all_same_nodes' helper function, which returns an iterator

  Show description for be36c3a
  greschd

  committed
  be36c3a

  Copy full SHA for be36c3a

• Rename 'cached_from' and 'hash' extras by prepending '_aiida_'.

  greschd

  committed
  c413145

  Copy full SHA for c413145

• Define constant for '_aiida_hash' key

  greschd

  committed
  acde1c3

  Copy full SHA for acde1c3

• Fix filter in '_iter_same_node': Rename hash to _aiida_hash

  greschd

  committed
  4101b15

  Copy full SHA for 4101b15

• Fix tests which compared extras and didn't expect the '_aiida_hash' extra.

  greschd

  committed
  107e444

  Copy full SHA for 107e444

• Fix TestDbExtrasDjango to take into account '_aiida_hash'.

  greschd

  committed
  1f5bf07

  Copy full SHA for 1f5bf07

• Fix SQLA test_replacement_1 test to work with '_aiida_hash'

  greschd

  committed
  89c3626

  Copy full SHA for 89c3626

Commits on Jan 31, 2018

• Fix error message where 'fast-forwarding' is mentioned.

  greschd

  committed
  7b4d164

  Copy full SHA for 7b4d164

• Remove 'retrieve_temporary_list' from ignored hash attributes.

  greschd

  committed
  e515f14

  Copy full SHA for e515f14

• Add doc for what to do when caching is triggered in error, add clear_cache function

  greschd

  committed
  99bad1e

  Copy full SHA for 99bad1e

Commits on Feb 1, 2018

• Add max_memory_kb and priority to the attributes ignored in JobCalculations

  greschd

  committed
  fe1f5e8

  Copy full SHA for fe1f5e8

• Add description of '_hash_ignored_inputs'

  greschd

  committed
  b7c1e71

  Copy full SHA for b7c1e71

• Add code to show how to get the full class name

  greschd

  committed
  1b4839e

  Copy full SHA for 1b4839e

• Remove workchain caching test, add caching section to dev guide

  greschd

  committed
  01f9004

  Copy full SHA for 01f9004

• Merge branch 'develop' into node-hashing

  greschd

  authored
  eef22f0

  Copy full SHA for eef22f0

• Merge branch 'develop' of github.com:aiidateam/aiida_core into node-hashing

  greschd

  committed
  5a2a840

  Copy full SHA for 5a2a840

• Merge branch 'node-hashing' of github.com:greschd/aiida_core into node-hashing

  greschd

  committed
  1c0e6bc

  Copy full SHA for 1c0e6bc

Commits on Feb 8, 2018

• Add 'retrieve_temporary_list' to updatable attributes

  greschd

  committed
  8596f77

  Copy full SHA for 8596f77

• Merge branch 'develop' into node-hashing

  greschd

  authored
  419b20a

  Copy full SHA for 419b20a