Code

Dataloaders with socket conection#803

Code

Merged

dfalbel merged 7 commits intomainmlverse/torch:mainfrom compressmlverse/torch:compressCopy head branch name to clipboardMar 31, 2022

+334-139

Commits

Commits on Mar 28, 2022

• Do not compress tensor by default when saving to a raw vector.

  dfalbel

  committed
  9a2a374

  Copy full SHA for 9a2a374

• Actually compress is already = FALSE when writing to a raw vector.

  dfalbel

  committed
  2fce17c

  Copy full SHA for 2fce17c

Commits on Mar 30, 2022

• Implement socket conections to transfer data from workers to the main process.

  dfalbel

  committed
  7dc52ef

  Copy full SHA for 7dc52ef

• slighly improve string handdling.

  dfalbel

  committed
  60e99fe

  Copy full SHA for 60e99fe

• add compress argument.

  dfalbel

  committed
  4deba61

  Copy full SHA for 4deba61

• We now avoid base64 encoding the serialized torch tensor before passing to R. This gives us ~10x speedup on serialization - which is very important for the dataloader performance.

  dfalbel

  committed
  218c44f

  Copy full SHA for 218c44f

• Add NEWS bullets + possibility to use older serialization format.

  dfalbel

  committed
  29efd83

  Copy full SHA for 29efd83