Code

Multithreaded downloads#6794

Code

Merged

lhoestq merged 13 commits intomainhuggingface/datasets:mainfrom multithreaded-downloadshuggingface/datasets:multithreaded-downloadsCopy head branch name to clipboardApr 15, 2024

+158-25

Commits

Commits on Apr 9, 2024

• multithreaded downloads

  lhoestq

  committed
  f72bffb

  Copy full SHA for f72bffb

• fix

  lhoestq

  committed
  b05e626

  Copy full SHA for b05e626

• fix again

  lhoestq

  committed
  db97134

  Copy full SHA for db97134

• fix tests

  lhoestq

  committed
  cd42a37

  Copy full SHA for cd42a37

• fix

  lhoestq

  committed
  e4a1d01

  Copy full SHA for e4a1d01

• fix 16 workers

  lhoestq

  committed
  a2e921b

  Copy full SHA for a2e921b

• enable multithreading only for small files

  lhoestq

  committed
  bd5a779

  Copy full SHA for bd5a779

• pin uv

  lhoestq

  committed
  0ccb360

  Copy full SHA for 0ccb360

Commits on Apr 11, 2024

• fix tests

  lhoestq

  committed
  6ccc067

  Copy full SHA for 6ccc067

Commits on Apr 12, 2024

• unpin uv

  lhoestq

  committed
  8e56b3e

  Copy full SHA for 8e56b3e

• add HF_DATASETS_MULTITHREADING_MAX_WORKERS

  lhoestq

  committed
  0ecc64f

  Copy full SHA for 0ecc64f

• rename _download_single

  lhoestq

  committed
  d8e31fb

  Copy full SHA for d8e31fb

• minor

  lhoestq

  committed
  89c21fa

  Copy full SHA for 89c21fa