Code

MPI/TCP/FS for NCCL-init#632

Code

Merged

karpathy merged 20 commits intokarpathy:masterkarpathy/llm.c:masterfrom gordicaleksa:multi_node_mygordicaleksa/llm.c:multi_node_myCopy head branch name to clipboardJun 24, 2024

+683-72

Commits

Commits on Jun 24, 2024

• Use NCCL where possible

  gordicaleksa

  committed
  0addb50

  Copy full SHA for 0addb50

• Can not call set device after NCCL init

  gordicaleksa

  committed
  52db158

  Copy full SHA for 52db158

• Use NCCL stream

  gordicaleksa

  committed
  ac72875

  Copy full SHA for ac72875

• Mini refactor

  gordicaleksa

  committed
  1bd7375

  Copy full SHA for 1bd7375

• Free up unified buffer mem

  gordicaleksa

  committed
  08c055c

  Copy full SHA for 08c055c

• fs working

  gordicaleksa

  committed
  5500b61

  Copy full SHA for 5500b61

• TCP working

  gordicaleksa

  committed
  1c6c5ff

  Copy full SHA for 1c6c5ff

• Add all 3 init methods

  gordicaleksa

  committed
  1bc3a5e

  Copy full SHA for 1bc3a5e

• Add launch scripts - tested

  gordicaleksa

  committed
  bfdea88

  Copy full SHA for bfdea88

• Add usage for new args

  gordicaleksa

  committed
  75a2c8b

  Copy full SHA for 75a2c8b

• Add gpus per node arg

  gordicaleksa

  committed
  3abdf9a

  Copy full SHA for 3abdf9a

• Fix CI errors

  gordicaleksa

  committed
  a259eea

  Copy full SHA for a259eea

• Add MPI under multi gpu guard

  gordicaleksa

  committed
  fbf4f59

  Copy full SHA for fbf4f59

• Fix arg permutation

  gordicaleksa

  committed
  c420a52

  Copy full SHA for c420a52

• Fix makefile

  gordicaleksa

  committed
  05ef1e7

  Copy full SHA for 05ef1e7

• Add socket logic for Windows

  gordicaleksa

  committed
  f9d0e8b

  Copy full SHA for f9d0e8b

• Add a multi node readme section

  gordicaleksa

  committed
  60c95c8

  Copy full SHA for 60c95c8

• Remove linux socket funcs when windows

  gordicaleksa

  committed
  ac1e37b

  Copy full SHA for ac1e37b

• Fix windows include issue

  gordicaleksa

  committed
  2667aa1

  Copy full SHA for 2667aa1

• Remove /tmp as default init value for fs_path

  gordicaleksa

  committed
  4af6a6a

  Copy full SHA for 4af6a6a