Vitaliy/sync foundry tests

[vchiley]

ports all other tests from llm-foundry

# pytest tests/*
=========================================================================== test session starts ===========================================================================
platform linux -- Python 3.9.15, pytest-7.2.0, pluggy-1.0.0
rootdir: /workdisk/vitaliy/benchmarks/llm
collected 13 items                                                                                                                                                        

tests/compare_hf_v_mosaic_gpt.py ....                                                                                                                               [ 15%]
tests/dataloader_tests.py .                                                                                                                                         [ 23%]
tests/model_tests.py ...                                                                                                                                            [ 46%]
tests/test_file_loading_scripts.py ..                                                                                                                               [ 61%]
tests/tokenizer_tests.py .                                                                                                                                          [ 69%]
tests/training_integration_tests.py ..

===================================================================== 13 passed in 104.30s (0:01:44) ======================================================================

lmk if you think we should merge all of these or what we should do with them.

[vchiley]

rm unittests

benchmarks/llm# pytest tests/*
====================================================================== test session starts =======================================================================
platform linux -- Python 3.9.15, pytest-7.2.0, pluggy-1.0.0
rootdir: /workdisk/vitaliy/benchmarks/llm
collected 13 items                                                                                                                                               

tests/c4_data_prep_script.py ....                                                                                                                          [ 15%]
tests/compare_hf_v_mosaic_gpt.py ..                                                                                                                        [ 30%]
tests/dataloader_tests.py .                                                                                                                                [ 38%]
tests/model_tests.py ...                                                                                                                                   [ 61%]
tests/tokenizer_tests.py .                                                                                                                                 [ 69%]
tests/training_integration_tests.py ..

================================================================= 13 passed in 104.06s (0:01:44) =================================================================

[dblalock]

Tried a fresh checkout of the branch in an interactive instance and the tests choked on my-copy-c4 not being present. Excited to merge the improved testing once this works (or xfails?) though.

root@interactive-a100-40gb-1-v15y-5bm5c:/workspace/vitaliy-benchmarks/llm# pytest tests/* --tb=short
...
tests/c4_data_prep_script.py ....
tests/compare_hf_v_mosaic_gpt.py ..
tests/dataloader_tests.py F
tests/model_tests.py FFF
tests/tokenizer_tests.py .
tests/training_integration_tests.py FF
...
----------------------------------------------------------- Captured stdout call ------------------------------------------------------------
Initializing model...
cfg.n_params=1.64e+06
Building train loader...
========================================================== short test summary info ==========================================================
FAILED tests/dataloader_tests.py::test_correct_padding - FileNotFoundError: [Errno 2] No such file or directory: './my-copy-c4/val/index.json'
FAILED tests/model_tests.py::test_full_forward_and_backward - FileNotFoundError: [Errno 2] No such file or directory: './my-copy-c4/val/index.json'
FAILED tests/model_tests.py::test_attention_mechanism - FileNotFoundError: [Errno 2] No such file or directory: './my-copy-c4/val/index.json'
FAILED tests/model_tests.py::test_full_forward_and_backward_gpt_neo - FileNotFoundError: [Errno 2] No such file or directory: './my-copy-c4/val/index.json'
FAILED tests/training_integration_tests.py::test_train[cpu] - FileNotFoundError: [Errno 2] No such file or directory: './my-copy-c4/val/index.json'
FAILED tests/training_integration_tests.py::test_train[cuda] - FileNotFoundError: [Errno 2] No such file or directory: './my-copy-c4/val/index.json'
================================================== 6 failed, 7 passed in 61.96s (0:01:01) ===================================================

Also, as long as we're changing stuff, could you have all the files with tests start with test_? This way pytest tests/ works, not just pytest tests/*.

[vchiley]

updated with xfail

(venv) root@interactive-a100-40gb-1-2loe-dwpsm:/workdisk/vitaliy/benchmarks/llm# pytest tests/*
============================================== test session starts ===============================================
platform linux -- Python 3.9.15, pytest-7.2.0, pluggy-1.0.0
rootdir: /workdisk/vitaliy/benchmarks/llm
collected 13 items                                                                                                

tests/c4_data_prep_script.py ....                                                                           [ 15%]
tests/compare_hf_v_mosaic_gpt.py ..                                                                         [ 30%]
tests/dataloader_tests.py .                                                                                 [ 38%]
tests/model_tests.py ...                                                                                    [ 61%]
tests/tokenizer_tests.py .                                                                                  [ 69%]
tests/training_integration_tests.py ..

========================================= 13 passed in 102.14s (0:01:42) =========================================
(venv) root@interactive-a100-40gb-1-2loe-dwpsm:/workdisk/vitaliy/benchmarks/llm# ls
README.md  __pycache__  assets  convert_c4.py  main.py  my-copy-c4  requirements.txt  src  tests  venv  yamls
(venv) root@interactive-a100-40gb-1-2loe-dwpsm:/workdisk/vitaliy/benchmarks/llm# mv my-copy-c4 tmp_my-copy-c4
(venv) root@interactive-a100-40gb-1-2loe-dwpsm:/workdisk/vitaliy/benchmarks/llm# ls
README.md  __pycache__  assets  convert_c4.py  main.py  requirements.txt  src  tests  tmp_my-copy-c4  venv  yamls
(venv) root@interactive-a100-40gb-1-2loe-dwpsm:/workdisk/vitaliy/benchmarks/llm# pytest tests/*
============================================== test session starts ===============================================
platform linux -- Python 3.9.15, pytest-7.2.0, pluggy-1.0.0
rootdir: /workdisk/vitaliy/benchmarks/llm
collected 13 items                                                                                                

tests/c4_data_prep_script.py ....                                                                           [ 15%]
tests/compare_hf_v_mosaic_gpt.py ..                                                                         [ 30%]
tests/dataloader_tests.py x                                                                                 [ 38%]
tests/model_tests.py ...                                                                                    [ 61%]
tests/tokenizer_tests.py .                                                                                  [ 69%]
tests/training_integration_tests.py xx

==================================== 10 passed, 3 xfailed in 148.99s (0:02:28) ===================================
(venv) root@interactive-a100-40gb-1-2loe-dwpsm:/workdisk/vitaliy/benchmarks/llm#