GemmaMLP uses 'tanh` approximation for GeLU activation

[Andrei-Aksionov]

Hi there 👋

Fixes #999

As @carmocca found out, the original Keras implementation of Gemma used tanh approximation for GeLU activation, but neither PyTorch implementation nor HF Transformers variant had it.
Recently the official PyTorch variant was updated to also use this approximation and this PR reflects this change.

[carmocca]

Thank you! You'll have to xfail the HF comparison tests for now

[Andrei-Aksionov]

You'll have to xfail the HF comparison tests for now

First I need to understand why test_model.py doesn't fail, but test_convert_lit_checkpoint.py does fail.
Maybe there is something wrong with the conversion script (or the test) and this approximation just makes it more pronounced?

[Andrei-Aksionov]

I looked through the code maybe a dozen of times and could find anything wrong.
Still don't understand why when we convert from HF to Lit the test passes, but when from Lit to HF - doesn't.
If in the single test I first convert HF -> Lit and then right after Lit -> HF - test passes 🤷.

[rasbt]

Thanks for looking into this! Before merging, should we run a few eval harness tasks to compare the before/after performance? E.g., hellaswag and TruthfulQA should be sufficient. If you don't have access to GPUs, I am happy to do that @Andrei-Aksionov, just let me know.

[Andrei-Aksionov]

I have access, but I'm just busy with the other task (drop interleaved placement in QKV, should unlock your OLMo integration work).
So, if you have an already prepared bash script/notebook to do it (meaning that it won't take too much time for you), then do it.
If not, I can do it, but a bit later.

[rasbt]

No worries I can do it

[rasbt]

It doesn't seem to perform differently without and with the PR:

# before

hellaswag acc norm: 0.4230233021310496
truthful qa mc1: 0.24724602203182375

# after
hellaswag acc norm: 0.4230233021310496
truthful qa mc1: 0.24724602203182375

Btw the hellaswag score is really bad, with tinyllama I get 0.60. Maybe it's a weird benchmark.

[carmocca]

It is: https://www.surgehq.ai/blog/hellaswag-or-hellabad-36-of-this-popular-llm-benchmark-contains-errors

[rasbt]

Arg ... 🤦‍♂️. Need to find something else ... but in any case, I think the change via the PR seems to ok!?

[kashif]

see also here: https://twitter.com/danielhanchen/status/1765446273661075609

[Andrei-Aksionov]

Thanks for the link, @kashif.
I intend to examine this carefully.