OpenCoder series

[ysjprojects]

OpenCoder is an open and reproducible code LLM family which includes 1.5B and 8B base and chat models, supporting both English and Chinese languages. Starting from scratch, OpenCoder is trained on 2.5 trillion tokens composed of 90% raw code and 10% code-related web data, reaching the performance of top-tier code LLMs. We provide not only model weights and inference code, but also the reproducible training data, the complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols. Empowering researchers to build and innovate, OpenCoder is your open foundation for advancing code AI.

State of the art code LLM that beats Qwen2.5-Coder of equivalent size.

https://opencoder-llm.github.io/
https://arxiv.org/pdf/2411.04905

https://huggingface.co/infly/OpenCoder-1.5B
https://huggingface.co/infly/OpenCoder-1.5B-Instruct
https://huggingface.co/infly/OpenCoder-8B
https://huggingface.co/infly/OpenCoder-8B-Instruct

Issue #1858

[Andrei-Aksionov]

Suggested change

	repo_id, cache_dir=cache_dir / "hf", local_files_only=True, token=access_token, trust_remote_code=True
	repo_id, cache_dir=cache_dir / "hf", local_files_only=False, token=access_token, trust_remote_code=True

Not sure how exactly it works inside, but I suspect that it wants to download additional files (since trust_remove_code is True) and limiting to only local files prevents it.

[Andrei-Aksionov]

Hello @ysjprojects

Thanks for another great PR.

But, in this case, it's a bit difficult.
The repo for this model only contains tokenizer.model, which is the tokenizer vocab file for SentencePiece.
In addition to this, there are additional tokens that aren't in the vocab, but in the added_tokens.json.
The problem is that it's a pain to extend SentencePiece, once the tokenizer is already trained.

How does HF team extends SentencePiece. Well, they don't 🙂.
AutoTokenizer converts from SentencePiece to Tokenizers format and then extends the vocabulary.

In practice that means that this model might generate tokens infinitely, since one of the bos tokens are in added_tokens.json and thus the tokenizer doesn't know about them 🫠.

[ysjprojects]

Hello @ysjprojects

Thanks for another great PR.

But, in this case, it's a bit difficult. The repo for this model only contains tokenizer.model, which is the tokenizer vocab file for SentencePiece. In addition to this, there are additional tokens that aren't in the vocab, but in the added_tokens.json. The problem is that it's a pain to extend SentencePiece, once the tokenizer is already trained.

How does HF team extends SentencePiece. Well, they don't 🙂. AutoTokenizer converts from SentencePiece to Tokenizers format and then extends the vocabulary.

In practice that means that this model might generate tokens infinitely, since one of the bos tokens are in added_tokens.json and thus the tokenizer doesn't know about them 🫠.

I see. In that case would you consider this problem completely intractable or just requires a lot of work?

[Andrei-Aksionov]

I see. In that case would you consider this problem completely intractable or just requires a lot of work?

There is nothing intractable about it 🙂.
We just need to support all the conversion steps from AutoTokenizer, since we need to match it.

Or, we can cheat a little bit for now, and just check if there is only tokenizer.model available, but also added_tokens.json, then just load into AutoTokenizer, save into tokenizer.json and proceed with initialization. Later we can extend the logic to get rid of AutoTokenizer and have more control.

But, what bothers me, is that in the repo there is another file - tokenization_inflm.py. As I see it, it contains a custom logic for tokenization, as-like *_modeling.py files that you might have encountered with other models (for custom model implementation, that is not yet supported by HF).

For now, I raised a question in their repo. Perhaps there is an easier way to do all this 🤷.