How to prepare a dataset to train "Quality Scorer" classifier?

[kdcyberdude]

I want to know the implementation details of the "Quality Scorer" and "Document Coherence Scorer" filters.

[renxiaoyi]

Basically, "Quality Scorer" is a fasttext classifier that was trained to assign high scores to pages that are similar to "high quality" content like Wikipedia pages and books. "Document Coherence Scorer" is a scorer to assign high scores to pages where paragraphs are more "consistent", bases on their embedding cosine similarity.

[kdcyberdude]

Hi @renxiaoyi, Did you guys use LLM to generate the dataset, or was it manually created? Alternatively, is there any available dataset for the classifier?