[Frontend][Misc] Don't Repeat Yourself (DRY) Sampling

[AlpinDale]

FIX #8581

This PR adds support for the DRY sampler, a modern repetition penalty to prevent repetitive outputs at a sequence-level, and performs much better than older penalty methods.

Showcase:

Prompt:

All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work

No DRY output:

 and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy.\n\nSomewhere between the repetition and the mundane task of George Orwell’s 1984 and the evocative yet disturbing image of a mess of typewriter keys

dry_multiplier=1.0

... (more)\nIn a remote part of town there once lived a lonely young boy named Danny who wrote the phrase \"All work and no p(l)ay makes Jack a d(u)ll boy.\" on a typewriter for what he would shortly come to realize was the remainder of his life. Danny, who was 8 years old at the time (1984), had not yet learned how to turn off his own typed phrase.\nAs I, a good six years older (1990), like many others who experienced The Shining, witnessed the two year and two month documented torment that befell this isolated young soul. A remote"

Currently, using DRY can incur an overhead of up to ~25%. We can probably optimize this.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[AlpinDale]

/ready

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @AlpinDale.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[p-e-w]

Probably makes more sense to link to the original PR (oobabooga/text-generation-webui#5677), as that contains a detailed description and discussion, whereas this is just code similar to the code below.

[p-e-w]

50 is more than enough. In practice, top logits tend to be on the order of 10^2, and 1.75^50 is already 10^12. This will double performance for pathological inputs.

[p-e-w]

Here's an idea: Instead of matching a single token above with nonzero, try to match the last allowed_length tokens. Shorter matches don't need to be penalized by definition, and this should drastically cut down on the number of matches that must be looped over. I'm not sure how to perform subsequence matching in PyTorch, but there should be a way to do this as a vectorized operation.

[p-e-w]

What if ngram_lens[penalty_mask] - min_ngram < 0? Perhaps I'm misunderstanding the implementation, but that case doesn't appear to be accounted for.

[akkikiki]

+1 on confirming this.
If I understand correctly, we want something like the following to filer out the cases when ngram_lens[penalty_mask] - min_ngram < 0:

# Apply penalties
penalty_mask = ngram_lens > 0
exponent = ngram_lens[penalty_mask] - min_ngram

# Ensure that when exponent < 0, scales is set to 0
valid_exponent_mask = exponent >= 0
scales = torch.zeros_like(exponent, dtype=torch.float32)
scales[valid_exponent_mask] = bases[irow] ** exponent[valid_exponent_mask]

# Apply penalties
logits[irow][penalty_mask] -= multipliers[irow] * scales

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @AlpinDale.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork