Generate: add missing logits processors docs (#25653)

huggingface · Aug 28, 2023 · 6836e9d · 6836e9d
1 parent e82040e
commit 6836e9d
Show file tree

Hide file tree

Showing 6 changed files with 339 additions and 49 deletions.
diff --git a/docs/source/en/internal/generation_utils.md b/docs/source/en/internal/generation_utils.md
@@ -75,39 +75,104 @@ values. Here, for instance, it has two keys that are `sequences` and `scores`.
 We document here all output types.
 
 
-### GreedySearchOutput
-
-[[autodoc]] generation.GreedySearchDecoderOnlyOutput
+### PyTorch
 
 [[autodoc]] generation.GreedySearchEncoderDecoderOutput
 
-[[autodoc]] generation.FlaxGreedySearchOutput
+[[autodoc]] generation.GreedySearchDecoderOnlyOutput
 
-### SampleOutput
+[[autodoc]] generation.SampleEncoderDecoderOutput
 
 [[autodoc]] generation.SampleDecoderOnlyOutput
 
-[[autodoc]] generation.SampleEncoderDecoderOutput
+[[autodoc]] generation.BeamSearchEncoderDecoderOutput
 
-[[autodoc]] generation.FlaxSampleOutput
+[[autodoc]] generation.BeamSearchDecoderOnlyOutput
 
-### BeamSearchOutput
+[[autodoc]] generation.BeamSampleEncoderDecoderOutput
 
-[[autodoc]] generation.BeamSearchDecoderOnlyOutput
+[[autodoc]] generation.BeamSampleDecoderOnlyOutput
 
-[[autodoc]] generation.BeamSearchEncoderDecoderOutput
+[[autodoc]] generation.ContrastiveSearchEncoderDecoderOutput
 
-### BeamSampleOutput
+[[autodoc]] generation.ContrastiveSearchDecoderOnlyOutput
 
-[[autodoc]] generation.BeamSampleDecoderOnlyOutput
+### TensorFlow
 
-[[autodoc]] generation.BeamSampleEncoderDecoderOutput
+[[autodoc]] generation.TFGreedySearchEncoderDecoderOutput
+
+[[autodoc]] generation.TFGreedySearchDecoderOnlyOutput
+
+[[autodoc]] generation.TFSampleEncoderDecoderOutput
+
+[[autodoc]] generation.TFSampleDecoderOnlyOutput
+
+[[autodoc]] generation.TFBeamSearchEncoderDecoderOutput
+
+[[autodoc]] generation.TFBeamSearchDecoderOnlyOutput
+
+[[autodoc]] generation.TFBeamSampleEncoderDecoderOutput
+
+[[autodoc]] generation.TFBeamSampleDecoderOnlyOutput
+
+[[autodoc]] generation.TFContrastiveSearchEncoderDecoderOutput
+
+[[autodoc]] generation.TFContrastiveSearchDecoderOnlyOutput
+
+### FLAX
+
+[[autodoc]] generation.FlaxSampleOutput
+
+[[autodoc]] generation.FlaxGreedySearchOutput
+
+[[autodoc]] generation.FlaxBeamSearchOutput
 
 ## LogitsProcessor
 
 A [`LogitsProcessor`] can be used to modify the prediction scores of a language model head for
 generation.
 
+### PyTorch
+
+[[autodoc]] AlternatingCodebooksLogitsProcessor
+    - __call__
+
+[[autodoc]] ClassifierFreeGuidanceLogitsProcessor
+    - __call__
+
+[[autodoc]] EncoderNoRepeatNGramLogitsProcessor
+    - __call__
+
+[[autodoc]] EncoderRepetitionPenaltyLogitsProcessor
+    - __call__
+
+[[autodoc]] EpsilonLogitsWarper
+    - __call__
+
+[[autodoc]] EtaLogitsWarper
+    - __call__
+
+[[autodoc]] ExponentialDecayLengthPenalty
+    - __call__
+
+[[autodoc]] ForcedBOSTokenLogitsProcessor
+    - __call__
+
+[[autodoc]] ForcedEOSTokenLogitsProcessor
+    - __call__
+
+[[autodoc]] ForceTokensLogitsProcessor
+    - __call__
+
+[[autodoc]] HammingDiversityLogitsProcessor
+    - __call__
+
+[[autodoc]] InfNanRemoveLogitsProcessor
+    - __call__
+
+[[autodoc]] LogitNormalization
+    - __call__
+
 [[autodoc]] LogitsProcessor
     - __call__
 
@@ -123,61 +188,63 @@ generation.
 [[autodoc]] MinNewTokensLengthLogitsProcessor
     - __call__
 
-[[autodoc]] TemperatureLogitsWarper
+[[autodoc]] NoBadWordsLogitsProcessor
     - __call__
 
-[[autodoc]] RepetitionPenaltyLogitsProcessor
+[[autodoc]] NoRepeatNGramLogitsProcessor
     - __call__
 
-[[autodoc]] TopPLogitsWarper
+[[autodoc]] PrefixConstrainedLogitsProcessor
     - __call__
 
-[[autodoc]] TopKLogitsWarper
+[[autodoc]] RepetitionPenaltyLogitsProcessor
     - __call__
 
-[[autodoc]] TypicalLogitsWarper
+[[autodoc]] SequenceBiasLogitsProcessor
     - __call__
 
-[[autodoc]] NoRepeatNGramLogitsProcessor
+[[autodoc]] SuppressTokensAtBeginLogitsProcessor
     - __call__
 
-[[autodoc]] SequenceBiasLogitsProcessor
+[[autodoc]] SuppressTokensLogitsProcessor
     - __call__
 
-[[autodoc]] NoBadWordsLogitsProcessor
+[[autodoc]] TemperatureLogitsWarper
     - __call__
 
-[[autodoc]] PrefixConstrainedLogitsProcessor
+[[autodoc]] TopKLogitsWarper
     - __call__
 
-[[autodoc]] HammingDiversityLogitsProcessor
+[[autodoc]] TopPLogitsWarper
     - __call__
 
-[[autodoc]] ForcedBOSTokenLogitsProcessor
+[[autodoc]] TypicalLogitsWarper
     - __call__
 
-[[autodoc]] ForcedEOSTokenLogitsProcessor
+[[autodoc]] UnbatchedClassifierFreeGuidanceLogitsProcessor
     - __call__
 
-[[autodoc]] InfNanRemoveLogitsProcessor
+[[autodoc]] WhisperTimeStampLogitsProcessor
     - __call__
 
-[[autodoc]] TFLogitsProcessor
+### TensorFlow
+
+[[autodoc]] TFForcedBOSTokenLogitsProcessor
     - __call__
 
-[[autodoc]] TFLogitsProcessorList
+[[autodoc]] TFForcedEOSTokenLogitsProcessor
     - __call__
 
-[[autodoc]] TFLogitsWarper
+[[autodoc]] TFForceTokensLogitsProcessor
     - __call__
 
-[[autodoc]] TFTemperatureLogitsWarper
+[[autodoc]] TFLogitsProcessor
     - __call__
 
-[[autodoc]] TFTopPLogitsWarper
+[[autodoc]] TFLogitsProcessorList
     - __call__
 
-[[autodoc]] TFTopKLogitsWarper
+[[autodoc]] TFLogitsWarper
     - __call__
 
 [[autodoc]] TFMinLengthLogitsProcessor
@@ -192,10 +259,30 @@ generation.
 [[autodoc]] TFRepetitionPenaltyLogitsProcessor
     - __call__
 
-[[autodoc]] TFForcedBOSTokenLogitsProcessor
+[[autodoc]] TFSuppressTokensAtBeginLogitsProcessor
     - __call__
 
-[[autodoc]] TFForcedEOSTokenLogitsProcessor
+[[autodoc]] TFSuppressTokensLogitsProcessor
+    - __call__
+
+[[autodoc]] TFTemperatureLogitsWarper
+    - __call__
+
+[[autodoc]] TFTopKLogitsWarper
+    - __call__
+
+[[autodoc]] TFTopPLogitsWarper
+    - __call__
+
+### FLAX
+
+[[autodoc]] FlaxForcedBOSTokenLogitsProcessor
+    - __call__
+
+[[autodoc]] FlaxForcedEOSTokenLogitsProcessor
+    - __call__
+
+[[autodoc]] FlaxForceTokensLogitsProcessor
     - __call__
 
 [[autodoc]] FlaxLogitsProcessor
@@ -207,27 +294,30 @@ generation.
 [[autodoc]] FlaxLogitsWarper
     - __call__
 
-[[autodoc]] FlaxTemperatureLogitsWarper
+[[autodoc]] FlaxMinLengthLogitsProcessor
     - __call__
 
-[[autodoc]] FlaxTopPLogitsWarper
+[[autodoc]] FlaxSuppressTokensAtBeginLogitsProcessor
     - __call__
 
-[[autodoc]] FlaxTopKLogitsWarper
+[[autodoc]] FlaxSuppressTokensLogitsProcessor
     - __call__
 
-[[autodoc]] FlaxForcedBOSTokenLogitsProcessor
+[[autodoc]] FlaxTemperatureLogitsWarper
     - __call__
 
-[[autodoc]] FlaxForcedEOSTokenLogitsProcessor
+[[autodoc]] FlaxTopKLogitsWarper
     - __call__
 
-[[autodoc]] FlaxMinLengthLogitsProcessor
+[[autodoc]] FlaxTopPLogitsWarper
+    - __call__
+
+[[autodoc]] FlaxWhisperTimeStampLogitsProcessor
     - __call__
 
 ## StoppingCriteria
 
-A [`StoppingCriteria`] can be used to change when to stop generation (other than EOS token).
+A [`StoppingCriteria`] can be used to change when to stop generation (other than EOS token). Please note that this is exclusivelly available to our PyTorch implementations.
 
 [[autodoc]] StoppingCriteria
     - __call__
@@ -243,7 +333,7 @@ A [`StoppingCriteria`] can be used to change when to stop generation (other than
 
 ## Constraints
 
-A [`Constraint`] can be used to force the generation to include specific tokens or sequences in the output.
+A [`Constraint`] can be used to force the generation to include specific tokens or sequences in the output. Please note that this is exclusivelly available to our PyTorch implementations.
 
 [[autodoc]] Constraint