812 implement de berta for zero shot classification annotator

[ahmedlone127]

adding DeBertaForZeroShotClassification ( note needs to be reviewed to see why predictions( Very different from huggingface predictions and largely incorrect ) are not being made correctly , same as BartForZeroShotClassifications

[maziyarpanahi]

@DevinTDHa Could you please have a look and see why the predictions are different from HuggingFace?

[DevinTDHa]

Hi @maziyarpanahi @ahmedlone127, I encountered this issue before and propose a solution:

Underlying Cause

I believe the issue could be part of the following. Note that this might be specific to implementations that I looked at, namely RoBertaForQA and MPNetForQA. It might differ for different models.

1. Attention mask tensors are wrongly encoded. They should be an array of ones, but instead we are creating it based on the token ids (token 0 could be a valid ID and not a special token, depending on the model)

   spark-nlp/src/main/scala/com/johnsnowlabs/ml/ai/DeBertaClassification.scala

   Line 410 in 67e67e1

   batch.map(sentence => sentence.map(x => if (x == 0L) 0L else 1L)).toArray)

2. Logits are not pre-processed before calculating probabilities with softmax. In the original implementations, special tokens and question tokens are not included in the start/end probabilities. This is done by setting the log-probability for these tokens to -10000 which is 0 probability.

   spark-nlp/src/main/scala/com/johnsnowlabs/ml/ai/DeBertaClassification.scala

   Lines 330 to 346 in 67e67e1

   	def tagSpan(batch: Seq[Array[Int]]): (Array[Array[Float]], Array[Array[Float]]) = {
   	val batchLength = batch.length
   	val (startLogits, endLogits) = detectedEngine match {
   	case ONNX.name => computeLogitsWithOnnx(batch)
   	case _ => computeLogitsWithTF(batch)
   	}
   	val endDim = endLogits.length / batchLength
   	val endScores: Array[Array[Float]] =
   	endLogits.grouped(endDim).map(scores => calculateSoftmax(scores)).toArray
   	val startDim = startLogits.length / batchLength
   	val startScores: Array[Array[Float]] =
   	startLogits.grouped(startDim).map(scores => calculateSoftmax(scores)).toArray
   	(startScores, endScores)
   	}

3. Final Score is not correctly calculated. it should be the likelihood (startScore * endScore) instead of the mean.

   spark-nlp/src/main/scala/com/johnsnowlabs/ml/ai/XXXForClassification.scala

   Line 303 in 060cf6d

   "score" -> ((startIndex._1 + endIndex._1) / 2).toString)))

Fix

I fixed these issues for RobertaForQA and the upcoming MPNetForQA by overriding the base functions in the specific implementation file (RobertaClassification.scala and MPNetForClassification.scala).

They are included in the respective PR: #14147

I chose not to fix the XXXForClassification.scala file, as I don't know if the other annotators are also affected that use it. It could be model specific.

So going forward, I'd suggest looking at my implementation and checking if the scores are better than before. The implementation is here:

spark-nlp/src/main/scala/com/johnsnowlabs/ml/ai/MPNetClassification.scala

Lines 429 to 477 in a2d08d6

	override def predictSpan(
	documents: Seq[Annotation],
	maxSentenceLength: Int,
	caseSensitive: Boolean,
	mergeTokenStrategy: String = MergeTokenStrategy.vocab,
	engine: String = TensorFlow.name): Seq[Annotation] = {
	val questionAnnot = Seq(documents.head)
	val contextAnnot = documents.drop(1)
	val wordPieceTokenizedQuestion =
	tokenizeDocument(questionAnnot, maxSentenceLength, caseSensitive)
	val wordPieceTokenizedContext =
	tokenizeDocument(contextAnnot, maxSentenceLength, caseSensitive)
	val contextLength = wordPieceTokenizedContext.head.tokens.length
	val questionLength = wordPieceTokenizedQuestion.head.tokens.length
	val encodedInput =
	encodeSequence(wordPieceTokenizedQuestion, wordPieceTokenizedContext, maxSentenceLength)
	val (rawStartLogits, rawEndLogits) = tagSpan(encodedInput)
	val (startScores, endScores) =
	processLogits(rawStartLogits.head, rawEndLogits.head, questionLength, contextLength)
	// Drop BOS token from valid results
	val startIndex = startScores.zipWithIndex.drop(1).maxBy(_._1)
	val endIndex = endScores.zipWithIndex.drop(1).maxBy(_._1)
	val offsetStartIndex = 3 // 3 added special tokens
	val offsetEndIndex = offsetStartIndex - 1
	val allTokenPieces =
	wordPieceTokenizedQuestion.head.tokens ++ wordPieceTokenizedContext.flatMap(x => x.tokens)
	val decodedAnswer =
	allTokenPieces.slice(startIndex._2 - offsetStartIndex, endIndex._2 - offsetEndIndex)
	val content =
	mergeTokenStrategy match {
	case MergeTokenStrategy.vocab =>
	decodedAnswer.filter(_.isWordStart).map(x => x.token).mkString(" ")
	case MergeTokenStrategy.sentencePiece =>
	val token = ""
	decodedAnswer
	.map(x =>
	if (x.isWordStart) " " + token + x.token
	else token + x.token)
	.mkString("")
	.trim
	}
	val totalScore = startIndex._1 * endIndex._1

Just as a note, the tolerance for the scores of my implementation are not that low (0.01 float tolerance). Close enough I suppose but I believe there might be some numerical issues for the implementation in general.

It can be tested like so:

spark-nlp/src/test/scala/com/johnsnowlabs/nlp/annotators/classifier/dl/MPNetForQuestionAnsweringTestSpec.scala

Line 103 in a2d08d6

implicit val tolerantEq = TolerantNumerics.tolerantFloatEquality(1e-2f)

[maziyarpanahi]

• Thanks @DevinTDHa appreciate sharing your experience
• @ahmedlone127 could you please apply the changes and re-test?

[ahmedlone127]

Thanks for the detailed explanation @DevinTDHa !

I will implement these steps and get back to you @maziyarpanahi

[ahmedlone127]

Hello @maziyarpanahi , I took another look at the source code and found that by changing the offset value for the tokenizations (from 1 to 0 ) the predictions from the model became a lot more accurate.

[coveralls]

Pull Request Test Coverage Report for Build 7799480957

Warning: This coverage report may be inaccurate.

We've detected an issue with your CI configuration that might affect the accuracy of this pull request's coverage report.
To ensure accuracy in future PRs, please see these guidelines.
A quick fix for this PR: rebase it; your next report should be accurate.

• -1 of 1 (0.0%) changed or added relevant line in 1 file are covered.
• 7 unchanged lines in 5 files lost coverage.
• Overall coverage decreased (-0.01%) to 62.756%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/main/scala/com/johnsnowlabs/nlp/pretrained/ResourceDownloader.scala	0	1	0.0%

Files with Coverage Reduction	New Missed Lines	%
src/main/scala/com/johnsnowlabs/nlp/pretrained/S3ResourceDownloader.scala	1	49.12%
src/main/scala/com/johnsnowlabs/util/Benchmark.scala	1	70.59%
src/main/scala/com/johnsnowlabs/util/ZipArchiveUtil.scala	1	87.67%
src/main/scala/com/johnsnowlabs/nlp/util/io/ResourceHelper.scala	2	47.02%
src/main/scala/com/johnsnowlabs/storage/StorageHelper.scala	2	59.18%

Totals
Change from base Build 7799297990:	-0.01%
Covered Lines:	8954
Relevant Lines:	14268

---

💛 - Coveralls