Same token can be matched multiple times in stringMatchesReference (for evaluating answers)

[alopezlago]

website/server/scorer.js

Lines 428 to 450 in 12f2e68

	// check if every token in the string is in the reference
	for (let i = 0; i < stringTokens.length; i++) {
	let tokenMatches = false;
	for (let j = 0; j < referenceTokens.length; j++) {
	let errors;
	if (useStemmer) {
	errors = distance(stemmer(stringTokens[i]), stemmer(referenceTokens[j]));
	} else {
	errors = distance(stringTokens[i], referenceTokens[j]);
	}
	if (strictness * errors <= referenceTokens[j].length || (acceptSubstring && referenceTokens[j].includes(stringTokens[i]))) {
	tokenMatches = true;
	break;
	}
	}
	if (!tokenMatches) {
	return false;
	}
	}

When comparing stringTokens against referenceTokens in Room.stringMatchesReference, we never remove the reference token that matches the string token. This means that the same reference token can match different string tokens. This will lead to inocrrect matches. For example, there will be a match when the reference string "Major Tom" is compared against the string "Major Major Major", even though they are different.

There are a few ways to do this. One way to do it is to keep track of the last reference token we should use, and when we find a matching reference token, we swap it with the token at the last position and decrease the number of reference tokens we check against.

[geoffrey-wu]

In a similar vein, token order should (probably) matter - i.e. "potential chemical" shouldn't match against "chemical potential". Probably easiest to keep an index for both arrays and increment them (never going backwards) to check for a match.

[geoffrey-wu]

For now, I'm going to assume that token order does not matter; otherwise, answers like "Australian Prime Minister" don't get accepted for "Prime Minister of Australia", for example.