autocomplete: In @-mention results, match insensitively to diacritics

[chrisbobbe]

If there's a user with Étienne in their name, we should still include it in the results for the query etienne.

The matching is already case-insensitive, but it should be insensitive to diacritics too.

[chrisbobbe]

(In principle this should also apply to user-group mentions, not just user mentions. User-group mention autocomplete hasn't been implemented yet: #233.)

[Khader-1]

Can we use something like diacritic?
BTW I'm new here (and new to open source in general) so what is the policy when it comes to packages and dependencies?

[gnprice]

It looks like that package consists of a hand-maintained set of mappings:
https://github.com/agilord/diacritic/blob/8cc02fb838f69bfffd25a7629787267814669b28/lib/src/replacement_map.dart#L59-L60
with no particular documentation of where the list comes from. So that isn't something I'd be eager to rely on.

The Zulip web client and the legacy zulip-mobile app have functionality like this. So let's implement the same behavior they have.

That behavior has been stable for a long time and I'm not aware of any bug reports we've gotten about it. If we later learn about ways the behavior can be improved, we can make the improvement for web too at the same time.

[Khader-1]

The Zulip web client and the legacy zulip-mobile app have functionality like this. So let's implement the same behavior they have.

After taking a look back at the mobile app implementation for this feature I found that it depends on @zulip/shared to handle this. check:
file: zulip-mobile/src/users/userHelpers.js

import * as typeahead from '@zulip/shared/js/typeahead';

const loweredFilter = filter.toLowerCase();
const isAscii = /^[a-z]+$/.test(loweredFilter);
return users.filter(user => {
  const full_name = isAscii ? typeahead.remove_diacritics(user.full_name) : user.full_name;
  return user.user_id !== ownUserId && full_name.toLowerCase().startsWith(loweredFilter);
});

Which uses the convenient Js String built in method normalize

file: @zulip/shared/lib/typeahead.js

const unicode_marks = /\p{M}/gu;
export function remove_diacritics(s) {
    return s.normalize("NFKD").replace(unicode_marks, "");
}

Which unfortunately dart does not have for now at least so we might have to consider third party packages, I am still searching and evaluating the options but I found several candidates:

• unorm-dart
• normalizer

So I will do some research to be able make an educated suggestion

[gnprice]

Updating this thread with a conclusion from chat and the PR thread #588 (comment) :

We'd use package:unorm_dart for this.

[chrisbobbe]

When making the search "fuzzy" like this, ideally we'd prioritize non-fuzzy matches over fuzzy matches. (zulip/zulip#32634 gave me the idea to comment here; we can split this to a new issue if helpful.)