Custom emotes

[not-layla]

Allows instance admins to define a list of custom emotes.

Can be typed out in text:

This text gets rendered as an emote:

When typing, there is an auto-complete pop-up:

There is also an emote picker:

Notes on implementation

The emote picker's performance is extremely poor, whatever our implementation of it sucks.

This already exists in ActivityPub, Mastodon has this feature, and it should ideally be implemented in a way such that it federates with Mastodon.

Mastodon's implementation is somewhat similar with Hexbear's, though their emote picker is significantly more performant. Both support animated emotes.

[PorkrollPosadist]

I'm going to write down a couple observations from a recent discussion on Matrix.

I'm not particularly experienced with Firefox's profiling tools but from a cursory glance, it appears the vast number of HTTP requests being made when the emote picker is opened is the primary cause of performance issues. On Hexbear, this is upward of 1000 requests. On my DESKTOP, this takes around 3 to 4 seconds. On less performant mobile devices, this only gets worse (≈10 seconds on my Pixel 4a 5G, a fairly recent "flagship" phone. Older, more affordable, used phones will suffer even more, as well as lower power devices designed deliberately for longer battery life).

This can probably be mitigated with better HTTP caching settings on the server, but it is also worth looking into if these requests could be paginated or otherwise done in a more lazy fashion.

The current implementation is basically federation-agnostic. It doesn't break federation as far as I know, but users on other instances would only see codes such as :reddit-logo: instead of an actual image. There could be a few potential approaches to federation. A lazy approach might involve simply replacing the emote codes with embedded image markdown, but this does not seem ideal.

If we want this feature to truly federate, there must be some API to retrieve emote metadata from an instance. At the very least, this would provide a list of emote names and their corresponding URLs. Additionally, it might provide categories (for grouping in the picker), alternate keywords (to improve search results in the picker and drop-down), and alternate text (for accessibility). Before getting carried away here, it would be good to review what some other Fediverse projects like Mastodon are doing. Anyway, we should ensure the information provided here is sufficient to browse the emote list without causing any major bottlenecks.

When it comes to federation, there are some behavioral / logical / administration issues we need to sort out as well. Currently on Hexbear, there is one set of emotes used throughout the instance. In a federated environment we need to decide, where will these emotes be available? Only on communities located on the same instance? Or can they be cross-referenced on communities outside of that instance? Also, It is worth considering that most Lemmy instances aren't operated with as much centralized control as Hexbear. It is probably more appropriate for communities to be able to set their own emotes, either exclusively, or in addition to an instance-wide set.

[PorkrollPosadist]

Here's a tentative proposal for how me might implement this:

High level thoughts:

• We stick to the "emoji" nomenclature (vs. "emote") throughout the implementation, since this is what Mastodon (the largest federated network) uses.
• Unlike the current implementation in Hexbear, we want to create a user interface so admins and/or community moderators can create / remove custom emojis. This implies that information about custom emojis should live in the database, rather than in a configuration file or the source code.
• I am making the assumption that emojis can be added by communities in addition to at the instance level (unlike Hexbear or Mastodon, where there is only one set at the instance level). Implementation details can be changed accordingly if this is not the path we want to take.
• Custom emoji images will be uploaded, stored, and hosted by Pictrs using the same mechanisms Lemmy uses for icons, avatars, banners, etc. This means we will lose support for SVG and GIF emojis, however animated emojis can be converted to WebP, SVGs can be rasterized (:disappointed:), and this would be workable.

Database:

We add a table to the database named 'custom_emoji,' which looks something like this

int id: Unique ID for emote
int community_id: Community this emote belongs to, or zero if it is defined at the instance level (is zero guaranteed to be free?)
varchar(128) shortcode: The text (excluding colon syntax) which this emoji is meant to replace
text url: The URL of the emoji image
text alt_text: Alternate text for the emoji image, to aid accessibility (e.g. "Smiling pile of poo" for 💩)
text category: Name of the category this emoji should be displayed under in the picker.
text keywords: Additional keywords to improve discoverability when using the search function in the picker (i.e. so you can find 🍌 when searching for "fruit").

This roughly reflects the Emoji object described in the Mastodon API, however we add a field for the owning community (Mastodon doesn't have communities), and search keywords. Hexbear is a bit of an extreme case with the number of emojis it has, but even comparing the emoji search functions in applications like Discord and Element, additional search tags (or whatever they're doing under the hood) make the picker in Discord much easier to use.

REST/WS API:

Based on the Mastodon docs, they just provide one API endpoint, GET custom_emojis, which simply returns an array of Emoji objects. I see nothing wrong with this interface (except this doc isn't very clear on the CRUD aspects), however we might want separate API endpoints for community and site. Admins can POST to site/custom_emojis, moderators can POST to community/custom_emojis, users can GET either.

Federation / ActivityPub:

Honestly, this is where my experience is lacking. Most of my experience hacking on Hexbear was from before Federation was implemented. I need to get up to speed on the architecture and implementation details. Until then, all I can say is that we need to decide the scope of information sharing which is going to take place here. At the very least, emote lists need to be made available between instances. Additionally, assuming it is possible to moderate communities on other instances, there needs to be a mechanism to add/remove emojis from those communities. We might also want to consider the privacy/performance trade-offs between hot-linking and caching the actual images.

Front-End / UI:

The Emoji picker we are currently using (emoji-mart) suffers from poor performance, so we will need to review alternatives or come up with an original solution. The interface will keep in line more-or-less with the standard experience found in applications like Discord, Element, Android Messenger, etc.

Displaying emoji images in comments is fairly trivial using the markdown-it library already employed by Lemmy. The front-end will grab the custom_emojis list for the instance and the current community, and replace the syntax if a match is found. If the instance and a community both define emojis with the same shortcode, the community emoji will take precedence.

I need to review how the auto-completion function works, but I remember it didn't take too long to implement the first time.

Then, there is the user interface where admins and mods can actually create / remove the emotes.

End:

I guess I'll leave it here for now. Admittedly, this is still fairly under-spec'ed, but A: you gotta start somewhere, and B: I'd prefer to hear some feedback before really drilling down the minute details.

[dessalines]

The federation stuff is the tricky part... if these are community rather than instance-based, then there needs to be some way of sending those DB rows to other instances that have connected to that community.

If @Nutomic thinks that's not doable (or wayy too complicated to implement), then I'd say we stick to emojis being instance-based, and have users request instance admins rather than community mods to add them.

[dessalines]

As far as an emoji picker, I've had terrible luck finding a good pure-js one, so we'd have to do a lot of searching. The last one lemmy-ui used (and I later removed), was bigger than the entire UI itself.

[Nutomic]

Emoji federation as done by Mastodon looks pretty simple. Basically it just includes all emojis used in the message in a tag field (search for "Custom emojis"). It doesnt seem necessary at all to synchronize the full list of emojis of each instance/community. But we will probably need an emoji db table which stores all local and known remote emojis.

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "toot": "http://joinmastodon.org/ns#",
      "Emoji": "toot:Emoji"
    }
  ],

  "id": "https://example.com/@alice/hello-world",
  "type": "Note",
  "content": "Hello world :kappa:",
  "tag": [
    {
      "id": "https://example.com/emoji/123",
      "type": "Emoji",
      "name": ":kappa:",
      "icon": {
        "type": "Image",
        "mediaType": "image/png",
        "url": "https://example.com/files/kappa.png"
      }
    }
  ]
}

[makotech222]

I'm looking to pick this one up after Featured Posts is finished. Porkroll's post pretty much lines up with what I would expect to implement. I would say I'd like to keep emoji's as instance-specific, rather than allowing communities to have their own lists, which seems like overkill and may increase complexity a bit. I'd also make the emoji tags into a many-to-one table relation, so adding a 'emoji_tags' table. I am also deficient in the apub stuff, so I'll probably need some more direction around that part. But otherwise, are there any objections to that design?

[makotech222]

I'm digging in a bit on the frontend. It looks like the most popular library for a picker is https://github.com/missive/emoji-mart, which is unfortunately a React-dependent library. There doesn't seem to be any native one for Inferno. There seems to be ways to 'host' the component in Inferno, using something like remount library, or maybe inferno-compat. Are either of these options that you guys want to bring into lemmy? Or should I try writing my own emoji picker component entirely in inferno?

edit: actually, I may be able to import the library just using a <script> element and use raw js to interact with it. I'll give that a try instead.

[Nutomic]

For federation you will have to add the tag array shown in my previous comment to Page, Note and other places where emojis can be used. It should contain a list of all emojis which are used in that item, so that the placeholder text (:kappa: in the example) can be replaced with the emoji image by each federated platform.

[dessalines]

We used to use an emoji-picker in lemmy-ui, but it was larger than the entire UI itself so I removed it. It might unfortunately be necessary to make an inferno specific one, which you could probably mostly copy-paste from the react one.

Or code one from scratch. All you'd need is a library that has a json list of all the emoji shortnames (I think I found one before), as well as the groupings.

As far as federation for custom emojis, these will have to be picture URLs I think, for them to work across different servers.

[makotech222]

We used to use an emoji-picker in lemmy-ui, but it was larger than the entire UI itself so I removed it. It might unfortunately be necessary to make an inferno specific one, which you could probably mostly copy-paste from the react one.

Or code one from scratch. All you'd need is a library that has a json list of all the emoji shortnames (I think I found one before), as well as the groupings.

As far as federation for custom emojis, these will have to be picture URLs I think, for them to work across different servers.

I have a prototype working with the emoji-mart v5, the .js file is only 95kb, though it may be downloading the default emojis separately, ill have to double check that.