"content" field is markdown but is advertised as HTML

[lanodan]

This was seen in #1220.

Currently in lemmy, the content field contains markdown which is tolerable but sadly it's advertised as text/html.

$ curl -s -H 'Accept: application/activity+json' https://ds9.lemmy.ml/comment/2207 | jq . 
{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "sensitive": "as:sensitive",
      "pt": "https://join-lemmy.org#",
      "sc": "http://schema.org#",
      "stickied": "as:stickied",
      "moderators": "as:moderators",
      "comments_enabled": {
        "type": "sc:Boolean",
        "id": "pt:commentsEnabled"
      },
      "matrixUserId": {
        "type": "sc:Text",
        "id": "as:alsoKnownAs"
      }
    },
    "https://w3id.org/security/v1"
  ],
  "type": "Note",
  "id": "https://ds9.lemmy.ml/comment/2207",
  "attributedTo": "https://ds9.lemmy.ml/u/lanodan",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public"
  ],
  "content": "Formatting test.\n\n#hashtag ?\n\n- zero\n- *one*\n- **two**\n- ***three***\n- ****four****\n- *****five*****\n\n~~strike~~\n\n> quote\n\n`code`\n\n```\ncode\nblock\n```\n\nCO~2~\nEMC^2^\n\n\n::: spoiler details\ncontent\n:::\n\n\n# h1\n## h2\n### h3\n#### h4",
  "mediaType": "text/html",
  "source": {
    "content": "Formatting test.\n\n#hashtag ?\n\n- zero\n- *one*\n- **two**\n- ***three***\n- ****four****\n- *****five*****\n\n~~strike~~\n\n> quote\n\n`code`\n\n```\ncode\nblock\n```\n\nCO~2~\nEMC^2^\n\n\n::: spoiler details\ncontent\n:::\n\n\n# h1\n## h2\n### h3\n#### h4",
    "mediaType": "text/markdown"
  },
  "inReplyTo": "https://ds9.lemmy.ml/comment/2206",
  "published": "2021-10-29T18:56:56.678095+00:00"
}

[Nutomic]

Thanks for finding this. Will definitely fix it before 0.14. If markdown is tolerable for you thats great, then i will just set mediatype markdown, put markdown in content and get rid of the source field.

[lanodan]

[2021-11-01 04:07:15-0700] Nutomic:

Thanks for finding this. Will definitely fix it before 0.14. If markdown is tolerable for you thats great, then i will just set mediatype markdown, put markdown in content and get rid of the source field.

Well it's tolerable (slightly because of Peertube also doing it) but as said before having HTML would be much better, specially as your usage of markdown is non-standard (as in not CommonMark).

[Nutomic]

Alright, then i will just fix the html conversion.

[dessalines]

Just a note, but if there's something we're doing that's not commonmark, then we need to open up an issue on are markdown parser, markdown-it, which should follow the commonmark spec.

[lanodan]

[2021-11-02 08:26:29-0700] Dessalines:

Just a note, but if there's something we're doing that's not commonmark, then we need to open up an issue on are markdown parser, `markdown-it`, which should follow the commonmark spec.

Well it might follow the spec for it's base but the subscript, superscript, spoiler and maybe few other features aren't part of common mark. And should be noted that even though CommonMark is a standard there is few adjustments which can be done on it, for example rendering line breaks as either `<br />` or `<br />\n`. (I got in this one because PleromaFE preserves newlines because of some legacy, even thought that's quite breaking how HTML should render)

[lanodan]

[2021-11-03 08:23:38-0700] Dessalines:

Closed #1871 via 1c6f74c.

Confirmed for all but the spoiler feature. ::: spoiler details content ::: Should render as: <details><summary>details</summary> content </details> While currently it is just not transformed.

[Nutomic]

That is because we use two different libraries for markdown rendering. One in javascript in the frontend, which supports spoiler tags, and another one in the backend in Rust which apparently doesnt.

So we should probably open an issue about it in the comrak repo. @dessalines Do you know if there is some documentation/specification for this spoiler syntax?

[dessalines]

The spoiler tag is the only thing that's not CommonMark, and its totally custom so I wouldn't expect anything else to support it. It's using markdown-it custom containers: https://github.com/markdown-it/markdown-it-container

[Nutomic]

I opened a separate issue about the spoiler tag #1884