Custom tokenizer/renderer for @mention and #hashtags?

[ysk3a]

Hello,
I thought I try asking this community for some help as I think my problem may be more closer to how to use marked.
I am using a svelte library https://github.com/pablo-abc/svelte-markdown#props which uses marked and I am trying to use the tokenizer(?) to catch markdown text that has @mention and #hashtags and render a custom component (so i can add listeners and other complex logic).

I believe what I am trying to do is similar to #2888, #2817, #2543, #2532 ?

My progress so far:

• No extra customization i.e. Just Markdown version just shows markdown. no problem when just want simple md.
• Svelte Marked Renderer on Text uses the renderer along with my SpanTag.svelte component following https://github.com/pablo-abc/svelte-markdown#renderers but this attempt will style the whole paragraph or url (if it has @ or # in it) instead of just mentions and tags
• The Tokenizer version is where I try the tokenizer method (Markdown.svelte line 5) but when rendering, the @ and # are just being repeated or overwriting the other text. I think this is where i'm struggle with the lack of understanding of how to Marked?
• With the custom tokenizer from the third point, I thought I would be able to pass to https://github.com/pablo-abc/svelte-markdown#renderers and use it like <SvelteMarkdown {source} renderers={{ tagtext: SpanTag }} />

Stackblitz: (maybe a stackblitz issue or hmr but sometimes manual refresh is needed for changes to apply)
https://stackblitz.com/edit/vitejs-vite-cbg7vh?file=src%2Flib%2FMarkdown.svelte

End goal:
have an editor/textarea just like githubs textarea and render user input as comments/messages but that should fall into place once the above is resolved 🤔

[UziTech]

There are two types of extension:

1. Extensions that change existing tokenizers/renderers
2. Extensions that add new tokenizers/renderers

The Tokenizer in marked options is for the extensions that change existing tokenizers. To add new tokenizers you will need to use marked.use to add an extension.

It doesn't look like SvelteMarkdown allows the type of extension that adds new tokenizers/renderers.

I would suggest adding a feature request for SvelteMarkdown to be able to send extensions to marked.use.

[ysk3a]

Thanks for the reply.
I'll see around SvelteMarkdown issues if there was already some request made.

But I was wondering if anyone could shine some light on how to properly the custom tokenizer in general?
I think it maybe some mistake in the regex? the regex seems to catch it when testing the regex https://regex101.com/r/eL6cL8/1 i.e. finds words containing @ but below, the custom tokenizer seems to always start from beginning of the sentence instead of beginning of the @.

	const tagtext = {
		name: 'tagtext',
		level: 'inline', // Is this a block-level or inline-level tokenizer?
		start(src) { return src.indexOf('@'); },   
		tokenizer(src: string, tokens: any) {
		const rule = /(^@[\w_-]+)/;
    	       const match = rule.exec(src);
			if (match) {
				console.log('::tagtext', match);
				return {
					type: 'tagtext', // Should match "name" above
					raw: match[0] // Text to consume from the source
				};
			}
			return false;
		},
		renderer(token: any) {
			console.log('::tagtext renderer', token); //marked.parseInline(token)
			return `<span class="tag-item">::${token.raw}</span>`; // Return the final HTML for this token
		}
	};

or
line 15 in https://stackblitz.com/edit/vitejs-vite-jkregk?file=src%2Flib%2FMarkdown.svelte

[UziTech]

I don't know what svelte is doing but this seems to work without it https://runkit.com/uzitech/marked-tagtext

[ysk3a]

yeah the part i was asking about wasn't anything to do with svelte. that tokenizer snippet seemed to be working.
after coming back later to the above stackblitz link, it seemed to work.
I also try out in a blank ts project, https://stackblitz.com/edit/typescript-ehcvvz?file=index.ts
and that seemed to show that it was working.
and your sample runkit also shows it is working.

i am super 😕 and wondering if i was seeing things 🤣 but i feel like i had that tokenizer code way before and didn't work. 🤷
idk why it wasn't working last time. maybe it was stackblitz's svelte webcontainer template project or had to hard refresh browser.