feat: make the open graph tags dynamic based on the `base64` and `url` query parameters #1122

helios2003 · 2024-07-09T17:18:46Z

This PR addresses the issue mentioned in #224. I have added the following features in this PR.

The studio version I am testing on is hosted here: https://studio-studio-next.vercel.app. (I will switch this to Netlify deployment once we are able to resolve #1118)
The Image Generator service in charge of creating the dynamic image is hosted here: https://ogp-studio.netlify.app
The code for the image Generator service can be found here: https://github.com/helios2003/ogp-studio

Features Added

Add a middleware that captures Open Graph crawler requests when they hit the website and rewrites the URL to the route api/crawler. This is done to serve minimal HTML when the crawler hits the page, reducing the chance of timeout.
Parse the URL and send it to the Image Generator service to add the dynamic image.
Show the default og:image if there is an invalid base64 string provided.
Fall back on the usual behavior added in #1106 if there isn't a base64 string.

Todos

Address unusual behavior observed like Slack showing the default image even when the document is valid.
- For example: if we go to https://studio-studio-next.vercel.app and use MQTT template, then the default behavior is observed instead of dynamic tags.
- Sometimes, Slack crawlers arent able to detect the open graph image.
Figuring out why sometimes the dynamic og:image does not appear on site like Whatsapp.
Add additonal crawlers to the list.

cc : @smoya

changeset-bot · 2024-07-09T17:18:49Z

⚠️ No Changeset found

Latest commit: 19858da

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

netlify · 2024-07-09T17:19:02Z

✅ Deploy Preview for modest-rosalind-098b67 ready!

Name	Link
🔨 Latest commit	`19858da`
🔍 Latest deploy log	https://app.netlify.com/sites/modest-rosalind-098b67/deploys/66e98efe014adb0008b47576
😎 Deploy Preview	https://deploy-preview-1122--modest-rosalind-098b67.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify · 2024-07-09T17:19:03Z

✅ Deploy Preview for asyncapi-studio-design-system ready!

Name	Link
🔨 Latest commit	`19858da`
🔍 Latest deploy log	https://app.netlify.com/sites/asyncapi-studio-design-system/deploys/66e98efebd2526000885dfba
😎 Deploy Preview	https://deploy-preview-1122--asyncapi-studio-design-system.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify · 2024-07-09T17:19:03Z

✅ Deploy Preview for studio-next ready!

Name	Link
🔨 Latest commit	`19858da`
🔍 Latest deploy log	https://app.netlify.com/sites/studio-next/deploys/66e98efe0094a20008517e8b
😎 Deploy Preview	https://deploy-preview-1122--studio-next.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

KhudaDad414

left some comments.

KhudaDad414 · 2024-07-15T13:43:31Z

apps/studio-next/next.config.js

+  async rewrites() {
+    return [
+      {
+        source: '/:base64',


I think we can support URLs here as well, I mean when a user shares a file with url parameter.

Yes, we should support both base64 and url use cases as stated in #224

Yes sure, I will add it.

apps/studio-next/src/app/api/crawler/route.tsx

KhudaDad414 · 2024-07-15T13:48:09Z

apps/studio-next/src/helpers/parser.ts

+import { Parser } from '@asyncapi/parser';
+import { DocumentInfo } from '@/types';
+
+export default async function parseURL(base64Document: string): Promise<DocumentInfo | null> {


We also have this parsing logic to parse the AsyncAPI file from base65 or a url in the studio as well, maybe we can re-use this functionality in both places.

So do you recommend keeping the studio version, or using this code?

They both serve the same purpose, so I guess you can look at both of them and combine them into a util function.

KhudaDad414 · 2024-07-15T14:05:09Z

apps/studio-next/src/middleware.ts

The configuration and this middleware achieve the purpose, right? I don't understand why we are adding both.

helios2003 · 2024-07-15T19:47:39Z

Thank you for the review @KhudaDad414. I will add the suggested changes.

helios2003 · 2024-07-15T19:57:08Z

Also, I had a few questions:

Is it okay if I hardcode the popular social media site crawlers as I have done now or is it better if we use a library like https://github.com/monperrus/crawler-user-agents as suggested to me by @smoya. The main problem I could think of is that by including all the potential sets of crawlers as mentioned in the repository we could be causing rewrites for nonopen graph crawlers as well which is not required. But @smoya mentioned that he does not expect other bots to use the query parameters like that of open graph crawlers.
Also, could you please let me know your thoughts on https://github.com/helios2003/ogp-studio and its code?

smoya · 2024-07-17T06:39:28Z

Is it okay if I hardcode the popular social media site crawlers as I have done now or is it better if we use a library like https://github.com/monperrus/crawler-user-agents as suggested to me by @smoya. The main problem I could think of is that by including all the potential sets of crawlers as mentioned in the repository we could be causing rewrites for nonopen graph crawlers as well which is not required.

If you don't use a complete list of crawlers, you will end up facing issues like the one you mentioned:

Figuring out why sometimes the dynamic og:image does not appear on site like Whatsapp.

This is happening most probably because you are not considering Whatsapp User-Agent(s).

But @smoya mentioned that he does not expect other bots to use the query parameters like that of open graph crawlers.

AFAIK, indexer robots such as Google only follow links, meaning that unless there is a published link somewhere to a link such as https://studio.asyncapi.com/?base64=<base64_doc> or the variant with url query param, the parsing won't happen. In fact, it does make sense to do the parsing and return the proper title, description, etc to the results of such indexer bot request so it appears in the results (like google results).

smoya · 2024-07-17T06:41:04Z

Also, could you please let me know your thoughts on https://github.com/helios2003/ogp-studio and its code?

Will that work in a Netlify setup? I mentioned an alternative in case it doesn't in #224.

Additionally, I think we can include more statistics in the rendered image like # of operations

helios2003 · 2024-07-17T06:50:08Z

Will that work in a Netlify setup? I mentioned an alternative in case it doesn't in #224.

Yes, it works. The service is currently hosted on Netlify: https://ogp-studio.netlify.app.

Additionally, I think we can include more statistics in the rendered image like # of operations

Okay, I will add it.

helios2003 · 2024-07-25T10:57:32Z

Added modifications:

Using a package for detecting crawlers.
Using sharp to optimize the image generation as mentioned here: https://nextjs.org/docs/messages/install-sharp.
Add the dynamic behavior for url query param as well.

smoya · 2024-07-29T10:35:17Z

apps/studio-next/src/app/api/crawler/route.tsx

+        <meta property="og:title" content="AsyncAPI Studio" />
+        <meta property="og:description" content="Studio for AsyncAPI specification, where you can validate, view preview documentation, and generate templates from AsyncAPI document." />
+        <meta property="og:url" content="https://studio-next.netlify.app" />
+        <meta property="og:image" content="https://raw.githubusercontent.com/asyncapi/studio/master/apps/studio-next/public/img/meta-studio-og-image.jpeg" />


Can those values be grabbed from

studio/apps/studio-next/src/app/page.tsx

Line 12 in a428b15

url: 'https://studio-next.netlify.app',

?

Done, @smoya

KhudaDad414 · 2024-07-30T09:17:12Z

apps/studio-next/src/middleware.ts

+export const config = {
+  matcher: ['/:base64', '/:url'],
+}


Is this matcher configuration correct? I feel like it is matching to path parameters instead of query parameters that we use for url and base64.

Oh yes, you are right, I didn't interpret the documentation correctly, I will make the changes.

smoya · 2024-07-30T09:44:04Z

apps/studio-next/src/app/api/crawler/route.tsx

+      const websiteTitle = "AsyncAPI Studio";
+      const websiteDescription = "Studio for AsyncAPI specification, where you can validate, view preview documentation, and generate templates from AsyncAPI document.";
+      const websiteUrl = "https://studio-next.netlify.app";
+      const ogImage = "https://raw.githubusercontent.com/asyncapi/studio/master/apps/studio-next/public/img/meta-studio-og-image.jpeg";


As I commented previously, those values are already present in the source code in

studio/apps/studio-next/src/app/page.tsx

Lines 4 to 25 in a8c8cb5

import ogImage from '@/img/meta-studio-og-image.jpeg';

export const metadata: Metadata = {

metadataBase: new URL('https://studio-next.netlify.app'),

openGraph: {

type: 'website',

title: 'AsyncAPI Studio',

description: 'Studio for AsyncAPI specification, where you can validate, view preview documentation, and generate templates from AsyncAPI document.',

url: 'https://studio-next.netlify.app',

images: [

{

url: ogImage.src,

width: 800,

height: 600,

alt: 'AsyncAPI default image',

},

]

},

twitter: {

site: '@AsyncAPISpec',

}

}

Please reuse them somehow instead of hardcoding them here again.

helios2003 · 2024-08-08T21:29:11Z

The PR is ready for review @KhudaDad414, @smoya.

helios2003 · 2024-08-09T03:38:52Z

Also, I'm unsure if the middleware code I wrote is the best way to achieve this functionality. Any suggestions or improvements would be appreciated.

…into fix/dynamic-og

smoya · 2024-08-25T06:49:12Z

apps/studio-next/src/app/api/crawler/route.tsx

+      </head>
+      </html>
+    `;
+    console.log(crawlerInfo);


Suggested change

console.log(crawlerInfo);

smoya · 2024-08-25T06:51:51Z

@helios2003 Are you planning to move the OG Image generator service (Netlify Edge Function) into this PR?

merge the tests with the original PR

helios2003 · 2024-09-13T08:34:32Z

Maintainers, can you please review the PR? I have addressed all the comments above and included the unit tests. Additionally, I had to change the version of @asyncapi/converter after this commit: asyncapi/converter-js@b3592ef.
I did not include these changes in the studio repository since we are soon switching to studio-next as the default version for studio based on this issue: #1134

Amzani · 2024-09-13T08:59:32Z

Some tests are still failing @helios2003

helios2003 · 2024-09-13T09:13:14Z

@Amzani the tests are failing in studio which I did not update as mentioned above.

Shurtu-gal · 2024-09-13T11:18:03Z

I suppose this can me merged as soon as the merging of studio and studio-next is done.

KhudaDad414

tests are faliling since you are installing a newer version of jest-environment-jsdom on studio-next app explicitly.

to resolve install the older version in studio app.
pnpm install -D jest-environment-jsdom@27.5.1

sonarcloud · 2024-09-17T14:16:05Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

helios2003 · 2024-09-17T14:38:30Z

Maintainers, the tests are passing now. PTAL at the PR.

helios2003 requested review from magicmatatjahu, KhudaDad414, Amzani and asyncapi-bot-eve as code owners July 9, 2024 17:18

KhudaDad414 requested changes Jul 15, 2024

View reviewed changes

helios2003 changed the title ~~feat: make the open graph tags dynamic based on the base64 url~~ feat: make the open graph tags dynamic based on the base64 and url query parameters Jul 25, 2024

helios2003 requested review from KhudaDad414 and smoya July 26, 2024 07:38

smoya reviewed Jul 29, 2024

View reviewed changes

KhudaDad414 reviewed Jul 30, 2024

View reviewed changes

smoya reviewed Jul 30, 2024

View reviewed changes

refactor

730df15

helios2003 force-pushed the fix/dynamic-og branch from fd2de17 to 730df15 Compare August 8, 2024 21:08

helios2003 and others added 3 commits August 14, 2024 21:24

change the route

fdc4295

Merge branch 'master' into fix/dynamic-og

9c9b3fd

add tests

2d3c6e0

helios2003 mentioned this pull request Aug 20, 2024

chore: add tests for dynamic open graph tags #1130

Closed

Merge branch 'master' into testing

e319a47

helios2003 added 4 commits August 24, 2024 13:33

testing the breaking points

4cd0d6f

Merge branch 'fix/dynamic-og' of https://github.com/helios2003/studio …

7eb005a

…into fix/dynamic-og

add logging for testing purposes

1f718ba

changes in parser

1367312

smoya reviewed Aug 25, 2024

View reviewed changes

helios2003 added 11 commits August 30, 2024 18:29

fix certain edge cases

fc7da7e

minor fix

f6c3113

add vercel for image generator service

a4b3581

set up configuration for tests

ce5d906

add og tests

67d5960

change lock file

032a980

configure open graph tags to work with normal tests

37f5b2d

Merge branch 'fix/dynamic-og' into testing

2efc878

Merge pull request #5 from helios2003/testing

945a0b8

merge the tests with the original PR

change the imports based on the new converter version

16eea66

change the website URL

b7da6cc

KhudaDad414 reviewed Sep 15, 2024

View reviewed changes

helios2003 added 3 commits September 17, 2024 19:04

change jest-env version

1cdb9b8

update

1363905

Merge branch 'master' into fix/dynamic-og

19858da

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: make the open graph tags dynamic based on the `base64` and `url` query parameters #1122

feat: make the open graph tags dynamic based on the `base64` and `url` query parameters #1122

helios2003 commented Jul 9, 2024 •

edited

Loading

changeset-bot bot commented Jul 9, 2024 •

edited

Loading

netlify bot commented Jul 9, 2024 •

edited

Loading

netlify bot commented Jul 9, 2024 •

edited

Loading

netlify bot commented Jul 9, 2024 •

edited

Loading

KhudaDad414 left a comment

KhudaDad414 Jul 15, 2024

smoya Jul 17, 2024

helios2003 Jul 17, 2024

KhudaDad414 Jul 15, 2024

helios2003 Jul 16, 2024

KhudaDad414 Jul 17, 2024

KhudaDad414 Jul 15, 2024

helios2003 commented Jul 15, 2024

helios2003 commented Jul 15, 2024

smoya commented Jul 17, 2024

smoya commented Jul 17, 2024 •

edited

Loading

helios2003 commented Jul 17, 2024

helios2003 commented Jul 25, 2024

smoya Jul 29, 2024

helios2003 Jul 30, 2024

KhudaDad414 Jul 30, 2024

helios2003 Jul 30, 2024

smoya Jul 30, 2024

helios2003 commented Aug 8, 2024

helios2003 commented Aug 9, 2024

smoya Aug 25, 2024

smoya commented Aug 25, 2024

helios2003 commented Sep 13, 2024

Amzani commented Sep 13, 2024

helios2003 commented Sep 13, 2024

Shurtu-gal commented Sep 13, 2024

KhudaDad414 left a comment

sonarcloud bot commented Sep 17, 2024

helios2003 commented Sep 17, 2024

	import ogImage from '@/img/meta-studio-og-image.jpeg';

	export const metadata: Metadata = {
	metadataBase: new URL('https://studio-next.netlify.app'),
	openGraph: {
	type: 'website',
	title: 'AsyncAPI Studio',
	description: 'Studio for AsyncAPI specification, where you can validate, view preview documentation, and generate templates from AsyncAPI document.',
	url: 'https://studio-next.netlify.app',
	images: [
	{
	url: ogImage.src,
	width: 800,
	height: 600,
	alt: 'AsyncAPI default image',
	},
	]
	},
	twitter: {
	site: '@AsyncAPISpec',
	}
	}

feat: make the open graph tags dynamic based on the base64 and url query parameters #1122

Are you sure you want to change the base?

feat: make the open graph tags dynamic based on the base64 and url query parameters #1122

Conversation

helios2003 commented Jul 9, 2024 • edited Loading

Features Added

Todos

changeset-bot bot commented Jul 9, 2024 • edited Loading

⚠️ No Changeset found

netlify bot commented Jul 9, 2024 • edited Loading

✅ Deploy Preview for modest-rosalind-098b67 ready!

netlify bot commented Jul 9, 2024 • edited Loading

✅ Deploy Preview for asyncapi-studio-design-system ready!

netlify bot commented Jul 9, 2024 • edited Loading

✅ Deploy Preview for studio-next ready!

KhudaDad414 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helios2003 commented Jul 15, 2024

helios2003 commented Jul 15, 2024

smoya commented Jul 17, 2024

smoya commented Jul 17, 2024 • edited Loading

helios2003 commented Jul 17, 2024

helios2003 commented Jul 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helios2003 commented Aug 8, 2024

helios2003 commented Aug 9, 2024

Choose a reason for hiding this comment

smoya commented Aug 25, 2024

helios2003 commented Sep 13, 2024

Amzani commented Sep 13, 2024

helios2003 commented Sep 13, 2024

Shurtu-gal commented Sep 13, 2024

KhudaDad414 left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Sep 17, 2024

Quality Gate passed

helios2003 commented Sep 17, 2024

feat: make the open graph tags dynamic based on the `base64` and `url` query parameters #1122

feat: make the open graph tags dynamic based on the `base64` and `url` query parameters #1122

helios2003 commented Jul 9, 2024 •

edited

Loading

changeset-bot bot commented Jul 9, 2024 •

edited

Loading

netlify bot commented Jul 9, 2024 •

edited

Loading

netlify bot commented Jul 9, 2024 •

edited

Loading

netlify bot commented Jul 9, 2024 •

edited

Loading

smoya commented Jul 17, 2024 •

edited

Loading