API endpoint to get metadata of a brew

[DSPaul]

Your idea:

I am working on a library app that would allow users to import official RPG rulebooks and homebrew content from all over the web into one place where they can sort, filter, ect based on metadata like author, release date, ect. To get metadata for homebrewery documents, I currently scrape the https://homebrewery.naturalcrit.com/share/:id endpoint and get the metadata in json format with this little bit of C# code

//Select script tag with all metadata in JSON format
string script = src.SelectSingleNode("/html/body/script[2]").InnerText;
//json is encapsulated by "start_app() function, so cut that out
string rawData = script[10..^1];
JObject metadata = JObject.Parse(rawData);

This would break easily however if the client/template.js file were to be changed so having a new endpoint like https://homebrewery.naturalcrit.com/info/:id that returns all the metadata as json would eliminate this rather janky way of doing it. I'm sure other 3rd party projects could also benefit from this. If there is already a way to do this with the current API, please tell as it isn't really documented anywhere so I could have missed it.

[ericscheid]

You would better off with https://homebrewery.naturalcrit.com/download/:id .. that returns the raw source with no UI or fiddling (unlike /share/:id or even /source/:id)

[G-Ambatte]

I had a bit of a poke at the library app, it appears that it's just scraping the brew's metadata.

Homebrewery doesn't force updates to brews in storage, so older brews may not have every metadata option in the code-fenced metadata block in the brew text - for example, pageCount is a relatively new metadata feature, and may not exist on every brew. Similarly, the codefenced metadata block in the brew text (which is visible via the /download/:id endpoint) will not exist on older brews (that is, brews that have not been edited or updated since the change went live).

A better API endpoint would be for Homebrewery to implement a /metadata/:id endpoint which returns a JSON object with only the brew metadata - title, authors, version, pageCount, description, thumbnail image URL, publication status, and so on.

However, something to consider for the library app itself is that Homebrewery is completely open source and free, and anyone can run it locally on just about any OS (Windows, macOS, Ubuntu, Debian, FreeBSD, RaspBian, or anything that will run a Docker image), so there is no guarantee that a user's Homebrewery document will always exist at https://homebrewery.naturalcrit.com/share/:id.

[DSPaul]

Thanks a lot for the helping me with this and even checking out my code, I will check out the /download/:id and /source/:id endpoints, I did not know they were a thing. I might be worth it to write a quick little wiki entry with a list of the endpoints and a one sentence explanation of what they do because they seem to be quite scattered in the code so its easy to miss one.

It's also good that you told me that older brews might lack some data, I only tested it with a handful of documents so I never ran into problems but others might. I'll change my code to account for that. A /metadata/:id endpoint that always has all the metadata fields, even if some of them are empty, would be perfect.

As for homebrewery deployments that are not on the https://homebrewery.naturalcrit.com domain, would there be an easy way to verify if any given URL is a valid homebrewery deployment? I guess not because the source code of the self deployed version could have been changed so any check I would do might not hold true any more so I think I will keep the domain check by default for now so that the vast majority of users that use the official deployment will have the convenience of getting a warning if they make a typo or formatting mistake and add a checkbox that disables the domain check for self-hosted users that know what they are doing.

[5e-Cleric]

@DSPaul Is that project still being worked on? Can we get an update? Should we close this issue if its not going to be worked on?

[DSPaul]

Yes I am still actively working on the project, and still using the scraping method as I described above. It has proven reliable enough as it hasn't broken in the past year. I would still like to see this implemented so I wouldn't close the issue but given that there is a working workaround, you can treat this as very low priority.

[5e-Cleric]

As for homebrewery deployments that are not on the https://homebrewery.naturalcrit.com domain, would there be an easy way to verify if any given URL is a valid homebrewery deployment? I guess not because the source code of the self deployed version could have been changed so any check I would do might not hold true any more so I think I will keep the domain check by default for now so that the vast majority of users that use the official deployment will have the convenience of getting a warning if they make a typo or formatting mistake and add a checkbox that disables the domain check for self-hosted users that know what they are doing.

I am confused as to what do you want in relation to PR deployments, those are temporary domains to test stuff, why would you or any user of your app want to access them?

From what i gather, you want a /metadata/:id (or other name) endpoint which should return a JSON object with only the brew metadata - title, authors, version, pageCount, description, thumbnail image URL, publication status, and so on. Is that correct?

Also, thanks for the very fast reply.

[5e-Cleric]

Working example

Astonished as to how simple that turned out to be, i'm more the CSS guy

[DSPaul]

Awesome, that is exactly what I wanted! Thanks for implementing this. You can go ahead and merge the PR and close this issue as far als I'm concerned.

Ps.
You can forget what I said about self hosted instances of homebrewery, it was pretty far fetched and irrelevant, I was thinking at the time that some people might be hosting their own fork, mastodon style but that is clearly not the case, everyone just uses the official instance.

[5e-Cleric]

Sorry this took this long, could you share a link to the app? I'm interested.

[Gazook89]

Stealing their thunder: https://www.compassapp.info/

[5e-Cleric]

@DSPaul New api endpoint ready and live

[DSPaul]

Awesome, just updated my code to use it, works great

[5e-Cleric]

@DSPaul currently working on implementing a strict CORS policy to the project, so i will need your requests' origin so i can whitelist it.

[DSPaul]

Thanks for considering me, but because the requests are made directly by the desktop application, and not a server/website, I'm pretty sure the requests don't have an origin right now. I'll look into it this evening. Maybe I can manually add an origin to the request, and if not I could try to use my own api as a proxy. I'll keep you posted.

[ericscheid]

CORS (Cross-Origin Resource Sharing) is a browser-enforced security mechanism, and it specifically applies to requests made by browsers running scripts (e.g., JavaScript) within web pages. Here's how it applies across different contexts:

1. Browser Access — Applies: CORS primarily governs browser-based requests to protect users from malicious cross-origin interactions.

2. Standalone Applications — Does Not Apply: Applications like mobile or desktop apps (e.g., a React Native app, Postman, or a backend service) are not subject to CORS restrictions because they don’t rely on browsers.
   These apps can freely send HTTP requests to your API regardless of CORS settings.
   However, you can enforce additional security measures such as API keys, OAuth tokens, or IP whitelisting for these types of clients.

3. Command-Line Tools and Scripts — Does Not Apply: Tools like curl, wget, or custom scripts do not enforce CORS rules. They can directly interact with your API regardless of the CORS headers your server sends.

Why CORS is Browser-Specific

CORS is designed to mitigate risks in browser environments, such as:

• Cross-Site Request Forgery (CSRF):
  Prevents a malicious web page from sending unauthorized requests to your API using a user's credentials (e.g., cookies).
• Data Leakage:
  Ensures that responses are only visible to approved origins.

Standalone applications and scripts are not vulnerable to these browser-based security issues, which is why CORS is not enforced for them.

(per chatGPT, not me)

[5e-Cleric]

Alright, so yeah CORS does not apply to your app, that's one less problem.

[DSPaul]

Great, saves me work as well.

[5e-Cleric]

Would you mind providing some contact info, mail or whatever, to contact you in case we need? Should we contact you through disclord or reddit?

[DSPaul]

You can reach me on discord @paulds but sometimes discord doesn't notify me of message requests from users I haven't talk to before so if I don't respond, you can just hop in the discord server for my app and talk there, I'll see it. You can access the server by clicking the discord badge in the readme of the repo which is pinned on my github account.