Netlify URLs asking for password

[Cartmanishere]

https://clojuredocs-edn.netlify.app/export.compact.edn

https://clojuredocs-edn.netlify.app/export.edn

These URLs are asking for password.
This breaks thing in cider-nrepl clojuredocs middleware when it tries to refresh the cache.

Has anything changed in the netlify deployment?

[bbatsov]

I've enabled this password, as someone started to fetch data from the site every second and consumed all the site's bandwidth + an additonal data package. I'm not willing to pay for so much data usage for something so trivial, so I'll likely kill the site and drop the daily exports functionality for the time being.

I know this sucks, but it's also insane that someone would be downloading 500gb of ClojureDocs data a month. Clearly that's some malicious usage of the site and it's not something I can easily restrict either. The damn script has generated 10 gigs of bandwidth just by re-rendering constantly the 301 page....

[bbatsov]

@liquidz Let me know if you have any ideas on the subject - I was thinking of either of maybe just serving the exports from the Git repo, as we won't have to deal with bandwidth limits this way.

[coder11]

Hi Bozhidar!

Is it possible to make CIDER to load this file from the local filesystem rather then Netlify as a workaround?
It looks like the script is pretty straightforward and it would be easy to perform the conversion by hand!

[bbatsov]

Everything is possible. :-) I liked the idea of the central place with a fresh snapshot, as this was the most convenient experience from a user standpoint. Btw, I have to say I'm surprised someone uses the refresh functionality, as the data in ClojureDocs doesn't change that often anyways.

[coder11]

Well, I'm just using spacemacs with the default CIDER setup. Autocompletion and doc functionality stopped working for me and I saw lots of 403 errors in repl. So here is how I got here in the first place :)

[bbatsov]

@coder11 What's your CIDER version? Normally CIDER won't interact with this site at all unless you request a manual update of the ClojureDocs cache, as Orchard bundles a copy of the export.

[coder11]

@bbatsov

(cider-version) gives me this: CIDER 0.22.0snapshot (package: 20190829.613) Seems like it's very outdated isn't it?

[bbatsov]

@coder11 Yeah, indeed. The old versions used to fetch this data when starting, which I assume is the problem you encountered. You should update to 1.0.

[bbatsov]

Seems it's going to be easy to trigger a daily job with GHA (https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#schedule) and have it commit the fresh exports to the repo.

[bbatsov]

I've moved the exports under the repo, for whoever needs them. I'll get to automating the export process down the road, unless @liquidz beats me to it. I don't care that much about having fresh data, so it's not a big priority for me and I've very short on time these days.

I'll kill the old site completely as well. All the useful information from it is now in the README.

[liquidz]

@bbatsov I've tried automatic commit using GitHub Actions in the following repositories, so I will automate the update on the repository.
https://github.com/liquidz/liquidz/blob/master/.github/workflows/build.yml

[bbatsov]

@liquidz Thanks! Much appreciated! 🙇‍♂️

[Cartmanishere]

Btw, I have to say I'm surprised someone uses the refresh functionality, as the data in ClojureDocs doesn't change that often anyways.

I don't think anyone uses this as their workflow. I had a colleague of mine for whom the CIDER started breaking with errors on nrepl due to this. So I reported it. I don't think this causes any issues with newer version of CIDER and middlewares.

An upgrade should resolve the issue for anyone facing same thing. A different workaround to stop middleware from doing auto refresh, I did touch <cache-file> (located at ~/Library/Caches/orchard/clojuredocs/export.compact.edn on Macos) and that also resolved the problem for time being.

Thank you @bbatsov and @liquidz for taking care of it! 🙇