Replies: 5 comments 2 replies
-
Hey this is awesome! I didn't even think of checking if there's a SaaS for scraping around rate limits. I just use mobile API that doesn't have rate limiting (well it didn't back in January). Looking how USPSA and SCSA mobile apps are very similar, maybe you can use their mobile api too and save money on the scraper (don't know if it costs you anything). Let me know if you need help with that. Link you posted didn't open for me for some reason. |
Beta Was this translation helpful? Give feedback.
-
Could you paste a sample mobile api link ? I wish they had a swagger doc, but from what I see they ( USPSA / SCSA ) don't really have genuine api's ( maybe you can prove me wrong ). |
Beta Was this translation helpful? Give feedback.
-
OK, this is getting good ! Here is my sample on HTTP Req ( I feed into this 17k urls which I retrieve from AWS RDS ) client_key = "xxx" def make_request_with_retry(url, retries=9, backoff_factor=1.9, timeout=50):
|
Beta Was this translation helpful? Give feedback.
-
Hey it looks like you are using Zenrows; how do you like it ? Have you taken any different directions as a result of SaaS scraping ? |
Beta Was this translation helpful? Give feedback.
-
Congrats on the launch of your site. your UI work is outstanding. I having been trying various scraping methods with and without zenrows. |
Beta Was this translation helpful? Give feedback.
-
so I do something very similar here: http://www.steelrankings.com/
I too ran into the http429 issue. Cloudflare is checking IP; I ended up using zenrows.com. I process 17000 USPSA numbers for steel challenge classification rankings 4x / week. I use 25 concurrent threads and it averages about .07 per request; Happy to have more discussion. I did take a look at your data files and happy to help if you wanted to rebuild them on-demand.
Beta Was this translation helpful? Give feedback.
All reactions