Pismo Redirection Not Allowed

[sidbatra]

Any fix planned for allowing redirects? thanks!

"redirection forbidden: http://www.bettiepageclothing.com -> https://www.bettiepageclothing.com/"

[Jeewes]

I would also appriciate allowing redirects. This guestion on stackoverflow seems to deal the same problem. Suppose open-uri is the cause of the problem.

[andyatkinson]

@peterc I think this has been a problem forever in open-uri. Do you have a recommendation for what to do here? I tried adding this gem that monkey patches open-uri and provides a allow_redirects option. I tried this out with a New York Times article and it followed their redirect, but hit their paywall, which means the NYT link isn't useful in our app, but it still feels like there should be a built-in way to allow redirects, at least "safe" ones. https://github.com/andyatkinson/pismo/commit/39da3f68380e8260a787562e4f3e21c752a2791d

[peterc]

Hmm, well they must be detecting it somehow, so I'm guessing it's probably through the User-Agent on the request? If so, spoofing that User-Agent with that from a regular browser could work.

Pismo essentially includes open-uri to act as a sort of fallback if you didn't obtain the data in any other way or with a better/proper library, so I don't think going too far in implementing a more powerful HTTP client is a good idea, when they already exist separately (such as Typhoeus or even net/http). That said, spoofing the User-Agent is most likely the quick/cheap fix.

[andyatkinson]

Ok. I'll mess around with the user-agent and try out a separate HTTP library. Thanks for the ideas!

[peterc]

If you find one that's pretty customizable and pure Ruby, let me know as it could become a dependency here. Any native/compiled stuff would be a no-no. Though to be fair, net/http could be used in a more advanced way here I guess..