capybara-webkit doesn't work on Heroku

[andrewhubbs]

Any thoughts on how to get this working on Heroku? What would it take to add a phantomjs driver? Pretty sure that will run on Heroku.

[benkitzelman]

Hi Andrew - thanks for contributing! There is a base class /lib/google_ajax_crawler/drivers/driver.rb ... really all that is needed is to implement the following methods using phantom

• visit(url)
• default_page_loaded_test
• html

The capybara-webkit driver does this by mixing in the Capybara::DSL....
You might be interested in using Poltergeist - a phantom capybara bridge.... It is probably better writing it as a separate gem, otherwise the dependencies for the crawler will involve the installation of two headless browsers (QT for capybara-webkit and Phantom for yours)... In time I might break the capybara one out into another repo as well

[ChristopheBelpaire]

Hello,
Ok, I made a branch with the poltergeist driver here : https://github.com/ChristopheBelpaire/google-ajax-crawler/tree/poltergeist_driver
But It think it could be better to have one geme per driver with dependencies to google-ajax-crawler (something like google-ajax-crawler-poltergeist gem and google-avjax-crawler-capybara-webkit) Seems a good idea ?

[andrewhubbs]

Thanks @benkitzelman for getting back to me. Sorry I have been very slow to respond.

I think @ChristopheBelpaire is right. We need to separate the driver gems from the core. I did a nasty hack job on my fork of ripping out Webkit and putting in Poltergeist. Not sure it is useful to salvage but I'd be happy to help split things.

[dobber1134]

@ChristopheBelpaire Thanks for the effort on the Poltergeist driver, we are having too many issues with Capybara as well.

I'm trying your branch, but in running the specs we get an error as lib/google_ajax_crawler/drivers/poltergeist is missing. Is the code complete or is there something I can do?

[ChristopheBelpaire]

oops forgot to git add . ;)
Should be ok now

[benkitzelman]

yeah definitely a +1 on breaking out a separate gem - when its ready add a url to the crawler's readme pointing to the repo, and the name of the published rubygem :)

[ChristopheBelpaire]

Ok I'll do that, but maybe another gem for the capaybara-webkit driver should be created also, otherwise if I create a gem for phantom-js driver with dependencies to google-ajax-crawler I'll have dependencies on capybara-webkit (and webkit qt etc ...) and also phantom js.

[ChristopheBelpaire]

Ok I have first alpha version (witch seems to work for my app) here : https://github.com/ChristopheBelpaire/google_ajax_crawler_phantomjs
It has a dependency on your gem, so on capybara-webkit. We should have the same kind of gem driver for capybara webkit.

[vitorbaptista]

@benkitzelman Are you still planning to break out the plugins? I'm trying to install @ChristopheBelpaire's plugin in Heroku, but I can't because it depends on your project, which depends on Qt...

[ChristopheBelpaire]

@vitorbaptista if you want, you can try this branch : https://github.com/ChristopheBelpaire/google-ajax-crawler/tree/poltergeist_driver witch use poltergeist instead of capybara webkit (without the dependencies).

[vitorbaptista]

Cool, I'll try that later. Thanks :-)

[aminariana]

I'm having a lot of trouble installing even the main gem. My hope was to use the poltergeist_driver branch, but there may be some issues I'm having with the Ruby version (mine is Ruby 2) so I can't even get it to install properly.