Solr search implementation

[ujithaperera]

• All tests pass -- rake test:all
• code is in uniquely-named feature branch, and has been rebased on top of latest master (especially if you've been asked to make additional changes)
• pull request are descriptively named
• if possible, multiple commits squashed if they're smaller changes
• reviewed/confirmed/tested by another contributor or maintainer
• schema.rb.example has been updated if any database migrations were added

[ujithaperera]

This PR is created to represent #663 with latest rebase

[jywarren]

Oh, cool - so is this ready to be merged? I'm about to merge a bunch of
smaller PRs, but I can hold off if that's helpful.

On Fri, Sep 2, 2016 at 11:33 AM, Ujitha Perera notifications@github.com
wrote:

This PR is created to represent #663
#663 with latest rebase

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#781 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABfJ-H6rB34TJ2I4ekYFYP1VK_dR3sSks5qmEHjgaJpZM4Jz0wL
.

[ujithaperera]

Yes Jeff ! we can merge this PR. All the tests are passed in my local. have to run solr server in test environment to avoid solr connection errors. Instructions are in the README.md

[ujithaperera]

We have to add RAILS_ENV=test bundle exec rake sunspot:solr:start this line to the Cl script to run tests successfully, I think

[jywarren]

Ujitha - do you know which file to add that line to, or do you need to ask @icarito for guidance? It's either a Travis or Docker configuration, i think.

[ujithaperera]

we can add docker-compose run web RAILS_ENV=test bundle exec rake sunspot:solr:start line to .travis.yml.

As far as I know this should start solr in test env. Since solr is separate server running in different port in the server, perhaps docker may need additional configurations inside the server.
let's verify from @icarito

[jywarren]

Ujitha - try adding that and pushing here, to see if it passes TravisCI
tests. If not, we can just roll back that commit.

On Fri, Sep 2, 2016 at 12:01 PM, Ujitha Perera notifications@github.com
wrote:

we can add docker-compose run web RAILS_ENV=test bundle exec rake
sunspot:solr:start line to .travis.yml
https://github.com/publiclab/plots2/blob/master/.travis.yml#L13.

As far as I know this should start solr in test env. Since solr is
separate server running in different port in the server, perhaps docker may
need additional configurations inside the server.
let's verify from @icarito https://github.com/icarito

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#781 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABfJwEdz7pQTJVBS9_-nqiM-6sJsGLsks5qmEhUgaJpZM4Jz0wL
.

[jywarren]

That worked, but new error:

Error: test_update_wiki_selecting_previous_image(WikiControllerTest):
  RSolr::Error::Http: RSolr::Error::Http - 500 Internal Server Error
  Error:     {msg=SolrCore 'test' is not available due to init failure: Index locked for write for core 'test'. Solr now longer supports forceful 
unlocking via 'unlockOnStartup'. Please verify locks manually!,trace=org.apache.solr.common.SolrException: SolrCore 'test' is not available due to
 init failure: Index locked for write for core 'test'. Solr now longer supports forceful unlocking via 'unlockOnStartup'. Please verify locks manu
ally!
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:974)
        at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:250)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:417)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)

  URI: http://localhost:8981/solr/test/update?wt=ruby
  Request Headers: {"Content-Type"=>"text/xml"}
  Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><add><doc><field name=\"id\">DrupalNodeRevision 1027611808</field><field name=\"type\">DrupalNodeRevision</
field><field name=\"type\">ActiveRecord::Base</field><field name=\"class_name\">DrupalNodeRevision</field><field boost=\"5\" name=\"title_text\">New Title</field><fie
ld name=\"body_text\">Editing about Page</field><field name=\"teaser_text\"/></doc></add>"

  Backtrace: /home/warren/.rvm/gems/ruby-2.1.2/gems/rsolr-1.1.2/lib/rsolr/client.rb:288:in `adapt_response'
  /home/warren/.rvm/gems/ruby-2.1.2/gems/rsolr-1.1.2/lib/rsolr/client.rb:189:in `execute'
  /home/warren/.rvm/gems/ruby-2.1.2/gems/rsolr-1.1.2/lib/rsolr/client.rb:175:in `send_and_receive'
  /home/warren/.rvm/gems/ruby-2.1.2/gems/sunspot_rails-2.2.5/lib/sunspot/rails/solr_instrumentation.rb:16:in `block in send_and_receive_with_as_instrumentation'
  /home/warren/.rvm/gems/ruby-2.1.2/gems/activesupport-3.2.22.2/lib/active_support/notifications.rb:123:in `block in instrument'
  /home/warren/.rvm/gems/ruby-2.1.2/gems/activesupport-3.2.22.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  /home/warren/.rvm/gems/ruby-2.1.2/gems/activesupport-3.2.22.2/lib/active_support/notifications.rb:123:in `instrument'
  /home/warren/.rvm/gems/ruby-2.1.2/gems/sunspot_rails-2.2.5/lib/sunspot/rails/solr_instrumentation.rb:15:in `send_and_receive_with_as_instrumentation'
  (eval):2:in `post'
  /home/warren/.rvm/gems/ruby-2.1.2/gems/rsolr-1.1.2/lib/rsolr/client.rb:84:in `update'
  /home/warren/.rvm/gems/ruby-2.1.2/gems/rsolr-1.1.2/lib/rsolr/client.rb:104:in `add'
app/controllers/wiki_controller.rb:141:in `block in update'
app/controllers/wiki_controller.rb:140:in `update'
test/functional/wiki_controller_test.rb:132:in `block in <class:WikiControllerTest>'

[jywarren]

Hi, Ujitha - it looks like you have some viable next steps:

• put java deps into Dockerfile
• put RAILS_ENV=test bundle exec rake sunspot:solr:start into Dockerfile
• rearrange README steps
• investigate Solr error (maybe to do with a deprecated version?)

I think we can get the tests to pass if we work through this checklist. And since all of your commits are your own, with none mixed in from master, rebasing should be pretty clean and straightforward, so I'm going to go ahead and merge other PRs while you work on this. Just ensure your commits (and only yours) are cleanly rebased and you can work on this a bit without holding things up. Thanks!

[ujithaperera]

Yes. we can follow this plan. I'll take care of these issues.

[ujithaperera]

Hi @jywarren, I tried by putting command (sudo apt-get install -y openjdk-7-jre) for installing java in several places. Since I have very minor knowledge about docker and travis-ci, still figuring out the correct place to put this command.

Can you or @icarito check my last commits and guide me to solve this matter.

Thank you.

[jywarren]

I'm not sure myself, but it makes most sense to me to put it after # Install dependencies in the Dockerfile. Did that not work? Can you push
that up and we can look at the output?

Thanks!

On Sep 5, 2016 11:45 AM, "Ujitha Perera" notifications@github.com wrote:

Hi @jywarren https://github.com/jywarren, I tried by putting command (sudo
apt-get install -y openjdk-7-jre) for installing java in several places.
Since I have very minor knowledge about docker and travis-cl, still
figuring out the correct place to put this command.

Can you or @icarito https://github.com/icarito check my last commits
and guide me to solve this matter.

Thank you.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#781 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABfJ6Ng-WWzSRv9CirEgJmqtJG4MElgks5qnDkggaJpZM4Jz0wL
.

[icarito]

I'll have a look today at the dockerfile / travis conf.

Regards,

Sebastian

El 05/09/16 a las 10:45, Ujitha Perera escribió:

Hi @jywarren https://github.com/jywarren, I tried by putting command
(|sudo apt-get install -y openjdk-7-jre|) for installing java in
several places. Since I have very minor knowledge about docker and
travis-cl, still figuring out the correct place to put this command.

Can you or @icarito https://github.com/icarito check my last commits
and guide me to solve this matter.

Thank you.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#781 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMMS_q6SwZCXcH5EA1lwqe-PzmTuMFhks5qnDkhgaJpZM4Jz0wL.

[jywarren]

Hi, Ujitha - did you try pushing a version which puts the apt-get command in the Dockerfile where I suggested? I'd like to get this merged as soon as we can, and I think we're close!

[ujithaperera]

yes Jeff, you can find this change in b10dfd5

[icarito]

Hi,
Sorry I took so long. I reviewed your .travis.yml file and Dockerfile and, your dependencies should be installed inside the Docker container (defined in Dockerfile), not the test server (defined in .travis.yml).

I'll try it.

[icarito]

Hi,
I just tried it and it got further ahead:
https://travis-ci.org/icarito/plots2/builds/158369043

Successfully started Solr ...
(...)
rake aborted!
RSolr::Error::ConnectionRefused: Connection refused - {:data=>"<?xml version=\"1.0\" encoding=\"UTF-8\"?><add><doc><field name=\"id\">DrupalUsers 1</field><field name=\"type\">DrupalUsers</field><field name=\"type\">ActiveRecord::Base</field><field name=\"class_name\">DrupalUsers</field><field name=\"name_s\">admin</field><field name=\"mail_s\">admin@example.com</field><field name=\"status_s\">1</field></doc></add>", :headers=>{"Content-Type"=>"text/xml"}, :method=>:post, :params=>{:wt=>:ruby}, :query=>"wt=ruby", :path=>"update", :uri=>#<URI::HTTP:0x005622dfe292c8 URL:http://localhost:8981/solr/test/update?wt=ruby>, :open_timeout=>nil, :read_timeout=>nil, :retry_503=>nil, :retry_after_limit=>nil}
(eval):2:in `post'
/app/app/models/user.rb:63:in `create_drupal_user'
/app/db/seeds.rb:13:in `<top (required)>'
Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 8981
(eval):2:in `post'
/app/app/models/user.rb:63:in `create_drupal_user'
/app/db/seeds.rb:13:in `<top (required)>'
Tasks: TOP => db:setup => db:seed

I'm guessing Solr requires some configuration to work?

Note Docker runs everything within a container. You can install Docker and docker-compose if you like and manually follow the steps detailed in the .travis.yml file.

@jywarren It might be worthwhile to add a section to the README for building and using a container for development/testing, or comments to the Dockerfile / travis.yml file.

[icarito]

@ujithaperera I've issued a pull request to your pull request (I didn't know you could do that!).

Here's the changes I made for reference:
david-days@5ee1f58

The app still seems to not find Solr, but at least Solr finds Java now. ;-)

[jywarren]

I'm poking through this now... going to try to see where/why the solr server doesn't start. This line looks fine: https://github.com/publiclab/plots2/pull/781/files#diff-3254677a7917c6c01f55212f86c57fbfR14 but isn't it missing icedtea-7-plugin ?

[jywarren]

OK -- I've rebased this entire PR on top of the latest master, in #810, adding the icedtea line. We'll see how this goes.

[jywarren]

All merged in #810!