Solr

[jywarren]

Based on #781 781

• All tests pass -- rake test:all
• code is in uniquely-named feature branch, and has been rebased on top of latest master (especially if you've been asked to make additional changes)
• pull request are descriptively named
• if possible, multiple commits squashed if they're smaller changes
• reviewed/confirmed/tested by another contributor or maintainer
• schema.rb.example has been updated if any database migrations were added

[jywarren]

Oops, mistyped "icedtea-7-plugin". Trying again.

[jywarren]

One more time with RAILS_ENV=test rake sunspot:solr:start in .travis.yml

[jywarren]

Making:

env:
- RAILS_ENV=test

line in .travis.yml instead of prefixing the command with RAILS_ENV=test

[jywarren]

It's been running 22 minutes!

[jywarren]

Ok, well we probably have to turn off solr for tests at least if it's going to take that long... Resources:

https://github.com/collectiveidea/sunspot_test/blob/master/README.md

http://www.subelsky.com/2007_10_01_archive.html?m=1

https://github.com/dcrec1/acts_as_solr_reloaded/blob/master/README.markdown

[jywarren]

I'm trying using rake sunspot:solr:start RAILS_ENV=test instead of a default var in an env setting in .travis.yml.

[jywarren]

Hmm, so by seeding the database, we seem to need to start up Solr for development as well as for test. Do we really need to run rake db:setup and rake db:migrate if we're just running tests @icarito? Shouldn't the test db be able to install and run without the development db?

[jywarren]

I pushed ddb14b4 which removes rake db:setup and rake db:migrate to see what happens.

[jywarren]

Ok I get error You have 47 pending migrations

Taking a break!

[icarito]

Do we really need to run rake db:setup and rake db:migrate if we're just running tests @icarito?

I assumed so because otherwise you won't have a fully working environment. There is no "testing" database, it is setup on the fly for testing.

[jywarren]

Hmm, I keep getting Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 8981 on rake db:setup -- but I've tried starting solr (and it says successful) in both development and test modes:

- docker-compose run web rake sunspot:solr:start RAILS_ENV=test
- docker-compose run web rake sunspot:solr:start

and it says Successfully started Solr:

https://travis-ci.org/publiclab/plots2/builds/158977363#L2297

Any ideas here?

[jywarren]

Port 8981 is the test environment, as specified in config/sunspot.yml. @ujithaperera -- any ideas here? If you do make a fix, please create it as a commit based off this PR, rather than your own, so that we can preserve the rebase.

[jywarren]

This indicates that we might need to insert a "try" check to ensure Solr has had a chance to start up properly:

http://stackoverflow.com/questions/7325885/connection-rejected-from-solr-in-rspec#7327654

It's for rspec, but we could likely add a similar "rescue" for our own tests...

before :all do
  `sunspot-solr start`
  begin
    Sunspot.remove_all!
  rescue Errno::ECONNREFUSED
    sleep 1 && retry
  end
end

[jywarren]

Or perhaps we really do need to install https://github.com/collectiveidea/sunspot_test to automatically manage solr startup for tests.

[icarito]

You can recreate the testing environment if you install docker and docker-compose in your environment.

Those are the only requirements for running the tests locally and interacting with the testing environment interactively and directly.

The commands for setting up the testing environment (which creates a Debian 8 container complete with dependencies), are detailed in the .travis.yml file. (Starting with docker-compose build).

You can enter the container with docker-compose run web /bin/bash and treat it like a server, locally.

[jywarren]

Sebastian, do you happen to know if the install steps are run asynchronously, or if it really is returning Successfully started Solr before we're seeing the Connection refused error?

[icarito]

It really is returning success. It might be that Solr is not ready to respond to clients yet. A similar issue happens with Mysql, that's why I had to introduce the sleep 5 step in .travis.yml. Maybe try introducing a long sleep (try 20 or 30s first) between Solr startup and attempts at using it.

Another, cleaner option would be to use a "wait-for-port-to-open" script, such as this one https://github.com/vishnubob/wait-for-it

[jywarren]

I tried sleep 25 to no avail. These are seconds, not ms, right?

[icarito]

Yup. I suggest building the container locally and experimenting directly
with it.

El 10/09/16 a las 12:03, Jeffrey Warren escribió:

I tried sleep 25 to no avail. These are seconds, not ms, right?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#810 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMMS_pciBdOivQqLktxJE4ULk_JxcE6ks5qouLjgaJpZM4J5RB0.

[ujithaperera]

Since this is a docker container based issue, Couldn't we use an image of solr or sunspot_solr from docker hub. after this PR, solr is going to stay in the system for every build that we're making. right ?
So just an idea, can we go for this kind of solution. @icarito what do think about this. Actually I didn't check for practical difficulties here.

[jywarren]

Well, I think solr is actually running ok now. I think it's either a timing
issue or perhaps a port number issue?

On Sep 10, 2016 1:27 PM, "Ujitha Perera" notifications@github.com wrote:

Since this is a docker container based issue, Couldn't we use an image of
solr or sunspot_solr from docker hub. after this PR solr is going to stay
in the system for every build that we making. right ?
So just an idea, can we go for this kind of solution. @icarito
https://github.com/icarito what do think about this. Actually I didn't
check for practical difficulties here.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#810 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABfJ6B9p28HHkVFQBsuOj-gMGnWReRTks5qouh8gaJpZM4J5RB0
.

[jywarren]

Like, you could wget to the different ports and echo the response codes in
a new commit, but setting up local docker may really be easier.

On Sep 10, 2016 2:22 PM, "Jeffrey Warren" jeff@unterbahn.com wrote:

Well, I think solr is actually running ok now. I think it's either a
timing issue or perhaps a port number issue?

On Sep 10, 2016 1:27 PM, "Ujitha Perera" notifications@github.com wrote:

Since this is a docker container based issue, Couldn't we use an image of
solr or sunspot_solr from docker hub. after this PR solr is going to stay
in the system for every build that we making. right ?
So just an idea, can we go for this kind of solution. @icarito
https://github.com/icarito what do think about this. Actually I didn't
check for practical difficulties here.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#810 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABfJ6B9p28HHkVFQBsuOj-gMGnWReRTks5qouh8gaJpZM4J5RB0
.

[icarito]

@ujithaperera Yes, using a prebuilt image from docker hub would be great. It would need to be defined in docker-compose.yml. My recommendation is, install Docker and Docker Compose locally and try it until it works, then setup travis.yml to replicate.

[jywarren]

I'd do this myself, but I am running in Ubuntu on a Chromebook, and docker is not supported. @ujithaperera -- I'm going to try running:

wget http://localhost:8982/solr/ && less index.html at various points -- it took my chromebook 13 seconds to return an admin console at that address, even after it returned successful. So maybe we just need more waiting.

[jywarren]

Still trying different variations; i'll probably squash all these commits at some point, but this preserves a record of everything I've tried.

http://stackoverflow.com/questions/9041722/solr-reports-connection-refused-when-running-import#9041769 has some tips on unresponsive solr applications, but this one was pretty interesting -- it says it can't connect to mysql, while running the solr startup command with RAILS_ENV=development -- https://travis-ci.org/publiclab/plots2/builds/159440471#L2300

Does this indicate that solr is not running in development, which would be required for rake db:setup?

[jywarren]

Hmm, actually, why is it trying to run mysql at all? The config at config/database.yml.example which is being used specifies sqlite3 for both test and development environments. @icarito or @ujithaperera -- any thoughts on that?

[jywarren]

Oh, sorry, misread -- config/database.yml.example specifies mysql for both. Oops, never mind! But we only have one database set up on docker -- "plots" -- and development environment asks for one called publiclaboratory. Kind of confused how rake db:setup ever works without a matching database in the docker config?

[jywarren]

Great, working on the failing tests now -- they seem pretty minor.

[jywarren]

OK -- @ujithaperera -- can you take a look at the currently failing tests? They seem relatively minor, and we're almost there. Please make your changes to a new branch and pull request against this one using:

git checkout -b jywarren-solr master
git pull https://github.com/jywarren/plots2.git solr

Thanks!!

[jywarren]

Actually I was able to fix the time-based errors using Timecop -- but there are two remaining errors (SO CLOSE!) to do with a mismatch between search and search_record -- this sounds like @david-days might be able to help?

======================================================================================================================================================================
.....................E
======================================================================================================================================================================
Error: test_should_get_questions_search_and_render_template_if_question_match_found(QuestionsSearchControllerTest): ActionView::Template::Error: undefined method `len
gth' for nil:NilClass
app/views/searches/normal_search.html.erb:16:in `_app_views_searches_normal_search_html_erb__887431197_66770904'
app/controllers/questions_search_controller.rb:25:in `index'
test/functional/questions_search_controller_test.rb:12:in `block in <class:QuestionsSearchControllerTest>'
======================================================================================================================================================================
........E
======================================================================================================================================================================
Error: test_should_get_show(SearchesControllerTest): ArgumentError: wrong number of arguments (1 for 0)
app/models/search_record.rb:31:in `notes'
app/controllers/searches_controller.rb:44:in `show'
test/functional/searches_controller_test.rb:25:in `block in <class:SearchesControllerTest>'
======================================================================================================================================================================
...........................................................................................

We'd /really/ like to get this merged and running, if possible. David, are you at all able to take a brief look at this? I think it's probably very simple for you as you wrote this part of the API, I believe?

[david-days]

I'll take a look right now.

[david-days]

Finally able to get the tests running--for some reason, I had to change the port for config/sunspot.yml to 8982 for all instances. I followed the instructions on your branch, but the port assignment was off.

Getting into the real tests, now, so we can figure out that issue (or not--just not include it in the update) when/if I get a fix put together.

[jywarren]

Ah, hmm, for running tests I've been booting solr with "rake
sunspot:solr:start RAILS_ENV=test" once that's running, do the default
sunspot.yml settings work? Sorry, I should've made that more clear!!!

On Sep 19, 2016 10:36 PM, "David C Days" notifications@github.com wrote:

Finally able to get the tests running--for some reason, I had to change
the port for config/sunspot.yml to 8982 for all instances. I followed the
instructions on your branch, but the port assignment was off.

Getting into the real tests, now, so we can figure out that issue (or
not--just not include it in the update) when/if I get a fix put together.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#810 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABfJ3078r0WsdeCrXKtjKwcG07tfkzkks5qr0a1gaJpZM4J5RB0
.

[david-days]

Ok--think I've figured out the problem for the question_search_controller_test.rb:

1. Call to questions_search_controller pulls up the information, then sends the call to searches/normal_search.html.erb as a template (":index" call in the test, line 12).
2. However, a look at questions_search_controller index() shows that the value for @Questions is not being filled--it is nil. questions_search_controller is actually just setting @notes.
3. Template gets to line 16, and tries to find the length of a nil, and craps out.
4. Short-term fix is changing line 16 from "<% if @questions.length == 0 %>" to "<% if @questions.nil? || @questions.length == 0 %>"
5. Long term fix is reworking questions_search_controller to fill @Questions instead of (or in addition to) @notes.

[david-days]

I'm testing a small fix now, but it looks like the problem is that the searches_controller.rb "def show" uses SearchRecord (app/models/search_record.rb), when you actually want your new class Search (app/models/search.rb).

For a quick fix, I'm just adding "@search = Search.new" just under "def show", line 42.

Testing that now.

Heh...like I said, "search" in all its forms gets used a lot--that's why I made the RESTful api use "srch", so it could be differentiated.

[jywarren]

Odd, I get one more error,

=======

Error: test_should_get_show(SearchesControllerTest): ActionView::Template::Error: undefined method `body' for nil:NilClass

app/views/searches/_node_result.html.erb:20:in `block in _app_views_searches__node_result_html_erb___252723600144578527_46994973454380'

app/views/searches/_node_result.html.erb:1:in `each'

app/views/searches/_node_result.html.erb:1:in `_app_views_searches__node_result_html_erb___252723600144578527_46994973454380'

app/views/searches/show.html.erb:21:in `_app_views_searches_show_html_erb__327083743399389248_46994983933580'

test/functional/searches_controller_test.rb:25:in `block in <class:SearchesControllerTest>'

====================================

[jywarren]

OK - -changed line 20 of _node_result.html.erb to:

<%= node.drupal_node_revision.try(:first).body.first(350) %>...

The .try(:first) should catch if there are no revisions.

[jywarren]

TESTS PASSED!!!!!! WOOOOHOOOOOOOO!!!

[jywarren]

OK, this is great, but when in development, I go to "advanced search" and type the name of a page and hit enter, i get a blank page showing "Required parameter missing: search".

That said, the Advanced Search page already doesn't return anything on the production server, so perhaps we should move forward with this.

@icarito, i'm a little worried that if the Solr server goes down, we'll see the "Connection refused" error, as apparently that'll happen any time we even load the /dashboard page. I guess for every page on the site. Could/should we implement some kind of try/catch so that if the Solr service goes down, the site still mostly works?

Otherwise, all tests are passing and I'm ready to pull this in.