Fix international strings test / mishandling of international strings

[eileenmcnaughton]

Overview

Fix mishandling of non-English charsets in very specific searches - e.g organization_name

Before

Specific mysql / server environments cannot successfully search for organization_name of 'Scarabée'

This is replicated by the environment-specifc errors on the testInternationalStrings test

After

testInternationalStrings now passes locally & hopefully will also pass on Jenkins where it was failing.... holds breath.

Technical Details

In trying to review PR #12364 I dug into the strotolower behaviour. I found that
in the path altered in this PR we are setting the value to lower case but not the
mysql clause.

I ran through ContactTest without these lines & found it
only hit this strtolower in one test - testInternationalStrings which searches
by organization name. I found the this test actually failed for me
when I ran it locally - but it passed when I removed the strotlower lines.

Wonder if jenkins finds the same

Comments

@seamuslee001 @monishdeb

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[eileenmcnaughton]

@seamuslee001 @monishdeb @colemanw So we now have some more information about the use of LOWER in mysql.

What we knew

1. wrapping a where param in LOWER() means the index cannot be used, resulting in a performance hit
2. we have removed various instances of the lower casing over time with no blow back or bug reports
3. lower casing strings was messing with regex

What we ALSO know now

1. the use of LOWER actively hurts our searches on some non-english alpha strings and that is the cause of the intermittent issues on those strings

Where to?

1. I think we should actively remove the last 6 places in the query object where LOWER is used. I'm happy to spread that over a few months just in case anything pops up

[seamuslee001]

@eileenmcnaughton i still have a question about whether the locale of the server plays some role in interpreting the string (especially with accents)

[eileenmcnaughton]

@seamuslee001 I don't know - what I DO know is that the strtolower() php function seems to mess with the string locally. Perhaps when mbstrtolower is installed it gets 'lowered' without mucking up the string

[eileenmcnaughton]

@seamuslee001 I propose to close this for now & we merge #12503 for 5.5 & then another one of a similar scope next month

[eileenmcnaughton]

I'm going to close this for now as we have a plan to phase out lower slowly based on this I believe

[eileenmcnaughton]

I have re-opened this because we have done as discussed above & removed some other places without any issues arising. We have had a related PR arise (#12930) so that is encouragement to push on with phasing out lower.

At the moment this will fail as I'm trying to flush out the tested places that touch on this

[eileenmcnaughton]

@mfb @seamuslee001 OK - this is the last one! I think we should merge this & it will be done!

I searched for LOWER( & there are only 2-3 places left and they all seem very contained - ie. Contribution_BAO_Query still does the bad thing when you search on contribution note. I was going to leave that out of scope though unless on of you want to do it.

[seamuslee001]

This is all consistent with all the other work going on. These parts all look to be well backed up by tests so the fact this has passed Jenkins makes me feel comfortable with this. I note we are re-enabling a couple of tests and all tests have passed and i'm good with this enough to merge

[eileenmcnaughton]

Thanks @seamuslee001

@mfb I'm not going to work on the last few places that LOWER still happens at the moment but am happy to review if you wish to