Opcache optimization

[jamesmontalvo3]

Changes

• Creates config variable m_use_production_settings which is True or False. At this point, it really only controls PHP's opcache. It could be used for other general dev versus prod settings. It defaults to production settings (faster opcache, more annoying for development)
• Sets m_use_production_settings to True by default if not specified, and is specified as True in the baseline public.yml. In Vagrant's public.yml it is set to False.
• Important for development on Meza: If m_use_production_settings is True, you must now either sudo systemctl reload httpd (preferred for production setups) or sudo systemctl restart httpd (may cause current client's requests to fail) in order for Apache to be made aware of any changes to any PHP files. If you modify a PHP file your server will not see this change until Apache is reloaded/restarted.
• An Apache httpd reload has been added near the end of deploy to pick up any changes to PHP files
• Sets better defaults for opcache and makes some of them configurable via public.yml and such
• Adds ServerPerformance/opcache.php for monitoring opcache. Ultimately this should be moved into a restricted special page, just like everything in ServerPerformance.

Issues

• Ref PR Feature optimize opcache #943
• Ref PR Add Opcache status monitor #944

Post-merge actions

Post-merge, the following actions need to be addressed:

• Update documentation at https://www.mediawiki.org/wiki/Meza
  • Specifically document m_use_production_settings: True and its impact on opcache.validate_timestamps

[jamesmontalvo3]

@freephile can you take a look at this? It's all your commits, plus some changes I've made. I'm testing it on one of my dev servers over the next day or so. If all is well I'll merge later in the week (maybe tomorrow).

Some minor changes I want to make before merging into master:

1. Change m_mode to m_environment_type or something more descriptive like that
2. Set m_mode (or whatever it ends up being called) to development by default in Vagrant
3. Set m_mode (or whatever) to production in the default public.yml file when a new server/environment is setup. Yes, it's in defaults.yml, but I think it'd be good to make it as obvious as possible to new users. Keeping it in defaults.yml is required for backwards compatibility, though.
4. Add m_mode (or whatever) to automated tests: dev in one or two, prod in one or two, and unset in one or two.

[jamesmontalvo3]

I changed the var name to m_use_production_settings, which is boolean and made it False in Vagrant and True in baseline public.yml.

The following situations will be handled in jobs 1 through 5:

1. m_use_production_settings: True because it runs meza setup env monolith and thus will get the baseline value
2. m_use_production_settings: False because gets public config from enterprisemediawiki/meza-test-config and I just added that variable
3. m_use_production_settings: False because gets public config from enterprisemediawiki/meza-test-config
4. m_use_production_settings is not set, because public.yml is manually generated. This will cause it to default to True
5. m_use_production_settings: False because gets public config from enterprisemediawiki/meza-test-config

[freephile]

Yay, looks good. 👍 I'll try to get to the documentation on mw.org but have a million things on my @todo list. One important aspect to document is that validate frequency and validation on/off go together: you can have frequency every millisecond.... but if (file change) validation is off, then nothing will get checked. No validation is perfect for fast production servers. Validation on every request is perfect for development.

[jamesmontalvo3]

@freephile after two days on production (which isn't really enough to make any definitive statements from) our median page load time is unchanged, and our average page load time is up about 20%. Do you have any data on improvements from using these settings? I'm going to let a few more days play out and see what happens, then I'll tweak the settings a bit to see what happens. The ServerPerformance/opcache.php that you added (which is awesome, btw) shows about 50% memory utilization and less than 50% of keys are used. Cache hits to misses is 86,000,000:3,000. No restarts. Only cached 2350 files of the ~19000 I counted, so perhaps php_opcache_max_accelerated_files can be reduced, but I can't imagine that accounts for a 20% slower average page load.

[jamesmontalvo3]

Here's our data on the page response times of our largest internal wiki, 7 weekdays prior to the opcache changes and 4 weekdays after. I've thrown out all data points not during regular business hours, since the traffic is low which makes the data more chaotic (the first hit after an hour of inactivity takes considerably longer). Note that we make extensive use of Semantic MediaWiki, with many queries per page. This considerably slows down our response times.

Blue dots are actual page hits. Black line is median of the last 1000 hits. Red is average of the last 1000 hits.

@freephile any insights?

[freephile]

I think the issue w opcache is too many files allocated. If it reserves too much memory, leaving less memory for Apache / MySQL, then it could impact performance with more I/O (swapping) or waiting as these memory hungry beasts have less to "eat".

There is a big curve (long tail) for the application in terms of files and code paths so that say 50% of the files are used 90% of the time, while the other 50% of files are special cases. e.g. all admin functions are used far less than regular views. Some files will hardly ever be accessed, and thus won't really benefit from being cached.

So, in short, try adjusting your file allocation way down to ~3,000 and see what happens compared to 6,000, 12,000.

[jamesmontalvo3]

Copy that. I'm going to let another day or so play out to get the data, then I'll start tweaking settings to see how things go. Did you measure improvements from these changes?

[freephile]

I do not have current metrics to show before/after comparisons. I believe I took some a while ago. Anyway, I'll have a chance to compare and test a few implementations between now and December.

[jamesmontalvo3]

Actually I'm having some weird behavior. It appears that opcache is not picking up all the changes to php.ini.

php.ini has:

opcache.memory_consumption=256`
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=28001

But ServerPerformance/opcache.php is saying the following:

opcache.memory_consumption	128.00 MB
opcache.interned_strings_buffer	8
opcache.max_accelerated_files	4000

Changing these values in php.ini has NO EFFECT in opcache.php!

The following settings in php.ini are being correctly set as shown in opcache.php:

opcache.validate_timestamps=0
opcache.revalidate_freq=0
opcache.log_verbosity_level=1

And here's a copy/paste of their values from ServerPerformance/opcache.php. Changing any of these in php.ini DOES REFLECT changes in opcache.php.

opcache.validate_timestamps	false
opcache.revalidate_freq	0
opcache.log_verbosity_level	1

[jamesmontalvo3]

Here's the problem: https://serverfault.com/questions/813208/opcache-ignoring-certain-settings-in-php-ini-after-httpd-restart

Basically, /etc/php.d/10-opcache.ini is overriding values in /etc/php.ini. Pull request in work...