[BUG] epicli upgrade fails when upgrading Open Distro for Elasticsearch

[to-bar]

Describe the bug
There is a fatal error when upgrading Open Distro for Elasticsearch:

TASK [upgrade : Open Distro for Elasticsearch | Start elasticsearch service]
ERROR cli.engine.ansible.AnsibleCommand - fatal: [ci-06todevazubflannel-logging-vm-0]: FAILED! => {"changed": false, "msg": "Unable to start service elasticsearch: Job for elasticsearch.service failed because the control process exited with error code.\nSee \"systemctl status elasticsearch.service\" and \"journalctl -xe\" for details.\n"}

On host:

[operations@ci-06todevazubflannel-logging-vm-0 ~]$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2020-09-11 07:21:14 UTC; 23s ago
     Docs: https://www.elastic.co
  Process: 66006 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
 Main PID: 66006 (code=exited, status=78)

Sep 11 07:20:51 ci-06todevazubflannel-logging-vm-0 systemd[1]: Starting Elasticsearch...
Sep 11 07:20:55 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[66006]: OpenJDK 64-Bit Server VM warning: Ignoring option UseConcMarkSweepGC; support was removed in 14.0
Sep 11 07:20:55 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[66006]: OpenJDK 64-Bit Server VM warning: Ignoring option CMSInitiatingOccupancyFraction; support was removed in 14.0
Sep 11 07:20:55 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[66006]: OpenJDK 64-Bit Server VM warning: Ignoring option UseCMSInitiatingOccupancyOnly; support was removed in 14.0
Sep 11 07:21:14 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[66006]: ERROR: [1] bootstrap checks failed
Sep 11 07:21:14 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[66006]: [1]: JVM is using the serial collector but should not be for the best performance; either it's the default for the VM [OpenJDK 64-Bit Server VM] or -XX:+UseSeriSep 11 07:21:14 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[66006]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/EpiphanyElastic.log
Sep 11 07:21:14 ci-06todevazubflannel-logging-vm-0 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Sep 11 07:21:14 ci-06todevazubflannel-logging-vm-0 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 11 07:21:14 ci-06todevazubflannel-logging-vm-0 systemd[1]: Failed to start Elasticsearch.

[operations@ci-06todevazubflannel-logging-vm-0 ~]$ sudo tail /var/log/elasticsearch/EpiphanyElastic.log
[2020-09-11T07:21:14,067][INFO ][o.e.b.BootstrapChecks    ] [ci-06todevazubflannel-logging-vm-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-09-11T07:21:14,077][ERROR][o.e.b.Bootstrap          ] [ci-06todevazubflannel-logging-vm-0] node validation exception
[1] bootstrap checks failed
[1]: JVM is using the serial collector but should not be for the best performance; either it's the default for the VM [OpenJDK 64-Bit Server VM] or -XX:+UseSerialGC was explicitly specified
[2020-09-11T07:21:14,093][INFO ][o.e.n.Node               ] [ci-06todevazubflannel-logging-vm-0] stopping ...

[2020-09-11T09:08:08,581][INFO ][o.e.n.Node               ] [ci-06todevazubflannel-logging-vm-0] version[7.8.0], pid[70815], build[oss/deb/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Linux/5.3.0-1028-azure/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[2020-09-11T09:08:08,593][INFO ][o.e.n.Node               ] [ci-06todevazubflannel-logging-vm-0] JVM home [/usr/share/elasticsearch/jdk]
[2020-09-11T09:08:08,594][INFO ][o.e.n.Node               ] [ci-06todevazubflannel-logging-vm-0] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-9877401543197881936, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/elasticsearch/plugins/opendistro_performance_analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb, -Des.bundled_jdk=true]

To Reproduce
Steps to reproduce the behavior:

1. create cluster using epicli 0.6
2. run epicli upgrade using a build from develop branch

OS (please complete the following information):

• OS: RHEL7 & Ubuntu 18.04

Cloud Environment (please complete the following information):

• Cloud Provider: Azure

Additional context
https://www.elastic.co/guide/en/elasticsearch/reference/master/_use_serial_collector_check.html

[to-bar]

I was able to start elasticsearch service after replacing jvm.options file:
cp -a /etc/elasticsearch/jvm.options.dpkg-dist /etc/elasticsearch/jvm.options

See elastic/elasticsearch#36828

root@ci-06todevazubflannel-logging-vm-0:~# systemctl status elasticsearch.service
[...]
Sep 11 18:18:58 ci-06todevazubflannel-logging-vm-0 systemd[1]: Starting Elasticsearch...
Sep 11 18:19:24 ci-06todevazubflannel-logging-vm-0 systemd[1]: Started Elasticsearch.
Sep 11 18:19:25 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[27984]: WARNING: An illegal reflective access operation has occurred
Sep 11 18:19:25 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[27984]: WARNING: Illegal reflective access by com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.MasterServiceEventMetrics (file:/usr/share/elasticsearcSep 11 18:19:25 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[27984]: WARNING: Please consider reporting this to the maintainers of com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.MasterServiceEventMetrics
Sep 11 18:19:25 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[27984]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Sep 11 18:19:25 ci-06todevazubflannel-logging-vm-0 systemd-entrypoint[27984]: WARNING: All illegal access operations will be denied in a future release

[mkyc]

It's possible that it also appears in other versions.

[atsikham]

Unable to upgrade, bug #1741 is created.

[rafzei]

Fix merged into develop #1750

[to-bar]

Test OK (upgrade from v0.7.2 to develop). Additional changes planned, see comment in #1719.

[rpudlowski93]

Tested from my side. Upgrade completed properly.