Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Somehow a change to a plugin is restarting jenkins quite often #250

Closed
dkowis opened this issue Feb 13, 2015 · 20 comments · Fixed by #396
Closed

Somehow a change to a plugin is restarting jenkins quite often #250

dkowis opened this issue Feb 13, 2015 · 20 comments · Fixed by #396
Labels
needs-feedback Further information is requested

Comments

@dkowis
Copy link

dkowis commented Feb 13, 2015

I have the version of the plugin fixed in the puppet manifest:

    jenkins::plugin{ 'ghprb':
        version => '1.16-8'
    }

But I will see this happen on occasion. I don't know why, because I didn't delete that plugin, or anything else. I don't know why it's getting the plugin again, because it shouldn't be. But this is causing our jenkins to restart in some cases every 20 minutes....

Feb 13 13:07:49 jenkins-proto puppet-agent[12424]: (/Stage[main]/Repose_jenkins::Master/Jenkins::Plugin[ghprb]/Exec[download-ghprb]/re
turns) executed successfully
Feb 13 13:07:49 jenkins-proto puppet-agent[12424]: (/Stage[main]/Repose_jenkins::Master/Jenkins::Plugin[ghprb]/File[/var/lib/jenkins/p
lugins/ghprb.hpi]/owner) owner changed 'root' to 'jenkins'
Feb 13 13:07:51 jenkins-proto jenkins: jenkins: client (pid 9935) exited with 143 status
Feb 13 13:07:52 jenkins-proto puppet-agent[12424]: (/Stage[main]/Jenkins::Service/Service[jenkins]) Triggered 'refresh' from 1 events
Feb 13 13:07:52 jenkins-proto puppet-agent[12424]: (/Stage[main]/Base/Service[chrony]/ensure) ensure changed 'stopped' to 'running'
Feb 13 13:07:52 jenkins-proto puppet-agent[12424]: Finished catalog run in 5.94 seconds
@dkowis
Copy link
Author

dkowis commented Feb 13, 2015

As best as I can tell, from reading the puppet manifests, the ghprb plugin isn't being found in the jenkins_plugins fact. I can't figure out how to run that fact locally to figure out what's going on and why it's not there.

Also, I don't understand why it's not hitting any of the other plugins, if there were something wrong with getting the list of plugins...

@dkowis
Copy link
Author

dkowis commented Feb 13, 2015

All this happened on 1.2.0 of the module.

I have updated to 1.3.0 of the puppet module, trying that out.

@dkowis
Copy link
Author

dkowis commented Feb 13, 2015

I think this might have been due to a missing dependency for the ghprb plugin, but I couldn't figure out how to reproduce it. I had one error in the jenkins logs indicating that the ghprb plugin wouldn't load due to a missing dependency. It's loaded now.... I'll update if I still keep getting this problem.

@jchristi
Copy link
Contributor

@dkowis, to run facter locally, use facter -p

@dkowis
Copy link
Author

dkowis commented Feb 16, 2015

The problem is that a jenkins plugin was missing a dependency, puppet would then put the plugin back, and this would cycle back on itself every 20 minutes. It was just a difficult error to pin down, ultimately my fault for not making sure a jenkins plugin had the proper dependencies.

@dkowis dkowis closed this as completed Feb 16, 2015
@dkowis dkowis reopened this Mar 30, 2015
@dkowis
Copy link
Author

dkowis commented Mar 30, 2015

So I've run into this again.

It still claims that the ghprb plugin has something being updated in the exact same way as before, however, we haven't updated the ghprb plugin. So it either updated underneath us and didn't tell us, or something else is going on.

It's resulting in restarting the jenkins service every 30 minutes (on each puppet run.) Is there some way to get more information about why it's deciding to update the plugin? Is there a way to prevent the service from restarting so we can at least have some time to figure out what's going on?

@jchristi
Copy link
Contributor

@dkowis what is the output of facter -p on the host? What is the version of ghprb you are using and what is puppet trying to update the version to?

@dkowis
Copy link
Author

dkowis commented Mar 31, 2015

We're using 1.16-8, and that's the version on the host, and we aren't actually trying to update anything...
https://github.com/rackerlabs/repose-infrastructure-ng/blob/master/myModules/repose_jenkins/manifests/master.pp#L83

A puppet run would look like the following, even though there were no actual changes to the ghprb plugin. The last time we had this problem, the ghprb plugin was a red herring and it was a missing dependency instead (which we solved.) this time it just started showing up without us changing any configurations.

Mar 29 06:54:22 jenkins-proto puppet-agent[13696]: (/File[/var/lib/puppet/facts.d]) Could not evaluate: Could not retrieve information
 from environment production source(s) puppet://puppet/pluginfacts
Mar 29 06:54:30 jenkins-proto puppet-agent[13696]: (/Stage[main]/Repose_jenkins::Master/Jenkins::Plugin[ghprb]/Exec[download-ghprb]/returns) executed successfully
Mar 29 06:54:30 jenkins-proto puppet-agent[13696]: (/Stage[main]/Repose_jenkins::Master/Jenkins::Plugin[ghprb]/File[/var/lib/jenkins/plugins/ghprb.hpi]/owner) owner changed 'root' to 'jenkins'
Mar 29 06:54:32 jenkins-proto jenkins: jenkins: client (pid 6300) exited with 143 status
Mar 29 06:54:33 jenkins-proto puppet-agent[13696]: (/Stage[main]/Jenkins::Service/Service[jenkins]) Triggered 'refresh' from 1 events
Mar 29 06:54:33 jenkins-proto puppet-agent[13696]: (/Stage[main]/Base/Service[chrony]/ensure) ensure changed 'stopped' to 'running'
Mar 29 06:54:33 jenkins-proto puppet-agent[13696]: Finished catalog run in 6.40 seconds

I took out a few things, but here's what is in facter -p:

architecture => amd64
augeasversion => 0.10.0
bios_release_date => 11/28/2013
bios_vendor => Xen
bios_version => 4.1.5
blockdevice_xvda_size => 42949672960
blockdevice_xvde_size => 85899345920
blockdevices => xvda,xvde
concat_basedir => /var/lib/puppet/concat
domain => openrepose.org
facterversion => 2.4.1
filesystems => ext3,ext4
gemhome => /var/lib/gems/1.9.1
gid => root
hardwareisa => unknown
hardwaremodel => x86_64
hostname => jenkins-proto
id => root
interfaces => eth0,eth1,lo
ip6tables_version => 1.4.14
ipaddress_lo => 127.0.0.1
iptables_persistent_version => 0.5.7
iptables_version => 1.4.14
is_pe => false
is_rsc => true
is_virtual => true
jenkins_plugins => ant 1.2, antisamy-markup-formatter 1.1, conditional-buildstep 1.3.3, copyartifact 1.33, credentials 1.18, cvs 2.11, dashboard-view 2.9.4, envinject 1.90, external-monitor-job 1.4, ghprb 1.16-8, git 2.3.4, git-client 1.15.0, github 1.10, github-api 1.59, jacoco 1.0.18, javadoc 1.1, jenkins-multijob-plugin 1.16, join 1.15, jquery 1.7.2-1, junit 1.2-beta-4, ldap 1.6, m2release 0.14.0, mailer 1.11, mask-passwords 2.7.2, matrix-auth 1.1, matrix-project 1.4.1, maven-plugin 2.7.1, pam-auth 1.1, parameterized-trigger 2.25, publish-over-ssh 1.12, run-condition 1.0, scm-api 0.2, scm-sync-configuration 0.0.7.3, script-security 1.13, simple-theme-plugin 0.3, sonar 2.1, ssh 2.4, ssh-agent 1.5, ssh-credentials 1.10, ssh-slaves 1.9, subversion 1.54, swarm 1.22, token-macro 1.10, translation 1.10, windows-slaves 1.0
kernel => Linux
kernelmajversion => 3.2
kernelrelease => 3.2.0-4-amd64
kernelversion => 3.2.0
lsbdistcodename => wheezy
lsbdistdescription => Debian GNU/Linux 7.8 (wheezy)
lsbdistid => Debian
lsbdistrelease => 7.8
lsbmajdistrelease => 7
lsbminordistrelease => 8
manufacturer => Xen
maven_version => 3.2.1
memoryfree => 6.38 GB
memoryfree_mb => 6529.14
memorysize => 7.57 GB
memorysize_mb => 7748.98
mtu_eth0 => 1500
mtu_eth1 => 1500
mtu_lo => 16436
operatingsystem => Debian
operatingsystemmajrelease => 7
operatingsystemrelease => 7.8
os => {"name"=>"Debian", "family"=>"Debian", "release"=>{"major"=>"7", "minor"=>"8", "full"=>"7.8"}, "lsb"=>{"distcodename"=>"wheezy", "distid"=>"Debian", "distdescription"=>"Debian GNU/Linux 7.8 (wheezy)", "distrelease"=>"7.8", "majdistrelease"=>"7", "minordistrelease"=>"8"}}
osfamily => Debian
partitions => {"xvda1"=>{"uuid"=>"497441ed-9739-4cb3-97b2-440ba257b38c", "size"=>"83873317", "mount"=>"/", "filesystem"=>"ext3"}, "xvde1"=>{"uuid"=>"40f816dd-456d-4089-874e-fa8cbf3ad296", "size"=>"167772159", "mount"=>"/var/lib/jenkins", "filesystem"=>"ext4"}}
path => /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
physicalprocessorcount => 8
pip_version => false
processor0 => Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
processor1 => Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
processor2 => Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
processor3 => Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
processor4 => Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
processor5 => Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
processor6 => Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
processor7 => Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
processorcount => 8
processors => {"models"=>["Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz", "Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz", "Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz", "Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz", "Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz", "Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz", "Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz", "Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz"], "count"=>8, "physicalcount"=>8}
productname => HVM domU
ps => ps -ef
puppet_vardir => /var/lib/puppet
puppetversion => 3.7.5
python_version => 2.7.3
root_home => /root
rsc_instance_id => 4bb96df2-4aaf-4d19-b77c-bee0d16c7968
rsc_region => dfw
rubyplatform => x86_64-linux
rubysitedir => /usr/local/lib/site_ruby/1.9.1
rubyversion => 1.9.3
selinux => false
swapfree => 4.00 GB
swapfree_mb => 4096.00
swapsize => 4.00 GB
swapsize_mb => 4096.00
system_python_version => 2.7.3
system_uptime => {"seconds"=>2378615, "hours"=>660, "days"=>27, "uptime"=>"27 days"}
timezone => CDT
type => Other
uniqueid => d00a6a21
uptime => 27 days
uptime_days => 27
uptime_hours => 660
uptime_seconds => 2378615
uuid => 1CDF73EF-A712-6B8B-02FD-5EE73C7627FE
virtual => xenhvm
virtualenv_version => absent

@jchristi
Copy link
Contributor

I suppose the version of puppet-jenkins you are using would also be an important detail too :)

@dkowis
Copy link
Author

dkowis commented Mar 31, 2015

Certainly, it's still 1.3.0. Haven't updated from the last time :)

@jchristi
Copy link
Contributor

My first guess is that it has something to do with the ghprb version string having a dash in it. I swear I saw that issue come up before but I thought it was fixed. I can't find it though.

@dkowis
Copy link
Author

dkowis commented Apr 2, 2015

This feels related to #219 . I wonder if somehow it's failing to get ahold of the plugins list for facter and is bombing?

@dkowis
Copy link
Author

dkowis commented Apr 2, 2015

  describe 'when the jenkins_plugins fact is empty' do
    let(:params) {{ :version => '1.2.3'}}
    let(:facts) {{ :jenkins_plugins => ''}}

    it { should_not contain_exec('download-myplug')}
    it { should_not contain_file('/var/lib/jenkins/plugins/myplug.hpi')}
  end

Stick this spec in jenkins_plugin_spec.rb and it fails, and might be why it's derping up. If for some reason the JSON cannot be found/parsed, it will attempt to install the plugin again. I'd prefer the behavior to be "do nothing." I think that is far less disruptive upon a jenkins box in production. Not exactly sure how to solve this, digging into it a bit. @jchristi WDYT?

@dkowis
Copy link
Author

dkowis commented Apr 2, 2015

Nope, I think I've got this backwards, don't I? The jenkins_plugins fact is what's on the local box? Yeah, nevermind.

@dkowis
Copy link
Author

dkowis commented Apr 2, 2015

A mitigating factor would be if I could just not have the puppet module manage the service for me. Is there a way to do that?

@dkowis
Copy link
Author

dkowis commented Apr 2, 2015

Also, in the lib/puppetjenkins/plugins.rb is there a way to log in the the self.available method what might have happened if we had a Standard Error in there that would result in a smaller or empty plugins list? I don't know what would cause it to skip the ghprb plugin resulting in a reinstall of it...

@jchristi
Copy link
Contributor

jchristi commented May 5, 2015

@dkowis I'm pretty sure @rtyler heavily refactored my plugins custom fact, but you are probably on the right track. My suggestion would be to try running that fact on the affected host, perhaps adding binding.pry to step debug through and see where it's actually failing.

@jhoblitt
Copy link
Member

@dkowis Is this reproducible with the current master? Plugin handling has been reworked a bit.

@jhoblitt jhoblitt added the needs-feedback Further information is requested label Oct 11, 2015
jhoblitt added a commit to jhoblitt/puppet-jenkins that referenced this issue Oct 12, 2015
* add `manager_user`, `user`, `manage_group`, `group` params to
  `jenkins` class.

* improve consistency of `file` resouce group ownership throughout
  the module

* add 'localstatedir' param to `jenkins` class and consistently use it
  to set the 'data' base dir throughout the module

* Deprecate/noop `username`, `group`, `create_user` params to
  `jenkins::plugin` define; replaced by `user`,`group`,etc. params to
  `jenkins` class

* Deprecate/noop `plugin_dir` param to `jenkins::plugin`; replaced by
  `localstatedir` param to `jenkins` class

closes voxpupuli#356 -- alternative approach
resolves voxpupuli#362
resolves voxpupuli#365
resolves voxpupuli#219
resolves voxpupuli#249
resolves voxpupuli#250
@dkowis
Copy link
Author

dkowis commented Oct 12, 2015

I'll give it a shot soonish, I'm not exactly on that project any more, but I'll see what I can do.

@jhoblitt
Copy link
Member

@dkowis It would be great if you have the time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-feedback Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants