Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make bamboo verbose on haproxy reload errors #45

Merged
merged 2 commits into from
Oct 20, 2014

Conversation

sttts
Copy link
Contributor

@sttts sttts commented Oct 20, 2014

On haproxy reload errors, only the exit status was printed, no "Exec cmd.." and not stdout/stderr of the command, i.e.:

2014/10/20 13:02:31.794232 exit status 1
2014/10/20 13:02:31.794513 HAProxy: Configuration updated

This patch changes the error output into:

2014/10/20 13:02:31.767447 Exec cmd: read PIDS < /var/run/haproxy.pid; haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS && while ps -p $PIDS; do sleep 0.2; done 
2014/10/20 13:02:31.794232 exit status 1
2014/10/20 13:02:31.794329 Output:
[ALERT] 292/130231 (9426) : parsing [/etc/haproxy/haproxy.cfg:38] : unknown keyword 'asdf' in 'defaults' section
[ALERT] 292/130231 (9426) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] 292/130231 (9426) : Fatal errors found in configuration.

2014/10/20 13:02:31.794513 HAProxy: update failed

Fixes #44

We move the haproxy update and reload logic into a go routine which listens for
update requests on updateChan. If an update is already queued, the channel is
drained and another update request is sent. This way there can only be one
pending update queued. At the same time, no update can be missed.

By moving the real haproxy reload work into a go routine, the actual HTTP
handler remains fast and slim. Before this patch in real world environements it
took over 2 seconds to return. Marathon sometimes sends a lot of events
sequentially, leading to HTTP POST timeouts on Marathon's side if the HTTP
handler takes so long. This patch will fix this and at the same time reduce the
number of reloads to a minimum.
@j1n6
Copy link
Contributor

j1n6 commented Oct 20, 2014

Thanks for this! This was one of the task in my todo list.

j1n6 pushed a commit that referenced this pull request Oct 20, 2014
Make bamboo verbose on haproxy reload errors
@j1n6 j1n6 merged commit 58e198c into QubitProducts:master Oct 20, 2014
@ankurcha
Copy link

Are there any tests for this? Seems like a pretty important functionality got touched. It makes me a little uneasy that no tests were put in place

@j1n6
Copy link
Contributor

j1n6 commented Oct 20, 2014

nope - i'm more than happy to accept pr on adding unit test.

On 20 October 2014 17:10, Ankur Chauhan notifications@github.com wrote:

Are there any tests for this? Seems like a pretty important functionality
got touched. It makes me a little uneasy that no tests were put in place


Reply to this email directly or view it on GitHub
#45 (comment).

This email may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone to

the wrong person. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Haproxy reload error output not verbose enough and misleading
3 participants