Schedule Webhook Deploys in Threads

[vinayhegde90]

Hi Team,

We're using Samson (v2488) for our deployments for about 12 apps with 4-5 stages for each. For automated deploys, we trigger the Generic web-hook via a scheduled bash-script to all the stages every morning. Below is a small snippet of the same:

#!/bin/bash
curl -v http://$webhook_url_1/integrations/generic/$random_hash_1 -d @deploy.json --header "Content-Type: application/json"
sleep 180

curl -v http://$webhook_url_2/integrations/generic/$random_hash_2 -d @deploy.json --header "Content-Type: application/json"
sleep 180

curl -v http://$webhook_url_X/integrations/generic/$random_hash_X -d @deploy.json --header "Content-Type: application/json"

The Issue: While the script just fires a curl call to trigger the deployments and about 50% succeed, the remaining fail with errors like below:

[12:37:21] Your configuration specifies to merge with the ref 'refs/heads/master'
[12:37:21] from the remote, but no such ref was fetched.

Can someone here please help me with enable scheduling via an option in Samson or an alternate logic in the script such as multi-threading so that the triggered webhook finishes deploy to all stages successfully in a phased manner?

Do let me know if any more info is required.

[grosser]

samson has periodic deploys which might help PERIODICAL=...,periodical_deploy:86400,... but it does them all at once

idk what these errors are or where they come from ... sounds like git problems :(

you could try using curl -X POST foo.com/projects/foo/stages/bar/deploys and see if that's better

[vinayhegde90]

I already had a look at periodic deploys as you said but was confused by its tooltip.

Please correct me if I'm wrong but periodic deploys will only happen every 24 hrs from the last deploy that should be successful otherwise there won't be any, thus triggering a failure email.

For e.g: Let's say if a successful deploy happens for 1 stage today at 7 PM, then the periodic one will be triggered at 7 PM on the next day right?

Also, is there a way to schedule this at a particular time?

[grosser]

It schedules at the same time for all deploys, I'd think it would run midnight UTC I think it would be easy to add a kind of offset so it runs at the time you want.

…

On Mon, Jul 8, 2019 at 12:40 AM Vinay Hegde ***@***.***> wrote: I already had a look at periodic deploys as you said but was confused by its tooltip. [image: samson-periodic-deploys] <https://user-images.githubusercontent.com/43435196/60791154-ace3b200-a180-11e9-9824-df3b8ad7d2bf.png> Please correct me if I'm wrong but periodic deploys will only happen every 24 hrs from the last deploy that should be successful otherwise there won't be any, thus triggering a failure email. For e.g: Let's say if a *successful* deploy happens for 1 stage today at *7 PM*, then the periodic one will be triggered at *7 PM* on the *next day* right? Also, is there a way to schedule this at a particular time? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3436?email_source=notifications&email_token=AAACYZZL7MDAHFSGLQBSL53P6LVO7A5CNFSM4H6LBW6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZMHZWQ#issuecomment-509115610>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAACYZ4GCTQFV7OUHBFT3JDP6LVO7ANCNFSM4H6LBW6A> .

[grosser]

fyi added a PR to show next execution time in the tooltip

[vinayhegde90]

I suppose we'll need to update Samson to the latest, i.e: v2845 to avail the fix pushed in #3438

It schedules at the same time for all deploys, I'd think it would run midnight UTC I think it would be easy to add a kind of offset so it runs at the time you want.

How can this be added? Also as we'd want the runs to be in IST, does it depend on what timezone Samson uses?

[vinayhegde90]

you could try using curl -X POST foo.com/projects/foo/stages/bar/deploys and see if that's better

tried it as you recommended but it didn’t work, sharing the commands & status codes for reference.

user@server:~$ curl -X POST http://endpoint/projects/<project-name>/stages/<qa>/deploys
HTTP/1.1 422 Unprocessable Entity
Content-Type: text/html; charset=UTF-8
X-Request-Id: 78552476-22fc-46bd-9cc5-88121df3ae7c
X-Runtime: 0.112386
Set-Cookie: __profilin=p%3Dt; path=/
Content-Length: 103248

user@server:~$ curl --write-out %{http_code} http://endpoint/projects/<project-name>/stages/<qa>/deploys -d @deploy.json --header "Content-Type: application/json"

while running this as a one-off command is fine but won't this approach be problematic to scale up in a script since there'd be multiple apps across multiple stages?

[grosser]

will need to send some data like `-d '{"deploy":{"reference":"master"}}'` etc idk why this would be problematic ... need lots of deploys -> do lots of api calls :D

…

On Tue, Jul 9, 2019 at 1:33 AM Vinay Hegde ***@***.***> wrote: you could try using curl -X POST foo.com/projects/foo/stages/bar/deploys and see if that's better tried it as you recommended but it didn’t work, sharing the commands & status codes for reference. ***@***.***:~$ curl -X POST http://endpoint/projects/<project-name>/stages/<qa>/deploys HTTP/1.1 422 Unprocessable Entity Content-Type: text/html; charset=UTF-8 X-Request-Id: 78552476-22fc-46bd-9cc5-88121df3ae7c X-Runtime: 0.112386 Set-Cookie: __profilin=p%3Dt; path=/ Content-Length: 103248 ***@***.***:~$ curl --write-out %{http_code} http://endpoint/projects/<project-name>/stages/<qa>/deploys -d @deploy.json --header "Content-Type: application/json" while running this as a one-off command is fine but won't this approach be problematic to scale up in a script since there'd be multiple apps across multiple stages? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3436?email_source=notifications&email_token=AAACYZ7PB4GBZKSCZRIR2NLP6RENZA5CNFSM4H6LBW6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZPRBFY#issuecomment-509546647>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAACYZ6YX4RYSYIYQZPANJ3P6RENZANCNFSM4H6LBW6A> .

[grosser]

also the response body should have detailed errors on what exactly is missing

…

On Tue, Jul 9, 2019 at 4:14 PM Michael Grosser ***@***.***> wrote: will need to send some data like `-d '{"deploy":{"reference":"master"}}'` etc idk why this would be problematic ... need lots of deploys -> do lots of api calls :D On Tue, Jul 9, 2019 at 1:33 AM Vinay Hegde ***@***.***> wrote: > you could try using curl -X POST foo.com/projects/foo/stages/bar/deploys > and see if that's better > > tried it as you recommended but it didn’t work, sharing the commands & > status codes for reference. > > ***@***.***:~$ curl -X POST http://endpoint/projects/<project-name>/stages/<qa>/deploys > > HTTP/1.1 422 Unprocessable Entity > > Content-Type: text/html; charset=UTF-8 > > X-Request-Id: 78552476-22fc-46bd-9cc5-88121df3ae7c > > X-Runtime: 0.112386 > > Set-Cookie: __profilin=p%3Dt; path=/ > > Content-Length: 103248 > > > > ***@***.***:~$ curl --write-out %{http_code} http://endpoint/projects/<project-name>/stages/<qa>/deploys -d @deploy.json --header "Content-Type: application/json" > > > while running this as a one-off command is fine but won't this approach > be problematic to scale up in a script since there'd be multiple apps > across multiple stages? > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#3436?email_source=notifications&email_token=AAACYZ7PB4GBZKSCZRIR2NLP6RENZA5CNFSM4H6LBW6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZPRBFY#issuecomment-509546647>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAACYZ6YX4RYSYIYQZPANJ3P6RENZANCNFSM4H6LBW6A> > . >

[vinayhegde90]

will need to send some data like -d '{"deploy":{"reference":"master"}}' etc idk why this would be problematic ... need lots of deploys -> do lots of api calls :D

doing lots of API calls isn't the issue here, what I was looking for a way to do them in parallel (with or without threads) with a single endpoint thereby avoiding failures to deploy on the same app on multiple stages

[grosser]

are you talking about a single deploy that goes to all stages of an app ?

…

On Wed, Jul 10, 2019 at 2:01 AM Vinay Hegde ***@***.***> wrote: will need to send some data like -d '{"deploy":{"reference":"master"}}' etc idk why this would be problematic ... need lots of deploys -> do lots of api calls :D doing lots of API calls isn't the issue here, what I was looking for a way to do them in parallel (*with or without threads*) with a single endpoint thereby avoiding failures to deploy on the same app on multiple stages — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3436?email_source=notifications&email_token=AAACYZ4SOY2MVS7G2TBSDFLP6WQORA5CNFSM4H6LBW6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZS2A3Q#issuecomment-509976686>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAACYZ5EV4URH2NQYKFA3FDP6WQORANCNFSM4H6LBW6A> .

[vinayhegde90]

I meant one app, i.e: (backend-app-1) is deployed to multiple stages (server-1, server-2... server-n) with master branch & similarly, (frontend-app-1) to above stages

[grosser]

So you want to deploy the next stage when the first stage finishes ?
Using the webhook or the /deploys endpoint should return a deploy id, that can be used to check if the deploy is succeeded ... need to implement a loop to wait until it is either succeeded or failed/errored and then continue/stop.

[vinayhegde90]

No, we want to deploy it in parallel in a threaded manner on all stages for a particular app. We do have a loop that sleeps for X seconds post triggering the webhooks, i.e: sequentially but that isn't very effective.

idk what these errors are or where they come from ... sounds like git problems :(

We also figured a probable cause for the Git issues that appear due to Samson doing Git IO like fetch/pull/checkout still ongoing for deploys on 1 stage but fails on any of the others due to Git locking.

A workaround to solve this would be separating and/or cloning the repos in a folder structure based on stage which has the following pros & cons:
Pros:

• Git locking will be avoided as each app is deployed to all stages without conflicts or waiting for an operation to finish.
• Removes a single point of failure in case of repo deletion as dependent deploys will fail due to directory being absent until it's recloned

Cons:

• Keeping multiple copies of same repo segregated in different folders will increase disk usage.
• Will get tedious to manage for a large number of stages.
• Increases IO as team scales since there'll be more pull/push

Hoping this helps anyone solving a similar problem, cheers!