Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split brain handling #44

Merged
merged 13 commits into from
Feb 27, 2020
Merged

Split brain handling #44

merged 13 commits into from
Feb 27, 2020

Conversation

ytsarev
Copy link
Member

@ytsarev ytsarev commented Feb 16, 2020

  • Use TXT as 'heartbeat' for spefic gslb deployment
  • Use UTC timestamp with second level precision
  • Switch to automatically generated geo tag based fqdns
  • Eventually we will detect 'stale' gslb deployments by an
    outdated timestamp
  • The timestamp is about to handle situation of split brain
    between multiple gslb deployments in case of network partition
    between them but not between gslb and edgeDNS

* Planned to be used for handling split-brain
* Use TXT as 'heartbeat' for spefic gslb deployment
* Use UTC timestamp with second level precision
* Switch to automatically generated geo tag based fqdns
* Eventually we will detect 'stale' gslb deployments by an
  outdated timestamp
* The timestamp is about to handle situation of split brain
  between multiple gslb deployments in case of network partition
  between them but not between gslb and edgeDNS
@ytsarev
Copy link
Member Author

ytsarev commented Feb 16, 2020

@donovanmuller it's not completed implementation. Before proceeding further I would like to validate the design.

@donovanmuller
Copy link
Contributor

@ytsarev 👍let's discuss and walk through the permutations.

{Address: "10.0.0.1", Name: "test-gslb-ns-eu.example.com"},
{Address: "10.0.0.2", Name: "test-gslb-ns-eu.example.com"},
{Address: "10.0.0.3", Name: "test-gslb-ns-eu.example.com"},
{Address: "10.1.0.1", Name: "test-gslb-ns-sa.example.com"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

County code for South Africa is za 😀

* Test logic injection
* Stripped implementation for fakeInfoblox connector from
  https://github.com/infobloxopen/infoblox-go-client/blob/master/object_manager_test.go
* Why we need to implement fakeConnector on our own?
  - infoblox-go-client package does not export fakeConnector
  - even if we export it with mass rename to FakeConnector
    object_manager_test.go will not be installed by `go mod`
    golang/go#26913
  - moving into main ibclient package file does not seem to be good idea
    as it is purely testing code with dependenies like gingko/omega
  - also upstream will probably never accept it
* Temporary comment out yet unused fields of the fake Infoblox connector
  to obey the lint

Switch github actions pipeline back to upstream

As shoukoo/golang-pipeline#5 was merged
* fakedns can serve TXT with 10m old timestamp withing
* associated test suite extensions
@ytsarev ytsarev changed the title Split brain handling prototype Split brain handling Feb 27, 2020
@ytsarev
Copy link
Member Author

ytsarev commented Feb 27, 2020

@donovanmuller commits are squashed to more or less important semantic sets.

Btw that's how relevant part of test suite execution looks like

=== RUN   TestGslbController/Can_check_external_Gslb_TXT_record_for_validity_and_fail_if_it_is_expired
2020-02-27T14:43:55.314+0100	INFO	controller_gslb	Query for TXT test-gslb-ns-eu.example.com.

2020-02-27T14:43:55.314+0100	INFO	controller_gslb	TXTs found: [2020-02-27T13:33:55]

2020-02-27T14:43:55.314+0100	INFO	controller_gslb	Split brain TXT raw record: test-gslb-ns-eu.example.com.	3600	IN	TXT	"2020-02-27T13:33:55"
2020-02-27T14:43:55.314+0100	INFO	controller_gslb	Split brain TXT raw time stamp: 2020-02-27T13:33:55
2020-02-27T14:43:55.314+0100	INFO	controller_gslb	Split brain TXT parsed time stamp: 2020-02-27 13:33:55 +0000 UTC
2020-02-27T14:43:55.314+0100	INFO	controller_gslb	Split brain TXT time diff: 10m0.314661s
=== RUN   TestGslbController/Can_check_external_Gslb_TXT_record_for_validity_and_pass_if_it_is_not_expired
2020-02-27T14:43:55.314+0100	INFO	controller_gslb	Query for TXT test-gslb-ns-za.example.com.

2020-02-27T14:43:55.314+0100	INFO	controller_gslb	TXTs found: [2020-02-27T13:40:55]

2020-02-27T14:43:55.314+0100	INFO	controller_gslb	Split brain TXT raw record: test-gslb-ns-za.example.com.	3600	IN	TXT	"2020-02-27T13:40:55"
2020-02-27T14:43:55.314+0100	INFO	controller_gslb	Split brain TXT raw time stamp: 2020-02-27T13:40:55
2020-02-27T14:43:55.314+0100	INFO	controller_gslb	Split brain TXT parsed time stamp: 2020-02-27 13:40:55 +0000 UTC
2020-02-27T14:43:55.315+0100	INFO	controller_gslb	Split brain TXT time diff: 3m0.315007s

@ytsarev ytsarev merged commit 0420523 into master Feb 27, 2020
@ytsarev ytsarev deleted the split-brain branch February 27, 2020 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants