Skip to content

Commit

Permalink
Merge pull request #69 from linki/excluded-dates
Browse files Browse the repository at this point in the history 
Allow to exclude certain dates from chaos
  • Loading branch information
@linki
linki authored Mar 8, 2018
2 parents 1c7a239 + 04f7317 commit 46e70ef
Show file tree
Hide file tree
Showing 6 changed files with 211 additions and 11 deletions.
7 changes: 4 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ INFO[4804] Killing pod chaoskube/nginx-701339712-51nt8
...
```

`chaoskube` allows to filter target pods [by namespaces, labels and annotations](#filtering-targets) as well as [exclude certain weekdays or times of day](#limit-the-chaos) from chaos.
`chaoskube` allows to filter target pods [by namespaces, labels and annotations](#filtering-targets) as well as [exclude certain weekdays, times of day and days of a year](#limit-the-chaos) from chaos.

## How

Expand Down Expand Up @@ -140,9 +140,9 @@ spec:

## Limit the Chaos

You can limit the time when chaos is introduced by weekdays, time periods of a day or both.
You can limit the time when chaos is introduced by weekdays, time periods of a day, day of a year or all of them together.

Add a comma-separated list of abbreviated weekdays via the `--excluded-weekdays` options and/or a comma-separated list of time periods via the `--excluded-times-of-day` option and specify a `--timezone` by which to interpret them.
Add a comma-separated list of abbreviated weekdays via the `--excluded-weekdays` options, a comma-separated list of time periods via the `--excluded-times-of-day` option and/or a comma-separated list of days of a year via the `--excluded-days-of-year` option and specify a `--timezone` by which to interpret them.

Use `UTC`, `Local` or pick a timezone name from the [(IANA) tz database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). If you're testing `chaoskube` from your local machine then `Local` makes the most sense. Once you deploy `chaoskube` to your cluster you should deploy it with a specific timezone, e.g. where most of your team members are living, so that both your team and `chaoskube` have a common understanding when a particular weekday begins and ends, for instance. If your team is spread across multiple time zones it's probably best to pick `UTC` which is also the default. Picking the wrong timezone shifts the meaning of a particular weekday by a couple of hours between you and the server.

Expand All @@ -156,6 +156,7 @@ Use `UTC`, `Local` or pick a timezone name from the [(IANA) tz database](https:/
| `--namespaces` | namespace selector to filter pods by | (all namespaces) |
| `--excluded-weekdays` | weekdays when chaos is to be suspended, e.g. "Sat,Sun" | (no weekday excluded) |
| `--excluded-times-of-day` | times of day when chaos is to be suspended, e.g. "22:00-08:00" | (no times of day excluded) |
| `--excluded-days-of-year` | days of a year when chaos is to be suspended, e.g. "Apr1,Dec24" | (no days of year excluded) |
| `--timezone` | timezone from tz database, e.g. "America/New_York", "UTC" or "Local" | (UTC) |
| `--dry-run` | don't kill pods, only log what would have been done | true |

Expand Down
26 changes: 20 additions & 6 deletions chaoskube/chaoskube.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ type Chaoskube struct {
ExcludedWeekdays []time.Weekday
// a list of time periods of a day when termination is suspended
ExcludedTimesOfDay []util.TimePeriod
// a list of days of a year when termination is suspended
ExcludedDaysOfYear []time.Time
// the timezone to apply when detecting the current weekday
Timezone *time.Location
// an instance of logrus.StdLogger to write log messages to
Expand All @@ -50,21 +52,26 @@ var (
msgWeekdayExcluded = "weekday excluded"
// msgTimeOfDayExcluded is the log message when termination is suspended due to the time of day filter
msgTimeOfDayExcluded = "time of day excluded"
// msgDayOfYearExcluded is the log message when termination is suspended due to the day of year filter
msgDayOfYearExcluded = "day of year excluded"
)

// New returns a new instance of Chaoskube. It expects a kubernetes client, a
// label, annotation and/or namespace selector to reduce the amount of affected
// pods as well as whether to enable dryRun mode and a seed to seed the randomizer
// with. You can also provide a list of weekdays and corresponding time zone when
// chaoskube should be inactive.
func New(client kubernetes.Interface, labels, annotations, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, timezone *time.Location, logger log.FieldLogger, dryRun bool) *Chaoskube {
// New returns a new instance of Chaoskube. It expects:
// * a Kubernetes client to connect to a Kubernetes API
// * label, annotation and/or namespace selectors to reduce the amount of possible target pods
// * a list of weekdays, times of day and/or days of a year when chaos mode is disabled
// * a time zone to apply to the aforementioned time-based filters
// * a logger implementing logrus.FieldLogger to send log output to
// * whether to enable/disable dry-run mode
func New(client kubernetes.Interface, labels, annotations, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, excludedDaysOfYear []time.Time, timezone *time.Location, logger log.FieldLogger, dryRun bool) *Chaoskube {
return &Chaoskube{
Client: client,
Labels: labels,
Annotations: annotations,
Namespaces: namespaces,
ExcludedWeekdays: excludedWeekdays,
ExcludedTimesOfDay: excludedTimesOfDay,
ExcludedDaysOfYear: excludedDaysOfYear,
Timezone: timezone,
Logger: logger,
DryRun: dryRun,
Expand Down Expand Up @@ -146,6 +153,13 @@ func (c *Chaoskube) TerminateVictim() error {
}
}

for _, d := range c.ExcludedDaysOfYear {
if d.Day() == now.Day() && d.Month() == now.Month() {
c.Logger.WithField("dayOfYear", now.Format(util.YearDay)).Debug(msgDayOfYearExcluded)
return nil
}
}

victim, err := c.Victim()
if err == errPodNotFound {
c.Logger.Debug(msgVictimNotFound)
Expand Down
86 changes: 84 additions & 2 deletions chaoskube/chaoskube_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ func (suite *Suite) TestNew() {
namespaces, _ = labels.Parse("qux")
excludedWeekdays = []time.Weekday{time.Friday}
excludedTimesOfDay = []util.TimePeriod{util.TimePeriod{}}
excludedDaysOfYear = []time.Time{time.Now()}
)

chaoskube := New(
Expand All @@ -48,6 +49,7 @@ func (suite *Suite) TestNew() {
namespaces,
excludedWeekdays,
excludedTimesOfDay,
excludedDaysOfYear,
time.UTC,
logger,
false,
Expand All @@ -60,6 +62,7 @@ func (suite *Suite) TestNew() {
suite.Equal("qux", chaoskube.Namespaces.String())
suite.Equal(excludedWeekdays, chaoskube.ExcludedWeekdays)
suite.Equal(excludedTimesOfDay, chaoskube.ExcludedTimesOfDay)
suite.Equal(excludedDaysOfYear, chaoskube.ExcludedDaysOfYear)
suite.Equal(time.UTC, chaoskube.Timezone)
suite.Equal(logger, chaoskube.Logger)
suite.Equal(false, chaoskube.DryRun)
Expand Down Expand Up @@ -102,6 +105,7 @@ func (suite *Suite) TestCandidates() {
namespaceSelector,
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
false,
)
Expand Down Expand Up @@ -134,6 +138,7 @@ func (suite *Suite) TestVictim() {
labels.Everything(),
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
false,
)
Expand All @@ -150,6 +155,7 @@ func (suite *Suite) TestNoVictimReturnsError() {
labels.Everything(),
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
false,
)
Expand All @@ -176,6 +182,7 @@ func (suite *Suite) TestDeletePod() {
labels.Everything(),
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
tt.dryRun,
)
Expand Down Expand Up @@ -210,6 +217,7 @@ func (suite *Suite) TestTerminateVictim() {
for _, tt := range []struct {
excludedWeekdays []time.Weekday
excludedTimesOfDay []util.TimePeriod
excludedDaysOfYear []time.Time
now func() time.Time
timezone *time.Location
remainingPodCount int
Expand All @@ -218,6 +226,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
1,
Expand All @@ -226,6 +235,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Friday},
[]util.TimePeriod{},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
2,
Expand All @@ -234,6 +244,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{afternoon},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
2,
Expand All @@ -242,6 +253,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Friday},
[]util.TimePeriod{},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(24 * time.Hour) },
time.UTC,
1,
Expand All @@ -250,6 +262,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Friday},
[]util.TimePeriod{},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(7 * 24 * time.Hour) },
time.UTC,
2,
Expand All @@ -258,6 +271,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{afternoon},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(+2 * time.Hour) },
time.UTC,
1,
Expand All @@ -266,6 +280,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{afternoon},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(+24 * time.Hour) },
time.UTC,
2,
Expand All @@ -274,6 +289,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Friday},
[]util.TimePeriod{},
[]time.Time{},
ThankGodItsFriday{}.Now,
australia,
1,
Expand All @@ -282,6 +298,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{afternoon},
[]time.Time{},
ThankGodItsFriday{}.Now,
australia,
1,
Expand All @@ -290,6 +307,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Monday, time.Friday},
[]util.TimePeriod{},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
2,
Expand All @@ -298,6 +316,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{morning, afternoon},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
2,
Expand All @@ -306,6 +325,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{midnight},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(-15 * time.Hour) },
time.UTC,
2,
Expand All @@ -314,6 +334,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{midnight},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(-17 * time.Hour) },
time.UTC,
1,
Expand All @@ -322,17 +343,75 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{midnight},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(-13 * time.Hour) },
time.UTC,
1,
},
// this day of year is excluded, no pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
ThankGodItsFriday{}.Now(), // today
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
2,
},
// this day of year in year 0 is excluded, no pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
time.Date(0, 9, 24, 0, 00, 00, 00, time.UTC), // same year day
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
2,
},
// matching works fine even when multiple days-of-year are provided, no pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
time.Date(0, 9, 25, 10, 00, 00, 00, time.UTC), // different year day
time.Date(0, 9, 24, 10, 00, 00, 00, time.UTC), // same year day
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
2,
},
// there is an excluded day of year but it's not today, one pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
time.Date(0, 9, 25, 10, 00, 00, 00, time.UTC), // different year day
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
1,
},
// there is an excluded day of year but the month is different, one pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
time.Date(0, 10, 24, 10, 00, 00, 00, time.UTC), // different year day
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
1,
},
} {
chaoskube := suite.setupWithPods(
labels.Everything(),
labels.Everything(),
labels.Everything(),
tt.excludedWeekdays,
tt.excludedTimesOfDay,
tt.excludedDaysOfYear,
tt.timezone,
false,
)
Expand All @@ -356,6 +435,7 @@ func (suite *Suite) TestTerminateNoVictimLogsInfo() {
labels.Everything(),
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
false,
)
Expand Down Expand Up @@ -406,13 +486,14 @@ func (suite *Suite) assertLog(level log.Level, msg string, fields log.Fields) {
}
}

func (suite *Suite) setupWithPods(labelSelector labels.Selector, annotations labels.Selector, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, timezone *time.Location, dryRun bool) *Chaoskube {
func (suite *Suite) setupWithPods(labelSelector labels.Selector, annotations labels.Selector, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, excludedDaysOfYear []time.Time, timezone *time.Location, dryRun bool) *Chaoskube {
chaoskube := suite.setup(
labelSelector,
annotations,
namespaces,
excludedWeekdays,
excludedTimesOfDay,
excludedDaysOfYear,
timezone,
dryRun,
)
Expand All @@ -430,7 +511,7 @@ func (suite *Suite) setupWithPods(labelSelector labels.Selector, annotations lab
return chaoskube
}

func (suite *Suite) setup(labelSelector labels.Selector, annotations labels.Selector, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, timezone *time.Location, dryRun bool) *Chaoskube {
func (suite *Suite) setup(labelSelector labels.Selector, annotations labels.Selector, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, excludedDaysOfYear []time.Time, timezone *time.Location, dryRun bool) *Chaoskube {
logOutput.Reset()

return New(
Expand All @@ -440,6 +521,7 @@ func (suite *Suite) setup(labelSelector labels.Selector, annotations labels.Sele
namespaces,
excludedWeekdays,
excludedTimesOfDay,
excludedDaysOfYear,
timezone,
logger,
dryRun,
Expand Down
Loading

0 comments on commit 46e70ef

Please sign in to comment.