Instances shutting down prematurely and not following the scheduled period

[JCKrause]

I have a schedule with two periods. Period 1 shuts down any instances associated to it everyday at 17:00 local time.
Period 2 turns on any instances associated to it on the first Thursday of the month at 18:50 local time and turns off the instance at 23:00 local time.

Period 1 works correctly.

Period 2 turns on the instances correctly. However the instances turn off at 20:00 local time.

The schedule and the Dynamo Config are set to the US/Eastern timezone.

Schedule configuration
Off1700x7LP1:
Type: Custom::ServiceInstanceSchedule
Properties:
Name: Off1700x7-LP1
NoStackPrefix: True
Description: Manual start, shutdown at 17:00 everyday, maintenance window 18:50 - 23:00 the 1st Thursday of the month
ServiceToken: !ImportValue asap-instance-scheduler-ServiceInstanceScheduleServiceToken
Timezone: US/Eastern
Periods:
- Description: Shutdown time
EndTime: '17:00'
- Description: Maintenance Window
BeginTime: '18:50'
EndTime: '23:00'
WeekDays: Thu#1

DynamoDB Config
{
"create_rds_snapshot": {
"BOOL": true
},
"default_timezone": {
"S": "US/Eastern"
},
"name": {
"S": "scheduler"
},
"regions": {
"SS": [
"us-east-1"
]
},
"schedule_clusters": {
"BOOL": false
},
"schedule_lambda_account": {
"BOOL": true
},
"scheduled_services": {
"SS": [
"ec2"
]
},
"started_tags": {
"S": "SchedulerMessage=Started by {scheduler} on {year}/{month}/{day} at {hour}:{minute} {timezone}"
},
"stopped_tags": {
"S": "SchedulerMessage=Stopped by {scheduler} on {year}/{month}/{day} at {hour}:{minute} {timezone}"
},
"tagname": {
"S": "Schedule"
},
"trace": {
"BOOL": false
},
"type": {
"S": "config"
},
"use_metrics": {
"BOOL": true
}
}

[gockle]

Hi @JCKrause This is an existing issue with instance scheduler, when a period is defined using a weekday value as Mon#1 scheduler is supposed to start/stop the instances based on the timezone configured but the scheduler ends up using UTC. Hence you can see as soon as the time in UTC is beyond the day in your case it is 20:00 US/Eastern (UTC-4:00) the instances will be stopped, this is issue in the solution, and is currently in the backlog and will be fixed in a future release, The work around for your scenario will be to convert the particular schedule and period(s) you require to be converted into UTC time and configured accordingly. Let me know if the work around resolves the issue.

[JCKrause]

Hi @gockle I changed the scheduler DynamoDB default time zone to UTC and I have set the schedule to UTC. I tested the following schedule last night.
      Periods:
        - Description: Shutdown time
          EndTime: '21:00'
        - Description: Maintenance Window part 1
          BeginTime: '22:50'
          EndTime: '23:59'
          WeekDays: Tue#4
        - Description: Maintenance Window part 2
          BeginTime: '00:00'
          EndTime: '03:00'
          WeekDays: Wed#4

The instance shutdown at 21:00 (17:00 EDT) and started up at 22:50 (18:50 EDT) but never stopped at 03:00 (23:00 EDT).

Tonight I am going to try the following two schedules with the period 3 changes in the hopes that one of them will work.
Schedule 1
Periods:
- Description: Shutdown time
EndTime: '21:00'
- Description: Maintenance Window part 1
BeginTime: '22:50'
EndTime: '23:59'
WeekDays: Wed#4
- Description: Maintenance Window part 2
EndTime: '03:00'
WeekDays: Thu

Schedule 2
Periods:
- Description: Shutdown time
EndTime: '21:00'
- Description: Maintenance Window part 1
BeginTime: '22:50'
EndTime: '23:59'
WeekDays: Wed#4
- Description: Maintenance Window part 2
EndTime: '03:00'

[JCKrause]

Test results from last night:
Both instance turned off Wed at 21:00 UTC (17:00 EDT).
Both instances turned on Wed at 22:50 UTC (18:50 EDT).
Neither instance turned off Thu at 03:00 UTC (Wed 23:00 EDT)

I am going to try this schedule tonight to see if it makes difference.

Periods:

• Description: Maintenance Window part 2
  EndTime: '03:00'
• Description: Shutdown time
  EndTime: '21:00'
• Description: Maintenance Window part 1
  BeginTime: '22:50'
  EndTime: '23:59'
  WeekDays: Wed#4

[gockle]

Hi @JCKrause Can you try using the SSM Maintenance Windows for setting up the weekly maintenance window, there are some known limitations for the solution with SSM Maintenance Windows, such as all the maintenance windows which are defined should be in UTC timezone, and the SSM Maintenance Window should be defined in the same region as the solution is deployed?

Refer to the implementation guide, section SSM Maintenance Window Field.

[JCKrause]

Hi @gockle I originally was using SSM Maintenance Windows and yes they do have some limitations with the time zone changes and time being defined in UTC. They also do not run automatically when an instance is turned on that has been off. I have moved away from SSM Maintenance Windows and am now using SSM State Manager and the AWS-RunPatchBaselineAssociation associations for all my patching and scanning. The down side to SSM State Manager is that it does not support window of time like Maintenance Windows as it only has a cron schedule time and the cron schedule time does not support specific day of the month (no # support). The up side is that whenever an instance turns on the association will run if it is not in the association resource list. So I need to turn on and off the instances that might be off during a given time. It just so happens that the time period is once a month from 18:50 ET to 23:00 ET which crosses 00:00 UTC.

I am hoping that reordering the periods will resolve the issue. I will update with tomorrow with the test results.

Let me know if you can think of anything else.

Thanks!

[JCKrause]

Tested using the following schedules last night:
Test 1:
      Periods:
        - Description: Maintenance Window part 2
          EndTime: '03:00'
          WeekDays: Fri
        - Description: Shutdown time
          EndTime: '21:00'
        - Description: Maintenance Window part 1
          BeginTime: '22:50'
          EndTime: '23:59'
          WeekDays: Thu#4
Test 2:
      Periods:
        - Description: Maintenance Window part 2
          EndTime: '03:00'
        - Description: Shutdown time
          EndTime: '21:00'
        - Description: Maintenance Window part 1
          BeginTime: '22:50'
          EndTime: '23:59'
          WeekDays: Thu#4

Both instance turned off Thu at 21:00 UTC (17:00 EDT).
Both instances turned on Thu at 22:50 UTC (18:50 EDT).
Neither instance turned off Fri at 03:00 UTC (Thu 23:00 EDT)

I am at a loss.

[gockle]

Hi @JCKrause

I was able to figure out the issue, i.e why the solution is shutting down at 20:00 PM EST, even when there is a period defined to ensure the instance should be available until 23:00 PM EST. In the method _build_schedule(..) in the file source/lambda/configuration/scheduler_config_builder.py. the datetime reference is not adjusting the datetime instance to the correct timezone, following is the line 215.

    # use current date and time for timezone of schedule
    current_schema_dt = dt.replace(tzinfo=SchedulerConfigBuilder._get_timezone(timezone))

This should be as follows,

  # use current date and time for timezone of schedule
  current_schema_dt = dt.now(SchedulerConfigBuilder._get_timezone(timezone))

We will include this fix in our next release, Let me know if you can modify the code and then update the schedules as you have defined in the first comment i.e.

• Use US/Eastern as the default timezone.
• Period 1 Thu#1 09:00 to 17:00
• Period 2 Thu#1 19:00 to 23:00

[JCKrause]

Hi @gockle

That worked perfectly!

Thank you,
@JCKrause