Allow a fixed number of sync dirs instead of removing by date

[taldcroft]

I ran into a problem with ccdm15eng data not syncing correctly. This was due to my local copy missing one sync dir that had since been removed from the primary sync archive because it was older than 60 days.

Ccdm15eng is dwell-mode telemetry that (mostly) only shows up when we have a momentum dump, which as been quite infrequent of late.

Testing

Regression

Passes unit tests (including sync tests)

Functional

$  python -m cheta.update_server_sync --content=acis4eng --max-days=1.0 --max-sync-dirs=3 --date-start=2019:001 --date-stop=2019:004
$  python -m cheta.update_server_sync --content=acis4eng --max-days=1.0 --max-sync-dirs=3 --date-stop=2019:005
$  python -m cheta.update_server_sync --content=acis4eng --max-days=1.0 --max-sync-dirs=3 --date-stop=2019:005
$  python -m cheta.update_server_sync --content=acis4eng --max-days=1.0 --max-sync-dirs=3 --date-stop=2019:006
$  python -m cheta.update_server_sync --content=acis4eng --max-days=1.0 --max-sync-dirs=3 --date-stop=2019:007
$ ls sync/acis4eng
2019-01-04T1308z/    2019-01-05T1246z/    2019-01-06T1227z/    index.ecsv           last_rows_5min.pkl   last_rows_daily.pkl

$ python -m cheta.update_server_sync --content=ccdm15eng --date-start=2018:001 --date-stop=2019:300
$ ls sync/ccdm15eng
2018-01-02T1454z/    2018-02-06T0129z/    2018-03-28T0945z/    2018-05-02T0034z/    2018-10-12T2005z/    2019-02-19T0801z/
2018-01-05T0612z/    2018-02-08T1456z/    2018-04-05T0725z/    2018-05-07T0045z/    2018-10-26T0407z/    2019-02-21T2001z/
2018-01-15T2130z/    2018-02-11T0523z/    2018-04-08T0142z/    2018-05-15T0350z/    2018-10-28T1500z/    2019-04-13T0751z/
2018-01-18T1558z/    2018-02-16T1630z/    2018-04-10T1001z/    2018-05-23T0308z/    2018-10-31T0539z/    2019-07-01T0916z/
2018-01-21T0253z/    2018-02-19T1100z/    2018-04-13T0710z/    2018-06-29T0255z/    2018-12-07T0700z/    2019-09-08T0029z/
2018-01-24T0225z/    2018-02-27T0207z/    2018-04-15T1928z/    2018-08-18T0603z/    2019-01-15T1800z/    index.ecsv
2018-01-26T1015z/    2018-03-01T2221z/    2018-04-18T1216z/    2018-09-10T1927z/    2019-01-29T0201z/    last_rows_5min.pkl
2018-01-29T0609z/    2018-03-07T0500z/    2018-04-26T1217z/    2018-09-26T1856z/    2019-01-31T2001z/    last_rows_daily.pkl
2018-02-01T0123z/    2018-03-15T0354z/    2018-04-29T0338z/    2018-10-09T2010z/    2019-02-03T0901z/

$ python -m cheta.update_server_sync --content=ccdm15eng --max-sync-dirs=10 --date-stop=2019:301
2020-01-09 15:29:28,319 No updates available for content ccdm15eng
2020-01-09 15:29:28,444 Removing sync directory /Users/aldcroft/git/eng_archive/sync/ccdm15eng/2018-01-02T1454z
2020-01-09 15:29:28,445 Removing sync directory /Users/aldcroft/git/eng_archive/sync/ccdm15eng/2018-01-05T0612z
2020-01-09 15:29:28,446 Removing sync directory 
...
2020-01-09 15:29:28,486 Removing sync directory /Users/aldcroft/git/eng_archive/sync/ccdm15eng/2018-10-31T0539z
2020-01-09 15:29:28,487 Writing 10 row(s) to index file /Users/aldcroft/git/eng_archive/sync/ccdm15eng/index.ecsv

$ cat sync/ccdm15eng/index.ecsv 
# %ECSV 0.9
# ---
# datatype:
# - {name: date_id, datatype: string}
# - {name: filetime0, datatype: int64}
# - {name: filetime1, datatype: int64}
# - {name: row0, datatype: int64}
# - {name: row1, datatype: int64}
# schema: astropy-2.0
date_id filetime0 filetime1 row0 row1
2018-12-07T0700z 660553253 660553253 133632 134656
2019-01-15T1800z 663962453 663962453 134656 135680
2019-01-29T0201z 665114487 665114487 135680 140416
2019-01-31T2001z 665352090 665352090 140416 145152
2019-02-03T0901z 665571686 665571686 145152 149888
2019-02-19T0801z 666950500 666950500 149888 154496
2019-02-21T2001z 667166488 667166488 154496 159232
2019-04-13T0751z 671529085 671529085 159232 160256
2019-07-01T0916z 678359817 678359817 160256 161280
2019-09-08T0029z 684289795 684289795 161280 167168

[taldcroft]

I'd like to get this into 2020.1 if possible.

[taldcroft]

Never mind the 2020.1 request. I just installed this to /proj/sot/ska3/test where it will be run in normal cron processing (for sync repo generation ONLY). Version is Ska.engarchive-4.47.4.dev7+gcdef460.

I will let this soak for a couple of weeks and then we can promote. It doesn't really matter since sync repo processing is still happening out of test, but at some point we should just put this into flight. (But having some agility to update that sync processing is still proving useful).

[taldcroft]

The two-week soak has turned into 8 weeks. All looks good. I checked the files and see the expected result of this PR. No processing errors have turned up. Therefore I am merging this.