Enforce retention policies #1562

otoolep · 2015-02-10T23:58:33Z

No description provided.

otoolep · 2015-02-11T00:02:08Z

This is work-in-progress, as I have some questions about this untested implementation, and also need to add unit-testing.

otoolep · 2015-02-11T00:06:13Z

A summary of the implementation follows.

Each data node runs the "retention policy enforcement" loop, and looks for shard groups within each Retention Policy, which have an "end time", plus the retention period, earlier than now. For all matching shard groups, a command is sent to the Raft cluster, requesting deletion of this shard group. Each data node, upon receipt of this command, deletes any shards it has associated with this shard group, and updates its local copy of the metastore.

Because all data nodes will run the loop, it does mean that multiple deletion commands may be generated for a given shard group (when its end time passes) -- the data nodes account for this by simply ignoring requests to delete non-existent shard groups.

otoolep · 2015-02-11T01:41:10Z

server.go

+	}
+
+	// Remove from metastore.
+	rp.removeShardGroupByID(c.ID)


It is these lines I have questions about for @benbjohnson around serlialization of database meta to metastore.

Answered by @benbjohnson and fixed by beefdee

otoolep · 2015-02-11T01:41:43Z

Unit tests added, green build.

Still need to perform some system testing.

pauldix · 2015-02-11T14:41:09Z

cmd/influxd/config.go

+		MaxOpenShards         int                       `toml:"max-open-shards"`
+		PointBatchSize        int                       `toml:"point-batch-size"`
+		WriteBatchSize        int                       `toml:"write-batch-size"`
+		Engines               map[string]toml.Primitive `toml:"engines"`


Should just remove the WriteBufferSize, MaxOpenShards, PointBatchSize, WriteBatchSize, and Engines members on the Data struct

pauldix · 2015-02-11T14:41:26Z

One small comment, but other than that LGTM

If shard groups were stored in a map by ID, this lookup could be sped up.

otoolep · 2015-02-11T18:23:08Z

I performed a system test this morning, using an actual influxd process. Code looks good. I created a long (1 hour) and short (1 minute) retention policy, and wrote data to each. Within a couple of minutes (system had a retention check interval of 30 seconds) the short policy's shard groups were deleted. Log follows.

[srvr] 2015/02/11 10:18:04 broker enforcing retention policies with check interval of 30s
...
[srvr] 2015/02/11 10:18:34 retention policy enforcement check commencing
[srvr] 2015/02/11 10:19:04 retention policy enforcement check commencing
[srvr] 2015/02/11 10:19:34 retention policy enforcement check commencing
[srvr] 2015/02/11 10:20:04 retention policy enforcement check commencing
[srvr] 2015/02/11 10:20:04 shard group 1, retention policy short, database foo due for deletion
[srvr] 2015/02/11 10:20:34 retention policy enforcement check commencing

Queries of data in short returned no results, but queries of data in long did.

otoolep · 2015-02-11T18:46:11Z

This appears to be an integration test failure, I haven't seen it before. The integration tests may still be racy.

I'll kick Travis and try again.

otoolep · 2015-02-11T18:50:44Z

Green build, we appear to have some racy tests. This is currently captured by issue #1556.

Enforce retention policies

pauldix · 2015-02-25T18:51:29Z

server.go

+	for _, db := range s.databases {
+		for _, rp := range db.policies {
+			for _, g := range rp.shardGroups {
+				if g.EndTime.Add(rp.Duration).Before(time.Now()) {


Should be using UTC here, otherwise it's going to be off if their server isn't set to that.

Fixed the task id path, Added RunController instance to pAdapter

Fixes merge issue from #1562

otoolep added the 2 - Working label Feb 10, 2015

otoolep force-pushed the enforce_retention_policies branch 4 times, most recently from 388d3dc to cb3d19f Compare February 11, 2015 01:38

otoolep reviewed Feb 11, 2015
View reviewed changes

otoolep added review and removed 2 - Working labels Feb 11, 2015

otoolep force-pushed the enforce_retention_policies branch 3 times, most recently from bd478c8 to f544bea Compare February 11, 2015 05:57

otoolep self-assigned this Feb 11, 2015

pauldix reviewed Feb 11, 2015
View reviewed changes

otoolep and others added 12 commits February 11, 2015 09:54

Add config control for retention policies

61e1fa0

Add stubbed enable of retention policy enforcement

50ef167

Fix minor typo in unit test comment

20bbbe7

Add function to return shard group by ID

37647cc

If shard groups were stored in a map by ID, this lookup could be sped up.

Implement "delete shard group" command

6a20100

Detect shard groups due for deletion

65edd6b

Close shard before deletion

91f8632

Explain why shard deletion may fail

ddb57b4

Unit-test deletion of shard groups

8dae2c2

Refactor retention policy enforcement for testing

8cb06c5

Unit test retention policy enforcement

0c8ded4

Retention policy and database were reversed in log

19cd550

otoolep force-pushed the enforce_retention_policies branch from f544bea to 19cd550 Compare February 11, 2015 18:23

Remove obsolete data config options

3b64f34

otoolep added a commit that referenced this pull request Feb 11, 2015

Merge pull request #1562 from influxdb/enforce_retention_policies

af41030

Enforce retention policies

otoolep merged commit af41030 into master Feb 11, 2015

otoolep removed the review label Feb 11, 2015

otoolep deleted the enforce_retention_policies branch February 11, 2015 18:51

pauldix reviewed Feb 25, 2015
View reviewed changes

mark-rushakoff pushed a commit that referenced this pull request Jan 11, 2019

Merge pull request #1562 from bednar/task_service

31518b7

Fixed the task id path, Added RunController instance to pAdapter

mark-rushakoff pushed a commit that referenced this pull request Jan 11, 2019

fix(http): update task test to use :id

edfb323

Fixes merge issue from #1562

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enforce retention policies #1562

Enforce retention policies #1562

otoolep commented Feb 10, 2015

otoolep commented Feb 11, 2015

otoolep commented Feb 11, 2015

otoolep Feb 11, 2015

otoolep Feb 11, 2015

otoolep commented Feb 11, 2015

pauldix Feb 11, 2015

pauldix commented Feb 11, 2015

otoolep commented Feb 11, 2015

otoolep commented Feb 11, 2015

otoolep commented Feb 11, 2015

pauldix Feb 25, 2015

Enforce retention policies #1562

Enforce retention policies #1562

Conversation

otoolep commented Feb 10, 2015

otoolep commented Feb 11, 2015

otoolep commented Feb 11, 2015

otoolep Feb 11, 2015

Choose a reason for hiding this comment

otoolep Feb 11, 2015

Choose a reason for hiding this comment

otoolep commented Feb 11, 2015

pauldix Feb 11, 2015

Choose a reason for hiding this comment

pauldix commented Feb 11, 2015

otoolep commented Feb 11, 2015

otoolep commented Feb 11, 2015

otoolep commented Feb 11, 2015

pauldix Feb 25, 2015

Choose a reason for hiding this comment