Importer for 0.8.9 data via the CLI #3502

corylanou · 2015-07-29T15:11:09Z

This PR allows you to import an exported file from 0.8.9 (#3477).

Caveats

For the export/import to work, all requisites have to be met. For export, all series names in 0.8 should be in the following format:

<tagName>.<tagValue>.<tagName>.<tagValue>.<measurement>

for example:

az.us-west-1.host.serverA.cpu

or any number of tags

building.2.temperature

Additionally, the fields need to have a consistent type (all float64, int64, etc) for every write in 0.8. Otherwise they have the potential to fail writes in the import. See below for more information.

Running the import command

To import via the cli, you can specify the following command:

influx -import -path=metrics-default.gz -compressed

If the file is not compressed you can issue it without the -compressed flag:

influx -import -path=metrics-default

To redirect failed import lines to another file, run this command:

influx -import -path=metrics-default.gz -compressed > failures

It will import using the line protocol in batches of 5,000 lines per batch.

Understanding the results of the import

The batch will give some basic stats when finished:

2015/07/29 23:15:20 Processed 2 commands
2015/07/29 23:15:20 Processed 70207923 inserts
2015/07/29 23:15:20 Failed 29785000 inserts

Most inserts fail due to the following types of error:

2015/07/29 22:18:28 error writing batch:  write failed: field type conflict: input field "value" on measurement "metric" is type float64, already exists as type integer

This is due to the fact that in 0.8 a field could get created and saved as int or float types for independent writes. In 0.9 the field has to have a consistent type.

beckettsean · 2015-07-29T22:11:19Z

@corylanou how are errors reported? STDOUT?

corylanou · 2015-07-29T22:14:47Z

@beckettsean no, STDERR... get it? errors ... :-)

beckettsean · 2015-07-29T22:28:15Z

but of course...

:dunce_hat: should be an emoji, I need one.

dgnorton · 2015-07-31T18:32:49Z

client/influxdb.go

 // Query is used to send a command to the server. Both Command and Database are required.
 type Query struct {
 	Command  string
 	Database string
 }

+// ParseConnectionString will parse a string to create a valid connection URL
+func ParseConnectionString(path string, ssl bool) (url.URL, error) {


Could the path just require http://... or https://... and then no need for ssl bool?

It could, but then you have to parse and validate, and if they don't give you either, you aren't sure what they wanted (you could default to http, but that might not be right). Making them explicitly choose makes it their responsibility to get right. Otherwise, if they don't provide either, it's now on us to try to make the right decision without enough information.

I was thinking that http or https would be required but no strong feeling here.

pauldix · 2015-07-31T19:36:38Z

Looks mostly good. The one other thing I think would be good to add is to output any of the lines that fail to parse into a new file. That way the developer can write their own custom script to handle just those data points.

And if there are write timeouts that just can't be overcome, to output all the timed out data to a different file.

benbjohnson · 2015-07-31T19:40:55Z

cmd/influx/importer/v8.go

+
+	// Process the scanner
+	v8.processDDL(scanner)
+	v8.processDML(scanner)


Does the DDL and DML need to be processed in order? Right now it sends it to goroutines that are running concurrently and it seems like they could be run out-of-order.

Yeah, I might of over engineered that a little. I might refactor that to straight method calls instead after seeing the final product.

corylanou · 2015-07-31T19:41:20Z

@pauldix none of the lines can fail to parse as they were written out by the same line protocol code we use to ingest. I could check for it, but imo that would be the equivalent of a corrupt input file at that point.

Do you want them to specify a file for "failed writes" that we write to? Otherwise I can write failed writes to STDOUT and all other info to STDERR and they can just pipe failed writes to any file they want.

pauldix · 2015-07-31T19:43:15Z

@corylanou oh right, never mind what I said about the parse thing. For failed writes I think let them specify a file to output to? That way they can just try again and point it at that file.

benbjohnson · 2015-07-31T19:43:19Z

cmd/influx/main.go

+		} else {
+			c.Line.Close()
+			os.Exit(0)
+		}


You can drop the else since the if block will exit.

corylanou · 2015-08-03T19:01:05Z

@benbjohnson ok, I think I have your comments addressed. Please take another look when you get a chance. Thanks!

…d to it

…og/import file

corylanou · 2015-08-06T14:55:49Z

@pauldix this now writes all failed lines to STDOUT, and messages to STDERR so the end users can do what they want depending on what they care about.

pauldix · 2015-08-06T15:44:55Z

+1, just give @beckettsean the invocation example that outputs the failed writes to another file.

Importer for 0.8.9 data via the CLI

beckettsean · 2015-08-12T00:07:26Z

@corylanou is there any way to throttle or back off on the import? Most users will be running the import against a live 0.9.3 server.

corylanou added the 2 - Working label Jul 29, 2015

corylanou force-pushed the import branch from 38b8384 to fc46fd3 Compare July 29, 2015 20:21

corylanou changed the title ~~WIP - Importer for 0.8.9 data via the CLI~~ Importer for 0.8.9 data via the CLI Jul 30, 2015

corylanou force-pushed the import branch 2 times, most recently from 884b269 to fe05a53 Compare July 31, 2015 18:18

dgnorton reviewed Jul 31, 2015
View reviewed changes

benbjohnson reviewed Jul 31, 2015
View reviewed changes

cmd/influx/main.go

} else {

c.Line.Close()

os.Exit(0)

}

Copy link

Contributor

benbjohnson Jul 31, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can drop the else since the if block will exit.

corylanou force-pushed the import branch from fe05a53 to 2e3b595 Compare August 3, 2015 18:41

corylanou added 10 commits August 6, 2015 09:19

start of a v8 data importer via the cli

30a0ca0

add database/retention policy context switching. log failed imports

05a8c10

update client tests

c2f90ed

add godoc comments for v8 importer

b0b229b

update changelog

ea58609

refactoring based on feedback

76367d5

NewConfig should return a default config without needing params passe…

296f16d

…d to it

update useragent string for importer

0682c2a

minor fix to help for influx

b29f9a4

send failed import lines to stdout for users to redirect to another l…

96aaad4

…og/import file

corylanou force-pushed the import branch from c475cd1 to 96aaad4 Compare August 6, 2015 14:21

silence output when connecting

05fde32

corylanou added a commit that referenced this pull request Aug 6, 2015

Merge pull request #3502 from influxdb/import

08f84a2

Importer for 0.8.9 data via the CLI

corylanou merged commit 08f84a2 into master Aug 6, 2015

corylanou removed the 2 - Working label Aug 6, 2015

corylanou deleted the import branch August 6, 2015 15:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Importer for 0.8.9 data via the CLI #3502

Importer for 0.8.9 data via the CLI #3502

corylanou commented Jul 29, 2015

beckettsean commented Jul 29, 2015

corylanou commented Jul 29, 2015

beckettsean commented Jul 29, 2015

dgnorton Jul 31, 2015

corylanou Jul 31, 2015

dgnorton Jul 31, 2015

pauldix commented Jul 31, 2015

benbjohnson Jul 31, 2015

corylanou Jul 31, 2015

corylanou commented Jul 31, 2015

pauldix commented Jul 31, 2015

benbjohnson Jul 31, 2015

corylanou commented Aug 3, 2015

corylanou commented Aug 6, 2015

pauldix commented Aug 6, 2015

beckettsean commented Aug 12, 2015

Importer for 0.8.9 data via the CLI #3502

Importer for 0.8.9 data via the CLI #3502

Conversation

corylanou commented Jul 29, 2015

Caveats

Running the import command

Understanding the results of the import

beckettsean commented Jul 29, 2015

corylanou commented Jul 29, 2015

beckettsean commented Jul 29, 2015

dgnorton Jul 31, 2015

Choose a reason for hiding this comment

corylanou Jul 31, 2015

Choose a reason for hiding this comment

dgnorton Jul 31, 2015

Choose a reason for hiding this comment

pauldix commented Jul 31, 2015

benbjohnson Jul 31, 2015

Choose a reason for hiding this comment

corylanou Jul 31, 2015

Choose a reason for hiding this comment

corylanou commented Jul 31, 2015

pauldix commented Jul 31, 2015

benbjohnson Jul 31, 2015

Choose a reason for hiding this comment

corylanou commented Aug 3, 2015

corylanou commented Aug 6, 2015

pauldix commented Aug 6, 2015

beckettsean commented Aug 12, 2015