Ensure all lines fed to marian are well formed

[XapaJIaMnu]

No description provided.

[jelmervdl]

Suggested alternative:

while True:
	line = self._fh.readline()
	if line == '':
		raise StopIteration
	self.line += 1

	# assert that the line is well formed, meaning non of the fields is the empty string
	# If not, try to get a new line from the corpus
	if any(field == '' for field in line.rstrip('\r\n').split('\t')):
		logging.warning("Empty field in {self.dataset.name}")
		continue

	return line

Not sure about the warning(), but I'm afraid that you'll have to do a lot of debugging otherwise to figure out what is going on. Ideally we would tell you which line, but since the data comes from the shuffled output, that line number would be meaningless.

Maybe shuffle.py should have a line number on the first column, so you can trace the data. But that's a lot of string parsing just for debugging once in a while.

Also I'm hinting to move all our info printing to logging so we can give the user a bit more control over what is printed. And we can filter repeated warnings more easily.

[jelmervdl]

This continue will only skip the for field in testline loop, right? Not go to the next line?

[XapaJIaMnu]

Yep... That should be breakout 🤦

[jelmervdl]

Should be \ instead of /. Also the strip() will mess things up here because empty columns will be stripped entirely, yielding just a row with fewer fields. Only when a column that is surrounded by non-empty columns is empty, this will be found.

[XapaJIaMnu]

🤦

[XapaJIaMnu]

Note to self, do not push PRs without sleeping well.

I don't think line number is necessary when outputting the erroneous sentence, because it can very easily be later identified via grep (i've done this several times). Logging is nice though

[XapaJIaMnu]

We should start moving to logger, but annoyingly it doesn't have a LOG_ONCE feature AND on top of that we'd probably have to redo all the test outputs...

[XapaJIaMnu]

This is still not fool proof. The reader doesn't know how many fields there are, and if we are missing a field altogether this would fail...