18_Text Processing

awk

delete all repeated lines in a file, unless it is a newline:

awk 'NF == 0 || !seen[$0]++' input_file > output_file