Mind Mining Medium

archives
May 28, 2009

How To: Remove Duplicate Lines From A File

Sure you can load up a bulky editor and use its tools to do that, but why not do it in much faster in bash with awk.

In Bash:

awk '!x[$0]++' in.txt > out.txt

Wasn’t that easy?