How long it will take to sort uniq a 62GB file? [closed]
For 5 hours I ran the command sort file1 | uniq > file2 on a 62GB file and I'm just wondering how long it will take.
I have a Intel® Core™ i7-4510U CPU @ 2.00GHz × 4 with 8GB RAM.
It will be faster if I'll run sort --parallel=4 -uo file2 file1?
1 Answer
It will take a long time. The exact time depends on a lot of things, like whether the file has a lot of duplicate lines or whether it's already sorted or very disordered. There's no way to tell from your system specs.
Yes, the last command will be much faster, especially since it can drop duplicates as it works rather than having to hang on to them. However, you need file2 file1 on the end since the argument to -o's option comes right after -o.
See also another answer that adds in nice and ionice to speed things up at the expense of slowing everything else down (perhaps to the point of freezing other tasks).
