The client
was running Windows and the sort function in DOS did not seem to
have a -u parameter as it does in Linux.
Since the client already had PERL installed (for the other
script), I decided to write a sort by unique script in PERL. I
was quite surprised by the results. The file in question was
326MB* (a pipe-delimited scrape of the business listings on Yellow Pages.
Using time cat listings.csv | sort -u >> test.csv took
approximately 6 minutes and 30 seconds. Sorting the same file
with my PERL script took approximately 10 seconds.
The Linux sort function is written in C. I find this interesting
as C is generally much faster than PERL (although design is far more
important for optimization than is the speed of the language). Since
my little script obviously isn't the result of some ingenious design,
I think what this best illustrates is that certain languages are
best for certain jobs because of their inherent data-structures.
Some data-structures are simply better suited for certain jobs
and allow for simpler algorithms.
#!/usr/bin/perl
use strict;
(our $input, our $output) =@ARGV;
our %uniques = ();
open(INPUT, "< $input") or die "Cannot open input file $input...n";
open(OUTPUT, ">> $output") or print "Cannot open output file $output...n";
while(my $line = )
{ $uniques{$line} = $line; }
close(INPUT);
foreach my $key (sort (keys %uniques))
{ print OUTPUT $key; }
close(OUTPUT);
sub ksort()
{ $uniques{$b} <=> $uniques{$a}; }
* This file contained a large amount of redundancy - the resulting
output was only 1MB (.003 of the original).
