Python: De-duping in one line of code using pop method.

Submitted by john on Mon, 03/20/2017 - 16:26

Python LogoIn 1 line of code we can de-duplicate this list of first names:

dup = ['John','Jimmy','Carly','John','John', 'Michael','Jimmy','John', 'David', 'Lisa', 'John', 'John', 'John', 'Lisa', 'David']

Note that we do not need to import anything. How this works is it is altering the original list so if you have a HUGE list, it will carve it down. If you absolutely need all the dups to count them or something be sure to do that first because they will be gone with this technique. Most other techniques create a clean list external to the dirty list and append to it. This rips the duplicates out of the dirty list without creating a new list. Know that difference. 

[dup.pop(dup.index(x)) for x in sorted(dup) if dup.count(x) > 1]
print dup

OUTPUT: [’Carly’, ‘David’, ‘Jimmy’, ‘John’, ‘Lisa’, ‘Michael’]

Also note that the list is sorted in the loop but the output is not sorted. That is because sorted([]) is a temporary sort and [].sort() is a persistent sort. You will have to do a variable.sort() after you de-dup it if you want it sorted.