Is the data small enough that it will all fit in memory?
Read data, save data for each "ID", discard any duplicates, write out new file from saved data.