Remove Duplicate Lines
Detect and eliminate repeated lines from your text
Reduce a list to its unique entries
Lists collect repeats without anyone noticing: email addresses merged from two exports, keywords copied from several sources, log entries, URLs or IDs. This tool reads your text one line at a time and keeps only the first appearance of each line, preserving the original order. Nothing is sorted, so your list stays in the sequence you gave it, just without the repeats.
How to dedupe a list
- Paste one item per line into Input Lines, or bring in a .txt file using File. The header shows characters, words and lines as you type.
- Pick your matching rules with the three checkboxes: Case-insensitive, Trim whitespace and Ignore blank lines. All three begin ticked.
- Click Remove Duplicates.
- Read the summary, which reports how many duplicate lines were removed and how many unique lines remain, then compare the Before and After counts.
- Press Copy or .TXT to keep the result.
What the options do
- Case-insensitive treats Apple and apple as the same line. The version that appeared first is the one kept. Untick it when capitalization matters, such as with case-sensitive codes.
- Trim whitespace strips spaces from both ends of every line before comparing, so an entry with a trailing space matches its clean twin. The kept lines are output in their trimmed form.
- Ignore blank lines removes empty lines altogether. If you untick it, the first blank line survives and later blanks count as duplicates.
Practical uses
- Merging two mailing lists and keeping each address once.
- Cleaning a keyword or tag list before uploading it to an ad platform or CMS.
- Removing repeated error messages from a copied log so the distinct problems stand out.
- Deduplicating a list of domain names, SKUs or file names.
Things to keep in mind
Only whole lines are compared. Two lines that differ by even one character, such as a trailing period or a different quote style, count as different. With Trim whitespace turned off, invisible trailing spaces can stop lines from matching and leave what look like duplicates behind, so leave that box ticked unless indentation matters. Because the first occurrence wins, the tool is not the right choice when you want to keep the last occurrence. The processing runs as JavaScript in your browser page.
Related Tools
Related Tools