Remove Duplicate Words

Clean up repeated words from your text

0Chars

0Words

0Lines

Keep each word once

Repeated words are common in keyword lists, tag collections, scraped text and rough drafts. Remove Duplicate Words reads your text word by word and keeps the first occurrence of each, dropping later repeats. The words stay in the order they first appeared, so the result is a de-duplicated version of what you pasted rather than a sorted or reshuffled one.

How to use it

Enter your text in Input Text. Sample loads two lines full of doubled words, Paste pulls from the clipboard, File reads a .txt file and Clear empties the box.
Set Case-sensitive. When it is ticked, Apple and apple count as different words. Untick it to treat them as the same.
Set Preserve lines. It is ticked by default.
Click Remove Duplicates.
The summary shows how many duplicate words were removed, with Before and After counts. Use Copy or .TXT to save cleaned-words.txt.

Line mode versus whole-text mode

With Preserve lines on, each line is treated separately. A word can appear once on every line, so the line structure of your text survives.
With it off, the whole text is treated as one stream. A word is kept only the first time it appears anywhere, and all line breaks are replaced by single spaces, giving you one long line.

Realistic uses

Cleaning a pasted keyword list so each term appears once before you submit it to an ad or SEO tool.
Reducing a set of tags or labels gathered from several sources.
Fixing accidental doubled words such as 'the the' in a draft, when you are working on short snippets.
Building a quick vocabulary list from a passage.

Cautions

Words are split on whitespace only, so punctuation stays attached: dog and dog, are different words, and both will be kept. Strip punctuation first for a stricter result. The tool cannot tell a mistake from a legitimate repeat, so running it on ordinary sentences will also delete common words like the, and or of after their first use, which makes prose read badly. It suits lists better than paragraphs. Extra spaces between words are collapsed into one in the output. The processing runs in your browser page and leaves the input box untouched, so you can try both line modes on the same text.

Related Tools

Related Tools

Frequently Asked Questions

What does Preserve lines change?

When ticked, duplicates are checked within each line, so a word can appear once per line and your line breaks stay. When unticked, the whole text is checked as one and the result becomes a single line with spaces between the words.

Is matching case-sensitive?

By default, yes: Apple and apple are treated as different words and both are kept. Untick the Case-sensitive checkbox to treat them as the same word, in which case the first version seen is retained.

Why are dog and dog, both kept?

Words are separated by whitespace, so punctuation stays attached to the word. A trailing comma or period makes it a different token. Remove punctuation from the text first if you want those counted as the same word.

Will it damage normal sentences?

It can. It removes every repeat, including legitimate ones like the, and or of, so paragraphs of prose can become hard to read. It works best on keyword lists, tags and other lists of words.

Does it keep the original word order?

Yes. Words are output in the order in which they first appeared, with later repeats dropped. Nothing is sorted. Extra spaces between the remaining words are reduced to a single space.