In this article
A mailing list with the same address three times. A keyword sheet where half the rows repeat. A class list built from two exports. Duplicates creep into every list that is copied from more than one place, and checking them by eye does not work past about thirty lines. The Duplicate Line Remover does it in one click. It is simple, but its four options change the answer in ways that are easy to miss, so this guide walks through each one with real examples.
How to use it, step by step
- Open the Duplicate Line Remover and paste your list, one item per line. The Try a sample button fills in a small example if you want to see it work first.
- Leave the switches as they are for a first run. The defaults suit most lists: ignore case, trim spaces, drop blank lines.
- Press Remove Duplicates.
- Read the three counts: Original, Unique and Removed. If Removed is zero, the list had no duplicates under these settings.
- Copy the Unique lines result. If something looks wrong, change a switch and press the button again.
What the four switches do
The tool always works in the same order: first it trims (if that is on), then it drops blank lines (if that is on), and only then does it compare lines. That order explains most surprises.
| Switch | Default | When on | When off |
|---|---|---|---|
| Ignore case | On | “Apple” and “apple” count as the same line. The first spelling you typed is the one kept. | “Apple” and “apple” are different lines and both stay. |
| Trim spaces | On | Spaces at the start and end of each line are removed first, so “apple ” and “apple” match. Kept lines come out without those spaces. | A trailing space makes a line different, even though you cannot see it. |
| Remove blank lines | On | Empty lines are deleted before comparing. | The first blank line is kept and later blank lines are removed as duplicates of it. |
| Keep only lines that appear once | Off | Every line that occurs more than once is deleted, including its first copy. Only true one-offs remain. | Normal mode: each line is kept once, at its first position. |
Ignore case compares lower-case versions of the lines, but the output keeps the original capital letters of the line it keeps.
A worked example you can check
The sample list on the page has six lines. Here is what each setting does to it:
| List | Settings | Result | Original / Unique / Removed |
|---|---|---|---|
| apple, banana, Apple, apple, cherry, banana | Defaults (ignore case on) | apple, banana, cherry | 6 / 3 / 3 |
| Same list | Ignore case off | apple, banana, Apple, cherry | 6 / 4 / 2 |
| Same list | Ignore case on, Keep only lines that appear once on | cherry | 6 / 1 / 5 |
| Same list | Ignore case off, Keep only lines that appear once on | Apple, cherry | 6 / 2 / 4 |
In the “only once” mode, apple (three copies) and banana (two copies) vanish completely. That is the point of the mode: it shows you the lines that have no twin.
- “Give me a clean list with no repeats”
- First copy stays, later copies go
- Use it for mailing lists, keywords, names
- “Which lines appear in this list just once?”
- Every copy of a repeated line goes, even the first
- Use it to find odd ones out between two lists
A handy trick: paste list A and list B together, turn on only-once mode, and what remains is every line that was in just one of them.
Reading the three counts
The counts are easy to misread, so it helps to know how they are made. Original is the number of lines the tool actually compared, which is *after* trimming and after blank lines were removed. Unique is how many lines are in the result. Removed is Original minus Unique.
That means blank lines you had in your paste are not part of Removed when the blank-line switch is on. If you paste 120 lines and 10 of them are empty, Original shows 110. This is not a bug: the empty lines were taken out before the count began. If you want a count of the blanks too, compare the number of lines you pasted with Original.
What it will not catch
The tool compares lines as whole units. Two lines are the same only if they match character for character, after the optional trim and case step. Everything else counts as different, even if a person would say it is the same.
If a list still has repeats after cleaning, look for one of these four causes first.
Preparing a list so nothing slips through
- 1One item per lineSplit anything separated by commas or semicolons onto its own line.
- 2Fix the spacingRun the Whitespace Cleaner so double spaces and odd gaps do not hide duplicates.
- 3Remove duplicatesUse the defaults. Look at the Removed count and see if it is what you expected.
- 4Sort if you need toThe result keeps your order. Put it in order afterwards with the Line Sorter.
- 5Count the resultCheck the final number of lines with the Word Counter.
Doing the spacing before the duplicates step is what catches the repeats that look identical on screen but are not.
If you are comparing repeated words inside a paragraph rather than repeated lines, the Duplicate Word Finder is the right tool. To turn a tidy list back into a single comma-separated line, the Line Break Remover joins lines with a comma, and its guide explains the options.
Good uses, and one warning
- Mailing and contact lists: merge two exports, then remove repeated addresses before you send.
- Keywords and tags: combine lists from several sources into one clean set. The Hashtag Generator output often has repeats when you run it twice.
- Student or attendee names: one name per line, remove repeats, then use the Random Name Picker so nobody gets two chances.
- Logs and URLs: shrink a long list of repeated entries into the distinct ones.
- Spreadsheets: copy a single column out of Excel, paste it here, and paste the clean result back. For files, the PDF to Excel guide shows how to get a table into a sheet first.
Privacy: the list is processed in your browser, and Toolcloak does not receive it. That matters for customer lists and email addresses, which you should not paste into a site that uploads them.
Questions people ask
How do I remove duplicate lines online?
Open the Duplicate Line Remover, paste your list with one item per line, and press Remove Duplicates. The first copy of every line is kept in its original order, and you can copy the result.
Duplicate lines kaise hataye?
Duplicate Line Remover kholiye, list paste kijiye (har line mein ek item), aur Remove Duplicates dabaiye. Har line ki pehli copy rahegi aur baaki hat jayengi. Upar Original, Unique aur Removed ki ginti dikhegi.
Does it keep the original order?
Yes. It keeps the first copy of each line where it first appeared and removes the later copies. Nothing is sorted. Use a line sorter afterwards if you want alphabetical order.
Why were “Apple” and “apple” treated as the same line?
Ignore case is on by default, so lines are compared in lower case. The first spelling is kept. Turn Ignore case off if you want capital letters to count as a difference.
What does “Keep only lines that appear once” do?
It removes every line that occurs more than once, including its first copy, so only lines with no twin remain. It is useful for finding the odd ones out between two lists.
Why does Original show fewer lines than I pasted?
Original is counted after trimming and blank-line removal. If you pasted blank lines and the Remove blank lines switch is on, they are taken out before the count begins.
Why were two lines that look the same not removed?
They differ in something you cannot see, usually a double space inside the line, a different accent, or a spelling variant. Trim only removes spaces at the ends. Clean inner spaces first, then run it again.
Is my list uploaded anywhere?
No. The list is processed in your browser and is not sent to Toolcloak, which is important for email lists and customer names.