# Remove duplicate lines: how to clean a list of emails, names or keywords, and what Ignore case and Trim spaces really do

> Paste a list, press one button, and every repeated line disappears while the original order stays. Here is exactly how the four options decide what counts as a duplicate, what the Original, Unique and Removed counts mean, the kinds of near-duplicates it will not catch, and how to prepare a list so nothing slips through.

Published 2026-10-23 by Toolcloak. 6 minute read. Original: https://toolcloak.com/articles/remove-duplicate-lines-how-to-clean-a-list-of-emails-names-or-keywords-and-what-ignore-case-and-trim-spaces-really-do
Related free tool: [Duplicate Line Remover](https://toolcloak.com/tools/remove-duplicate-lines)

A mailing list with the same address three times. A keyword sheet where half the rows repeat. A class list built from two exports. Duplicates creep into every list that is copied from more than one place, and checking them by eye does not work past about thirty lines. The [Duplicate Line Remover](https://toolcloak.com/tools/remove-duplicate-lines) does it in one click. It is simple, but its four options change the answer in ways that are easy to miss, so this guide walks through each one with real examples.

> **In short**
> - Paste one item per line, press **Remove Duplicates**, and copy the **Unique lines** it gives back. The first copy of each line is kept and the **original order is preserved**. Nothing is sorted.
> - Four switches: **Ignore case** (on), **Trim spaces** (on), **Remove blank lines** (on) and **Keep only lines that appear once** (off).
> - Three counts appear above the result: **Original**, **Unique** and **Removed**. Original is counted *after* trimming and blank-line removal.
> - It matches **whole lines, exactly** (apart from case and outer spaces). It does not catch typos, accents or double spaces inside a line.
> - It works on lines, so a list separated by commas must be put one per line first.

## How to use it, step by step

1. Open the [Duplicate Line Remover](https://toolcloak.com/tools/remove-duplicate-lines) and paste your list, **one item per line**. The **Try a sample** button fills in a small example if you want to see it work first.
2. Leave the switches as they are for a first run. The defaults suit most lists: ignore case, trim spaces, drop blank lines.
3. Press **Remove Duplicates**.
4. Read the three counts: Original, Unique and Removed. If Removed is zero, the list had no duplicates under these settings.
5. Copy the **Unique lines** result. If something looks wrong, change a switch and press the button again.

Try it: [Duplicate Line Remover](https://toolcloak.com/tools/remove-duplicate-lines). Paste your list, one item per line, and press Remove Duplicates. Change the switches if the result is too strict or too loose.

## What the four switches do

The tool always works in the same order: first it **trims** (if that is on), then it **drops blank lines** (if that is on), and only then does it compare lines. That order explains most surprises.

| Switch | Default | When on | When off |
| --- | --- | --- | --- |
| Ignore case | On | “Apple” and “apple” count as the same line. The first spelling you typed is the one kept. | “Apple” and “apple” are different lines and both stay. |
| Trim spaces | On | Spaces at the start and end of each line are removed first, so “apple ” and “apple” match. Kept lines come out without those spaces. | A trailing space makes a line different, even though you cannot see it. |
| Remove blank lines | On | Empty lines are deleted before comparing. | The first blank line is kept and later blank lines are removed as duplicates of it. |
| Keep only lines that appear once | Off | Every line that occurs more than once is deleted, **including its first copy**. Only true one-offs remain. | Normal mode: each line is kept once, at its first position. |

*Ignore case compares lower-case versions of the lines, but the output keeps the original capital letters of the line it keeps.*

## A worked example you can check

The sample list on the page has six lines. Here is what each setting does to it:

| List | Settings | Result | Original / Unique / Removed |
| --- | --- | --- | --- |
| apple, banana, Apple, apple, cherry, banana | Defaults (ignore case on) | apple, banana, cherry | 6 / 3 / 3 |
| Same list | Ignore case **off** | apple, banana, Apple, cherry | 6 / 4 / 2 |
| Same list | Ignore case on, **Keep only lines that appear once** on | cherry | 6 / 1 / 5 |
| Same list | Ignore case off, Keep only lines that appear once on | Apple, cherry | 6 / 2 / 4 |

*In the “only once” mode, apple (three copies) and banana (two copies) vanish completely. That is the point of the mode: it shows you the lines that have no twin.*

**Two different questions**

| Normal mode | Only-once mode |
| --- | --- |
| Keep one of each | Drop anything repeated |
| “Give me a clean list with no repeats” | “Which lines appear in this list just once?” |
| First copy stays, later copies go | Every copy of a repeated line goes, even the first |
| Use it for mailing lists, keywords, names | Use it to find odd ones out between two lists |

*A handy trick: paste list A and list B together, turn on only-once mode, and what remains is every line that was in just one of them.*

## Reading the three counts

The counts are easy to misread, so it helps to know how they are made. **Original** is the number of lines the tool actually compared, which is *after* trimming and after blank lines were removed. **Unique** is how many lines are in the result. **Removed** is Original minus Unique.

That means blank lines you had in your paste are not part of Removed when the blank-line switch is on. If you paste 120 lines and 10 of them are empty, Original shows 110. This is not a bug: the empty lines were taken out before the count began. If you want a count of the blanks too, compare the number of lines you pasted with Original.

## What it will not catch

The tool compares lines as whole units. Two lines are the same only if they match character for character, after the optional trim and case step. Everything else counts as different, even if a person would say it is the same.

**Near-duplicates that survive**

| Box | What it means |
| --- | --- |
| Double spaces inside a line | “New  York” (two spaces) and “New York” are different. Trim only touches the ends. Clean inner spaces first with the [Whitespace Cleaner](/tools/whitespace-cleaner). |
| Typos and spelling variants | “Jon Smith” and “John Smith” stay as two lines. There is no fuzzy matching, so a person must decide. |
| Accents and look-alikes | “café” and “cafe” are different characters, and so are letters that merely look alike in different alphabets. |
| Several items on one line | “a@x.com, b@x.com” is one line. Put one item per line before you start, or the tool has nothing to compare. |

*If a list still has repeats after cleaning, look for one of these four causes first.*

> **Email addresses and case:** Ignore case is usually what you want for emails, because almost every mail service treats “Priya@Example.com” and “priya@example.com” as the same inbox. Strictly, the part before the @ is allowed to be case-sensitive, so if you ever manage an unusual system, turn the switch off and check by hand.

## Preparing a list so nothing slips through

**A clean-up order that works**

| When | What |
| --- | --- |
| 1 | One item per line. Split anything separated by commas or semicolons onto its own line. |
| 2 | Fix the spacing. Run the [Whitespace Cleaner](/tools/whitespace-cleaner) so double spaces and odd gaps do not hide duplicates. |
| 3 | Remove duplicates. Use the defaults. Look at the Removed count and see if it is what you expected. |
| 4 | Sort if you need to. The result keeps your order. Put it in order afterwards with the [Line Sorter](/tools/line-sorter). |
| 5 | Count the result. Check the final number of lines with the [Word Counter](/tools/word-counter). |

*Doing the spacing before the duplicates step is what catches the repeats that look identical on screen but are not.*

If you are comparing repeated **words** inside a paragraph rather than repeated **lines**, the [Duplicate Word Finder](https://toolcloak.com/tools/duplicate-word-finder) is the right tool. To turn a tidy list back into a single comma-separated line, the [Line Break Remover](https://toolcloak.com/tools/remove-line-breaks) joins lines with a comma, and its [guide](https://toolcloak.com/articles/how-to-remove-line-breaks-from-text-copied-from-a-pdf-or-email-join-lines-into-one-paragraph-or-a-comma-separated-list) explains the options.

## Good uses, and one warning

- **Mailing and contact lists:** merge two exports, then remove repeated addresses before you send.
- **Keywords and tags:** combine lists from several sources into one clean set. The [Hashtag Generator](https://toolcloak.com/tools/hashtag-generator) output often has repeats when you run it twice.
- **Student or attendee names:** one name per line, remove repeats, then use the [Random Name Picker](https://toolcloak.com/tools/random-name-picker) so nobody gets two chances.
- **Logs and URLs:** shrink a long list of repeated entries into the distinct ones.
- **Spreadsheets:** copy a single column out of Excel, paste it here, and paste the clean result back. For files, the [PDF to Excel guide](https://toolcloak.com/articles/how-to-convert-a-pdf-table-to-excel-or-csv-free-bank-statements-invoices-and-why-columns-shift) shows how to get a table into a sheet first.

> **Keep the original:** Removing duplicates is a one-way change to the list you paste. Keep the original file or sheet until you have checked the Removed count. If a line was removed that you wanted to keep, it almost always means a switch was on (usually Ignore case) that you wanted off.

Privacy: the list is processed in your browser, and Toolcloak does not receive it. That matters for customer lists and email addresses, which you should not paste into a site that uploads them.

## Frequently asked questions

### How do I remove duplicate lines online?

Open the Duplicate Line Remover, paste your list with one item per line, and press Remove Duplicates. The first copy of every line is kept in its original order, and you can copy the result.

### Duplicate lines kaise hataye?

Duplicate Line Remover kholiye, list paste kijiye (har line mein ek item), aur Remove Duplicates dabaiye. Har line ki pehli copy rahegi aur baaki hat jayengi. Upar Original, Unique aur Removed ki ginti dikhegi.

### Does it keep the original order?

Yes. It keeps the first copy of each line where it first appeared and removes the later copies. Nothing is sorted. Use a line sorter afterwards if you want alphabetical order.

### Why were “Apple” and “apple” treated as the same line?

Ignore case is on by default, so lines are compared in lower case. The first spelling is kept. Turn Ignore case off if you want capital letters to count as a difference.

### What does “Keep only lines that appear once” do?

It removes every line that occurs more than once, including its first copy, so only lines with no twin remain. It is useful for finding the odd ones out between two lists.

### Why does Original show fewer lines than I pasted?

Original is counted after trimming and blank-line removal. If you pasted blank lines and the Remove blank lines switch is on, they are taken out before the count begins.

### Why were two lines that look the same not removed?

They differ in something you cannot see, usually a double space inside the line, a different accent, or a spelling variant. Trim only removes spaces at the ends. Clean inner spaces first, then run it again.

### Is my list uploaded anywhere?

No. The list is processed in your browser and is not sent to Toolcloak, which is important for email lists and customer names.