How to Find Duplicate Rows in Pandas

Handling duplicate data is a crucial step in data analysis and preprocessing. Pandas, a powerful data manipulation library in Python, offers several methods to find and manage duplicate rows. In this article, we will explore the different ways to identify duplicate rows in a DataFrame using the duplicated() method. We'll cover scenarios where you want to find duplicates based on the entire DataFrame, specific columns, and even customize the behavior further.

Understanding the duplicated() Method

The duplicated() method in Pandas is a powerful tool for identifying and filtering out duplicate rows. When called on a DataFrame without any arguments, it returns a Boolean series indicating which rows are duplicates. The default behavior is to return all rows that are duplicates, keeping only the first occurrence.

Example of Using duplicated() Without Arguments

To find and select all duplicate rows based on all columns, you can use:

DfNew df.duplicated()

This will return a Boolean series where True is present at every place of the duplicate rows, excluding the first occurrence. Here's an example:

import pandas as pddata  {'Name': ['John', 'Anna', 'Anna', 'Mike', 'Mike', 'Mike'],        'Age': [28, 22, 22, 30, 30, 30],        'City': ['New York', 'London', 'London', 'Berlin', 'Berlin', 'Berlin']}df  (data)# Find duplicate rowsduplicates  df.duplicated()df[duplicates]

The output will be:

NameAgeCity Anna22London Mike30Berlin

Finding Duplicates Based on a Single Column

Often, you may be interested in finding duplicates based on a specific column. This can be done using the duplicated(column) method, where you specify the name of the column. For instance, to find duplicates based solely on the 'Name' column:

DfNew df[df.duplicated('Name')]

Here’s an example to demonstrate this:

import pandas as pddata  {'Name': ['John', 'Anna', 'Anna', 'Mike', 'Mike', 'Mike'],        'Age': [28, 22, 22, 30, 30, 30],        'City': ['New York', 'London', 'London', 'Berlin', 'Berlin', 'Berlin']}df  (data)# Find duplicate rows in 'Name' columnduplicates_name  df[df.duplicated('Name')]df[duplicates_name]

The output will be:

NameAgeCity Anna22London Mike30Berlin

Finding Duplicates Based on Multiple Columns

In many cases, you might want to find duplicates based on multiple columns. To achieve this, you can pass a list of column names to the method. Here’s an example:

DfNew df[df.duplicated(['Name', 'Age'])]

This will return all rows that are duplicates based on the 'Name' and 'Age' columns. Here’s a practical example:

import pandas as pddata  {'Name': ['John', 'Anna', 'Anna', 'Mike', 'Mike', 'Mike'],        'Age': [28, 22, 22, 30, 30, 30],        'City': ['New York', 'London', 'London', 'Berlin', 'Berlin', 'Berlin']}df  (data)# Find duplicate rows based on 'Name' and 'Age'duplicates_name_age  df[df.duplicated(['Name', 'Age'])]df[duplicates_name_age]

The output will be:

NameAgeCity Anna22London Mike30Berlin

Further Customizations

By default, duplicated() keeps the first occurrence of the duplicate and marks the rest as True. However, you can specify different behaviors using the `keep` argument. The `keep` argument can take the following values:

first: Keep the first occurrence (default) last: Keep the last occurrence False: Mark all duplicates as True

Here’s an example of how to keep the last occurrence:

DfNew df.duplicated(keep'last')

And here’s an example of marking all duplicates as True:

DfNew df.duplicated(keepFalse)

Here’s how you can apply these customizations:

import pandas as pddata  {'Name': ['John', 'Anna', 'Anna', 'Mike', 'Mike', 'Mike'],        'Age': [28, 22, 22, 30, 30, 30],        'City': ['New York', 'London', 'London', 'Berlin', 'Berlin', 'Berlin']}df  (data)# Keep last occurrenceduplicates_last  df.duplicated(keep'last')df[duplicates_last]# Mark all duplicates as Trueduplicates_all  df.duplicated(keepFalse)df[duplicates_all]

Conclusion

Identifying duplicate rows is a critical step in data preprocessing and analysis. Pandas provides a robust and flexible way to accomplish this through the duplicated() method. By customizing the method, you can efficiently manage and clean your data to improve the accuracy and reliability of your analysis. With the examples provided, you should now have a comprehensive understanding of how to find and handle duplicate rows in Pandas.