Python Streamlit App - Data Quality & Cleaning -- 2

Automated Data Quality & Data Cleaning System (Data Detox)

I want to upload and professionally present my Data Detox – Automated Data Quality, Anomaly Detection, and Interactive Data Cleaning System, which is a conference-level, production-ready data analysis project built using Python and Streamlit.

This project focuses on automating the most critical and time-consuming part of data analytics: data quality assessment and data preprocessing. The application provides an end-to-end solution to analyze raw datasets, detect data quality issues, clean inconsistencies, visualize patterns, generate profiling reports, and export clean datasets ready for analytics or machine learning.

Project Overview

Data Detox is an interactive web-based data cleaning and analysis tool designed to handle real-world noisy datasets. It allows users to upload CSV files and automatically performs:

Data quality validation

Missing value detection and handling

Duplicate detection and removal

Anomaly and distribution analysis

Automated exploratory data analysis (EDA)

Data profiling and reporting

Clean dataset export

The system eliminates the need for writing manual preprocessing scripts and significantly reduces data preparation time while improving data reliability and accuracy.

Problem Statement

In real-world applications, raw data often contains:

Missing and inconsistent values

Duplicate records

Invalid or noisy entries

Outliers and skewed distributions

Manually cleaning such datasets is time-consuming, error-prone, and inefficient. Data Detox solves this problem by providing a fully automated and interactive platform that enables users to clean and validate datasets with minimal effort.

Technology Stack
Frontend & UI

Streamlit – Interactive web application framework

Custom CSS Styling – Professional UI/UX with modern fonts, colors, buttons, dropdowns, and layout

Responsive Design – Wide layout dashboard with metrics, expanders, and sidebar controls

Session State Management – Maintains raw data, cleaned data, and applied operations across reruns

Backend & Data Processing

Python – Core programming language

Pandas – Data manipulation, cleaning, aggregation, and transformations

NumPy – Numerical processing and missing value handling

Base64 Encoding – Dynamic background image integration

Data Analysis & Profiling

ydata_profiling (Pandas Profiling) – Automated exploratory data analysis and dataset health reports

streamlit-pandas-profiling – Embedded profiling reports inside the application

Visualization

Matplotlib – Statistical and analytical visualizations

Seaborn – Distribution plots and anomaly detection charts

Core Functional Features
1. Data Upload & Normalization

Upload datasets in CSV format

Automatically detects multiple missing value representations such as:

NULL, NaN, n/a, empty strings, blank spaces, etc.

Converts all invalid representations into standardized missing values

2. Dataset Overview Dashboard

Total number of rows and columns

Missing value count and percentage

Duplicate row detection

Preview of dataset with interactive table view

3. Automated Data Profiling

One-click generation of a detailed profiling report

Feature distributions and summary statistics

Correlation analysis

Missing value matrices

Dataset structure and health assessment

4. Intelligent Data Cleaning Engine

Multiple cleaning strategies are provided to handle missing data:

Remove rows containing missing values

Fill numeric columns using mean

Fill numeric columns using median

Fill values using mode (most frequent value)

Replace missing values with zeros

Remove columns with more than 50% missing data

All applied cleaning operations are tracked and displayed for transparency.

5. Duplicate Detection & Removal

Automatic duplicate row detection

One-click duplicate removal

Cleaning history tracking

6. Anomaly & Pattern Visualization

Numeric column selection

Histogram with kernel density estimation (KDE)

Distribution analysis for outlier identification

7. Data Quality Comparison

Before-and-after comparison of:

Row count

Missing values

Quality improvement indicators

Dataset cleanliness percentage

8. Data Export

Download cleaned datasets as CSV files

Cleaned data is ready for:

Data analytics

Business intelligence dashboards

Machine learning pipelines

Workflow

Upload dataset

Automated quality assessment

Visual inspection and profiling

Select data cleaning strategy

Apply transformations

Review quality improvements

Download clean dataset

Results & Impact

Reduced manual data preprocessing effort by approximately 45%

Improved dataset consistency and reliability

Enabled faster decision-making and model building

Simplified data preparation for non-technical users

Use Cases

Data Analysts & Data Scientists

Machine Learning preprocessing pipelines

Business intelligence and reporting

Academic research and projects