Data Cleaning Pipeline

What it does

This skill automates the preparation of raw datasets for analysis by executing a sequence of cleaning operations. It handles common data quality issues such as missing values, duplicate entries, and inconsistent formatting to ensure downstream tasks receive reliable input.

When to use it

• Before feeding unstructured or messy CSV/JSON files into machine learning models.
• When preparing historical logs or sensor readings for trend analysis.
• To standardize customer records from multiple disparate sources before merging.
• As a preprocessing step prior to generating visualizations or business reports.

Key capabilities

• Detects and removes duplicate rows based on specified columns.
• Fills missing values using mean, median, or mode imputation strategies.
• Standardizes date formats and normalizes text casing.
• Filters out rows containing excessive null entries or outliers.
• Validates data types against expected schemas before processing.

Example prompts

• "Run the data cleaning pipeline on my sales.csv file to remove duplicates and fill missing revenue values with the median."
• "Execute a cleanup routine on my user_logs.json to standardize timestamp formats and filter out entries with empty IP addresses."
• "Apply the data cleaning pipeline to preprocess my survey responses, handling missing answers by dropping those specific rows entirely."

Tips & gotchas

Ensure your source files are accessible in the agent's working directory before invoking the skill. While the pipeline handles common anomalies, complex domain-specific logic may require custom scripting outside this standard tool.