Monitor Data Integrity

What it does

The monitor-data-integrity skill allows an AI agent to verify the consistency and accuracy of data within a specified dataset. It can identify missing values, detect outliers based on defined rules, and flag potential errors requiring human review. This helps ensure data quality for downstream processes like analysis or model training.

When to use it

• Data Pipelines: Regularly check data flowing through ETL pipelines for unexpected changes or corruption.
• Post-Ingestion Validation: After importing data from external sources, confirm its integrity before further processing.
• Anomaly Detection: Identify unusual patterns in datasets that might indicate errors or malicious activity.
• Model Training Data Quality: Ensure the training dataset is clean and accurate to improve model performance.

Key capabilities

• Missing value detection
• Outlier identification based on rules
• Data consistency checks
• Error flagging for review

Example prompts

• "Monitor the 'sales_data' dataset for missing values in the 'customer_id' column."
• "Check the 'inventory' data and flag any quantities below zero."
• "Verify that all dates in the 'transaction_log' are within the last year."

Tips & gotchas

The skill requires a clear definition of rules for outlier detection. Ensure these rules accurately reflect expected data patterns to avoid false positives.