Data Quality Frameworks

What it does

This skill enables AI agents to analyze data quality using established frameworks and tools like Great Expectations, dbt tests, and data contracts. It helps ensure reliable data pipelines by identifying potential issues and suggesting remediation strategies. The skill focuses on implementing data quality checks within existing pipelines and establishing clear agreements between teams regarding data standards. Detailed implementation patterns can be found in the included playbook.

When to use it

• Implementing data quality checks as part of a data pipeline.
• Setting up Great Expectations validation suites.
• Building comprehensive dbt test suites for your data transformations.
• Establishing and enforcing data contracts between different teams or systems.
• Automating data validation processes within a CI/CD workflow.

Key capabilities

• Utilizes established frameworks (e.g., DAMA-DMBOK, TDQM - not explicitly detailed but implied).
• Integrates with Great Expectations for data validation.
• Supports the creation and use of dbt tests.
• Facilitates the definition and enforcement of data contracts.
• Provides guidance on automating validation within CI/CD pipelines.

Example prompts

• "Implement a Great Expectations check to validate the 'customer_id' column in my sales table."
• "Create a dbt test suite for our product catalog transformation process."
• "Define a data contract between the marketing and analytics teams regarding the format of lead generation data."

Tips & gotchas

• This skill is specifically designed for tasks related to data quality checks, testing, and contracts. Ensure your task aligns with this scope before using it.
• The output from this skill should not replace environment-specific validation or expert review.
• Refer to the resources/implementation-playbook.md file for detailed frameworks, templates, and examples.