Data Engineering

What it does

This skill enables AI agents to automate data engineering tasks, including building data pipelines, designing data warehouses, and implementing ETL/ELT processes. It supports various components of a modern data stack, from ingestion and storage to transformation and serving. The skill provides examples of batch processing using Airflow and streaming processing with Kafka, as well as dbt modeling patterns for structuring data transformations.

When to use it

• Building automated data pipelines.
• Designing the architecture for data warehouses.
• Implementing ETL/ELT processes to move and transform data.
• Setting up data lakes for storing large volumes of raw data.
• Optimizing existing data infrastructure for efficiency.

Key capabilities

• Supports building batch processing pipelines using Airflow.
• Provides examples of streaming data processing with Kafka.
• Demonstrates dbt modeling patterns for organizing and transforming data (staging, intermediate, marts).
• Covers components including ingestion, storage, transformation, orchestration, and serving layers within a modern data stack.
• Includes example code snippets for common data engineering tasks.

Example prompts

• "Create an Airflow DAG to extract data from [source], transform it using [transformation logic], and load it into [destination]."
• "Show me the recommended dbt model structure for a project with orders and customers data."
• "Give me an example of how to consume events from Kafka in Python."

Tips & gotchas

• This skill focuses on providing examples and guidance for data engineering practices; it doesn't directly execute these tasks. You’ll need additional tools (like Airflow, dbt, Kafka) installed and configured separately.
• The provided code snippets are illustrative and may require adaptation to your specific environment.