Data Engineer

What it does

This skill automates the creation of data pipelines, Extract, Transform, Load (ETL) processes, and overall data infrastructure. It handles data ingestion from various sources, supports both batch and real-time processing, validates data quality, and facilitates analytics workflows like report generation and data export. The skill provides example implementations for common ETL patterns, such as daily synchronization and incremental updates.

When to use it

• You need to regularly move data from an external API into a database.
• You want to automate the process of cleaning and transforming raw customer data.
• You require a system for aggregating data and calculating metrics for reporting purposes.
• You're building or maintaining a data warehouse and need assistance with schema design and data replication.

Key capabilities

• ETL Processes: Extract, Transform, Load (ETL) pipelines.
• Data Ingestion: Ingestion from multiple data sources.
• Processing Modes: Supports both batch and real-time data processing.
• Data Quality Validation: Includes mechanisms for validating data quality during the ETL process.
• Database Schema Design: Assists in designing database schemas.
• Analytics Support: Facilitates data aggregation, metrics calculation, report generation, and data export.

Example prompts

• "Create a daily sync pipeline from this external API to my customer database."
• "Build an incremental ETL process that only updates records changed since the last run."
• "Design a database schema for storing product information with these fields..."

Tips & gotchas

• The skill provides example code snippets, but you may need to adapt them to your specific data sources and environments.
• API keys and secrets are required for authenticating with external services; ensure these are properly configured as environment variables.
• The provided examples use TypeScript and Vercel Cron for scheduling – familiarity with these technologies might be helpful.