The student will be able to:

1. Explain data engineering roles, the data lifecycle, and the modern data stack.
2. Design and evaluate analytical data storage architectures and modeling patterns.
3. Implement batch, streaming, and API-based data ingestion techniques.
4. Assess scaling strategies and distributed computing fundamentals for data infrastructure.
5. Develop and validate ETL pipelines using Python and SQL.
6. Automate and manage scheduled data workflows using orchestration tools.
7. Package and deploy data applications using containerization techniques.
8. Integrate pre-trained machine learning models for batch data enrichment.
9. Apply data privacy, security, and access control principles to data pipelines.
10. Evaluate the ethical, social, and environmental impacts of data engineering workflows and infrastructure.

1. Fundamentals of data engineering
   1. Data roles and career pathways
   2. The data engineering lifecycle
   3. Overview of the modern data stack and open-source ecosystems
2. Data storage architectures and modeling
   1. Relational databases, NoSQL, and object storage
   2. Data warehouses, data lakes, and lakehouses
   3. Pipeline architecture patterns: medallion architecture
   4. Data modeling for analytics (star schema, snowflake schema)
   5. Columnar vs. row-oriented storage formats
3. Data ingestion techniques
   1. Batch processing vs. stream processing
   2. API integration and flat file ingestion
   3. Event streaming concepts (e.g., Apache Kafka)
   4. Incremental vs. full data loads
4. Scaling strategies and infrastructure
   1. Vertical scaling (scale-up) vs. horizontal scaling (scale-out)
   2. Distributed computing fundamentals
   3. Cloud-agnostic infrastructure concepts
5. ETL pipeline development
   1. Extracting data from disparate sources
   2. Data quality and validation (handling nulls, duplicates, and schema enforcement)
   3. Data transformation techniques (joining, aggregating) using Python and SQL
   4. Loading data into analytical storage
   5. ETL vs. ELT and transformation-in-place
   6. SQL analytical functions for data transformation: window functions and common table expressions (CTEs)
6. Workflow orchestration
   1. Scheduling and dependency management
   2. Directed Acyclic Graphs (DAGs)
   3. Designing robust workflows using tools like Apache Airflow
   4. Reliability concepts (idempotency, retries, and backfills)
   5. Version control and Continuous Integration and Continuous Deployment (CI/CD) for data engineering workflows
7. Containerization
   1. Introduction to containers and image-based deployment (e.g., Docker)
   2. Managing dependencies and environment reproducibility
   3. Connecting multiple containers (e.g., Docker Compose)
8. Machine learning workflows in data engineering
   1. Using MLOps to serve data and manage artifacts
   2. Integrating pre-trained models for data enrichment (batch inference vs. real-time)
   3. Tracking and versioning model artifacts within a pipeline environment
9. Data governance and security
   1. Data privacy regulations (e.g., GDPR, CCPA) and compliance basics
   2. Role-Based Access Control (RBAC) and least privilege principles in data pipelines
   3. Identifying and masking Personally Identifiable Information (PII) during transformation
10. Social and ethical considerations
    1. Ethical implications of broad data collection and automated decision-making
    2. Identifying and mitigating bias within data pipelines and downstream models
    3. Environmental impact and sustainability of large-scale cloud infrastructure
    4. Promoting transparency and accountability in data workflows

1. Environment setup and containerization
   1. Installing and configuring a local development environment
   2. Writing Dockerfiles to containerize Python and SQL applications
   3. Running multi-container applications using basic container orchestration tools
2. Data storage and modeling
   1. Provisioning an open-source relational database (e.g., PostgreSQL)
   2. Interacting with a cloud object storage bucket
   3. Designing and implementing a star schema for an analytical dataset
   4. Writing SQL queries to aggregate and analyze data
3. Developing ETL pipelines
   1. Writing Python scripts to extract data from public APIs and flat files
   2. Transforming and cleaning raw data using Python libraries (e.g., Pandas)
   3. Loading transformed data into a target database
   4. Adding data quality assertions (e.g., checking row counts or uniqueness) to a data transformation script
   5. Writing SQL queries using window functions and CTEs to perform analytical transformations
   6. Restructuring a flat dataset into Bronze, Silver, and Gold layers to practice the medallion architecture pattern
   7. Identifying and masking Personally Identifiable Information (PII) before loading the data into the target database
4. Workflow orchestration
   1. Installing and configuring an orchestration tool (e.g., Apache Airflow)
   2. Converting standalone ETL scripts into a scheduled DAG
   3. Implementing error handling, alerts, and retries in the workflow
   4. Tracking orchestration scripts with version control and demonstrating a repeatable backfill
5. Integrating pre-trained models into data workflows
   1. Loading a pre-trained ML model (e.g., Hugging Face sentiment analysis) into a Python transformation script
   2. Applying the model to a batch dataset to generate predictions and enrich the data
6. Capstone project
   1. Designing an end-to-end automated data workflow based on a real-world scenario
   2. Integrating ingestion, storage, orchestration, and a pre-trained ML model for batch data enrichment
   3. Documenting pipeline architecture and deployment instructions

Tests and quizzes
Lab notebook
Written laboratory assignments which include source code, sample runs, and documentation
Reflective papers
Final examination or project

Instructor-authored lectures which include technical foundations, theoretical motivation, and coding implementation of data engineering principles
Detailed review of assignments which includes solutions and specific comments on the student submissions
Discussion which engages students and instructor in an ongoing dialog about data engineering
Instructor-authored labs that rigorously demonstrate a student's ability to implement data engineering objectives

Reis, Joe, and Matt Housley. Fundamentals of Data Engineering. 2022.

Nwokwu, Chisom. Data Engineering for Beginners. 2025.

1. Reading
   1. Textbook assigned reading averaging 30 pages per week
   2. Reading the supplied handouts and modules averaging 10 pages per week
   3. Reading online resources as directed by instructor though links pertinent to programming
   4. Reading library and reference material directed by instructor through course handouts
2. Writing
   1. Writing technical prose documentation that supports and describes the programs that are submitted for grades