Data Engineering: XML, Data Formats, Analytical SQL, Spark, Kafka, Airflow and the Lakehouse

Data Engineering is the seventh and final volume of the Web Coding Series. You transform a publisher's XML catalog with XPath, XQuery and XSLT, write the same orders as JSON, CSV, Avro, Parquet and Arrow, model and query them in PostgreSQL and DuckDB with dbt, process them at scale with Apache Spark, stream them through Apache Kafka, schedule the whole pipeline with Airflow, Dagster and Prefect, and land it in an Apache Iceberg lakehouse with data quality checks, lineage and governance. One business, BookNest, a small online bookshop, runs through all eight chapters, and the final capstone runs its entire data platform in a single pipeline.

Every example was run. The commands were executed on one modest 4-CPU machine in Ubuntu 26.04 under WSL2 and Docker, with Saxon-HE 12.10, PostgreSQL 18.6, DuckDB 1.5.5, Spark 4.2.0, Kafka 4.3.1, Airflow 3.3.2, Iceberg 1.12.0 and Trino 483, and the listings show their actual output. Managed cloud warehouses, Spark platforms, Kafka services and lakehouses are compared from their documentation with verified prices and marked as not run here; BookNest's order events also ran on Aiven's free Kafka plan, which needs no credit card.

What you will learn

Foundations: the data engineering lifecycle, OLTP and OLAP, batch and streaming, Lambda and Kappa, the modern data stack, warehouses, lakes, lakehouses, data mesh, data fabric and the real cost of tools.

XML: namespaces, XPath 3.1, XQuery 3.1 with Full Text and Update, XSLT 3.0 with streaming, DTD, XSD 1.1, RELAX NG and Schematron, parsers, web services and XML security.

Data formats: JSON pitfalls, JSON Schema, jq, CSV, Avro, Protocol Buffers, Parquet, ORC and Arrow internals, codecs and schema evolution.

Analytical SQL: column stores, PostgreSQL 18 and DuckDB for analytics, dimensional modeling, slowly changing dimensions, dbt, ClickHouse, semantic layers and the cloud warehouses compared.

Apache Spark: how a job runs, PySpark DataFrames, Spark SQL, UDFs and Arrow, the Spark UI, tuning, Spark on Kubernetes, and Polars, DuckDB and Dask as alternatives.

Apache Kafka: the log on disk, KRaft, producers, transactions and exactly-once, consumer and share groups, Connect, Schema Registry, Kafka Streams, Flink, security, tiered storage, Confluent Cloud, Amazon MSK, Azure Event Hubs, Aiven and Redpanda.

Orchestration: idempotent pipelines, Airflow 3 in Docker Compose, dbt with Cosmos, Dagster, Prefect, change data capture with Debezium, Airbyte and dlt.

The lakehouse: MinIO and the S3 API, Iceberg and its catalogs, Delta Lake and Hudi, Trino, streaming Kafka into Iceberg, data quality with GX Core, Soda and dbt tests, contracts, lineage and governance.

Why this book is different

The commands were executed, not imagined. Where the documentation and the real behavior disagreed, the book reports what actually happened.

One BookNest data set is carried from an XML catalog to a lakehouse, where files, Spark, Trino and DuckDB agree on its sales total to the cent.

It explains what happens under the hood, from XML parsers and Parquet pages to Spark's planner, Kafka's replicated log and Iceberg's metadata tree.

Honest comparisons of open-source and commercial alternatives, with licences and verified prices.

Hundreds of listings, diagrams and screenshots, reference appendices, interview questions and an index of open-source tools.

Ten "Test Yourself!" questions end every chapter, and every answer was produced by running the code.