A Data Guide to AWS on oblako

Introduction

This book is a guide to the AWS data services as they run on oblako, which runs those services on your own machine. S3 calls reach a real object store, Redshift SQL runs in a real database, and Athena queries run on a real query engine that reads the files. Your code receives the responses these engines produce, and nothing is billed.

You can create a bucket, a table, a warehouse schema and a streaming pipeline in an afternoon, break them and start again, without an AWS account or a credit card and with nothing billable left to clean up. When the code works, you point it at AWS by changing an endpoint and a set of credentials, and the code itself stays the same.

Who this book is for

The book is for data engineers, analytics engineers and data scientists who work with AWS or plan to. It assumes you write Python, know SQL and have used a terminal, but it assumes no prior knowledge of AWS, since each chapter explains its service before it uses it. If you already run these services in production, you can use the book to develop and test against them locally, with more confidence than a mock gives you.

A disclosure

I wrote oblako, which is open source under the Apache 2.0 license with its code public. This book is therefore a companion to software I maintain, and you should read it with that in mind. I have tried to state exactly where oblako behaves like AWS and where it differs. Each chapter on a service or tool has a section on the differences, and every listing that shows something working was run on oblako before it went into the text.

How the book is organized

Part I covers oblako itself: what it is, how a request travels from your code to a real engine, and how to install and run it. Read it first, because the rest of the book assumes you know which engine sits behind which port.

Part II covers the data services one at a time: S3; S3 Tables, Iceberg and S3 Vectors; the Glue Data Catalog and Athena; Glue jobs; Redshift; RDS and Aurora; replicating PostgreSQL; DynamoDB; and streaming with Kinesis and Firehose.

Part III covers the data engineering tools that run on those services: Spark writing Iceberg tables to S3 Tables, a multi-node Redshift built on Citus, a feature store with Feast over Redshift and DynamoDB, and Airflow pipelines on MWAA.

Part IV covers practice: testing a pipeline in CI, describing its infrastructure as code with SAM, Terraform and Pulumi, and running the same code and tests on AWS.

Appendix A covers services outside the book’s data focus that oblako also runs: Lambda, Step Functions, EventBridge, Bedrock, ECS and EKS, with one working example each.

The examples

Instead of a running example, each chapter uses small, self-contained data chosen for its own topic, such as a few files for S3, a partitioned table for Athena or a handful of rows for Redshift. Each chapter’s code runs on its own, and its listings create their data or a short script next to them does.

Conventions

Code is plain boto3 or the service’s own driver, such as redshift-connector for Redshift, and listings never call oblako-specific APIs. The only oblako commands in the book are calls to the oblako command-line tool in setup steps, so the same code runs on AWS.

Python listings look like this:

import boto3

s3 = boto3.client("s3")
print(s3.list_buckets()["Buckets"])

and shell commands like this:

oblako up s3

The listings were run against oblako 0.1.0 from its main branch, most recently at commit 3117e6f. oblako changes quickly, and where a later release behaves differently from the book, its release notes say so.