Cloud platforms separate compute from storage, enabling elastic scaling and pay-per-use pricing. The "Big Three" (AWS, GCP, Azure) plus Snowflake provide the foundation for modern data engineering.
Service Model Comparison
| On-Premise | AWS | GCP | Purpose |
| HDFS | S3 | Cloud Storage | Object storage |
| Spark | EMR | Dataproc | Distributed compute |
| Kafka | MSK / Kinesis | Pub/Sub | Event streaming |
| PostgreSQL | RDS | Cloud SQL | Managed RDBMS |
Cloud DWH Options
| System | Type | Key Feature |
| Athena | Query service | Managed Presto, queries S3 without loading |
| Redshift | MPP DWH | Closest to Greenplum architecture |
| BigQuery | Serverless DWH | No compute control - true serverless |
| Snowflake | Separated compute/storage | Spins up clusters per query, data in S3 |
| Presto/Trino | Federated query engine | Queries data in source systems without ETL |
S3 - Simple Storage Service
- 5-10x cheaper than HDFS
- 99.999999999% durability (11 nines)
- Practically unlimited elasticity
- Separates storage from compute
- Access via
boto3 (Python), CLI, or Spark (s3a://)
Cloud Service Models for DE
| Approach | Examples | DE Responsibility |
| IaaS | Self-managed ClickHouse on VMs | OS, database, monitoring, backups |
| PaaS | Managed ClickHouse, RDS | Service configuration |
| SaaS | BigQuery, Snowflake | Query writing only |
Snowflake Architecture
- Three layers: Storage, Compute, Services
- No direct access to storage layer
- Automatic micro-partitioning
- Virtual clusters (warehouses) for different workloads
- Pay per query compute time
VM and Disk Best Practices
Disk performance scales with size (QoS). A 50GB SSD may be slow because IOPS are throttled proportionally. Fix: increase disk size for more IOPS.
| Disk Type | Best For |
| HDD | Cheapest, archives |
| SSD | Dev/test (cost-effective) |
| High-IOPS SSD | Production workloads |
| Local NVMe | Maximum performance, no replication |
Cost Management
- Always pause/stop unused VMs, databases, K8s clusters
- Transient clusters for one-off jobs - create, use, destroy
- HDD for dev/test, SSD for staging, High-IOPS for production
- Monitor QoS limits - small disks have lower IOPS quotas
Cloud-Native Data Pattern
Storage Layer: S3 (or compatible)
Compute Layer: Kubernetes (Spark, Presto run here)
Gotchas
- Redshift nodes include compute + storage by default; Snowflake separates natively
- Vertical scaling requires VM restart (downtime)
- Always configure firewall - any VM with public IP is a target
- Network latency kills cross-datacenter cluster performance with synchronous replication
- BigQuery: no compute control means unexpected costs on complex queries
See Also