Spark + S3

/tldr: S3 is fast as hell in 2025 — if you’re not stupid

s3a:// Committer v2 Multipart 2025

2025 LAW

s3a:// + Fast Committer + Multipart
= S3 is faster than HDFS

The Golden S3 Config (2025)

spark.hadoop.fs.s3a.impl                        org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.fast.upload                 true
spark.hadoop.fs.s3a.fast.upload.buffer          array
spark.hadoop.fs.s3a.fast.upload.active.blocks   8
spark.hadoop.fs.s3a.multipart.size              128M
spark.hadoop.fs.s3a.multipart.threshold         256M
spark.hadoop.fs.s3a.connection.maximum          1000
spark.hadoop.fs.s3a.threads.max                 256

# Zero-rename commit (mandatory!)
spark.hadoop.fs.s3a.committer.name              directory
spark.hadoop.fs.s3a.committer.staging.tmp.path  /tmp/staging
spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory

# Optional but golden
spark.hadoop.fs.s3a.block.size                  134217728   # 128MB
spark.sql.files.maxPartitionBytes               256MB

S3A Committer v2 = No More _SUCCESS Hell

Directory Committer

Zero-rename · 10–100× faster writes

Old Way (Dead)

_temporary → rename → 1 hour for 100k files

Result

10TB write in < 5 min

Reading S3 — The Right Way

DO THIS

spark.read
  .option("recursiveFileLookup", "true")
  .parquet("s3a://bucket/path/to/data/")

NEVER DO

s3:// (old, slow)
dbutils.fs.ls + loops
WholeTextFiles

Writing: Partition Smart or Die Slow

50–200 files per partition · ~128–256 MB each

partitionBy("date", "region") → not 1000 columns

1 file per partition = 1 task → 1 hour delay

EMR & Databricks — Already Done For You

Databricks Runtime 13+

All this config = default

EMR 6.10+

s3a committer v2 enabled by default

FINAL ANSWER:

s3a:// + Fast Upload + Directory Committer
= S3 is now the fastest storage

HDFS is dead.
S3 won.