Compare Hadoop-era concepts with modern data-platform needs

“Hadoop versus modern data stack” is a poor architecture question. Hadoop bundled answers to storage, distributed computation, metadata and resource management for an earlier operating context. Current platforms may separate those responsibilities, but the underlying problems—partitioning, movement, recovery, replay and governance—remain.

Compare responsibilities and constraints before products.

Translate four concepts

The local data-engineering lab records a compact mapping.

from data_engineering_cases import platform_concepts_case

result = platform_concepts_case()
assert result["concepts"] == 4
for row in result["mappings"]:
    print(row["hadoop_era"], "->", row["modern_need"])

Distributed file blocks map to durable storage layout and data-locality cost. MapReduce stages map to execution DAGs and query engines, where shuffle boundaries still influence time and expense. Hive tables map to catalogued analytical datasets whose schema, partitions and access policy must remain discoverable. Long batch jobs map to a mix of batch and streaming chosen from the decision’s latency tolerance.

The mapping is conceptual. This lab does not run Hadoop, Spark or a cloud warehouse and supplies no platform benchmark.

Start from a workload contract

Describe the workload with evidence:

• data volume now and expected growth;
• arrival pattern and permitted lateness;
• query shapes, concurrency and latency targets;
• replay, recovery and availability requirements;
• privacy, residency and retention constraints;
• team skills, operating capacity and cost envelope.

A 5,000-row local aggregation does not need a cluster. A multi-terabyte join with repeated shuffles cannot be assessed from laptop timing. A low-latency fraud decision may need event processing, while monthly retraining can remain a bounded batch even when its dataset is large.

Preserve useful principles

HDFS was designed for high-throughput access to large datasets and assumed hardware failure. Those design pressures still matter even when durable object storage replaces directly managed HDFS. MapReduce made movement between stages visible; modern optimizers hide more machinery, so execution plans and runtime metrics become essential review evidence.

Modern managed services can reduce hardware and scheduler operations, yet they do not remove data contracts, key skew, small-file problems, access control or recovery tests. Separating storage and compute can improve flexibility while introducing network and request costs that must be measured on the intended workload.

The Data Science course teaches this comparison so model builders can identify when a local pipeline is sufficient and when to collaborate with data-platform owners. Product selection requires a measured proof of concept, not a fashionable diagram.

Exercise

Write two workload contracts: a daily 20 GB feature build and a 30-second review queue. Map storage, compute, catalog, orchestration and observability responsibilities. Identify likely shuffle or replay boundaries, then state what benchmark would reject each proposed design.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Create a backfill plan with bounded impact.
• Continue with Build a reproducible data pipeline on a local machine.

References: Apache Hadoop HDFS architecture and Apache Spark SQL performance tuning.