Data engineer role in India: pipelines, on-call and data contracts

What breaks, and why it lands on you

The clearest way to understand the role is to look at what goes wrong. A typical month for a data engineer at a hypothetical Indian e-commerce company includes:

• A source team renames a column in the orders database; the nightly load fails.
• A payment provider's API starts returning pages of 100 instead of 500; the job still succeeds but loads a fifth of the data.
• A retry after a timeout loads the same day's data twice, and the finance dashboard doubles for a day.
• Warehouse costs jump because a new query scans a year of events every hour.

Each of these is an engineering problem with an engineering answer: schema checks, row-count reconciliation, idempotent loads and query review. The second and third are the dangerous ones, because nothing looks broken.

The artefact: a runbook entry for the nightly orders load

Mature teams keep a runbook so that whoever is on call can act without guessing. A realistic entry:

PIPELINE: orders_daily_load
Schedule: 01:30 IST daily. Downstream SLA: tables fresh by 06:30 IST
Owner: data-platform team. Consumers: finance daily pack, ops dashboard, demand forecast job

What it does
  Extracts orders updated in the previous IST day from the orders replica,
  writes to staging.orders_<date>, then MERGEs into core.orders on order_id.

Why a rerun is safe
  The load is idempotent: MERGE on order_id with updated_at comparison.
  Rerunning the same date produces the same result. Never switch to INSERT.

Checks that fail the run
  1. Source row count for the date vs staging row count: must match exactly
  2. Null order_id count in staging: must be 0
  3. Expected columns present (schema check against contract v3)
  4. Total order value within ±40% of the trailing 7-day median (warn, not fail)

If it fails
  Schema check failed  -> compare with contract; contact orders-service on-call; do not patch mapping silently
  Row count mismatch   -> check replica lag; rerun after lag < 5 min
  Value check warning  -> check for sale events on the marketing calendar before escalating
  Missed 06:30 SLA     -> post in #data-incidents with ETA; finance pack owner is notified automatically

Two lines deserve attention. "Why a rerun is safe" exists because the most common on-call mistake is rerunning a non-idempotent job and duplicating data. And the value check is a warning, not a failure, because a genuine festive-sale spike should not stop the pipeline.

Data contracts: agreeing before things break

A data contract is a written agreement between the team that produces data and the teams that consume it: which fields exist, their types and meanings, freshness, and how changes are announced. Data engineers increasingly push for these because most pipeline failures start as an unannounced upstream change. Even a short versioned schema file in the source team's repository, with a review step for changes, prevents a lot of 3 AM alerts.

A normal week, when nothing is on fire

Roughly: building a new ingestion from a SaaS tool the sales team adopted, reviewing a colleague's pull request, optimising one expensive query, a planning meeting with analytics engineers about new tables, and a rotation of on-call duty. Documentation is part of the week, not an extra.

Freshness, failed runs and time to recover

Pipeline success rate, freshness against SLAs, number of data incidents that reached consumers, time to recover, cost per workload, and delivery of new sources on the roadmap. Senior engineers are also judged on how much they reduce future incidents, not just how fast they fix current ones.

Platform leads, analysts and source-system owners

Data engineers report to a data engineering manager or head of data platform. They work with application engineers who own source systems, analytics engineers and analysts who consume the tables, data scientists who need features, and security teams for access and personal data handling.

Confused with analytics engineer and backend engineer

An analytics engineer mostly works inside the warehouse, modelling tables for business use with SQL and testing tools. A data engineer owns getting data in and the platform it runs on. A backend engineer builds application services; data engineers build the flows around them. In smaller Indian companies one person may do all three.

Spark, Airflow and cloud warehouse signals

• "Build scalable ETL/ELT pipelines, Airflow, Spark" — core data engineering.
• "Kafka, streaming, CDC" — real-time ingestion; a more specialised variant.
• "Data modelling, dbt, semantic layer" — analytics engineering focus.
• "Informatica, SSIS, DataStage" — traditional ETL tools, common in services and banks.
• "Maintain dashboards and reports" — probably an analyst role with a data engineer title.

Own one pipeline end to end, including its pager

Take the on-call rotation early, shadowed, because it teaches the system fastest. Add row-count and schema checks to one pipeline that lacks them. By the end of the third month, own one new ingestion end to end, including its runbook.

Senior data engineer, platform or architecture

Senior data engineer, data platform engineer, data architect, or a move towards ML platform and MLOps work. Some move into forward-deployed roles, where integrating messy client systems is the daily work; the FDE hub explains that path. The FDE for Freshers programme covers SQL, Python services, testing, APIs and deployment foundations that underpin this role.