Data · Since 2017 · 18 TB/day in production

Apache Spark

Distributed processing for the workloads that genuinely need it — batch and streaming.

Our position

Spark is the workhorse of our large-scale data processing: batch transformations over historical data, structured streaming from Kafka into lakehouse tables, and the heavy joins and aggregations that produce the gold-layer marts our clients report from.

It is also frequently the wrong tool, and we say so. Spark carries real overhead — cluster startup, shuffle cost, operational complexity — that only pays back above a certain volume. Below that threshold, DuckDB or Polars on a single machine will finish faster and cost a fraction. We set that threshold explicitly during architecture rather than defaulting to a cluster.

Where it does fit, the difference between a Spark job that costs ₹200 a day and an identical-output job that costs ₹9,000 is entirely in how it is written: partitioning, join strategy, shuffle avoidance and file layout. That tuning is the substance of our Spark practice.

Where it fits

Batch transformation over terabytes of historical plant or transaction data

Structured Streaming from Kafka into bronze and silver lakehouse layers

Large joins across sources that do not fit a single machine

Backfills and reprocessing of historical data after a logic change

Feature generation for forecasting and machine learning

Why we choose it

What Apache Spark genuinely gives us

Genuine horizontal scale

Workloads that exceed a single machine run across a cluster without the job being rewritten.

One engine, batch and streaming

Structured Streaming uses the same API as batch, so logic is written once and the delivery mode is a configuration choice.

Native lakehouse writes

First-class support for Iceberg and Delta, including merge operations, schema evolution and time travel.

Runs everywhere

EMR, Dataproc, Databricks, Azure Synapse or self-managed on Kubernetes — the same code deploys to all of them.

Where Spark cost actually goes

Shuffle dominates. Any operation that redistributes data across the cluster — a wide join, a groupBy on a high-cardinality key, a repartition — writes to disk and moves over the network, and a job that shuffles four times when it could shuffle once costs roughly four times as much.

Our tuning follows a consistent order: read only the columns and partitions actually needed, broadcast the small side of joins so the large side never moves, pre-partition on join keys where the same join recurs, and compact small files, which are the silent killer of both cost and read performance in lakehouse tables.

Predicate and partition pushdown verified in the physical plan, not assumed.

Broadcast joins for dimension tables under the threshold, with the threshold tuned to the cluster.

Adaptive Query Execution enabled and its effect measured rather than trusted.

Small-file compaction scheduled as routine table maintenance.

Spot or preemptible instances for interruptible batch, on-demand for streaming.

Structured Streaming with real guarantees

Streaming jobs that work in a demo and lose data in production usually miss three things. Watermarking with an agreed late-arrival tolerance, so late events are handled by a documented rule rather than dropped arbitrarily. Idempotent sinks with checkpointed offsets, which is what actually delivers exactly-once rather than the phrase appearing in a design document. And bounded state, because unbounded aggregation state grows until the executor dies.

We also keep raw events in Kafka with retention long enough to replay, so a logic error can be corrected by reprocessing rather than by accepting a permanent gap.

Honest warnings

How Apache Spark projects usually go wrong

These are the failure modes we look for first when we are called in to rescue somebody else's implementation.

Using Spark for small dataCluster overhead exceeds the work. Polars or DuckDB finishes faster on one machine, at a fraction of the cost.

collect() on a large DataFramePulls the whole dataset into the driver and kills the job. Almost always a design mistake rather than a necessity.

Skewed join keysOne partition does all the work while the cluster idles. Salting or adaptive skew handling resolves it once identified.

Thousands of tiny output filesCripples subsequent reads and inflates storage API costs. Compaction is not optional maintenance.

Straight answers

Apache Spark questions we get asked

Including where we would recommend something other than Apache Spark. Call +91 70033 91355 and you will get the same answer from an engineer.

+91 70033 91355 WhatsApp

As a rough guide, when a single transformation regularly exceeds what a large single machine can hold comfortably — call it a few hundred gigabytes of working set — or when the job must run alongside an existing Spark estate for operational consistency. Below that, DuckDB or Polars will usually finish faster and cost far less. We benchmark both during architecture rather than assuming.

Databricks if you want the least operational burden and are using its notebook, Delta and governance ecosystem — you pay a premium for that. EMR or Dataproc if you are already committed to AWS or Google Cloud and have platform engineering capability. Self-managed on Kubernetes when data residency or cost at very large scale justifies the operational effort. We run all four for clients.

Yes. Spark on Kubernetes with MinIO or HDFS for storage and Iceberg tables gives you a full lakehouse with no external connectivity. It requires more operational effort than a managed service, which we will quantify honestly, and for clients whose data genuinely cannot leave the premises it is the correct architecture.

Services built on Apache Spark

Data EngineeringPetabyte-scale pipelines, lakehouses and warehouses — built by engineers who have run them in production for cement, steel, solar and commerce clients.ETL & Data PipelinesBatch and real-time pipelines that run every day without drama — orchestrated, monitored, idempotent, and honest about failure.Data & Warehouse MigrationMoving warehouses, lakes and databases without downtime, without data loss, and with proof that every row arrived.Business Intelligence & AnalyticsDashboards people actually open, built on a modelled semantic layer so two departments never bring different numbers to the same meeting.

Technologies we pair it with

PythonThe language our data, scraping and AI work is written in.Apache KafkaThe durable, replayable spine every real-time system we build sits on.Databricks & Delta LakeLakehouse platform where Spark, streaming and machine learning share one governed layer.SnowflakeWarehouse of choice where SQL analytics dominates and operational simplicity is valued.

Next step

Tell us what is slowing your business down.

A 30-minute call with a senior engineer — not a salesperson. You leave with an architecture sketch and an honest cost range, whether or not you hire us.

Book a discovery call