Why I Didn’t Start with Spark, Pinot, or ClickHouse

-


This question comes up a lot, so it’s worth answering directly:

If you’re doing analytics at scale, why not Spark, Pinot, or ClickHouse?

They’re powerful tools.

They’re also optimized for a different problem.

The Kind of Scale That Matters

There are two kinds of scale:

  1. Data volume

  2. Semantic complexity

Most modern analytics stacks optimize for the first.

This work is mostly about the second.

Spark: Great for Pipelines, Not Reasoning

Spark excels at:

  • Batch processing

  • Large transformations

  • Schema-on-read workloads

But semantic analytics needs:

  • Low latency

  • Fine-grained validation

  • Interactive feedback

You can build that on Spark — but you’ll spend most of your time:

  • Managing jobs

  • Handling latency

  • Debugging execution graphs

It’s a mismatch for question-driven systems.

Pinot and ClickHouse: Fast, But Opinionated

Pinot and ClickHouse are impressive.

They shine when:

  • Queries are known in advance

  • Dimensions are stable

  • Aggregation paths are predictable

Semantic systems break those assumptions:

  • Users define models dynamically

  • Hierarchies evolve

  • New measures appear

You end up fighting the engine:

  • Rebuilding segments

  • Re-indexing constantly

  • Encoding semantics indirectly

Speed doesn’t help if the answer is conceptually wrong.

Semantics First, Engines Later

My bias has been:

  1. Make meaning explicit

  2. Make queries explainable

  3. Make performance incremental

Once semantics are stable, then:

  • Specialized engines make sense

  • Pre-computation becomes safe

  • Cost optimization becomes meaningful

Starting with a heavy engine too early locks you in.

PostgreSQL + Redis as a Control Plane

This combination works well as:

  • A semantic control plane

  • A correctness baseline

  • A performance sandbox

It lets you ask:

  • “What should this system do?”
    Before:

  • “How fast can it go?”

That ordering matters.

This Is Not an Anti-Big-Data Argument

At some point:

  • Pinot might be perfect

  • ClickHouse might slot in cleanly

  • Spark might power offline simulation

The key is timing.

You don’t want your infrastructure to decide your semantics for you.

Final Thought

Fast answers to the wrong question aren’t useful.

Semantic systems live or die on:

  • Trust

  • Clarity

  • Explainability

For that, simpler stacks often win — at least at the beginning.

If you’ve made different trade-offs and they worked, I’d genuinely love to hear about them.


Comments

Post a Comment

Popular posts from this blog

A Secure Blazor Server Azure Deployment Pipeline

-
Image
I wanted to kick off the year with something practical, modern, and production-ready. This post is based on my own experience deploying my new website, dotnetconsult.tech , using GitHub Actions into Azure App Service . In this post, I’ll walk through how to set up a secure development and deployment flow using: Blazor Server GitHub Azure App Service GitHub Actions with OpenID Connect (OIDC) This approach avoids stored secrets entirely and reflects how I now set up every new .NET project  whether it’s for a client, a SaaS product, or an internal tool.
Read more

Stop Wrapping EF Core in Repositories: Use Specifications + Clean Architecture

-
GitHub project which accompanies this article is available here    Wrapping Entity Framework Core in repositories has become a default in many .NET codebases. But defaults deserve to be challenged. This post shows: why repository-over-EF breaks down how Clean Architecture + Specification Pattern fixes it and how EF Core InMemory tests prove the approach works The problem with “Repository Pattern over EF Core” EF Core already gives you: DbSet<T> → repository behavior DbContext → unit of work Yet many projects add another repository layer anyway. It usually starts simple: Add(order) GetById(id) Update(order) Then order-processing requirements arrive: “Get open orders for customer” “Get orders awaiting payment older than 7 days” “Get paged orders sorted by date with items” Soon you’re staring at this: OrderRepository ├── GetOpenOrdersForCustomer(...) ├── GetOrdersAwaitingPayment(...) ├── GetOrdersWithItemsAndPayments(...) ├── ...
Read more

Server-Sent Events in .NET 10: Do You Really Need SignalR?

-
Image
Real-time updates have become a default expectation in modern web applications. Dashboards refresh automatically, notifications arrive instantly, and long-running operations stream progress back to the UI. In the .NET ecosystem, SignalR has traditionally been the go-to solution for these scenarios. However, with .NET 10 , Server-Sent Events (SSE) have become a first‑class, production-ready option that deserves serious consideration. In many cases, SignalR may actually be overkill . This post explores: What Server-Sent Events are How SSE works in .NET 10 The benefits of SSE over SignalR When SignalR is the right choice How to decide which approach fits your application What Are Server-Sent Events (SSE)? Server-Sent Events provide a simple, standards-based way for a server to push data to a browser or client over HTTP. Key characteristics: One-way communication (server → client) Built on plain HTTP Uses the text/event-stream content type Automatically reconnects if the connection dr...
Read more