Why Faster Algorithms Aren't Enough to Build High-Performance Data Platforms

[Megaladata]

There is a very common way to solve performance problems in software. Buy a bigger machine. If that does not work, add more RAM. Then more CPU. Then another machine. Eventually somebody says “cluster,” and now you have a new architecture, a monitoring stack, distributed execution, and several new job openings.

Sometimes that is exactly the right answer. But sometimes we are using infrastructure to compensate for decisions that were made much earlier – in the way data is represented, moved, allocated, calculated, and eventually shown to a user.

We have been building analytical platforms since 1998. Megaladata is the third generation of that technology. Over that time, one lesson has repeated itself often enough that we no longer think of performance as a feature.

You cannot take a fundamentally slow system, replace one Join implementation, install a better allocator, add multithreading, and suddenly call it a high-performance platform.

Performance is an architectural property. It emerges from dozens of decisions that reinforce one another. And some of the most important ones are surprisingly far away from the algorithm that appears to be doing the actual calculation.

This article is about those decisions. Not benchmarks. Not “our engine is X times faster.” Instead, this article looks at what happens underneath a data workflow: how data is represented, why some tables should never physically exist, why moving data can cost more than calculating it, how memory allocation can become a 15-minute problem, why splitting a task across CPU cores is the easy part, and why your backend can finish its job perfectly and still lose the performance battle in the browser.

Fast algorithms do not automatically produce fast pipelines​

Modern data platforms have access to excellent building blocks. Python gives us an enormous ecosystem. The JVM provides an extremely mature runtime. There are optimized libraries for machine learning, numerical methods, networking, storage formats, databases, compression, and almost anything else you might want.

We use existing libraries where they make sense. Writing every algorithm yourself is not a serious performance strategy. Quite often, someone else has already spent ten years optimizing exactly the function you need.

The interesting question is what happens between those functions. Suppose you have three extremely fast operations:

Operation A → Operation B → Operation C

Each operation may be beautifully optimized in isolation. But between A and B, and then again between B and C, the system still needs to answer a lot of questions:

• What does the result physically look like?
• Does it need to be materialized?
• Do we make a copy?
• Who owns the memory?
• Can the next operation read the existing representation directly?
• Does the result need conversion?
• Can execution happen concurrently?
• When can the memory be released?
• What happens if the next operation accesses rows in a completely different order?

That “glue” becomes part of your execution time. And on large workflows, it can become a very significant part.

This is why the most valuable part of our own core is not that every algorithm is proprietary. It is that the engine controls the dataflow execution architecture:

• how data is represented,
• where it physically lives,
• how nodes access one another,
• what is materialized,
• what is never materialized,
• how memory is allocated,
• how work is scheduled,
• and how CPU cores are used.

That distinction becomes especially important once we look at something as innocent as a table.

The table the user sees may not exist​

Imagine a simple visual workflow:

Import
↓
Filter
↓
Sort
↓
Calculation
↓
Aggregation


A perfectly reasonable mental model is this:

Table A
↓
Filter
↓
Table B
↓
Sort
↓
Table C
↓
Calculation
↓
Table D


Every node receives a table and produces another table. This is intuitive. It is also potentially very expensive.

Megaladata uses a different model. Nodes expose a common programmatic data-access interface.

Conceptually, the downstream node can say: “Give me row 10.” It does not need to know whether the upstream node physically stores row 10 in a conventional table at all. Under that common interface, different components can use very different physical representations.

Consider sorting. A naive implementation might create an entirely new table containing the same values in a new order. But the values already exist. Why duplicate them? Instead, the sorting component can store an array of indexes that defines the new access order. The source remains untouched. Conceptually:

Source:
[A][B][C][D]

Sorted representation:
[3][1][4][2]

The next node still sees rows in sorted order. But there is no second physical copy of the source table.

Now consider a calculated column. Suppose the workflow defines:

Revenue = Quantity × Price

One option is to immediately calculate Revenue for every row and allocate memory for the whole resulting column. But what if the downstream path never reads most of those values? In some cases, it is more efficient to keep the parsed expression and calculate a value only when it is requested.

So what looks logically like:

| Quantity | Price | Revenue |

may physically be closer to:

Quantity column
Price column
Expression: Quantity × Price

The Revenue column exists at the logical level. It may never exist as a fully materialized physical column.

This leads to one of the principles that has become central to the engine:

Universal access. Specialized representation.

The outside contract remains stable. The physical representation underneath can change. A text import can have one representation. Sorting can have another. A calculated field can have another. The next node does not need to care.

This separation has two important consequences. First, we can often store less. Second, the component can choose a representation optimized for the work it actually performs rather than forcing every operation into the same physical model.

Stop moving data​

Once you separate logical access from physical storage, another optimization becomes possible. Do not move data unless you actually need to.

In a straightforward pipeline implementation, data can easily travel like this:

Dataset
↓ COPY
Node A
↓ COPY
Node B
↓ COPY
Node C
↓ COPY
Node D

The problem is that a copy is not free. The CPU has to move the bytes. Memory bandwidth is consumed. New memory has to be allocated. The old memory eventually has to be released. Caches are affected. The working set grows. And if you repeat this across a large workflow, data movement itself becomes a substantial workload.

Our preferred rule is simple: If unique data already exists somewhere, leave it there whenever possible. Other nodes should access it rather than automatically creating another physical copy.

Internally, one of the mechanisms used for this is called Readers. The particular name is not very important. The architectural idea is. A downstream operation does not necessarily receive a newly materialized table. Instead, it receives a way to access the required data through a common interface.

This means a visual workflow with 100 nodes does not imply that the platform has created 100 complete copies of the dataset. The visual graph and the physical execution graph are not the same thing. That distinction can be summarized very simply: The workflow defines the logic. The core defines how data actually moves.

And the more complicated the workflow becomes, the more valuable that separation becomes.

In-memory is easy. Efficient in-memory is hard​

If performance is the goal, keeping data in memory seems obvious. RAM is much faster than disk. It has also historically been relatively cheap.

Well… At least that used to be the comforting part of the argument. Recent memory prices are doing their best to ruin this paragraph.

Either way, RAM has one property that has never disappeared: it is finite. So, an in-memory engine cannot simply mean: “Load everything into RAM.” That is not an architecture. That is a shopping strategy.

The real problem is how to get the performance advantages of RAM while minimizing the amount of memory required to execute the workflow. This is where the previous design decisions start working together.

• If sorting only needs indexes, do not create another table.
• If a calculated column can be evaluated lazily, do not materialize it.
• If the same values already exist upstream, do not copy them.
• And if an operation can work in blocks, do not require the entire dataset at once:

read block
↓
process
↓
release
↓
next block

The important metric is therefore not simply: “How many gigabytes of RAM does this server have?” A much more useful question is: “How much useful work can the engine perform per gigabyte of RAM?”

That is a different way of thinking about in-memory processing.

Allocating memory is also work​

Even after we reduce the amount of data stored in memory, another problem remains. Something still needs to allocate that memory. And later, something needs to free it. Applications create objects constantly:

allocate
allocate
allocate
allocate
...
free
free
free

At modest workloads, allocator behavior can remain almost invisible. At large workloads, it becomes part of the engine.

In 2017, our engineering team benchmarked different general-purpose memory allocators. TCMalloc eventually became the primary allocator. But raw allocation throughput was not the only property we cared about.

Consider memory returned to the operating system. An allocator can be extremely good at obtaining new memory and still be less effective at releasing unused memory back to the OS. For a short-lived command-line tool, that may not matter very much. For a long-running analytical platform processing changing workloads, it does. A workflow may temporarily consume a large amount of memory. After it completes, you want that memory to become useful again.

But general-purpose allocators are still designed to solve a very broad problem. Sometimes the engine knows something they do not.

When a worse allocator is better​

Imagine that one component repeatedly creates millions of objects. All of them have exactly the same size.

A general-purpose allocator must support many different object sizes and many possible allocation patterns. But our component does not need any of that flexibility. It needs one very specific operation to be extremely fast. That allows a much simpler strategy.

Allocate one large memory region:

+---------------------------------------------------+
| |
| LARGE BLOCK |
| |
+---------------------------------------------------+


Then carve fixed-size objects out of it:

+----+----+----+----+----+----+----+----+----+
| o1 | o2 | o3 | o4 | o5 | o6 | o7 | o8 | o9 |
+----+----+----+----+----+----+----+----+----+

There is an obvious limitation. This allocator is much less universal. It cannot efficiently allocate every arbitrary size in every possible pattern. But that is precisely why it can be faster at its one job.

And when the component is destroyed, instead of freeing thousands or millions of tiny allocations individually, the whole large region can be released in one operation.

This sounds like a low-level optimization that no user should ever care about. Until the user has to care about it.

The 15-minute shutdown​

We had a case in an older generation of the system where the actual calculation finished successfully. The user was done. The application was closed. And then the application spent an enormous amount of time… closing. In one case the team still remembers, around 15 minutes.

The computation was not running anymore. There was no analytical workload left. The program was simply freeing a huge structure, object by object.

From a user's point of view, this is absurd: “The work finished ten minutes ago. Why is the program still alive?” But from the allocator's point of view, it was still performing a very large amount of legitimate work.

Changing the memory model so that these structures could be released in blocks turned that long cleanup into something close to immediate.

This story shows how real optimization usually happens. Nobody sat down in 1998 and produced a 30-year master plan containing: “In twenty years we shall require a fixed-size specialized allocator.”

The system grows. The datasets grow. Usage patterns change. And eventually a bottleneck appears in a place that was irrelevant when the architecture was younger. Then you fix it.

Performance engineering is often archaeology performed on your own software.

Parallel by default​

Now let's move from memory to CPU. Multicore processors are not exotic hardware anymore. They are the normal hardware. Even inexpensive laptops have several CPU cores.

So our default philosophy is straightforward: If something can reasonably be executed in parallel, we try to execute it in parallel by default.

By default is important. The user should not have to write thread-management code. They should not need to manually divide a table into eight pieces because the machine has eight available cores. They should not need to understand the engine's internal thread pool. They build the workflow. The engine decides how to use the available CPU.

Of course, not every operation can be parallelized efficiently. Some operations have dependencies. Some workloads are too small for the scheduling overhead to make sense. Some phases eventually have to synchronize. But wherever parallel execution is useful, it should be an execution property of the engine, not another programming task imposed on the analyst.

And this leads to another interesting lesson.

Splitting work is usually easier than merging it​

Imagine sorting a very large dataset. Splitting the work is conceptually simple:

┌── Part 1 → Thread 1
│
Large dataset ───────┼── Part 2 → Thread 2
│
├── Part 3 → Thread 3
│
└── Part 4 → Thread 4


Each thread sorts its own partition. Great. Now we have four sorted sequences. The actual problem has merely moved:

Sorted 1 ──┐
Sorted 2 ──┤
Sorted 3 ──┼── ??? ──→ Final sorted result
Sorted 4 ──┘

How do we merge them without destroying the benefit we just gained? We want to avoid copying huge arrays. We must preserve the correct ordering. And the resulting representation must still be efficient for the next downstream operation. A beautiful parallel sort is not very useful if every following node now pays a penalty to access its output.

This pattern appears in many parallel algorithms. The interesting engineering is not only: “How can we divide this into N jobs?” It is also: “What representation will exist after those N jobs finish, and how expensive will it be to consume?”

So a useful rule is:

Splitting work is relatively easy. Merging the result efficiently is hard.

Your thread pool is also a memory-management system​

Memory and parallelism are often discussed as different optimization topics. In real systems, the boundary is much less clean.

Modern allocators can associate memory regions or caches with individual threads. Imagine a thread that just processed a very large task. The work ends. The thread becomes idle. CPU utilization drops to zero. Everything looks fine.

Except a large amount of memory may still be effectively associated with that idle thread. Another part of the workflow needs memory, while RAM is still tied up in resources attached to a worker that is doing nothing.

So the lifecycle of threads also becomes part of memory management. Our thread pool can respond to workload conditions. If work is waiting, there are no free workers, and CPU capacity remains available, the engine can add a thread. If a thread stays idle for long enough, the engine can remove it so its associated resources can be released.

That creates one of our favorite examples of systems behavior: CPU optimization suddenly becomes RAM optimization.

This is why optimizing an execution engine is rarely a collection of isolated tricks. Subsystems interact.

Python: paying a local cost to win globally​

A nice example of this interaction came from Python integration. Users wanted to execute Python code inside workflows. The initial product requirement sounded trivial:

Input data
↓
Python node
↓
User code
↓
Output data

Pass data in. Execute the script. Return the result. Done.

Except it was not done. Megaladata workflows can contain independent branches. Suppose two independent branches both reach Python nodes. From the engine's point of view, there is no dependency:

┌── Python A
Workflow ───┤
└── Python B

They should be able to run concurrently.

The original integration used the standard Python API inside the process. The result was an architectural mismatch: the Megaladata execution model wanted to run independent tasks in parallel, while the Python runtime integration effectively forced those nodes to wait for one another. That was not acceptable for an engine whose normal rule is “parallel when possible, by default.”

So the integration architecture changed. Python execution moved to separate processes:

┌── Python Process A
Engine ───────┤
└── Python Process B

This creates overhead. The data must be prepared. It must cross a process boundary. The Python process must read it. If you benchmark only one node, this may look like a regression.

But that is the wrong scope for the benchmark. Now independent Python tasks can actually run at the same time. The entire workflow scales better.

That is an important engineering trade-off:

Sometimes the optimal design for one node is not the optimal design for the whole workflow.

And sometimes you should deliberately make one local operation slightly more expensive to preserve system-level parallelism and scalability.

Backend performance is only half the story​

Suppose all of this works perfectly. The engine processes 50 million rows. Memory usage is efficient.

Parallelism works. The calculation finishes quickly. We are done, right?

Not yet. The user wants to see a chart. What happens if the server sends 50 million points to the browser? Nothing good.

Even if the network were infinitely fast, the idea itself does not make sense. Suppose the chart is 1,000 pixels wide. A screen with 1,000 horizontal pixels physically cannot display 50 million independent X positions. Most of the transferred information cannot possibly become visible. So why send it?

The same applies to a table. The dataset might contain 10 million rows. The browser may currently display 50. Sending all 10 million rows is not “giving the user the data.” It is wasting network bandwidth, browser memory, serialization time, and rendering work.

So the server-browser layer has its own performance architecture. Data is transferred in binary form. Small requests can be batched. And different visualization types can have specialized server-side providers.

For a table, the client receives the window of rows it actually needs. As the user scrolls, additional windows can be requested. The experience looks like a normal large table even though the browser never holds the entire dataset.

For charts, the server can use decimation. If the original dataset contains tens of millions of points but the chart has only 1,000 pixels of horizontal resolution, the server prepares the amount of information needed to represent the graph correctly rather than blindly transferring every source point.

And here the architectural principle from the beginning appears again: Universal access. Specialized representation. One conceptual interaction for the user. Different internal representations optimized for different tasks.

Performance is the whole path​

At this point, we can trace almost the entire lifecycle of a piece of data:

SOURCE
↓
PHYSICAL REPRESENTATION
↓
ACCESS BETWEEN NODES
↓
MEMORY MANAGEMENT
↓
COMPUTATION
↓
PARALLEL EXECUTION
↓
RESULT PREPARATION
↓
NETWORK
↓
BROWSER
↓
HUMAN

Every transition can create work. Every representation can create memory pressure. Every unnecessary copy can become bandwidth. Every synchronization point can waste CPU. Every visualization can send data that nobody can see.

This is why asking: “Which algorithm makes your platform fast?” is almost the wrong question.

There is no single secret algorithm. There is no magic button. There are many architectural decisions accumulated over years of running real workloads:

• Keep unique data where it already lives.
• Do not create a full table if an index is enough.
• Do not materialize a calculated column if it can be evaluated on demand.
• Do not copy data just because two workflow nodes are visually separate.
• Use RAM aggressively, but economically.
• Use specialized allocators when a general-purpose solution is unnecessary.
• If work can be parallelized, execute it in parallel by default.
• Manage the lifecycle of worker threads, not just their CPU work.
• Accept local overhead when it improves global scalability.
• Do not send the browser information the human cannot physically see.

There is one architectural principle behind all of them: Control the Entire Lifecycle of Data

That, much more than any individual optimization, is what high-performance data engineering looks like.

Niklaus Wirth, quoting his colleague Martin Reiser, captured the problem decades ago: “Software is getting slower more rapidly than hardware becomes faster.”