Databricks Is Improving Lakebase Postgres Compute Cache, but Autoscaling Is the Real Test

Databricks is improving Lakebase Postgres compute cache performance with a new configuration that produced up to twice the throughput in its production measurements. The change is already active on fixed-size Databricks computes with at least 80 compute units. Yet the harder problem remains unresolved: bringing the same memory strategy to computes that expand and contract while Postgres is running.

The update changes where frequently requested database pages spend their time. Large fixed-size computes now place up to 75% of available memory into Postgres shared buffers. These buffers are the database engine’s fastest in-memory cache. Databricks also backs that memory with 2 MB huge pages to reduce address-translation work inside the operating system.

That combination targets a central tension in disaggregated databases. Separating durable storage from compute enables rapid restarts, independent scaling, and inexpensive object storage. However, it also places more distance between a running query and the data it needs. Amazon Aurora, AlloyDB, and other cloud databases face versions of this problem, but Databricks is applying its fix to the Neon architecture beneath Lakebase.

Improving Lakebase Postgres Compute Cache Starts With Fixed Computes

Databricks has moved the hottest part of the Lakebase cache from local disk into Postgres memory, but only on sufficiently large fixed-size machines.

The change became public on September 9, 2026. According to the company’s compute cache update, it is active on fixed-size Lakebase computes with at least 80 compute units. The equivalent Neon rollout covers fixed computes with at least 18 compute units.

Lakebase uses disaggregated storage, meaning Postgres compute and durable data storage operate as separate layers. A compute node runs the database engine but does not own the authoritative copy of the data. That design lets operators replace, restart, or resize compute without moving an entire database.

Reads travel through a hierarchy. Postgres first checks shared buffers in DRAM. If the page is absent, the earlier configuration checks a local file cache stored on the compute node’s NVMe drive. A further miss sends the request to the distributed storage layer, where pageservers reconstruct and return database pages.

That local file cache solved an important early problem. Standard Postgres installations normally benefit from both shared buffers and the operating system’s page cache. Lakebase does not route remote page reads through a conventional local filesystem, so it cannot depend on the normal second layer.

Databricks and Neon therefore built the local file cache as an elastic substitute. It could occupy capacity beyond the static Postgres buffer allocation and adjust alongside serverless compute. This arrangement supported autoscaling without requiring Postgres to resize shared buffers while running.

However, the previous configuration capped shared buffers at roughly 1 GB. Most of the remaining compute-cache capacity, which could reach 75% of DRAM, went to the local file cache. On large machines, that balance sent many otherwise cacheable reads through NVMe instead of memory.

The new fixed-compute setup removes the local file cache from that path. It assigns 75% of DRAM to shared buffers, keeping a much larger working set inside Postgres. An 80-unit Databricks endpoint should report 120 GB when an administrator runs show shared_buffers, according to the announcement.

DRAM access avoids the disk input and output required by the local cache. It also gives Postgres direct knowledge of cached pages and their use. An operating-system cache or separate disk layer has less information about database state when deciding which page to evict.

This is the immediate meaning of improving Lakebase Postgres compute cache. It is not a new index, query planner, or storage format. Databricks changed the memory allocation and virtual-memory machinery surrounding the existing Postgres execution engine.

The scope matters. Fixed-size deployments can allocate shared buffers at startup because their memory ceiling remains known. Autoscaling endpoints cannot make the same assumption. Their available memory changes with workload demand, while the standard Postgres setting remains static after startup.

That is why the first release is meaningful without being complete. Databricks has shown that more direct use of memory can improve large Lakebase endpoints. It has not yet delivered the same design across the serverless operating model that makes disaggregated Postgres attractive.

Why Lakebase Cache Performance Became a Priority

Storage separation creates operational flexibility, but every cache miss exposes the latency and CPU costs hidden by that flexibility.

Traditional Postgres usually combines the query engine, write-ahead log, and database files on one machine or tightly connected storage. The operating system can retain recently accessed file pages in its page cache. Postgres also keeps selected pages in shared buffers.

That arrangement can duplicate data in memory. Databricks gives the example of a machine with 4 GB of RAM and 1 GB assigned to shared buffers. Reading 1 GB through the filesystem can leave another copy in the operating-system cache, consuming 2 GB to cache 1 GB of database pages.

Lakebase changes the path. Its storage architecture divides the system into stateless Postgres compute and a durable storage service. Safekeepers replicate write-ahead log records, while pageservers reconstruct page versions and persist them to object storage.

The architecture allows compute to scale independently from stored data. It supports features such as scale-to-zero, rapid branching, read replicas, and failover without copying an entire database. Those benefits depend on keeping durable state outside the compute node.

Yet stateless compute still needs state close at hand while processing queries. Applications repeatedly access indexes, table pages, catalog records, and other structures. If those pages remain in DRAM, reads complete through the lowest-latency part of the hierarchy.

A miss in shared buffers takes a longer route. The local NVMe cache is faster than remote storage, but it still requires disk access and extra software processing. A miss there reaches the pageserver, potentially adding network traffic and page reconstruction work.

The resulting pressure falls on teams using large Lakebase instances for latency-sensitive applications. These customers provision substantial memory, but the previous 1 GB shared-buffer ceiling prevented Postgres from using most of that memory through its fastest cache path.

The mismatch becomes more visible as the working set grows. A working set is the collection of pages an application accesses frequently during a given period. When it exceeds the small memory cache, requests spill into slower layers even if the machine has ample DRAM.

Databricks needed a better answer because Lakebase now serves operational workloads, not only analytical jobs. User-facing applications care about tail latency, which reflects slower requests near the far end of a latency distribution. Averages can remain acceptable while p99 requests still create visible pauses.

AI applications also place uneven pressure on operational databases. An agent can trigger bursts of retrievals, writes, checkpoints, and concurrent tool calls. Predictable cache behavior becomes more important when traffic arrives in clusters rather than through a steady request stream.

This does not mean every workload benefits equally. A dataset that already fits within the earlier shared-buffer allocation has less room to improve. A scan-heavy workload with little page reuse might continue reaching lower cache tiers regardless of buffer size.

The strongest candidates are large endpoints with reusable working sets that exceeded the former 1 GB limit. Their existing memory can now hold more hot pages directly inside Postgres. This is why Databricks presented production examples rather than promising a universal doubling.

Lakebase cache performance therefore depends on workload shape as much as machine size. The new configuration removes one architectural bottleneck. It does not repeal the basic rules governing cache locality, query design, indexing, or memory contention.

Huge Pages Make the Larger Buffer Pool Practical

Allocating more memory to Postgres would create its own overhead unless Lakebase also reduced the cost of mapping that memory across database processes.

Postgres uses a process-based architecture. Each active connection generally receives a backend process, and each backend maps the shared-buffer region into its virtual address space. The operating system maintains page-table entries that translate those virtual addresses into physical memory locations.

Standard Linux memory pages are commonly 4 KB. With that page size, 1 GB of shared buffers requires 262,144 page-table entries for each process mapping the region. Databricks calculates that 32 GB of shared buffers across 512 backends can require roughly 4.3 billion entries.

The company estimates those entries would consume about 32 GB of page tables merely to map a 32 GB cache. That is an extreme illustration of how a larger buffer allocation can shift overhead elsewhere. More cache capacity is not automatically useful when memory management consumes excessive RAM and CPU time.

The processor also maintains a translation lookaside buffer, or TLB. This hardware cache stores recent virtual-to-physical address translations. A data page can be present in Postgres memory while the CPU still pays a penalty to locate it after a TLB miss.

Huge pages reduce that pressure by mapping memory in larger units. Lakebase uses explicit 2 MB HugeTLB pages for the new fixed-compute configuration. Each huge page covers 512 times as much memory as a standard 4 KB page, reducing the number of required mapping entries by the same factor.

PostgreSQL already exposes operating-system huge-page controls. Its resource documentation explains that explicit huge pages can reduce overhead associated with large contiguous shared-memory regions. The benefit is especially relevant when shared_buffers becomes very large.

Virtualization makes implementation harder. Lakebase runs Postgres inside lightweight guest virtual machines on bare-metal hosts. Address translation therefore crosses both guest and host layers. Huge pages must remain consistently backed across the host, hypervisor, and guest to preserve their intended benefit.

Databricks says it added dedicated huge-page support throughout that stack. Large fixed-size virtual machines start with a predetermined quantity of huge pages. After Postgres initializes, the compute system releases capacity that the database does not require.

The company chose explicit HugeTLB pages instead of transparent huge pages. Transparent huge pages let the operating system promote memory automatically, but that behavior is best effort. Explicit reservation gives the database environment tighter control over page availability and layout.

In Databricks benchmark tests, huge pages reduced tail read latency by as much as approximately 40%. CPU utilization declined by as much as approximately 30%. These are company measurements, not independent guarantees for every Lakebase workload.

The distinction matters because huge pages do not explain the entire reported improvement. Two changes arrived together: more data remained in shared buffers, and access to that larger region required fewer page translations. Workloads can benefit from each mechanism in different proportions.

An administrator can inspect the deployed setting through Postgres. Running show huge_pages should return on for an eligible 80-unit Lakebase endpoint. Combined with the shared_buffers value, this offers a direct way to confirm whether the new configuration reached a compute.

This mechanism also explains why simply raising shared_buffers is not a complete answer for ordinary Postgres deployments. Memory reserved for the database must coexist with connections, query operations, maintenance jobs, and operating-system needs. A large allocation can create new constraints if the surrounding system is not designed for it.

Databricks controls the virtual machines, compute images, cache path, and storage protocol. That end-to-end control lets it coordinate huge-page reservation with Postgres startup. A self-managed team would need to tune these layers independently and validate the result under its own workload.

How Lakebase caching works is therefore more complicated than “use more RAM.” The improvement depends on placing hot data in the right memory region, mapping that region efficiently, and preserving enough memory for everything outside the buffer pool.

Production Results Show Gains, Not a Universal Baseline

Databricks reports substantial improvements from three production endpoints, but the published examples do not establish a platform-wide performance average.

The first example received the new configuration around 06:10 UTC on August 11. Accessed Postgres blocks per second doubled, which Databricks used as a throughput proxy. Storage GetPage requests fell from about 8,000 per second to about 1,500.

The customer also reported lower median and p99 latency compared with the previous day, week, and month. The release does not provide the underlying latency values, workload definition, query mixture, or controlled comparison environment. Readers should treat the observation as a production result rather than a standardized benchmark.

A second endpoint changed around 01:30 UTC on August 14. Its reported throughput increased by approximately 43%, while its compute cache hit rate approached 100%. Requests were served almost entirely from shared buffers after the rollout.

The third endpoint changed on August 15. Databricks says CPU consumption fell from 20 cores to four, the cache hit rate approached 100%, and measured throughput doubled. The fivefold reduction in CPU use stands out, but the public post does not specify whether every external workload condition remained constant.

Together, the examples support a credible mechanism. More requests hit DRAM, fewer reads reached the distributed storage service, and the processors spent less time on address translation. Those outcomes align with the design changes.

They do not show that every qualifying compute becomes twice as fast. Databricks uses “up to” language, and the three results vary. One recorded a 43% throughput increase, while two reached roughly double the prior measurement.

Workload composition remains the largest variable. Cache-sensitive queries with repeated access to a large but bounded working set should gain more. Write-heavy work, low-reuse scans, lock contention, inefficient queries, or network-bound application logic can limit the visible improvement.

The old local file cache also performed useful work. It offered greater capacity than the original shared-buffer allocation and avoided many storage-layer requests. Moving hot pages from NVMe into DRAM improves the fastest path, but removing that secondary tier changes behavior when the working set exceeds available memory.

Databricks says a miss now proceeds from shared buffers to distributed storage on these fixed computes. That raises an important question for unusually large or shifting workloads. A higher memory hit rate can coexist with a steeper penalty for pages that fall outside the enlarged buffer pool.

The published endpoints appear to have benefited because their active data fit well within the larger allocation. Nearly 100% hit rates in two examples suggest strong locality. Applications with less locality might produce a different balance between faster hits and remote misses.

Restart behavior deserves attention as well. Shared buffers are transient, so a newly started compute normally begins without its hot working set in memory. Databricks separately documents automatic cache prewarming, which repopulates frequently used data during planned updates.

Prewarming can reduce the cold-cache penalty after an update, but applications can still experience a short connection interruption during the restart. Drivers, pools, and retry logic must handle that event. Cache improvements do not remove the need for connection resilience.

Independent benchmark results would strengthen the case. Useful tests would disclose dataset size, connection count, query distribution, buffer state, compute size, and latency percentiles. They would also compare the old and new cache paths under both steady and changing working sets.

For now, the evidence supports a narrower conclusion. Improving Lakebase Postgres compute cache appears effective for the large production endpoints Databricks measured. The magnitude for another application remains an empirical question that operators should answer with their own latency, hit-rate, CPU, and storage-read metrics.

The Main Conflict Is Fixed Memory Versus Autoscaling

The current release improves the simpler half of Lakebase, while the product’s serverless promise depends on resizing cache memory without restarting Postgres.

The shared_buffers setting is normally chosen before Postgres starts. Changing it requires a restart because the database establishes its shared-memory region during initialization. That behavior conflicts directly with an autoscaling compute that changes memory capacity while processing traffic.

A fixed compute avoids the conflict. Databricks knows how much memory the virtual machine has, allocates 75% to shared buffers, reserves the corresponding huge pages, and starts Postgres. The allocation can remain unchanged for the machine’s lifetime.

An autoscaling endpoint must expand and shrink. When demand rises, Postgres should gain buffer capacity and the guest should receive the exact quantity of huge pages required to map it efficiently. When demand falls, both resources should return without corrupting active state or forcing disruptive restarts.

That is a substantially different engineering task. Shared memory can contain pages that active backends are reading, modifying, pinning, or inspecting. Shrinking the region requires safe coordination with cache eviction and concurrent database activity.

Databricks says it has developed a protocol that scales huge pages alongside dynamic shared buffers. The company plans to explain that implementation in a second technical post. It also intends to work with the open-source PostgreSQL community on the underlying machinery.

Until that work ships, Lakebase has two cache strategies. Large fixed computes receive the enlarged shared-buffer design. Autoscaling computes continue relying on the existing combination of conservatively sized shared buffers and the local file cache.

This split puts pressure on the product’s positioning. Fixed capacity offers the clearest performance improvement today, while autoscaling offers the flexibility associated with a serverless database. Customers cannot yet assume they receive both characteristics from the same compute mode.

That does not make autoscaling inferior for every deployment. Variable or intermittent applications can value scale-to-zero and elastic capacity more than the lowest possible cache latency. A steady, memory-intensive production service may prefer predictable fixed resources.

The decision also depends on workload growth. Fixed computes require operators to select sufficient capacity in advance. Autoscaling can absorb traffic changes, but the current cache path can send more hits through local NVMe when the working set exceeds the small shared-buffer allocation.

This is the real competitive test for Lakebase cache performance. Other managed Postgres services also combine distributed durability, local caching, replicas, and elastic resource management. Architectural details differ, so headline benchmark numbers rarely support a clean product ranking.

Databricks must instead show that its storage separation does not impose an avoidable penalty on the workloads it targets. Delivering dynamic shared buffers would let Lakebase retain its stateless-compute model while putting more elastic memory under direct Postgres control.

Upstream collaboration could extend the impact beyond Lakebase. Dynamic buffer sizing has relevance for containerized and elastic Postgres environments where assigned memory changes over time. However, Databricks has not yet published the code, review status, or release path described in its announcement.

That uncertainty should remain explicit. The company has stated a direction and says the autoscaling protocol exists. It has not provided a release date, supported compute range, or production measurements for autoscaling shared buffers.

The fixed-compute rollout is therefore both an improvement and a preview. It validates the cache placement and huge-page mechanisms under stable memory conditions. The next phase must prove those mechanisms can follow a changing machine without sacrificing availability or predictable performance.

What Lakebase Users Should Measure Now

The relevant question is not whether the published benchmark looks impressive, but whether an eligible workload becomes faster without developing a new miss penalty.

Eligible users can first confirm the configuration. show shared_buffers reveals the current Postgres buffer allocation, while show huge_pages reports whether explicit huge pages are active. An 80-unit Databricks endpoint should show 120 GB and on, based on the company’s example.

Configuration alone does not establish value. Teams should compare cache-hit rate, storage GetPage activity, CPU consumption, throughput, and p50 and p99 query latency across equivalent traffic windows. A comparison should account for deployments, data growth, maintenance, and application changes.

Cache-hit rate deserves context. A rate near 100% can indicate that the active working set fits in memory. It can also hide differences in request cost, query frequency, or workload mix if examined without throughput and latency.

Storage reads provide another direct signal. A drop suggests more pages remain inside Postgres rather than reaching pageservers. Databricks reported approximately 5.3 times fewer GetPage requests in its first example, based on the decline from about 8,000 to 1,500 per second.

CPU measurements can reveal the benefit of huge pages and avoided disk-cache handling. However, lower CPU use is most useful when paired with stable or increased throughput. A quiet period can reduce both CPU and completed work without reflecting an efficiency gain.

Teams should also inspect restart and warm-up behavior. Planned updates trigger compute restarts, although Databricks says they typically take only a few seconds. Testing connection retries and tail latency around those windows can expose operational effects that steady-state charts miss.

A concrete scenario is a transactional application whose active indexes and frequently accessed rows occupy tens of gigabytes. Under the former configuration, only a small portion could remain in shared buffers. Many hits descended into the local NVMe cache despite unused potential in DRAM.

After the update, that working set might fit almost entirely within the enlarged buffer pool. The team should expect fewer storage requests, lower read latency, and reduced CPU overhead. If those signals do not move, another bottleneck likely dominates.

An AI agent service offers a second scenario. It might store conversation state, tool results, job status, or vector metadata in Postgres. Bursty concurrent reads can benefit when frequently reused pages remain in memory, but connection counts and query patterns still determine pressure on backend processes.

Engineers reviewing this kind of change need shared evidence, not isolated screenshots. A searchable engineering knowledge base can preserve benchmark conditions, query plans, configuration snapshots, and rollout observations for later comparison.

The evaluation should also include the failure case. If the working set exceeds the enlarged shared buffer, fixed computes no longer have the previous local file cache in the middle. Measuring latency during cold starts, large scans, and sudden working-set changes will show whether remote misses become more visible.

None of these checks requires accepting or rejecting the company’s headline claim. They translate the proposed mechanism into observable signals. If more reads hit shared buffers while CPU and tail latency decline under equivalent demand, the update is working for that workload.

If throughput remains flat, teams should examine lock waits, application networking, query plans, indexes, and write pressure before attributing the outcome to Lakebase. A cache change cannot resolve every source of database latency.

Three Signals Will Decide Whether the Cache Redesign Matters

Autoscaling delivery, independent workload evidence, and upstream PostgreSQL progress will determine whether this becomes a broad Lakebase advantage.

The first signal is a production release of dynamic shared buffers for autoscaling computes. Databricks needs to show that shared-buffer capacity can grow and shrink alongside memory while huge-page backing remains correctly sized. A release with disclosed eligibility, rollout behavior, and operational limits would strengthen the company’s architectural case.

Production measurements should accompany that release. The useful comparison is not autoscaling against an unrelated fixed benchmark. It is the same elastic workload before and after dynamic buffers, including scale-up events, scale-down events, cache-hit rates, CPU use, and p99 latency.

If autoscaling reaches comparable cache efficiency without disruptive restarts, the present analysis becomes stronger. It would show that disaggregated Postgres can combine elastic compute with a large engine-managed memory cache. A prolonged delay would leave the fastest path confined to fixed capacity.

The second signal is broader benchmark evidence. Databricks has published three favorable production examples, but users need results across different working-set sizes and query patterns. Independent tests should include read-heavy transactions, mixed reads and writes, high connection counts, cold starts, and workloads larger than available DRAM.

Evidence of consistent gains would support the claimed mechanism. Highly variable results would not invalidate the release, but they would narrow the set of applications likely to benefit. A regression on cache misses would require closer attention after removal of the local disk tier.

The third signal is visible progress in open-source PostgreSQL. Databricks says it plans to collaborate upstream on dynamic shared buffers. Concrete proposals, patches, technical discussion, and reviewer feedback would reveal how much of the solution belongs in Postgres itself.

Upstream acceptance would give the design wider technical scrutiny and make it useful beyond one vendor. It could also reduce the long-term gap between elastic cloud environments and Postgres settings designed around a fixed machine.

Failure to upstream the work would not prevent Databricks from shipping a platform-specific implementation. It would, however, make compatibility, maintenance, and portability harder for outsiders to assess.

Improving Lakebase Postgres compute cache is already producing measurable results on selected fixed computes. The more consequential test is whether Databricks can make the cache dynamic without weakening the elasticity that storage separation provides.

For teams running eligible endpoints, the next action is straightforward: verify the settings, capture a stable baseline, and compare real workload metrics after rollout. For autoscaling users, watch for the second engineering release before assuming the same gains apply. Which matters more to your application today, fixed-memory performance or the freedom to scale capacity with demand?